Not a bug - not a script problem - sourceforge issues

fronobulax

Developer
Staff member
They say it is fixed. They asked for ip address of any "new" failures. There was a comment that there is an SVN specific protocol that is easier on their servers than https: It is probably worth investigating as well as some kind of throttling on mafia's end - limit number of simultaneous requests, introduce delay between etc.
 
Last edited:

Darzil

Developer
Interesting. Certainly for mafia itself I use svn+ssh://, I seem to recall you can use svn:// if you aren't committing to mafia.
 

xKiv

Active member
Do we want mafia to automatically convert https URLs to svn URLs on (first) script checkout? Repository URL is stored in .svn metadata.
Or can svn:// be forced on an operation-by-operation basis (in svnkit)?

How well does svnkit support svn:// ?
 

fronobulax

Developer
Staff member
I'm inclined to run an experiment where the the repository is given as svn+ssh:// instead of https:// and see what happens. I would try it for a new (to a user) repository and then on an existing one. If the change is transparent then, since the json file belongs to mafia anyway, I'd update that file to use the preferred protocol for any SourceForge hosted scripts. That still leaves scripts that are not in the json but script authors or users who are bothered can just check out the script with the different protocol. I like that better than having mafia munge the input.
 
... That still leaves scripts that are not in the json but script authors or users who are bothered can just check out the script with the different protocol. ...

Predicated on the "if the change is transparent" question: Is there a way to identify such (not-in-json) scripts among already checked out scripts? Is there a way of just changing protocols, or would existing scripts like that need to be removed and then added using the new protocol? What about the script forum posts that still use the https protocol?

Just trying to think about potential tasks ahead and persistence of recurring issues.
 

Magus_Prime

Well-known member
And....whatever Sourceforge did has come undone again. I tried running svn update and, after the first twenty, or so script checks, was blocked again.
 

xKiv

Active member
Is there a way of just changing protocols

SVN supports a switch/relocate command, which just changes the URL that needs to be changed for that. Worst case, you can do it from command line (you might need to get command line tools that support the same version of local repository as mafia's copy of svnkit does; I think svnkit has commandline tools that can do that).
 

fronobulax

Developer
Staff member
And....whatever Sourceforge did has come undone again. I tried running svn update and, after the first twenty, or so script checks, was blocked again.

Well it worked for me this morning. Do you have a SourceForge account, do you know what your internal IP was when it failed and is it failing now?

I'm more than happy to change KoLmafia to be a good netizan but fundamentally this is SourceForge's problem and they have not been forthcoming (perhaps justifiably if this is part of an effort to reduce attack vectors) with specifics.

I am very much distracted by shiny objects and the local weather has cleared and there are lots of them outside. Nevertheless, if no one runs some tests and reports, I will spend some time determining whether Kolmafia supports svn-ssl and whether it recognizes svn-ssl and https protocols that point to the same project as actually being the same project. If that is the case then there is some low hanging fruit tat can be harvested over the weekend.
 

Magus_Prime

Well-known member
Hi Frono. I do have a Sourceforge account and reported it in the thread along with my IP address. The block expired after, approximately, an hour.

I agree that it's yet another frustrating change on the part of Sourceforge.
 

lostcalpolydude

Developer
Staff member
Interestingly, the reason I have always refused to enable automatic updating of all scripts every day is that it seemed like wasteful server hits for sourceforge/github/whoever to deal with, when plenty of scripts are never updated. I see why script writers would want to be sure that script users are regularly updating, but I can also understand why sourceforge would specifically not want this type of thing from random users.
 

fronobulax

Developer
Staff member
I am still surrounded by shiny objects.

I have reached out to the SF engineer who worked the ticket in case he has other suggestions.

Introducing a delay between requests will definitely help. I'm OK with doing that, especially if the delay can be tweaked by the user. I will work on that momentarily.

Limiting the maximum number of simultaneous requests might also get done although I have not dug deeply into how I would do that.

I've done some playing with replacing https: with svn: I deleted a script and then checked it out with svn: It worked but the SVN version on disk is older than what my copy of TorotiseSVN wants to deal with and I'd like to understand why before I go too far.

I note that svn is Read Only and svn+ssh will commit. If all KoLmafia SVN operations on a "project" must use the same URL then changing the URL to svn might break KoLmafia's ability to commit. On the other hand a commit will require a user name and password and I don't know if/where KoLmafia is getting those so... Still more questions than answers.
 

heeheehee

Developer
Staff member
I've done some playing with replacing https: with svn: I deleted a script and then checked it out with svn: It worked but the SVN version on disk is older than what my copy of TorotiseSVN wants to deal with and I'd like to understand why before I go too far.

SVNKit probably needs updating :/
 

heeheehee

Developer
Staff member
Actually, looks like you could probably just svn upgrade those, and it'd probably work just fine. Unless your version of SVN is sufficiently ahead of the version of SVNKit currently bundled with Mafia.
 

fronobulax

Developer
Staff member
SVNKit probably needs updating :/

Quite possibly. There is a new version circa May 2018 but I am not quite ready to look at the differences and incorporate it. Neither am I willing to change the build process to maven or something similar where we can just grab a jar file as a third party dependency. And I just noticed mafia uses some third party jars so maybe that is the way forward.
 

fronobulax

Developer
Staff member
Actually, looks like you could probably just svn upgrade those, and it'd probably work just fine. Unless your version of SVN is sufficiently ahead of the version of SVNKit currently bundled with Mafia.

I think I saw this before and Tortoise upgraded and mafia could handle it but I'm not in a position to try something and have it breal everything. Eventually...
 

fronobulax

Developer
Staff member
OK.

I saw several 429 errors this morning so Sourceforge may be reporting a useful error message.

I decided the easiest thing was to use a ThreadPool to limit the requests. The attached patch does that and has worked for me. If anyone feels like reviewing it or trying it, please do so. I am especially concerned that I am swallowing errors that were at least reported before.

The pool size is set by the global variable svnThreadPoolSize. I don't expect users to really be messing with it so have not added it to a GUI.

I'll fix the extra space before a finally and commit in the next day or two.

I don't have a good programmatic fix for svn: vs https: yet. Not sure there will be one.

Patch.
View attachment SVNThrottle.patch
 

fronobulax

Developer
Staff member
r18678

I moved the update all to a thread pool and made the pool size a preference (svnThreadPoolSize). The default value of 2 actually seems to run faster on my system than the old way. That's probably because update all is reusing threads rather than creating a whole bunch and then destroying them. I decided there was no reason to introduce additional delay, but if there is a reason to do so it is a simple change to the ExecutorService.

The recommendation from SourceForge was to use svn: instead of https: as the connection protocol because it makes fewer connections. However there are comments in the SVN code that imply changing the protocol is not going to work, even with the switch command. So the only safe way, AFAIK, to change protocol is to delete the local copy and then grab a new copy using the svn: protocol.

So I think we need to gradually migrate published URLs to use the svn protocol but the user will have to decide to delete and update. At some point in the future support for automation can be considered but it might not be needed.

Needless to say, if SourceForge is complaining about too many connections or something similar that is probably worthy of further investigation. I think one error updating a script will not stop the update all from trying the others, but I may be wrong and need to reconsider implementation.

But as poorly trained customer service reps who don't want to deal with your problem say "It is working for me" :)
 
Top