Not a bug - not a script problem - sourceforge issues

They say it is fixed. They asked for ip address of any "new" failures. There was a comment that there is an SVN specific protocol that is easier on their servers than https: It is probably worth investigating as well as some kind of throttling on mafia's end - limit number of simultaneous requests, introduce delay between etc.
 
Last edited:
Interesting. Certainly for mafia itself I use svn+ssh://, I seem to recall you can use svn:// if you aren't committing to mafia.
 
Do we want mafia to automatically convert https URLs to svn URLs on (first) script checkout? Repository URL is stored in .svn metadata.
Or can svn:// be forced on an operation-by-operation basis (in svnkit)?

How well does svnkit support svn:// ?
 
I'm inclined to run an experiment where the the repository is given as svn+ssh:// instead of https:// and see what happens. I would try it for a new (to a user) repository and then on an existing one. If the change is transparent then, since the json file belongs to mafia anyway, I'd update that file to use the preferred protocol for any SourceForge hosted scripts. That still leaves scripts that are not in the json but script authors or users who are bothered can just check out the script with the different protocol. I like that better than having mafia munge the input.
 
fronobulax said:
... That still leaves scripts that are not in the json but script authors or users who are bothered can just check out the script with the different protocol. ...
Click to expand...

Predicated on the "if the change is transparent" question: Is there a way to identify such (not-in-json) scripts among already checked out scripts? Is there a way of just changing protocols, or would existing scripts like that need to be removed and then added using the new protocol? What about the script forum posts that still use the https protocol?

Just trying to think about potential tasks ahead and persistence of recurring issues.
 
And....whatever Sourceforge did has come undone again. I tried running svn update and, after the first twenty, or so script checks, was blocked again.
 
fractalnavel said:
Is there a way of just changing protocols
Click to expand...

SVN supports a switch/relocate command, which just changes the URL that needs to be changed for that. Worst case, you can do it from command line (you might need to get command line tools that support the same version of local repository as mafia's copy of svnkit does; I think svnkit has commandline tools that can do that).
 
Magus_Prime said:
And....whatever Sourceforge did has come undone again. I tried running svn update and, after the first twenty, or so script checks, was blocked again.
Click to expand...

Well it worked for me this morning. Do you have a SourceForge account, do you know what your internal IP was when it failed and is it failing now?

I'm more than happy to change KoLmafia to be a good netizan but fundamentally this is SourceForge's problem and they have not been forthcoming (perhaps justifiably if this is part of an effort to reduce attack vectors) with specifics.

I am very much distracted by shiny objects and the local weather has cleared and there are lots of them outside. Nevertheless, if no one runs some tests and reports, I will spend some time determining whether Kolmafia supports svn-ssl and whether it recognizes svn-ssl and https protocols that point to the same project as actually being the same project. If that is the case then there is some low hanging fruit tat can be harvested over the weekend.
 
Hi Frono. I do have a Sourceforge account and reported it in the thread along with my IP address. The block expired after, approximately, an hour.

I agree that it's yet another frustrating change on the part of Sourceforge.
 
Interestingly, the reason I have always refused to enable automatic updating of all scripts every day is that it seemed like wasteful server hits for sourceforge/github/whoever to deal with, when plenty of scripts are never updated. I see why script writers would want to be sure that script users are regularly updating, but I can also understand why sourceforge would specifically not want this type of thing from random users.
 
I am still surrounded by shiny objects.

I have reached out to the SF engineer who worked the ticket in case he has other suggestions.

Introducing a delay between requests will definitely help. I'm OK with doing that, especially if the delay can be tweaked by the user. I will work on that momentarily.

Limiting the maximum number of simultaneous requests might also get done although I have not dug deeply into how I would do that.

I've done some playing with replacing https: with svn: I deleted a script and then checked it out with svn: It worked but the SVN version on disk is older than what my copy of TorotiseSVN wants to deal with and I'd like to understand why before I go too far.

I note that svn is Read Only and svn+ssh will commit. If all KoLmafia SVN operations on a "project" must use the same URL then changing the URL to svn might break KoLmafia's ability to commit. On the other hand a commit will require a user name and password and I don't know if/where KoLmafia is getting those so... Still more questions than answers.
 
fronobulax said:
I've done some playing with replacing https: with svn: I deleted a script and then checked it out with svn: It worked but the SVN version on disk is older than what my copy of TorotiseSVN wants to deal with and I'd like to understand why before I go too far.
Click to expand...

SVNKit probably needs updating :/
 
Actually, looks like you could probably just svn upgrade those, and it'd probably work just fine. Unless your version of SVN is sufficiently ahead of the version of SVNKit currently bundled with Mafia.
 
heeheehee said:
SVNKit probably needs updating :/
Click to expand...

Quite possibly. There is a new version circa May 2018 but I am not quite ready to look at the differences and incorporate it. Neither am I willing to change the build process to maven or something similar where we can just grab a jar file as a third party dependency. And I just noticed mafia uses some third party jars so maybe that is the way forward.
 
heeheehee said:
Actually, looks like you could probably just svn upgrade those, and it'd probably work just fine. Unless your version of SVN is sufficiently ahead of the version of SVNKit currently bundled with Mafia.
Click to expand...

I think I saw this before and Tortoise upgraded and mafia could handle it but I'm not in a position to try something and have it breal everything. Eventually...
 
OK.

I saw several 429 errors this morning so Sourceforge may be reporting a useful error message.

I decided the easiest thing was to use a ThreadPool to limit the requests. The attached patch does that and has worked for me. If anyone feels like reviewing it or trying it, please do so. I am especially concerned that I am swallowing errors that were at least reported before.

The pool size is set by the global variable svnThreadPoolSize. I don't expect users to really be messing with it so have not added it to a GUI.

I'll fix the extra space before a finally and commit in the next day or two.

I don't have a good programmatic fix for svn: vs https: yet. Not sure there will be one.

Patch.
View attachment SVNThrottle.patch
 
r18678

I moved the update all to a thread pool and made the pool size a preference (svnThreadPoolSize). The default value of 2 actually seems to run faster on my system than the old way. That's probably because update all is reusing threads rather than creating a whole bunch and then destroying them. I decided there was no reason to introduce additional delay, but if there is a reason to do so it is a simple change to the ExecutorService.

The recommendation from SourceForge was to use svn: instead of https: as the connection protocol because it makes fewer connections. However there are comments in the SVN code that imply changing the protocol is not going to work, even with the switch command. So the only safe way, AFAIK, to change protocol is to delete the local copy and then grab a new copy using the svn: protocol.

So I think we need to gradually migrate published URLs to use the svn protocol but the user will have to decide to delete and update. At some point in the future support for automation can be considered but it might not be needed.

Needless to say, if SourceForge is complaining about too many connections or something similar that is probably worthy of further investigation. I think one error updating a script will not stop the update all from trying the others, but I may be wrong and need to reconsider implementation.

But as poorly trained customer service reps who don't want to deal with your problem say "It is working for me" :-)
 
You must log in or register to reply here.
Back
Top