Sourceforge downtime -- discussion of alternatives

heeheehee

Developer
Staff member
Yeah, I remember us discussing this at the time. It was opt-in, and I remember thinking that if I ever used an open source project that did opt-in, I'd immediately switch to a competing open source project, but until and unless Sourceforge somehow required projects to use their build tools to insert extra stuff not controlled by the project itself into binaries, it was completely irrelevant to THIS project.

That opinion stands. I don't see how their "anti-consumer practices" have anything to do with us using them as a svn server for source code. Doubly so for ASH scripts which will don't even HAVE a "binary" form to download.

Now, if they start to require all source code hosted on their site to use the GPL license - rather than Berkeley (as we do) or Apache, or the non-virus license of the project's choice - THAT would be a reason to cease using them for SVN.

Given the recent (and current) Sourceforge downtime, I'd like to resume this conversation, but in a separate thread. I don't have much faith in a service that reports updates more or less every twelve hours, and only has part of the problem addressed more than 48 hours after the initial report.

One alternative that's been discussed is for fewyn to set up a svn server on the same server that hosts kolmafia.us.

I've considered trying to setup an SVN server for usage here but I've yet to find a software package which would make managing them not a pain in the rear. If anyone has seen any I'd be open to giving it a try.

Some googling revealed the following options that provide a web interface for SVN repository management:
  • VisualSVN; Windows-only.
  • SCM-Manager.
  • Several other projects hosted on Sourceforge, which, surprise, are currently unavailable :(

SCM-Manager looks like it'd fit our needs rather nicely. I literally downloaded the package, unzipped the tarball, and ran bin/scm-server, and I had a server running on localhost:8080 in a matter of seconds. The getting started page may also be of some use (namely, initial user / password).
 

Veracity

Developer
Staff member
I, too, am extremely unimpressed with sourceforge's handling of its current issue. And given that they are in "disaster recovery mode" (don't remember where I read that), I am even wondering if they will succeed in recovering all of the projects they host. Ours, in particular. The project page and the download sections are back, but all of the developer areas - the source code repositories, in particular - continue to be unavailable.

(Perhaps I am extra suspicious, having gone through the ezboard meltdown, years ago. I had a private message board on that host, and it was adequate for my needs - and easy to administer - but when they lost all of the message boards on the system - and all of their backups - due to what they claimed was a "hacker" (although I have heard other stories, like "disgruntled former employee"), they never did recover so much as a single thread from my message board, and I ended up installing phpBB on my own server and running it myself. I am still depending on my web host - Network Solutions - to keep backups , but I hope/expect that they are more reliable. I think you have to actually control the hardware yourself and do your own backups, if you don't want to trust the reliability of others.)

This has also inspired me to look at recent (June) discussions about how sourceforge is taking "abandoned" projects and pointing the download page to their own downloads, complete with bundled malware. So, yeah - I have a lot more concern about their "anti-consumer policies" than I did before. Too bad the person who originally commented on them here never followed up with some links to support their assertions at the time - even though I specifically asked for some links!

So, I am all for moving the repository. There seem to be several general options:

1) Move to a different public host, like github
2) Install it on a private site hosted on a public web server - like mine, assuming I can install an SVN server there
3) Install it on a private site hosted on a private server - like fewyn's, right here.

For option 1, my big question is: what is to stop it from going the way of sourceforge in the future?
For option 2, what happens when the person who owns the site gets bored?
For option 3, ditto.

That is not intended to be mean; after 11 years, I am very tired of spending so much time on KoL (and KoLmafia, alas), and want to scale WAY back. I wouldn't want to commit to owning and maintaining a site devoted to KoL for as long as KoL exists, long after I have otherwise left the game. Consider weas and Faxbot - although weas was driven from the game by KoL admins, from what I understand.

How does fewyn feel about KoL and his ability to commit to spending time (and money!) providing and maintaining this server for the benefit of the Kolmafia community?

Again, that is not intended to be mean, but it applies to any private place we decide to host an SVN repository; on sourceforge, I made lost an admin, so even after I go, he could carry on - and he could assign a new admin if/when he gets tired. But if the admin owns the hardware, it's a different level of commitment.
 

fronobulax

Developer
Staff member
For clarity are we talking about migrating the entire KoLmafia project or just the scripts that KoLmafia will automatically update?

I have been playing KoL for almost 11 years and been associated with KoLmafia for about 10. I have seen numerous individuals come, make significant contributions and then leave. Having to deal with SourceForge's down time sucks but over 10 years it has been pretty reliable. I'm not sure if we'd even be here having this conversation if KoLmafia itself hadn't remained at SourceForge.

If we are only talking about scripts I can't make a strong argument either way. But I will note the developers have been unwilling to make GitHub an unsupported repository even though the problems updating Guide were numerous. Unless there is a willingness to mandate that all scripts must be hosted at a single repository I am not sure that moving will do anything more than reduce the frequency because some scripts will still be a Sourceforge.

I could argue that a constructive response to this would be to evaluate SVN support for script authors. One option would be to remove all SVN support from ash and the gCLI. Mafia would be responsible for updating all registered scripts on login and there would be no preference because mafia would detect and handle errors. An alternative would be to provide enough SVN status information that the script author who needed to manage the process had the error and status reporting necessary to do it properly.
 

heeheehee

Developer
Staff member
I did find some more information regarding the current Sourceforge outage, but it really shouldn't be this difficult to figure out what is going on.
Robert Galoppini said:
Below some facts about the actual situation.

1. Fault with storage platform was due to a platform bug; we're working on data validation and where necessary restoring from backups 2. Operations staff worked overnight and are continuing work on service restoral. Operations staffing has been split between rapid recovery (expediting restoral) and data validation. 3. Slashdot site service was restored. 4. Work is in-progress to restore SF site, download data, mailing lists, and project web and every other service.

[Disclosure: I do work for SourceForge]
(https://news.ycombinator.com/item?id=9901999)

For reference, here are some links regarding Sourceforge's anti-consumer (and, in fact, anti-developer) practices (and a writeup, too).

For clarity are we talking about migrating the entire KoLmafia project or just the scripts that KoLmafia will automatically update?
I started this thread under the assumption that we'd consider migrating the entire project, since Sourceforge's current status means that any KoLmafia commits will have to wait.

I have been playing KoL for almost 11 years and been associated with KoLmafia for about 10. I have seen numerous individuals come, make significant contributions and then leave. Having to deal with SourceForge's down time sucks but over 10 years it has been pretty reliable. I'm not sure if we'd even be here having this conversation if KoLmafia itself hadn't remained at SourceForge.

That may be the case, and I don't have anything bad to say about the past 10.5 years of Sourceforge as a whole, but just considering the past three months are problematic, especially since there are better hosting services these days.

I'm not saying that Github, for instance, has a flawless track record in terms of uptime -- on the contrary, it's been spotty for reasons from getting DDoSed by the Chinese government to emergency maintenance. However, both github and bitbucket have dedicated status trackers, as opposed to just a twitter feed.
 
The issue I have is that we have scripts hosted on Sourceforge that have not been maintained in some time but are still listed in the Script manager and there is no one who actively maintains them and a current active maintainer does not have access so these can't be moved to a new SVN.
 

fewyn

Administrator
Staff member
I personally would rather see us using Github. I can setup private SVN repos here at kolmafia.us (and I don't plan on going anywhere anytime soon I've been here what... 8 years now hosting the site?) The problem with hosting them myself is I've yet to find a good solution to easily manage users/projects as good as Github etc. If anyone knows of one I'd be more than willing to set it up.
 
I personally would rather see us using Github. I can setup private SVN repos here at kolmafia.us (and I don't plan on going anywhere anytime soon I've been here what... 8 years now hosting the site?) The problem with hosting them myself is I've yet to find a good solution to easily manage users/projects as good as Github etc. If anyone knows of one I'd be more than willing to set it up.


What OS are you running? As for Atlassisan would every script have to make a request? If that is the case I would think github would be the best choice. I would also recommend appointing a community manager per se that has access to all scripts so in case they needs to be moved again there would be someone active to move them.
 

fewyn

Administrator
Staff member
I'm not sure, we might be able to create different repos and such. I'll look into it.
 

hallehoopma

New member
As just a consumer of mafia and of many scripts through the years, I can only comment about the user end. As a used, I do not install anything that forces me to Sourcefourge for installation. I have been tricked, and it screwed up my computer in the past. Quick downloads from what had always been a decent repository, had me clicking too fast to realize what was about to happen, and I could not stop it in time. If I am forced to go there, I do without the script/program/whatever.

This probably is no help for you guys, but before I knew about their horrible developer abuse, this was a site I just stayed away from. Not supporting them in any way is a nice bonus, but I do not get a vote here. I just wanted to say that I thank you all for doing such an awesome job, for being so consistent, for what must be a massive amount of time. I would not play KoL without mafia, so I truly thank you and hope that nobody has to get totally burnt out.

Scale back for sure, but having left for years once, just splitting will leave a tiny hole in your heart until you return. ;)

Thank you all again - Halle
 

fewyn

Administrator
Staff member
With Bitbucket we can setup multiple repositories one for each script (I can setup several people to be able to add them including myself).
 

DaveK

New member
I'd recommend switching to github:

i - It's big and likely to be around for some time. Obviously any firm can go bust (or just turn rubbish, like SF) but GH has a lot of buy-in so it's as good a bet for the longterm as any other, and probably a better one than a less-used service like Atlassian.
ii - No worry about backups if the service does go down, because with git your local working copy can be a full repo copy with all history and everything. This also makes any future migration easy.
iii - GH provides SVN-over-http access, so the existing code in mafia for fetching and updating scripts can stay the same even though the repo switches to git.

So yeah, I'm with option 1. People need to be able to stand down without worrying about how the community can carry on without their resources, which rules out a private server to me. Maybe we'll have to migrate again in another decade or so, but that doesn't seem frequent enough to worry about to me, and migration could be a lot easier in future if we're using a DCVS like git.
 

lostcalpolydude

Developer
Staff member
iii - GH provides SVN-over-http access, so the existing code in mafia for fetching and updating scripts can stay the same even though the repo switches to git.

I guess you aren't aware of the issue where github sometimes doesn't report an updated revision number, so people report already-fixed issues in scripts because mafia believes the script is fully updated. I don't know whether it's an issue with github or with our SVN package, but until that can be resolved (roippi is the one who added that support to begin with, and he doesn't seem to have that much time to devote to this now) it's a bit of an issue. Enough of an issue that github scripts don't get included in the Script Manager.
 

roippi

Developer
Here are the alternatives, as I see it.

1) stay on sourceforge, and stay on SVN.
2) Migrate to native git (including the library within mafia itself), force users/scripters to change over, and migrate to some other host. GitHub, bitbucket, whatever.

That's it. SVN-over-git is a non-starter, and there are simply no other hosting alternatives as far as I'm concerned.

My stance on the anti-whatever practices of sourceforge remain a resounding "meh." To be clear, these practices do not affect direct SVN repo access, and will never do so. And if the argument is that we are in some way supporting sourceforge... ah, no. We are using their free bandwidth and storage space (a surprising amount, given how SVN works) and giving nothing in return. Kolmafia and associated scripts are decidedly a net negative in SF's pocketbooks.

If the objections are that the SF platform is unstable (either for tech reasons or for business reasons) and we need to GTFO while we still can, fine. But that's going to be a painful migration so we all need to be on board.
 

Darzil

Developer
Sourceforge has definitely had periods of instability before this in recent weeks. It does make me wonder if they were early signs of a failing environment, and that maybe it'll be more stable once this is resolved. That is not uncommon with storage issues.
 

fronobulax

Developer
Staff member
I was surprised to learn that "Total Inability to Support Usual Performance" had been around a lot longer than computers.

I find the reddit post very interesting having worked with storage and consultants from an Extremely Massive Company. I'm not sure I'd enjoy repeating the experience.

I expect the question of resolving hosting for the project will be resolved by the devs. It doesn't matter where it is hosted if no one is willing to maintain it there. So far I'm not seeing much of a consensus for moving and mercifully no one has thrown a temper tantrum and said they will stop continuation of their contributions unless their hosts of choice is used.

The question of where to host scripts impacts a larger community and there are things that could and probably should be done differently once SourceForge is back.
 
Top