Sourceforge downtime -- discussion of alternatives

heeheehee

Developer
Staff member
Darzil: downloading from svn:// provides you simply with read-only access to the repository. You'd need to checkout with svn+ssh://darzil@... (and have your private key handy) in order to check out with rw permissions. Or, as you noted, you can just check out with https.
 

Veracity

Developer
Staff member
On one hand, I don't know how I feel about including yet another SCM library in Mafia, because the problem then becomes where we draw the line. Do we include a Mercurial library? How about Darcs? Or Bazaar? Hell, there's even a version control system called Veracity.
We have a lot of existing SVN-controlled scripts. We also have a few SVN-over-GIT scripts - Ezandora's Guide being the prime example - which have had repeated VC issues. It's those latter scripts that I'm thinking of; having native GIT support would (hopefully) fix the source control reliability issues. (Or, perhaps, the newer version of SVN we now support will make that work better, if the github hosted scripts can use that. I don't know.)

If we have an "svn" command to let you do svn-specific things, a "git" command to let you do git-specific things - and have a "vcs" or "script" command, say, to be the front end to either, depending on what the individual script needs - it would not be hard to add other VCSs, if needed - but I would be surprised if there was much call for anything else; I think people will be satisfied to use SVN at sourceforge or GIT at github

On the other hand, it could be an interesting project, but it'd probably take some significant effort to implement fully.
I think you are right. I have not looked at all at what we expose of SVN via CLI or ASH; I know we have a script registry - which assumes SVN - and that you can type "svn update" to update all of your registered scripts - and that is, literally, the only thing I ever use, other than following the published instructions for particular scripts that say "type this svn checkout command into the gCLI to install this script."

It would be a project to design a "vcs" or "script" command; perhaps you would use the "svn" or "git" command to actually install each individual script, but would use "script update" or "script synch" or whatever to do "operate over every installed script" operations.

A project that I will not be doing, although if somebody thinks it would be "fun", I'd certainly encourage the discussion.

For what it's worth, I'm currently using git-svn, which allows me to fetch SVN updates into a local git repository and even make commits from there. In fact, this is how hola mirrored Mafia's source code to github, apparently. This... could be a decent way to grasp the learning curve before committing to migrating, I guess.
Yeah, I saw that - and it would probably be adequate for our needs; the other tool preserves SVN branches and tags - but we don't use that. In retrospect, it would have been reasonable to tag all of the commits that go into a release with the release number, or something, but I don't think we really care; we have the "all development goes onto the trunk" super-simple approach to commits.

But, whether or not we move KoLmafia source to git is orthogonal to whether (and how) KoLmafia supports git for its script registry.
 

xKiv

Active member
Actually, as far as I can tell, the recent fun actually involved Sourceforge losing a few commits from shortly before the outage; Lost's commits got added to the centralized repository as the new "ground truth" for r16064-5 or whatever, which led to all sorts of checksum issues.

And how would that look like with git?

I am guessing:
1) everyone has something checked out
2) central repository is restored, is missing some commits
3) somebody who didn't yet have the missing commits makes a new commit (and pushes it to central)
4) somebody else who *did* have those commits makes a new commit (and pushes)
At this point, I think git will either say "you are not at head, you have to fetch/update (pull) first"
5) you do that
6) then you commit (and push), which will also push all the missing commits back into central repository
7) then everyone will get everything again

(this is possible because git a) doesn't insist on labelling commits with sequential numbers, but rather generates "unique" identification by hashin the parent commit and this commit's content, ii) keeps all history locally, and 3) doesn't insist that your local repository is always based on a snapshot of something that already exists in a central repository)

(Or, perhaps, the newer version of SVN we now support will make that work better, if the github hosted scripts can use that. I don't know.)

I suspect this to not be the case, because the root of the problem was an error while retrieving info about a particular revision from the remote repository (even when doing just svn log). A revision that had previously checked out without an error. And seems to be error-free now.
 

fronobulax

Developer
Staff member
Speaking about KoLmafia as a project, there are two things I'd like to understand in The World According To Git. The first question is how does Git make one file available to anyone on the internet who wants it? This is the analog to downloading the "latest release" from SourceForge. Second will Git support some kind of continuous build process so that a jar file is always generated (and "published") after each commit or will we continue to look to fenwyn (Thank You) for this?

Git seems to support a lot of collaboration and reporting features. Should bug reporting and dev collaboration also migrate if migrating was the decision?
 

heeheehee

Developer
Staff member
And how would that look like with git?

I am guessing:
1) everyone has something checked out
2) central repository is restored, is missing some commits
3) somebody who didn't yet have the missing commits makes a new commit (and pushes it to central)
4) somebody else who *did* have those commits makes a new commit (and pushes)
At this point, I think git will either say "you are not at head, you have to fetch/update (pull) first"
5) you do that
At this point, pulling would try to automatically merge (or, if you pull --rebase, it'll rewind your changes, then try to apply them to head). If it fails, then it's up to you to manually merge. Pretty much the same workflow as with svn if you tried to commit without updating first. No checksum mismatches that you get in svn, since revisions aren't structured in a list so much as a dag.
6) then you commit (and push), which will also push all the missing commits back into central repository
7) then everyone will get everything again

(this is possible because git a) doesn't insist on labelling commits with sequential numbers, but rather generates "unique" identification by hashin the parent commit and this commit's content, ii) keeps all history locally, and 3) doesn't insist that your local repository is always based on a snapshot of something that already exists in a central repository)
Seems more or less correct.

I suspect this to not be the case, because the root of the problem was an error while retrieving info about a particular revision from the remote repository (even when doing just svn log). A revision that had previously checked out without an error. And seems to be error-free now.
Part of the problem was that somehow SVNKit was corrupting the repository. Ezandora mentioned that a test case she has no longer fails with the updated version in Mafia.

Speaking about KoLmafia as a project, there are two things I'd like to understand in The World According To Git. The first question is how does Git make one file available to anyone on the internet who wants it? This is the analog to downloading the "latest release" from SourceForge.
git has no such notion of this. On the other hand, github does allow for you to provide downloads for certain (typically tagged) releases (see: https://github.com/houeland/kolproxy/releases for an example of that)

Second will Git support some kind of continuous build process so that a jar file is always generated (and "published") after each commit or will we continue to look to fenwyn (Thank You) for this?
You could use a post-commit hook that does all this. Apparently we could even be doing that with subversion.

Git seems to support a lot of collaboration and reporting features. Should bug reporting and dev collaboration also migrate if migrating was the decision?
I'm not entirely sure I understand the question. Are you thinking about pull requests on github?
 

fronobulax

Developer
Staff member
I'm not entirely sure I understand the question. Are you thinking about pull requests on github?

My original question might have been clearer if I had said GitHub. At this point in my thinking git and GitHub are pretty much the same thing. There is no benefit to me to use git to access SourceForge and no one has proposed using git without GitHub although I understand that is possible.

https://github.com/ said:
Great collaboration starts with communication.

Review changes, comment on lines of code, report issues, and plan the future of your project with discussion tools.
 

xKiv

Active member
Part of the problem was that somehow SVNKit was corrupting the repository. Ezandora mentioned that a test case she has no longer fails with the updated version in Mafia.

How? Did anyone use svnkit to *commit* to github, or did github's svn support have a bug that corrupts the repository on read-only operations?
 

fronobulax

Developer
Staff member
How? Did anyone use svnkit to *commit* to github, or did github's svn support have a bug that corrupts the repository on read-only operations?

I observed corrupt local repositories when then only code that had managed the local copy was mafia's update which used svnkit. Someone smarter than me traced it back to a known bug in git's support for svn. If the problem has gone away it is because git has fixed their bug or svnkit has made a workaround that is now included in our code base.
 

Ezandora

Member
I observed corrupt local repositories when then only code that had managed the local copy was mafia's update which used svnkit. Someone smarter than me traced it back to a known bug in git's support for svn. If the problem has gone away it is because git has fixed their bug or svnkit has made a workaround that is now included in our code base.
To further expand on this, I have a test case where I update the same file repeatedly until "svn update" stops working. The only change is updating the version string.

Some results from testing today:

Before SVNKit upgrade, r16070:
Code:
simpleSvnUpdate=true, same as the default.

r16070: (almost immediately before the SVN update bug)
Checked out 3.7. (svn checkout https://github.com/Ezandora/Guide/branches/SVNTest/)
Updated to 3.7.1, 3.7.2. Tried to update to 3.7.3, failed.

r16070:
Deleted SVNTest svn project in mafia.
Checked out 3.8. Updated to 3.8.1, 3.8.2. Tried to update to 3.8.3, failed.

r16070:
Deleted SVNTest svn project in mafia.
Checked out 3.9. Updated to 3.9.1, 3.9.2. Tried to update to 3.9.3, failed.

r16070:
Deleted SVNTest svn project in mafia.
Checked out 3.10. Updated to 3.10.1, 3.10.2. Tried to update to 3.10.3, failed.
Deleted SVNTest svn project in mafia.

There's the update bug. I'm not sure if these test cases are representative - I vaguely remember updates failing at different intervals than the third one.

After SVNKit upgrade, r16073:
Code:
r16073:
No projects currently installed with SVN.
Checked out 3.11. Updated to 3.11.1, 3.11.2, 3.11.3, 3.11.4, 3.11.5, 3.11.6, 3.11.7, 3.11.8, 3.11.9, 3.11.10, 3.11.11.
Updated the project to 3.11.12, but did not try to update in mafia to see if it handles skipped versions.
Updated the project to 3.11.13, but did not try to update in mafia.
Updated to 3.11.14, 3.11.15. Still no errors.

No sign of the update bug for fifteen revisions, thirteen of which mafia updated to.

What happens if we combine the two?
Code:
r16070
------
(no SVN projects installed)
Checked out 3.12. Updated to 3.12.1, 3.12.2. Failed to update to 3.12.3.
Switch to r16073. Failed to update to 3.12.3.
The local installation is still corrupt, so it won't update. So, users will probably have to re-checkout my guide again, once they're on newer copies of mafia.

This test isn't conclusive. Each version was a single line of code changed, without large blocks of additions and removals typical to updates. But, improvement!
 

heeheehee

Developer
Staff member
Looks like I can still commit, too. It was definitely down for some number of additional hours beyond the scheduled (?? You'd think that they'd send out emails or something!) 1-hour maintenance, though.
 
Even with SF up & running again, this discussion is still probably worth pursuing - maybe a new thread? Issues: (a) source hosting, (b) source control technology (SVN v. Git, centralized v. distributed). At the very least, a fail-over strategy would be good. The strategy used during this outage appears to have been "wait & see", which may be appropriate for short initial period (whatever that is), but it would be good to have an idea of what to do beyond that, other than talk about it some more when it happens.

Or not, that would be ok. The main thing is to say something about it up front.

For starters, maybe having an official mirror on a separate host would be a good thing, to be used as a starting point if the primary system is gone, even if no other plans beyond that exist.
 
Top