Feature RFC: script dependency remapping

[Rinn]

To prepare for github dropping svn support I've been thinking about how to deal with repos that don't update and the dependencies will no longer be able to be resolved and how we could work around that. I'd like some feedback before I make any additional changes.

I was thinking to add something like a property "remapScriptDependency" (or a data file? a property could get incredibly unwieldy due to the length), which is a comma separated list of kvp pairs that will be replaced when trying to install dependencies if one of the dependencies matches exactly which would allow you to use git instead or even remap it to a different repo entirely if the script needs to be forked.

For example https://github.com/Ezandora/Genie depends on an svn install https://github.com/Ezandora/Choice-Override/branches/Release/ which could be installed with git, and with a remapping I can change it to instead install the dependency with git:

> set remapScriptDependency=https://github.com/Ezandora/Choice-Override/branches/Release/=github Ezandora/Choice-Override Release

remapScriptDependency => https://github.com/Ezandora/Choice-Override/branches/Release/=github Ezandora/Choice-Override Release

> git checkout Ezandora/Genie Release

Starting
remote: Enumerating objects
Receiving objects
Resolving deltas
Updating references
Checking out files
Copying: scripts/genie.ash
Copying: images/genie/genie_shaq_colour.png
Copying: images/genie/genie_shaq.png
Copying: relay/genie.js
Copying: relay/choice.1267.ash
Installing dependencies
Starting
remote: Enumerating objects
remote: Counting objects
remote: Compressing objects
Receiving objects
Resolving deltas
Updating references
Checking out files
Copying: relay/choice.example.ash
Copying: relay/choice.ash
Cloned project Ezandora-Choice-Override-Release
Cloned project Ezandora-Genie-Release

 

[Ryo_Sangnoir]

Folk should fork the repo that doesn't work and users should delete the broken version and install the fork instead. If this requires recursive forking so be it.

I did this for Genie; you can install "midgleyc/Genie". I have a PR up for the main version if Ezan ever returns.

 

[Rinn]

fair enough

 

[fronobulax]

I'm perhaps not seeing the problem.

Right now we specify both a repository and a protocol. The user friendly option would be just to specify a script and let KoLmafia handle repository and protocol. We aren't there yet but my knee jerk reaction to deprecating SVN access to GitHub would be to try and access GitHub with git and if that fails escalate the issue to the user. If that happens while resolving a dependency then the script writer really needs to address where the dependencies need to come from.

There are already tools that will help identify svn and git versions of what are probably the same script. What is lacking is an automated response to the situation when detected.

Tangentially, SourceForge has been pretty aggressive telling me about tools that will mirror my GitHub repository to SourceForge. It is allegedly a one click operation. With volunteers or something like LASS that might also be a simple solution.

 

[Rinn]

If the option was to migrate something to sourceforge to keep it svn or fork it on github and fix it, i'd never chose to put anything back on sourceforge.

 

[fronobulax]

If the option was to migrate something to sourceforge to keep it svn or fork it on github and fix it, i'd never chose to put anything back on sourceforge.

Agreed.

I would be perfectly fine in declaring that all scripts hosted on GitHub were going to be updated via git and allowing KoLmafia to use git no matter was specified.

Having a git version and a SVN version of the same script does not even have to be a problem. There are versions in two local repositories and the most recently updated one is likely to be in scripts/ . When GitHub is updated that update will replace what is in scripts/

 

[Ryo_Sangnoir]

I think it might be worth doing something when GitHub removes SVN support, but only to the extent of telling the user that SVN update will fail for (scripts X, Y, Z), skipping update attempts, and telling the user to install using git if they still want updates (if updates are possible).

I don't want to do anything complicated like installing a different script to the one the user has requested to install, because this involves work both on our side (for the feature) and for every user that wants to install said script (configuring it), whereas forking a script just requires work from one user (plus advertisement).

Automatic redirects of SVN to git where possible runs into issues of edge cases: we can't necessarily figure out the correct repo / branch from the installed repo. We could do "good enough" but again I'd rather just get the affected user to do it.

 

[fronobulax]

Automatic redirects of SVN to git where possible runs into issues of edge cases: we can't necessarily figure out the correct repo / branch from the installed repo. We could do "good enough" but again I'd rather just get the affected user to do it.

Keep hammering that point. I will eventually remember that it is not just as simple as accessing the same "url" with a different protocol and stop making the suggestion. Thanks.

 

[MCroft]

I agree with Chris. Error messages and warnings are appropriate. Also nudges.

There's not a lot we can do if they're installing a script from the gCLI. At best we could be clippy: "It looks like you're trying to use SVN to install a script from GitHub. Would you like help forking the project and installing via git? Too bad, I'm an autoresponder, but here's a forum thread."

The ones we list in the Script Manager via the svnrepo.json file we may want to plan to take over and update or find an owner for or just remove. We don't want the users to see these (currently) 26 scripts and try to install them.

I took the svnrepo.json file and stripped out all "type":"git" and source forge hosted projects. There are 26, some of which aren't going to be updated. But active/ LASS developers should definitely have their scripts that are in the repo modernized. Autoscend, for instance, is listed as an SVN project, but they recommend installing that via GIT in their support channel.

 

[Ryo_Sangnoir]

I've gone through and filed https://github.com/kolmafia/kolmafia/pull/1693, converting where possible.

The unconverted are:

CONSUME needs a manifest file (PR from taltamir)
sl_ascend would need a manifest file but probably wants removing (with autoscend the recommended replacement)

FantasyRealm has an SVN dep on Gain (no PR)
PirateRealm needs a manifest file, and also has an SVN dep on Gain (PR from Veracity)
Voting-Booth needs a manifest file (PR from midgleyc)
Bastille has an SVN dep on Choice-Override (PR from midgleyc)
Genie has an SVN dep on Choice-Override (PR from midgleyc)
Source-Terminal-GUI has an SVN dep on Choice-Override (PR from Veracity)

 

[Veracity]

I have other PRs for Ezandora scripts:

Latte needs a manifest file, has an SVN dep on Choice-Override, and has a data error
Mummery has an SVN dep on Choice-Override

I could have sworn I did FantasyRealm, but apparently not.

 

[fronobulax]

My memory is not what it used to be or never was as good as I think I remember, but I think there is an unresolved issue when converting from svn to git. So there may be some idiot proof steps, or better yet automation, out there that I just have forgotten about.

If a user has local changes that are managed/respected by a svn update/merge what does the user have to do so they end up with a git repository that has local and respects local changes?

 

[Veracity]

Here is a comment from an hour ago from Ryo_Sangnoir's PR documenting how to do this with "excavator":

To migrate your local changes (all file paths are relative to Mafia root):

• you currently have an excavator folder in svn/. It has a scripts/ folder, and files in this folder are copied to the main scripts/ folder.
• you have local changes in the svn/gausie-excavator folder
• you should checkout the git version of excavator in the Mafia CLI: git checkout gausie/excavator. This will download files to git/gausie-excavator and override the version in scripts/
• you should copy your local changes from svn/gausie-excavator. Just dump the entire svn/gausie-excavator/scripts folder on top of git/gausie-excavator/scripts
• run git sync excavator in the Mafia CLI. Confirm that your scripts/ folder looks the way you expect and that excavator behaves the way you expect
• run svn delete excavator to remove the SVN version

I think you have to run "git sync excavator" again, since "svn delete excavator" will remove the files from scripts, and you want to restore them (again) from git. I am also not completely convinced that that worked for me last time I tried it

 

[fronobulax]

Here is a comment from an hour ago from Ryo_Sangnoir's PR documenting how to do this with "excavator":

Thank you. The curse/benefit of having conversations in two places at once.

 

[xKiv]

I would also recommend to first explicitly update the svn version, in case there was a new commit since the last update. If you copy old version of files from svn/ to git/ you are basically saying that you have a new local change that reverts them. That's not going to help with clean, conflictless updates in the future. (only applicable while the svn repo is still accessible, though)
(or possibly run a merge tool between svn/ and git/, and accept only the changes that you know are yours)