roippi
Developer
Interesting thought. I'll ponder. Been a while since I thought about why the system was designed the way it is.
Mafia SVN for Scripts?
- Thread starter bumcheekcity
- Start date
Interesting thought. I'll ponder. Been a while since I thought about why the system was designed the way it is.
roippi
Developer
So the proposal is to expand the /scripts namespace to include any projects that contain a /scripts folder, likewise for /relay, /ccs... And then slowly phase out the "local copy" concept. That transition certainly would be... bumpy.
I think the reason we never originally contemplated this as an option was that the svn system was so unfamiliar to everyone. The svn system had to "just work" with the legacy system of copying files into your /scripts folder. It wasn't clear that users would be comfortable not knowing where a given relay override was coming from, for example; it could be residing in any one of their svn-installed projects.
Now, the system is more mature, people are more comfortable with it, and a GUI manager is on the horizon. Maybe it's worth transitioning, I can see pros and cons. I'll let interested parties discuss.
I think the reason we never originally contemplated this as an option was that the svn system was so unfamiliar to everyone. The svn system had to "just work" with the legacy system of copying files into your /scripts folder. It wasn't clear that users would be comfortable not knowing where a given relay override was coming from, for example; it could be residing in any one of their svn-installed projects.
Now, the system is more mature, people are more comfortable with it, and a GUI manager is on the horizon. Maybe it's worth transitioning, I can see pros and cons. I'll let interested parties discuss.
Theraze
Active member
The way things currently work, if a user moves an SVN file into a new folder to sort things, does it keep updating? I thought that it did, as long as the name itself isn't broken... this would remove that sortable option for those who want to remove items from the Scripts list but not use the MRU version.
Bale
Minion
Yes? It depends on how I interpret your question since your terminology is ambiguous. If you are referring to moving the local copy in the /scripts directory, then it is true that if you move it to a subdirectory it will continue to update since mafia will recognize as the same script as an identically named working copy.
Excellent point. I do not want to lose the ability to sort my scripts menu. Perhaps if Roippi were to also take on that other project he suggested to make the script menu organizable without actually modifying the directory tree and moving files, but that sounds like a lot of work.
Theraze
Active member
Yes, I believe you understood me correctly. 
I was trying to distinguish between the person who makes an OCD folder and puts the files in there, and someone who renames the script itself to OCD or 1_OCD. In the first case it should still work because the namespace matcher finds the script, but in the second case the script is no longer the 'same' as its name doesn't match anymore.
I was trying to distinguish between the person who makes an OCD folder and puts the files in there, and someone who renames the script itself to OCD or 1_OCD. In the first case it should still work because the namespace matcher finds the script, but in the second case the script is no longer the 'same' as its name doesn't match anymore.
bleary
Member
alternative to internal svn
I decided that Mafia's internal svn stuff just wasn't going to work for me, so I decided to start handling svn updates of scripts outside of mafia, using command-line versions of svn and GNU stow (http://www.gnu.org/software/stow/), which maintains libraries of symlinks. I svn checkout each script pacakge into a /packages subdirectory under the mafia library directory, and point stow at that in order to symlink the files where they ought to be elsewhere in the mafia directory. This way there's only a single copy of each script and library file, I have access to all of my command-line svn tools to make diffs, update, debug, and resolve conflicts, and I have complete control over when/how each script updates and I can even read commit logs or revert to earlier versions if I need to. I edited the shell script I run to update mafia to handle updating the scripts under svn control. I found it desirable to edit a few scripts to turn off checks for updated versions.
To make this work, I had to disable part of RelayRequest.java in order for the relay server to serve symlinked files properly.
I decided that Mafia's internal svn stuff just wasn't going to work for me, so I decided to start handling svn updates of scripts outside of mafia, using command-line versions of svn and GNU stow (http://www.gnu.org/software/stow/), which maintains libraries of symlinks. I svn checkout each script pacakge into a /packages subdirectory under the mafia library directory, and point stow at that in order to symlink the files where they ought to be elsewhere in the mafia directory. This way there's only a single copy of each script and library file, I have access to all of my command-line svn tools to make diffs, update, debug, and resolve conflicts, and I have complete control over when/how each script updates and I can even read commit logs or revert to earlier versions if I need to. I edited the shell script I run to update mafia to handle updating the scripts under svn control. I found it desirable to edit a few scripts to turn off checks for updated versions.
To make this work, I had to disable part of RelayRequest.java in order for the relay server to serve symlinked files properly.
Code:
// Make sure that the file is actually in the relay directory
// try
// {
// String overridePath = override.getCanonicalPath();
// String relayPath = KoLConstants.RELAY_LOCATION.getCanonicalPath();
// if ( !overridePath.startsWith( relayPath ) )
// {
// this.sendNotFound();
// return;
// }
// }
// catch ( IOException e )
// {
// }
Code:
#!/bin/sh
cd ~/Documents/Projects/kol/kolmafia
svn log -r HEAD:BASE
svn update
JAVA_HOME=/Library/Java/Home ant
cp dist/KoLmafia-*.jar /Applications/KoLmafia.app/Contents/Resources/Java/KoLmafia.jar
echo Updating packages....
echo
cd ~/Library/Application\ Support/KoLmafia/packages/
svn update *
echo
echo Stowing packages....
echo
stow -v --ignore="dependencies.txt" *
Last edited:
I bet a lot of your problems with merging changes comes from conflicting line endings.
bleary
Member
Probably. 50+ years of computing and we're still trying to figure this out. This is why we don't have jetpacks.
Nah, they are just too hard to steer.
That said, the line ending problem has been solved for almost 50+ years but people have to be aware enough that it exists to actually seek out the solutions. While I don't recall the specific bits to fiddle with it is possible to to tell SVN what the local line ending convention is and have it automagically convert between the local and remote. Perhaps there is a FR to expose that so mafia's embedded SVN will do the right thing or maybe there is a BR because it is already trying and there is something about bleary's environment that makes it fail.
roippi
Developer
As an FYI, I'm going to be making some pretty radical changes to how mafia resolves SVN-installed script calls. But I'll do my best to ease the migration. It'll go something like this:
0) Include svn/ directory in namespace searches, only if there is no match in the standard namespace (i.e. scripts/, relay/ and friends). This is very trivial to code, already have this done locally.
1) Refactor scripts menu to use bookmarks (i.e. internal symbolic links) instead of relying on physical subfolder usage. This will take me a while. (Sigh.)
2) Stop pushing local updates into /scripts and /relay. If a copy of a script exists in both places, preferentially use the copy in svn/. No warnings or errors yet.
2.5) Start emitting warnings that duplicate scripts in the "normal" script locations (scripts/, relay/) that are also installed in svn/ will soon be errors. i.e.
3) Warnings become errors. Scripts won't run until you delete one copy (i.e. the non-svn copy).
The whole goal here is to wean mafia off the idea that the scripts/ folder is where scripts have to live - it's just massively simpler if we can run scripts directly from the svn/ folder. For one, "svn sync" will no longer be necessary, as the svn copy is the only copy. It also makes the workflow for authors who publish directly out of their svn/ folder far simpler, and just generally reduces the amount of "surprising" behavior mafia's SVN implementation performs.
0) Include svn/ directory in namespace searches, only if there is no match in the standard namespace (i.e. scripts/, relay/ and friends). This is very trivial to code, already have this done locally.
1) Refactor scripts menu to use bookmarks (i.e. internal symbolic links) instead of relying on physical subfolder usage. This will take me a while. (Sigh.)
2) Stop pushing local updates into /scripts and /relay. If a copy of a script exists in both places, preferentially use the copy in svn/. No warnings or errors yet.
2.5) Start emitting warnings that duplicate scripts in the "normal" script locations (scripts/, relay/) that are also installed in svn/ will soon be errors. i.e.
Code:
Warning: script svn/bumdoodad/scripts/bumdoodad.ash conflicts with $ROOT/scripts/bumdoodad.ash. You should delete $ROOT/scripts/bumdoodad.ash.The whole goal here is to wean mafia off the idea that the scripts/ folder is where scripts have to live - it's just massively simpler if we can run scripts directly from the svn/ folder. For one, "svn sync" will no longer be necessary, as the svn copy is the only copy. It also makes the workflow for authors who publish directly out of their svn/ folder far simpler, and just generally reduces the amount of "surprising" behavior mafia's SVN implementation performs.
roippi
Developer
So slight change in plans on step 0.
I'm creating a preference svnMigration that will ease transferring to the new way.
Defaults off. Turning it on does a few things. First, no files will be pushed out of your svn/ folder. Instead, namespace searches will include svn projects. Since at the start everyone will have duplicate files, no warnings will be emitted. Instead, if there is exactly one "local" file and exactly one svn file, just use the svn file. Again this is all mediated by having that pref active; if it's false, things operate as they work now.
So then migration goes like this:
0) pref defaults off. No warnings for duplicates.
1) refactor scripts menu.
1.5) pref defaults on.
2) remove the pref, start emitting warnings. Introduce a "svn dedupe" command that will go through and remove all your local files that have a svn/ counterpart. Mention it in the warning.
3) warnings become errors.
I think this is finally palatable. Making it opt-in at the start is good, and the dedupe command will make cleaning up local files painless.
I'm creating a preference svnMigration that will ease transferring to the new way.
Defaults off. Turning it on does a few things. First, no files will be pushed out of your svn/ folder. Instead, namespace searches will include svn projects. Since at the start everyone will have duplicate files, no warnings will be emitted. Instead, if there is exactly one "local" file and exactly one svn file, just use the svn file. Again this is all mediated by having that pref active; if it's false, things operate as they work now.
So then migration goes like this:
0) pref defaults off. No warnings for duplicates.
1) refactor scripts menu.
1.5) pref defaults on.
2) remove the pref, start emitting warnings. Introduce a "svn dedupe" command that will go through and remove all your local files that have a svn/ counterpart. Mention it in the warning.
3) warnings become errors.
I think this is finally palatable. Making it opt-in at the start is good, and the dedupe command will make cleaning up local files painless.
I can't find any reason to object. Thank you.
Clarification - this will actually apply to items in scripts, relay and data, right, or are the latter two down the road?
Clarification - this will actually apply to items in scripts, relay and data, right, or are the latter two down the road?
Winterbay
Active member
The only issue I have with this, and it is a minor issue, is that it'll lead to having to look in more places for where to edit scripts. But I guess not any more than now if I want the edits to svn-enabled scripts to persist over updates. So, as I said, a very minor issue
Bale
Minion
I look forward to removing the problems for people who want to know how to save edits they make to their scripts. Accidentally editing the wrong copy will be a thing of the past. The need for instruction will be reduced to information on where to find the /svn directory. And how to resolve conflicts.