Mafia SVN for Scripts?

[roippi]

ant is really badly documented. By that I mean that the ant manual is actually very complete but unintuitively laid out, and googling what I need is hard.

I believe I figured it out. First, compile needs to know about the .class files in the jars, so it needs to be added to the classpath:

			<classpath>
			      <pathelement path="${classpath}"/>
			      <fileset dir="${lib}/jar">
			        <include name="**/*.jar"/>
			      </fileset>
			</classpath>

And then jar needs to know about it since compile will make sure that all the necessary .class files are there but won't actually unpack the .jars:

		    <zipgroupfileset dir="lib/jar">
		        <include name="*.jar"/>
		    </zipgroupfileset>

Totally intuitive, yeah.

I'll be committing this soonish and hopefully it compiles for others. Knock on wood.

 

[roippi]

Well strike one, the .jar files didn't unpack on fewyn's side. The .jar compiled but none of the binary libraries got loaded, so svn stuff is broken.

I'm not sure why, zipgroupfileset worked for me. Hmm.

Fewyn, can you make sure your build.xml properly updated and whatnot?

Others who build from source: easy way to check if the binaries are properly installed for you is if you open up the .jar, you will see a sqljet.build.properties file in the top level of the .jar, and also a /de/ folder. Anyone getting those?

 

[lostcalpolydude]

NetBeans complained about some errors. The first file I looked at is SVNManager.java, and it looks like java.util.ArrayDeque and java.util.Deque require Java 6.

 

[roippi]

Oh yeah. I'll fix that.

That's not really the issue though, fewyn compiles under 7.

As for netbeans, you'll need to tell it that lib/jar is in your classpath, however that works. .classpath tells eclipse that, and build.xml tells ant that, but I don't know what tells netbeans that.

 

[lostcalpolydude]

It looks like NetBeans uses .classpath also, I just have it set up to use Java 5 to make sure I don't add Java 6/7 features.

 

[Theraze]

Compiling 12142:

    [javac] Note: C:\kolmafia\src\net\sourceforge\kolmafia\svn\SVNManager.java uses or overrides a deprecated API.

 

[roippi]

Yes, that is expected. It's just a note.

Think I discovered what the issue is, I think fewyn's build does "ant daily", and I only addressed "ant jar".

 

[holatuwol]

I think the reason there is a lib branch with non-KoLmafia source is because no one really figured out how to build our jar "with dependencies". Maven will do that but I'm not sure how to bridge the gap between Maven and the associated infrastructure I use at work and KoLmafia.

I created the lib folder so I could differentiate between source code that I was modifying from some other library and source code that originated in KoLmafia. Also avoids the problem where JARs are built using a later version of Java, making it not useful when you're trying to get things to run on older versions of Java.

For all intents and purposes, building a JAR with dependencies is nothing more than creating a different manifest file for the JAR, which Maven does for you but is a little bit trickier in Ant. Or you could do what roippi is doing and just unzip the whole JAR.

 

[lostcalpolydude]

For anyone else using NetBeans, I had to add all of the .jar files to the Compile tab in Libraries for the project, rather than just adding lib/jar.

 

[roippi]

svn stuff should be working in daily builds now after r12144.

 

[lostcalpolydude]

Running 12144. I discovered that svn list and update throw a NPE if you don't have any projects checked out yet, which is just a minor thing. I tried checking out the SVN repo I set up of my own, but that silently failed; maybe I have that misconfigured.

So I tried checking out the test folder you posted here. It got as far as printing "Repo validated." I now have a svn folder, and inside it is a project folder, but that folder is empty. Mafia is green but the checkout command is stuck in the queue.

After restarting mafia, I can use svn list to see the one project I have so far. Using "svn update" just gets it stuck in the queue.

 

[roippi]

Sounds like you haven't imported the jars correctly, I imagine if you look at the CLI (not mafia CLI) you will see some ClassNotFoundExceptions.

Try downloading an hourly .jar and compare with that.

 

[Bale]

> svn checkout https://svn.code.sf.net/p/mafiasvntest/code/myvalidproject1/

Validating repo...
Repo validated.
C:\Documents and Settings\David\My Documents\Dropbox\KolMafia\svn\mafiasvntest-myvalidproject1
A https://svn.code.sf.net/p/mafiasvntest/code/myvalidproject1/scripts
A https://svn.code.sf.net/p/mafiasvntest/code/myvalidproject1/scripts/valid2.txt
https://svn.code.sf.net/p/mafiasvntest/code/myvalidproject1
At revision 7

Successfully checked out working copy.
Pushing local updates...
valid2.txt => C:\Documents and Settings\David\My Documents\Dropbox\KolMafia\scripts\valid2.txt
Done.

Sweet!

 

[roippi]

So svn stuff should be working now, report bugs relating to it over in bug reports.

Where svn features are going:

Short term plans-
I want to spend a while hammering out the behavior of the current CLI commands. I expect mainly advanced users to be using the feature. I want to improve error handling (some stack traces only happen in the CLI, some happen in DEBUG logs, etc) and gCLI output when an error happens. Specifically if an SVN repo is unreachable, I think that should be handled gracefully.

I also want to make repo validation less permissive of file types. Right now validation allows any file type through. I think we should have a whitelist, with

• .txt
• .ash
• .php
• .js
• .css
• .gif
• .png
• .jpg
• .html
• directories

allowed, and others disallowed. I probably forgot an extension or two, let me know.

Medium term plans-
I expect a majority of the "prominent" scripts to have migrated to the svn system, though forum downloads may be maintained in parallel. Lay users should be encouraged to use the feature. Major bugs should not exist, error handling should be pretty good. A preference will exist to run "svn update" on login (this is trivial to put in a loginScript, but people will want it).

Long term plans-
GUI support for a script manager. Users can at a glance see what scripts they have installed and what scripts can be updated. One-click updates. Other metadata will be shown at a glance, like date of last update, author info, number of files in the project, etc.

I'd like to have installation of new scripts handled through this interface as well. It's pretty easy to have a .txt file hosted somewhere that has a list of links to repos with scripts, maybe with a quick description attached. A name filter field lets users get to the script that they want quickly. Here's where I'm not 100% sure on the direction to take with it. Who is responsible for maintaining that .txt file? How do we do quality control on it, prune obsoleted scripts, etc? Food for (eventual) thought, we have plenty of time to discuss it.

 

[Bale]

I've only ever used a subset of those file types for my scripts, so I cannot suggest any file types missing from that list. This will make updating of ChIT so much easier. It's got a gazillion files in 3 different directories. With this any changed files can be updated with a single CLI command or eventually by clicking on the appropriate GUI button.

So... I wonder if I should get a sourceforge account, or would an svn hosted by fewn or lost be better?

It's pretty easy to have a .txt file hosted somewhere that has a list of links to repos with scripts, maybe with a quick description attached. A name filter field lets users get to the script that they want quickly. Here's where I'm not 100% sure on the direction to take with it. Who is responsible for maintaining that .txt file? How do we do quality control on it, prune obsoleted scripts, etc? Food for (eventual) thought, we have plenty of time to discuss it.

Sounds like a pita for a single person to have to handle, but if all the minions and devs are given control over it I am sure that it would be no big deal. We'd also have a thread on this forum where people are encouraged to point out any scripts that need to be pruned, etc so that minions can take care of things. Shouldn't be hard once there is a system set up that allows multiple people to alter it as necessary.

 

[lostcalpolydude]

svn checkout didn't work for me with a svn:// URL (even though TortoiseSVN could browse it fine), but it worked once I set up WebDAV (http URL). Unfortunately, along with not working, there was no output to try to track down the issue. Also, once I set up http, svn suddenly worked (up to the error about that repository already existing, which wasn't at all surprising but could eventually be handled better than a debug log maybe). My only guess for a starting point is

String testURL = "http" + url.substring(index);

(line 284 in SVNURL.java).

I haven't actually posted a URL because I probably want a different domain name for this stuff.

 

[zarqon]

You might allow .nfo files -- it would allow us all to be h4ckZ0rZ.

This is kind of amazing. I'm going to sit back and watch as this unfolds, as I'm clueless about SVN and have very little to suggest. I do wonder how much control it takes away from users/scripters at the expense of automation. It seems like something I'd probably want to participate in, however, if it allows updates to automatically trickle out to users.

I'm leery, however, of other people's updates trickling to me. At certain times, my scripts are in a state of functional disarray as I'm refactoring or adding something, and if another user's changes came to my script at that time it could cause me to lose track of what I was doing. There have been no shortage of people posting altered versions of my scripts over the years -- which is fine, in the end it's about having the best ideas in a script and it's not surprising when those ideas aren't all mine -- but I don't want to lose control of my own development process, and I don't want to make it easy for people to fork my scripts. Forks are for people who can't cooperate; I can. I would rather consolidate all the best ideas in one place, which is what I've historically worked towards when adding code from other scripters.

I'm sure by now it's apparent I have no idea what SVN is or how it works. As long as the final implementation proves my concerns groundless, I'll be happy to participate. Thanks for your revolutionary work, roippi.

 

[Winterbay]

As long as you set up the repository (be it sourceforge or fewyn or your own) in the correct way only the one given access to the repository can update the script so un-authorized merges won't happen.

 

[xKiv]

I do wonder how much control it takes away from users/scripters at the expense of automation.

I think, as it is now, it will take away my ability to put scripts in (sub)folders of my choosing (library, consult, turnburning, info, ...).

 

[roippi]

svn checkout didn't work for me with a svn:// URL (even though TortoiseSVN could browse it fine), but it worked once I set up WebDAV (http URL). Unfortunately, along with not working, there was no output to try to track down the issue. Also, once I set up http, svn suddenly worked (up to the error about that repository already existing, which wasn't at all surprising but could eventually be handled better than a debug log maybe). My only guess for a starting point is

String testURL = "http" + url.substring(index);

(line 284 in SVNURL.java).

I haven't actually posted a URL because I probably want a different domain name for this stuff.

I had similar issues with svn:// URLs, I never did track it down. I do remember svn:// magically start working at some point, so there's some way to make it work at least.

So... I wonder if I should get a sourceforge account, or would an svn hosted by fewn or lost be better?

There's likely going to be some growing pains with non-sourceforge repos, as above. And I don't know of any features that you'd be getting with hosting your own repo. There could be something I suppose.

As long as you set up the repository (be it sourceforge or fewyn or your own) in the correct way only the one given access to the repository can update the script so un-authorized merges won't happen.

Correct. The reason I like this multiple-individual-repo model is that scripters have ultimate control over their own stuff. The only way for two scripts to interfere is if you checkout two projects with an identically-named file.

I do plan on adding more protection against users accidentally doing this. It's corner-case, but could happen with relay scripts. I've put in a lot of code that protects users from unwanted files showing up via svn update, but haven't really protected against svn checkout installing conflicts.

It does sound like I should make an svn 101 post as there's bound to be confusion on what it is, what it isn't, and what it can do.

I think, as it is now, it will take away my ability to put scripts in (sub)folders of my choosing (library, consult, turnburning, info, ...).

Eh? No, I've stated several times that nesting files is okay within the four permissible folders. You are restricted from putting files in the top level, and in places like ccs/ or settings/. I think this is reasonable.