Feature - Implemented allow kolmafia to handle ash bundles

This is a "speculative" feature request, in that I'm shooting for feedback on the value/direction of the approach before writing code. The problem I see is (and this is notably for relay scripts, although presumably if done properly the type wouldn't matter) that for ash scripts that bundle any additional resources (javascript, images, css), it's currently unwieldy to manage them - you have to unzip the files into your scripts directory, hope they're all in the right place, and then if there's a later version you have to figure out which files to remove and/or replace, etc. I propose extending the scripting system such that it will also recognize zip files, and transparently (to the end kolmafia user) expose whatever ash scripts/resources are in the zip. The outcome would be now after downloading the latest best_relay_interface_ever.zip, you'd just drop it somewhere in your .kolmafia structure instead of having to unzip it to make it usable by kolmafia. And if a new version comes out, you just delete the original zip and drop in the new one.

I'm handwaving around the implementation details for now, just figured I'd test the temperature on if this is a useful problem to solve or not. My apologies if I'm not explaining this well...

Anyone who uses chit would undoubtedly appreciate this.

This would be super-mega-awesomesauce; external css & js is so much easier to maintain, but the headache for users is... appreciable. I can see this being difficult to implement in a reliably cross-platform way, though. Maybe...

Yeah.. I think it's an okay idea. I think there's a few considerations, like how KoLmafia should handle a zip file that contains more than one ASH file, or if a size restriction should be placed on the zip file, or how KoLmafia should treat a zip file that contains no ASH file, how it should handle directories, etc.

StDoodle said:

I can see this being difficult to implement in a reliably cross-platform way, though. Maybe...

Click to expand...

java.util.zip helps with that

Edit: I have a question, are you talking about being able to automatically unzip and run an ASH bundle, or are you talking more about being able to "install" ash bundles and then run the actual ASH file later on?

What you're saying sort of reminds me of the SMZIP format.

When I saw "ash bundle" I wondered if this was about new content. Two comments. First it is a nominal security feature that KoLmafia will only access a selected set of directories. Will the implementation of this break that? Since unpacking a zip file has been an attack vector for malware is this feature increasing risk/exposure? Second I seem to recall a lack of unanimity on compressed file formats among posters here. Would implementing this force script writers to have a zip utility or opt out?

As a veer - there was some discussion of embedding a SVN client in mafia that got tabled because the Open Source client required at least 1.5 and at the time mafia was still Java 1.4. If that is revisited, would it provide an alternative distribution path that supports bundling?

fronobulax said:

When I saw "ash bundle" I wondered if this was about new content. Two comments. First it is a nominal security feature that KoLmafia will only access a selected set of directories. Will the implementation of this break that?

Click to expand...

Not with structure mimicking. Let's say awesome.zip has 5 files in it:

awesome.ash
data1.txt
data2.txt
data3.txt
awesomerelay.ash

awesome comes out with a new version, so just overwrite the old .zip and voila.

If the .zip's internal directory structure have to mimic Mafia's:
scripts/awesome.ash
data/data1.txt
data/data2.txt
data/data3.txt
relay/awesomerelay.ash

Anything in the root directory of the zip would be wholly ignored and would be the perfect place for a ReadMe if anyone felt like opening it up to examine it.

Since unpacking a zip file has been an attack vector for malware is this feature increasing risk/exposure?

Click to expand...

Compressed files are already available from the mafia forum, so the risk is at least as bad as that.

Second I seem to recall a lack of unanimity on compressed file formats among posters here. Would implementing this force script writers to have a zip utility or opt out?

Click to expand...

From the OP, I would assume this is for an additional option. ZIP utilities are a standard option for Windows systems since XP IIRC, pretty sure they're easily available for Macs if not included as well, and I don't think I can name a Linux distro that doesn't offer one unless it's a real lightweight. Heck, I think even my Android can open .zips. AS it is, .zips do seem the de facto standard with the occasional .rar on these forums.

I'd suggest, in line with Fluxxdog's internal directory structure, that there's a new folder, probably named "zips" in line with the "scripts" (for plurality) folder, and this is the only folder where zips are parsed.

Java has a java.util.zip package. Using classes from that, KoLmafia would not need an external program to extract zip files. It could examine the directory of entries and extract only those that it liked: those with paths relative to scripts, data, and relay, say.

Implementation-wise, extracting at execution time is probably not going to work, in terms of performance. Unless there's some way to do, like, a hash check comparison on compressed files versus already extracted files, and skip the ones that are already extracted.

Especially when you're dealing with relay overrides, where you have no idea what it might need, since one semi-unrelated file might involve a bunch of other files. To do this well/easily, it would need to pull out the whole folder on initial run, and keep it available until closing.

I had to ask the security questions because That's What I Do, but I concur that this FR is not going to make anything worse. I was thinking that mafia would look for zip files in a special place, unpack any one that is "newer" and that would be it. Thus the performance hit only occurs when there is a new file and a user edits in the scripts directory would override anything in a zip. The definition of "newer" could be explored but a preference could probably deal with that.

The simplest and least intrusive way would be to have an ash function unbundle_zip(filename). It would look for filename in .\zips, unpack anything that belonged in .\data, .\relay and .\scripts and that was it. The script writer would presumably manage whether it was called or not. Of course, that might lead to a situation where unbundle_zip tries to overwrite the file that is running it. Never mind. Mafia has to trigger the update when nothing else is going on.

Wow, this spawned a lot more feedback than I expected, which is awesome. So - my thought right now is that I'd want to extract and open the files in memory on startup, to avoid a whole host of issues with having to worry about someone else breaking parts of a bundle outside of kolmafia, and/or having to have kolmafia keep track of what files map to what packages. But, as I'm writing this, there's a downside there anyways of now a lack of clarity in what scripts you've got.

So, tweaked proposal - create a .kolmafia/zip folder, that, on startup, gets parsed for new zips which then get unzipped to your .kolmafia folder. To the security point, I'd have to ensure that it was effectively chroot'd - only unzipping in .kolmafia, and only the approved folders. I'd also store in prefs a md5sum (or whatever's easier in java) + filename w/ whatever files were created, and if I found a zip with the same name but different hash, I'd remove the old files associated with the zip and unzip the new ones. Similarly, if the zip was removed, I'd remove the old files. Zips would have to have the appropriate relay/script/etc. folder structure in them, and then could have whatever readme/docs they'd otherwise like to include.

thoughts? I need to spend more time reviewing the thread to see if there were other questions I should address. For those of you who are linux users, I feel like I'm re-implementing RPM.

I've been busy, so I'm a bit late to the party, but I would love this for the sake of publishing relay scripts with css and java files. Mostly I'd love this to make ChIT and easier install. I'll like to tell the users to simply put it in the /zip drectory -- and that is all they need to do.

Other points that occur to me: What if there is a file in the zip folders root directory? Will that be ignored like it was in the KoLmafia's root directory, or is there some use for a file there?

Second, what if there is a file /relay/charapane.ash and a file /zip/relay/charpane.ash? You'd have to make sure that they don't conflict, so which one would take precedence? I'd recommend the zip file since otherwise un-installation might be confusing to simple minded users.

Oh hey! A thought just occured to me! If this is done, the master relay override format that bumrats uses will be deprecated since a zip file is even easier to install (no property to set) and makes it easy to use only part of by deleting any relays from the file that you don't want to use. I'd like that since I would use some parts of bumrats if they could be easily separated from the rest.

It will be a slight hassle if a script gets run, determines that it needs to be updated, and then the update isn't applied as soon as the new zip is put in place.

I'm probably in the minority preferring a SVN solution, since every time I update CHIT I have to merge the changes into my modified version instead of simply downloading the new file.

I'm thinking along the same lines, lost. I think there's too many problems with the zip implementation anyway. Requiring a restart is blah, for several reasons - not the least of which is the boatload of server hits added. And I don't see a good way to do it without a restart.

An SVN checkout solution could be relatively easy to write and end up being way more user-friendly. SVNKit is available open-source and seems like a pretty easy plug-in solution.

roippi said:

An SVN checkout solution could be relatively easy to write and end up being way more user-friendly. SVNKit is available open-source and seems like a pretty easy plug-in solution.

Click to expand...

I already use SVN and symbolic links to keep my BumcheekAscend modified and up to date, so I'm also in favour of this.

Okay - different direction than I was thinking, but my overall goal is "make it easy to install/manage extensions to kolmafia", not "make kolmafia use zips".

So, what then does implementation of "svn support" look like? adding in a ash function that lets scripts download resources normally? and now the "distributed" file that script developers post is just a wrapper that defines the svn respository and resources it should download? Is this something that the devs would be willing to include in the main kolmafia repository, or should everyone register their own projects (or should someone create a kolmafia-scripts sf project?).

Perhaps it is time to reference the previous discussion here. That discussion petered out because any solution beyond integrating SVNKit was more work than I wanted to do, no one else was volunteering and I realized SVNKit would not work under the previous Java 1.4 target.

In the current discussion, I would imagine a list of files in hosted in the Kolmafia.us SVN and an ash command to update every one of those files that had not been locally modified. A script with dependencies would just have to add the dependency to that list. If there was enough script developer buy in then doing the updates based on the list could be a breakfast option.

This also brings up the possibility once again of enabling mafia to internally update to the latest SVN build. Without relying on an bit of bandwidth from fewyn's servers, which is nice.

Although.. that might require users to have a JDK. So maybe nevermind on that. Maybe.

Maybe we could use Java_Web_Start to do auto-updates