Official script registry - discussion

roippi

Developer
So. I am beginning work on a GUI frontend for the SVN manager. (calm down, it's still a long ways off.) Still, it is not too soon to bring up the elephant in the corner - how are we going to handle the script registry? There needs to be one file (or I suppose several) that mafia can fetch that says "here are the scripts available to download, and here is some metadata about each one."

i.e. the contents of the table in this thread, in machine-friendly format.

Now, before I enumerate the possible options for the implementation of such a thing, let me remind everyone of the unofficial mission statement for incorporating an SVN client into mafia in the first place. The goal is to minimize the barriers between scripters and those who would use scripts. That goes both ways, mind. It should be as easy as possible for Joe Sixpack to install and update a script. It should also be as easy as possible for Steve the Scripter to publish and maintain his script.

Now, the options, off the top of my head. If there are others, mention them, I will add them to this list.

  1. Registry file is maintained within mafia's source tree.
  2. Registry file is hosted on kolmafia.us.
  3. Registry file is hosted in some SVN repository, somewhere.

My thoughts, below.
 

roippi

Developer
So pursuant to the mission statement I posted above, I believe there are three critical factors to consider in any proposed implementation: maintainability, accessibility, and security.

Maintainability is hugely important. Who is doing the maintaining? How hard is it for them to make updates to the repository?

Accessibility is similar in nature to maintainability. How hard is it for a new person to get "in" to the system? Or to be given update privileges to the repository file?

Security is somewhat important as well, though our community is small and mostly relies on common decency. Still, there needs to be a human in the loop. Someone has to give the "okay" for a new scripter to enter their script into the registry (and, likewise, for obsolete scripts to be removed from the system).

With all that in mind, my takes on the first three (at the point of this writing, only three) options above:

1) Is not happening. Simply put, I won't be involved in coding anything to do with such a system. The burden of maintaining the script registry can't be solely on mafia devs. Also, issuing updates to the repository file shows up in the SVN update history, mixed in with actual updates to mafia that people want to read. Decreasing the signal-to-noise ratio is not a happy side effect. All together: no.

2) Is actually my favored solution right now. I asked around a while ago and it is possible to set up a subforum wherein people can be made administrators solely for it. Thusly a group of people could all be given edit permissions to a communal document. The downsides are that it is.. clunky. The workflow goes one of two ways. One: scripter posts what they want to go into the repository, maintainer takes that and transcribes it into file. Two: scripter posts that they want edit access, maintainer okays them. Both are less than ideal (see: barriers) but not too bad.

3) Has some upsides over 2). The file itself being in a version control system means.. well, all the upsides of a VCS. We should be familiar with them by now: you track who did what, see exactly what changed, can rollback things you don't like, etc. There are downsides. The workflow is no better than in 2; worse actually, since they probably need to sign up for another account somewhere, yada yada. Security is a mixed bag - VCS limits vulnerability to one guy irrecoverably deleting something, but hosting the file in an external place brings all the headaches therein.

Those are my thoughts at the moment.
 
Last edited:

lostcalpolydude

Developer
Staff member
If the separate SVN repository is on sourceforge, then there's probably no new account for the maintainer to make. I expect that anyone taking on that role would have SVN-hosted scripts of their own, and so far everyone has gone with sourceforge. It's likely that the person publishing the script will host it on sourceforge, and therefore have an account there already, but I expect the script/author would come to the maintainer's attention due to a post/thread on this forum, making the sourceforge account unnecessary. If we have someone willing to take on the role of maintaining the list (something that would be a bit more official than the forum post in use currently), I don't see any issues for #3.
 

Bale

Minion
Personally, I favor 3. Registry file is hosted in some SVN repository, somewhere.

An SVN repository on sourceforge would work or fewyn might be willing to set it up on kolmafia.us. If fewyn sets it up here we completely bypass the downside you mention. Potentially it could use the same login that we have here on the forum so all that would be necessary is for someone with access to flip a switch granting write access. If it is on sourceforge, well the script is already on SVN so most scripters are already registered there.

1. Maintainability: This would become the responsibility of minions plus some others that the minions & devs have decided to trust. With a sufficient base of people (addressed in the following point) it will not be difficult for someone to find the time necessary to add or remove a script within a 24 hour window. We would also maintain a thread in the Community Forum where scripters can ask for a new script to be added to our list. Whenever someone posts to suggest adding a script to the list, someone responsible will consider it.

2 Accessibility: Specifically we'd add script wranglers from the ranks of committed scripters like zarqon and Winterbay who have proven to be both prolific and committed to the community. Before anyone is added a minion would bring up their name in the Mod Lounge and if there is doubt, we'd vote on it. All minions and devs would be granted the ability to grant editing permission -- that just means creating a permission group for minion which allows adding people to the "script wrangler" permission group which has editing ability, but not the ability to alter user permissions. As for creating an account on the svn repository... anyone who thinks it is too much trouble to create a login, even with need for email verification... well, we just don't need anyone who thinks that barrier for entry is too high. Seriously, that's like 5 minutes of work at most.

3. Security: I believe I addressed some of these issues in the previous point. It does rely on trust and it is reliant on minions to have some idea of whom to trust. Having an attentive pool of minions and script wranglers will allow us to swiftly correct violations as soon as they are noted. Quite frankly this works because our community is good.

I am reassured that we have only had two scripting security incidents in all the time that KoLmafia has been around. One of those was intended as a joke since he hadn't expected anyone to actually use the example inside his code tags and he altered the text of his helpful example as soon as someone pointed out that the idea of code sending him their Mr. As wasn't a good idea, just in case someone actually copy/pasted it and tried to run his code. The other one was intentional, but that time wasn't malicious either, just bad judgment since he thought he was merely charging a small one time fee for using his farming script and he removed that code as soon as others pointed out he shouldn't have done so. Those are our two worst disasters. The former led to no problems at all and the later resulted in the refund of a small amount of meat to a few people. That's why I trust security in this community. Since we'll have minions and script wranglers as a buffer, we should be fine.
 
Last edited:

fronobulax

Developer
Staff member
I vote for a text file hosted somewhere in SVN. I'm comfortable with Bale's comments and suggestions.
 

roippi

Developer
I'm fine with option 3.

Next question is the markup of the file. Two options that I can think of:

  • CSV (or any delimiter, including tab)
  • JSON

JSON is more reasonable to work with on the parsing end, but it can be very annoying to hand-code JSON for people who are maintaining the file. So unless everyone is comfortable with firing up a python interpreter to edit it...
 

Bale

Minion
I was thinking of mafia's usual text file format which would be tab delimited, but... Can you recommend a user friendly python interpreter I can use in Windows? I'm at least willing to consider the alternative since it would remove incidences of people confusing tabs and spaces: a not uncommon source of bugs.
 
Last edited:

roippi

Developer
Can you recommend a user friendly python interpreter I can use in Windows?

On my windows box I use virtualbox to get me into a ubuntu shell to do most things, so I can't help with the exact installation procedure.

But I can in general recommend ipython running in qtconsole mode. It's the best interpreter I've used, by a rather wide margin.

This is a useful tangent but we should remember that we want to make this system accessible; I'm not sure that making people install and learn to use a python interpreter satisfies that litmus test. Though hand-editing JSON is always an option for those so inclined.. so I dunno. Worth considering.
 
Last edited:

Bale

Minion
Your point about accessibility is well taken. We'd best stick to the tab delimited format utilized by other KoLmafia files. That is the method best suited to keeping a low barrier to entry.
 

Winterbay

Active member
I guess CSV would remove the problem of people using spaces instead of tabs and whatnot if we aren't overly concerned about looking the same as all other Mafia files.
 

Bale

Minion
Commas might definitely be used in script descriptions. Perhaps tildes are safe? If someone wants to use a tilde we can just substitute an emdash and he'll live with it. Though tabs are more intutive.
 

Fluxxdog

Active member
Question: What information will you be having? I'm expecting an SVN checkout command (parsed as plain text), a user friendly name (parsed as plain text), and a description.
For the descriptions, will they be parsed as simple text or will we be allowed to use some html formatting?

Sorry, questions spring forth and I thought I should ask now in planning rather than FReq it later.
 

Bale

Minion
You forgot the name of the scripter. I created this thread thinking that it was the same information as we'd be using for the official script registry. (Though perhaps longer descriptions might be used.) I think we'd want to categorize the scripts that way also to make it easier for a user to find the sort of script he wants.

I'm not sure why we'd want to need formatting for the descriptions. Take a look at that table and tell me which descriptions would benefit from html. Of course, if it is really easy for Roippi to do, I guess there isn't any reason not to add the feature since the future might bring us something interesting and we'd want it then.

Side note: Although I listed function libraries in that thread, I think we shouldn't list them in the data file. Those will simply be added as dependencies if he needs them so he doesn't need to get confused by excess data.
 
Last edited:

Fluxxdog

Active member
You forgot the name of the scripter.
Derp! :p That's why you're a minion and I'm a lowly... uh... nothing ^^
I created this thread thinking that it was the same information as we'd be using for the official script registry. (Though perhaps longer descriptions might be used.) I think we'd want to categorize the scripts that way also to make it easier for a user to find the sort of script he wants.
Categories are definitely good. as for longer descriptions, there would be people who'd want to explain some important aspect. I guess I was thinking people would add the scripts from mafia without accessi9ng the thread on the forums that contains information they'd need, like configuring.
I'm not sure why we'd want to need formatting for the descriptions. Take a look at that table and tell me which descriptions would benefit from html. Of course, if it is really easy for Roippi to do, I guess there isn't any reason not to add the feature since the future might bring us something interesting and we'd want it then.
It was just a thought. I was originally thinking of longer descriptions.
Side note: Although I listed function libraries in that thread, I think we shouldn't list them in the data file. Those will simply be added as dependencies if he needs them so he doesn't need to get confused by excess data.
But what if people want those libraries? Should kep them just as available. zlib is an excellent example of something you want to highlight.
I just wanted to point out that FaxBotDatabase knows how to parse XML files.
I like where that line of thinking is heading ^^
 

xKiv

Active member
Side note: Although I listed function libraries in that thread, I think we shouldn't list them in the data file. Those will simply be added as dependencies if he needs them so he doesn't need to get confused by excess data.

Scripter would be still expected to watch this forum for usable libraries and their svn links?
 

Bale

Minion
Well I do expect that scripters are more connected to this community so they'd notice those things. Roippi is talking about providing a tool for the end user. End users are confused when you provide a function library that they cannot simply install and use. Therefore adding function libraries will confuse the end user making it harder for them to understand what is being offered and increasing our need for customer support. I don't want us to need to answer more questions and council more people because we show them something they do not need.

Random people just message me in chat when something about KoLmafia confuses them, so I have a clue about what confuses users. Please, let's not add function libraries to the official registry.
 

zarqon

Well-known member
Bale, users of KoLmafia are scripters. By default. Excluding libraries would be silly.

I like the idea of hosting on kolmafia.us. I don't think SVN is really designed for communal database editing where each datum has differing ownership.

I also think a link field would be a good idea, since no doubt authors would like to direct people to a place to get more information or discuss the script (most likely a thread here). If we allow markup in descriptions, that link could be provided there, but having it as a separate field would perhaps be more useful to KoLmafia, depending on the eventual GUI.

One possibility: a PHP file on kolmafia.us that parses a forum post (perhaps Bale's preexisting post could be the starting point, but we'd have to agree on a standard format that this file would parse) and returns a file in whatever format Roippi wants to receive, likely JSON. Thus, the middleman (a script) would eliminate formatting work to be done by either KoLmafia on the client side, or users on the provision side. It doesn't solve the delay between submission and existence in KoLmafia's data though, as Bale or a minion would still need to add the script to the forum post.

This is probably better: a PHP file on kolmafia.us which provides an entry form for logged-in users to register scripts. Would be pretty easy to do, and we could make it work with the forum software so that only logged-in members could submit. The desired level of control/security could be built into the script. I like the idea of reducing entry requirements to almost nothing, then giving some users administrative power to remove unwanted entries from the list (or, if necessary, block users or IPs from submitting). Maximum convenience!

Just throwing out some ideas that haven't been thrown yet, in case they help.
 
Top