Feature filename to hide a folder in scripts menu

[taltamir]

In mafia GUI there is a button at the top that says scripts. it shows the contents of your /scripts/ directory and lets you click on scriptname.ash to run it.
It would be great if we could include a file called "MAFIA_NO_EXECUTE_FOLDER" in a subfolder under /scripts/ to make it so that this subfolder will be hidden and not shown when viewing the scripts dropdown in the gui.
for example in autoscend we have
/scripts/autoscend.ash
which is the only file that should be run. then we have a bunch of other files in
/scripts/autoscend/
which should not be run. so hiding the entire folder would be great.

naturally an exception should be made so that the base /scripts/ folder could not be hidden.

 

[fronobulax]

Why?

Do we really need to change KoLmafia to defend against users who do not read the autoascend instructions that tell them how to run it?

There has been much discussion about refactoring how KoLmafia handles scripts and the underlying file system and this might be better deferred until some of those ideas come up for discussion and implementation.

 

[taltamir]

Why?

Do we really need to change KoLmafia to defend against users who do not read the autoascend instructions that tell them how to run it?

There has been much discussion about refactoring how KoLmafia handles scripts and the underlying file system and this might be better deferred until some of those ideas come up for discussion and implementation.

it is also convenient. as it makes the gui neater if it is not listing folders full of unexecutable

 

[MCroft]

Hey, yeah, I'm one of the people who is unsatisfied with the current solution, so I'm happy to talk about it.

I'd like to see something much more like a registration process which provides the entry points for a script.

My thought is we'd have a file, MANIFEST.json, that lived in the root directory for each script with some or all of the following data:

• The Script Manager Fields
  • Script Name
  • Authors
  • Short Description
  • Long Description
  • Forum (or repo?) Thread
  • Category
  • Repo
• dependencies
• script root
• menu name
• menu entry point

0 or more menu items, depending if you want it on the menu. Probably nice to include required mafia version and installed version, but not necessarily. This could let a package have multiple scripts in it (eg. "eat", "drink", "be merry").

script root is "this is where we start syncing stuff from /svn/scriptname (or /git/scriptname) to our scripts folder". That keeps us from having to worry too much about "is there a readme or other files in the /svn/scriptname directory" like we do now.

That last one, incidentally, allows us to not have issues with GITSVN, which is ironic since it probably won't come into KoLmafia until we also support git clone as an alternative to svn install.

Should make it easier to build svnrepo.json for maintenance, and also to add local scripts to the menu and manager.

 

[MCroft]

Hey, yeah, I'm one of the people who is unsatisfied with the current solution, so I'm happy to talk about it.

I'd like to see something much more like a registration process which provides the entry points for a script.

My thought is we'd have a file, MANIFEST.json, that lived in the root directory for each script with some or all of the following data:

and if there's no manifest.json, follow the current rules, so we can get there neatly.

 

[gausie]

I think an explanatory JSON file is a really really good idea. We should look at package.json for inspiration as it provides existing keynames for everything in your list

 

[taltamir]

Hey, yeah, I'm one of the people who is unsatisfied with the current solution, so I'm happy to talk about it.

I'd like to see something much more like a registration process which provides the entry points for a script.

My thought is we'd have a file, MANIFEST.json, that lived in the root directory for each script with some or all of the following data:

• The Script Manager Fields
  • Script Name
  • Authors
  • Short Description
  • Long Description
  • Forum (or repo?) Thread
  • Category
  • Repo
• dependencies
• script root
• menu name
• menu entry point

0 or more menu items, depending if you want it on the menu. Probably nice to include required mafia version and installed version, but not necessarily. This could let a package have multiple scripts in it (eg. "eat", "drink", "be merry").

script root is "this is where we start syncing stuff from /svn/scriptname (or /git/scriptname) to our scripts folder". That keeps us from having to worry too much about "is there a readme or other files in the /svn/scriptname directory" like we do now.

That last one, incidentally, allows us to not have issues with GITSVN, which is ironic since it probably won't come into KoLmafia until we also support git clone as an alternative to svn install.

Should make it easier to build svnrepo.json for maintenance, and also to add local scripts to the menu and manager.

This sounds awesome. I like it way better than my own suggestion.

 

[MCroft]

prior conversation about it as part of JGit

 

[fronobulax]

The curmudgeon expresses a slight sense of unease at a mechanic by which a third party can both place executable files on the system and then hide their presence, at least from the first tool likely to discover them.

If you invoke the Scripts option there are actually three options to choose a file. One is "Load script..." which is basically a Swing file chooser driven by what is on the local disk in selected directories. The other is the list that is displayed immediately. This "immediate" list is either driven by directory contents or, if the user opts in, by KoLmafia's dynamic list of most recently used scripts instead.

So this request could be framed better - is this a change to one of these displays or an additional display triggered by a new Scripts menu item or a user selected alternative to the immediate display list? My guess is that it is a change to the immediate display for users who are not using the MRU but that could be clarified. So long as the Load script..." option remans unchanged there is no real good case for making this an opt in alternative like the MRU.

Assuming this is a change to the "immediate" menu then the behavior when a locally created script exists needs to be explored. Does the implementation display everything unless a file or its parent directory is explicitly hidden by the configuration? Does it ignore things that are not mentioned in the JSON and then provide a way to autogenerate the JSON for locally created scripts? Does it ignore the disk contents completely and only build menus and submenus according to the JSON - thus requiring users who write their own scripts to take the additional step of creating the JSON or only invoking their own scripts from the MRU or Load script... options?

Tangentially, does this have implications for the Relay Scripts menu? If a similar request is made for Relay Scripts can this be extended or reused?

Finally, in the event that the ambitious rethinking of where scripts live on disk generates actual code changes, will much of this implementation survive? (While I agree that the rethinking is a long way off, it seems closer with the impending switch to GitHub and the opportunity for greater collaboration, code reviews and the ability to easily shelve changes).

On a practical basis there is a known, but as yet inconsistently reproducible, bug concerning the ScriptsMRU menu. There are circumstances where the menu should update but does not or that the update is somehow incomplete (only displaying a portion of the MRU list). Some thought should be given as to whether this new thing is always rebuilt every time the menu is displayed or whether there are listeners that trigger on disk and JSON changes or whether there is a hybrid approach. The choice might lead to a fix for the MRU issue.

 

[gausie]

The curmudgeon expresses a slight sense of unease at a mechanic by which a third party can both place executable files on the system and then hide their presence, at least from the first tool likely to discover them.

If they're not in the manifest they would not be executable

 

[fronobulax]

If they're not in the manifest they would not be executable

We have different expectations of how the manifest would be generated and used, I guess. No big deal. My intent was to get this fleshed out rather than shoot it down the way that I usually do for things I wouldn't spend my time implementing.

 

[MCroft]

The curmudgeon expresses a slight sense of unease at a mechanic by which a third party can both place executable files on the system and then hide their presence, at least from the first tool likely to discover them.

I don't think this is a problem exclusive to a manifest/registration model (which seems a lot more like a package or plugin manager than a script manager with the way scripts have evolved). But I'm definitely open to ways to minimize risk. Perhaps a future version of svnrepo.json could have a cryptographic hash that we can verify and tell that at least it hasn't been tampered with.

If you invoke the Scripts option there are actually three options to choose a file. One is "Load script..." which is basically a Swing file chooser driven by what is on the local disk in selected directories. The other is the list that is displayed immediately. This "immediate" list is either driven by directory contents or, if the user opts in, by KoLmafia's dynamic list of most recently used scripts instead.

So this request could be framed better - is this a change to one of these displays or an additional display triggered by a new Scripts menu item or a user selected alternative to the immediate display list? My guess is that it is a change to the immediate display for users who are not using the MRU but that could be clarified. So long as the Load script..." option remans unchanged there is no real good case for making this an opt in alternative like the MRU.

My thought is that I want to change how the scripts menu looks and works and how the scripts work under the hood and for developers. I still want the gCLI and actual CLI to be able to run a script.

Assuming this is a change to the "immediate" menu then the behavior when a locally created script exists needs to be explored. Does the implementation display everything unless a file or its parent directory is explicitly hidden by the configuration? Does it ignore things that are not mentioned in the JSON and then provide a way to autogenerate the JSON for locally created scripts? Does it ignore the disk contents completely and only build menus and submenus according to the JSON - thus requiring users who write their own scripts to take the additional step of creating the JSON or only invoking their own scripts from the MRU or Load script... options?

My first thought is that there's a "register script" command, (or perhaps script register, for the noun-verb fans). Minimum data would be a filename, and most of the data would be null. if register script pointed to a manifest.json file, it would load that.

If we combine that with Gausie's statement that we only run registered scripts, we would treat the scripts the users registered as first class citizens like the ones we pulled in from the ScriptManager.

It could be that we have to compromise on that and register a script as it is run, but I'm not sure all of the implications of that.

Tangentially, does this have implications for the Relay Scripts menu? If a similar request is made for Relay Scripts can this be extended or reused?

Sure. The data structure we're looking at would know the script type and name, so it's easy to build a menu.

Finally, in the event that the ambitious rethinking of where scripts live on disk generates actual code changes, will much of this implementation survive? (While I agree that the rethinking is a long way off, it seems closer with the impending switch to GitHub and the opportunity for greater collaboration, code reviews and the ability to easily shelve changes).

It seems like it would. You point to a directory in the repo directory as your script's script-root, and the app would do what was necessary. There would probably need to be some sort of "migrate to script 2.0 structure" function, and maybe a "discover old scripts", but basically we're stealing a lot wholesale from package managers.

On a practical basis there is a known, but as yet inconsistently reproducible, bug concerning the ScriptsMRU menu. There are circumstances where the menu should update but does not or that the update is somehow incomplete (only displaying a portion of the MRU list). Some thought should be given as to whether this new thing is always rebuilt every time the menu is displayed or whether there are listeners that trigger on disk and JSON changes or whether there is a hybrid approach. The choice might lead to a fix for the MRU issue.

I'd want to have a more typical menu structure, with registered scripts on the menu and a "Recent Scripts >" item that pops out and which users who do not want to use it can ignore. But if we search our data structure for registered scripts to run, prepending the script_id to the MRU list makes it a lot more manageable. We don't have to remember where the script is, just what it's id is.

 

[fronobulax]

I don't think this is a problem exclusive to a manifest/registration model (which seems a lot more like a package or plugin manager than a script manager with the way scripts have evolved). But I'm definitely open to ways to minimize risk. Perhaps a future version of svnrepo.json could have a cryptographic hash that we can verify and tell that at least it hasn't been tampered with.

I'm just coming at this from Hacking 101. If I want to run malicious code on your system I try and create and hide a directory to contain it. That does not guarantee the malicious code will run before detection but it increases the chances it will. As an Enterprise Manager an easy defense is to not allow a third party installer to create hidden directories. So I am just looking at a security concern because a third party can create hidden directories. Pretty generic. And it may be, given gausie's comments that I have incorrect expectations as to how this would work. Allowing the script author to specific that files they are installing will be hidden from the menu structure that runs scripts is a different security issue than allowing the system owner to say they no longer wish to see certain subdirectories included in the run scripts menu.

My thought is that I want to change how the scripts menu looks and works and how the scripts work under the hood and for developers. I still want the gCLI and actual CLI to be able to run a script.

The reason for my previous wall of text is that there are always two different menus to run a script and a user configuration can replace one of those with a third. So I was hoping further discussion would be more specific as to which one was being modified.

Perhaps this is about a brand new way that would add a menu option not currently available? Or maybe the Script Manager display that allows third party scripts to be installed (via svnrepo.json) gets a new tab, Run. and that new tab is populated with installed third party scripts?

My first thought is that there's a "register script" command, (or perhaps script register, for the noun-verb fans). Minimum data would be a filename, and most of the data would be null. if register script pointed to a manifest.json file, it would load that.

If we combine that with Gausie's statement that we only run registered scripts, we would treat the scripts the users registered as first class citizens like the ones we pulled in from the ScriptManager.

It could be that we have to compromise on that and register a script as it is run, but I'm not sure all of the implications of that.

I write a script, drop it in scripts. Run it once from the file chooser menu and the subsequently from the MRU. If I have to do anything else to run it then the changes that force me to do something else are not A Good Idea.

If I choose to run third party scripts in the same way that should be my choice.

So "only run registered scripts" concerns me. "only run registered scripts from the new, opt in system" doesn't concern me. See previous comments about "where" this is going to be done.

 

[MCroft]

I'm just coming at this from Hacking 101. If I want to run malicious code on your system I try and create and hide a directory to contain it. That does not guarantee the malicious code will run before detection but it increases the chances it will. As an Enterprise Manager an easy defense is to not allow a third party installer to create hidden directories. So I am just looking at a security concern because a third party can create hidden directories. Pretty generic. And it may be, given gausie's comments that I have incorrect expectations as to how this would work. Allowing the script author to specific that files they are installing will be hidden from the menu structure that runs scripts is a different security issue than allowing the system owner to say they no longer wish to see certain subdirectories included in the run scripts menu.

If we want to go back to Taltamir's original post, we can do what Unix systems have done for ages and just hide directories that start with ".", but we've sort of jumped past his request into the greater menu revamp discussion.

I don't see it that way, and I don't see any difference between this and allowing users to svn install from arbitrary URLs. The files from whatever project gets installed will be on the user's computer, regardless of if KoLmafia copies them.

But even if this causes risk our flat namespace makes this risk more serious. If I sneak a file into CHiT and I carefully call it wildfire.ash, we currently will overwrite Autoscend's (or anyone else's) wildfire.ash with my code. We do ask, but it's really easy to miss that CHiT shouldn't be writing Autoscend's path files. And then two months later when the player returns to that path and autoscend, they run the code. We already have potential issues with scripts.

The reason for my previous wall of text is that there are always two different menus to run a script and a user configuration can replace one of those with a third. So I was hoping further discussion would be more specific as to which one was being modified.

Perhaps this is about a brand new way that would add a menu option not currently available? Or maybe the Script Manager display that allows third party scripts to be installed (via svnrepo.json) gets a new tab, Run. and that new tab is populated with installed third party scripts?

In other menu paradigms, there may be a list of registered objects and a single "Open Recent" item which shows a cascading menu with some number of recent documents. Our options of "every file in the scripts directory, hierarchically" or "the last 5 or 25 things you used" seem to have room for improvement. The change I am thinking of is a menu of script names, sorted alphabetically, with a recent scripts item to take the place of the MRU style.

If a script was registered, it would have a name and might even have useful (ScriptManager) information as a tool tip.

We could work with unregistered scripts by adding them the old fashioned way.

I would think we would want to remove the deep dives into the heirarchy for dependent scripts that were probably not meant to be called at all.

I write a script, drop it in scripts. Run it once from the file chooser menu and the subsequently from the MRU. If I have to do anything else to run it then the changes that force me to do something else are not A Good Idea.

If I choose to run third party scripts in the same way that should be my choice.

I am going to suggest an open mind might be more conducive to design and not being seen as intending to shoot down ideas you personally don't care for.

I did say we should still support this, perhaps with a "register on first run" feature for exactly your case.

But we're not done thinking through it and you're pushing pretty hard into the XKCD "workflow" comic: https://xkcd.com/1172/ .

So "only run registered scripts" concerns me. "only run registered scripts from the new, opt in system" doesn't concern me. See previous comments about "where" this is going to be done.

We might need to have a hybrid at the start, because abandonware scripts may be useful and never get updated to have a manifest. We might never be able to stop having it.

I'm not convinced that we can't build this in a way that will satisfy you (and however-many users share your concern), but "I must never be asked to do anything differently" is an impediment to incremental improvements.

 

[fronobulax]

If we want to go back to Taltamir's original post, we can do what Unix systems have done for ages and just hide directories that start with ".", but we've sort of jumped past his request into the greater menu revamp discussion.

I don't see it that way, and I don't see any difference between this and allowing users to svn install from arbitrary URLs. The files from whatever project gets installed will be on the user's computer, regardless of if KoLmafia copies them.

But even if this causes risk our flat namespace makes this risk more serious. If I sneak a file into CHiT and I carefully call it wildfire.ash, we currently will overwrite Autoscend's (or anyone else's) wildfire.ash with my code. We do ask, but it's really easy to miss that CHiT shouldn't be writing Autoscend's path files. And then two months later when the player returns to that path and autoscend, they run the code. We already have potential issues with scripts.

No argument from me. Our script model has issues. Allowing a third party to tell KoLmafia to hide a directory at the OS level or exclude certain files from view magnifies the existing problems. But my original comment was intended as a throwaway aside and not a serious impediment to implementation, provided that the existing file chooser based "Load scripts..." menu option continues to exist and remains unchanged. (And that KolMafia does not support making directories hidden at the OS level).

We would be having a different conversation if the OP had asked for a new menu for launching scripts that excluded scripts in a specified subdirectory, or at least a more focused and less verbose one.

...

I would think we would want to remove the deep dives into the heirarchy for dependent scripts that were probably not meant to be called at all.

The need for a deep dive come from the design decision that if a user tries to run a script from the command line, KoLmafia will do all it can to find and run it. Accommodating the reasonable preference of JavaScript authors to have filenames that are not unique but need a portion of a path to be disambiguated contributes. We have no real way to determine which files contain scripts that were intended to be called directly by a user and which files are just needed by scripts.

I suspect Model View Controller might help clarify our thinking, knowing that we have a lot of work to implement that. The View that contains files that can be invoked directly by the user and the View that contains files that can be referenced by a script are different yet the existing code does not (and perhaps cannot, without additional work) make the distinction.

I am going to suggest an open mind might be more conducive to design and not being seen as intending to shoot down ideas you personally don't care for.

I find this offensive and a misrepresentation of my intent and motivation. I will accept that unclear writing on my part may have been misunderstood.

To try once more...

The current state is that a user selects the Scripts menu from the main display. They are then presented with a submenu that has

• Script Manager
• Load script...
• Refresh menu
• (Shift key to edit)
• A list of scripts that can be launched

Are we talking about

• Replacing this menu entirely?
• Adding a new item that will also display a list of script that can be launched, this one filtered by location?
• Changing Load script... to exclude selection from specified locations on disk?
• Unconditionally replacing the list of scripts that can be launched with a list filtered by location?
• For someone who is not using the MRU List, replacing the list of scripts with a list filtered by location?

IMO some of these are genuinely Bad Ideas but some aren't. Conflating the discussion of what a launch menu should look like with a major refactoring of the relationship between scripts and the file system probably isn't helping me.

 

[MCroft]

I did not mean to offend you. My apologies.

My thought is the menu would be something like this:

• Script Manager
• Load script...
• Refresh menu
• (Shift key to edit)
• Recent Scripts >(submenu I can’t model on the phone)
• List of registered scripts (including register-by-running them)

 

[Laserrobotics]

We would be having a different conversation if the OP had asked for a new menu for launching scripts that excluded scripts in a specified subdirectory, or at least a more focused and less verbose one.

As best as I can tell, Taltamir just wanted to be able to have them not show up in the dropdown list. I can see how it could be misunderstood though given that "hidden and not shown" is redundant, and one might assume that it therefore was actually referring to two separate actions.


For my two cents about a possible refactor of the scripts menu, I'd like it if there was a way for the individual user to remove a script from that scrollable list while keeping it installed, or some other method of customizing what scripts show up there.

I've got quite a few scripts installed at this point, but very few of them actually get run through that menu - some of them are run as commands with arguments, some of them are run by other scripts, some of them shouldn't be run at all, etc. Scrolling through the list right now takes a long time to get to the lower half, since it doesn't register the scroll wheel for this. Just turning it into a 'recently used' list would mostly solve it, but would then require using the file browser for any scripts rarely-used enough to not be kept in that menu.

 

[taltamir]

As best as I can tell, Taltamir just wanted to be able to have them not show up in the dropdown list. I can see how it could be misunderstood though given that "hidden and not shown" is redundant, and one might assume that it therefore was actually referring to two separate actions.

That is indeed what I meant

For my two cents about a possible refactor of the scripts menu, I'd like it if there was a way for the individual user to remove a script from that scrollable list while keeping it installed, or some other method of customizing what scripts show up there.

I've got quite a few scripts installed at this point, but very few of them actually get run through that menu - some of them are run as commands with arguments, some of them are run by other scripts, some of them shouldn't be run at all, etc. Scrolling through the list right now takes a long time to get to the lower half, since it doesn't register the scroll wheel for this. Just turning it into a 'recently used' list would mostly solve it, but would then require using the file browser for any scripts rarely-used enough to not be kept in that menu.

Very true, that would be very convenient as well. I was originally planning on relying on the authors to trim down the list of scripts (that currently is more than a full screen in height and requires scrolling through)

But having the user able to trim down that list without uninstalling scripts would be great too

 

[fronobulax]

What I called the "list of scripts that can be launched" is derived from the file system. It can be replaced by the MRU list by setting the preference scriptMRULength to some value greater than 0. On my system something around 30 keeps the scrolling from being activated. The list itself is maintained as a semi-colon delimited list stored in scriptMRUList. A file that is executed by user action - selected from a script menu, typed in the gCLI or called by cli_execute in a script - gets added at the head of the list and the list is trimmed from the tail to be the specified length. Scripts can be launched from this list.

There are some bugs. Launching fails if the target script is in planting/ or in a subdirectory of scripts. Sometimes something that was not intended by the author to be callable sneaks in. I have seen zlib on the list a few times but don't know why. At one time case sensitivity was not consistent and I don't remember if that was fixed, or not. Relay scripts don't appear, in part because they can't be launched from the MRU list at the moment. The list sometimes does not display properly for undiagnosed reasons believed to be threading or Swing issues.

The list is easily edited outside of mafia but using a script to manage it can get interesting because the managing script's name will be added. There will be timing behaviors associated with reading and writing scriptMRUList.

The MRU List was originally intended to address several of the issues raised above but the implementation has not kept up with changes that allow scripts to be anywhere besides scripts/ (and planting/ was missed from the beginning) or file names that are not unique across the file space and need pathnames to disambiguate.

Folks who log preference changes are particularly unhappy with the frequency of changes to scriptMRUList but I think the log changes logic was extended to have preferences to be ignored and thus not logged.

 

[fronobulax]

OK. So the high level design for the OP's request is to change (or create an alternative to) the "list of scripts that can be launched". The list will be driven by some to be determined data source. One candidate is to continue to build the list from the file system contents but provide a way to specify that selected files or directories be excluded. The other way is basically driven by a menu configuration file. There are several ideas about how to allow a script to be added to the configuration at installation or when first run.

I think I've got it?