Feature Support for scripting in JavaScript

ikzann

Member
I have been playing around with support for scripting in other languages. I have a preliminary version of JavaScript support working, based on Mozilla's Rhino project. If this ends up getting merged, I think this will enable some pretty cool things - like using modern web frameworks to build relay scripts, or by allowing easier programming of complex data structures and algorithms.

In the current draft, access to the ASH runtime library is available via an "ash" global in JavaScript. So, for example, the following is a valid JS script:

Code:
ash.print("abc" + ash.get_property("banishedMonsters"));

I am still working on the representation of proxy record values in JS, which to my mind is the single biggest open question. Curious for folks' feedback on that.

I know there have been security concerns with similar changes folks have proposed in the past. I think this works around most of them - the only objects available in the scope are the normal JavaScript objects available via rhino's "initSafeStandardObjects" function. There won't be any direct access to Java objects from scripts - just the ash global and standard JS class objects, none of which allow access to e.g. the filesystem.

I've attached the patch of my current draft. I've also put it up on my github for easier browsing, but my understanding of standard practice is that discussion stays here. This is a pretty complicated patch. One of the more irritating changes was refactoring the runtime-control aspects of Interpreter into a RuntimeController interface that both Interpreter and the new JavascriptRuntime implement. Happy to take any feedback folks are willing to give on this.
 

Attachments

  • javascript-v1.txt
    389.7 KB · Views: 2

zarqon

Well-known member
This is CLI Links after eating spinach! Would probably be a prohibitive amount of work for me to convert my relay scripts to use this, but am still following this thread with interest.
 

fronobulax

Developer
Staff member
This makes me apprehensive but a lot of that is my declining ability to assess security risks. While ikzann has not been a prolific contributor to kolmafia.us I note they did write the Cargo Shorts GUI and that establishes some level of familiarity and trust.

Reading the documentation I think initSafeStandardObjects will protect us from many opportunities for malice, for example deliberately modifying mafia objects and associated states.

I do wonder about access to the local file system. mafia spends a lot of effort to restrict that access. JavaScript can access local files. Can this be implemented to impose similar restrictions?

Similarly mafia restricts access to some URLs. Can those restrictions be imposed or is this a vector around them?

Thanks,
 

ikzann

Member
Thanks, frono, for those questions. JS only gives access to local files through the File interface, which is not initialized as part of initSafeStandardObjects. Similarly, fetch and XMLHttpRequest are not initialized. My intent is that the only way to access the outside world will be via ash.visit_url and ash.file_to_buffer/ash.file_to_map. You can see the list of things that are initialized in the Rhino source here.

The one aspect I'd flag from that list is XML parsing, but the Rhino folks appear to have configured it correctly to avoid the usual issues with XML - see this pull request for details. If you all feel it is necessary, I could entirely disable XML parsing, but I do think it will prove to be a very useful feature if available. For example, you can use it to parse pages coming down from KoL and make changes to the DOM instead of doing the usual regex-string-replace that relay scripts do.

As a bit of an intro, my IGN is "worthawholebean"; I've been around KoL off and on for the past decade, but mostly in the past 9 months or so. My personal scripting projects have mostly been writing ascension scripts, including a 1-day HCCS script and 1-day casual script that wraps autoscend. I haven't really advertised those for others to use, but they're there as a resource if anyone wants them.
 

philmasterplus

Active member
This is very interesting! The static typing of ASH gave me some trouble before, so I'll gladly welcome any means of dynamism. I assume it supports the CommonJS require()?
 

ikzann

Member
It does not support any module system right now, and I don't think that is a high priority given that folks could bundle scripts using an off-the-shelf bundler (maybe even Babel!) and distribute the bundle. As a comparison, that is how Guide is distributed in the ASH world.
 

fronobulax

Developer
Staff member
It does not support any module system right now, and I don't think that is a high priority given that folks could bundle scripts using an off-the-shelf bundler (maybe even Babel!) and distribute the bundle. As a comparison, that is how Guide is distributed in the ASH world.

Do you mean this Guide which KoLmafia publishes as being available via svn? Maybe we need a feature request to delete it from the Script Menu or deprecate it there and add the new one (using UR as a precedent/model, perhaps).

Snark aside, do your comments about modules and bundle apply to this patch (in which case, please explain/elaborate) or to the time in the future when artifacts that use this new capability will be distributed and shared? If KoLmafia's existing support is not going to be sufficient then there is probably more to this patch than discussed so far?
 

ikzann

Member
I was just using Guide as an analogy: it is written across multiple files and compiled into one file for distribution, as is the TourGuide fork. My current draft of the patch does not implement require or any other module system for JS, and I am arguing that such an import system is a low priority because folks can just use off-the-shelf module bundlers like webpack. Much like Guide/TourGuide, bundlers resolve module imports and bring all the code in a project into one file. I could, of course, be convinced that my instinct on prioritization is wrong. And now that I think about it, importing ASH scripts from JS would be extremely useful - but quite difficult to implement.

Does that answer your question? I might be confused as to what you're asking.
 

fronobulax

Developer
Staff member
What is a bundler? What is being bundled? Why does it matter in the context of this patch?

I don't write or distribute JavaScript and "module system" is not vocabulary I use. I infer from some searches that it is one solution to the dependency issue in JavaScript but I don't know how it relates to this patch.

My vision, perhaps incorrect, is that a user selects a local file, something knows or detects that the file is a JS script file and fires up the brand new interpreter. So does the thing that reads this file need to know about bundlers or is a bundled file plain text and support for unbundling already exists?
 

ikzann

Member
Apologies for not explaining. JS bundlers (webpack being the most prominent one) package a script with all of its dependencies into a single file that can be run in e.g. a web browser without installing any of those dependencies. The idea is that the user doesn't need to know anything about the dependencies of the script. Bundlers resolve all uses of imports, including require, so we wouldn't need to implement either of the two main options (import and require) for JS scripts in this patch.

Your last paragraph is correct. Bundled scripts are just javascript files without any dependencies. My goal here was just to point out a cool thing you would be able to do if a final version of this patch were merged, not to point out an implementation complexity.
 

Ryo_Sangnoir

Developer
Staff member
In git I'd at least want the renames in a separate commit, so I could more easily see the other changes. The replacement of "interpreter" with "runtimecontroller" adds a lot of noise to what I think is the important part of the PR.

Currently you've got an interface RuntimeController implemented by JavascriptRuntime and Interpreter. I don't like this in a naming sense: I don't think it has a shared language. I'd prefer an interface Interpreter implemented by JavascriptInterpreter and AshInterpreter (and the current NamespaceInterpreter becomes AshNamespaceInterpreter). If Interpreter needs to stay as-is, at least InterpreterController implemented by JavascriptInterpreter and Interpreter?

There's a lot of instances of "Interpreter" still in the codebase. "parsetree" is clearly related solely to ASH parsing; I assume that javascript scripts won't support debugging or "buy scripts" or "chat scripts" or "spading scripts" or many other things, allowing for the lack of replacement there: there'll be a focus on relay / runnable scripts.

Could you give an example of a script you've got running?
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
In git I'd at least want the renames in a separate commit, so I could more easily see the other changes. The replacement of "interpreter" with "runtimecontroller" adds a lot of noise to what I think is the important part of the PR.
Perhaps @ikzann you could work commit by commit in your git repository and then we can always squash it down to svn's resolution at merge time.
There's a lot of instances of "Interpreter" still in the codebase. "parsetree" is clearly related solely to ASH parsing; I assume that javascript scripts won't support debugging or "buy scripts" or "chat scripts" or "spading scripts" or many other things, allowing for the lack of replacement there: there'll be a focus on relay / runnable scripts.
While this seems to be true of the current work, I don't see why it couldn't be extended to support lifecycle scripts too.
 

ikzann

Member
Yeah, my intent was to submit a single patch to be merged - the github PR is just so people can use that UI to review changes if they wish.

Ryo, thanks for the suggestion on the naming. I'll do that.

Gausie is correct that lifecycle scripts are not currently supported but certainly could be. This patch is still very much a work in progress, so I don't have anything very complicated working yet. I'm trying to ensure that devs here are OK with the general direction before I go any further down the garden path.
 

fronobulax

Developer
Staff member
Lifecycle scripts? You realize using jargon is a classic way to exclude people :)

I am getting more and more confused about the purpose and use of this.

I can currently write an ash script to be executed in the relay browser. I have access to the ash runtime library. I can use HTML in the script although I have to "wrap" it in write statements.

The obvious generalization would be to allow my ash relay script to access things written in JavaScript. That gives me lots of tools and it is likely easier than embedding HTML line by line in a write statement.

But I don't think that is what is being proposed.

So I flip things. This lets me write in JavaScript. It allows me to access ash functions that are currently in the RuntimeLibrary. Something will allow me to select and execute ash enhanced JavaScript in the relay browser. So someone who knows JavaScript can use JavaScript to build a relay script (instead of having to embed HTML).

If that last is correct, I'm not sure why ash scripts that have certain functionality need to be a special case. Is ash.cli_execute("<my script or command here>") not going to be supported? Are there conventions associated with scripting jargon that allow or prohibit certain types of actions in scripts of a particular type? (For example, ash scripts of "type" headless should not expect to get input from the user during execution).

To clarify, or regurgitate what I thought I heard - a JS script for KoLmafia will be a text file that will be read and executed by this capability. It is the responsibility of the script author to create the text file in such a way that everything needed to run the script is available in the text file. "bundling" is an action taken by the script author and the new capability just reads and processes a text stream.

I am fundamentally lazy but I probably should just suck it up and apply the patch. If I do, do you have a simple test script so I can "play" before I relearn all the JavaScript I have gladly forgotten?
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
Lifecycle scripts? You realize using jargon is a classic way to exclude people :)
This is literally a phrase I invented on the spot to describe those scripts because it seemed self-explanatory. Sorry if I excluded you or anyone else in doing so, that was not my intention
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
@ikzann I think this is really cool and extremely useful. If you can solve the issue of aggregates and proxy records I would be happy to test and commit it. Would you like any help with anything?
 

ikzann

Member
I think maybe we've gotten a little distracted by the discussion on bundlers? I think your overall understanding is correct, fronobulax. This patch is a first attempt at making JavaScript a second option for writing scripts for Mafia. Eventually, I'd like to have all avenues for executing ASH scripts also support JS, so that would include relay scripts, consult scripts, event-triggered/lifecycle scripts, and the current route via the CLI. The patch I posted is a work-in-progress and shouldn't be committed yet - just trying to confirm that folks here are OK with the approach before I work on it further.
 

fronobulax

Developer
Staff member
This is literally a phrase I invented on the spot to describe those scripts because it seemed self-explanatory. Sorry if I excluded you or anyone else in doing so, that was not my intention
Well the snark is strong today but since it was immediately used in response I figured I had a chance to learn from two people. No harm, no foul and I guess no definition either :)
 

ikzann

Member
Oh, I assumed that meant what Ryo had mentioned: scripts triggered via events in Mafia like the betweenBattleScript etc.
 

fronobulax

Developer
Staff member
I think maybe we've gotten a little distracted by the discussion on bundlers? I think your overall understanding is correct, fronobulax. This patch is a first attempt at making JavaScript a second option for writing scripts for Mafia. Eventually, I'd like to have all avenues for executing ASH scripts also support JS, so that would include relay scripts, consult scripts, event-triggered/lifecycle scripts, and the current route via the CLI. The patch I posted is a work-in-progress and shouldn't be committed yet - just trying to confirm that folks here are OK with the approach before I work on it further.

Agreed on the distraction.

Is

"This lets me write in JavaScript. It allows me to access ash functions that are currently in the RuntimeLibrary. Something will allow me to select and execute ash enhanced JavaScript in the relay browser. So someone who knows JavaScript can use JavaScript to build a relay script (instead of having to embed HTML)."

basically correct?
 
Top