Feature Extend KoLmafia with other scripting languages

lostcalpolydude

Developer
Staff member
I'm not very familiar with python, or jython... but my first thought is to wonder about the security of adding this. If someone has jython for some other reason, does this suddenly allow scripts to do harmful things outside of influencing a KoL account? I've worked pretty hard to make sure nothing outside of KoLmafia's files are accessible (on the local system) from scripts, and I can't tell if that's true with this.

On a separate note, I see that all of the leading whitespace is spaces instead of tabs in the new file.
 

Sytras

Member
I can't speak to the security implications, but it would be very cool to be able to write python scripts for automating mafia instead of needing to learn ash and its idiosyncrasies.
 

Veracity

Developer
Staff member
I’ve never been able to advance very far into Python, given its (numerous) idiosyncrasies.

To each their own! :)
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
Accidentally deleted my post trying to edit it ugh mobile. Anyway in summary - security concerns can be addressed, didn't consider someone might have jython in their default classpath and thus not knowingly opt in to the experiment. I will revert this until I can sort the AccessController
 

fronobulax

Developer
Staff member
When I supported a desktop Java application with BeanShell embedded, all of the heavy lifting was done by knowledge of the application's internals. Is jython interaction with KoLmafia done in a similar fashion - i.e. the script writer is going to have access to KoLmafia Java objects and classes?

I'm in violent agreement that security issues should be addressed before (re)releasing this.
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
When I supported a desktop Java application with BeanShell embedded, all of the heavy lifting was done by knowledge of the application's internals. Is jython interaction with KoLmafia done in a similar fashion - i.e. the script writer is going to have access to KoLmafia Java objects and classes?

I'm in violent agreement that security issues should be addressed before (re)releasing this.

Current plan is:
1) Limit the jython jar using a security policy to not do.... well not do anything really
2) Only allow python files access to classes under net.sourceforge.kolmafia
3) Write a standard library in Python to bridge the gap API wise

It might take a while to get this all in place so sorry for the early fanfare!
 

Veracity

Developer
Staff member
You are proposing to allow Python scripts access to any class/method in the Kolmafia source tree?
Shit. Time to make a major pass over the source declaring a buttload of methods “private”, I guess.
 

Veracity

Developer
Staff member
And all the work we’ve done to NOT expose internal functions to enable mallbots? I guess that goes away if we allow access to internal functions.

So far, I am hating this.

Can you explain exactly what your motivation is for this feature? What do you hope to provide that ASH doesn’t provide?

If it is simply catering to people who are too lazy to learn ASH, fuck it.

What IS your plan?
 

Veracity

Developer
Staff member
gausie, please create a Feature thread where we can discuss this.

Are you proposing to give python scripts access to internal KoLmafia data structure? read and write? If so, perhaps, I need to to look at every single data structure and declare lots of fields "final" so that rogue python scripts can't fuck us over? Probably a good idea anyway, rather than assuming that internal code will not fuck us over, but you are adding a new threat vector.

Are your proposing that Python scripts will be able to call any "public" method in KoLmafia, as opposed to only those that we explicitly encapsulate in RuntimeLibrary functions or ProxyRecords?

If so, I preemptively ask fewyn for a new forum response code:

We have "Not a Bug". I respectfully request "Fuck You, Not a Bug" to be used as a response for bug reports when an external Python script calls an internal - but public - method - which has gone away when I (for example) refactor the source and fix all the internal users. External (python script) users who are affected? Sucks to be you if I change internal functions and your script breaks. You going to write a bug report about it? I guarantee you will. Thererefore, I want a "Fuck you, Not a Bug" response to close, with prejudice, the ineveitable bug reports.

Open a Feature. Explain EXACTLY the problem that needs to be solved, what your proposed solution is, and exactly how it will NOT give more internal access to functions and data structures than ASH currently provides - or how, exactly, you will introduce additional access which will not allow user-provided scipts to completly trash our internal model or do things that we intentionally restrict.
 

fronobulax

Developer
Staff member
So my previous post was supposed to be an intro but things are sorted chronologically.

If I understand relay scripts they do allow arbitrary, user written JavaScript so they are a threat vector, correct? The threat is to the user's system but not to KoLmafia internals, except what is exposed by ASH. correct?

I am not sure I understand the motivation for this request. Is there something people can't do in ASH that they can do in Python or are they expressing a preference to script in Python rather than ASH?

If Jython is going to have access to "mafia internals" I do not see this as a Very Good Thing and consider the effort to protect the internals from rogue or malicious scripts to be a lot of work for very little benefit.
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
I'm not necessarily proposing that, I didn't fully think through the consequences of giving access to everything inside net.sourceforge.kolmafia.* I certainly don't want to introduce attack vectors. I think my key issue here was doing anything experimental without starting a discussion here first as I would have heard all these arguments! It would be easy to expose a single class to Jython (much like RuntimeLibrary) and just do it that way.

In terms of why one might want to use Python instead of ASH, there are lots of reasons. I write ASH all the time and frequently find myself wishing I could use it differently. I have tried lots of times to extend the language in various ways, or improve its extensibility but I always hit my own skill ceiling. To be more specific, lambdas, dynamic typing, iterables, "catch", the wider python ecosystem if possible, namespacing, classes, optional parameters, mixed-typed collections and much more. I just find myself hitting an ASH ceiling often and I was trying to carve a cool new future for mafia.

I accept some of the mistakes but I stand by my desire to make scripting easier and more powerful. Anyway, sorry to have caused so much concern and I hope you'll take my speedy reversion as proof that I'm not trying to do anything untoward!
 
Last edited:

Aenimus

Member
I think there are valid concerns here, and I believe another large issue is that gausie didn't properly discuss his intentions with other devs(?).

But I vehemently disagree with Veracity regarding laziness to learn ASH. I learned ASH before any other language. Now, I'm not saying I am amazing or even good at ASH, but I can mostly do what I want it to do. I appreciate the thought, time and intelligence that people poured into ASH over the years to make a scripting language for Mafia, and it is (and probably always will be) a fantastic and pivotal addition to the project.

But I don't think anyone here can defend ASH as a "ideal" language. It's a limited means to an end, and it lacks a plethora of features that you can find in most non-propriety languages. One of the best examples I can think of are classes. The closest thing ASH has, as far as I know, are records. There's a reason I wrote my KoL simulator in Python and not ASH. In fact, would anyone ever choose to write in ASH if they had a choice?

If it's possible to address any security or exploitation concerns, I don't see why being able to use Python to script in Mafia would be anything but good. Heck, it might allow people to learn something useful outside of a program created to play KoL.
 
Last edited:

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
I guess the point is - I am interested in interacting with mafia through Python and I'm willing to put the time into making it happen in a safe and maintainable fashion. I'm not trying to be a cowboy or anything and I just made a mistake putting anything into SVN so early.

EDIT: I think I will just park my efforts for now, so there's no need to be concerned even if you disagree with me. I'm sorry for causing an issue.
 
Last edited:

fronobulax

Developer
Staff member
I have a set of guidelines by which I evaluate KoLmafia proposed features. These include but are not limited to:

  • No mall bots.
  • Reduce server hits compared to doing the same thing without KoLmafia.
  • Think carefully before adding a feature that can be done by a script.
  • Don't make the user's system any more vulnerable because they run Kolmafia and scripts it supports.
  • Listen to Veracity.

So the first problem with this suggestion is just that it is a surprise and not especially well thought out.

After flogging gausie/innerdreams with a wet noodle for not discussing this first, and complaining about people who want "me" to do work because "they" can't be bothered to learn ASH or "they" can't be bothered to propose changes to deal with ASH limitations, I can see some merit. Or at least an opportunity for peaceful coexistence.

So what I wonder about is whether there is a way to limit jpython to only accessing things that are exposed to ASH via Runtimelibrary? If so then I am comfortable with spending the effort to lock down runtime library and then exposing it to jpython. The goal would be that any exploit in jpython was also an exploit in ASH and both could be fixed when discovered.

I also note that projects evolve. This may be the first stage of a transition from the devs who worked with hola and are ok with SourceForge and SVN to those who think "hola" is a greeting in Spanish and git is the way best way to go.
 

Aenimus

Member
Frono, the Jython implementation has been completely dropped indefinitely. My understanding was that jython was a way to allow everyone to use Python to script without having to install it. But now I believe it's a case of needing Python installed if you want to use Python to script in Mafia.

P.S. listening to Veracity is definitely something I can concur with.
 
Last edited:

fronobulax

Developer
Staff member
Frono, the Jython implementation has been completely dropped indefinitely. My understanding was that jython was a way to allow everyone to use Python to script without having to install it. But now I believe it's a case of needing Python installed if you want to use Python to script in Mafia.

P.S. listening to Veracity is definitely something I can concur with.

gasusie/innerdreams did not make it clear the idea was abandoned. I infer from your comment that it was not just their idea but since they have commit access and did commit and revert...

But if it is off the table then fine. I was looking to see if there was a way ahead.

I won't lock this but it sounds like we are done. Thanks.
 

Veracity

Developer
Staff member
I may gotten more vehement than was called for. I apologize.

My irritation came from not understanding what the intent was for this feature since it was inserted without any advance discussion. Speaking for myself, when I have a "big"-ish idea, I tend to make a thread in Scripting Discussion where I lay out my thoughts and ask for feedback. For example Overriding methods with typedef arguments or Implicit conversion to strings.

I don't understand where (some of you) thought I was getting defensive about ASH. It is certainly a smaller language than, say, Java. I have certainly felt limited, sometimes, when coding in it. Fortunately, I have the power to improve it for my (and everybody's) advantage. heeheehee has done that as well; if I recall "array literals" were theirs, originally. I extended them to general maps and made them work in more contexts, and they are the cat's meow, to my eye. But I'm quite sure that I didn't intend to imply that ASH is the "one true language for scripting KoLmafia" - especially since it is "one of two languages" we provide. :)

But a strength - or weakness - of ASH is that it has a controlled and curated access to which features of KoLmafia - and therefore KoL - you can access.

The security issues that lost brought up? You can read or write files - from within KoLmafia's data directories, but cannot use general file system operations. We closed a security hole, years ago, where a script could set the "command to invoke an editor on a script" property to any shell command and then say "edit this script", which would execute that shell command. I know that Python has tons of libraries and built-in support for files and such. I (and lost) worried that if Jython has such file system access, that would reopen up that security hole

I looked a little at the doc for Jython and saw that it lets you "import" any Java class on your classpath. I note several issues with that. The first of which are security issues of the sort frono pointed out.

For example, KoLmafia certainly has the internal ability to do mall-bot stuff; when we look up the mall price for something, we get a page of results from KoL. You can see that on the Purchase frame. If you tell KoLmafia to buy something, it will get you the cheapest one. But if you ask for the current price, you get the fifth-cheapest price. You can't use that to detect a single mis-priced Mr. A, for example, and snatch it up.

If Jython can import any class in KoLmafia and call any method from a class, boom; that protection goes away.

Similarly, we also prevent chat scripts from spamming public channels - or harrassing individual players in chat. I don't know for sure, but I suspect we have internal methods which could be used for such.

My second observation is what prompted some of my vehemence: If Jython scripts can call ANY method on ANY class, that turns into a potential maintenance nightmare for the devs. If a knowledgable user sees a "useful" internal method they want to call, their expectation will be that it will persist. What if I - or any dev - wants to completely refactor some internal aspect of KoLmafia? I did that for paths and tower door locks recently, for example. I see old old code where we have arrays of Objects, which are eminently suitable for conversion to classes or enums. Well, some of those correspond to ASH data types. If I refactor the code to make it better and more maintainable Java, I fix ASH. But Jython scripts accessing such data (without the benefit of a built-in data type like ASH provides) will break. Or, perhaps I change an internal facility and add an argument to a method. I fix all the Java inside KoLmafia. I do not fix your Jython script. And I really, really, really do not want to get bug reports - or even simple forum posts - asking me to explain - or, worse, justify - such changes. If you are familiar enough with the codebase to call methods inside KoLmafia, you are familiar enough to follow change logs and figure it out yourself - until you publish (or otherwise share) your Jython script with a "civilian". So to speak. :)

We can discuss all of thse things. Let's just have the "specification" and "design" discussions up front.

Thanks.
 

Aenimus

Member
I am certain that the jython-implementation has been dropped, but implementing a way to use Python (3.8) has not. Gausie mentioned to me that he would work on non-public implementation (as would I) while there is discussion with other Mafia devs regarding any concerns they might have.

To be clear, this non-public implementation does not use jython and requires the users to have Python installed on their device. I don't believe there is any intention to make features protected behind ASH accessible with Python. We just want to script more easily/achieve scripting methods ASH currently cannot handle.

If any of the above is incorrect, I am sure Gausie will correct me in the morning, and I apologise in advance.
 
Top