Feature ASH language server features

fredg1

Member
MCroft, BigD and I are planning to try to add Language Server features to mafia, so that making scripts is easier. This would be done using Language Server Protocol for (4) Java.

We're starting with ASH (we'll have to see how adding the javascript part goes afterwards).

The first problems encountered are :
  1. textui is very poorly documented. I'm currently trying to understand it as best as I can, but if possible, can anyone with a greater understanding of this explain the inner workings of textui/parsetree? More precisely, Scope, BasicScope and ParseTreeNode, if possible?
  2. I noticed that script, notify, since and/or import statements are allowed at the start of literally any block, like so:
    Java:
    script this_script_is_called_abc.ash_now;
    notify Fronobulax;
    since 20.1;
    import a.ash;
    import b.ash;
    
    static {
        script this_script_is_called_aaa.ash_now;
        notify Veracity;
        since 20.2;
        import c.ash;
        import d.ash;
    }
    
    string a_funtion() {
        script this_script_is_called_foo.ash_now;
        notify MCroft;
        since 20.3;
        import e.ash;
        import f.ash;
        
        for x from 1 to 10 by 5 {
            script this_script_is_called_bar.ash_now;
            notify Gausie;
            since 20.4;
            import g.ash;
            import h.ash;
        }
        
        return "";
    }
    
    {
        import i.ash;
        {
            import j.ash;
            {
                import k.ash;
            }
        }
    }

    (the script name and notify recipient don't actually get changed, since they can only be set once per script)

    Is this what-I-can-only-call-an-abheration intentional?
 

fronobulax

Developer
Staff member
Who is BigD? :)

How does LSP make writing scripts easier? In particular how does it help a writer who only has a text editor and no understanding of the kinds of things people learn in My First Programming Course?

The ash design was somewhat ad hoc and added to. Does the fact that it does not conform to any grammar or set of language rules mean LSP is doomed to be riddled with exceptions and special cases or does this mean ash is going to be regularized?
 

fredg1

Member
ASH will not be changed. This is an attempt at letting mafia tell users what its language, ASH, can do. This is not an attempt at imposing a standard on ASH.
 

MCroft

Developer
Staff member
Fred is too modest. I am consulting on distributed computing, and am interested as a possible consumer.

But it's his project.
How does LSP make writing scripts easier? In particular how does it help a writer who only has a text editor and no understanding of the kinds of things people learn in My First Programming Course?
ASH syntax highlighting was the feature I picked up on, with the possibility also of autocomplete as well. These seem like useful things for newbies, but you have to be the kind of newbie who gloms onto an IDE. If that's who you are, then LSP seems to be a way to write syntax highlighting and autocompletion once and not once-per-editor. Also, Fred's plan, IIRC, will regenerate the syntax from a mafia build and thus keep the LSP up to date.
The ash design was somewhat ad hoc and added to. Does the fact that it does not conform to any grammar or set of language rules mean LSP is doomed to be riddled with exceptions and special cases or does this mean ash is going to be regularized?
I wouldn't think it would be a problem. There are more complex languages with (many) more ways to do things that succeed at this. Java. PERL. BASH.
 

fredg1

Member
The ash design was somewhat ad hoc and added to. Does the fact that it does not conform to any grammar or set of language rules mean LSP is doomed to be riddled with exceptions and special cases or does this mean ash is going to be regularized?
LSP is only a communication protocol. The requests communicated are only things like "send me all the references of *this* variable", "generate the folding for this file", and things like that. While some of these are incompatible with ASH, they are NOT intended to be forced on the language server. The language server is the one to tell what feature it supports, and it implements whichever it wants.
LSP4J only handles the implementation of LSP in java, i.e. handling the communication between server and client. We're the ones who decide how we implement the server.
I wouldn't think it would be a problem. There are more complex languages with (many) more ways to do things that succeed at this. Java. PERL. BASH.
That, too.
 

MCroft

Developer
Staff member
My main concern is: who *made* the docs? Is it someone who truly *knew* what ash was supposed/wants to do, or is it someone who *interpreted* it?
Well, turns out that hola wrote notify and hola wrote that text on the forum, and Zarqon copied hola's text from 2006 into the wiki in 2009. But in general, we only need to rely on the wiki for things we can't deduce from code.

But it would also be really handy to see where they two diverge (as in the case you mention). Again, recommend we split this off, because it's really a different topic...
 

philmasterplus

Active member
Intriguing. Sorry for potentially driving the thread off course, but what are your end goals? I thought LSP was for implementing editor features, such as syntax highlighting, code suggestions, navigation (go to declaration/find references...), and refactoring. If LSP was built into KoLmafia, does it mean that when I edit an ASH file in VS Code, a background Java process would provide such features?
 

fredg1

Member
I thought LSP was for implementing editor features, such as syntax highlighting, code suggestions, navigation (go to declaration/find references...), and refactoring.
LSP, Language Server Protocol, is, as the name says, just a protocol dictating broadly how the *communication* between a client and a server goes, but that's about right.

The server gets from the client (for example, a VS Code extension) which file gets opened/when they get edited, when the user wants to go to a symbol (variable/function/etc)'s definition/references, when the user wants to refactor something, etc., and returns the respective expected information (locations of the definition/references, suggested refactoring, errors/warnings/infos, or "hey-why-did-you-ask-for-this-I-told-you-I-didn't-support-that-go-to-hell")
If LSP was built into KoLmafia, does it mean that when I edit an ASH file in VS Code, a background Java process would provide such features?
Again, LSP isn't what's being "built in" mafia, but basically, yes (if you have an extension that can act as a language client).
 

philmasterplus

Active member
This is exciting news, but I'm still puzzled about your roadmap. Are you planning to build an editor plugin for ASH? What features can we expect?
 

fredg1

Member
Frankly, I don't know!

We'll start by having it simply send warnings and errors (the only thing Parser currently does), then we'll go from there.

Almost every feature will require an update of Parser to make it able to log where symbols and tokens are. We'll probably go for that, see what doors this opens us (if any), and then see what next is within reach, then what next is within reach, then what (...)
 

fronobulax

Developer
Staff member
I remain totally clueless as to the need, benefits and motivation.

Client, server and protocol are mentioned. What is the client? What is the server? If this is an internal feature, then why is client-server the architecture choice? If this is an external feature than what's outside?

Is the goal an IDE for ASH? Perhaps a symbolic debugger for ash? A visual editor that builds ASH from drag and drop operations? Is whatever is being planned static - it is driven by the ASH specification or dynamic - it won't work unless there is a running instance of KoLmafia available?
 

Ryo_Sangnoir

Developer
Staff member
I remain totally clueless as to the need, benefits and motivation.
In user story style, I would say:

As a script developer, I want my editor to syntax highlight ASH scripts and provide autocomplete functionality, so that I can develop scripts more easily.

The LangServer Protocol (LSP) is a protocol that aims to allow a language community (e.g. KoLMafia ASH users) to develop a single library which works across many editors (e.g. VSCode, vim, emacs, atom) without having to develop a plugin for each editor. I presume the goal is to develop a single language server (LS) and plugins for frequently used editors.

One example of something that may be produced from this effort is a VSCode extension that would allow a developer to edit ASH files in VSCode with syntax highlighting and autocomplete. This could also be written directly, without involving a language server.

The client is the editor, such as nvim (or a plugin associated to that editor). The server is something that gets created that a client can send messages to and receive responses from. I would have guessed this was entirely separate from Mafia, but I'm not clear based on what's been said in this thread.
 

fredg1

Member
This could also be written directly, without involving a language server.

The client is the editor, such as nvim (or a plugin associated to that editor). The server is something that gets created that a client can send messages to and receive responses from. I would have guessed this was entirely separate from Mafia, but I'm not clear based on what's been said in this thread.
The language server would be integrated into mafia.
It "could" be something completely different, but doing it like this allows it to stay up-to-date.

Is whatever is being planned static - it is driven by the ASH specification or dynamic - it won't work unless there is a running instance of KoLmafia available?
I initially thought that it would require a running instance of mafia, but we could simply make it possible to launch mafia with a different "main" class, which would launch it in Language Server mode.
 

fronobulax

Developer
Staff member
In user story style, I would say:

As a script developer, I want my editor to syntax highlight ASH scripts and provide autocomplete functionality, so that I can develop scripts more easily.

The LangServer Protocol (LSP) is a protocol that aims to allow a language community (e.g. KoLMafia ASH users) to develop a single library which works across many editors (e.g. VSCode, vim, emacs, atom) without having to develop a plugin for each editor. I presume the goal is to develop a single language server (LS) and plugins for frequently used editors.

One example of something that may be produced from this effort is a VSCode extension that would allow a developer to edit ASH files in VSCode with syntax highlighting and autocomplete. This could also be written directly, without involving a language server.

The client is the editor, such as nvim (or a plugin associated to that editor). The server is something that gets created that a client can send messages to and receive responses from. I would have guessed this was entirely separate from Mafia, but I'm not clear based on what's been said in this thread.

Motivation - edit (and run?) ash in an IDE like environment? Fine. Not clear why mafia has to be changed to support that but that could be me interpreting questions as Requests For Change.
 

fronobulax

Developer
Staff member
The language server would be integrated into mafia.
It "could" be something completely different, but doing it like this allows it to stay up-to-date.


I initially thought that it would require a running instance of mafia, but we could simply make it possible to launch mafia with a different "main" class, which would launch it in Language Server mode.

To the extent that ash features are more static than dynamic, why integrate into mafia to keep things up to date? Why not a separate utility or code base that looked at mafia source and updated the spec in response to observed changes? How does LSP in general handle externals (in an IDE context)? I can imagine and IDE that knew about zlib but is that knowledge derived dynamically based upon where, or if, zlib is in my script file system?

Presumably if this were done right there would be a plugin that could be created that would extend language support in IntelliJ or Eclipse to ash?
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
I think this is a great idea! Thanks for working on it. @fredg1 I know you like to be very thorough but if I were you in this instance I would leave frono's questions until the work is done so he can learn by using
 

fronobulax

Developer
Staff member
I think this is a great idea! Thanks for working on it. @fredg1 I know you like to be very thorough but if I were you in this instance I would leave frono's questions until the work is done so he can learn by using

Well if a requirement to understanding this is to use a new tool to create ash scripts then it is probably not how I will get answers. Questions answered now keep a civil war from breaking out among those who currently have commit access at SourceForge if the results are committed there ;-)
 

Ryo_Sangnoir

Developer
Staff member
How does LSP in general handle externals (in an IDE context)? I can imagine and IDE that knew about zlib but is that knowledge derived dynamically based upon where, or if, zlib is in my script file system?

Presumably if this were done right there would be a plugin that could be created that would extend language support in IntelliJ or Eclipse to ash?
To use the LS, you open a script file in an editor. If this file uses zlib, there will be a line "import <zlib.ash>;". Mafia reads that, figures out where it should import zlib from, and includes all of zlib's functions at the top level. You can then use these functions in your script. Similarly, the LS looks at the file, sees the line "import <zlib.ash>", finds the zlib file, and marks all the zlib functions as available and coming from zlib (and what parameters they take, etc.). None of this is hardcoded: it can work for any file you choose to import.

Both IntelliJ and Eclipse have LSP clients, so after this is done you should be able to create a plugin that lets them communicate with the LS, yes.
 
Top