Scripts in tests

[fronobulax]

I know we have a framework for having a test that runs a script and then compares the output with the expected output.

I haven't really wrapped my head around how. If I were going to do it (which makes a lot of sense in my Coverage++ effort) I would want to know where to put the script (/test/root/scripts) and the expected output (/test/root/scripts). What role do /test/root/data. images, relay and settings play? Are there file naming conventions to be followed? If I am reading CustomScriptTest correctly then I should be able to just drop in the script and expected output and my script will be run as a test, correct?

How do I set up the "mafia conditions" that would be part of the test environment? I've seen examples of adding items to inventory but how/where would I do that as part of the setup for CustomScriptTest? If I remember how to Mock instead of having a logged in character or comms with KoL where do I set it up for a script?

Is the answer write my own test and try not to break CST which is driven by directory contents?

Thanks.

 

[heeheehee]

I haven't really wrapped my head around how. If I were going to do it (which makes a lot of sense in my Coverage++ effort) I would want to know where to put the script (/test/root/scripts) and the expected output (/test/root/scripts).

Script goes in test/root/scripts, expected output goes in test/root/expected.

In short, it uses the method that handles script resolution (for calling or imports), then calls all of the scripts it found. It then redirects the script output into a buffer, which it then compares against the corresponding expected output file.

There aren't really any naming conventions at this time (other than autodetection of expected output as input_script.ash.out). The "test_" prefix is redundant, and I would prefer if that didn't become a convention.

What role do /test/root/data. images, relay and settings play?

none, beyond the role that they might normally play (e.g. loading datafiles, or configuring preferences). I think we autocreated those directories when I first set up CustomScriptTest.

How do I set up the "mafia conditions" that would be part of the test environment? I've seen examples of adding items to inventory but how/where would I do that as part of the setup for CustomScriptTest? If I remember how to Mock instead of having a logged in character or comms with KoL where do I set it up for a script?

The `test` CLI command is likely useful here.

In particular, it seems that there's a magical `inventory (add|remove)` subcommand that allows you to manipulate the state of your inventory. (There's also `test stats` that let you manipulate your stats, `test equipment`, and possibly a few others.


I have two overall comments about this thread:
1. This points to a need for better documentation.
2. I think this would be broadly useful for anyone who's interested in contributing patches. Thoughts on moving it to, say, Scripting Discussion?

 

[fronobulax]

Trivial lessons learned so far.

Put script file in test/root/scripts named <something>.ash

Put file containing expected output in test/root/expected named <something>.ash.out

Run CustomScriptTest.

The Clover optimization does not understand that a new or changed script or output file means CustomScriptTest needs to be rerun. So I just ran that one test from the IDE until I needed to see code coverage changes and then I "cleaned" Clover and than ran it for coverage.

The easiest thing for me was to run the script from KoLmafia until I was happy with it and then copy/paste gCLI output from a run into the expected file. Don't forget to get script changes from the mafia runs back into test/root/scripts

I had my best results with a script that would run when I was not logged into KoLmafia. That mean the script was not relying on dynamic or character specific data.

I scripted the cli test command because TestCommand.java had very little test coverage.

Since my focus is test coverage and not functionality I can see using this technique to cover portions of RuntimeLibrary.java. Some of the date, time and string manipulations don't depend upon dynamic data so could be scripted for a coverage test.

Conceptually we ought to be able to use Mockito to provide dynamic data to a script but I have not figured out a framework yet. CustomScriptTest is driven by the directory contents. I can imagine a parameterized test where the parameters are script file name and Mocked objects. But my imagination is not leading to anything concrete yet.

 

[heeheehee]

The Clover optimization does not understand that a new or changed script or output file means CustomScriptTest needs to be rerun. So I just ran that one test from the IDE until I needed to see code coverage changes and then I "cleaned" Clover and than ran it for coverage.

That's accurate.

r20291 disables test optimization for tests with external dependencies such as this one, as well as the various DataFile* tests.