KolMafia Pre-Release Test Script

cjswimmer

Member
I'm not sure if this is the right place to start this idea, but I've been having some concerns with bugs that have appeared in the latest releases of KOLMafia. There are multiple developers that can check in changes to the program, and I know that one of the senior developers reviews the changes before allowing them into the final version, but I think what might be missing is a comprehensive test script for the entire application that is run through during a code freeze period before releasing a new version. The term 'script' in this context does not refer to the files that are executed by KOLMafia to perform certain actions in the game, but refers to a checklist used by people to test all the major functional components of the program itself.
I manage several large web application development projects at my job, and we rely heavily on test scripts for component and regression testing when we're releasing new versions of our apps. The test scripts consist of Excel spreadsheets that are sent out to customer focus groups for them to run through and record their findings. We are in the process now of setting up QuickTest Pro to automate our regression testing procedures, but for now its a lot of manual work. FYI, regression testing refers to the testing of older components of an application to make sure they still work alongside new components or enhancements. Now, creating test scripts is the least favorite part of my job. They are incredibly time-consuming, and often logistical nightmares to organize depending on the size of the application. they are however, crucial to the success of our releases.
I am willing to offer my support in the creation of such a test script for KOLMafia.

The first step is the identification of all the major interfaces of the application (i.e. Adventure tab, Graphical Cli, Scripting System etc.)

We then identify groups of the functional components in the app(i.e. Inventory Management, Adventuring, Spellcasting, etc.).

Those components can be broken down into groups of Actions (Using Items, Moving items between storage locations, casting buffs, ...).

After that you can define logical test cases which have a list of expected results and actual results based on conditions
For buying a item:
Conditions: enough meat on hand, purchase location is accessible
Expected result: meat on hand reduced by item cost
Expected result: item added to inventory

You further define your logical test cases to include sample data and the interfaces defined in the first step:
buy a cool whip using the mall tab interface
buy a cool whip using the Graphical Cli
buy a cool whip using the web browser
buy a cool whip using ash script
...
and you show your results in each interface as well
character pane shows meat reduced by 30
ash script my_meat() shows meat reduced by 30
...

We'll want to make sure we include as many expected failure conditions in the testing as well (i.e. buying from an NPC store that is not accessible, equipping items you don't have etc..).

As you can see, this is one huge pain in the ass. The nice thing about it is, you only have to do the bulk of it once. After you make changes to the program you just have to make new test cases for the new functionality, but all your regression test plans are done already. I'm not sure if it would work for this type of app, but there might be an open source automated test program that could be used for regression testing so that the workload is reduced significantly for each release. Maybe its the glue I'm sniffing, but I'd be really interested in taking part in this kind of undertaking. Let me know if this sounds interesting to you and I'll put together a post similar to this in the Mafia dev forum.

cj-"biting off more than he can chew"-swimmer
 

Veracity

Developer
Staff member
[quote author=cjswimmer link=topic=141.msg672#msg672 date=1146770156]
I'm not sure if this is the right place to start this idea, but I've been having some concerns with bugs that have appeared in the latest releases of KOLMafia.  There are multiple developers that can check in changes to the program, and I know that one of the senior developers reviews the changes before allowing them into the final version, but I think what might be missing is a comprehensive test script for the entire application that is run through during a code freeze period before releasing a new version.  The term 'script' in this context does not refer to the files that are executed by KOLMafia to perform certain actions in the game, but refers to a checklist used by people to test all the major functional components of the program itself.[/quote]

Both holatuwol and myself look at every line of code that is submitted to the project. Each of us feels no restraint about changing or modifying any submitted code whatever.

Neither of us is immune to submitting bugs; each has corrected the code of the other. On the other hand, neither of us is perfect at detecting bugs; each of us has submitted bugs that made it into a released.

Holatuwol is the only one who packages up releases. He tends to do this after a long night's coding. Even if I weren't three timezones farther east than he, I'd probably be asleep when he makes releases. So, there's generally a group of changes which have never been tried by anybody else, much less been code inspected, which go into each release. And I think he'd be the first to admit that that is frequently where new bugs appear.

For example, the problem with the Inventory on the Item Manager in 7.3 is the result of just such a last-minute change. When I woke up, I saw that a new release was out. I updated my source, saw the problem, and fixed it.

I think a test suite would be a fine thing.
I also think that a code freeze for at least a day before making a new release would be a good idea, as well.

I'm not sure if it would work for this type of app, but there might be an open source automated test program that could be used for regression testing so that the workload is reduced significantly for each release.  Maybe its the glue I'm sniffing, but I'd be really interested in taking part in this kind of undertaking.  Let me know if this sounds interesting to you and I'll put together a post similar to this in the Mafia dev forum.

cj-"biting off more than he can chew"-swimmer

If you are volunteering to look for said "open source automated test program" and to start filling in the test cases, I bow to you in respect. That's a great idea.

By all means, write a project proposal in the dev forum. I can't imagine you'd not get a positive response.
 

cjswimmer

Member
[quote author=Veracity link=topic=141.msg679#msg679 date=1146782735]
Both holatuwol and myself look at every line of code that is submitted to the project. Each of us feels no restraint about changing or modifying any submitted code whatever.

Neither of us is immune to submitting bugs; each has corrected the code of the other. On the other hand, neither of us is perfect at detecting bugs; each of us has submitted bugs that made it into a released.[/quote]

I reread my post and it sounded a little bitchy. I have enormous respect for you, holatuwol, the rest of the dev team and everyone else involved. This program is a stunning achievement, especially since you face an ever-changing set of interfaces in the game.

I'll start searching for some testing programs. If I find anything that's worthwhile, I post about it.

Keep up the great work!
 

holatuwol

Developer
Holatuwol is the only one who packages up releases. He tends to do this after a long night's coding. Even if I weren't three timezones farther east than he, I'd probably be asleep when he makes releases. So, there's generally a group of changes which have never been tried by anybody else, much less been code inspected, which go into each release. And I think he'd be the first to admit that that is frequently where new bugs appear.
Quoted for truth.

But, if you'd like to prepare a test suite, one of the problems is reproducible output. But, KoLmafia was designed to be debug-able in a very specific manner (and some of the developers discovered this when I accidentally committed changes) that correlates directly with the debug logs it produces.

All it needs for fully automated testing, rather than the direct testing I do now is something that can parse out a debug log and spit out the files, in order of their receipt along with the CLI commands that were executed to generate them. Then, for validation, mirror stream output would be used.

This is because all you would need for verifying regression test issues is to turn on mirror streams and use any standard diff utility to make sure the outputs on the mirror stream for the test suite is identical to the output on the mirror stream during testing.

For example, if you went adventuring somewhere once using the CLI command "adventure 1 wherever", hit a fight, and then used a moxious maneuver which finished it. The debug log would show that you visited adventure.php?snarfblat=1, fight.php and fight.php?action=moxman along with a whole bunch of HTML that resulted from those form requests.

If there were a utility that could parse the debug log and produce 1.html (that contained information stating that it was a redirect), 2.html, 3.html, an index file that listed adventure.php?snarfblat=1, fight.php and fight.php?action=moxman, a script file that showed "adventure 1 wherever", then it would be very easy for me to write something internal to KoLmafia which would reproduce the events.

So that means anyone running KoLmafia with debug streams, mirror streams, and while running ASH/CLI scripts could, at their leisure, provide a test suite which could be run with KoLmafia before release that would be almost trivial to implement within the JUnit framework.
 
Top