I'm not sure if this is the right place to start this idea, but I've been having some concerns with bugs that have appeared in the latest releases of KOLMafia. There are multiple developers that can check in changes to the program, and I know that one of the senior developers reviews the changes before allowing them into the final version, but I think what might be missing is a comprehensive test script for the entire application that is run through during a code freeze period before releasing a new version. The term 'script' in this context does not refer to the files that are executed by KOLMafia to perform certain actions in the game, but refers to a checklist used by people to test all the major functional components of the program itself.
I manage several large web application development projects at my job, and we rely heavily on test scripts for component and regression testing when we're releasing new versions of our apps. The test scripts consist of Excel spreadsheets that are sent out to customer focus groups for them to run through and record their findings. We are in the process now of setting up QuickTest Pro to automate our regression testing procedures, but for now its a lot of manual work. FYI, regression testing refers to the testing of older components of an application to make sure they still work alongside new components or enhancements. Now, creating test scripts is the least favorite part of my job. They are incredibly time-consuming, and often logistical nightmares to organize depending on the size of the application. they are however, crucial to the success of our releases.
I am willing to offer my support in the creation of such a test script for KOLMafia.
The first step is the identification of all the major interfaces of the application (i.e. Adventure tab, Graphical Cli, Scripting System etc.)
We then identify groups of the functional components in the app(i.e. Inventory Management, Adventuring, Spellcasting, etc.).
Those components can be broken down into groups of Actions (Using Items, Moving items between storage locations, casting buffs, ...).
After that you can define logical test cases which have a list of expected results and actual results based on conditions
For buying a item:
Conditions: enough meat on hand, purchase location is accessible
Expected result: meat on hand reduced by item cost
Expected result: item added to inventory
You further define your logical test cases to include sample data and the interfaces defined in the first step:
buy a cool whip using the mall tab interface
buy a cool whip using the Graphical Cli
buy a cool whip using the web browser
buy a cool whip using ash script
...
and you show your results in each interface as well
character pane shows meat reduced by 30
ash script my_meat() shows meat reduced by 30
...
We'll want to make sure we include as many expected failure conditions in the testing as well (i.e. buying from an NPC store that is not accessible, equipping items you don't have etc..).
As you can see, this is one huge pain in the ass. The nice thing about it is, you only have to do the bulk of it once. After you make changes to the program you just have to make new test cases for the new functionality, but all your regression test plans are done already. I'm not sure if it would work for this type of app, but there might be an open source automated test program that could be used for regression testing so that the workload is reduced significantly for each release. Maybe its the glue I'm sniffing, but I'd be really interested in taking part in this kind of undertaking. Let me know if this sounds interesting to you and I'll put together a post similar to this in the Mafia dev forum.
cj-"biting off more than he can chew"-swimmer
I manage several large web application development projects at my job, and we rely heavily on test scripts for component and regression testing when we're releasing new versions of our apps. The test scripts consist of Excel spreadsheets that are sent out to customer focus groups for them to run through and record their findings. We are in the process now of setting up QuickTest Pro to automate our regression testing procedures, but for now its a lot of manual work. FYI, regression testing refers to the testing of older components of an application to make sure they still work alongside new components or enhancements. Now, creating test scripts is the least favorite part of my job. They are incredibly time-consuming, and often logistical nightmares to organize depending on the size of the application. they are however, crucial to the success of our releases.
I am willing to offer my support in the creation of such a test script for KOLMafia.
The first step is the identification of all the major interfaces of the application (i.e. Adventure tab, Graphical Cli, Scripting System etc.)
We then identify groups of the functional components in the app(i.e. Inventory Management, Adventuring, Spellcasting, etc.).
Those components can be broken down into groups of Actions (Using Items, Moving items between storage locations, casting buffs, ...).
After that you can define logical test cases which have a list of expected results and actual results based on conditions
For buying a item:
Conditions: enough meat on hand, purchase location is accessible
Expected result: meat on hand reduced by item cost
Expected result: item added to inventory
You further define your logical test cases to include sample data and the interfaces defined in the first step:
buy a cool whip using the mall tab interface
buy a cool whip using the Graphical Cli
buy a cool whip using the web browser
buy a cool whip using ash script
...
and you show your results in each interface as well
character pane shows meat reduced by 30
ash script my_meat() shows meat reduced by 30
...
We'll want to make sure we include as many expected failure conditions in the testing as well (i.e. buying from an NPC store that is not accessible, equipping items you don't have etc..).
As you can see, this is one huge pain in the ass. The nice thing about it is, you only have to do the bulk of it once. After you make changes to the program you just have to make new test cases for the new functionality, but all your regression test plans are done already. I'm not sure if it would work for this type of app, but there might be an open source automated test program that could be used for regression testing so that the workload is reduced significantly for each release. Maybe its the glue I'm sniffing, but I'd be really interested in taking part in this kind of undertaking. Let me know if this sounds interesting to you and I'll put together a post similar to this in the Mafia dev forum.
cj-"biting off more than he can chew"-swimmer