Bug JUnit tests are not tested in a random order

[Irrat]

I'm not sure if this is a bug or not, but JUnit does not (At least on my local copy) run the tests in a random manner.

I'm not really the best to talk about it, but while trying to track down an issue with a memory leak in LockableListModel (More visible elements than there are elements) I noticed that the test order was always the same. The issue itself is probably related to rollover; Seeing as I can't seem to reproduce it after rollover.

I'm not sure why the test order is like this, given that default behavior is random and I can't seem to find something in the project specifying a specific test order.
But when running the tests in a random order, I'm seeing multiple tests fail.

Some of them, it's probably simple as data wasn't loaded? Others, I believe they're depending on other tests putting them in a good state to run.

Regardless, I don't think it's good practice. If mafia's tests rely on classes being called in a specific order that's not obvious, and not annotated to run before other classes; That's probably not good code practice.

This also exposes test leakage where stuff wasn't cleaned up but isn't obvious due to test order.

 

[gausie]

Agreed. I found a test recently that passed despite defective code due to ordering.

 

[gausie]

I don't think this is a design choice of course, we just have a configuration issue

 

[fronobulax]

The number of forks (threads) allowed when testing is also a factor. I found a number of leaks when running with only one fork. The cost was 15 minutes of wall clock time per test execution.

I will also note that vague memories plus Google searches suggest that tests should be run in a deterministic order. How do you know you have fixed something if the order is different next time? The preferred solution to find leakage is to run tests in small groups and manually adjust the order. I'm lazy and what I do is just monitor /test/root. Any file there that is not under git control was created by running a test and is this a candidate for leaks. There are several things there that should be cleaned up but doing so isn't my highest priority.

But it is probably worth deciding that random test order is A Good Thing and the only solution to a possibly minor problem before we change anything.

IMO. YMMV.

 

[gausie]

We could run them in order and random order, concurrently

 

[fronobulax]

We could run them in order and random order, concurrently

If someone who is looking for test leaks thinks random order would help them then I have no issues supporting that preference. But since there are other ways to find leaks I am unconvinced that random ordering is going to make the default GitHub processes any more reliable.

Tangentially, given the speed at which testing practices and support have evolved for KoLmafia, I might wonder if a better use of time might be to document "how to write a good KoLmafia test".