Linux/GitHub/Test Help

fronobulax

fronobulax

Developer
Staff member
See PR at https://github.com/kolmafia/kolmafia/pull/191

and test results at https://github.com/kolmafia/kolmafia/runs/4156747723?check_suite_focus=true

A portion of the test failure message is

Code: 
org.opentest4j.AssertionFailedError: show_skills.txt output does not match:  ==> expected: <
Torso Awareness (0 mp)
Summon Holiday Fun! (11 mp)
Communism! (0 mp)

Torso Awareness (0 mp)

> but was: <Torso Awareness (0 mp)
Summon Holiday Fun! (11 mp)
Communism! (0 mp)

Torso Awareness (0 mp)

>
    at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
    at app//org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
    at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
    at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1152)
    at app//net.sourceforge.kolmafia.CustomScriptTest.testScript(CustomScriptTest.java:59)

What I think is going on is that a test is failing on Ubuntu on GitHub but not locally on Windows 10. My hypothesis is the failure is due to a whitespace type of issue between Ubuntu and Windows. EOL comes to mind but I can't really confirm what the file GitHub/Ubuntu really is using for EOL.

I am running out of ideas.

Is it possible for me to run any of the OS checks separately or have the other ones run anyway if one fails?

Are there any ideas besides trying to make the comparison in CustomScriptTest ignore whitespace?

Anyone on a Linux box willing to grab the PR and help debug the test or point a finger at the GitHub environment?
 
MCroft

MCroft

Developer
Staff member
fronobulax said:
See PR at https://github.com/kolmafia/kolmafia/pull/191

and test results at https://github.com/kolmafia/kolmafia/runs/4156747723?check_suite_focus=true

A portion of the test failure message is

Code: 
org.opentest4j.AssertionFailedError: show_skills.txt output does not match:  ==> expected: <
Torso Awareness (0 mp)
Summon Holiday Fun! (11 mp)
Communism! (0 mp)

Torso Awareness (0 mp)

> but was: <Torso Awareness (0 mp)
Summon Holiday Fun! (11 mp)
Communism! (0 mp)

Torso Awareness (0 mp)

>
    at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
    at app//org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
    at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
    at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1152)
    at app//net.sourceforge.kolmafia.CustomScriptTest.testScript(CustomScriptTest.java:59)

What I think is going on is that a test is failing on Ubuntu on GitHub but not locally on Windows 10. My hypothesis is the failure is due to a whitespace type of issue between Ubuntu and Windows. EOL comes to mind but I can't really confirm what the file GitHub/Ubuntu really is using for EOL.
Click to expand...
Well, MacOS shares line endings with Linux, so that's not the issue.

As an experiment (which can be discarded), I added trim() to the strings.

Why? Looking at the first line of the success and failure...

Code: 
expected: <
Torso Awareness (0 mp)
...
Code: 
but was: <Torso Awareness (0 mp)

which looks like one case captured a leading EoL character and the other didn't.
 
MCroft

MCroft

Developer
Staff member
Aside: I want to improve the all-or-nothing model of matching exact file contents vs. script output.

Not sure exactly how. Maybe we could have a data object (possibly in a script.tests folder or somewhere) that had something like an array of values to check.
Something like [{assertContains, scriptOutput, TorsoAwareness}{assertThat, scriptOutput.contains(AnotherShow)}].

I know why we have what we have, I just would like the option to ignore the name of the logged in user (for instance) and only deal with the part I want to test.
 
H

heeheehee

Developer
Staff member
MCroft said:
Aside: I want to improve the all-or-nothing model of matching exact file contents vs. script output.
Click to expand...
That doesn't address the issue that golden tests should be hermetic and deterministic (and these scripts should satisfy these criteria). Nor does trimming the output contents figure out where the leading newline is coming from, and why it's not present in that one test failure. (Do we have to worry about a leading newline being dropped the second time Torso Awareness is printed? trim() won't fix that.)
 
fronobulax

fronobulax

Developer
Staff member
I remain unenlightened about the possibility of manually running tests for a specific OS, on demand.

I am mystified why a supposedly independent test can fail for one OS on GitHub but nowhere else and there is no obvious relationship between the changed code and the failing test.

The independence is certainly suspect as might be the GitHub provided OS environment or the testing scaffolding.

We'll figure things out
 
You must log in or register to reply here.
Top