Feature - Implemented Adding Clover Coverage to Jenkins

fewyn

fewyn

Administrator
Staff member
I decided to add Clover coverage to Jenkins so that it would run tests automatically on every build but I've run into a failure where UseItemRequest.java is exceeding the 64kb limit of byte code.

Code: 
set.version:
    [javac] Compiling 984 source files to /var/lib/jenkins/workspace/Kolmafia/build
   [clover] OpenClover Version 4.4.1, built on October 11 2019 (build-1020)
   [clover] Updating existing database at '/var/lib/jenkins/workspace/Kolmafia/.clover/clover4_4_1.db'.
   [clover] Processing files at 1.8 source level.
   [clover] Clover all over. Instrumented 984 files (27 packages).
    [javac] Ignoring source, target and bootclasspath as release has been set
    [javac] /tmp/clover7988318540920929393.tmp/net/sourceforge/kolmafia/request/UseItemRequest.java:1823: error: code too large
    [javac]     public static final void parseConsumption( final String responseText, final boolean showHTML )
    [javac]                              ^
    [javac] Note: /tmp/clover7988318540920929393.tmp/net/sourceforge/kolmafia/textui/RuntimeLibrary.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error
    [javac] Compile failed; see the compiler error output for details.
 
H

heeheehee

Developer
Staff member
Yeah. The current clover setup in build.xml currently excludes UseItemRequest and ChoiceManager.

XML: 
<target name="with.clover">
      <clover-setup source="1.8">
        <fileset dir="${src}" includes="**/*.java">
          <!-- Add a couple of exclusions for files with
               methods that are too large for instrumentation. -->
          <!-- parseConsumption is ~5000 lines long. -->
          <exclude name="**/UseItemRequest.java" />
          <!-- postChoice1 is ~5000 lines long. -->
          <exclude name="**/ChoiceManager.java" />
        </fileset>
        <testSources dir="${test}" includes="**/*.java" />
      </clover-setup>
    </target>

Does your Jenkins setup not just use the existing build.xml rules?
 
fewyn

fewyn

Administrator
Staff member
Not by default it seems, I'm working on a new version that uses pipelines which seems to be functioning so far. At least it ran all the tests, looking for a way to clean it up and get reports showing somewhere.
 
fewyn

fewyn

Administrator
Staff member
heeheehee said:
Yeah. The current clover setup in build.xml currently excludes UseItemRequest and ChoiceManager.

XML: 
<target name="with.clover">
      <clover-setup source="1.8">
        <fileset dir="${src}" includes="**/*.java">
          <!-- Add a couple of exclusions for files with
               methods that are too large for instrumentation. -->
          <!-- parseConsumption is ~5000 lines long. -->
          <exclude name="**/UseItemRequest.java" />
          <!-- postChoice1 is ~5000 lines long. -->
          <exclude name="**/ChoiceManager.java" />
        </fileset>
        <testSources dir="${test}" includes="**/*.java" />
      </clover-setup>
    </target>

Does your Jenkins setup not just use the existing build.xml rules?
Click to expand...
@heeheehee Could we get the XML report also added to the build? Part of the way Jenkins can display it is with the XML, otherwise I'll have to artifact the clover html directory and upload/copy it somewhere (which I prefer not to do).
 
H

heeheehee

Developer
Staff member
r20881 adds a `clover.xml` target. As noted in the commit description, typical CLI usage is probably something like `ant -lib lib/testjar clean coverage clover.xml`.

(Note that `ant coverage` bundles a bunch of dependent tasks so you don't have to manually `ant with.clover compile test clover.report` or whatever.)
 
fewyn

fewyn

Administrator
Staff member
I've got the pipeline up and running now and it tests fine and shows reports which you can now see at https://ci.kolmafia.us/job/Kolmafia Pipeline/

Below are the current status targets if you think they should be changed let me know.

Code: 
                    healthyTarget: [methodCoverage: 70, conditionalCoverage: 70, statementCoverage: 70],
                    unhealthyTarget: [methodCoverage: 50, conditionalCoverage: 50, statementCoverage: 50],
                    failingTarget: [methodCoverage: 0, conditionalCoverage: 0, statementCoverage: 0]


I'm going to let this run for a few days and if all goes well I will get the Jenkinsfile commited to the repo and we should be all good when I replace the current main builds with this, also shouldn't require any changes to any existing updaters/downloaders this way.
 
H

heeheehee

Developer
Staff member
Hm. It looks like the only file with coverage on the Jenkins build currently is KoLConstants.

Our current overall coverage numbers look like around 13%, with the additional test cases that I'm running locally for the language server thread (which so far have brought Parser.java up to 77%). I don't suppose those targets are incremental coverage? If they're absolute numbers, then our repo will be failing for a very long time...

From the test log:
Code: 
   [clover] Updating existing database at '/var/lib/jenkins/workspace/Kolmafia Pipeline/.clover/clover4_4_1.db'.
   [clover] Processing files at 1.8 source level.
   [clover] Clover all over. Instrumented 22 files (8 packages).

You may want to add a `clean` at the start of your `ant` command -- I've run into problems with coverage build artifacts clashing with non-coverage build artifacts.

(What we probably should do is shunt them to different directories, and then clean them both up when we run `ant clean`.)
 
fewyn

fewyn

Administrator
Staff member
If I run the clean command with the tests it empties the dist folder where the build is located so I can't store it once the tests are completed.
 
H

heeheehee

Developer
Staff member
Yeah, I've tweaked the build targets to put instrumented classes in build/clover. Hopefully r20882 does the trick. Seems to work locally, anyhow.

(And now I don't need to ant clean for every commit.)
 
H

heeheehee

Developer
Staff member
Code: 
coverage-compile:
    [javac] Compiling 984 source files to /var/lib/jenkins/workspace/Kolmafia Pipeline/build/clover
   [clover] OpenClover Version 4.4.1, built on October 11 2019 (build-1020)
   [clover] Creating new database at '/var/lib/jenkins/workspace/Kolmafia Pipeline/.clover/clover4_4_1.db'.

Oh yeah, that looks way better.
 
H

heeheehee

Developer
Staff member
But, it looks like we cached the previous bad execution. :/

Can you manually go in and nuke .clover/coverage.db.snapshot? (if not, it'll clear itself up after 10 or so executions in a row.)

Also note that you don't need to run `clover.report` if you don't plan to show the HTML report.
 
fewyn

fewyn

Administrator
Staff member
I nuked the entire workspace and it's still clearing out the build in the dist folder. Hrm....
 
H

heeheehee

Developer
Staff member
The inline CSS that Clover generates isn't loading because of the CORS headers your server's sending:

dashboard.html:2251 Refused to apply inline style because it violates the following Content Security Policy directive: "style-src 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-x...'), or a nonce ('nonce-...') is required to enable inline execution. Note that hashes do not apply to event handlers, style attributes and javascript: navigations unless the 'unsafe-hashes' keyword is present.

As a result, the numbers look funky:

https://ci.kolmafia.us/job/Kolmafia Pipeline/clover/dashboard.html for instance has 1.0 (100%) coverage but it's run together such that it looks like "1.0100%".

There's also some blocked JS:

Blocked script execution in 'https://ci.kolmafia.us/job/Kolmafia Pipeline/clover/dashboard.html' because the document's frame is sandboxed and the 'allow-scripts' permission is not set.
 
fewyn

fewyn

Administrator
Staff member
Probably has to do something with the reverse proxy probably. It seems to be working now with the latest build I manually ran. I'll look into the blocked script stuff tomorrow.
 
H

heeheehee

Developer
Staff member
Hrm. I pushed a new change, and test coverage went "down" as a result of test optimization. In particular, note that coverage of test code outside CustomScriptTest / ParserTest is at 0%.

Should we remove the snapshot db as part of `ant clean`? An easy enough change to make on my end, although that means that Jenkins will never get the benefit of optimized builds / caching test results.
 
You must log in or register to reply here.
Top