Feature - Implemented build.xml: have "daily" call "jar", and add a note about new JARs

F

fredg1

Member
Context:
XML: 
    <target name="jar" depends="compile">

        <jar
            destfile="${jarfile}"
            basedir="${build}"
            update="true"
            index="true"
            compress="true">

            <manifest>
                <attribute name="Main-Class" value="${main}" />
            </manifest>
            <zipgroupfileset dir="${lib}/jar">
                <include name="*.jar"/>
            </zipgroupfileset>
            <zipgroupfileset dir="${src}/jar">
                <include name="*.jar"/>
            </zipgroupfileset>
        </jar>

    </target>
XML: 
    <target name="daily">

        <antcall target="set.released.false"/>

        <antcall target="compile" />

        <jar
            destfile="${dist}/${ant.project.name}-${revision}.jar"
            basedir="${build}"
            update="true"
            index="true"
            compress="true">

            <manifest>
                <attribute name="Main-Class" value="${main}" />
            </manifest>
            <zipgroupfileset dir="${lib}/jar">
                <include name="*.jar"/>
            </zipgroupfileset>
            <zipgroupfileset dir="${src}/jar">
                <include name="*.jar"/>
            </zipgroupfileset>
        </jar>

    </target>
Since the <jar> call of these two is identical, minus the destfile, I replaced <daily> with
XML: 
    <target name="daily">

        <antcall target="set.released.false"/>

        <antcall target="jar">
            <param name="jarfile" value="${dist}/${ant.project.name}-${revision}.jar"/>
        </antcall>

    </target>
, which simply calls our <"jar"> with the parameter jarfile (which we use as the value of destfile) temporarily altered, for the length of the call.

This now causes the target "jar" to be our main "archive builder" (so that any change we would want to apply to it doesn't have to be mirrored somewhere else).


Additionally, this patch adds a warning/comment in <"jar"> about the addition of new JARs to this project (I got burnt by this myself, which is why I thought that it would be a good idea).
Diff: 
             <manifest>
                 <attribute name="Main-Class" value="${main}" />
             </manifest>
+            <!--
+            Note: when adding new jars, make sure to manually
+            remove any "META-INF/*.RSA" and "META-INF/*.DSA"
+            in them. We could take care of that here,
+            but since <restrict> doesn't support <zipgroupfileset>,
+            we would need to use <archives> instead,
+            which is WAAAAAYYY too slow...
+            (build time goes from 30 seconds to 2m 30s...)
+            -->
             <zipgroupfileset dir="${lib}/jar">
                 <include name="*.jar"/>
             </zipgroupfileset>
 

Attachments

  • build.xml.patch
    3.4 KB · Views: 0
H

heeheehee

Developer
Staff member
fredg1 said:
Additionally, this patch adds a warning/comment in <"jar"> about the addition of new JARs to this project (I got burnt by this myself, which is why I thought that it would be a good idea).
Click to expand...
Why do we need to do this? I would prefer not to change these jars so others can validate signatures / checksums. What does it break?
 
F

fredg1

Member
It breaks *our* jar. If you try to build a jar using other jars as dependency/ies, and they have their own signatures, the signatures are just added, as is, to the resulting jar.
Those signatures no longer being valid, the jar is unusable.
 
H

heeheehee

Developer
Staff member
Okay. I've implemented a workaround in our build.xml (since it seems zipgroupfileset doesn't support includes): instead, we bundle all our dependencies into one fat jar, then use zipfileset on that resulting output to exclude these signature files as part of building the executable jar.

In practice this increased build times by maybe 10-20%, which seems reasonable in exchange for not accidentally breaking everything in the future.

The Gradle analogue was much simpler.

r20911.
 
fronobulax

fronobulax

Developer
Staff member
fredg1 said:
Additionally, this patch adds a warning/comment in <"jar"> about the addition of new JARs to this project (I got burnt by this myself, which is why I thought that it would be a good idea).
Click to expand...

Moot point because it is already done but how did you get burned? I build with ant. I have added new jars and I have changed the version of jar files that we are using. I run the result from the OS. I never noticed anything broken. What are you doing differently?
 
F

fredg1

Member
fronobulax said:
Moot point because it is already done but how did you get burned? I build with ant. I have added new jars and I have changed the version of jar files that we are using. I run the result from the OS. I never noticed anything broken. What are you doing differently?
Click to expand...
Looking at the jars in lib/jar and src/jar, none of them seem to have an RSA, so they were either already removed, or weren't included in the source to begin with (or I'm dumb).
 
fronobulax

fronobulax

Developer
Staff member
Thanks.

Looking down from a high perch and remembering Software Engineering and Configuration Management practices that worked and solved more problems than they created I would like to avoid adding things to the Repository that appear to be needed by one developer to support one project in one development environment.

When I was using Maven and IntelliJ professionally we managed to set things up so that we could specify which jars were needed to develop, compile, run and test as separate "lists". Configuring IntelliJ was nuanced but possible. Same for the other IDEs used on the team, of which there were several. Only the jars required for runtime were actually packaged and available to the end user. Everything else just lived on the developer's computer and was pulled from a local or remote maven repository as needed. Third party jars were either used blindly (because something depended upon them) or not allowed by the project unless useful JavaDoc was provided by the jar maintainer. In the corporate environment people could be compelled to write useful JavaDoc doc for "company" code so JavaDoc and test coverage were things that were checked during reviews and sometimes before a commit was actually allowed.

I personally don't like clutter and what I perceive as inefficiency so I will be sad when I do a "svn update" and find files that *I* don't need in order to develop or build.

With the expectation that we will eventually be comfortable enough with Gradle to stop using Ant and start using Gradle we can defer these considerations until then, if indeed they even matter to anyone else.

Since the "no third party source" policy was mentioned (off site) as an impediment, it might be worth reviewing why. It is not an absolute policy as the source files in /lib attest.

It was adopted because
  • There was concern about inadvertently claiming someone else's work as KoLmafia's.
  • There was no good way to obtain a jar file from a maintainer (but there is now).
  • It required work to monitor the maintainer's code and update the copied source when there were relevant changes.
  • It was rare that KoLmafia actually needed to modify someone else's source to make it work in a mafia environment.
The success of the Maven approach suggests to me that "no third party source - Use a jar" is still good policy. If source is needed to support a developer's style or IDE then perhaps the effort should be made to assist the developer in tweaking their environment so third party source in the repository is not needed?

The same spirit that animates folks to use lint on source code makes me dream of a day when there are no java files in /lib and what is there now has been deprecated, replaced with someone else's jar or moved to /src since KoLmafia was willing and able to maintain it.

It is a maxim for folks who optimize code (usually for speed) that "it is easier to make a working program fast than get a fast program working" and in that spirit I do not intend my philosophical comments to be an impediment to progress.
 
gausie

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
I would love to move away from third-party source and manage our dependencies with Gradle when we're moved over.
 
fronobulax

fronobulax

Developer
Staff member
gausie said:
I would love to move away from third-party source and manage our dependencies with Gradle when we're moved over.
Click to expand...

I looked for low hanging fruit once dropping a jar was trivial - but most of what is there now is no longer supported by the originators or wasn't in a jar I could easily find. When I get tired of writing tests for coverage and tests I would want before I deprecated some code that is really OBE, this is likely to be my next shiny object.
 
H

heeheehee

Developer
Staff member
fronobulax said:
The same spirit that animates folks to use lint on source code makes me dream of a day when there are no java files in /lib and what is there now has been deprecated, replaced with someone else's jar or moved to /src since KoLmafia was willing and able to maintain it.
Click to expand...
My aspirational vision is that lib/ ceases to exist entirely.

As I mentioned elsewhere: org/jdesktop/{beans,swingx} is the low-hanging fruit as far as a jar we should be able to drop in.

Beyond that, I wonder how many of these dependencies are obsoleted by newer Java standard library features. If we move to Java 9, we get libraries for interfacing with the System Tray / Dock, potentially obviating apple.dts.samplecode.osxadapter and lib.com.jeans.trayicon.

JEP 272: Platform-Specific Desktop Features

openjdk.java.net openjdk.java.net

And lib.com.centerkey.BareBonesBrowserLaunch is mostly a lot of extra work around Java 5 back-compatibility for clients who can't use java.awt.Desktop.getDesktop().browse(java.net.URI.create(url));
 
H

heeheehee

Developer
Staff member
Yes -- the gradle build already does this.

edit: As it stands today, lib/ stores external sources (and their dependencies) that aren't bundled into jars. If/when we move away from Ant entirely, we can already discard src/jar (instead relying on the build system to bring those in for us). lib/testjar also can go away in that scenario, although maybe that should already live in src/testjar, or test/jar.
 
Last edited:
You must log in or register to reply here.
Top