Bug - Fixed r20911 causes jar size regression

[gausie]

This change seems to have brought the bundle size from 17MB to 46MB

See: https://ci.kolmafia.us/job/Kolmafia/410/

 

[ckb]

Did this deduplication actually duplicate? Because the jar doubled in size.

 

[fewyn]

jar:
      [jar] Building jar: /var/lib/jenkins/workspace/Kolmafia Pipeline/build/bundled-libs.jar
      [jar] Building jar: /var/lib/jenkins/workspace/Kolmafia Pipeline/dist/KoLmafia-20914.jar
      [jar] Warning: selected jar files include a META-INF/INDEX.LIST which will be replaced by a newly generated one.
   [delete] Deleting: /var/lib/jenkins/workspace/Kolmafia Pipeline/build/bundled-libs.jar

Maybe?

 

[PeKaJe]

This change seems to have brought the bundle size from 17MB to 46MB

Also, when you look inside the .jar, every file is there twice ... somehow. Confuses the heck out of some unzipping applications.

 

[fronobulax]

Also some discussion towards the end of https://kolmafia.us/threads/build-x...-add-a-note-about-new-jars.26273/#post-163814

 

[fronobulax]

I poked around.

The KoLmafia jar file contains bundled-libs.jar. There is a lot of duplication between bundled-libs and KoLmafia. I didn't check to see how close they were to identical.

Not that it matters but there is an empty clover subdirectory but several Clover generated html files are bundled in the jar. I'm not sure that is necessary.

 

[fredg1]

source:

    <target name="jar" depends="compile">
      <jar destfile="${build}/bundled-libs.jar"
           basedir="${build}"
           update="true"
           index="true"
           compress="true">
        <zipgroupfileset dir="${lib}/jar">
          <include name="*.jar"/>
        </zipgroupfileset>
        <zipgroupfileset dir="${src}/jar">
          <include name="*.jar"/>
        </zipgroupfileset>
      </jar>

      <jar
          destfile="${jarfile}"
          basedir="${build}"
          update="true"
          index="true"
          compress="true">

        <manifest>
          <attribute name="Main-Class" value="${main}" />
        </manifest>
        <zipfileset src="${build}/bundled-libs.jar">
          <exclude name="META-INF/*.RSA"/>
          <exclude name="META-INF/*.DSA"/>
          <exclude name="META-INF/*.SF"/>
        </zipfileset>
      </jar>

      <delete file="${build}/bundled-libs.jar" />
    </target>

---

Separation:
1-

      <jar destfile="${build}/bundled-libs.jar"
           basedir="${build}"
           update="true"
           index="true"
           compress="true">
        <zipgroupfileset dir="${lib}/jar">
          <include name="*.jar"/>
        </zipgroupfileset>
        <zipgroupfileset dir="${src}/jar">
          <include name="*.jar"/>
        </zipgroupfileset>
      </jar>

2-

      <jar
          destfile="${jarfile}"
          basedir="${build}"
          update="true"
          index="true"
          compress="true">

        <manifest>
          <attribute name="Main-Class" value="${main}" />
        </manifest>
        <zipfileset src="${build}/bundled-libs.jar">
          <exclude name="META-INF/*.RSA"/>
          <exclude name="META-INF/*.DSA"/>
          <exclude name="META-INF/*.SF"/>
        </zipfileset>
      </jar>

3-

      <delete file="${build}/bundled-libs.jar" />

---

"1" creates "our jar". It has everything, including unwanted .RSA/.DSA/.SF.
"2" creates another jar, using "bundled-libs.jar" AND /build as source.
In short, "2" should only be transforming "bundled-libs.jar", rather than re-fetching the source. (at least as far as I know)

 

[fredg1]

that, or "1" shouldn't be using basedir="${build}"

 

[heeheehee]

So, the intent of bundled-libs was to just bundle all the other jars.

When I try removing basedir, that does nothing. On the other hand, I have had success adding excludes to the two jar commands. I'll commit that, since it brings the jar size back down to 20MB.

 

[heeheehee]

r20915 should fix this regression, as far as I can tell

 

[heeheehee]

https://ci.kolmafia.us/job/Kolmafia/416/ indicates that the jar size is back down to 19.90MB, compared to 19.93MB in https://ci.kolmafia.us/job/Kolmafia/409/

 

[heeheehee]

(I've split off these discussions into a separate thread so they're all in one place.)

 

[heeheehee]

Related: how much do we care about jar sizes?

Apparently the gradle-built jars are 30MB, although by using the shadow plugin with its minimize feature, that went down to 17MB.

 

[fronobulax]

Related: how much do we care about jar sizes?

Apparently the gradle-built jars are 30MB, although by using the shadow plugin with its minimize feature, that went down to 17MB.

Ancient History tells of a time when a large portion of the user base was believed to have dial up connections and so the size of the jar file was a consideration. In 2021 I suspect it matters only if there are users with slow or unreliable connections. Like the refusal to update Java version I think the size is also a user community issue. How many people does it matter to and how many people do we alienate?

 

[ckb]

Related: how much do we care about jar sizes?

What would be the reason for not making it as small as possible? more efficient = more better, no?

 

[gausie]

What would be the reason for not making it as small as possible? more efficient = more better, no?

Efficiency has nothing to do with bundle size necessarily. A bigger bundle might be because there's more efficient code and lots of it!

Related: how much do we care about jar sizes?

My personal take is that we care enough to check in on it but not so much that we'll let it stand in the way of a well considered 3rd party library (for example, though I can't think what else would contribute that significantly)

 

[fronobulax]

Efficiency has nothing to do with bundle size necessarily.

Perhaps a different meaning of efficiency was being used? If you are looking at the efficiency of transmissions, smaller is better and more efficient. If you have ever had to live with dial up or support receivers with extremely low bandwidth then bundle size matters. When it takes 3 hours to download something, whether the result runs in 3 seconds or 30 seconds is much less a concern.

As alluded to above, if there are now KoL players and KoLmafia users that are constrained by bandwidth or storage then we don't seem to know about them. Until we find out otherwise, keeping bundle size small is more of a developer desire than anything an end user would know or care about.

 

[heeheehee]

What would be the reason for not making it as small as possible? more efficient = more better, no?

Well, one argument (per a commit I added earlier today that added about 150KiB) is maintainability.

One way to minimize jars is to pull in the dependency sources (e.g. svnkit, or swingx), then prune them to the specific files we need. This takes a lot of effort, and if we patch these dependencies to work around issues, we should track those fixes (or risk losing them if we need to upgrade in the future). Usually it means we just don't upgrade our dependencies unless we need some critical feature we aren't willing to backport.

The simpler approach is to bundle the entire jar. Upgrading is trivial (download a new jar, delete the old one).