Feature Build Process Changes

MCroft

Developer
Staff member
looking further at the changes to build.xml and build.properties, it looks like you're changing the ant build process to add the lib/jar directory to the classpath (via adding the classpath to the compile task).

The classpath file has a long history of not being required except for eclipse, and the project has generally had a model where compile-time dependencies on external jars were managed by importing the source and and runtime depenedencies were managed by adding a jar to lib/jar .

I can't tell you why, but I'm cautious about changing it. Does it need to be changed to get this patch to work? What are the implications of it for other jars? There are users (recently) who've had issues in Eclipse when they had two conflicting versions of some jar, so what complexity are we buying to get this?

tl;dr - I'd be more comfortable if we didn't change the build.xml, but I don't have a concrete failure to report.
 

ikzann

Member
My understanding, and please correct me if i'm wrong, is that the .classpath file played and plays no role in the build process. It is only there to assist editors' language support.

I am a little confused about your question about build.properties, as there aren't any changes to that file in the patch.

On substance, I'm not sure why we want to import the source rather than a compiled jar. But if that's longtime practice I can certainly revise the patch to include all the rhino source.
 

fronobulax

Developer
Staff member
FWIW, I thought at some point that the build process was set up in such a way that any jar file included in lib/jar was automatically available on the class path and bundled to be available at runtime. So if just putting the jar in that directory is not sufficient then we need to understand why the "old" build process doesn't work.

Also, FWIW, at some point if KoLmafia needed third party code the preferred solution was to grab and embed a jar file. The cases where we have source code are either because we had to change source to make things work or a jar file was not available at the time the decision was made.

For me the Gold Standard in building is "ant <target>" on a command line. If that works and an IDE generated process doesn't then the IDE is misconfigured. There are a couple of things, that are part of the managed assets, that exist to support a specific development environment but I think of them as exceptions to be avoided :)
 

ikzann

Member
For me the Gold Standard in building is "ant <target>" on a command line. If that works and an IDE generated process doesn't then the IDE is misconfigured. There are a couple of things, that are part of the managed assets, that exist to support a specific development environment but I think of them as exceptions to be avoided :)
That goal makes sense to me, but the .classpath file was already included in the repo so I updated it. The action consistent with that goal would be to remove the .classpath file entirely.
 

ikzann

Member
FWIW, I thought at some point that the build process was set up in such a way that any jar file included in lib/jar was automatically available on the class path and bundled to be available at runtime. So if just putting the jar in that directory is not sufficient then we need to understand why the "old" build process doesn't work.
The old build process doesn't work because it just doesn't put anything in lib/jar into the classpath. That's what my build.xml changes accomplish. Happy to revert that and include source or make whatever changes you all want; not something I really have strong views on.
 

fronobulax

Developer
Staff member
Something changed/broke in 2018 and .classpath and .project were added to the repository to fix it. I did it but don't recall why. I think someone inadvertently checked them in because their IDE changed them and then the build process broke for a lot of other people. I either did the revert or had pristine copies available locally. I think.
 

ikzann

Member
Sure. Those tests:
On a clean checkout with my patch + rhino jar, the build succeeds.
On a clean checkout with my patch + rhino jar, and build.xml changes reverted, the build fails because it can't find any of the rhino classes.
On a clean checkout with my patch + the rhino source, the build succeeds whether or not the build.xml changes are reverted.

Which is to say that, in my understanding, MCroft's assessment is exactly correct. I'm not an expert on this so I'll say again that I'm happy to take guidance from you all.

Oh, and the .classpath file doesn't appear to have any effect on the build. Removing it does not appear to change anything.
 

fronobulax

Developer
Staff member
I just deleted all the files in lib/jar, using a svn delete so the update in the build would not just fetch them again and "ant daily" fails with many package not found errors, from which I conclude that something is looking for files there.

I selectively unreverted some and the errors went away. I also note that I think some of the files are no longer necessary since their absence did not cause an error, but weeding that out is another project.
 

fronobulax

Developer
Staff member
Sure. Those tests:
On a clean checkout with my patch + rhino jar, the build succeeds.
On a clean checkout with my patch + rhino jar, and build.xml changes reverted, the build fails because it can't find any of the rhino classes.
On a clean checkout with my patch + the rhino source, the build succeeds whether or not the build.xml changes are reverted.

Which is to say that, in my understanding, MCroft's assessment is exactly correct. I'm not an expert on this so I'll say again that I'm happy to take guidance from you all.

Oh, and the .classpath file doesn't appear to have any effect on the build. Removing it does not appear to change anything.

We're talking at the same time but I suspect your build process is not the same as my ant process.
 

fronobulax

Developer
Staff member
My build process is running ant clean && ant. Is there something else I should be doing?
Hmmm. I do "ant daily" and sometimes "ant test".

Now to look and see if there is something in the daily target that should have been added to other targets as well.
 

MCroft

Developer
Staff member
I applied your patch and it told me that build.properties had changed. I didn't look into details. We did several iterations, so it might not have in -v5.

ant clean && ant runs the default target, which is debug

I tend to run from ant daily, which cleans and updates. I don't know what other users do, but @gausie might use debian, @Veracity has a process for releases and a different one for debugging, and some people may use the jrebel task, and I have no idea if the javadoc task is even valid.

I'm not sure I have an opinion yet, but I will eventually. I do want this surfaced and discussed, because it will be hard to extract once we do it, and I don't want to break something that someone has used for a while without understanding it.

I did run into "jars in lib/jar are not available at compile time" on my Darkmode patch. It was easy enough for me to work around, but I think this code may need deeper integration with the jars than I did.

I don't want to add all of the jars to the classpath when we need one, though.

Code:
Michaels-MBP:kolmafia mcroft$ ant -projecthelp
Buildfile: /Users/mcroft/projects/kolmafia/build.xml

Main targets:

Other targets:

 clean
 compile
 daily
 debian
 debug
 desktop
 dist
 dmg
 exec
 exec-jrebel
 gitsvn-version
 init
 jar
 jarbundler
 javadoc
 jsmooth
 notes
 set.dist.properties
 set.released.false
 set.version
 start
 svn-version
 tar
 test
 test-compile
 unset.properties
 update
 version
Default target: debug[code]
 

fronobulax

Developer
Staff member
I think the daily target sets a classpath that includes lib/jar and the debug target does not.

When we migrate this discussion elsewhere I think I'd suggest that daily be the default, not debug, but I think we know where the different results are coming from. Need see whether the targets were ever documented...
 

gausie

D̰͕̝͚̤̥̙̐̇̑͗̒e͍͔͎͈͔ͥ̉̔̅́̈l̠̪̜͓̲ͧ̍̈́͛v̻̾ͤe͗̃ͥ̐̊ͬp̔͒ͪ
Staff member
I just deleted all the files in lib/jar, using a svn delete so the update in the build would not just fetch them again and "ant daily" fails with many package not found errors, from which I conclude that something is looking for files there.

I selectively unreverted some and the errors went away. I also note that I think some of the files are no longer necessary since their absence did not cause an error, but weeding that out is another project.
This is because prior to ikzann's changes those jars are present at compile time but not shipped with the final jar. @ikzann do your changes bring all the library jars into the final jar or just the rhino one?
 

MCroft

Developer
Staff member
To Frono's point, I would love this to be a new discussion of build.xml, but I don't know how to fork a comment into a new post in XenForo.
This is because prior to ikzann's changes those jars are present at compile time but not shipped with the final jar. @ikzann do your changes bring all the library jars into the final jar or just the rhino one?
Ant daily packages the lib/jar files in the jar task.
All the jars are in the zipfilegroup.

Also, build documentation is (and already was when I started messing with it 10 years ago) done by XML comment in the build file above the targets.
 

fronobulax

Developer
Staff member
This is because prior to ikzann's changes those jars are present at compile time but not shipped with the final jar. @ikzann do your changes bring all the library jars into the final jar or just the rhino one?

You are right. But something in the build process is extracting class files from jars in lib\jar and bundling them in KoLmafia.jar right? Some of the classes end up under org in the jar file.
 

fronobulax

Developer
Staff member
To Frono's point, I would love this to be a new discussion of build.xml, but I don't know how to fork a comment into a new post in XenForo.

Ant daily packages the lib/jar files in the jar task.
All the jars are in the zipfilegroup.

Also, build documentation is (and already was when I started messing with it 10 years ago) done by XML comment in the build file above the targets.

I put on my Moderator hat if it is not there, start the new thread and move posts to the new thread.

Sometimes I will copy posts or copy an entire thread and prune it but I haven't discovered a fork type command when Moderating elsewhere on XenForo.
 

MCroft

Developer
Staff member
In evaluating the Scripting Support Feature Proposal, it has a change to build.xml that puts the library jars in the classpath of the compile target, which changes the longstanding but perhaps unnecessary division between strongly coupled library jars (i.e. those that must be available to the compiler to compile) and weakly coupled library jars (i.e. those that are packaged with the jar and are available at runtime).

the Scripting support currently needs this, but could be implemented differently at build-time. We wanted to create a new thread for this discussion because it's important to, but off topic for discussing the scripting piece.
 

fronobulax

Developer
Staff member
I see class files that are in jar files in lib/jar that somehow end up in KoLmafia.jar. I would like to understand how they get there and why the same technique cannot be done for the rhino jar.
 
Top