put something like
ulimit -v 200000 -m 200000
on a line between "cd /kolmafia" and "ant daily"
the numbers are max virtual memory and max memory (do ulimit -a for all settable limits) in kilobytes.
I am not sure how much it will help if your ant spawns many processes in parallel (the limits are per process, I am not seeing anything for "limit total memory for all subprocesses spawned from here" ... well, there are process groups in recent-ish linux kernells ...).
The limit should be high enough to permit it to finish under normal conditions - I don't know how much that is.
ETA: control groups
mount none /cgroup -t cgroup -o memory
cd /cgroup
mkdir restricted_ant
chown user_who_runs_the_build_job restricted_ant/*
then before running ant daily (in the bash script):
echo $$ > /cgroup/restricted_ant/tasks
echo ??? > /cgroup/restricted_ant/memory.max_usage_in_bytes
echo ??? > /cgroup/restricted_ant/memory.limit_in_bytes
cgroup does include all spawned subprocesses, but on the other hand, I think going over limit will just start swapping (and not killing), controlling swap needs another cgroup subsystem ... (enabled on kernel build)
then