Recent Downtime

fewyn

Administrator
Staff member
I'm going to chock up the recent downtime's culprit again to the auto build process I have setup on the server. Basically for some reason it goes crazy during the build process eats all my ram then the os starts killing things to free up memory and the server locks up :(

Not really sure what to do about it, any suggestions would be appreciated.
 

xKiv

Active member
Sic ulimit on the autobuild process to prevent it from stealing more than X amount of RAM?
Assuming that it's only one process.

If it w*rks, it will just kill the build process (when it requests too much memory) instead of something random.
 

fewyn

Administrator
Staff member
Not sure how I would do that tbh.

Code:
#!/bin/sh 

cd /kolmafia
ant daily
cd dist
mv -f * /var/www/builds.kolmafia.us/
cd /kolmafia
svn log -l 30 > /var/www/builds.kolmafia.us/svnlog.txt
 

xKiv

Active member
put something like
ulimit -v 200000 -m 200000
on a line between "cd /kolmafia" and "ant daily"
the numbers are max virtual memory and max memory (do ulimit -a for all settable limits) in kilobytes.

I am not sure how much it will help if your ant spawns many processes in parallel (the limits are per process, I am not seeing anything for "limit total memory for all subprocesses spawned from here" ... well, there are process groups in recent-ish linux kernells ...).

The limit should be high enough to permit it to finish under normal conditions - I don't know how much that is.

ETA: control groups

mount none /cgroup -t cgroup -o memory
cd /cgroup
mkdir restricted_ant
chown user_who_runs_the_build_job restricted_ant/*

then before running ant daily (in the bash script):
echo $$ > /cgroup/restricted_ant/tasks
echo ??? > /cgroup/restricted_ant/memory.max_usage_in_bytes
echo ??? > /cgroup/restricted_ant/memory.limit_in_bytes


cgroup does include all spawned subprocesses, but on the other hand, I think going over limit will just start swapping (and not killing), controlling swap needs another cgroup subsystem ... (enabled on kernel build)

then
 
Last edited:
Top