Bug - Cannot Reproduce java.lang. OutOfMemoryError

bumcheekcity

bumcheekcity

Active member
I'm trying to build a FaxBotSpadeArmy (log in all 100+ faxbot clan leaders, run some spading scripts, one at a time) and keep getting the out of memory error.

Is there anything that can be done for what is admittedly a fairly horrible method of using KoLMafia? Is it just not designed for this? I've thrown some cli_execute("gc") around the place, but it just can't handle it.

I'm willing to be told that this is a horrifying overuse of mafia and have to find a workaround, but if there's something that can be done or that I can do that'd be great.

I'm posting this under "Bug" not because I think it is, but because I genuinely don't know what to tag it as otherwise. Could be a feature request I suppose. But feel free to change it if you think appropriate.
 
fronobulax

fronobulax

Developer
Staff member
The knee jerk reaction to that error is to start tweaking the memory given to the Java VM from the operating system. There is a command line switch that controls how much memory is allocated and raising it is usually the first way to go. Since KoLmafia will let you trigger garbage collection, pressing that button once or twice will probably help.

I'm short of time and have not found the command line syntax on my fist search but I will track it down as soon as I can if no one beats me to it.
 
roippi

roippi

Developer
I don't run kolmafia from the command line, but I run some other java programs with the -Xmx%%%%m flag, where % is the number of megabytes of memory allocated to the program. Might try that.
 
R

Raijinili

Member
Are you running them all at the same time? Java's taking over 100MB on my machine right now (Mafia's memory sayer looks like 40MB, at rest). A hundred of them at the same time might be over 10GB. If you're doing it at the same time, can I say: Run ten at a time?
 
bumcheekcity

bumcheekcity

Active member
One single java window is open. It's a breakfast script that throws a cli_execute("login "+nextFaxbotName()) at the end of what it's doing, which is basically just an eatdrink() command and then cli_execute("adv * treasury")
 
jasonharper

jasonharper

Developer
Look for the instructions I've previously posted for using the jps and jstack tools to do a heap dump. Do so when you're in an out-of-memory state, and post the dump - I can analyze it to figure out exactly what's taking all the memory.
 
jasonharper

jasonharper

Developer
Preliminary results: hola did it...

Your heap dump included 217 MB of data from 12199 cached files with names like "aa/aa-19418.txt". As of r8962, unbounded use of file_to_map() / map_to_file() is basically illegal, since the data file cache that was added back then has absolutely no provision for ever removing anything from memory.
 
holatuwol

holatuwol

Developer
12,199 cached files? Whoa...

Update: Try r9148. I've limited the number of cached files to 500 (or at least, the code should theoretically do that), which should still save some disk access. Thinking about implementing a less naive recently used checker that actually tracks how many times a file's been accessed, but I'll do that once I know that 500 is a good enough limit.
 
Last edited:
Veracity

Veracity

Developer
Staff member
Perhaps some options to control caching manually might be in order, too. If BCC will explain how he is using these files, it might give help illuminate that. Here are some possible usage patterns:

- Perhaps some files are read once into a map, updated with new data, and written out again, not to be read again in this session. Those files would be good candidates nor not caching at all.
- Perhaps some files are are read and re-read repeatedly, as data to the same script, but are per-character data. These files would be usefully cached, but should be flushed on logout.
- Perhaps an analysis script will read and process the data from thousands of other files - once. Again, nothing is gained by caching them.

Perhaps file_to_map could take a "cache control" argument with values of, say, "don't cache", "cache until logout", "always cache". If the default is "always cache", your cache limit and "recently used" idea would manage that, but scripted applications (what a concept - but that seems to be what BCC has done) could have finer control and perform better.
 
bumcheekcity

bumcheekcity

Active member
Basically, the bots are dumping a massive amount of raw data (small, usually 15-40K) into .txt files which I am then parsing. In this particular case, as it's a data dump, I gain nothing from caching them.

An extra argument (perhaps a simple int flag 0/1/2) would be a fine solution. I can't think that many other people would ever need this, so always caching would be sensible, or always caching until logout possibly.
 
holatuwol

holatuwol

Developer
Well, if I flush the cache on logout, there's almost zero impact for regular users (since most users exist KoLmafia anyway once they logout of one character). So r9152 flushes the data file on cache on logout which, when combined with the maximum file limit, should handle most cases.
 
bumcheekcity

bumcheekcity

Active member
I'm getting the following error with the bots when I'm adventuring the file. The bots are no longer dumping the files that they were before, that was some spading that is long done.

They're just logging in, eating/drinking with eatdrink() and "adv * treasury"ing. I've got the heap.bin and am compressing it now.

http://hotfile.com/dl/111440209/69a0ff1/Desktop.7z.html
 
Last edited:
bumcheekcity

bumcheekcity

Active member
Yes. Though it wouldn't be for dumping loads of data into text files, because that was only for a fortnight's worth of spading.
 
You must log in or register to reply here.
Top