Memory leaks?

[xephyne]

Catch-22 : Try manually specifying a heapsize at runtime using the -Xmx switch.

It still leaks, and eventually fills 128MB as well (in fact, it doesn't seem to take that much longer - Like whatever causes it gets more and more aggressive once the leaking starts).

Incidentally, for me, it seems to happen mostly when (or shortly after) arranging things in my display case.

 

[greenrider]

I've just been ascending normally the past few days and have seen some pretty brutal slowdowns using the current build. I'm not running any ASH scripts and not really using any of mafia's features beyond adventuring in the relay browser and having a few chat windows opened.

When it gets bad, the only remedy appears to be to shut down mafia completely and restart, but that only helps for about 30 minutes before the slowdown returns.

 

[Catch-22]

I believe there's a memory leak involved with the GUI elements of KoLmafia. It's getting pretty tricky to pin it down and I'm not very familiar with how the KoLmafia GUI works. If any of you guys that are experiencing issues are confident enough to run KoLmafia in headless/CLI mode, please feel free and share your results. I've managed to get my Mafia up to 62mb using the GUI but I'm not experiencing slow-downs.

Okay, pushed it a bit further by opening up some chat panels (I don't normally chat). Managed to fill up 64mb, I still think the issue is related to the GUI elements not unloading correctly.

Steps to reproduce: Open as many GUI elements as you can, close them, open them, etc. Eventually the memory will fill up and will stay relatively full even after you have closed all additional GUI elements. I must admit that, for me, KoLmafia wasn't much less responsive than usual though.

 

[Heffed]

Steps to reproduce: Open as many GUI elements as you can, close them, open them, etc. Eventually the memory will fill up and will stay relatively full even after you have closed all additional GUI elements. I must admit that, for me, KoLmafia wasn't much less responsive than usual though.

Yes, the store manager, ice penguin express, and the purchase buffs functions tend to suck up the allocated memory quite quickly.

It's actually quite easy to max it using no scripts. Just GUI manipulation. I do however see slowdowns when the memory gets maxed. It tends to peg one of my CPU cores.

 

[codster62]

Well, I am using mafia on ubuntu 9.04, and it runs much faster then on my windows computer, but it does indeed still fill up the memory and slow down a bit. On windows XP, it seems to run about 25% slower then this, and when it fills up the memory, there is a huge drop in speed. Of course, I have not updated Java on it, so that may be a problem as well. Hmmmm. Does everybody experiencing this problem have the latest Java application available?

 

[greenrider]

I wiped Java and reinstalled fresh from the latest version. I still get really bad slowdown in mafia - after about 20 minutes of use, even something as simple as "sell facsim dict" takes about 6-7 seconds to process. When I close the program and restart, everything works fine again for another 20 minutes until it grinds to a slog.

 

[Catch-22]

I'm surprised that no devs have commented yet.

I haven't looked at how KoLmafia handles GUI elements yet, but honestly there's not a whole lot more I can do. A dev will need to commit the changes anyway.

My best guest at what is probably happening is that the GUI elements are not marked for garbage collection once they are no longer in use.

 

[Bale]

There kind've was a response, just not in this thread.

Rev 7636 - jasonharper (1 file(s) modified)
Revert to pre-r7621 behavior of clearing the "multiByte" property on HTML
documents after every append; that's actually necessary to avoid slow-downs
after displaying any non-ASCII text.

This thread got active again just after r7621. Are you using r7636 or later?

 

[jasonharper]

r7636 has nothing to do with memory leaks, though. I'm not experiencing this problem at all, which makes it rather difficult to debug. There is a possibility of doing it remotely, though. I need someone who:
* can reproduce the "memory fills up - garbage collection button doesn't help" problem at will;
* has a full JDK installation (I'm not sure that a Java runtime has the tools you'll need - specifically jps and jmap); and
* is comfortable working at your OS's command line.

What I'd need you to do:
1. Run KoLmafia and get it into the problem state.
2. Run jps to get a list of Java process IDs; make a note of KoLmafia's ID.
3. Run jmap, telling it to produce a heap dump in hprof binary format. On my system, that would be:
jmap -heap:format=b ID
- but I'm running Java 1.5, I'm not sure if the format will be the same on all versions. In any case, running jmap with no parameters should list the available options.
4. This should produce an enormous (tens of megabytes) file, called something like heap.bin. Zip it and attach it here.

 

[Veracity]

I also have no commented because it doesn't happen to me, either. I'll defer to Jason, who seems to be a Man with a Plan.

 

[Heffed]

Well, I hope this is the issue, but I can't say for sure...

I know that I was pretty close on memory usage, but I was reading a book so I didn't notice if the slowdown was happening. However, I did attempt to open the gCLI when I had four turns left and KoLmafia hard-locked. So I'm assuming it's the issue, but like I said, I'm not 100% certain.

I couldn't upload the .zip directly to the forum because each attempt failed because I got an error saying the file lacked a security token. So hopefully mediafire works. (never used it before)

http://www.mediafire.com/file/kinccmtyzym/heap.zip

 

[jasonharper]

Well, your heap dump allowed me to find a problem, don't know if it's the only one...

When substring() is used in Java, it doesn't actually copy anything - the new string refers to the same character data as the original, just with different starting/ending offsets. That's fast, and generally an efficient use of memory, but it creates problems in situations where only a tiny substring of some huge string is still in use - the entire original string is ineligible for garbage collection.

Specifically, it appears that you encountered about 300 distinctly-named monsters in that session, presumably most (if not all) of them being hobos. All of those names end up in an internal cache, which is technically a memory leak (since there's no upper bound to the cache size), but would be livable if just the text of the names was being cached - that would be only a few kilobytes wasted. What actually happened, however, is that each cached name continues to refer to the entire page text (at over 40 kilobytes each) where the name was originally seen - that's over 12 megabytes wasted, right there.

Give r7643 a try, it explicitly copies strings that are going to be in the cache, so that they don't carry any excess baggage around. If you still have problems, do another heap dump - with this specific problem out of the way, others will be easier to find.

 

[Heffed]

Yes, those were all hobos.

OK, I'll give r7643 a go and upload another heap dump if I get in a slowdown.

Thanks Jason!

 

[Catch-22]

If you still have problems, do another heap dump - with this specific problem out of the way, others will be easier to find.

Nice work. IMO there's also memory problems with the way KoLmafia currently handles buffbot XML files.

Steps to reproduce:
1. Open KoLmafia.
2. Open purchase buffs menu.
3. Wave goodbye to about 12mb until you restart KoLmafia.

I'm sure if you also added a custom buffbot to the buffbots list that the memory usage would increase further (presumably with each XML file).

In my experience, KoLmafia will always use at minimum roughly 30mb of RAM. If the first thing somebody does for the day is purchase a few buffs, that will put their RAM usage at 40mb out of 64mb straight away and it's not likely to ever go lower than that.

Add to it a fairly lengthy chat session and a few more memory intensive operations here and there, pretty soon you'll be pushing the 64mb upper limit.

 

[Alhifar]

I was able to reproduce this and get a heap dump of it.
You can find it at http://www.insidethepale.net/mafia/heap.bin.zip

 

[Heffed]

Definitely a step in the right direction! Ran through my characters without needing to completely exit Java at any point.

The garbage collection was able to function the entire session. RAM tended to hover within 20MB of max.

 

[greenrider]

I was also able to replicate the buffbot window causing what appears to be a memory leak.

 

[jasonharper]

As far as I can tell, the buff purchase window isn't actually a memory leak - it's a one-time increase in used memory, which legitimately must continue to be used (rather than hit all the buffbots' servers again if you purchase more buffs later). I've tweaked what I could in r7644, along with some other unnecessarily large retained objects that showed up in Alhifar's dump, but I don't think I saved more than about 4 MB in this scenario.

 

[Alhifar]

I honestly think the only reason I was able to get it to freeze up was by outrunning the garbage collector, I had to open and close the windows as fast as I could to even get it to freeze up. Maybe I'm not experiencing the same problem as previously reported?

If anyone has specific steps to reproduce a problem, but doesn't know how to create the heap dump, I'm more than willing to do so.

 

[Catch-22]

As far as I can tell, the buff purchase window isn't actually a memory leak - it's a one-time increase in used memory, which legitimately must continue to be used (rather than hit all the buffbots' servers again if you purchase more buffs later). I've tweaked what I could in r7644, along with some other unnecessarily large retained objects that showed up in Alhifar's dump, but I don't think I saved more than about 4 MB in this scenario.

jason, what about a local cache of the XML files stored in the data folder? For now it would suffice just to cache them on each session (so you're using a hard drive storage instead of RAM). Once you've closed the buff requests window, the memory used should be marked for garbage collection. If you open the buff request window again that session, use the locally stored file in data instead of downloading it again from the server. This brings down RAM usage when not buying buffs and also eliminates server hits.

Ultimately, it would be nice if KoLmafia could determine the modification date of the remote file from a response header. If it matches the one in the cache then it shouldn't update the file. This should result in faster load times of the buff purchasing window as well as less bandwidth usage for the servers hosting the XML files. Most buffbots don't update their XML files very often.