Bug - Not A Bug KoLmafia using up lots of CPU

fronobulax

Developer
Staff member
Still learning my tools. A snapshot that didn't burn many adventures seemed to have Equipment Manager and some Java/Swing as hotspots. The script meat farming run definitely showed a slight increase in memory usage over time. It wasn't huge but it definitely looked like Ye Olden Times when memory was allocated and deallocated manually and there was no garbage collection and there was a "leak" in the code.

P.S. I am running 64 bit Java and the OP's taskManager showed 32 bit so that might be a contributor.
 
Last edited:

Zyro

Member
P.S. I am running 64 bit Java and the OP's taskManager showed 32 bit so that might be a contributor.

Bingo! I installed the 64 bit Java and now everything appears to be good and stable. It appears that I have a lot more memory now: https://i.imgur.com/CnZWuVg.png

I will post here if the problem rears its head again. Otherwise, consider it solved :)

Thanks to everyone for looking at my issue and taking the time to help me with it.
 

fronobulax

Developer
Staff member
So, is this not a bug, or do we need to say we only support Java 64 bit ?

I'm saying it is not a bug. I'm also thinking that we support Java and it is not our problem if someone's hardware and OS do much better with x64 than with x32.
 

Saklad5

Member
32-bit software can only use numbers as large as 2^32. That includes memory addressing, so 32-bit programs can only directly access up to 4GiB of RAM.

64-bit software, on the other hand, can work with numbers up to 2^64. That means the theoretical maximum amount of RAM it can use is 17180000000GiB. This is generally considered Enough.

Moral of the story: stop using 32-bit software. Blame Microsoft for the continued prevalence of it in the Windows ecosystem.
 

Saklad5

Member
KoLmafia definitely uses more CPU than I would expect it to, but that is just a result of using Java. Java is inefficient, so that’s par for the course unfortunately. I love programming in it, but I hate running it.

I’m sure there are a bunch of idling optimizations that could be made, but I’m certainly not familiar enough with KoLmafia’s code to contribute much there.
 

finaltidus

New member
So I noticed this happening to me as well and I tried to find stuff out. When running a script with mafia normally I use at most ~15% CPU, but when I have the gCLI open I always use 25%, it doesn't even need to be a script that's doing anything. Here's a wait command: https://i.imgur.com/dnV0ndj.png

Also here is the same thing except I clicked off the gCLI to the adventure pane: https://i.imgur.com/Yefu0fq.png
All CPU usage dropped off even though the "script" is still running and you can also see by the pics the memory is fine. Something funky is going on with the gCLI and I don't think it is "just java things".
 

finaltidus

New member
Then what do you think it is and, more importantly, what do you expect someone to do about it?

I didn't write the code, why would I know what's wrong with it? I just would expect it to be a bigger issue than "who knows whats wrong, it's not a bug just deal with it" when something is obviously wrong with it.
 

lostcalpolydude

Developer
Staff member
"Knowing" that "something is definitely wrong" doesn't make it worth tens of hours to maybe (but probably not) find a solution. Unless it's worth spending that time for you, learning the code...
 

fronobulax

Developer
Staff member
I didn't write the code, why would I know what's wrong with it? I just would expect it to be a bigger issue than "who knows whats wrong, it's not a bug just deal with it" when something is obviously wrong with it.

I want to know what you think is wrong with it. For example Galactic Civilizations III routinely uses 100% of all cores available. Is that wrong? Should I be complaining to Stardock? They are explicit in saying that if allowed, the game's AI will use all the CPU resources available to it. Do we update the docs?

You may be smarter than I am which is why I willing to ask questions. But having run a profiler on KoLmafia, I am unable to identify a hot spot that could be cooled by a change in programming. As a professional Java programmer, just because a customer says "it uses too much CPU" does not mean there is something that I should have my team fix. So far the evidence that this is something a programmer could address by changing code is lacking. Hence the questions.
 

finaltidus

New member
I want to know what you think is wrong with it. For example Galactic Civilizations III routinely uses 100% of all cores available. Is that wrong? Should I be complaining to Stardock? They are explicit in saying that if allowed, the game's AI will use all the CPU resources available to it. Do we update the docs?

You may be smarter than I am which is why I willing to ask questions. But having run a profiler on KoLmafia, I am unable to identify a hot spot that could be cooled by a change in programming. As a professional Java programmer, just because a customer says "it uses too much CPU" does not mean there is something that I should have my team fix. So far the evidence that this is something a programmer could address by changing code is lacking. Hence the questions.

You are right, wasting CPU time doing next to nothing is fine. I only have a cs bachelors so with your java focus you know more than me on this particular topic so be sure to talk slow with me so I can try and keep up. Sorry that I brought up an OBVIOUS issue that mafia hogs cpu when it shouldn't. The next time I run into an issue I will think about if I should post about it. I'll think "Is this a complex problem that would require more work than changing 1 or 2 lines of code?" and if it is I wont bother posting about it. I know you don't want to waste time working on a problem that could take a long time or not be solved at all, but trying to be so disingenuous to say that this isn't a bug/issue is just wrong. Just say, "Yes it's a problem but I don't care enough and have the time to try and fix it, it's really not that big of an issue in my opinion". Sorry for bringing up the issue, I'll stop.
 

fronobulax

Developer
Staff member
mafia hogs cpu when it shouldn't.

If you have evidence to support that statement, then I would like to hear it, specifically "when it shouldn't". My efforts to gather such evidence failed. But if you have no evidence - and I don't consider your screen shots evidence, at least on their own - the go off in a snit. I won't mind.
 

Veracity

Developer
Staff member
finaltidus said:
When running a script with mafia normally I use at most ~15% CPU, but when I have the gCLI open I always use 25%
Sorry that I brought up an OBVIOUS issue that mafia hogs cpu when it shouldn't
When the gCLI is not open, KoLmafia is making requests of KoL - and doing whatever Java does to wait until the response come in.
When the gCLI is open, KoLmafia is also running a thread waiting for input from the terminal. Using Java calls. Doing what? Polling? Who knows? Not I.

I am completely unsurprised that KoLmafia uses more CPU when the gCLI is open than when it is not.
That it goes from 15% of your CPU to 25% of your CPU is higher than I would have guessed.

But, when you characterize "using 25% of your CPU" as "hogging" the CPU, I raise my eyebrows.

Really?

A program running a GUI (which has event polling) and doing net access (and polling the network and occasionally processing responses) and also with an open command line (polling the keyboard) and it's using 25% of the CPU?

I would not be surprised if we could work to cut back on CPU usage for all those threads, but is 25% really a problem to you? Enough to characterize the program as "hogging" the CPU?

You have a CS degree? Congratulations! I will welcome your code analysis/profiling/whatever to point out exactly where we are "wasting" CPU. I suggest you focus on that "Chat Buffer" code which manages the gCLI window.

However, I believe the other devs have politely informed you that we don't think it's worth spending OUR (unpaid volunteer) time to work on that issue.

If you feel it IS worth YOUR (unpaid volunteer) time to reduce that 25% CPU footprint to something you would not characterize as "hogging", by all means - spend that time, submit a patch, and I, personally, will look at at.

Thank you!
 

Veracity

Developer
Staff member
By the way - I DON'T have a CS degree. Classics/History (undergraduate) and Medieval Studies (Graduate).

:p
 

heeheehee

Developer
Staff member
I didn't write the code, why would I know what's wrong with it? I just would expect it to be a bigger issue than "who knows whats wrong, it's not a bug just deal with it" when something is obviously wrong with it.

Understanding the context of this thread, and why it was marked as "Not a Bug" may be useful.

I'm saying it is not a bug. I'm also thinking that we support Java and it is not our problem if someone's hardware and OS do much better with x64 than with x32.

In particular, this thread seemed to be related to CPU usage related to garbage collection, and the OP commented that upgrading to 64-bit Java resolved said issue. I agree with frono's decision to close the thread as he did.

If you have a clearly documented performance issue you want us to look at (e.g. "Graphical CLI has high CPU usage when idle"), then please file a clearly specified bug rather than appending to some arbitrary (previously-closed, no less!) thread you found via search.

However, please keep in mind (as several others have pointed out) that this is a volunteer-driven community project, so please try to exercise a bit more tact. We're more than happy to look over a patch if you do end up tracking down this particular performance issue -- Veracity provided what I think should be a useful starting point.

(Also, flaunting credentials isn't going to get you far. I'm fairly confident stating that a number of us devs do have degrees in CS / related fields as well as years of experience writing production-grade code in industry.)
 

Darzil

Developer
I don't have a degree in CS, though I did end up doing a computer based project in my Engineering degree. Which in hindsight was a terrible mistake, as the level of knowledge of the people marking my work was terrible. However, this coding is something I do in my free time, I haven't been a full time programmer since 1989. My day job is Cloud Networking.

There are almost certainly ways we can improve performance. I discovered the other day we were recalculating adventure ranges stupidly often (every api status hit among other things), and reduced it by adding a couple of checks first. That made a significant difference to speed for me. I know one day I'll find a few days to add a bunch of pre-checks for equipment that improves with the number of parts of a set (Brimstone, Smithsness, etc), which will make a huge difference in aftercore maximizer speed.

The trick is finding them.

I can definitely see SOMETHING weird with Java (1.8_171) and/or Mafia. Not running anything (other than Relay Browser), I'm seeing CPU generally <1%. However, that sometimes climbs to around 25%. When I bring Mafia window to the front, it seems to happen, especially (but not exclusively) with GCLI tab in focus, but it isn't 100% repeatable (leaving Mafia out of focus for a while seems to help cause it). When it does happen, clicking on another tab, or another Mafia window taking focus, does reduce CPU to normal levels.

Edit - Am tempted to find a way to video this now. It's pretty repeatable. Happens when I haven't had it in focus for a while, then switch with Mafia with GCLI selected, goes away when I change focus to something else in mafia, then stays away when I load up Mafia with GCLI.
 
Last edited:

antipasta

Member
I thought I would bump this thread with a recent observation: after a while I routinely see Mafia using ~25% of the CPU, which probably means 100% of a core in my quad-core laptop. But if I hit the "Refresh Status" button on the lower left of the gCLI, usage immediately drops to something like 1%.

Then it climbs back up eventually. What could mafia be doing that "Refresh Status" fixes?
 
Top