Bug Mafia Slowness

Just to quickly list optimizations that I am playing with in my script that I have been finding help A LOT:

1) CPU optimization - Out of the box most CPU's are set in a powersave mode on some or all cores. You can use utilities to set this to "performance" mode. Additionally some BIOS settings allow you to force your CPU's to be in this mode constantly.

2) localhost settings - Wherever your hosts are setup (linux is /etc/hosts) make sure you have an entry that ties 127.0.0.1 to localhost to your HOSTNAME.

3) Setup BBR - instead of using Reno or Cubic congestion Control algorithms switch to BBR, this algorithm was developed by google to maximize throughput in network congestion scenarios (helps with all the calls to KoL's servers)

4) Verify DNS settings - make sure you are using a good DNS. Generally when on a LAN or even with your ISP you will be redirected to their DNS, then they use google's DNS. If you directly change your DNS servers to use google's (8.8.8.8 and 8.8.4.4) your name resolution for KoL hits will be faster.

5) Turn off IPv6 support - (WARNING: if this is a company computer, or used in an application that uses IPv6 this may cause problems) Turing off IPv6 support in both your browser and the system itself can improve the relay aspect of KoLMafia (notably in Firefox, possibly Chrome)

6) Java optimization - You can change the parameters that JVM launches with. Notably:

-Xmx This is the maximum amount memory java can use and shouldn't generally be more than 1/2 your free memory. 512M is a decent setting, I've seen mine hit just under 2G with my settings. This must be an integer and reflect proper memory increments. Example: -Xmx2G or -Xmx2048M

-Xms This is the initial amount of memory java uses, it will then incrementally scale up to the -Xmx setting. Unfortunately there is a small amount of performance loss during scaling, for speed it is best to set -Xms the same as -Xmx. Example: -Xms2G or -Xms2048M

-Xmn This is where new objects in Java start, live, and maybe die. This IS usually a calculated number, but you can set it yourself. Usually 1/4 to 1/3 of -Xmx is an accepted value and must also reflect proper memory increments. Example: -Xmn512M

-XX:+/-DisableExplicitGC This will disable/enable KoLMafia from deliberately calling for Garbage Collection (clean up of unused objects in memory). by default this is off (-XX:-DisableExplicitGC) if you are allocating 1G of memory or more in -Xmx you should be able to safely disable this (-XX:+DisableExplicitGC) which can eliminate otherwise unnecessary pauses, and allow the system to clean up when it feels necessary.

Java 7 or higher:
You can use the more optimized G1 Garbage Collector with the following settings:
Code: 
-XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled
Java 6 or lower:
You can use CMS Garbage collector with the following settings:
Code: 
-XX:+UseConcMarkSweepGC -XX:+UseParallelGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=###
(### = 40 (Default) if you are running -Xmx of 512M, or 68 if you are above 512M)​


Anyways. This is what I have been plying with and researching over the last few days. If anyone else has any comments let me know.
 
-XX:+/-DisableExplicitGC is probably not going to do anything. KoLmafia only explicitly does garbage collection in two places, the Collect Garbage button and the cli command "gc". Both of those are only triggered by the user. It could be argued that explicit GC triggered by the user is no longer necessary but some people like watching bar graphs change and mashing buttons.

In a perfect world you would be able to take some kind of measurement before and after and have an inkling which of those things actually effect mafia's performance and which are educated guesses that really have no impact.

My focus was on Java because mafia has some control over that. I found the reminders at https://betsol.com/2017/06/java-memory-management-for-java-virtual-machine-jvm/ to be useful. My opinion is that non-heap memory might be something to look at, especially since it is effected by coding decisions. If the issue is the Heap and Garbage Collection I would expect mafia would drop a hint by the memory bar.

You mention scripts. Which ones? That might point to something that has changed recently.
 
fronobulax said:
You mention scripts. Which ones? That might point to something that has changed recently.
Click to expand...

The script I mention is the Launcher script I made to preset and optimize then Launch KoLMafia.

The slowness is general, It doesn't matter if I am manually running turns or running the more intensive sl_ascend, I still experience slowness. When I am manually running turns the only active scripts are CHit and Ezandora's Guide.

The CPU settings and Java memory settings helped with the local running of Mafia. When running scripts or intensive things like using maximizer I saw an increase in speed. However the next bottleneck seemed to be the Relay and the individual calls to the site from scripts. For this issue BBR, DNS, and disabling IPv6 helped in small amounts. (Since you can't really fix network slowness).

Good to know about the Explicit Garbage collection. I can remove it then.
 
I was asking about ash scripts. Two reasons. While I do remember things that didn't happen, I remember that there were tweaks to how ash handles object creation because some script "idioms" caused heap overflows. It has also been the case scripts that did certain things or interacted with new content/features exposed opportunities for optimization. If you say that manually running turns in the relay browser is sluggish then I doubt ash is a factor.

I would say about 90% of the time I think mafia is "sluggish" on my (Windows) system it correlates directly with a network hiccup. That's my excuse for living with it rather than trying to diagnose and fix :-)
 
Here is what I am playing with so far. This isn't perfect, this isn't clean, just what I am working with. Cleaning comes after.

Code: 
#!/bin/bash
## This script optimizes the CPU and sets java's memory usage for Linux
## This script MUST be located in the same directory as the KoLmafia .jar file

## Set our variables
MyKoLM=`ls -t KoLmafia*.jar | head -n 1`
MyCPU=`which cpufreq-set`

## Fail if no KoLmafia found
if [[ -z $MyKoLM ]]; then
        echo "Could not find a valid KoLmafia .jar file in `pwd`"
        echo "This script MUST be located in the same file where the .jar is located"
        exit 1
fi

## Make sure we can run KoLmafia in the event of a fresh download
echo "Ensuring KoLmafia is executable"
chmod 777 "$MyKoLM"

## Get Free Memory Data
echo "Getting Memory Stats"
Mem=`free -mh | grep Mem | awk '{print $4}'`
Round=`echo $Mem | sed -e 's/[a-zA-Z]//g' | awk -F "." '{print $1}'`
HRA=`echo $Mem | sed -e 's/[0-9\.]//g'`

## Fail if Free memory is too small, otherwise optimize
##      XMN should be 1/4 Full memory
##      HeapOccupancy should be smaller with the < 1G
##      With superfluous RAM turn off Explicit Garbage collection calls and let the system decide when to pause
echo "We have $Mem free memory"
if [[ "$HRA" != "G" ]] && [[ "$HRA" != "T" ]]; then
        echo "This script only allocates Gigabytes or Terabytes."
        exit 1
elif [[ "$Round" == "1" ]] && [[ "$HRA" == "G" ]]; then
        OptMem="512M"
        HeapOcc="40"
        XMN="128M"
else
        HeapOcc="68"

        if [[ "$Round" == "2" ]] && [[ "$HRA" == "G" ]]; then
                OptMem="1024M"
                XMN="256M"
        elif [[ "$Round" -ge "3" ]] && [[ "$Round" -lt "8" ]] && [[ "$HRA" == "G" ]]; then
                OptMem="2048M"
                XMN="512M"
        elif [[ "$Round" -ge "8" ]] || [[ "$HRA" == "T" ]]; then
                OptMem="4096M"
                XMN="1024M"
        fi
fi
echo "Optimizing to use $OptMem memory"

## Find Version for Garbage Collection Optimization
VerArray=( `java -version 2>&1 | awk -F '"' '/version/ {print $2}' | sed 's/\./ /g'` )
if [[ "${VerArray[0]}" == "1" ]]; then
        Ver="${VerArray[1]}"
else
        Ver="${VerArray[0]}"
fi

## Use optimized Garbage Collector
if [[ "$Ver" -ge "7" ]]; then
        echo "Running Java $Ver: Using G1 Garbage Collector Settings"
        GCOpts="-XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled $ExplicitGC"
else
        echo "Running Java $Ver: Using CMS Garbage Collector Settings"
        GCOpts="-XX:+UseConcMarkSweepGC -XX:+UseParallelGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=$HeapOcc"
fi

## Try to optimize the CPU if we have the required utility
if [[ -z $MyCPU ]]; then
        echo "WARNING: Would prefer to maximize CPU performance consider installing optimizer."
        echo -e "\tRun: sudo apt-get install cpufrequtils"
        sleep 3
else
        ## Switch "-r" for cpufreq-set does not seem to work properly. Check and optimize each core individually.
        for P in `grep ^processor /proc/cpuinfo | awk '{print $3}'`; do
                if [[ `cpufreq-info -c $P | grep "The governor" | awk -F """ '{print $2}'` == "performance" ]]; then
                        echo "CPU $P already optimized"
                else
                        echo "Setting CPU $P to Performance mode"
                        sudo $MyCPU -g performance -c $P
                fi
        done
fi

## Optimize /etc/hosts
HostsToVerify="127.0.0.1 ::1"
echo "Checking that /etc/hosts is configured correctly for accessing localhost addresses"
for HTV in $HostsToVerify; do
        HostEntry=`grep ^$HTV /etc/hosts | grep -v "#"`
        if [[ -z $HostEntry ]] || [[ "$HostEntry" != *"localhost"* ]] || [[ "$HostEntry" != *$(hostname)* ]]; then
                echo -e "\tEntry for $HTV not correct in /etc/hosts"
                CapCurrent=`echo "$HostEntry" | awk -F "$HTV" '{print $2}' | sed -e 's/^\t//g;s/^[ ]+//g;s/ $//g;s/\n//g'`
                if [[ "$HTV" == "127.0.0.1" ]]; then
                        sudo sed -i "s/^127\.0\.0\.1.*/127.0.0.1\t$CapCurrent $(hostname)/g" /etc/hosts
                        HostsChanged="1"
                elif [[ "$HTV" == "::1" ]]; then
                        sudo sed -i "s/^::1.*/::1\t$CapCurrent $(hostname)/g" /etc/hosts
                        HostsChanged="1"
                fi
        else
                echo -e "\tCorrect entry found for $HTV"
        fi
done
if [[ "$HostsChanged" == "1" ]]; then
        echo "Refreshing Network"

        sudo ifdown -a
        sleep 3
        sudo ifup -a
fi

## Optimize TCP CCA
CCA=`sysctl net.ipv4.tcp_congestion_control`
if [[ "$CCA" == *"reno"* ]]; then
        echo "WARNING: Currently running outdated "reno" algorithm for network congestion"
        echo -e "\tconsider upgrading to BBR. Follow the directions here:"
        echo -e "\thttps://www.techrepublic.com/article/how-to-enable-tcp-bbr-to-improve-network-speed-on-linux/"
elif [[ "$CCA" == *"bbr"* ]]; then
        echo "CCA already upgraded to BBR."
else
        echo "WARNING: I do not recognize the CCA: `echo $CCA | awk -F ' = ' '{print $2}'`"
        echo -e "\tconsider upgrading to BBR. Follow the directions here:"
        echo -e "\thttps://www.techrepublic.com/article/how-to-enable-tcp-bbr-to-improve-network-speed-on-linux/"
fi

## Run java with our optimized memory and GC settings (Xmx and Xms should be equal to prevent mid process resizing)
##      Start as server. Takes longer to boot but offers better performance.
echo "Starting KoLMafia JVM in Server Mode (Longer boot time, better performance)"
java -server -Xmx$OptMem -Xms$OptMem -Xmn$XMN $GCOpts -jar ./$MyKoLM
 
You must log in or register to reply here.
Back
Top