Feature - Implemented Relogging to improve ping

[Irrat]

Edit: I am not advocating the use of the antilag user script, work has been done in mafia after this thread was created that makes this script redundant.
There is a new tab "Connection" in the login menu, and in the preferences you can see options in "General > Connection Options".
New commands has also been added to mafia, such as "relog" and "ping". I myself still use the script, but it's a personal choice.


Recently @Veracity has made improvements to Mafia in the way of checking what your ping to the servers is.

GitHub - libraryaddict/KolAntiLag: A script to refresh kol to find an non-laggy server

A script to refresh kol to find an non-laggy server - libraryaddict/KolAntiLag

github.com

This script is one that was shared in the ASS discord, and what it does is it stores a cache of 5 (or more) pings across different login sessions, and then relogs you until your current session ping is within 1.1x of the lowest ping. So if the lowest was 100, it will aim to get you a ping of 110 or less.
It is a rolling cache, so it deletes older entries as it inserts newer entries.

What it would be nice to see, is if Mafia could
1. Detect what your ping should be
2. Relog you until the ping is acceptable

There's several thoughts on this.

First, to detect the ping we should have.

We could ping the login page, that is always on the same server as far as I believe. So the ping should be relatively static.
This way if you switch internet connections, we could tell what ping you should be getting by comparing it to the ping on the main page.
Such as "main page has ping of 200ms, when logged in we should have a ping of 220ms or less"

However, this requires further verification to ensure this is viable.
Alternatively, we'd want to cache the pings in a rolling list, I don't have a solid idea on how to do this with different internet connections with different latency.

The interest behind different connections is because of the following; A user of the script had the issue where the cached pings were inaccurate as they switched internet connections. One day it'll be blazing fast, the next day it'll be slow. When its fast, they could be on the worst kol server and it'll say the ping is acceptable because it was looking at the slow day.

Secondly, when we relog.

By no means do I ever mean "Restart the mafia client" or "Refresh everything as if we were constantly logging out to the login window and back"
I mean we simply refresh the kol session we're connected to, the script does this by visiting logout.php, then telling mafia that we are logged out which makes mafia try to log us back in again.

(Due to an internal mafia check, the script has to wait 30s between each relog. We can obviously bypass this.)

By simply refreshing our login session, we do not need to reload our inventory, or character status or anything. We only change the session ID. This should take less than a few seconds and will switch the server we're connected to.

Because of how I'm thinking this could work, it can be a command as well as part of the mafia login sequence. You run the command as part of your daily scripts and it checks to make sure your ping is still acceptable, and relogs you if not.

Third, the logout and logins.

I'm not sure if this is viable, but some people do not use "Stealth Login". That's fine, but when we relog it could be annoying to people if you're constantly relogging to get faster internet.
As such, we can probably avoid the "logout" part of the process, since we only care about getting a new session. We don't need to "logout" as such.
Then when we login as part of the relogging process, always perform it in stealth mode.

Lastly, the preferences.

Only a few obvious preferences spring to mind.
A preference on the maximum amount of times we'll try to relog in an attempt to get a better ping.
A preference if we want to do this as part of the login.
A possible preference for debug purposes where it'll store the recent outcomes, and can deliver a warning if needed.

Closing thoughts

I'm not sure if I explained this sufficiently well, the end goal is to simply hop across kol backends until we find one that is not overloaded and will give us fast results.
Sometimes you'll get a server where it'll take 600ms to perform a request. Sometimes you get a server where it'll take 200ms to perform a request.
We can't tell which server we're on from what I know, only how long it takes to perform a request.

 

[MCroft]

We're getting an AWSALB cookie (and an AWSALBCors cookie). Can we just clear our AWSALB cookies without logging out? If KoL is using the Application Load Balancer the way Amazon expects, that will cause the LB to send us to a potentially different server. That will either work just fine or totally break because of server-side state.

The AWSALB cookie is encrypted with a rotating key, so it's not useful for determining the server.

I hope we're not making everyone else's experience marginally worse by cherry picking amongst healthy servers. If there is a way to aggregate and identify slow servers and provide that to Asym's Site Reliability Engineers, it might be worth some sort of collect script to help them determine what percentage of their servers are "sick" at any time. If they want to provide an encrypted value in api.php that told them which server a call was from, we could add that to data we provided.

 

[fronobulax]

Random thoughts:

I have no idea what "Stealth Login" is and whether I use it or whether I want to use it. It's an undocumented checkbox as far as I can tell

Some of the things suggested above seem to me to be already implemented so there is probably something I am not understanding.

The current code can be configured to test and evaluate and prompt the user to accept the connection or try again. Keeping the person in the loop seems to address some of the concerns - specifically if I have have moved from a good to bad connection I can choose to accept what I have been offered. If I understand the statistics collection and generation, if I am on a slower connection and keep refusing to reconnect the average will rise to reflect the slower location. A rolling average with some time sensitivity might be helpful but the user can manually zero out the stats if they don't appear to reflect the current conditions.

I'm not sure I understand the relog issues. The test is implemented before things related to state are fetched so there is no real cost to running the test over and over. It may also be that my experience is colored by my willingness to use the gUI and log in interactively. I can imagine someone wanting to micromanage the ping process with a script and do things differently than veracity did.

I assume KOL is balancing with session persistence which means once a logging in client is assigned to a server that server will process all requests from that client until the server goes offline or the client logs out. We don't know how the load balancer decides which server gets a new connection. We also don't know why there are a slow servers in the pool. Maybe something is misconfigured and KoL will eventually address that. Maybe some servers are faster than others and trying to avoid the slow ones will generate a reaction from KoL?

What would be really useful would be if KoL would somehow identify the server being used so we could use Excavator like technology to identify the slow ones and feed that back to whoever is managing the load balancer.

 

[MCroft]

@fronobulax , I stepped through the code and the requests return AWSALB cookies.

Those cookies are documented in AWS's Sticky sessions for your Application Load Balancer, which is probably how they're doing it. Nobody wants to roll their own ALB if they don't have to.

I think the scripters are expecting to use the gCLI command ping to detect and relog themselves.

 

[MCroft]

Random thoughts:

I have no idea what "Stealth Login" is and whether I use it or whether I want to use it. It's an undocumented checkbox as far as I can tell

Lost said:
It means other people don't see that you logged in (but a /who or /whois will show you as normal). You get this by adding /q at the end of your username.

 

[heeheehee]

The actual targets need tuning (if they're even viable at all), since you're going to run into network variability (especially with, say, users on wifi that are close enough to the servers). Or, if you're experiencing bufferbloat (which can add tens of milliseconds even with traffic shaping in place).

I also have concerns regarding users who use the same KoLmafia directory across multiple locations / ISPs, whether because they sync via Dropbox, or even something as simple as "I took my laptop on my Hawaii vacation, oh no now my ping is so bad that Mafia is stuck in a login retry loop."

We're getting an AWSALB cookie (and an AWSALBCors cookie). Can we just clear our AWSALB cookies without logging out? If KoL is using the Application Load Balancer the way Amazon expects, that will cause the LB to send us to a potentially different server. That will either work just fine or totally break because of server-side state.

That will most likely force a logout due to the aforementioned state (and as mitigation against historical server state desynchronization exploits, e.g. "snowcone bug").

Asym's Site Reliability Engineers

Last I checked, these don't exist. This is nowhere near a big enough operation to warrant having more than, like, one devops guy. (And in practice, probably not even that.)

 

[Veracity]

I asked Irrat to post this here. I only recently started looking at discord. As an introvert, I despise "chat" - but discord has a chat history, and I can look at selected forums - in particular, the KoLmafia coding one - and answer the occasional question. Or, as in this case, learn about various peoples' scripts which are only announced there. I saw the discussion of KoL lag (something I was well aware of; with low lag, my daily farming script might take 15 minutes. With high lag, it can take well over an hour.)

I've never used his script, but its existence inspired me to write two PRs (so far) to build-in some support for ping testing and manual re-logging-in. I asked him to post here so that we can discuss options for automated re-logging-in.

1. Ping Infrastructure

Ping request, manager, and command by Veracity0 · Pull Request #1799 · kolmafia/kolmafia

Something I've noticed for a long time is that different KoL servers can have vastly different processing time; sometimes it takes me 15 minutes to automate a character's daily turns, and sometimes...

github.com

PingRequest can call api.php, main.php, or council.php (for grins, since, Irrat's script calls that.)
- api.php can be called any time, whether or not you are in a fight or choice. The only exception is that it will redirect to afterlife.php if you are in Valhalla. It's also really fast: here in Boston, I've seen it as low as 22 msec (late at night), but 30 msec is an achievable average any time.
- main.php (council.php) take at least 3 times as long, transfer up to 4 times as much data, and can't be called if you are in a fight or choice; they will redirect.

PingManager will execute a PingTest(page, count) and return a data structure that contains: page, count, list of elapsed times, lowest time, highest time, total time, summed bytes. That structure can also calculate average time (total / count) and bytes per second (bytes*1000/total)
It will save the ping results in the pingLatest property (PAGE:COUNT:LOW:HIGH:TOTAL:BYTES:AVERAGE).
If it is a "standard" test - api.php with at least 10 pings - it will, as appropriate, save the result in pingLongest and pingShortest properties.

PingCommand

ping
ping 100
ping 10 main
ping 10 api true

Defaults to 10 pings, api.php, not verbose. You can run this logged in to see what you get - and to save the above mentioned properties.
A script could call this command and look at the pingLatest property.

2. Ping Test integration with the LoginFrame.

Optional ping test at login by Veracity0 · Pull Request #1801 · kolmafia/kolmafia

Four new settings: pingLogin - (boolean) run a ping test at login before refreshing session. Prints result to gCLI. pingLoginCheck - none, goal, threshold (Only if pingLogin is true) Require aver...

github.com

Properties:

1. pingLogin - (boolean) run a ping test at login before refreshing session.
   Prints result to gCLI.
2. pingLoginCheck - none, goal, threshold
   (Only if pingLogin is true) Require average lag to be below goal or historical minimum + threshold
3. pingLoginGoal - (int) msecs
4. pingLoginThreshold - (float) fractional percent above historical lowest average (pingShortest)

If pingLogin is true, we'll do a api.php (10) PingTest after you log in but before we do anything further to refresh the session data. This will save pingLatest, pingLongest, and pingShortest, as above.

If pingLoginCheck is not "none" (which is the default), we will evaluate the ping test and decide if it satisfies your configured requirements.
If so, we proceed with logging in and refreshing the session.
If not, we tell you what the ping was and what your configured maximum is and ask you if you are satisfied with the current connection.
Yes/No/Cancel will either continue logging in, logout and attempt a new login, or simply log out and leave you at the LoginFrame.

If you know what a "good" ping time is - In Boston, I use 30 msec - you can use "goal" and set the goal to the time in msec
If you want to simply use "something not too much higher than the lowest time I've seen", you can use "threshold" and set it to a fractional percent. That defaults to 0.20 - 20% slower than your best seen value. (Irrat uses 10%, but his script does not use api.php, which is has a way shorter elapsed time.)

3. Known issues

1. If api.php redirects to afterlife.php, you are in Valhalla. We can certainly run a ping test - in fact, if api.php redirects, we'll save it as a "afterlife.php" ping test - but we can't evaluate the quality of the connection. We should skip the pingLoginCheck.
2. As coded, this will recursively submit logout.php and then login.php - from within processing the previous login.php. That's not a big deal if the user fairly quickly accepts a ping test (how many would be too many?), since it will unroll harmlessly once that happens, but it should probably be coded to put the loop outside, so to speak. Some how.
3. If you get an anomalously low ping test - I usually get 27-33, say, but late at night I got a 22 - which gets saved as your "shortest" test, perhaps a "threshold" test will never pass. 22 msec with a 20% threshold would require 26 or 27 - and that is by no means guaranteed.

4. Summary

I need to ponder your note some more.

I like the idea of being able to login (and re-login) without requiring user confirmation - and it would be easier once I deal with issue #2.

Switching login servers by logging out and back in without reloading user state is very much like what we do already when we "time you in" after KoL logged you out. That is really fast. And, by the way, my existing "process login.php" ping testing should automatically kick in when we do that; once you accept the connection, it will continue to finish the timein, rather than finish the login. Huh. What will happen if you say "No" on a timein and it issues a logout? That just might screw up the timein. I'll have to test/fix that.

I hadn't considered Stealth login. I always use it, myself, but yes - I can see how those who don't would prefer that re-logs be done in stealth mode. That should probably go to "known issues". (Considering that we simply run the last LoginRequest in order to time in - or relog - people are already seeing time-ins as new logins, if you are not in stealth mode. That might be an existing issue.

More later after further ponderation.

 

[fronobulax]

I like the idea of being able to login (and re-login) without requiring user confirmation - and it would be easier once I deal with issue #2.

I thought about asking for this and decided against it. If a user's KoLmafia environment varies, for example: using Dropbox or equivalent and different computers; or using a laptop, it might not be possible to achieve the times observed on other networks. So if I am in the coffee shop and it asks me I'll accept the slower time because I expect things to be slower.

If the number of retries is limited and if a failure on the last one generates an unconditional login I could live with this but I would prefer not to have a potential infinite loop or a situation where I have to change a setting to login because the retries failed.

Might also be worth pondering the stats as well. It is simple to define what is collected but harder to assign meaning to them.

 

[MCroft]

That will most likely force a logout due to the aforementioned state (and as mitigation against historical server state desynchronization exploits, e.g. "snowcone bug").

I may be experimenting with this, but not with my main... The life on those cookies is like 60 seconds,

Last I checked, these don't exist. This is nowhere near a big enough operation to warrant having more than, like, one devops guy. (And in practice, probably not even that.)

Forgot the [sarcasm] tags again...

 

[Veracity]

Switching login servers by logging out and back in without reloading user state is very much like what we do already when we "time you in" after KoL logged you out. That is really fast. And, by the way, my existing "process login.php" ping testing should automatically kick in when we do that; once you accept the connection, it will continue to finish the timein, rather than finish the login. Huh. What will happen if you say "No" on a timein and it issues a logout? That just might screw up the timein. I'll have to test/fix that.

I just tried timing in.

I let my session sit idle for a long time and then hist the "Refresh Status" button, which submits api.php.
KoL redirected to "login.php?notloggedin=1" we timed in by resubmitting login.php.
It did a ping test which was acceptable, so it did not give me a dialog box to ask about it.
I still need to test that.

Seems like a "relog" command could be almost as simple as submitting logout.php and then timing in.
Almost, because what happens to your GUI if the user does not accept the ping and leaves you logged out?

I'll write that command and experiment.

 

[MCroft]

I'd like to request that "verbose" to be an alias for "true" in the gCLI parameter list. It took me a few moments looking at the help command to figure out that verbose wasn't literally verbose, but the value of a boolean and I needed to use "true"

If I have timed out, the first request to ping is interrupted by the login. Perhaps that ping should not count, either because we detect the login during the test or because we decide that anything > than 2 std deviations from the mean is an outlier and should be ignored. Or any outlier detection method that we like. We could also run one or two pings before counting the ones we want to keep, to get better accuracy.

> ping 10 api true

Ping #1 of 10...
Sending login request...
Ping test: average delay is 47 msecs.
Loading character status...
-> 1569 msec (1962 bytes)
Ping #2 of 10...
-> 44 msec (1962 bytes)
Ping #3 of 10...
-> 44 msec (1962 bytes)
Ping #4 of 10...
-> 44 msec (1962 bytes)
Ping #5 of 10...
-> 44 msec (1962 bytes)
Ping #6 of 10...
-> 46 msec (1962 bytes)
Ping #7 of 10...
-> 45 msec (1962 bytes)
Ping #8 of 10...
-> 46 msec (1962 bytes)
Ping #9 of 10...
-> 47 msec (1962 bytes)
Ping #10 of 10...
-> 48 msec (1962 bytes)
10 pings to api.php at 44-1569 msec apiece (total = 1977, average = 197) = 9924 bytes/second
Requests complete.

 

[Veracity]

That is funny! The login did its own ping test.

I have also usually seen that the first ping test after login is above my goal, but #2 works fine.

Here is an initial implementation of a "relog" command:

  @Override
  public void run(final String cmd, String parameters) {
    RequestThread.postRequest(new LogoutRequest());
    LoginRequest.retimein();
  }

"retimein" is like a timein that is not triggered by KoL redirecting.
Some tests:

1) relog where I rejected the first ping and the second was good enough

> relog

Sending logout request...
Logout request submitted.
Sending login request...
Ping test: average delay is 122 msecs.
Sending logout request...
Logout request submitted.
Sending login request...
Ping test: average delay is 22 msecs.
Loading character status...
Requests complete.

2) relog where I canceled the first ping and was left logged out.

> relog

Sending logout request...
Logout request submitted.
Sending login request...
Ping test: average delay is 31 msecs.
Sending logout request...
Logout request submitted.

I then hit the "refresh status button" and, since we were logged out, KoL redirected and KoLmafia timed us in, as usual.

Loading character status...
Sending login request...
Ping test: average delay is 31 msecs.
Sending logout request...
Logout request submitted.

I rejected the first ping check and the second was good enough.

Sending login request...
Ping test: average delay is 22 msecs.
Loading character status...
Requests complete.

The status request succeeded

3) relog where I canceled the first ping and was left logged out.

> relog

Sending logout request...
Logout request submitted.
Sending login request...
Ping test: average delay is 33 msecs.
Sending logout request...
Logout request submitted.

I then hit the "refresh status button" and, since we were logged out, KoL redirected and KoLmafia timed us in, as usual.

Loading character status...
Sending login request...
Ping test: average delay is 31 msecs.
Sending logout request...
Logout request submitted.

I canceled the first ping check and KoLmafia resubmitted the status refresh and it tried to time in again.

Sending login request...
Ping test: average delay is 82 msecs.
Sending logout request...
Logout request submitted.

I canceled the second ping check and KoLmafia resubmitted the status refresh and it tried to time in again.

Sending login request...
Ping test: average delay is 30 msecs.
Loading character status...
Requests complete.

The status request succeeded

 

[MCroft]

If I use relog until I get the dialog because my ping is too high and then I choose cancel, it allows me to try things in the gCLI and throws errors.

I'd like it to go back to the login window like typing logout in the gCLI does.

> relog

Sending logout request...
Logout request submitted.
Sending login request...
Ping test: average delay is 64 msecs.
Sending logout request...
Logout request submitted.
> pull line

Pulling items from storage...
Sending login request...
Ping test: average delay is 47 msecs.
Loading character status...
Server returned response code 404 for storage.php

 

[MCroft]

Is there a way other than editing the prefs to change the threshold or goal value when doing a relog? Either parameters or a Preference page option to set the same things once the character has logged in.

 

[Veracity]

If I use relog until I get the dialog because my ping is too high and then I choose cancel, it allows me to try things in the gCLI and throws errors.

I'd like it to go back to the login window like typing logout in the gCLI does.

If you choose cancel, you are logged out, but the GUI/gCLI still exist.
If KoL times you out and logs you out behind the scenes, you are in the same state.
They SHOULD behave identically.

Which is to say, if you try to do something that sends a request, we'll time you back in and when it resubmits the request, it will succeed. That is how it worked when I hit the "refresh status" button after canceling. Your 404 error on the pull surprises me. I will investigate.

I feel strongly that sending you back to the Login frame on a cancel is incorrect. That is how it works if you cancel when you are logging in initially, but making relog do that would require, essentially, that the "logout" command execute.

 

[heeheehee]

I'd like to request that "verbose" to be an alias for "true" in the gCLI parameter list. It took me a few moments looking at the help command to figure out that verbose wasn't literally verbose, but the value of a boolean and I needed to use "true"

I did the same thing, so I support this request. As an alternative, I'd also support printing an error if the value is neither "true" nor "false".

 

[heeheehee]

Ah, looks like you made that change in r27450.

I haven't fully decided how I feel regarding cancel on timein, but my gut agrees with MCroft's suggestion to present the Login frame, based purely on the wording in the prompt:

    buf.append(" Press 'Yes' if you are satisfied with the current connection.");
    buf.append(" Press 'No' to log out and back in to try for a better connection.");
    buf.append(" Press 'Cancel' to simply log out.");

 

[fronobulax]

Anecdotally, I built r27450 and ran with it today.

pingLogin=true
pingLoginCheck=threshold
pingLoginGoal=0
pingLoginThreshold=0.2

I wanted it to check the ping and prompt me if it was higher than I wanted. Everything worked as expected. The anecdote is all three characters initially had a bad ping of 35 +/- 1 and on the second attempt succeeded with a reported 25 +/- 1

No significance to this observation beyond the discussion about whether the first ping often took longer than subsequent pings.

Since the behavior is being discussed if I cancel I would like the state to be logged out with the KoLmafia gUI on screen. I believe that is the status quo.

 

[Veracity]

I've pondered this enough that I think I see how I want things to work.

1) User scripts

Given the "ping" command, the current ping login properties, and the "relog" command, I think a user script like Irrat's could be easily written/improved.

• You can invoke a "ping" command - even for a single request - and parse pingLast to see how it did.
• If your script wants a rolling average, so to speak, asking for multiple requests - like 10 - will calculate it for you.
• If you don't like the result, you can cli_execute "relog" and it will log you out and time you back in.
• note that if pingLogin is false, KoLmafia will not do any of this automatically; your script has complete control.

I am hoping to make scripts like this obsolete, but I'm not going to mandate you use KoLmafia's built-in ultimate support for the equipvalent.

2) Useful ASH functions

PingTest ping(string page, int count)
PingTest ping()

The second just uses the "default" page and count, which qualify for saving in pingLongest and pingShortest

The return value is an anonymous record - like we return for certain other ash functions, like item_drops_array, maximize, svn_info, and git_info.

This would obviate using cli_execute to call the ping command and having to fetch and parse the pingLast property.

3) Historical ping data

Insight about saving historical dats: the results are valid for a particular computer with connected to a particular network.

For example, I have desktop Mac mini with an M1 processor, connected by wired Ethernet to a router which is connected to the Internet via gig ethernet. I happen to be in Boston. Given the combination of all those factors, I get 30 msec api.php pings, which is, anecdotally, quite fast. If I run KoLmafia on my (much slower) laptop and connect to various wireless networks in random other cities around the world, I would expect wildly varying ping timings.

I conclude that if you have a fixed computer/location, historical measurements, as currently collected, should work well. But if you switch computers and/or networks, you presumably want to clear the historical data and start collecting it anew.

• A program could do that by setting pingShortest (and pingLongest, if desired) to "".
• I could add "ping reset" to do that.
• How about a way to do that on the LoginFrame Connection tab?

4) Improvements to the Connection tab

The Connection panel lets you change the 4 properties controlling ping testing and checking

• Put that panel in Preferences, also, for easy changes while already logged in.
• Add a text field which displays the current historical measured ping times - lowest (and highest?)
  (that field can listen to pingShortest and pingLongest properties and auto-update)
• Provide a button - "Clear" - which clears the historical data.
• Given that, if your mobile computer logs into a new network, just just press that button.

5) The "relog" command

It's also called "relogin". I think that "timein" is actually a better name - with the other two as synonyms. I like that because it tells you that it logs you out (like KoL does when it times out out) and then times you in, just like our current functionality for KoL-induced timeing out.

6) Bugs

• When we run a ping test at login and you are in Valhalla, do not check the "afterlife.php" result for goodness.
• Any "timein" - whether spurred by a KoL timeout or by the user looking for a bettwe connection - uses the last saved LoginRequest (as always - but with "/q", so it is a Stealth login.

7) End goal

Provide login/timein options to configure/allow doing multiple login/logout/timein requests until you get a good enough connection or have made enough attempts.

Details of that are still TBD, since I think every else I mentioned are necessary precursors, but I think that is the ultimate goal of this Feature Request.

 

[Veracity]

Is there a way other than editing the prefs to change the threshold or goal value when doing a relog? Either parameters or a Preference page option to set the same things once the character has logged in.

I moved ping configuration into a new panel class and now include that in both Login/Connection and Preferences/General.
I also added a Text Area which displays your historical shortest and longest ping tests.
That auto-updates when the preferences change.
I added a Clear button on that panel that resets the history.
You'd use this when logging in on a new network or computer.