Feature - Implemented Relogging to improve ping

[Veracity]

With the exception of one niche case, I believe this feature is complete. Highly configurable, takes a little setup to tune for your own needs, requires some (trivial) attention if you switch computers and/or networks, but I’m not seeing how a built-in integrated solution could work better. And I think the Preferences/General/Connection Options panel is super sweet.

The core technologies of this are ping testing and timing in a session. I think the former is complete, but there are a few improvements of interest to the latter.

1) Login and Timein to Valhalla. api.php redirects to afterlife.php. I want to experiment - and test - this myself.

(With almost 90,000 banked karma, all skills HC permed, and a complete Manuel, I only ascend for fun, these days. Manually, in browser; I have never used an ascension script. So, 2-3 hours a day of attention. Perhaps I’ll have a lengthy Heavy Rains experience to fill up my locket using Rain Man, by and by, and will linger a bit in Valhalla, first, to address this topic.)

2) Timein needs to refresh your password hash. It calls lchat.php and examines on-page links to learn it. Irrat points out that api.php has it - and timing in calls that -- but we don’t parse it from there. Why not? Well, Valhalla - but surely we could call api.php and only call lchat.php if it redirects.

3) MCroft is examining timein interactions with the preferences file. Now that I am not completely focused on ping testing, perhaps I should look at his PR.

Only point #1 specifically deals with this feature. I guess when I investigate Valhalla, by and by, I’ll be able to close this.

Modulo bug reports, of course.

And I suppose a little document (post) explaining how to use configure it for (all?) common use cases will help.

 

[Ethelred]

With the exception of one niche case, I believe this feature is complete. Highly configurable, takes a little setup to tune for your own needs, requires some (trivial) attention if you switch computers and/or networks, but I’m not seeing how a built-in integrated solution could work better. And I think the Preferences/General/Connection Options panel is super sweet.

[...]

And I suppose a little document (post) explaining how to use configure it for (all?) common use cases will help.

Oh yes please! This feature is complex enough that I've postponed updating because I have no idea how to begin with the configuration. I eagerly await a configuration how-to. Thanks for all your work implementing this and I'm looking forward to using it. I think it will solve an ongoing problem I've been having for several months where one (or more) of my chars takes 2 or more hours to run my daily aftercore script, as opposed the the typical 30 to 40 minutes.

 

[fronobulax]

r27470

I was used to getting times under 30msec. I was getting times in the 50s and decided to accept them and move on. I then got a lot of "Server returned response code 502 for api.php" messages. I think running auto_mushroom caused the first one. I logged out and cleared my stats and tried again. Getting 502s for closet, questlog, campground, desc_effect, standard and api.

All of the preferences that start with "ping":

pingDefaultTestPage=api.php
pingDefaultTestPings=10
pingLatest=api\:10\:16\:78\:522\:15696\:52
pingLogin=true
pingLoginCheck=threshold
pingLoginCount=0
pingLoginFail=login
pingLoginGoal=0
pingLoginThreshold=0.2
pingLongest=api\:10\:16\:78\:522\:15696\:52
pingShortest=api\:10\:16\:78\:522\:15696\:52
pingStealthyTimein=false
pingTestPage=api
pingTestPings=10

I cleared the stats, set the page for the login test to be main, shut down KoL, logged in I did not see any 502s. The reported time is about 10x what I expect but I am willing to adjust my expectations.

Reporting because the 502 breaks a couple of scripts that are not prepared to handle a bad response.

Thanks.

 

[Veracity]

Yes. KoL made a change yesterday afternoon which nullifies being able to pin your session to a fast server. As a temporary bonus, it was buggy, resulting in 502 errors - which, per the definition of HTTP response codes, denote server bugs.

We’ll see how this all shakes out.

 

[fronobulax]

Yes. KoL made a change yesterday afternoon which nullifies being able to pin your session to a fast server. As a temporary bonus, it was buggy, resulting in 502 errors - which, per the definition of HTTP response codes, denote server bugs.

We’ll see how this all shakes out.

Still running the same sessions and a couple of 504s have crept in. Standing by.

 

[snooty]

Could someone explain how to use this please? I entered the "prefref antilag_" into the gCLI, and "Name Value Default Scope" pop up, but, I have no idea what to do with any of that. I've been trying to run one account for the last hour and a half now, and manage to get literally *one* adv done before the 504s run rampant, and mafia claims I'm in a fight and it just freezes. When I log back in, I'm not in a fight at all. Updated to the latest version (27471) with the same effect.

 

[Veracity]

antilag is Irrat's script. KoLmafia has built-in login ping-testing support that is a strict superset of that script.
However, neither antilag nor KoLmafia's built-in equivalent works any more.
KoL has changed how it assigns servers to execute user requests and you can no longer relogin until you get a "fast" one.

504's are, essentially, I/O errors, if KoLmafia submits a request and gets such an error in return, it has no way of knowing whether the request succeeded - and KoL simply did not report it - or failed. I am not surprised that you might have finished a fight, but because of the KoL bug that returned a 504, KoLmafia does not know that it is done.

I am sorry you are afflicted so heavily with KoL's 504 bug. There is nothing you can do except hope that KoL fixes it soon.

 

[snooty]

Thanks for the reply, and, darn it! I stepped away to get some real life stuff done, so hopefully they've fixed it by now.

 

[Veracity]

KoL has, for now, rolled back its attempt to assign a new server to every request you submit.

The goal was to address the issue where two of the servers were always at 100% CPU usage and the other 3 were at 40%.
Which is to say, fix the thing that observant users - and now KoLmafia - noticed and attempted to workaround by re-logging in until you got one of the "good" servers.

Doing that would also allow autoscaling during periods of increased user activity - at Crimbo, for example.

Unfortunately (since I support both of those goals), they discovered major issues in their software (or the packages it depends on), with the result we observed for several days: 502/504 server errors, and the worst lag I've seen in years.

Work will continue on that project - on a server which is NOT user-facing - but, for the next few months, at least, this project remains useful.

Revision 27482 (or is it 27481) adds a new option: if you fail to get a fast enough connection after the configured number of retries, instead of simply accepting or rejecting it, you can "confirm" via a user dialog. Based on whether you have a "goal" or a "threshold" configured, you will be given the option to clear your ping history - which will automatically let KoLmafia re-learn how fast connections should be on whatever (new) network you are connected to.

I'm going to write an Announcement, by and by, explaining what this project does and how to configure it - and how to test and fine-tune, if you are so inclined.

But the tl;dr; configuration recommendation is this:

If you don't really want to make the effort of experimenting and fine-tuning - as easy as that SHOULD be from the Preferences/General/Connection Options panel - the following are good settings, which you can set on either the Login Frame/Connection Options, or from the aforementioned Preferences panel, which also gives the ability to run ping tests and Time In, to try for a better connection. (You can use the "ping" and "timein" (or "relog") commands from the gCLI, as well, but the Connection Options panel is your one-stop shop.)

Run ping test at login to measure connection lag - CHECKED
Kol Page to ping - api
(You have 5 options. This is fastest, does not vary depending on user state, and can be run any time.)
How many times to ping that page - 10
(There are always anomalously long pings. More pings in a test minimizes their effect.)
Login ping check type - threshold
(Compares ping times to the lowest you've ever seen. Reliable - unless you change networks.)
Allowed threshold above minimum historical lag - 0.2
(20%. If your best api test was 25, 30 is acceptable. If best was 40, 48 is acceptable.)
Attempted ping checks before giving up - 5
(I've never seen this fail, given an acceptable goal or threshold - until KoL broke things, late last week.)
Action after ping check failure - confirm
(If you DO fail to get a connection - perhaps because you changed networks - helps you diagnose it.)

With the above, this should be "set and forget".

I have a couple more things I MIGHT do:

- The ping command gives more raw info that the ping test on the Preferences panel. Interesting, but not really necessary. But I might beef up the Preferences panel...
- I am content to save averages as integers, but when calculating them - or comparing to thresholds - I should probably use Round, rather than Floor.
- And, yeah - api.php does not work in Valhalla. I'll jump the gash, eventually, and will experiment.

I will make the Announcement soon.

 

[phreddrickk]

Ran into an interesting bug: my character was trapped in the bookmobile noncombat, and I absentmindedly hit the "refresh all" button after I had presumably logged out due to inactivity:

Visiting The Bookmobile
Encounter: The Bookmobile
Requesting store inventory...
Sending login request...
Preference pingLatest changed from main:10:143:199:1647:154120:165 to main:10:0:0:0:0:0
Preference pingShortest changed from main:10:81:116:951:125590:95 to main:10:0:0:0:0:0
Ping test: average delay is 0 msecs.

Because I was unable to hit main.php, I got some pretty unexpected results

 

[MCroft]

Ran into an interesting bug: my character was trapped in the bookmobile noncombat, and I absentmindedly hit the "refresh all" button after I had presumably logged out due to inactivity:


Because I was unable to hit main.php, I got some pretty unexpected results

I had the same problem during rollover. We probably ought to abort on 0 and not save the value.

 

[Veracity]

Yeah. This is among the reasons main.php is not a good choice for a ping page: you can’t call it from a choice or fight.

We supposedly detect redirecting to afterlife.php, but other redirects should abort ping testing and not affect statistics.

 

[BigVinnie]

I have to say that I like this solution, and I am very grateful to the KoLMafia dev team for implementing it. The first best outcome would be for TPTB to fix it server-side. Until then, the Mafia fix is an excellent workaround. A hearty thank you to everyone involved!

 

[MCroft]

One thing I just noticed, and I'm not sure if there's an ideal way to address it, is that if I timein by pressing the "Load in Web Browser" toolbar button and the connection is too slow, the dialog box is opened in the app that is now behind my web browser.

The app will switch to a browser window and sit on a partially loaded (but not displayed or rendered) page for me to notice that it hasn't loaded, go back to the Swing App, and accept or retry to find a faster server.

 

[Veracity]

Perhaps that button should submit api.php to force a timein, if necessary, before passing a URL to the browser to request.

I expect if you already have the Relay Browser open and let your session timeout, when you come back to the browser and have it submit something, the same thing will happen, but the GUI will already be behind the browser.

In both cases, it is a RelayRequest which triggers the timein. That’s always been a bit tricky, since your password hash and session id change.

 

[MCroft]

Getting back to this, 2 months later.

I've got working code that runs a ping test before opening the relay browser if we have the pingtest pref set. This causes the too-slow dialog to pop up before the relay browser loads, which I think narrowly solves my problem.

I haven't created a PR yet (I need to capture enough data to write a test for it), but I have it working and would like some comments/pre-review, if you think there's anything we should add.


1: Should this have it's own pref or should it piggyback on pingLogin?
2: Should it go to the gCLI, the SessionLog, or the DebugLog if debugging?
3: Any notes on how to test this well would be appreciated.

RelayLoader.java change:

public static void openRelayBrowser() {
KoLmafia.forceContinue();
// TODO make sure ping fails don't wait for user input behind loading browser window
if (Preferences.getBoolean("pingLogin")) {
PingManager.PingTest pingTest = PingManager.runPingTest(false);
RequestLogger.updateSessionLog("Relay Loader Pre-Ping Average: " + pingTest.getAverage());
}
openSystemBrowser("game.php", true);
}

 

[MCroft]

It doesn’t solve the timein on relay reload when the browser is in the foreground.

Thinking through that one.

 

[Irrat]

I'm not sure but it feels like we might need to actually log out now, instead of just doing a fresh login.
I'm getting logged out errors whenever I time back in.

 

[MCroft]

I'm not sure but it feels like we might need to actually log out now, instead of just doing a fresh login.
I'm getting logged out errors whenever I time back in.

Can you clarify this? Or explain when?


I am seeing what you might mean, but not always.

> timein

Sending login request...
Ping test: average delay is 65.50 msecs.
Loading character status...
Requests complete.

> timein

Sending login request...
Ping test: average delay is 68.10 msecs.
Sending login request...
Ping redirected to 'login.php?notloggedin=1'; ping test aborted
Loading character status...
Requests complete.

> timein

Sending login request...
Ping test: average delay is 668.00 msecs.
Ping test aborted because 1 ping exceeded 397.00 msec.
Sending login request...
Ping test: average delay is 46.00 msecs.
Loading character status...
Requests complete.

 

[Irrat]

Can you clarify this? Or explain when?


I am seeing what you might mean, but not always.

> timein

Sending login request...
Ping test: average delay is 65.50 msecs.
Loading character status...
Requests complete.

> timein

Sending login request...
Ping test: average delay is 68.10 msecs.
Sending login request...
Ping redirected to 'login.php?notloggedin=1'; ping test aborted
Loading character status...
Requests complete.

> timein

Sending login request...
Ping test: average delay is 668.00 msecs.
Ping test aborted because 1 ping exceeded 397.00 msec.
Sending login request...
Ping test: average delay is 46.00 msecs.
Loading character status...
Requests complete.

Yeah, I am probably wrong.

I think they may have changed the backend stuff again, my ping has been remarkably consistant.

In discord, people were reporting login issues after rollover, being logged out after they had logged in.

But I also noticed this problem on a non-mafia script which only switches between multiple accounts, if they login too close to each other they seem to run into the issue.

I've attached a picture below on what it basically looked like for me. It happened earlier in the day too after I logged in, but this picture is after I've completed an entire leg and am about to jump the gash as part of my looping script.

To be clear, my script isn't doing anything special other than calling "relog" via CLI and visitUrls to try test ping.
The script itself

That said, on reflection I believe that I am just incorrect. That this is more of an issue in the backend instead of something mafia has to think about.