Bug - Fixed KoLmafia Version Check via SourceForge disabled,

fronobulax

Developer
Staff member
As of r18250 the call to visit https://sourceforge.net/p/kolmafia/code/HEAD/tree/src/net/sourceforge/kolmafia/KoLConstants.java to get version information (to check point releases) has been commented out. It was failing (handshake failure, suggesting an untrusted certificate) and a side effect of the failure was that the fix to allow file_to_map to handle https:// did not work. (Note that is is possible the point release update check has not worked since 2013 since that is when the change was made from http:// to https://).

See http://kolmafia.us/showthread.php?22155-file_to_map-needs-https-handling for additional discussion.

I was unable to figure out why this failure had side effects that effected file_to_map. I did not want to disable security checking or distribute a certificate with KoLmafia (the only suggestions I could find to deal with the handshake error) without input from others. It could also be argued that point release checking could be declared obsolete, but again, that is not a decision I feel like making unilaterally. Hence this report.
 

fronobulax

Developer
Staff member

fronobulax

Developer
Staff member
The good news is that the revised URL does work, in that KoLmafia successfully fetches the file and checks the version.

The bad news is when I restore the check using the revised URL, the price update fails with a 403.
 

heeheehee

Developer
Staff member
Well, that's interesting.

403'd result said:
GET /scripts/updateprices.php?action=getmap HTTP/1.1
Connection: close
User-Agent: Java/1.8.0_144
Host: kolmafia.us
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

200'd result said:
GET /scripts/updateprices.php?action=getmap HTTP/1.1
Connection: close
User-Agent: KoLmafia v17.6 Java/1.8.0_144
Host: kolmafia.us
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

You'll note that the user agent strings differ. I wonder what's screwing with the UA string.
 

heeheehee

Developer
Staff member
And by screwing with the UA, I mean causing it to not be set (indirectly) via GenericRequest.reset() in the case where it's erroring. Maybe some class is caching the value of the environment variable?

Also worth noting: this doesn't seem reproducible under --CLI conditions. Possibly the invocation of DEFAULT_SHELL.attemptLogin("")?

Adding GenericRequest.setUserAgent() right before the UpdateCheckRunnable invocation seems to do the trick, although that feels like a workaround for the buggy behavior.

Aha.

http://hg.openjdk.java.net/jdk8/jdk...www/protocol/http/HttpURLConnection.java#l210

It's initialized in a static block, which means it won't be executed until the class is loaded. Which I guess happens the first time we do any sort of HTTP(S) request
 

heeheehee

Developer
Staff member
To prevent similar issues from cropping up in the future, I recommend putting the GenericRequest.setUserAgent() invocation as early in main() as possible.
 

zarqon

Well-known member
I seem to recall adding code to the price update script that checks that the user-agent is KoLmafia. Some of my other mafia-interactive server-side scripts do that.
 

fronobulax

Developer
Staff member
r18259.

I set the user agent and restored version checking and everything seems behaved. The code to only version check once a day was incomplete, so I finished it. Lightly tested.

If the price checking script cares about the user agent than an alternative fix would have been to tell the script not to do so. Catch me in a cranky mood and I'd argue that should have been done, but this works too.

heeheehee - what did you do to capture the packets? My tool caught a size difference but your display made it clear User Agent was the difference?

I'll mark this and the related file_to_map issue as fixed since sometimes optimism is justified.
 

zarqon

Well-known member
Yes. I haven't had access to said script since it was moved to kolmafia.us years ago. If I were in a cranky mood I might mention that this is exactly the sort of thing I warned might happen when people wanted it to be moved. Fortunately I'm feeling great and not the least bit cranky, particularly now that we seem to have solved everything. :)
 

Veracity

Developer
Staff member
I seem to recall adding code to the price update script that checks that the user-agent is KoLmafia. Some of my other mafia-interactive server-side scripts do that.
How does that interact with people like me who tell KoLmafia to use a specific User Agent which is NOT "KoLmafia"? I added that feature - and use it - as a (paranoid) defense against KoL deciding to throttle requests, say, from KoLmafia.

I haven't look at my HTTP headers recently to verify it's actually still working.
 

heeheehee

Developer
Staff member
heeheehee - what did you do to capture the packets? My tool caught a size difference but your display made it clear User Agent was the difference?

I also used wireshark, but I looked at the HTTP stream, since it seemed to be an error on that level -- you had reported an actual HTTP status code (403), which indicated that it wasn't a TCP issue.

How does that interact with people like me who tell KoLmafia to use a specific User Agent which is NOT "KoLmafia"? I added that feature - and use it - as a (paranoid) defense against KoL deciding to throttle requests, say, from KoLmafia.

I haven't look at my HTTP headers recently to verify it's actually still working.

My understanding is that it should work just fine if you're just using a browser useragent -- everyone who tested in a browser was unable to reproduce.
 

fewyn

Administrator
Staff member
Here's a current copy of the script.
 

Attachments

  • updateprices.php
    4.6 KB · Views: 29

zarqon

Well-known member
Veracity, if memory serves (can't view the attachment on my phone at the moment), other user agents will get either an error message or an empty page. So simply removing the agent check should be fine and would avoid similar problems in the future. At the time it seemed an easy way to filter out unwanted access. I believe it predated the ability to specify user agent from within KoLmafia, and was never changed to accommodate that ability after the fact.
 

zarqon

Well-known member
And having just looked at it (thanks Fewyn), there's no user-agent check at all. Sorry for the red herring.
 

fewyn

Administrator
Staff member
Ooo... that might be Cloudflare (to stop bots and such)? Let me check some settings and I'll get back to you guys.
 

heeheehee

Developer
Staff member
Ooo... that might be Cloudflare (to stop bots and such)? Let me check some settings and I'll get back to you guys.

Yeah, that was my gut reaction too. Doesn't really matter, since we already have a mitigation in place.
 
Top