New Content - Implemented api.php

[StDoodle]

CDM has apparently rolled out a new way for tools such as KoLmafia to query KoL for account data, and it's "api.php". Will post more info as it becomes available.

Results of map_to_file() on api.php : View attachment _mafiaAPI.txt

 

[Veracity]

I posted a thread about api.php on the dev forums on November 30. And I followed up this morning when CDM told me that he'd added selected account flags to the "status" request for it. I looked at it and told him about the two things we currently get that are not there - and he added them on the spot.

We still need to go to the Account tab to get Bad Moon/Skills Recalled/Pathed Run info, but we could eliminate the Interface/Inventory/Combat tab visits in favor of a single call to api.php.

 

[StDoodle]

We still need to go to the Account tab to get Bad Moon/Skills Recalled/Pathed Run info, but we could eliminate the Interface/Inventory/Combat tab visits in favor of a single call to api.php.

Badmoon: wouldn't a combination of "sign" and "roninleft" do the trick? (I could be missing something...)
SKills Recalled: yeah, I don't see anything to help with that. Could probably do for a request... I'll put it in as soon as the thread is up on GD, as CDM promised. Actually, I have no idea where this appears on the regular page anymore...
Pathed Run: there's a "path" for that

I'm guessing some of these are probably new since you last looked; seems to be a work-in-progress. Probably not worth considering as of yet, but if it gets to the point -- soon, hopefully -- where all info can be grabbed from here, that would be awesome.

Edit to add:
This is kinda nifty so far. I was worried by the (sparse) documentation that getting inventory with the api would be limited to 100 items, but the limit doesn't appear to apply to that. Seems like a rather quick way to parse non-closeted non-dc items. Which should probably get requested.

 

[Veracity]

I added reading of custom outfits as part of startup partly to enable this: in theory we could simply read api.php?what=inventory as replacement for reading the three pages of equipment.php individually, but by skipping page 2, we would no longer pick up the list of custom outfits. So now we go straight to the page where you can manage them and get the list there - which has the advantage that we now know all the outfit pieces. That adds a page load to login, but replacing the 3 equipment page loads with one call to api.php would end up in a net decrease of one page load and more info available. Score!

Except: api.php's inventory page is simply a list of pairs: itemId, count. No descid - and that means that we can't learn new items from it, as we can from the inventory page. That's a problem! Right now, you have a new item in inventory, we see the itemId and the descid, look up the description, and register and override the item and you can use it as if the item were already built-in to KoLmafia source.

I'm considering asking CDM for a "verbose" inventory api that would give itemids, descids, and count; the "status" page already gives effect names and descid, just like the charpane, so why not let inventory give descids?

 

[StDoodle]

That seems like a good request to me, too. What's the point of an api page if it won't save page loads? I imagine it's more of an issue of "never thought of that being important" rather than "didn't want to share that" from CDM's point of view.

 

[Veracity]

Revision 9021 is the start of this: it asks for api.php?what=status and gets the config flags from it that we otherwise get from visiting the Interface, Inventory, Combat, and Account tabs of Options.

I might be able to eliminate the initial CharPane update.
I'll be able to get a lot of stuff that is on the CharSheet, but we still need to go there to get your skills.

Getting ascension count, password hash, and such from the very first request should simplify some of the rest of initialization.

This is far from done, but the framework is in place and I did eliminate 2 page loads from refreshing the session.

 

[Veracity]

Revision 9042 does more, following some things CDM added at my request:

- It eliminates the last call to account.php - the Account Tab - because we can now get "recalled skills" from api.php?what=status
- (We also get "king liberated" from there now. We still notice when you do it and open up things for you immediately, but if you did it outside of KoL, say, we notice when you log in as part of the initial status load.)
- We read equipment and stickers from api.php?what=status
- We read all inventory from api.php?what=inventory
- Therefore, "refreshing inventory" is now closet.php + api.php?what=inventory, instead of closet.php, equipment.php page 1, 2, and 3, and bedazzle.php.
- If we notice you have an unknown item equipped, we can scrape the description and override it. That's new!
- For that matter, if we notice an unknown item in inventory, we can detect and override it, whether or not you have inventory images enabled.

I need one more inventory thing from CDM - familiar item locked flag.
And I should try to eliminate the initial load to charpane.php; CDM added the current MCD setting to api.php?what=status at my request, so I could do that.

So, this is still not "done" - but we're definitely making good use of api.php now. In total, so far, I have replaced 8 page loads at login with 2 smaller and much server-friendlier (no database hits!) page loads.

 

[gemelli]

I think that CDM is doing some tinkering on the dev server which is temporarily returning bad JSON strings from api.php requests. When using the dev server, I can use Mafia builds up to r9039 without experiencing significant problems, but build r9040 gives me visible instances of the "Auto-Sell ('+i.s+' meat): ['+one+']'; if (showsome)" etc. code throughout the character and menu panes.

Is it possible to add fallback logic to account for api read errors? I mean, I realize that this is currently affecting a single-digit number of users here, and that I could always log into a live server instead But I also assume that fail-proofing code might be helpful for other scenarios down the road.

 

[Veracity]

Well... all the code to scrape info from normal user requested pages - account.php, equipment.php, and so on - is still there and still scrapes the same info as ever from those pages. We just don't call them at login any more.

I'm puzzled about why strange things would go into the charpanes and menu panes, since KoLmafia does not create the HTML for those pages. Are you seeing messages like the following:

api.php?what=inventory returned a bad JSON string!
Error parsing JSON string!

in your gCLI after you log in? If we truly get "bad" JSON strings back - including the data not being what we expect (integers rather than arrays, etc.) - we get & trap and log JSON exceptions like that.

 

[gemelli]

I'm seeing the "api.php?what=status returned a bad JSON string!" message in the CLI, but not the "Error parsing JSON string!" message. I don't know if that's helpful.

It's entirely possible that there are other changes taking place in those panes that I don't know about, in which case I cry your pardon. I just thought it was interesting that the weird parsing tags were visible on builds 9040 and later, and assumed there might be something happening behind the scenes.

 

[Veracity]

Well, yeah: if it gets the first error, it didn't get any structured data to look at and generate the second error for.

I suppose I could set up a "slow login" option which will go to all the old pages, just as before. I'd have to think how to make it an automatic "fallback".

 

[gemelli]

Is it worthwhile to provide a debugging output feature that logs the specific JSON string that was found to be invalid? I've been poring over the results of raw calls to api.php?what=status&for=gemelli+testing (my account) from both the live server and the dev server, and I honestly can't see any different in the outputs at all. Feeding both into JSONLint, they both validate just fine.

This is admittedly me being a little selfish But I'm hoping to ensure that we can continue to have active Mafia users on the dev team so we can identify changes that will potentially cause Mafia breakage ahead of time. Right now, we've all rolled back to r9039, or chosen not to use Mafia when looking at testing content on the dev server. It's a decent enough workaround in the short term, but not ideal.

Thanks in advance for any suggestions you might have! I'm happy to post the actual JSON output strings here if you think that's helpful.

 

[Veracity]

Post it here or email it, if you prefer, to me: veracity at hambo dot com

 

[holatuwol]

It looks like there's some HTML in the JSON string talking about a missing variable. I'll commit a fix.

 

[gemelli]

Thanks, Hola. I don't see the HTML myself, but I trust your instincts JSON string outputs from live and dev servers attached anyway.

 

[holatuwol]

I just logged in to the dev server, saw the error described in this thread, added some printouts to see the response text, saw a missing variable notice, added some logic to skip over that to get to the actual JSON text, logged in again, and saw no error. But your error could be different, and the fix I committed might not work for you. xD

 

[Veracity]

Thanks, Hola. Considering that it was a bug on the dev server - which I cannot get to - I couldn't have debugged this on my own; the status strings Gem uploaded here did not have the HTML error message preceding the JSON string.

 

[gemelli]

Problem confirmed to be fixed in r9065. Thank you both for your help!

 

[Veracity]

I'm marking this Implemented, since I think we do everything that makes sense with what is currently available. I have half a dozen little suggestions/requests in to CDM that would allow us to do more - obviate the initial charpane.php load during login, auto-learn plurals and multi-usability for unknown items, and so on - but unless/until he adds them, I'm done. If he DOES add them, I'll use them without needing this thread to remind me.

There's also api.php calls for reading events and kmail in bulk without having to parse HTML, which might be more reliable and/or quicker, but I'm not going to touch those. Somebody more familiar than I with the kmail interface in KoLmafia might enjoy looking at it. Or not.