JSON

[fronobulax]

@gausie has implemented an alternative to the Display Case Database, hosted at Coldfront. https://museum.loathers.net/

I like to know where my characters are on various Display Case leaderboards and I currently do this by one query per item to Coldfront for each character. The Museum allows me to make one query per character and then process the results and end up with the same information.

I can webscrape the Museum but the results are also available in JSON so I could ask for JSON and process that instead.

I can't find much ash support getting from JSON to a map. Maybe it is there and is so obvious that I am just clueless. But if it is there I would like some pointers, please.

I'm told JSON is not a problem for JavaScript so I could learn enough JS to fetch the JSON and write a map and then I can do then other things in ash by reading the file. I learn best by example so is there a JS snippet that deals with JSON and maps I could look at?

Thanks.

 

[MCroft]

I started writing something up and googled for a piece I needed and found this fine guide:

How to Convert JSON to/From a Map in JavaScript - Coding Beauty

Learn how to quickly convert a JSON string to a Map in JavaScript.

codingbeautydev.com

here's the meat of it:

const json =  '{"user1":"John","user2":"Kate","user3":"Peter"}';
const map = new Map(Object.entries(JSON.parse(json)));
// Map(3) { 'user1' => 'John', 'user2' => 'Kate', 'user3' => 'Peter' }
console.log(map);

 

[MCroft]

It would be possible to write a JSON_to_map function for ASH, but it's not a perfect fit.

1: JSON doesn't require each object to have the same format, so parsing needs to be careful.
2: JSON doesn't enforce any structure on the keys, and map will overwrite a key-value pair if the key is a match.

So there would have to be some guardrails, but it might be simple enough for your use case.

 

[heeheehee]

map will overwrite a key-value pair if the key is a match.

To be fair, JSON.parse will clobber any duplicate keys before you get to that point.

You shouldn't need to go through a Map -- JS Objects operate similarly to Maps (that's what we did 10 years ago, before ES2015 brought Map to the language).

As MCroft mentions, I'd suggest doing some preprocessing of the underlying data first to make sure your outputs are uniformly structured, since otherwise you'll probably run into some sharp edges of JS <> Java <> ASH conversions.. Something like...

let kolmafia = require("kolmafia");

let results = {};

for (let playerId of [1, 354981]) {
  let url = `https://museum.loathers.net/player/${playerId}?_data=routes%2Fplayer.%24id`;
  let obj = JSON.parse(kolmafia.visitUrl(url, false, true));
  let collection = obj.player.collection.map(x => { return {item: x.item.id, quantity: x.quantity, rank: x.rank}; });
  results[playerId] = collection;
}

kolmafia.mapToFile(results, "player-collection-ranks.txt");

 

[fronobulax]

Thank you very much. I run and get an error about an empty JSON string.

I tried via ash using

string url = "https://museum.loathers.net/player/215957?_data=routes/player.$id";
buffer out = visit_url(url, false, true);

and the return is of length zero.

The URL works in a browser.

Brute force being a tool of choice I tried all four combinations of the booleans and visit_url returns length zero in all four cases.

My next debugging step is a breakpoint in an IDE but if anyone has some other suggestions I'll try them too.

 

[heeheehee]

The above only works for me if I'm logged in, for whatever* reason. Otherwise it works just fine.

* The actual reason is at the start of GenericRequest.run().

  public void run() {
    if (GenericRequest.sessionId == null

 

[fronobulax]

That was it. Thank you.

Tangentially GenericRequest.sessionId == null and visit_url being a generic request means some scripts will need to be logged in to run. Not sure whether that is a bug or a feature but...

 

[fronobulax]

I've got something working so thank you. Not ready for prime time yet but it will eventually replace a portion of DCQuest. Thank you.

 

[fronobulax]

See https://kolmafia.us/threads/dcquest.3640/#post-25745

 

[fronobulax]

Bump.

let kolmafia = require("kolmafia");
let results = {};
let playerId = kolmafia.myId();
let playerName = kolmafia.myName();
let url = `https://museum.loathers.net/player/${playerId}?_data=routes%2Fplayer.%24id`;
let obj = JSON.parse(kolmafia.visitUrl(url, false, true));
let collection = obj.player.collection.map(x => { return {item: x.item.id, quantity: x.quantity, rank: x.rank}; });
results[playerId] = collection;
kolmafia.mapToFile(results, playerName + "_DCRanksRaw.txt");

When running I get an error "JavaScript error: Cannot call method "map" of undefined" on line 7.

JS not being one of my fluent languages I'm not really sure how to go about addressing the error. I can visit the URL manually in a web browser and what I get back looks like well formed JSON. This started happening today and, like most people, I think I didn't change anything except build and run a new version of KoLmafia. Most of the mafia related JS on my system isn't helping me guess at how to break things apart and debug.

Ideas? Suggestions? Help?

For extra credit this could be slightly more robust, in context, if it always wrote a file, specifically an empty one if if it could not fetch or parse the data.

Thanks.

 

[xKiv]

the json I am getting has collections (with an s at the end) where the code expects collection (without the s)
(IOW it's not a mafia change, it's a change on the remote side)

 

[fronobulax]

the json I am getting has collections (with an s at the end) where the code expects collection (without the s)
(IOW it's not a mafia change, it's a change on the remote side)

Thank you. The good news is I changed collection to collections and the code ran as expected. Thank you for confirming that it was not a change on my side

But running the script generates "Cannot invoke "net.sourceforge.kolmafia.textui.parsetree.Value.getType()" because the return value of "net.sourceforge.kolmafia.textui.javascript.ValueConverter.fromJava(Object)" is null" and the associated stack trace. I know there is code that maps Ash types into JavaScript types and visa versa and I wonder if that code needs to know about "collections"?

I'm tagging @gausie because https://museum.loathers.net/ is theirs and I may be the only user of the API

 

[fronobulax]

Problem fixed. The JSON from the server had changed and it took me some time to adjust to that.