Kol Data Sources

How do places like the koldb or jickenwings get their data?
I can't imagine them requesting every display case or profile page individually... and even if they did do it that way, how do they know which playerids are valid... or know the maximum player id if they are just hitting every playerid and discarding unused player id's?

Is there any sort of public read-only access to any kol databases?

KoLdb checks the Recent Ascensions boards regularly (either every 30 minutes, or every hour on the hour), and then it checks the ascension history of everyone on one of those boards.

I'm pretty sure jickenwings has non-public access to stuff, but I don't really know.

I believe there are publicly accessible data feeds which are updated at rollover each day. I have no idea where they are though, so I guess you could say they are secured through obscurity.

The jickenwings database is updated about half an hour after rollover and the data is a static feed from the day prior.

Edit:

jickenwings FAQ said:

How often is the database updated?

All display cases are refreshed on a nightly basis, right after rollover.

Click to expand...

jickenwings newsfeed said:

New Faster Update Script Running
Posted 2006-05-16 06:00:35

Thanks to the generous help of Jick and Xenophobe, I've now got a daily data feed of collection data from the KOL servers.

Click to expand...

So yeah, the kolDB is pretty much bruteforce updated as lostcalpolydude said (but I think it's every 2 hours). The jickenwings DB used to be bruteforce updated, but since 2006 a feed is sent to their server and gets parsed. The jickenwings feed isn't publicly accessible, I think it's just an FTP transfer from KoL to jickenwings.

There are other feeds you can obscurely query though.