price.ash: learn how much an item costs, courtesy items.ofloathing.org

[dj_d]

After some discussion on the subject (http://kolmafia.us/index.php/topic,1922.0.html), here's a quick script to grab prices from items.ofloathing.org and the kolwiki. The cool parts are from Zarqon; the embarrassingly incompetent parts are from me.

In any case, this is a small piece of a bigger project but I thought I'd share. Feedback welcome!

<<updated with the latest - see thread below>>

[zarqon]

Nice! Didn't know that site existed. I may modify this and use it with my clan StashBot, perhaps as a supplement to my method.

To include a double quote in a string, just escape it with a backslash.

"Hi there!" = "\"Hi there!\""

I'm still a fan of mafia allowing us to access the mall minimum once per item per day. (Saying this again in case any devs hadn't seen the other thread and are now inspired... ha.)

Something like int value( item ) which returns the minimum mall price, once per day per item. This would still prevent mallbots, but would also eliminate the need for complex workaround scripts that hit remote servers or place meat in and out of the closet. These second-hand prices will never be as reliable as actually accessing the mall itself.

[dj_d]

Cool! If you don't get to it first, I'm thinking about a method that grabs both, and compares them to the last recorded price produced by either source (in case they've gone down), plus the last price that the player paid for the item (if available), and makes the best guess based on all those data points. Basically, keep a local cache of expected prices and update it whenever new data is available.

And I agree - Mafia should have some sort of price estimation feature. Once per day lookup, time-delayed results, a central clearinghouse of prices updated periodically (kind of what we're dong here) - something.

[zarqon]

Okay, I took your idea and ran with it. Originally I was just going to clean up the string parsing, but I ended up adding in your "cache the page locally, with a 1-day expiration" idea. In the end, I think none of the code is the same, but it's basically what you were doing.

This scriptlet features a pretty nifty function which I wrote to be transportable:

string visit_daily_url(string url, string filename)

Which works just like visit_url() except it will first check to see if this data is stored in filename. If the cached data is less than 1 day old, it returns the cached data, otherwise it returns the visited url.

This pricing solution works nicely, except like I said before, the prices are still not as accurate as internal pricing would be. Some items are grossly mispriced, for example, Mr. A's (and just about every other Mr. xxxx item) are valued at 100 meat! I can only imagine the complaints I would get if someone donated a Mr. A to my StashBot and they were only given 100 credit for it!

Incidentally, the wiki method (value.ash) values this item at 4794988. Perhaps a script that uses both of them, then goes with whichever is higher would be the best solution at present. (Accessing a total of three other servers to find item values! Whee!)

[dj_d]

Thanks for
1) the improvements
2) the suggestions
3) reminding me of how much I've forgotten since CS101. Good thing I don't do this for a living. That quote-mark code was just embarrassing. :)

The URL caching is brilliant. I think the stashbot has issues under load, so this is a great compromise. You can get all the values from items.ofloathing in one whack. Then, I added (your) code to check with stashbot if it finds a value of 100 or 0. The first post has that version now. Further improvements welcome.

This is all going to be very handy for a large chunk of code I've got sitting on my machine that currently spends like a drunken sailor. More on that as soon as I polish it enough for sharing.

[zarqon]

Warning: sometimes it's better to go with 100 rather than a higher value. For example, casino passes are always overvalued in the wiki because of them being sold for high prices in /games or as MMG debt. Their actual value should be 100 meat.

Also, let me clarify: the pricing workaround I shared before references the wiki. It is not StashBot, just a pricing method. It accesses both my own server and the wiki, and I wrote the method as part of my StashBot script.

[dj_d]

Gotcha.

I did a quick survey of mall prices and of return values for both servers, and found that there were MANY more things mispriced at 100 than properly priced at 100, so I think the "not 100" heuristic is reasonable - but suggestions/changes (even just to correct the comments re: your stashbot point) are welcome. Otherwise, I'll try to get to it when I give it another overhaul.

[Bale]

Gotcha.

I did a quick survey of mall prices and of return values for both servers, and found that there were MANY more things mispriced at 100 than properly priced at 100, so I think the "not 100" heuristic is reasonable - but suggestions/changes (even just to correct the comments re: your stashbot point) are welcome. Otherwise, I'll try to get to it when I give it another overhaul.


Here's a simple heuristic to deal with figuring if something is overpriced in the wiki due to reverse raffles or the like. If an item (eg, casino passes) can be bought in an NPC store and the wiki says that it sells for more than twice autosell, then the wiki price is the error. Make a list of items from NPC stores and that'll deal with 80% of the problems.

[dj_d]

Good point Bale.

Zarqon, I realized that our uses for this may be very different - it doesn't particularly matter for my use if the price error is high or low, whereas your stashbot might be exploitable if it errs high. Feel free, of course, to modify to taste.

[dj_d]

Another update, again inspired by Z. This writes out any lookups you do to a file, pricelist.txt, and if you look up the same item in the same day, grabs it from the text file. You might need to create a blank pricelist.txt with notepad in order for it to work, I'm not sure.

[zarqon]

@Bale: your heuristic isn't really very simple! It's a simple concept, but not simple to code. There is no mechanism in mafia for accessing the autosell value of an item.

@dj_d: Yes indeed, any pricing errors by StashBot (high or low) are exploitable. No one in my clan is the exploiting type, but I would rather not present anyone the opportunity. Also, cool idea caching the prices. Not sure if it helps though since the page data is already cached...?

I've incorporated a hybrid pricing system combining every pricing method we've come up with so far into StashBot (including support for Bale's idea of a list of NPC items) and he seems to be pricing things much more accurately now. I've attached a standalone version of the mashup I'm using. The great news is that it seldom needs to fall back on the "buy the item" method of pricing (that's the last resort). The bad news is it still gives grossly incorrect values for Mr. A's.

[dj_d]

It's fun collaborating with you, Z. Question - I'm going to integrate more of your stuff in to the version at the top of the post. Can you use notify with two arguments ("notify zarqon dj_d") and if so would you like to be notified from that version? You've obviously put a ton of work in to this and I don't want to hog credit. :)

I'm using this pricing script in a place where it may be called for on hundreds of items, multiple times per day. Your HTML caching fixes the problem for the itemsofloathing lookup, but not for wiki lookups (which only returns one result at a time, as I understand). While I could cache each response from your page as a separate file using your function, that'd get ugly in the filesystem fast.

The second reason is that I'm moving to a more complex system where different lookup types have different lifetimes: amounts from the wiki/itemsofloathing, manually entered amounts, and amounts found through actual purchases in the mall. So if you bought a Mr A from the mall two days ago for 4.3mm, it wouldn't pay attention to itemsofloathing telling you it was worth 3000 meat today. Or you could manually enter prices for key items that were being misvalued.

The class of problem I'm working on, BTW, is shopping on a budget. To do that, you have to get a LOT of prices.

[dj_d]

Just noticed from items.ofloathing:
"Nov 21 Wow! Prices haven't been recording since Nov 1, sorry guys! They should be fixed now."

[Bale]

@Bale: your heuristic isn't really very simple! It's a simple concept, but not simple to code. There is no mechanism in mafia for accessing the autosell value of an item.


As I said, make a list. Just include the prices in the list.

[asturia]

Is there a way to input your complete itemlist? and write the results to a file?
I want to calculate how much my things are worth.

[zarqon]

@dj_d: Yes, what we've come up with here is far better than what I had before with the wiki solution. And don't worry about notification credit. I don't need to be notified for this particular script, since it is just a small part of one of my larger scripts. I don't think notify works for multiple people anyway. (ps - you might consider using my Registry script to keep track of who uses your scripts... [/plug])

@Bale: Making and especially maintaining an autosell-price list for all tradeable items would be much more of a chore than getting item prices manually, negating the whole point of writing the script. In my case, I need to be able to accurately appraise any tradeable item on the fly, including new items as soon as mafia supports them. It would be better to open mafia's itemlist and read the autosell from there -- but that would make the script huge and clunky and slow.

@asturia: I know for a fact there is a Greasemonkey script floating around that will do this. A little searching would probably turn it up. If you can't find it, I know someone who uses it and I could ask her where she got it. If you don't want to use Greasemonkey and want a mafia solution, you could try adding this function to the value.ash script, then calling it with get_inventory() as the parameter.


int stack_value(int[item] stack) {
int result;
foreach i in stack
result = result + (stack[i]*get_value(i));
return result;
}


Note that this will only return the sum total of all your tradeable items in inventory (not storage or closet).

A plea to the devs:

Please consider introducing these functions to ASH:

int autosell_price( item ) - returns the autosell price of an item. Current methods for accessing this (creating an independent map of data, or loading mafia's entire item database only for one info point) are hugely inefficient.
int mall_price( item ) - returns the current lowest price for an item in the mall once per item per day, to prevent mallbots. This would allow item pricing without buying the item, but it would still be just as difficult as now to manipulate mall prices.


If there is some reason this cannot or should not be done, at least write us a note explaining why. Then I promise I won't whine about it again.

[zecious babaloni]

I just wanted to make some comments on this script.
I am Zecious Babaloni, the creator of Items of Loathing.
First off, I'd like to thank dj_d for creating this script, I'm glad to hear that people enjoy and use my site.
I have fixed some bugs the site was having, causing some items to be mispriced, and items with special characters (such as quotes) in their name to not be listed. These bugs should all be fixed now. Please kmail me in-game if they are not.
Also, if you plan to make a script similar to dj's that will be scraping my database, I would prefer that you gather the info from the following automatically updated files:
http://items.ofloathing.org/today.txt - This file lists the items and prices from today only
http://items.ofloathing.org/export.txt - This file shows the past fourteen days worth of data.

I wiped all data back to October 26th today, due to inaccuracies caused by bugs.
export.txt does not show the past 14 records of an item, it shows the past 14 days, meaning that if/when I do a wipe like this, it will not have the full 14 days worth of data in it. Sorry for the convenience.

Thank you, everybody for your support and interest. Please kmail me in-game if there is anything I can do to help you scrape data for your scripts.

[asturia]

Zargon, thanks for the help.
The greasemonkey script you're talking about is called UpUp.us Price Gun.
The problem is that it doesn't include the latest new items.

When I add your suggestion to the script, it gives me this error when I run it:

Undefined reference to function 'get_value' (value.ash, line 85)

[dj_d]

Hm... attempts to fix the script aren't working. visit_url returns nothing when I pass it the today.txt URL. I'm stumped. Zarqon?

[dj_d]

New version is ugly beyond belief, but it works. It's designed to scale a little better when multiday returns become available, though, as it can read them all in to a big map.

[jasonharper]

A plea to the devs:

Please consider introducing these functions to ASH:

int autosell_price( item ) - returns the autosell price of an item. Current methods for accessing this (creating an independent map of data, or loading mafia's entire item database only for one info point) are hugely inefficient.
int mall_price( item ) - returns the current lowest price for an item in the mall once per item per day, to prevent mallbots. This would allow item pricing without buying the item, but it would still be just as difficult as now to manipulate mall prices.


I have added the autosell_price() function in r6524 - it appears non-abusable, and genuinely useful.

I don't feel that I have the authority to overturn existing policy on making mall prices available to scripts.

[Bale]

I don't feel that I have the authority to overturn existing policy on making mall prices available to scripts.


Of course not, but you could give them a tiny nudge by simply answering this question: If they agreed to once per day pricing, would you do all the work of writing the function so that they only have to make the decision?

[zarqon]

Zecious Babaloni and jasonharper save the day!

This would be a perfect example of a time when it would be awesome if file_to_map() worked with URLs. (Kind of like a public update command.) Centralized data files would be super easy to access in scripts... it would even enable automatic updates for script data files... or perhaps I'm just dreaming. :)

I had a bit of discussion with Zecious and he is working on a script to generate another data file, which will probably be called average.txt, which will contain the average price for each item over the last 7 days. Which will unfortunately negate a lot of your work with parsing export.txt into a big map, dj_d.

In the meantime, I changed just a few lines so now it gets data from today.txt. It still caches this data, so it will only hit the ofloathing server once per day max. I also changed the order of things so that if the ofloathing value checks out, it won't bother with the wiki, which makes it the fastest, most server-friendly version of this script to date. Enjoy this until Zecious gets average.txt up, at which point only a few tiny edits will need to be made to move over.

@asturia: Sorry, I had already changed the name of price() to get_value() in my script before I wrote about your solution. Try it using this one and it should work. ps - it's Zarqon with a Q. :)

@jason: Thanks a bunch! I know I am one of many who appreciates your instant attention to user requests. This will be a genuinely useful feature. Once this appears in the daily builds, I'll be able to save even more server hits with some of my scripts.

I also understand your position on mall_price(). It has been a longstanding policy, as well as a longstanding request by those who didn't yet know the policy. The policy exists for good reason: just because something is created in a certain spirit, it does not follow that everyone will use it in that same spirit. Which is why proper safeguards must be in place. Previously, to my knowledge no one has requested mall price access with a suitable safeguard in mind. But I do think that a per-item time limit on price requests (per day or per session), using the cached value after that, is a suitable safeguard against the kind of abuse that Hola and Veracity want to prevent.

I can only think of two ways to bypass this safeguard, here made public along with ideas on prevention:

1) The obvious way would be to reset mafia's external price/date cache (assuming this data is stored in a file), the easiest method of which would be deleting the file. Of course, doing that would require something outside mafia, which is no different from the way things are now. Still, if a user could simply delete a file, run a script, delete the file, run a script, etc. (or make an external script to do that) that's too easy to abuse.
Solutions:
Store the price/date values as part of the application file. Not sure if that is easily doable in Java. I also don't think it's the best solution.
Instead of limiting mall requests to one per item per day, limit it to one per item per session. Then the pricing cache could be kept internally. Getting more than one current price a day would require restarting mafia. I think this is an excellent solution.
If this external data were added to GLOBAL_prefs.txt or some other file that people need, normal users would be loathe to delete it, and editing it would take more time than just visiting the mall manually. Editing the data could also be made very difficult by encrypting it (probably some kind of hash of price and date together).
If you are storing the data externally, I think this is the best solution. The price/date data is stored in a separate file (encrypted to prevent editing). If mafia is run without that data file, the file is created but an internal "pricedisallowed" flag is set for that session and mall_price() will abort with an error explaining the need to restart. If the file is deleted while mafia is running, the flag will also be set. Only after restarting mafia could you then use the function, and if you deleted the file while mafia was running, you'd have to restart twice. This would only affect normal users once, if at all -- they would have to call mall_price() their first time running mafia to encounter the error.

2) The other method involves running many separate instances of mafia, since each one gets one price per day. For this to work would require many dummy KoL accounts with mall access, and a complex script running on all machines that communicates pricing info to one master account that serves as the mallbot. Hugely impractical, expecially in light of the fact that if you have enough meat you can already make a mallbot of sorts using the closet trick.
Solutions: none needed, doing this would be ridiculous. It's about as easy to learn Java and write your own mallbot.


Intended behavior: mall_price() would check to see if the value is cached (or in memory) and has not expired. If the value is found, it returns that value, otherwise it returns the current minimum mall price (storing it in the cache/memory). This stored value would be used for every future call of mall_search() until 1) the cached value expired (one day), or 2) the value was no longer in memory (next session). This would have the effect of allowing only one server hit per item per day (or session), and would not enable manipulation of mall prices.

Furthermore, the script we have made here already makes once-daily item prices available. Adding this function to ASH would only make those prices more accurate, because the pricing would happen at the exact time of calling the function (the first time, which is all any script should really need).

What do you think, Hola, Ver, jason? Do I make a convincing argument?

[jasonharper]

Intended behavior: mall_price() would check to see if the value is cached (or in memory) and has not expired. If the value is found, it returns that value, otherwise it returns the current minimum mall price (storing it in the cache/memory).
I'm not entirely sure you're asking for the right thing here - the minimum price isn't necessarily meaningful if it's a 1/day limit, or there's simply only one available at that price. Frankly, an anomalously low price seems more useful to a mallbot than a legitimate user.

[zarqon]

We are looking to know what we could expect to pay for an item without needing to pay for it. If the current mall price is anomalously low, that is still the price we would pay for it if we acted quickly. Further, sometimes prices which may appear to a script to be anomalies may really be trends, as is often the case with relatively new items.

If you can think of some other method whereby users looking to budget or appraise items would be better served (perhaps some kind of average of the top X stores, the mode of the top X stores, etc) I'm all for it. Such a method would be even harder to abuse.

An anomalously low daily price would not be terrifically useful to a mallbot. What would it do? Buy some items? How many? It can't know. It could try buying a handful and checking again but it would get the same cached value every time so it wouldn't know if it had succeeded in buying out the lower-priced items until the following day.

[dj_d]

For the record, I love to have my work invalidated by people who redo it better.

@jasonharper: you're absolutely right. I think I suggested this in the other thread as a solution. Show the second-lowest price. Fine for budgetary purposes, nearly useless for mallbots.

@Zarqon: that big ugly hack is basically a way to make file_to_map work with URLs. It caches the context as a text file, then runs file_to_map on it.
Also, I think there's some value to parsing export.txt anyway - some people may want a high estimate of the price, some may want a low estimate, some may want the average. For example, if you're budgeting, you might prefer over-estimating to under-. The current mess seems to work pretty well, though, so I may get back to the original project (to be posted shortly).

[jasonharper]

If I was only allowed to know one number about a mall item, I think I'd want the price of the 5th or so available item - in other words, it would only be the lowest price if at least 5 were available at that price. That should skip over most price anomalies.

A mallbot could easily buy out a known underpriced item, using techniques that have been previously discussed here. Actually, there's nothing stopping one from trying that on every item that normally sells for more than min price - perhaps that loophole needs to be closed before considering anything that would make such behavior easier.

[Bale]

If I was only allowed to know one number about a mall item, I think I'd want the price of the 5th or so available item - in other words, it would only be the lowest price if at least 5 were available at that price. That should skip over most price anomalies.

And I'd want to know the price of the first that didn't have a limit per day. It would be pretty hard to satisfy everyone... I guess that's why this idea never goes anywhere. Still, if you were willing to add a function for the 5th item, everyone would be glad to at least get some sort of mallchecking ability for so many reasons. We understand that mallchecking will be hobbled and if this is the flavor of hobbling you prefer, I'd say thank you for offering it.

[zarqon]

@jason: Awesome idea. Knowing the nth item price would be great. N should probably scale depending on the ratio of price increase to n. For example, I'd want to know the 1st or 2nd cheapest Tome of Snowcone Summoning, not the 5th, as the price curve is usually steep on really pricey items -- and for really rare items, the 5th might be priced at 999,999,999. But for something cheap, knowing the 30th item price might be the best. Something like

n = (price of 1st / price of 10th)*30

I arbitrarily chose 30, but you could decide how "safe" to be with a little testing.

Or, something that looks at the difference between the 1st and 20th and what percentage of the 1st that is, if it's over 100% use a lower value for n, if it's under 100% use a higher value.

I might be getting excited about your idea and making up worthless ones of my own that I will fully think through later and reject, but I thought I would share it while the excitement was fresh. :)

[jasonharper]

This would be a perfect example of a time when it would be awesome if file_to_map() worked with URLs.
Apparently, it does already... I haven't actually tried this, but there is a special case in the code if the supplied filename starts with "http".

[zarqon]

there is a special case in the code if the supplied filename starts with "http".


That's odd because I actually tried it before writing that and didn't get any results, but testing it now shows that it works. Perhaps something else in the script wasn't working yet at that time.

I've rewritten the script to use file_to_map() with the remote today.txt. Unfortunately that meant adding a setting to your preferences file (filename => timestamp), since the timestamp was no longer stored in the data. Enjoy.

[zarqon]

A mallbot could easily buy out a known underpriced item, using techniques that have been previously discussed here. Actually, there's nothing stopping one from trying that on every item that normally sells for more than min price - perhaps that loophole needs to be closed before considering anything that would make such behavior easier.


How would it know it was underpriced? It would need something to compare against. A script could not arrive at a "normally sells for" number if all it had was a daily minimum from a random time. We are asking for one or the other, not both. Min price, or "normally sells for" price. For my purposes, the second one would be better, and even harder to exploit by bots.

[Bale]

I'd like to thank zarqon and dj_d for having made this script. I've used it many, many times and many scripts of mine would have been impossible without it. Now I'd like to bid it a fond adeiu since jasonharper has just implemented mall_price() in revision 6989.


item [12] super_drink;
super_drink [ 0 ] = $item[ prussian cathouse ];
super_drink [ 1 ] = $item[ Neuromancer ];
super_drink [ 2 ] = $item[ vodka stratocaster ];
super_drink [ 3 ] = $item[ Mon Tiki];
super_drink [ 4 ] = $item[ teqiwila slammer ];
super_drink [ 5 ] = $item[ Divine ];
super_drink [ 6 ] = $item[ Gordon Bennett ];
super_drink [ 7 ] = $item[ gimlet ];
super_drink [ 8 ] = $item[ yellow brick road ];
super_drink [ 9 ] = $item[ mandarina colada ];
super_drink [ 10 ] = $item[ tangarita ];
super_drink [ 11 ] = $item[ Mae West ];

int [item] price;
for i from 0 upto 11
price [ super_drink [i] ] = mall_price(super_drink [i] );
sort super_drink by price[value];

for i from 0 upto 11
print( price [super_drink [i]] + " "+ super_drink [i]);


Tee-hee! I'm now in price sorting heaven!

Farewell value.ash. Thanks to mafia's ongoing development I won't miss you at all, but I will always feel a little nostalgic for the good times we had.

Edit: Or if I want even more brevity, this can be reduced to a single line with cli_execute():


cli_execute ("cheapest prussian cathouse, Neuromancer, vodka stratocaster, Mon Tiki, teqiwila slammer, Divine, Gordon Bennett, gimlet, yellow brick road, mandarina colada, tangarita, Mae West");

[zarqon]

Yes indeed, the new mall_price() is just what this thread ordered. :)

I personally am keeping value.ash around. In certain info-gathering-intensive situations, such as for my clan StashBot, it is far more KoL-server-friendly to continue using this script, which only generates one server hit per day (and not on KoL servers at that). I have found it surprisingly accurate, although a few items are quite underpriced.

[dj_d]

WHOO F*ING HOO!

Thanks JH! I will be updating eatdrink.ash and others asap.

[mredge73]

Hey Guys
Do you guys know what it means when it says "do cmds with "it" replaced with best"?
Can any of you guys explain this function for me? Thanks

This is what I am taking about:

cheapest[?] [+]item [,[-]item]... [; cmds] - compare prices, do cmds with "it" replaced with best.
expensive[?] [+]item [,[-]item]... [; cmds] - compare prices, do cmds with "it" replaced with best.

[bigevildan]

By itself, the "cheapest" command will search the prices of the items you specify and print them out in order from cheapest to most expensive. If you follow that with one or more CLI commands, it will instead run those commands, replacing the word "it" with the name of the cheapest item.

For example, "cheapest hi mein; eat 3 it" will search the mall to find out which hi mein is cheapest, and then eat 3 of them.

[Bale]

Or my personal favorite:

cheapest +tangarita; buy 4 it; drink 4 it;
The + means it will search the zap group!

[mredge73]

Hey Zarqon
Did you kill value.ash, it no longer seems to work retrieving the wiki value?

[zarqon]

The wiki method was the worst of the three methods for getting pricing information (least consistent, most server-intensive), so I'm no longer using it. I may have deleted that functionality from my server, but I can't remember at the moment (I'm at work). You should be fine with just the ofloathing price and mall_price().

[ki77bot]

Or my personal favorite:

cheapest +tangarita; buy 4 it; drink 4 it;
The + means it will search the zap group!

Is there a way of doing that in ash, and I don't mean via cli_execute. I'm looking for a way I can use/define the 'variable' it in ash.

Cheers,
ki77bot.

[jasonharper]

Is there a way of doing that in ash, and I don't mean via cli_execute. I'm looking for a way I can use/define the 'variable' it in ash.
Unfortunately, no; there's no current ASH access to the zap group info, and the zapgroups.txt file doesn't have the structure required to be read into a map so you could parse it yourself.

If for some reason you can't look up the members of the zap group ahead of time and enter them into your script explicitly, the best you could do at the moment would be:
cli_execute("cheapest +tangarita; set temp=it");
and then look at get_property("temp").

If I've misunderstood your question, and what you were wanting to know is how to duplicate "cheapest" itself in ASH, rather than its +zapgroup feature in specific, what you'd need to do is build an array with all the items of interest, sort it by mall_price(value), then look at element 0 of the array for the cheapest item.

[ki77bot]

Yes, the first part of your post answers perfectly what I was looking for. (The second part describes what I've been doing so far. But as I try to look for solutions with less code, the cheapest function is greatly welcome :D).

Thanks a lot.

Cheers,
ki77bot.