Historical price data

[jasonharper]

Ok, big change in r7531 - the most recently seen Mall price for each item is now remembered between sessions, so that you can always get an approximate idea of the value of any tradeable item without needing a server hit. The ASH functions to use are:

int historical_price(item) - Returns price, or zero if no data is available. Will never hit the server.
float historical_age(item) - Returns age of price data, in days.

* The price has the same meaning as with mall_price(): it was 5th available item as of the time the search was performed, or the last one if there weren't that many in the Mall.
* You'll get zero for any newly-added item prior to the first search for it, or for any of the (currently 16) tradeable items that aren't available at any price - hand turkey outlines, for example.
* The returned price is variable for NPC-buyable items, depending on whether the search that last updated it was done by a player who had access to the store that carries it or not.

The price data is stored as mallprices.txt in mafia's data folder, and is intended to be shared: CLI command update prices URL will retrieve a copy of the file that's been made available somewhere, and merge newer prices into your copy. To start things off, I'm now automatically uploading mine after every session, and you can retrieve it via:
update prices http://nixietube.info/mallprices.txt
The idea is that if we can set up a network of people updating from each other, then everybody will be able to benefit from reasonably current price data, without anybody having to perform an excessive number of Mall searches.

If you choose to make your data public: there shouldn't be any privacy issues, as there is no personally identifiable data in mallprices.txt. It doesn't even reveal anything about your Mall usage habits, as there is no way to tell whether a given updated price came from a search you performed, or was received in an update from someone else.

If you choose to receive updates from others: please understand that this will expose you to some risk of undesired behavior due to falsified data. There is some consistency checking of the received file, but it can't (for example) tell that the price of twinkly wads isn't a million meat - if it could infallibly tell if a price was correct, there would be no need for the price data in the first place!

 

[Bale]

This is frankly brilliant.

I find one of the annoying things about mallmode recovery is that when it runs for the first time in a day I have to deal with a very long list of price checks. That troubles me because if 100 people use my script, then KoL has to deal with all those server hits from all 100 people. This can cut down on server hits a lot as well as do away with that tedious first-run delay.

I have a question about this. Does it update historical_price() when I check prices in the mall, purchase something or use mall_price(). Which of those update the file?

 

[jasonharper]

historical_price() is updated at exactly the same times that mall_price() is, so basically:

* any script call to mall_price(), that actually results in a search.
* cheapest/expensive commands, that actually result in a search.
* price checks for deciding buy vs. make.
* Modifier Maximizer/Basement helper when price checking is enabled.
* any successful purchase performed by mafia, whether due to a buy command or implicit acquisition.

It is NOT updated by the CLI searchmall command, since that is unrestricted.

It is NOT updated by relay browser searches or purchases. The game's search form has approximately an infinite number of search options, which would make it difficult to determine whether a given search produced usable results - you might have limited the stores to search, used a price limit, or sorted by something other than price, any of which would make "the price of the 5th available item from the list" not a representative price.

 

[dj_d]

So let's say twinkly wads show up as 500 meat, but due to some special content, the price spikes to 50,000 meat. eatdrink deals with this by closeting your meat before trying to purchase, then uncloseting, but this take multiple server hits and is pretty clunky (e.g. if you abort at the wrong time all your meat is still in the closet).

Could we get a BuyAtPrice(int price) that buys if the item is available at or below the ostensible mall price, and doesn't if it isn't? Then we can use BuyAtPrice(historical_price(it)), and if it fails, update the price with real data and recalculate to see if we still want to buy the item.

 

[Bale]

Good news dj_d. That function already exists.

int buy( int quantity, item this_item, int price_limit )

It will only purchase this_item if the price is equal to or less than price_limit. It will return the number of items that were actually purchased. Unfortunately it will only update prices if it actually purchases something.

 

[kain]

This is excellent!

 

[dj_d]

Um... whoops. I was using that already.

 

[Bale]

zarqon, how do you feel about creating a section on your mafia tools website where people can upload their mallprices.txt, along with a little script to automate that?

 

[mredge73]

Can this be used to speed up Fern's basement?

 

[Bale]

CLI command update prices URL will retrieve a copy of the file that's been made available somewhere, and merge newer prices into your copy. To start things off, I'm now automatically uploading mine after every session, and you can retrieve it via:
update prices http://nixietube.info/mallprices.txt
The idea is that if we can set up a network of people updating from each other, then everybody will be able to benefit from reasonably current price data, without anybody having to perform an excessive number of Mall searches.

To facilitate sharing, could you add a CLI command to allow someone to upload their mallprices.txt to a given location?

priceupload http://www.somesite.com/prices/

This would be convenient for someone who wanted to create a script that shares a pricelist. I was thinking that I could set my recoveryScript to update mallprice.txt and then upload the updated mallprice.txt to that same location once per day. That way people who use my script will benefit from automatically sharing recent price information with each other.

 

[jasonharper]

There is no generic way of doing that; a server that allows uploads might do so via a HTML form, WebDAV, FTP, etc., all of which are going to require more details than just the URL to upload to.

 

[Bale]

Well I'd like to be able to update it with map_to_file() like we can send other information to a website. (Y'know, how zarqon has it set up.) Unfortunately the first line in mallprices.txt messes with proper mappability.

Are you sure you can't set it up to use the same protocols? That might not be great for everyone, but at least it will be good for someone. Without some method, it helps noone.

If you'd do this, I think I could talk zarqon into setting up a shared resource for mallprice sharing.

 

[dj_d]

If that was the case, I'd happily update some of my more popular scripts to do that, too. We could be one big happy distributed pricebot.

 

[zarqon]

You could definitely talk me into it, Bale. You have quite a bit of accumulated persuasion power, since your OCD Inventory script keeps sending me bat parts.

 

[Bale]

You could definitely talk me into it, Bale. You have quite a bit of accumulated persuasion power, since your OCD Inventory script keeps sending me bat parts.

Well, you collect such cheap items that I practically don't notice the loss of those bat bits. (Do you get them from others who use my script, or just me?) And I really appreciate all you do for scripting, so I'm glad to send them.

So, what do you think of setting up a depository for anyone to send mallprices.txt to? What protocols would be best for your use?

 

[jasonharper]

Well, I could probably implement an upload feature specifically for zarqon's Map Manager, but it wouldn't work for any other server that used different field names in its upload form. I'm pretty sure that update prices could already read a file from the Map Manager.

However, it's not real clear how this would work - would everybody be uploading to the same filename? If so, updates would be lost - even if uploads to the server always immediately followed an update from that server, there would still be a window in which somebody else's update could be overwritten. And if everybody was uploading to a personalized filename, that would quickly overwhelm the site's user interface...

Basically, if this is going to be done via a centralized site (rather than various people uploading to their own personal webspace), the site would ideally be capable of merging the received files into a single downloadable update. That wouldn't be too difficult: basically, for lines with the same 1st field, keep the one with the highest 2nd field.

 

[zarqon]

Jason: could map_to_file() be made to work with URLs? Submitting the map as a file, the same way that an HTML file upload form would? The burden would be on the server-side script to comply with field names and however you wanted to give feedback for the return value. i.e. the server could output just "success" if the file was written. Just a thought.

Hahahaha! I started writing a long post about problems with the time-sensitivity of this data, before I realized that the first field in the map file is the time! Haha.

In that case, the problem is the first line of the file. It's not a problem for uploading (I could remove it server-side, or if the data is sent from within mafia it would not include that line). But for downloading and updating your map, the new map would not include it and it would be deleted. Is it necessary?

Assuming it's OK to remove that first line, it would be easy to upload the file and parse it for newer prices and update as needed server-side. I could also have the script write an updated map, accessible by file_to_map(), making it easy for scripts to interact with.

Problem: if this map is sent to my server every time a script is run (I foresee daily use by Universal Recovery and bits of Ascend, which are quite popular), my server (actually, my server's sysadmin) may be a little annoyed. Most of my server-interactive scripts are just data fetchers, or some kind of very brief interaction. This script would involve a little number-crunching to determine which prices were newer. So I would EITHER set it up like the Map Manager (you have to manually visit the web site and submit the file), OR ask that scripts only send updated maps once daily (in the logout script would be a good place for that, but I would still request a _property limitation).

Unless Jason has a better idea for a more mafia-integrated solution, the best way may just be adding upload_shared_prices() and download_shared_prices() to ZLib. Hmmm.

Fetching the maps can be done as often as you please, since I would write the script to have a file ready to send out.

 

[Bale]

Well it would be ideal if zarqon could merge mallprices.txt on his site. (One address for people to send their mallprices to and a second to download from.) However I was really thinking of a much simpler and slightly more troubled solution.

I imagined a single directory that users of my recoveryScript would update from and then upload their merged version to. That would, as you say, leave a small window during which some information could be lost if someone else uploaded in the meantime, but that window is fairly small and as long as that other user also updated from the old version of mallprices, little would get lost.

• Original copy of mallprices.txt
• User A updates mallprices.txt
• User B updates mallprices.txt
• User A uploads mallprices.txt
• User B uploads mallprices.txt
• mallprices.txt contains information from the original copy and user B.

In this instance all of the unique information from user A would get lost, but we wouldn't lose any of the information from the original version. Anyway, next time user A updates and uploads he'll probably manage to get his price info onto the server.

Not perfect, but pretty darn good nonetheless and a LOT better than the current situation. Naturally there's no reason that this should completely erase people uploading their own price data for others. This will however be an extremely effective means of dissemination.

 

[Bale]

Problem: if this map is sent to my server every time a script is run (I foresee daily use by Universal Recovery and bits of Ascend, which are quite popular), my server (actually, my server's sysadmin) may be a little annoyed. Most of my server-interactive scripts are just data fetchers, or some kind of very brief interaction. This script would involve a little number-crunching to determine which prices were newer. So I would EITHER set it up like the Map Manager (you have to manually visit the web site and submit the file), OR ask that scripts only send updated maps once daily (in the logout script would be a good place for that, but I would still request a _property limitation).

Using the code I just put in place in my Universal_recovery, it would do this only once per day, per character. Judging by my notifications, that would be about 100-150 times per day. I just updated my script a few hours ago if you want to check out daily_handling() on lines 151-189. (About 90% of that function is owed to you so you'd probably be interested in it anyway.) The update code would be inserted at line 187.

 

[jasonharper]

I don't see how map_to_file() could work with a URL - where would the filename (in the form header) come from, so that you could tell what map had been uploaded? This would have to be a separate function, with a filename parameter.

The first line in mallprices.txt is a version number, just as with all of the built-in data files - it's a much larger number than the others, to prevent accidental updating from the wrong file. It is absolutely required in any file retrieved via 'update prices', as it's the primary check to ensure that this is actually a mall price file. (If the user mistyped the URL, resulting in a server response of 404 Not Found, it would not be particularly desirable for this to cause the price of item #404 to be set equal to the version number of the server...)