Feature historical_price does not update on mall extinct items, can not load -1

[Irrat]

Is this a bug or feature?

This is marked as a feature rather than a bug as this could be intended behavior, just that it hadn't considered a future with mall extinction.

First Issue

When an item is mall extinct (or only 4 is available), then mall_price will return -1
Edit: I was wrong, an item with at least one in mall still updates.

However, historical price will not update to reflect this and will instead report the same historical price that was stored. Even if the historical price is say, 60m. For example "Jeppson's Malort" is reported at 450m, but is mall extinct. This is always going to be a thing for all quantity limited items, such as iotms.

> js const i = Item.get("Jeppson's Malort");    print("Historical: " + historicalPrice(i)); print("Mall: " + mallPrice(i,    0)); print("Historical: " + historicalPrice(i));

Historical:    450000000
Searching for "Jeppson's Malort"...
Search    complete.
Mall: -1
Historical: 450000000
Returned: null

Although we could update historical_price to say mall max 999_999_999, I think it should be updated to -1 to reflect that it is mall extinct, not mall max.

If this is corrected, then the next issue then comes into play.

Second issue

We can upload -1 to the historical price database on the site (Not sure if that's an issue), but mafia will not load -1 due to this check
Now, that check is actually valid, because historical_price should not save a price as -1. That's strictly a modded mafia or third party call thing.
So instead the line will fail to load while logging a warning which is no longer desired if -1 is now a thing.

In regards to the site, @Veracity is probably the one to poke about it from what I understand.

So why you looking at this?

The reason I was playing with the whole historical_price thing is due to a separate project which is basically aimed at an alternative source of "historical prices" without the 5th item limitation, but with its own inaccurate limitations and funny behavior.

The cliff notes is that it isn't a true snapshot of the lowest price in mall and will adjust the displayed price by factors such as sales, limits, availability, with some decay for price swings. Basically, it was created as a tool to value items while attempting to ignore abnormal pricing and without asking users to do their own batches of server hits. So it may see some item at 1/day, yet it may skip the first 5 stores if the item is popular.

I was looking into historical_price as it was brought to my attention that mafia's historical prices could benifit from also being updated.

It should be mentioned though that as I am not running this through mafia I have to reimplement mafia's pricing logic to ensure there's no abnormal prices ever reported, it must always reflect exactly what mafia thinks the price would be if mafia had been the one to look it up. Thus it's still in the testing stages until I'm absolutely confident, which will take some time.

I do aim to eventually make most of the logic open source for transparency when I've finished prettifying the code, with an exception for code useful for mall botting. No need to provide the tools for that.

 

[fronobulax]

Not directly dealing with your questions.

You might look at CFstat which will get price and volume from Coldfront and cache results. It was originally written as an alternative to historical_prices and a way to get sales volume. A Long Time Ago I did some experiments using mall_price, historical_price and the CFStat price for buy and mall price decisions. I finally decided that for my purposes historical_price was good enough. I still use sales volume, though.

I had a list of items that I did not have in my display case and their historical prices. I tried to automate buying and discovered what you call "mall extinct" items. I eliminated my frustration with buy failures by using historical prices that were younger than seven days and mall prices otherwise. So if I waited long enough mall extinct items in historical_prices was no longer a problem. The cost of that was a server hit for each extinct item, however.

I ran some experiments, years ago, about eliminating mall extinct items from historical_prices. I gave up because a) no one wanted to undo the work if something ever returned from mall extinction and b) there were players who found the last price in the mall to be useful no matter how old. I suspect the latter were selling items via trades and not the mall.

I've made my peace with the status quo and just script for the possibility that an historical_price might be irrelevant.

Changing the server code that is updating the prices is not part of the usual development cycle and a feature that requires changing it probably needs to be thought through before it is rolled out.

I have noted that my scripts check mall extinct items every time they are called. If there were a performance issue I might consider a local override that was available for subsequent calls but did not update the global prices, That seems like more trouble than it is worth so I haven't done it.

You observation that in most scripts "mall extinct" actually means there are less than five items in the mall is interesting.

 

[Irrat]

Not directly dealing with your questions.

You might look at CFstat which will get price and volume from Coldfront and cache results. It was originally written as an alternative to historical_prices and a way to get sales volume. A Long Time Ago I did some experiments using mall_price, historical_price and the CFStat price for buy and mall price decisions. I finally decided that for my purposes historical_price was good enough. I still use sales volume, though.

I had a list of items that I did not have in my display case and their historical prices. I tried to automate buying and discovered what you call "mall extinct" items. I eliminated my frustration with buy failures by using historical prices that were younger than seven days and mall prices otherwise. So if I waited long enough mall extinct items in historical_prices was no longer a problem. The cost of that was a server hit for each extinct item, however.

I ran some experiments, years ago, about eliminating mall extinct items from historical_prices. I gave up because a) no one wanted to undo the work if something ever returned from mall extinction and b) there were players who found the last price in the mall to be useful no matter how old. I suspect the latter were selling items via trades and not the mall.

I've made my peace with the status quo and just script for the possibility that an historical_price might be irrelevant.

Changing the server code that is updating the prices is not part of the usual development cycle and a feature that requires changing it probably needs to be thought through before it is rolled out.

I have noted that my scripts check mall extinct items every time they are called. If there were a performance issue I might consider a local override that was available for subsequent calls but did not update the global prices, That seems like more trouble than it is worth so I haven't done it.

You observation that in most scripts "mall extinct" actually means there are less than five items in the mall is interesting.

Yeah, I'm aware of the coldfront stuff. Which is where I'm pulling data for the volume etc. The main difference is that this is distributed as a file so that individual users are not hitting coldfront.

I think the main difference in your perspective with mine, is that I'm thinking in a perspective of "historical_price to quickly grab the price an item is in mall for" vs your "the last price it was seen in mall for".
The difference being that I'm viewing it as a mall_price cache instead of a "last sold for" cache.

As for the server code, I don't think that's actually needed? The server code seems to perform as desired already. So the only changes would be inside the kolmafia application itself. That being to just change the ability to load a negative number, and to store when an item wasn't found in mall.

I ran some experiments, years ago, about eliminating mall extinct items from historical_prices. I gave up because a) no one wanted to undo the work if something ever returned from mall extinction

I think that instead of being removed from historical prices, it should be able to store -1. This should be completely possible just by changing the linked code in the original comment, and allowing historical_price to be updated with mall_price returning -1.

I don't feel a need to remove it from the file.

players who found the last price in the mall to be useful no matter how old

I'm vaguely interested in the reasoning, because when the item isn't available at that price then it feels like it should be sold at a higher price. Such as source terminal which is 900m, but you'll find it for at least 1.5b with another player.

It could be useful for a "It was in mall X months ago" sure, but there's no protection I'm aware of on the prices a user can submit. Rendering it unreliable. Which does feel like it should invalidate most arguments about historical value when said historical value can be manipulated unless the serverside is storing information about who submitted it. Especially when we have coldfront who does store more information about sales, but the unfortunate behavior of not storing beyond 2 years.

Bad behavior can be done as simply by changing two numbers in the prices file after all, and I don't think there's a real way to secure that kind of behavior, nor should there be. historical_price isn't meant to be trusted in that sense afaik.

You observation that in most scripts "mall extinct" actually means there are less than five items in the mall is interesting.

I apologize, I clearly made an incorrect assumpion.

 

[fronobulax]

At one point I *thought* the server side code was doing some verification and validation so that "bad data" as it was defined at the time, was not inserted into the file. I also *thought* there was checking so that the latest entry, as defined by KoL time, was used regardless of the order of submission or delays in submission. Hence my general comment/concern about confirming that any proposed change did (not) require server side adjustments.

I am digesting the rest of your comments.

 

[heeheehee]

One concern I have with storing -1 as a valid (crowdsourced) mallprice is that if an item is temporarily extinct (very possible for a low-volume item) or hasn't been added to the mall yet, we may end up caching -1 for longer than intended. This is probably fine, but something to be aware of.

Similarly, existing scripts would need to be able to handle negative historical_price(). I bet there are some scripts out there that are explicitly checking `price == 0` instead of <= 0.

I personally like storing a big number -- if there are concerns about accidentally trying to buy a thing that costs 2 billion even though the mall doesn't support that, then just make it bigger than MAX_MEAT, e.g. 5 billion.

I do agree that 450M is a worse answer than "there are none left" but the exact details need more discussion.

In my scripts that take advantage of full mall prices (e.g. diet-related), I've taken to just doing giant batched mall queries up front via mall_prices() -- something like 30 page hits to get all the results I care about on a regular basis... although not all page hits are created equal. Still better than issuing a few thousand requests, each of which would be backed by a database query, though.

 

[Irrat]

One concern I have with storing -1 as a valid (crowdsourced) mallprice is that if an item is temporarily extinct (very possible for a low-volume item) or hasn't been added to the mall yet, we may end up caching -1 for longer than intended. This is probably fine, but something to be aware of.

Similarly, existing scripts would need to be able to handle negative historical_price(). I bet there are some scripts out there that are explicitly checking `price == 0` instead of <= 0.

I personally like storing a big number -- if there are concerns about accidentally trying to buy a thing that costs 2 billion even though the mall doesn't support that, then just make it bigger than MAX_MEAT, e.g. 5 billion.

I do agree that 450M is a worse answer than "there are none left" but the exact details need more discussion.

In my scripts that take advantage of full mall prices (e.g. diet-related), I've taken to just doing giant batched mall queries up front via mall_prices() -- something like 30 page hits to get all the results I care about on a regular basis... although not all page hits are created equal. Still better than issuing a few thousand requests, each of which would be backed by a database query, though.

Ah, I hadn't considered that aspect.

Probably should be reset to 0 then, which should sort that out. I find it interesting that mall price and historical price differ in that aspect, unreasonably so in my opinion.

Although -1 makes sense, I don't see a real reason we can't use 0. Historical age still exists after all, though I'm not sure how usable it is in ash given it returns infinity..

Having it "mall extinct" for a while feels like it should be a non issue, though it may be a consideration that we could update historical prices more. I'm not sure if we update it whenever possible or only when specified calls are made.

Backoffice requests for example, they could be used if we can verify that the lowest 5 prices are definitely in the returned response. It might also be a thing that we are not storing prices gained from a relay search. I hadn't checked the code and am likely wrong. Although, something like back office may not be easily feasible as it practically isn't usable for mall price due to containing a lack of info.

 

[Irrat]

One concern I have with storing -1 as a valid (crowdsourced) mallprice is that if an item is temporarily extinct (very possible for a low-volume item) or hasn't been added to the mall yet, we may end up caching -1 for longer than intended. This is probably fine, but something to be aware of.

Similarly, existing scripts would need to be able to handle negative historical_price(). I bet there are some scripts out there that are explicitly checking `price == 0` instead of <= 0.

I personally like storing a big number -- if there are concerns about accidentally trying to buy a thing that costs 2 billion even though the mall doesn't support that, then just make it bigger than MAX_MEAT, e.g. 5 billion.

I do agree that 450M is a worse answer than "there are none left" but the exact details need more discussion.

In my scripts that take advantage of full mall prices (e.g. diet-related), I've taken to just doing giant batched mall queries up front via mall_prices() -- something like 30 page hits to get all the results I care about on a regular basis... although not all page hits are created equal. Still better than issuing a few thousand requests, each of which would be backed by a database query, though.

Actually, looked at the script in question. Putting it at 0 may actually break libram. libram expects it to be -1 if its mall extinct.

 

[heeheehee]

Changing mall_price() to return 0 would definitely break existing scripts.

I just cited that as the first instance that I found that checked historical_price(it) == 0, even if it's probably fine here.

I find it interesting that mall price and historical price differ in that aspect, unreasonably so in my opinion.

I'm inclined to agree, but this has been observable behavior for so many years that it's... complicated to change without breaking things horribly along the way (see: Hyrum's law).