TrueMall: Using true, real, actual, dynamic mall pricing in KoLmafia scripts.

I've been enjoying the excellent scripts from Bale, dj_d, Zarqon, aquaelectrix and more and feeling the need to contribute back to the community.

I think I have a modest contribution that meets a need that comes out once in a while in Item Handling scripts like EatDrink, PriceAdvisor, etc: the actual price of an item in KoLmafia. Even with the new "share mall prices" capabilities I'm finding that the fleet of KoLmafia scripts are often hampered only by the quality of the mall pricing. Not only is it sometimes outdated but there is an opportunity for better pricing resolution besides "the fifth entry in the mall". Hint: I don't think "the first entry in the mall" cuts it, either.

I know the philosophy regarding mall pricing vs mallbots and so before releasing the utility I wanted to get the ok of the community at large. Let me put down first a teaser of what TrueMall is and isn't:

Keypoints:

1. Tweaks mallprices.txt with a true pricing of an item. Not just the fifth entry in the mall or the first entry in the mall.

2. Limited to being run at *most* once a day. I hope this removes the mallbot suspicions. This is a hard limitation, and cannot easily be subverted, I promise. If there are other ways to alleviate these suspicions please let me know.

3. Is not a KoLmafia script. Works outside of KoLmafia and needs a simple command-line program and preferably a bash (not ash!) script. For the Windows folks this may present a slight learning curve. Another way to look at it: there may be an opportunity for a hosted gateway to do this automatically if anyone actually is interested in this.

4. Is not a KoLmafia script. Is meant to play nice with KoLmafia but can and will break as KoLmafia evolves. I'm not even sure it is possible to tweak and spoof mallprices.txt with current builds so it may not even work out of the gate. Work-In-Progress.


Hopefully it puts to rest the occasional query about doing an "Economics Project" (yeah: the economics project entitled Making Me Meat!) and needing better mall price resolution, in a non-destructive way. I've been very interested lately to see how efficient the Kol market is. It's surprisingly VERY efficient.

My questions to you are:

a. Would this be an offensive utility to the KoLmafia philosophy?

b. Would you find this useful?


Even if it doesn't see the light of day, thank you to all the KoLmafia developers and script writers for producing such a fine product!

hroth

You may want to chat with Multi Czar before building this. He recently shut down Items of Loathing, which had similar functionality to what you're describing: http://items.ofloathing.org/

Granted, you've addressed limited running time, so this may indeed be different enough, but it "rhymes", so I'd recommend checking before investing too much time.

Cool idea though! And yes, I'd find this extremely useful (especially now that Items of Loathing is gone).

Might I suggest that the utility could first check for an online version, and use that if it's < 24 hours old? That seems like the best way to alleviate any concerns about server hits. As to mall-botting itself, I'm not sure exactly what the stance is on that. It may be viewed negatively simply for the sake of server hits, in which case it seems you may be ok. But the devs would need to weigh in on that.

yes
no

That said, my attempts to write something that had a lot of overlap with your idea were actually the experience that convinced me to try KoLmafia many years ago. The fact that I think it would be offensive to the philosophy should not obscure the fact that I am not offended by the fact that you asked.

The basic problem with any pricing scheme is that you have to be able to act on the price in real time or any decisions you make based upon the price will be out of date when you do act. I won't bother fleshing out the analogy but if you imagine your idea applied to real world stocks (rather than the KoL Mall) you might get some different ideas about the utility, or lack thereof.

I am amused by you statement that the "real" price and the "first" price (when sorted by price) are not the same. If I am going to actually buy one of an item the difference is irrelevant. If I am going to analyze the market then the difference could be interesting, or if I am going to buy more than one purchase quotas are a factor that effects price, but if I am going to analyze the market, how much better is the analysis if the prices are minutes old rather than hours?

I can guarantee that if you make a tool and try to limit it so that it can only be run once a day, someone will find a way around that and share the results. You may be smart but thousands of monkeys are going to try and break the limitation just because it annoys them and some will succeed. Hint - can I defeat your scheme by tweaking my clock or changing my IP address or running under a different user account or running from a different computer or running from a virtual machine with no persistent storage? You don't have to answer but if you have those cases covered then it might be 15 minutes more before things get broken.

For several years I ran a script that used wget to grab a page and then a variety of bash commands to process that page and correlate data with mafia data files. Basically I was trying to make a list of items that were in the game, but not in my display case and then weed out the ones that were unobtainable for a never ascended character. When I discovered mallprices I then wanted to price these things so I could buy the low hanging fruit. That was a PITA. Sometimes there was an item name string and sometimes there was a number. Some of the strings had spaces or special characters so correlation code had to have lots of special cases and exceptions. Turns out I rewrote the entire thing in ash (with help and code snippets from posters here) in a couple hours work. This was the second time I abandoned a project because it was so much easier in the mafia environment. YMMV but even if there are techie reasons to be external, I sincerely doubt that there will be many people who will go through the effort to use your work. The ones who do are probably trying to pervert it into a mallbot anyway.

By the way I know a lot of Linux users (think Ubuntu) who have a learning curve when it comes to the command line so it won't just be Windows folks who don't make the effort.

As for then utility, I am not doing economic analysis on the Mall. I don't do any serious automated buying and selling and have no plans to do so. For big ticket items I will manually purchase them anyway so I can see what the transaction price will be so an out of date price does not harm me. I admit to being limited in imagination but I cannot perceive of any application for up to the minute prices except for a tool that searches the market looking for opportunities to buy and sell and make a profit on the difference.

The fact that there's no explanation of where the true price comes from makes that claim less meaningful.

Thanks for the response. I should have stressed the term modest contribution when I originally posted - it doesn't hit any servers at all (enforcing the one / day rule rather conveniently).

It is a simple tool to take data that's already out there and repackage it for the mallprices.txt format. Hitting the servers with a mallbot would be someone else's job.

It's not particularly clever, but the way it calculates price I thought was somewhat unique because I keep seeing debates on the "fifth price" and "first price" and thought it would be nice to have alternatives.

A discussion on which price to use is always welcome: care to expand your idea?

hroth said:

It is a simple tool to take data that's already out there and repackage it for the mallprices.txt format. Hitting the servers with a mallbot would be someone else's job.

Click to expand...

Of course, this begs the question: where do the prices come from? =)

Sure. Here's the script documentation:


# TrueMall
# --------
#
# Convert raw data from mallbots for mallprices.txt to give "true" mall pricing.
#
# True mall pricing may be, for example:
#
# 1. The cheapest item in the mall (the "first" item in the mall)
# 2. The average cost of N items in the mall. For example, you buy 321 twinkly wads;
# 1 at 180 meat (store has limit=1), 20 at 380 meat and 300 at 400 meat. You paid
# 398 per wad (using whole numbers).
# 3. The average cost of M meat worth of items in the mall. For example, you buy
# 100,000 meat worth of twinkly wads and get 213. You paid 469 per wad.

and a little further on:

# (datafile) is formatted in the log file format based on Fnord7's mallbot
# http://www.brandybuck.com/kolbot/mallbot.html, for example data archived
# daily by Eleron (#1564433) at:
#
# http://www.houeland.com/kol/malldump/
#
# (Note that sometimes Eleron's bot quits early! Get one with a full run around 786k.)

There may be other dumps of daily prices, but this one seemed reasonable. When it worked.

hroth

It should probably also reject prices that are over 1000x the previous value, I guess. We don't usually experience that type of inflation, so it's usually people manipulating the market.

[Speaking of which, that's probably something Mafia should do, as well. Well, technically, updateprices.php]

slyz said:

Of course, this begs the question: where do the prices come from? =)

Click to expand...

Shoot, I was all ready to make fun of you for improperly using "begs the question," but then I realized that, depending on how one parsed the statement you replied to, it could be considered correct. No grammar-nazi'ing for me, I am sad panda.

Sentence ends with a preposition. There you go.
("from where do the prices come?")

StDoodle said:

Shoot, I was all ready to make fun of you for improperly using "begs the question," but then I realized that, depending on how one parsed the statement you replied to, it could be considered correct. No grammar-nazi'ing for me, I am sad panda.

Click to expand...

I was going to call him for the same misuse of the technical term, but I'm failing to see how to parse his statement such that it is not a misuse.

To "beg a question" is to "presuppose the assumptions in the conclusion." More or less. How did slyz correctly "beg the question"?

(here is Feng Shweez talking about begging the question...)

Veracity said:

(here is Feng Shweez talking about begging the question...)

Click to expand...

Thanks for that link Veracity. In my defense, English isn't my first language, and I always took the "question begging" expression to mean, very literally: "This statement's wording makes us ask this question".

Veracity said:

I was going to call him for the same misuse of the technical term, but I'm failing to see how to parse his statement such that it is not a misuse.

To "beg a question" is to "presuppose the assumptions in the conclusion." More or less. How did slyz correctly "beg the question"?

Click to expand...

hroth said:

... it doesn't hit any servers at all (enforcing the one / day rule rather conveniently).

... Hitting the servers with a mallbot would be someone else's job.

Click to expand...

slyz said:

Of course, this begs the question: where do the prices come from? =)

Click to expand...

Well, perhaps the specific question as stated isn't being begged. But, if you interpret the exchange as follows:

hroth: This doesn't cause server hits, it relies on other files.
slyz: That begs the question, how do these "other files" come into existence, without hitting the server?

...then it works. Like I said, it has to be interpreted a certain way, but I don't feel that my "translation" is fundamentally different than the actual exchange.

Ignore my previous post, this was exactly what I was going for! Thanks StDoodle!

heeheehee said:

It should probably also reject prices that are over 1000x the previous value, I guess. We don't usually experience that type of inflation, so it's usually people manipulating the market.

[Speaking of which, that's probably something Mafia should do, as well. Well, technically, updateprices.php]

Click to expand...

heeheehee,

(I suppose I have to be more careful with my grammar...)

It is easy to put this in, but I wanted to understand where this would be applied; how does one manipulate the market by offering a super-high price? For example, let's say we have a situation with:

Store #1: Twinkly Wads (100) Limit 1/day, Price 180
Store #2: Twinkly Wads (30) Price 380
Store #3: Twinkly Wads (50) Price 390
Store #4: Twinkly Wads (70) Price 400
Store #5: Twinkly Wads (1) Price 400000

If QTY_LIMIT=151 (one needs to buy 151 twinkly wads) the average price for one to purchase twinkly wads is 391 meat.

If QTY_LIMIT=152 (one needs to buy 152 twinkly wads) the pricing for twinkly wads is 3020 meat. You needed 152 twinkly wads.


On the other hand, your request is a good reminder to test the corner condition to make sure the program does reject the 400,000 twinkly wad if you just want to buy 100,000 meat worth of twinkly wads. Thanks.

hroth

In theory it could be important. In practice the mall will never look like that. Even for something like fat stacks of cash where it went from worthless to very valuable, the price was never nearly that high. Mr. Store stuff could be inflated to artificially high prices by someone sufficiently rich, but they're always priced at > 1m and the mall doesn't show prices of > 1b.

As far as I can tell, mallbots are plentiful and allowed, though discouraged. I certainly haven't heard of mass bannings and complaints about them. Nobody really knows why itemsofloathing.com was shut down and we just speculate that it is because of server hits. Maybe they didn't want the historical data available or wanted their own more official version. Who knows, until we get an official response which nobody (including the creator of itemsofloathing) has received yet.

The philosophy of KolMafia is to add safeguards so that people cannot use KolMafia to make mallbots, so that their hard work is not used in a manner that they disagree with. I can see opinions varying on the idea of your script.

My initial thought is that you are already taking mall dumps, so I'd think just setting the kol file prices to yours is good enough and we don't really need the latest/greatest prices.


What I would love more than anything would be some kind of KolMafia script that lets you enter a particular item and then it goes to your website and finds the trend-data for that item over time.
Basically a replacement to itemsofloathing, which I really miss and you already have a lot of good historical data.

SinginSally said:

As far as I can tell, mallbots are plentiful and allowed, though discouraged. I certainly haven't heard of mass bannings and complaints about them.

Click to expand...

Forget itemsofloathing, there have been publicized cases of mallbots being disabled (announced by those that were banned, verified by TPTB).

SinginSally said:

What I would love more than anything would be some kind of KolMafia script that lets you enter a particular item and then it goes to your website and finds the trend-data for that item over time.
Basically a replacement to itemsofloathing, which I really miss and you already have a lot of good historical data.

Click to expand...

I know this is a bit out of left field, but I would also like to see this. In my WoW-playing days I really enjoyed checking out www.wowecon.com. Historical data is very cool.

Also, I love to play on yahoo's finance page. Those graphs are friggen' sweet!