Missing drop rate spading

digitrev

Member
So I've noticed that several locations have items without drop rates. I've decided that this just will not do, and hence that I should start doing some item spading. This is the script I'm using to collect data:

Code:
//   where   , fight,   drop, drate, ndropped, times it happened
int [location, monster, item, float, int] timesDropped;
string fName = "dropSpading.txt";

string bayesJSON(int[float, int] toJSON){
	string buff = '[';
	foreach drate in toJSON{
		buff += '{counts: [';
		int maxDropped = 0;
		foreach nDropped in toJSON[drate]{
			if (nDropped > maxDropped) maxDropped = nDropped;
		}
		for n from 0 to maxDropped{
			buff += toJSON[drate, n].to_string() + ', ';
		}
		buff += '], boost: ' + drate.to_string() + '},';
	}
	buff += ']';
	return buff;
}

void makeJSONs(){
	string[location, monster, item] json;
	if (file_to_map(fName, timesDropped)){
		print("map loaded");
	} else {
		abort("map not loaded");
	}
	foreach l, m, i in timesDropped{
		json[l, m, i] = bayesJSON(timesDropped[l, m, i]);
	}
	if (map_to_file(json, "bayesjson.txt")){
		print("map saved");
	} else {
		print("map not saved");
	}
}

void main(string sloc, int advLen){
	if (file_to_map(fName, timesDropped)){
		print("map loaded");
	} else {
		abort("map not loaded");
	}
	location loc = sloc.to_location();
	for i from 1 to advLen{
		int[item] idrops;
		foreach i, mon in loc.get_monsters(){
			foreach it in mon.item_drops(){
				idrops[it] = it.available_amount();
			}
		}
		float drate = numeric_modifier("item drop");
		adventure(1, loc);
		monster mon = last_monster();
		foreach it in mon.item_drops(){
			int dropped = it.available_amount() - idrops[it];
			timesDropped[loc, mon, it, drate, dropped] += 1;
			//print("item spade," + drate + "," + it.to_string() + "," + last_monster().to_string() + "," + dropped.to_string());
		}
	}
	if (map_to_file(timesDropped, fName)){
		print("map saved");
	} else {
		print("map not saved");
	}
}

I'll be dropping the result JSON strings into the Bayes tester (http://ludonomicon.com/kol/CoffeeBayes.html) and start spading missing data. Now, since I'd like this data to ultimately make its way into KoLmafia proper, I was wondering if folks could tell me if there's any obvious mistakes I've made in the code.

Also, I'll only be satisfied for a given drop rate once it passes 95% belief in the Bayes tester (using the Full Range analysis). I'll gladly submit all my raw data so that folks can verify it.
 
Last edited:

digitrev

Member
For example, here's the output after several adventures in The Haunted Wine Cellar:

Code:
The Haunted Wine Cellar	mad wino	crazy hobo notebook	270.0	0	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	270.0	1	1
The Haunted Wine Cellar	mad wino	crazy hobo notebook	275.0	0	4
The Haunted Wine Cellar	mad wino	crazy hobo notebook	275.0	1	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	361.749371855331	0	6
The Haunted Wine Cellar	mad wino	crazy hobo notebook	361.749371855331	1	2
The Haunted Wine Cellar	mad wino	crazy hobo notebook	364.4456264653803	0	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	364.4456264653803	1	3
The Haunted Wine Cellar	mad wino	crazy hobo notebook	369.4404424085076	0	4
The Haunted Wine Cellar	mad wino	crazy hobo notebook	369.4404424085076	1	3
The Haunted Wine Cellar	mad wino	crazy hobo notebook	377.0	0	1
The Haunted Wine Cellar	mad wino	crazy hobo notebook	377.0	1	1
The Haunted Wine Cellar	mad wino	crazy hobo notebook	382.0	0	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	382.0	1	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	389.4456264653803	0	8
The Haunted Wine Cellar	mad wino	crazy hobo notebook	389.4456264653803	1	5
The Haunted Wine Cellar	mad wino	crazy hobo notebook	396.79130371550696	0	1
The Haunted Wine Cellar	mad wino	crazy hobo notebook	396.79130371550696	1	2
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	270.0	0	1
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	270.0	1	5
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	275.0	0	4
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	275.0	1	5
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	361.749371855331	0	3
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	361.749371855331	1	5
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	364.4456264653803	0	2
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	364.4456264653803	1	6
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	369.4404424085076	0	1
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	369.4404424085076	1	6
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	377.0	1	2
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	382.0	0	3
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	382.0	1	7
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	389.4456264653803	0	1
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	389.4456264653803	1	12
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	396.79130371550696	0	1
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	396.79130371550696	1	2
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	270.0	0	3
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	270.0	1	3
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	275.0	0	6
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	275.0	1	3
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	361.749371855331	0	4
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	361.749371855331	1	4
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	364.4456264653803	0	2
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	364.4456264653803	1	6
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	369.4404424085076	0	3
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	369.4404424085076	1	4
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	377.0	0	1
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	377.0	1	1
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	382.0	0	7
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	382.0	1	3
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	389.4456264653803	0	5
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	389.4456264653803	1	8
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	396.79130371550696	0	1
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	396.79130371550696	1	2
The Haunted Wine Cellar	mad wino	Psychotic Train wine	270.0	0	4
The Haunted Wine Cellar	mad wino	Psychotic Train wine	270.0	1	2
The Haunted Wine Cellar	mad wino	Psychotic Train wine	275.0	0	5
The Haunted Wine Cellar	mad wino	Psychotic Train wine	275.0	1	4
The Haunted Wine Cellar	mad wino	Psychotic Train wine	361.749371855331	0	5
The Haunted Wine Cellar	mad wino	Psychotic Train wine	361.749371855331	1	3
The Haunted Wine Cellar	mad wino	Psychotic Train wine	364.4456264653803	0	3
The Haunted Wine Cellar	mad wino	Psychotic Train wine	364.4456264653803	1	5
The Haunted Wine Cellar	mad wino	Psychotic Train wine	369.4404424085076	0	4
The Haunted Wine Cellar	mad wino	Psychotic Train wine	369.4404424085076	1	3
The Haunted Wine Cellar	mad wino	Psychotic Train wine	377.0	0	2
The Haunted Wine Cellar	mad wino	Psychotic Train wine	382.0	0	5
The Haunted Wine Cellar	mad wino	Psychotic Train wine	382.0	1	5
The Haunted Wine Cellar	mad wino	Psychotic Train wine	389.4456264653803	0	4
The Haunted Wine Cellar	mad wino	Psychotic Train wine	389.4456264653803	1	9
The Haunted Wine Cellar	mad wino	Psychotic Train wine	396.79130371550696	0	1
The Haunted Wine Cellar	mad wino	Psychotic Train wine	396.79130371550696	1	2
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	270.0	0	7
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	275.0	0	12
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	361.749371855331	0	15
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	364.4456264653803	0	7
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	369.4404424085076	0	7
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	377.0	0	2
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	382.0	0	12
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	389.4456264653803	0	7
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	396.79130371550696	0	4
The Haunted Wine Cellar	possessed wine rack	bottle of Chateau de Vinegar	427.2418693812442	0	1
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	270.0	0	8
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	275.0	0	11
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	361.749371855331	0	8
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	364.4456264653803	0	9
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	369.4404424085076	0	5
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	377.0	0	4
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	382.0	0	9
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	389.4456264653803	0	13
The Haunted Wine Cellar	skeletal sommelier	ghost accordion	396.79130371550696	0	5
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	270.0	0	7
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	270.0	1	1
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	275.0	0	10
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	275.0	1	1
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	361.749371855331	0	8
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	364.4456264653803	0	8
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	364.4456264653803	1	1
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	369.4404424085076	0	4
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	369.4404424085076	1	1
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	377.0	0	4
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	382.0	0	4
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	382.0	1	5
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	389.4456264653803	0	10
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	389.4456264653803	1	3
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	396.79130371550696	0	2
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	396.79130371550696	1	3
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	270.0	0	8
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	275.0	0	11
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	361.749371855331	0	8
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	364.4456264653803	0	8
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	364.4456264653803	1	1
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	369.4404424085076	0	3
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	369.4404424085076	1	2
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	377.0	0	3
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	377.0	1	1
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	382.0	0	9
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	389.4456264653803	0	11
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	389.4456264653803	1	2
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	396.79130371550696	0	5
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	270.0	0	8
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	275.0	0	11
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	361.749371855331	0	8
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	364.4456264653803	0	9
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	369.4404424085076	0	5
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	377.0	0	4
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	382.0	0	9
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	389.4456264653803	0	13
The Haunted Wine Cellar	skeletal sommelier	wine-soaked bone chips	396.79130371550696	0	5

here's the JSON strings
Code:
The Haunted Wine Cellar	mad wino	crazy hobo notebook	[{counts: [5, 1, ], boost: 270.0},{counts: [4, 5, ], boost: 275.0},{counts: [6, 2, ], boost: 361.749371855331},{counts: [5, 3, ], boost: 364.4456264653803},{counts: [4, 3, ], boost: 369.4404424085076},{counts: [1, 1, ], boost: 377.0},{counts: [5, 5, ], boost: 382.0},{counts: [8, 5, ], boost: 389.4456264653803},{counts: [1, 2, ], boost: 396.79130371550696},]
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	[{counts: [1, 5, ], boost: 270.0},{counts: [4, 5, ], boost: 275.0},{counts: [3, 5, ], boost: 361.749371855331},{counts: [2, 6, ], boost: 364.4456264653803},{counts: [1, 6, ], boost: 369.4404424085076},{counts: [0, 2, ], boost: 377.0},{counts: [3, 7, ], boost: 382.0},{counts: [1, 12, ], boost: 389.4456264653803},{counts: [1, 2, ], boost: 396.79130371550696},]
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	[{counts: [3, 3, ], boost: 270.0},{counts: [6, 3, ], boost: 275.0},{counts: [4, 4, ], boost: 361.749371855331},{counts: [2, 6, ], boost: 364.4456264653803},{counts: [3, 4, ], boost: 369.4404424085076},{counts: [1, 1, ], boost: 377.0},{counts: [7, 3, ], boost: 382.0},{counts: [5, 8, ], boost: 389.4456264653803},{counts: [1, 2, ], boost: 396.79130371550696},]
The Haunted Wine Cellar	mad wino	Psychotic Train wine	[{counts: [4, 2, ], boost: 270.0},{counts: [5, 4, ], boost: 275.0},{counts: [5, 3, ], boost: 361.749371855331},{counts: [3, 5, ], boost: 364.4456264653803},{counts: [4, 3, ], boost: 369.4404424085076},{counts: [2, 0, ], boost: 377.0},{counts: [5, 5, ], boost: 382.0},{counts: [4, 9, ], boost: 389.4456264653803},{counts: [1, 2, ], boost: 396.79130371550696},]
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	[{counts: [7, 1, ], boost: 270.0},{counts: [10, 1, ], boost: 275.0},{counts: [8, 0, ], boost: 361.749371855331},{counts: [8, 1, ], boost: 364.4456264653803},{counts: [4, 1, ], boost: 369.4404424085076},{counts: [4, 0, ], boost: 377.0},{counts: [4, 5, ], boost: 382.0},{counts: [10, 3, ], boost: 389.4456264653803},{counts: [2, 3, ], boost: 396.79130371550696},]
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	[{counts: [8, 0, ], boost: 270.0},{counts: [11, 0, ], boost: 275.0},{counts: [8, 0, ], boost: 361.749371855331},{counts: [8, 1, ], boost: 364.4456264653803},{counts: [3, 2, ], boost: 369.4404424085076},{counts: [3, 1, ], boost: 377.0},{counts: [9, 0, ], boost: 382.0},{counts: [11, 2, ], boost: 389.4456264653803},{counts: [5, 0, ], boost: 396.79130371550696},]

And finally, here are my results:
Code:
Location	Monster	Item	Belief	Drop rate
The Haunted Wine Cellar	mad wino	crazy hobo notebook	30.49	9%
The Haunted Wine Cellar	mad wino	page of the Necrohobocon	34.5	17%
The Haunted Wine Cellar	mad wino	pie man was not meant to eat	28.88	11%
The Haunted Wine Cellar	mad wino	Psychotic Train wine	29.82	11%
The Haunted Wine Cellar	skeletal sommelier	sommelier's towel	35.87	5%
The Haunted Wine Cellar	skeletal sommelier	tarnished tastevin	53.94	2%
 

Darzil

Developer
I've not looked at the code, but I've previously thought about auto drop spading.

You'll need to know:
Pickpocket chance and whether item was pickpocketed.
Whether item was rave stolen.
Whether item was yellow rayed.
Item drop chance at the time the drop took place (has to be before character pane refresh after the fight).
Everything you were wearing and every effect you had (could be something unknown).
Ideally it'd handle monsters who dropped multiple identical items.
 

digitrev

Member
I'm using a muscle class and not pickpocketing, so as to avoid complications with pickpocketing, rave steal, and yellow rays. The item drop chance I'm handling by storing it before I adventure (and since I'm not using anything that will change the rate mid-fight, that will be the same as the item drop chance at the time the item dropped). It doesn't log current equipment and effects. I suppose I could have it do that, but I'd have to rewrite a large chunk of it. I've mostly been relying on the fact that mafia knows the +item% of most items. As for monsters with multiple drops, it will handle them, but not properly. That needs to be fixed.
 

Darzil

Developer
Ah, I misread, you want the data, not the code, to reach mafia proper. Those were more the things I thought of when I considered allowing everyone's mafia to (optionally) become a spade army.

You don't have to worry about that stuff then.

I'm not statistics person so probably can't comment in detail. I tend to just look manually at the data and work it out with pen and paper, and compare to history and Jick's tendency to aim for 5% breakpoints, with occasional exceptions.
 

digitrev

Member
Excellent. In that case, I'll just occasionally submit a bug report with my spaded drop rates and the link to the raw data & JSON strings.
 
Top