BeachComber - fast and efficient beach combing

[Veracity]

Regarding the error, my session logs say:

(9077,2,9) is an 'uncommon' tile.
(9077,3,10) is an 'uncommon' tile.
(9077,8,9) is a 'unknown' tile.
(9077,8,10) is an 'uncommon' tile.
(9077,9,4) is an 'uncommon' tile.
(9077,10,2) is an 'uncommon' tile.
(9077,10,5) is a 'uncommon' tile.
(9077,10,6) is an 'uncommon' tile.
(9077,10,10) is an 'uncommon' tile.

Note that columns go from 1-10 - as logged in the session log - but as reported by KoLmafia, they are zero-indexed, and in my (and combo's) JSON data, they go from 0-9.

Looking at that session log:

Encounter: Comb the Beach (9077 minutes down the beach)
> 1 squares in beach 9077 contain combed sand
> 65 squares in beach 9077 contain rough sand
> 4 squares in beach 9077 contain rough sand with a twinkle
> Combing the square at coordinates (9077,8,9) which contains rough sand with a twinkle
Combing square 0,1 (0 minutes down the beach)
> (9077,8,9) is a 'unknown' tile.

I did not like that logging, so I fixed it and tried again.

Encounter: Comb the Beach (9077 minutes down the beach)
> 1 squares in beach 9077 contain combed sand
> 65 squares in beach 9077 contain rough sand
> 4 squares in beach 9077 contain rough sand with a twinkle
> Combing the square at coordinates 10,90765 which contains rough sand with a twinkle
Combing square 10,6 (9077 minutes down the beach)
You acquire an item: dull fish scale
> (9077,10,5) is a 'uncommon' tile.

I did not like that logging, so I fixed it and tried again.

Encounter: Comb the Beach (6270 minutes down the beach)
> 68 squares in beach 6270 contain rough sand
> 2 squares in beach 6270 contain rough sand with a twinkle
> Combing the square at coordinates 5,62696 which contains rough sand with a twinkle
Combing square 5,5 (6270 minutes down the beach)
You acquire an item: sand dollar
> (6270,5,5) is a 'uncommon' tile.

That was good and was how I logged it henceforth.

You found the single tile that got logged - and stored - incorrectly.
I will remove (9077,10,5) from the uncommons set.
Thanks!

I'm about to add more stuff for spading - which will not be used if you are simply looking for rares, like combo:
commons (rough sand and sand castle)
combed (unknown)
heads (well, these are static, but what the heck.)

Since there are about 950,000 commons, this is data bloat for people (not me) who are not actively spading.

Over the next 5 days or so, I'll have identified the vast majority of commons.
Subsequent passes will go only to beaches where I have seen (not yet identified) combed tiles.

And combed will be exclude things that I, personally, have combed, and have identified as rare, uncommon, or common.
That will eventually be the equivalent of what you are doing, in that all unknowns will come out of the combed set.

Did you want me to make a PR to add any new tiles to your lists?

Sure, thank you. That will be very helpful!

Regarding whales, I have seen 5 of them in the last 3 months.
I have seen both whales and normal rares in the same tile.
I have no reason to think that they only appear in certain rare tiles; I expect KoL randomizes what you get in a rare tile.

Encounter: Comb the Beach (4067 minutes down the beach)
> 3 squares in beach 4067 contain combed sand
> 76 squares in beach 4067 contain rough sand
> 1 squares in beach 4067 contain rough sand with a twinkle
> Combing the square at coordinates (4067,3,9) which contains rough sand with a twinkle
Combing square 3,9 (4067 minutes down the beach)
You acquire an item: piece of driftwood
> (4067,3,9) is a 'rare' tile.

and

Encounter: Comb the Beach (4067 minutes down the beach)
> 1 squares in beach 4067 contain a beached whale
> 2 squares in beach 4067 contain combed sand
> 77 squares in beach 4067 contain rough sand
> You found a beached whale!
> Saving page HTML to Veracity_4067_3_9_20230823234110454.html
> Combing the square at coordinates (4067,3,9) which contains a beached whale
Combing square 3,9 (4067 minutes down the beach)
You gain 10,723,919 Meat
> (4067,3,9) is a 'rare' tile.
> Saving page HTML to Veracity_4067_3_9_20230823234110636.html

 

[Irrat]

Hmm, comparing both datasets. We have 41 beaches that likely have a hoarded rare we're unaware of yet.
Number reached by filtering out the minutes that contain a known rare, and the minute contains a combed or unknown sparkle as per my own dataset.

I have about 900 beaches with unknown tiles left, including known rares as my dataset doesn't include anything it hasn't found itself. Only roughly 3% of the turns are finding a tile to spade now. As the number goes down, I expect to spade more hoarded tiles as this is a standalone script that runs on a cronjob, and will visit the same beach multiple times a day as more adventures are remaining left over as beaches are crossed off.

As for whales, yeah. I left it as a special tile instead of a rare cos I wasn't 100% if it was a normal rare and found it cute in the data.

 

[Veracity]

Thanks for the PR. I accepted & merged it.

I have added detection of commons - rough sand (if you have TwinkleVision™) and sand castles - from the beach map returned by KoLmafia.
I also save combed tiles - if they are not known rares, uncommons, or commons.
Seeing twinkles, rough sand, sand castles, and beached whales from that same beach map removes them from previously seen combed tiles.

I populated the set of commons with those I combed myself over the last 3 months of spading; BeachComberFix scraped the session logs of all my characters and extracted the common tiles.

I estimate there will end up being about 950,000 commons, so this is a big file - and it takes noticeable time to load at startup, but in memory, it is in a much faster data structure for searching, so it is not noticeably slower to visit beaches and comb tiles.

I've only been saving commons and combed tiles for a couple days.

Here's the current data:

> BeachComber data

Known rare tiles: 935
Locally discovered rare tiles: 44
Erroneous rare tiles: 1

Verified rare tiles: 352
Newly verified rare tiles: 8
Total: 360

Known uncommon tiles: 48732
New uncommon tiles: 11
Total: 48743

Known common tiles: 18159
New common tiles: 393167
Total: 411326

Known combed tiles: 341

Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 3
Last minutes down the beach spaded: 430

Beaches with rare tiles: 892
Beaches with verified rare tiles: 353
Beaches with uncommon tiles: 9944
Beaches with common tiles: 7935
Beaches with combed tiles: 271
Beaches with sand castles: 180
Beaches with unvisited twinkles: 3

I also souped up BeachComberUtilities to have 3 kinds of analysis.

As before:

> BeachComberUtilities uncommon

56 beaches have no known uncommons.
4 beaches with no uncommons have a verified rare tile.
2 beaches with no uncommons have an unverified rare tile.
50 beaches with no uncommons have no known rare tile.

This is with your new data.

> BeachComberUtilities combed

There are 341 unknown combed tiles on 271 beaches.

Ditto.

> BeachComberUtilities tides

There are 7935 beaches with observed commons.
1251 beaches have been spaded with tides = 0
977 beaches have been spaded with tides = 1
2542 beaches have been spaded with tides = 2
1750 beaches have been spaded with tides = 3
1412 beaches have been spaded with tides = 4
3 beaches have only seen higher rows

Note that those are not "fully spaded" beaches; as I said, I initialized the set of commons from my session logs, which have many beaches visited at lower tides.

After I've made a few passes over the whole tide cycle, I'll write another function which will determine which beaches have been fully spaded - known rares, uncommons, commons, beach head add up to 100 tiles.

It will take a while. You are definitely ahead of me.

 

[Veracity]

I made (a bit more than) 1 pass through all 10000 minutes of the beach.
"A bit more" because I ended up with tides = 0 today, and looked at 2,000 or so beaches that way.

You have to enable "commons and "combed" tracking, since the tiles.common.jason is almost 35 MB (!).
Noticeably slow to read the file into a fast data structure in memory, but, once loaded, actual combing is still zippy.

Revision 29 has this:

> BeachComber data

Known rare tiles: 935
Locally discovered rare tiles: 45
Erroneous rare tiles: 1

Verified rare tiles: 352
Newly verified rare tiles: 26
Total: 378

Known uncommon tiles: 48732
New uncommon tiles: 20
Total: 48752

Known common tiles: 18159
New common tiles: 771607
Total: 789766

Known combed tiles: 525

Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 2
Last minutes down the beach spaded: 8488

And:

> BeachComberUtilities combed

There are 525 unknown combed tiles on 383 beaches.

> BeachComberUtilities tides

There are 10000 beaches with observed commons.
3052 beaches have been spaded with tides = 0
2336 beaches have been spaded with tides = 1
2002 beaches have been spaded with tides = 2
1475 beaches have been spaded with tides = 3
1135 beaches have been spaded with tides = 4
0 beaches have only seen higher rows

I am glad the tides never exceeded the maximum.

I'm going to do the following:

1) Write another utility to see how many beaches are "incomplete" at tides 4, 3, 2, 1, 0
That is to say, all visible tiles are common (rough sand or sand castles), uncommon (twinkles), rare (twinkle or whale), or beach head.

2) Given that, based on the tide, only look at beaches which are "incomplete" at that tide level.
There is no need to combed the beach at tide=4 if there are no unknown "combed" tiles to be discovered.

This will result in faster discovery - visiting only beaches with previously unidentified combed patches.
I think this is more-or-less the equivalent of Irrat's approach.

It will eventually (once I have seen all tiles at tides=0) result in a single character being able to visit all of the "incomplete" beaches each day.

 

[Veracity]

Progress report:

I've added "completeness" checking based on all the beaches I've combed over the last three months at the various tidal levels. Had I been saving commons from the beginning, I'd be a lot closer, but tides = 0 happens once every 8 days and tides = 1 happens twice, so those are lagging.

> BeachComberUtilities complete

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading beach castle data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 937
Total uncommons = 48768
Total commons = 847417
Total heads = 11
Total combed = 322
Total unseen = 102545
Total: 1000000

row	rare	uncommon	common	heads	combed	beaches	unseen	beaches	unknown	beaches
10	170	4962	94825	0	43	42	0	0	43	42
9	160	4784	95017	3	36	36	0	0	36	36
8	123	4869	94962	8	38	37	0	0	38	37
7	110	4819	95037	0	34	33	0	0	34	33
6	92	4871	94987	0	50	47	0	0	50	47
5	57	4893	95012	0	38	36	0	0	38	36
4	76	4913	94990	0	21	21	0	0	21	21
3	65	4883	95044	0	8	8	0	0	8	8
2	50	4841	63618	0	45	35	31446	3367	31491	3402
1	34	4933	23925	0	9	8	71099	7561	71108	7569

Accumulating tidal data for 10 rows

row	rare	uncommon	common	heads	combed	beaches	unseen	beaches	unknown	beaches
10	170	4962	94825	0	43	42	0	0	43	42
9	330	9746	189842	3	79	75	0	0	79	75
8	453	14615	284804	11	117	110	0	0	117	110
7	563	19434	379841	11	151	138	0	0	151	138
6	655	24305	474828	11	201	175	0	0	201	175
5	712	29198	569840	11	239	203	0	0	239	203
4	788	34111	664830	11	260	216	0	0	260	216
3	853	38994	759874	11	268	223	0	0	268	223
2	903	43835	823492	11	313	253	31446	3367	31759	3566
1	937	48768	847417	11	322	260	102545	7561	102867	7692

That last table is cumulative:

tides = 4 -> rows 5-10 are visible
tides = 3 -> rows 4-10 are visible
tides = 2 -> rows 3-10 are visible
tides = 1 -> rows 2-10 are visible
tides = 0 -> rows 1-10 are visible

Which means that when tides are at 3 (like today), spading only needs to look at 217 beaches to look for the 241 ties which have only been seen as combed on those particular beaches.

I'm about to code that up: is you do "BeachComber spade all", it will determine what the tides are and will visit ONLY the list of beaches with combed and/or unseen tiles at that level.

This means I will only need my whole army of spading multis for tides = 0 or 1.
By and by, I should be able to get by on using two multies:
one to spade the (few remaining) beaches with combed tiles
one to visit unverifed rares.

 

[Veracity]

Revision 30 is now out.

1) I have now combed through tides = 1, 0, 1 and here is the current state of my spading:

row	rare	uncommon	common	head	castle	#	combed	#	unseen	#	unknown	#
10	171	4971	94825	0	0	0	33	33	0	0	33	33
9	160	4793	95017	3	0	0	27	27	0	0	27	27
8	123	4876	94962	8	0	0	31	30	0	0	31	30
7	110	4822	94945	0	93	84	30	29	0	0	30	29
6	92	4884	94991	0	0	0	33	32	0	0	33	32
5	57	4898	95012	0	0	0	33	32	0	0	33	32
4	76	4918	94794	0	196	156	16	16	0	0	16	16
3	65	4883	95049	0	0	0	3	3	0	0	3	3
2	51	4841	95101	0	0	0	7	7	0	0	7	7
1	34	4934	47931	0	222	177	11	10	46868	4974	46879	4984

Accumulating tidal data for 10 rows

row	rare	uncommon	common	head	castle	#	combed	#	unseen	#	unknown	#
10	171	4971	94825	0	0	0	33	33	0	0	33	33
9	331	9764	189842	3	0	0	60	59	0	0	60	59
8	454	14640	284804	11	0	0	91	87	0	0	91	87
7	564	19462	379749	11	93	84	121	116	0	0	121	116
6	656	24346	474740	11	93	84	154	143	0	0	154	143
5	713	29244	569752	11	93	84	187	170	0	0	187	170
4	789	34162	664546	11	289	164	203	184	0	0	203	184
3	854	39045	759595	11	289	164	206	187	0	0	206	187
2	905	43886	854696	11	289	164	213	193	0	0	213	193
1	939	48820	902627	11	511	180	224	202	46868	4974	47092	5108

I have looked at all beaches with tides 1 or higher.
With tides at 0 - which happens one day out of 8 - I have not inspected 4974 beaches.
I should be able to set all of my multis doing "BeachComber spade all" on the next two tides=0 days (Dec 8 & 16) and end up with only combed tiles - not unseen - left to spade.

BeachComber spade all will visit unseen tiles first and, if not remain in the database, combed.
BeachComber all rare unverified twinles will try to verify "known" rare.

Those will be the only "spading" things required.

I am sure that Irrat is ahead of me, here, but between the two of use, we WILL finish mapping the entire beach.

2) Note that I originally collected sand castle tiles, but thought they were unnecessary - but I'm doing it again.
Notice that sand castles are all in rows 7, 4, and 1. That's why it took so long to verify the beaches; row 1 is difficult.

3) I also added a couple of commands:

> BeachComberUtilities minutes=9375

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

minutes	rare	uncommon	common	head	castle	combed	unseen	unknown
9375	1	5	92	0	2	0	0	0

> BeachComberUtilities minutes=9375 verbose

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

Beach 9375

row	rare	uncommon	common	head	castle	combed	unseen	unknown
10	1	0	9	0	0	0	0	0
9	0	0	10	0	0	0	0	0
8	0	1	9	0	0	0	0	0
7	0	0	10	0	0	0	0	0
6	0	0	10	0	0	0	0	0
5	0	2	8	0	0	0	0	0
4	0	0	9	0	1	0	0	0
3	0	1	9	0	0	0	0	0
2	0	1	9	0	0	0	0	0
1	0	0	9	0	1	0	0	0

4) I have discovered some new rare tiles and verified a bunch more.
I have verified 408 out of the 939 "published" rares.
Slow process...

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

Known rare tiles: 939
Locally discovered rare tiles: 47
Erroneous rare tiles: 1

Verified rare tiles: 408
Newly verified rare tiles: 0
Total: 408

Known uncommon tiles: 48820
New uncommon tiles: 0
Total: 48820

Known common tiles: 902627
New common tiles: 0
Total: 902627

Known combed tiles: 224

Known sand castle tiles: 511
New sand castle tiles: 0
Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 1
Last minutes down the beach spaded: 2311

Beaches with rare tiles: 894
Beaches with verified rare tiles: 400
Beaches with uncommon tiles: 9944
Beaches with common tiles: 10000
Beaches with combed tiles: 202
Beaches with sand castle tiles: 180
Beaches with sand castles: 180
Beaches with beach heads: 11
Beaches with unvisited twinkles: 1

 

[Irrat]

Huh, I missed the notifications.

You're further ahead than me for the rare tiles funny enough. I've found only 335, and that's including the hoarded ones.

Sand: 949,500
Uncommon Sparkle: 48,548
Unknown tiles: 1,094
Combed: 1,058
Castle: 512
Rare Sparkle: 335
Unknown tiles not on rare beach: 38
Mystery Sparkle: 36
Head: 11

I have about 5 tiles that might be sand, but that looks like a nice clean number. Combo will comb the first tile in the beach if there's no good targets.
My % of spading has slowed down drastically unfortunately. Today I spent 724 turns visiting beaches, only 7 of those beaches had a tile I could look at.
I've adjusted my script so now it'll only hit the known beaches once a day, and the unknown beaches hourly..
I really want to finish unhoarding those beaches so I don't have to worry about fighting other combers for potential rares

 

[fronobulax]

In the realm of inconsistencies that hurt no one, the script said "You found a cursed pirate hoard!" but data\beach\html\README.txt says "Combing a pirate treasure trove". Or perhaps there are many types of pirate treasure trove and I don't know about the others?

Less trivial, the resulting HTML cannot be displayed directly by my browser. That is consistent with my own experience of saving pages for debugging or testing - there are lines present in the file that must be deleted before it will be rendered. This is not an action item as much as a "heads up" for other users who might not be prepared to edit the saved file.

And thank you again for writing and maintaining this.

 

[Veracity]

1) Irrat published a new data set for combo. I looked at it and noticed a fair number that I had published - and the error I found was deleted. Perhaps this came from my data or was independently found; either is fine.

I added the new tiles I had not seen to tiles.rare.json
I did not add them to tiles.verified.json, since I have not seen them personally, yet. Sorry.

2) Today I send my army of spading multis to take a look at unspaded beaches at tides = 0.

BeachComber spade no longer cycles sequentially over all beaches, but looks first at:
- beaches we have not seen at the current tide level
- beaches with combed tiles

Since there are more of the former than turns I had available, it was all "unseen" beaches.
(I aso deleted spade.minutes.txt, since that is no longer needed.)

After merging the newly discovered stuff in with my existing data:

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

Known rare tiles: 978
Locally discovered rare tiles: 50
Erroneous rare tiles: 1

Verified rare tiles: 425
Newly verified rare tiles: 0
Total: 425

Known uncommon tiles: 48866
New uncommon tiles: 0
Total: 48866

Known common tiles: 930796
New common tiles: 0
Total: 930796

Known combed tiles: 144

Known sand castle tiles: 511
New sand castle tiles: 0
Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 1

Beaches with rare tiles: 930
Beaches with verified rare tiles: 416
Beaches with uncommon tiles: 9944
Beaches with common tiles: 10000
Beaches with combed tiles: 128
Beaches with sand castle tiles: 180
Beaches with sand castles: 180
Beaches with beach heads: 11
Beaches with unvisited twinkles: 1

Here is my "completeness" report:

row	rare	uncommon	common	head	castle	#	combed	#	unseen	#	unknown	#
10	181	4977	94825	0	0	0	17	17	0	0	17	17
9	167	4794	95017	3	0	0	19	19	0	0	19	19
8	129	4882	94962	8	0	0	19	18	0	0	19	18
7	114	4828	94945	0	93	84	20	19	0	0	20	19
6	96	4895	94991	0	0	0	18	18	0	0	18	18
5	58	4909	95012	0	0	0	21	21	0	0	21	21
4	78	4922	94794	0	196	156	10	10	0	0	10	10
3	66	4884	95049	0	0	0	1	1	0	0	1	1
2	52	4841	95105	0	0	0	2	2	0	0	2	2
1	37	4934	76096	0	222	177	17	15	18694	1980	18711	1995

Accumulating tidal data for 10 rows

row	rare	uncommon	common	head	castle	#	combed	#	unseen	#	unknown	#
10	181	4977	94825	0	0	0	17	17	0	0	17	17
9	348	9771	189842	3	0	0	36	35	0	0	36	35
8	477	14653	284804	11	0	0	55	51	0	0	55	51
7	591	19481	379749	11	93	84	75	70	0	0	75	70
6	687	24376	474740	11	93	84	93	86	0	0	93	86
5	745	29285	569752	11	93	84	114	103	0	0	114	103
4	823	34207	664546	11	289	164	124	112	0	0	124	112
3	889	39091	759595	11	289	164	125	113	0	0	125	113
2	941	43932	854700	11	289	164	127	115	0	0	127	115
1	978	48866	930796	11	511	180	144	128	18694	1980	18838	2092

In 8 days, when tides are at 0 again, I will send the army to look at the 1980 beaches I have not seen row 1 of.

Henceforth, spading will visit only beaches with unidentified combed tiles.

Since I only know of 144 combed tiles on 128 beaches, that will be easily accomplished by a single multi.
Perhaps several times a day, to look for twinkles that have regenerated.

Also, there is still attempting verify the rest of the rares.
There are more unverified rares than I generate via consumption with any single character - and it takes a long time for rares to regenerate, and the universe of combo users is compting ("interfering") with my spading, so it will take a while, I expect.

I'll make another data dump in 8 days.

 

[Veracity]

In the realm of inconsistencies that hurt no one, the script said "You found a cursed pirate hoard!" but data\beach\html\README.txt says "Combing a pirate treasure trove". Or perhaps there are many types of pirate treasure trove and I don't know about the others?

I need to update README. I have made a LOT of changes - and added additional files, like BeachComberUtilities.ash, which have their own parameterized command lines.

Less trivial, the resulting HTML cannot be displayed directly by my browser. That is consistent with my own experience of saving pages for debugging or testing - there are lines present in the file that must be deleted before it will be rendered. This is not an action item as much as a "heads up" for other users who might not be prepared to edit the saved file.

Have you tried test showhtml data/beach/html/FILENAME?

That will pop up a mini-browser window. It doesn't render as well as a real browser, but the images and scripts on KoL's image servers are all available.

I have no idea what "lines must be deleted before it will be rendered". Can you elaborate?

And thank you again for writing and maintaining this.

I am definitely running out of new features. I still want to accumulate results for each character individually - in addition to combining them in a total results file. That's only an issue if you have more than one character combing the beach.

 

[fronobulax]

test showhtml data/beach/html/FILENAME worked as expected - cursed tricorn hat, some pork gems and some meat.

Double clicking on the file to display it in my default browser generated an error "ERR_FILE_NOT_FOUND" but the address bar included "/dist/data/game.php"

Deleting the lines

<!--
if (parent.frames.length == 0) location.href="game.php";
top.charpane.location.href="charpane.php";
//-->

from where they occur at the top of the files results in a file that my browser displays.

Those lines have caused me problems in preparing HTML for use in tests which I solve by deleting them but that may be a problem of my own making

 

[Veracity]

Revision 33 has the following changes:

1) Irrat submitted a package of new rares to combo.
11 of them were new to me, so I added them to tiles.rare.json

2) I have now looked at all beaches with tides = 0.

> BeachComberUtilities complete

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 991
Total uncommons = 48911
Total commons = 949499
Total beach heads = 11
Total sand castles = 512
Total combed = 76
Total: 1000000

row	rare	uncommon	common	head	castle	#	combed	#
10	183	4985	94825	0	0	0	7	7
9	170	4800	95017	3	0	0	10	10
8	130	4889	94962	8	0	0	11	11
7	117	4835	94945	0	93	84	10	10
6	97	4902	94991	0	0	0	10	10
5	60	4911	95012	0	0	0	17	17
4	79	4924	94794	0	196	156	7	7
3	66	4884	95049	0	0	0	1	1
2	52	4841	95105	0	0	0	2	2
1	37	4940	94799	0	223	178	1	1

Accumulating tidal data for 10 rows

row	rare	uncommon	common	head	castle	#	combed	#
10	183	4985	94825	0	0	0	7	7
9	353	9785	189842	3	0	0	17	17
8	483	14674	284804	11	0	0	28	27
7	600	19509	379749	11	93	84	38	37
6	697	24411	474740	11	93	84	48	46
5	757	29322	569752	11	93	84	65	61
4	836	34246	664546	11	289	164	72	68
3	902	39130	759595	11	289	164	73	69
2	954	43971	854700	11	289	164	75	71
1	991	48911	949499	11	512	180	76	72

I no longer have "unseen" or "unknown" columns.
That's not quite right, since this race condition exists:

I parse the beach and see that a formerly combed tile is now a twinkle.
I remove it from the list of combed tiles.
I visit another twinkle and when I look at the beach again, the previously combed tile is now combed again
I don't re-add it to the list of combed tiles. Instead, I save it as an unvisited twinkle.

I'll think about this.

Notice that there are a total of 76 tiles on 72 beaches that are unknown.
Ergo, BeachComber spade all will visit 72 beaches - and comb up to 76 times, in total.

Even with Crimbo now underway, I should be able to manage that at least once a day.

3) Here's the current summary:

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Known rare tiles: 991
Locally discovered rare tiles: 52
Erroneous rare tiles: 1

Verified rare tiles: 448
Newly verified rare tiles: 0
Total: 448

Known uncommon tiles: 48911
New uncommon tiles: 0
Total: 48911

Known common tiles: 949499
New common tiles: 0
Total: 949499

Known sand castles: 512
New sand castles: 0

Beach Heads: 11

Known combed tiles: 76

Beaches with rare tiles: 942
Beaches with verified rare tiles: 439
Beaches with uncommon tiles: 9946
Beaches with common tiles: 10000
Beaches with sand castles: 180
Beaches with beach heads: 11

- I discovered a couple more hoarded rares.
- I have verified 46.6% of the "known" rares.
- There remain 553 unverified rares. That is more than my cheap diet generates per day, BeachComber all rare unverified twinkles makes slow and steady progress.
- I have seen all 512 sand castles that Irrat has seen. That's a nice round number.

> BeachComberUtilities uncommon

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data
54 beaches have no known uncommons.
4 beaches with no uncommons have a verified rare tile.
0 beaches with no uncommons have an unverified rare tile.
50 beaches with no uncommons have no known rare tile.
0 beaches with no uncommons have an unknown combed tile.

No conclusions yet.

Speaking of Crimbo, I am finally ready to start spending turns on it for the first time.
I wanted to finish examining row 1, so today is mostly shot, but I'll be diving in tomorrow.

 

[Irrat]

Sand: 949,500
Uncommon Sparkle: 48,864
Unknown tiles: 613
Combed: 594
Castle: 512
Rare Sparkle: 500
Mystery Sparkle: 19
Head: 11
Unknown tiles not on rare beach: 4

This is only data that was actually visited, so it's missing quite a bit of the known rares. I'm 99% sure that the "unknown tiles" are actually rares. So yeah. 4 rares for sure.
I feel pretty confident in saying there's 1000 total rares, and 2 of the rares are on beaches with a rare already marked. Or it could be on those 4 beaches I haven't finished yet, I don't feel like adding code just to check that.
The sand looks like a nice number, so we can probably determine the amount of uncommon tiles just by adding all the other numbers together and finding the remaining unaccounted for after that is done.
Uncommon = 1,000,000 - (Sand + Rares + Castles + Heads)

 

[Veracity]

Having been through tides = 0 again, here's the current state of my data (revision 34).
Executive summary:

26 combed tiles on 24 beaches (so BeachComber spade all will only visit 24 beaches).
478 out of 998 rares verified (so BeachComber all rare unverified twinkles wants to visit 520 tiles on 510 beaches).

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Known rare tiles: 998
Locally discovered rare tiles: 52
Erroneous rare tiles: 1

Verified rare tiles: 478
Newly verified rare tiles: 0
Total: 478

Known uncommon tiles: 48953
New uncommon tiles: 0
Total: 48953

Known common tiles: 949500
New common tiles: 0
Total: 949500

Beach Heads: 11

Known sand castles: 512
New sand castles: 0

Known combed tiles: 26

Beaches with rare tiles: 949
Beaches with verified rare tiles: 466
Beaches with uncommon tiles: 9946
Beaches with common tiles: 10000
Beaches with beach heads: 11
Beaches with sand castles: 180
Beaches with combed tiles: 24

Detailed analysis:

> BeachComberUtilities complete

Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 998
(Unverified rares = 520)
Total uncommons = 48953
Total commons = 949500
Total beach heads = 11
Total sand castles = 512
Total combed = 26
Total unseen = 0
Total: 1000000

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	113	112	4990	94825	0	0	0	2	2
9	172	108	107	4806	95017	3	0	0	2	2
8	131	74	74	4894	94962	8	0	0	5	5
7	117	69	69	4842	94945	0	93	84	3	3
6	99	68	68	4907	94991	0	0	0	3	3
5	60	40	40	4920	95012	0	0	0	8	8
4	79	31	31	4929	94794	0	196	156	2	2
3	66	5	5	4884	95049	0	0	0	1	1
2	54	6	6	4841	95105	0	0	0	0	0
1	37	6	6	4940	94800	0	223	178	0	0

Accumulating tidal data for 10 rows

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	113	112	4990	94825	0	0	0	2	2
9	355	221	219	9796	189842	3	0	0	4	4
8	486	295	293	14690	284804	11	0	0	9	8
7	603	364	360	19532	379749	11	93	84	12	11
6	702	432	425	24439	474740	11	93	84	15	14
5	762	472	465	29359	569752	11	93	84	23	21
4	841	503	493	34288	664546	11	289	164	25	23
3	907	508	498	39172	759595	11	289	164	26	24
2	961	514	504	44013	854700	11	289	164	26	24
1	998	520	510	48953	949500	11	512	180	26	24

Shouldn't be TOO long before I can visit all the "combed" tiles (at which point I will remove those columns), but it will be a slog to verify all the remaining rares.

I'm leaving town from Jan 6 to mid-March and will not have access to my KoLmafia dev environment, but I should be able to run turns (and thus finish my Standard 2024 ascensions) and continue verifying more rares, at least.

 

[Irrat]

I've added the last rares to combo, which is a total of 998 rares.

There's a chance the last 2 rares are just not available, having a castle or beach head on top of them.
I think someone may have already spaded all rares too and said as such.

Otherwise, there's 26 tiles on a rare beach that I don't know, that are not marked as a rare. It could be 2 of those. I need to do a writeup somewhere.

 

[Veracity]

Here are two tiles I have in tiles.rare.json that you do not have in raretiles.json:

  { "minute": 2065, "row": 6, "column": 9 },
  { "minute": 3920, "row": 5, "column": 4 },

What do you have for them?

Here are session logs where I combed those tiles:

[38485] Wandering 2065 minutes down the beach
Encounter: Comb the Beach (2065 minutes down the beach)
> 4 squares in beach 2065 contain combed sand
> 85 squares in beach 2065 contain rough sand
> 1 squares in beach 2065 contain rough sand with a twinkle
> 173 rare tiles have already been verified
> 664 unverified rare tiles are candidates for combing
> Combing the square at coordinates (2065,6,10) which contains rough sand with a twinkle
Combing square 6,10 (2065 minutes down the beach)
You acquire an item: piece of driftwood
> (2065,6,10) is a 'rare' tile.

[189080] Wandering 3920 minutes down the beach
Encounter: Comb the Beach (3920 minutes down the beach)
> 5 squares in beach 3920 contain combed sand
> 84 squares in beach 3920 contain rough sand
> 1 squares in beach 3920 contain rough sand with a twinkle
> Combing the square at coordinates (3920,5,5) which contains rough sand with a twinkle
Combing square 5,5 (3920 minutes down the beach)
You acquire an item: piece of driftwood
> (3920,5,5) is a 'rare' tile.

You have one tile in raretiles.json that I have in tiles.combed.json:

  { "minute": 1105, "row": 8, "column": 9 },

If I move that to tiles.rare.json (but not tiles.rare.verified.json, since I haven't seen it yet ) I currently have this:

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Known rare tiles: 1000
Locally discovered rare tiles: 52
Erroneous rare tiles: 1

Verified rare tiles: 478
Newly verified rare tiles: 48
Total: 526

Known uncommon tiles: 48953
New uncommon tiles: 16
Total: 48969

Known common tiles: 949500
New common tiles: 0
Total: 949500

Beach Heads: 11

Known sand castles: 512
New sand castles: 0

Known combed tiles: 8

Beaches with rare tiles: 951
Beaches with verified rare tiles: 513
Beaches with uncommon tiles: 9946
Beaches with common tiles: 10000
Beaches with beach heads: 11
Beaches with sand castles: 180
Beaches with combed tiles: 8

I'll release this today.

I have 8 tiles that are not known to be rare, uncommon, or common that I have only seen as combed.
I have 474 tiles listed as rare that I have not yet verified. I.e., have only seen as combed.

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	104	103	4992	94825	0	0	0	0	0
9	355	199	197	9799	189842	3	0	0	1	1
8	488	270	268	14695	284804	11	0	0	2	2
7	605	337	334	19539	379749	11	93	84	3	3
6	704	397	391	24448	474740	11	93	84	4	4
5	764	434	428	29374	569752	11	93	84	6	6
4	843	464	456	34304	664546	11	289	164	7	7
3	909	469	461	39188	759595	11	289	164	8	8
2	963	471	463	44029	854700	11	289	164	8	8
1	1000	474	466	48969	949500	11	512	180	8	8

 

[Irrat]

For both those beaches, I have "combed"
So yeah, that'd be the last two rares!

 

[Veracity]

I did a few more days of spading. Revision 35 has this:

We are down to 4 tiles which have only been seen as combed and are not unverified rares.
We are also down to 553 verified and 447 unverified rares.

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Known rare tiles: 1000
Locally discovered rare tiles: 52
Erroneous rare tiles: 1

Verified rare tiles: 553
Newly verified rare tiles: 0
Total: 553

Known uncommon tiles: 48973
New uncommon tiles: 0
Total: 48973

Known common tiles: 949500
New common tiles: 0
Total: 949500

Beach Heads: 11

Known sand castles: 512
New sand castles: 0

Known combed tiles: 4

Beaches with rare tiles: 951
Beaches with verified rare tiles: 539
Beaches with uncommon tiles: 9946
Beaches with common tiles: 10000
Beaches with beach heads: 11
Beaches with sand castles: 180
Beaches with combed tiles: 4

> BeachComberUtilities complete

Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 1000
(Unverified rares = 447)
Total uncommons = 48973
Total commons = 949500
Total beach heads = 11
Total sand castles = 512
Total combed = 4
Total unseen = 0
Total: 1000000

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	97	96	4992	94825	0	0	0	0	0
9	172	91	91	4808	95017	3	0	0	0	0
8	133	67	67	4897	94962	8	0	0	0	0
7	117	61	61	4844	94945	0	93	84	1	1
6	99	56	56	4910	94991	0	0	0	0	0
5	60	36	36	4927	95012	0	0	0	1	1
4	79	29	29	4930	94794	0	196	156	1	1
3	66	5	5	4884	95049	0	0	0	1	1
2	54	2	2	4841	95105	0	0	0	0	0
1	37	3	3	4940	94800	0	223	178	0	0

Accumulating tidal data for 10 rows

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	97	96	4992	94825	0	0	0	0	0
9	355	188	187	9800	189842	3	0	0	0	0
8	488	255	254	14697	284804	11	0	0	0	0
7	605	316	314	19541	379749	11	93	84	1	1
6	704	372	367	24451	474740	11	93	84	1	1
5	764	408	403	29378	569752	11	93	84	2	2
4	843	437	430	34308	664546	11	289	164	3	3
3	909	442	435	39192	759595	11	289	164	4	4
2	963	444	437	44033	854700	11	289	164	4	4
1	1000	447	440	48973	949500	11	512	180	4	4

 

[Veracity]

Revision 36 does this:

1) 3 of the 4 "combed" tiles are "uncommon", as expected.
2) 65 more "rare" tiles have been verified.

> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Known rare tiles: 1000
Locally discovered rare tiles: 52
Erroneous rare tiles: 1

Verified rare tiles: 618
Newly verified rare tiles: 0
Total: 618

Known uncommon tiles: 48976
New uncommon tiles: 0
Total: 48976

Known common tiles: 0
New common tiles: 0
Total: 0

Beach Heads: 11

Known sand castles: 512
New sand castles: 0

Known combed tiles: 1

Beaches with rare tiles: 951
Beaches with verified rare tiles: 599
Beaches with uncommon tiles: 9946
Beaches with common tiles: 0
Beaches with beach heads: 11
Beaches with sand castles: 180
Beaches with combed tiles: 1

I need to visit 1 (!) "uncommon" tile I have only see as "combed"
I need to verify 382 "rare" tiles I have only seen as combed.

> BeachComberUtilities complete

Loading rare tile data...
Loading verified rare tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Loading combed tile data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 1000
(Unverified rares = 382)
Total uncommons = 48976
Total commons = 949500
Total beach heads = 11
Total sand castles = 512
Total combed = 1
Total unseen = 0
Total: 1000000

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	87	86	4992	94825	0	0	0	0	0
9	172	78	78	4808	95017	3	0	0	0	0
8	133	54	54	4897	94962	8	0	0	0	0
7	117	50	50	4845	94945	0	93	84	0	0
6	99	47	47	4910	94991	0	0	0	0	0
5	60	33	33	4928	95012	0	0	0	0	0
4	79	23	23	4931	94794	0	196	156	0	0
3	66	5	5	4884	95049	0	0	0	1	1
2	54	2	2	4841	95105	0	0	0	0	0
1	37	3	3	4940	94800	0	223	178	0	0

Accumulating tidal data for 10 rows

row	rare	(unverified	#)	uncommon	common	head	castle	#	(combed	#)
10	183	87	86	4992	94825	0	0	0	0	0
9	355	165	164	9800	189842	3	0	0	0	0
8	488	219	218	14697	284804	11	0	0	0	0
7	605	269	267	19542	379749	11	93	84	0	0
6	704	316	312	24452	474740	11	93	84	0	0
5	764	349	345	29380	569752	11	93	84	0	0
4	843	372	366	34311	664546	11	289	164	0	0
3	909	377	371	39195	759595	11	289	164	1	1
2	963	379	373	44036	854700	11	289	164	1	1
1	1000	382	376	48976	949500	11	512	180	1	1

Once I get that last "combed" tile, I will remove those columns and change BeachComber spade to verify the rest of the rare tiles.
There are few enough of those that my spading multi can visit all of them each day with a cheap diet.

I notice that combo only has 994 rare tiles.
I just opened a PR to share the last 6 tiles - with proof from my session logs.
Once that is merged, both BeachComber and combo will have all 1000 rares.

Combo started with 849 rares.
I discovered 1 error and 52 previously "hoarded" rares.
Irrat's bot discovered another 100 (before I did; I have subsequently "verified" many of them, so would have discovered them myself, soon enough. ).

Between the two of us, there are no more "hoarded" rares.

I'll make a general announcement to that effect once I finish verifying the last 382 rares I have not personally seen.
I'm getting, oh, 3 per day or so, so it will be several months yet.
Probably more; I've looked for that last combed "uncommon" scores of times, and uncommons regenerate much faster than rares.

 

[Irrat]

Actually that's weird.

My current rares file has 1000 rares in it, but 2065 is the only tile that I don't know that isn't in combo as a rare... Which is the one you've marked as a rare.

On the other hand.
{ "minute": 9063, "row": 3, "column": 6 },

This is a rare.
So that's 1001 rares?.. Probably another false rare somewhere.

Sand: 949,500
Uncommon Sparkle: 48,976
Rare Sparkle: 775
Castle: 512
Unknown tiles: 226
Combed: 219
Head: 11
Mystery Sparkle: 7
Unknown not rare tiles: 1

I have 226 tiles left to spade.
I was waiting for that tile to be found, because it would be the "last non-rare".
But now it seems like there's a false rare somewhere.