BeachComber - fast and efficient beach combing

Veracity

Developer
Staff member
Regarding the error, my session logs say:

Code:
(9077,2,9) is an 'uncommon' tile.
(9077,3,10) is an 'uncommon' tile.
(9077,8,9) is a 'unknown' tile.
(9077,8,10) is an 'uncommon' tile.
(9077,9,4) is an 'uncommon' tile.
(9077,10,2) is an 'uncommon' tile.
(9077,10,5) is a 'uncommon' tile.
(9077,10,6) is an 'uncommon' tile.
(9077,10,10) is an 'uncommon' tile.

Note that columns go from 1-10 - as logged in the session log - but as reported by KoLmafia, they are zero-indexed, and in my (and combo's) JSON data, they go from 0-9.

Looking at that session log:

Code:
Encounter: Comb the Beach (9077 minutes down the beach)
> 1 squares in beach 9077 contain combed sand
> 65 squares in beach 9077 contain rough sand
> 4 squares in beach 9077 contain rough sand with a twinkle
> Combing the square at coordinates (9077,8,9) which contains rough sand with a twinkle
Combing square 0,1 (0 minutes down the beach)
> (9077,8,9) is a 'unknown' tile.
I did not like that logging, so I fixed it and tried again.

Code:
Encounter: Comb the Beach (9077 minutes down the beach)
> 1 squares in beach 9077 contain combed sand
> 65 squares in beach 9077 contain rough sand
> 4 squares in beach 9077 contain rough sand with a twinkle
> Combing the square at coordinates 10,90765 which contains rough sand with a twinkle
Combing square 10,6 (9077 minutes down the beach)
You acquire an item: dull fish scale
> (9077,10,5) is a 'uncommon' tile.
I did not like that logging, so I fixed it and tried again.

Code:
Encounter: Comb the Beach (6270 minutes down the beach)
> 68 squares in beach 6270 contain rough sand
> 2 squares in beach 6270 contain rough sand with a twinkle
> Combing the square at coordinates 5,62696 which contains rough sand with a twinkle
Combing square 5,5 (6270 minutes down the beach)
You acquire an item: sand dollar
> (6270,5,5) is a 'uncommon' tile.
That was good and was how I logged it henceforth.

You found the single tile that got logged - and stored - incorrectly. :)
I will remove (9077,10,5) from the uncommons set.
Thanks!

I'm about to add more stuff for spading - which will not be used if you are simply looking for rares, like combo:
commons (rough sand and sand castle)
combed (unknown)
heads (well, these are static, but what the heck.)

Since there are about 950,000 commons, this is data bloat for people (not me) who are not actively spading.

Over the next 5 days or so, I'll have identified the vast majority of commons.
Subsequent passes will go only to beaches where I have seen (not yet identified) combed tiles.

And combed will be exclude things that I, personally, have combed, and have identified as rare, uncommon, or common.
That will eventually be the equivalent of what you are doing, in that all unknowns will come out of the combed set.

Did you want me to make a PR to add any new tiles to your lists?

Sure, thank you. That will be very helpful!

Regarding whales, I have seen 5 of them in the last 3 months.
I have seen both whales and normal rares in the same tile.
I have no reason to think that they only appear in certain rare tiles; I expect KoL randomizes what you get in a rare tile.

Code:
Encounter: Comb the Beach (4067 minutes down the beach)
> 3 squares in beach 4067 contain combed sand
> 76 squares in beach 4067 contain rough sand
> 1 squares in beach 4067 contain rough sand with a twinkle
> Combing the square at coordinates (4067,3,9) which contains rough sand with a twinkle
Combing square 3,9 (4067 minutes down the beach)
You acquire an item: piece of driftwood
> (4067,3,9) is a 'rare' tile.
and
Code:
Encounter: Comb the Beach (4067 minutes down the beach)
> 1 squares in beach 4067 contain a beached whale
> 2 squares in beach 4067 contain combed sand
> 77 squares in beach 4067 contain rough sand
> You found a beached whale!
> Saving page HTML to Veracity_4067_3_9_20230823234110454.html
> Combing the square at coordinates (4067,3,9) which contains a beached whale
Combing square 3,9 (4067 minutes down the beach)
You gain 10,723,919 Meat
> (4067,3,9) is a 'rare' tile.
> Saving page HTML to Veracity_4067_3_9_20230823234110636.html
 

Irrat

Member
Hmm, comparing both datasets. We have 41 beaches that likely have a hoarded rare we're unaware of yet.
Number reached by filtering out the minutes that contain a known rare, and the minute contains a combed or unknown sparkle as per my own dataset.

I have about 900 beaches with unknown tiles left, including known rares as my dataset doesn't include anything it hasn't found itself. Only roughly 3% of the turns are finding a tile to spade now. As the number goes down, I expect to spade more hoarded tiles as this is a standalone script that runs on a cronjob, and will visit the same beach multiple times a day as more adventures are remaining left over as beaches are crossed off.

As for whales, yeah. I left it as a special tile instead of a rare cos I wasn't 100% if it was a normal rare and found it cute in the data.
 

Veracity

Developer
Staff member
Thanks for the PR. I accepted & merged it.

I have added detection of commons - rough sand (if you have TwinkleVision™) and sand castles - from the beach map returned by KoLmafia.
I also save combed tiles - if they are not known rares, uncommons, or commons.
Seeing twinkles, rough sand, sand castles, and beached whales from that same beach map removes them from previously seen combed tiles.

I populated the set of commons with those I combed myself over the last 3 months of spading; BeachComberFix scraped the session logs of all my characters and extracted the common tiles.

I estimate there will end up being about 950,000 commons, so this is a big file - and it takes noticeable time to load at startup, but in memory, it is in a much faster data structure for searching, so it is not noticeably slower to visit beaches and comb tiles.

I've only been saving commons and combed tiles for a couple days.

Here's the current data:

Code:
> BeachComber data

Known rare tiles: 935
Locally discovered rare tiles: 44
Erroneous rare tiles: 1

Verified rare tiles: 352
Newly verified rare tiles: 8
Total: 360

Known uncommon tiles: 48732
New uncommon tiles: 11
Total: 48743

Known common tiles: 18159
New common tiles: 393167
Total: 411326

Known combed tiles: 341

Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 3
Last minutes down the beach spaded: 430

Beaches with rare tiles: 892
Beaches with verified rare tiles: 353
Beaches with uncommon tiles: 9944
Beaches with common tiles: 7935
Beaches with combed tiles: 271
Beaches with sand castles: 180
Beaches with unvisited twinkles: 3

I also souped up BeachComberUtilities to have 3 kinds of analysis.

As before:

Code:
> BeachComberUtilities uncommon

56 beaches have no known uncommons.
4 beaches with no uncommons have a verified rare tile.
2 beaches with no uncommons have an unverified rare tile.
50 beaches with no uncommons have no known rare tile.

This is with your new data.

Code:
> BeachComberUtilities combed

There are 341 unknown combed tiles on 271 beaches.

Ditto.

Code:
> BeachComberUtilities tides

There are 7935 beaches with observed commons.
1251 beaches have been spaded with tides = 0
977 beaches have been spaded with tides = 1
2542 beaches have been spaded with tides = 2
1750 beaches have been spaded with tides = 3
1412 beaches have been spaded with tides = 4
3 beaches have only seen higher rows

Note that those are not "fully spaded" beaches; as I said, I initialized the set of commons from my session logs, which have many beaches visited at lower tides.

After I've made a few passes over the whole tide cycle, I'll write another function which will determine which beaches have been fully spaded - known rares, uncommons, commons, beach head add up to 100 tiles.

It will take a while. You are definitely ahead of me. :)
 

Veracity

Developer
Staff member
I made (a bit more than) 1 pass through all 10000 minutes of the beach.
"A bit more" because I ended up with tides = 0 today, and looked at 2,000 or so beaches that way.

You have to enable "commons and "combed" tracking, since the tiles.common.jason is almost 35 MB (!).
Noticeably slow to read the file into a fast data structure in memory, but, once loaded, actual combing is still zippy.

Revision 29 has this:

Code:
> BeachComber data

Known rare tiles: 935
Locally discovered rare tiles: 45
Erroneous rare tiles: 1

Verified rare tiles: 352
Newly verified rare tiles: 26
Total: 378

Known uncommon tiles: 48732
New uncommon tiles: 20
Total: 48752

Known common tiles: 18159
New common tiles: 771607
Total: 789766

Known combed tiles: 525

Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 2
Last minutes down the beach spaded: 8488

And:

Code:
> BeachComberUtilities combed

There are 525 unknown combed tiles on 383 beaches.

> BeachComberUtilities tides

There are 10000 beaches with observed commons.
3052 beaches have been spaded with tides = 0
2336 beaches have been spaded with tides = 1
2002 beaches have been spaded with tides = 2
1475 beaches have been spaded with tides = 3
1135 beaches have been spaded with tides = 4
0 beaches have only seen higher rows

I am glad the tides never exceeded the maximum. :)

I'm going to do the following:

1) Write another utility to see how many beaches are "incomplete" at tides 4, 3, 2, 1, 0
That is to say, all visible tiles are common (rough sand or sand castles), uncommon (twinkles), rare (twinkle or whale), or beach head.

2) Given that, based on the tide, only look at beaches which are "incomplete" at that tide level.
There is no need to combed the beach at tide=4 if there are no unknown "combed" tiles to be discovered.

This will result in faster discovery - visiting only beaches with previously unidentified combed patches.
I think this is more-or-less the equivalent of Irrat's approach.

It will eventually (once I have seen all tiles at tides=0) result in a single character being able to visit all of the "incomplete" beaches each day.
 

Veracity

Developer
Staff member
Progress report:

I've added "completeness" checking based on all the beaches I've combed over the last three months at the various tidal levels. Had I been saving commons from the beginning, I'd be a lot closer, but tides = 0 happens once every 8 days and tides = 1 happens twice, so those are lagging.

Code:
> BeachComberUtilities complete

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading beach castle data...
Done loading tile data

Iterating over 10,000 beaches, 10 rows, 10 columns

Total rares = 937
Total uncommons = 48768
Total commons = 847417
Total heads = 11
Total combed = 322
Total unseen = 102545
Total: 1000000

rowrareuncommoncommonheadscombedbeachesunseenbeachesunknownbeaches
1017049629482504342004342
916047849501733636003636
812348699496283837003837
711048199503703433003433
69248719498705047005047
55748939501203836003836
47649139499002121002121
3654883950440880088
25048416361804535314463367314913402
134493323925098710997561711087569

Code:
Accumulating tidal data for 10 rows
rowrareuncommoncommonheadscombedbeachesunseenbeachesunknownbeaches
1017049629482504342004342
9330974618984237975007975
8453146152848041111711000117110
7563194343798411115113800151138
6655243054748281120117500201175
5712291985698401123920300239203
4788341116648301126021600260216
3853389947598741126822300268223
29034383582349211313253314463367317593566
1937487688474171132226010254575611028677692

That last table is cumulative:

tides = 4 -> rows 5-10 are visible
tides = 3 -> rows 4-10 are visible
tides = 2 -> rows 3-10 are visible
tides = 1 -> rows 2-10 are visible
tides = 0 -> rows 1-10 are visible

Which means that when tides are at 3 (like today), spading only needs to look at 217 beaches to look for the 241 ties which have only been seen as combed on those particular beaches.

I'm about to code that up: is you do "BeachComber spade all", it will determine what the tides are and will visit ONLY the list of beaches with combed and/or unseen tiles at that level.

This means I will only need my whole army of spading multis for tides = 0 or 1.
By and by, I should be able to get by on using two multies:
one to spade the (few remaining) beaches with combed tiles
one to visit unverifed rares.
 

Veracity

Developer
Staff member
Revision 30 is now out.

1) I have now combed through tides = 1, 0, 1 and here is the current state of my spading:

rowrareuncommoncommonheadcastle#combed#unseen#unknown#
101714971948250003333003333
91604793950173002727002727
81234876949628003130003130
7110482294945093843029003029
6924884949910003332003332
5574898950120003332003332
47649189479401961561616001616
365488395049000330033
251484195101000770077
13449344793102221771110468684974468794984

Accumulating tidal data for 10 rows

rowrareuncommoncommonheadcastle#combed#unseen#unknown#
101714971948250003333003333
933197641898423006059006059
84541464028480411009187009187
75641946237974911938412111600121116
66562434647474011938415414300154143
57132924456975211938418717000187170
4789341626645461128916420318400203184
3854390457595951128916420618700206187
2905438868546961128916421319300213193
19394882090262711511180224202468684974470925108

I have looked at all beaches with tides 1 or higher.
With tides at 0 - which happens one day out of 8 - I have not inspected 2974 beaches.
I should be able to set all of my multis doing "BeachComber spade all" on the next two tides=0 days (Dec 8 & 16) and end up with only combed tiles - not unseen - left to spade.

BeachComber spade all will visit unseen tiles first and, if not remain in the database, combed.
BeachComber all rare unverified twinles will try to verify "known" rare.

Those will be the only "spading" things required.

I am sure that Irrat is ahead of me, here, but between the two of use, we WILL finish mapping the entire beach. :)

2) Note that I originally collected sand castle tiles, but thought they were unnecessary - but I'm doing it again.
Notice that sand castles are all in rows 7, 4, and 1. That's why it took so long to verify the beaches; row 1 is difficult.

3) I also added a couple of commands:

Code:
> BeachComberUtilities minutes=9375

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

minutesrareuncommoncommonheadcastlecombedunseenunknown
9375159202000


Code:
> BeachComberUtilities minutes=9375 verbose

Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

Beach 9375
rowrareuncommoncommonheadcastlecombedunseenunknown
1010900000
9001000000
801900000
7001000000
6001000000
502800000
400901000
301900000
201900000
100901000

4) I have discovered some new rare tiles and verified a bunch more.
I have verified 408 out of the 939 "published" rares.
Slow process...

Code:
> BeachComber data

Loading tile data...
Loading rare tile data...
Loading verified tile data...
Loading uncommon tile data...
Loading common tile data...
Loading beach head data...
Loading sand castle data...
Done loading tile data

Known rare tiles: 939
Locally discovered rare tiles: 47
Erroneous rare tiles: 1

Verified rare tiles: 408
Newly verified rare tiles: 0
Total: 408

Known uncommon tiles: 48820
New uncommon tiles: 0
Total: 48820

Known common tiles: 902627
New common tiles: 0
Total: 902627

Known combed tiles: 224

Known sand castle tiles: 511
New sand castle tiles: 0
Known sand castle beaches: 180
New sand castle beaches: 0

Beach Heads: 11

Unvisited twinkle tiles: 1
Last minutes down the beach spaded: 2311

Beaches with rare tiles: 894
Beaches with verified rare tiles: 400
Beaches with uncommon tiles: 9944
Beaches with common tiles: 10000
Beaches with combed tiles: 202
Beaches with sand castle tiles: 180
Beaches with sand castles: 180
Beaches with beach heads: 11
Beaches with unvisited twinkles: 1
 
Top