BeachComber - fast and efficient beach combing

Veracity

Veracity

Developer
Staff member
I updated BeachComberFix to verify sand castle data.
It handles tiles when it sees 6399,3,1) is a 'common' tile.
It handles sand castles when it sees 2 squares in beach 6399 contain a sand castle
That program scraped my logs and determrined that I'd seen 172 beaches with sand castles.

I figured out how castles work

There are ten words.
Each word has its own stretch of beaches, separated by 11, 33, or 55 minutes, denoting dot, dash, end of letter.
The last letter of a word does not have an end of letter.
Instead, a gap greater than 55 means that the next word starts at the new beach.

For reference, here is the set of castles mapping to words:

Code: 
EVERY   (9375) 9364 (9309) 9298-9287-9276-9243 (9188) 9177 (9122) 9111-9078-9067 (9012) 8979*-8968-8935-8902
TWENTY  (8482) 8449 (8394) 8383-8350-8317 (8362*) 8251 (8196) 8163-8152 (8097) 8064 (8009) 7976-7965-7932-7899
SECOND  (7479) 7468-7457-7466 (7391) 7380 (7325) 7292-7281-7248-7237 (7182) 7149-7116-7083 (7028) 6995-6984 (6929) 6896-6885-6874
LETTER  (6454) 6443-6410-6399-6388 (6333) 6322 (6267) 6234 (6179) 6146 (6091) 6080 (6025) 6014-5981-5970
OF      (5550) 5517-5484-5451 (5396) 5385-5374-5341-5530
THE     (4910*) 4877* (4822) 4811-4800-4789-4778 (4723) 4712
BOTTLE  (4292) 4259-4248-4237-4226 (4171) 4138-4105*-4072* (4017) 3984 (3929) 3896 (3841) 3830-3797-3786-3775 (3720) 3709
MESSAGE (3289) 3256-3223 (3168) 3157 (3102) 3091-3080-3069 (3014) 3003-2992-2981 (2926) 2915-2882 (2827) 2794-2761-2750 (2695 2684
LOOPED  (2264) 2253-2220-2209-2198 (2143) 2110-2077-2044 (1989) 1956-1923-1890 (1835) 1824-1791-1758-1747 (1692) 1681 (1626) 1593-1582-1571
THRICE  (1151*) 1118* (1063) 1052-1041-1030-1019 (964) 953-920-909 (854) 843-832 (777) 744-733-700-689 (634) 623
Numbers in parenthesis are either the start of a new word or are a letter break.
Solo numbers or numbers separated by a dash are morse code for a single letter
Numbers with a * are sand castles I have not seen - 8 out of the required 180.
I hope to pick them up by and by, but they must exist; when I generate the beach sequence using the 10 specified word start beaches, I get something which encodes the input message as expected.

And when I insert those 8 messages into the sand castle beaches I have seen, it generates the exact same message.

Code: 
> BeachComberCastles

There are 180 beaches with sand castles
There are 180 castles encoding the puzzle hint
Encoded message = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'
Decoded message = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'
Decoded message from observed castles = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'

I encoded the message using the rules I described and decoded it - and got the identical message.
I decoded the sand castles I've seen (+8 other specific ones) - and got the same message.

I still hope to see the last 8 sand castles, by and by, but I believe sand castle spading is done.
 
Veracity

Veracity

Developer
Staff member
Revision 23 merges in all the sand castle stuff and also the data for another full traversal of the beach.

It's going a lot faster, now that I am (usually) finding either nothing new or at most one previously unknown twinkle.
I suspect what I am finding is uncommons on "rare" beaches that have been visited multiple times per day by combo users.
I am also making progress "verifying" rares, but that is slow going - for the same reason.

My main will be starting her series of smol runs, soon, and will not be trying to verify rare tiles, except in aftercore, before diving back into the next run. The multis have nothing better but traverse the whole beach, every 5 or 6 days, picking up the last unknown uncommons - and discovering new rares.

Beaches with uncommon tiles: 9892

I bet that all 10,000 beaches have uncommons - but see above comment.

Code: 
> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 18
Verified rare tiles: 186
Not previously verified rare tiles: 0
Known uncommon tiles: 44745
New uncommon tiles: 0
Known sand castle beaches: 180
New sand castle beaches: 0
Last minutes down the beach spaded: 748

Beaches with rare tiles: 823
Beaches with verified rare tiles: 184
Beaches with uncommon tiles: 9892
Beaches with sand castles: 180
Beaches with unvisited twinkles: 0
 
Veracity

Veracity

Developer
Staff member
My main has been looking for "rare unverified twinkles" - visit known "rare" tiles that we have not successfully visited before and, whether or not the rare tile is uncombed, visit any other "twinkle" in that section of the beach that we have not successfully visited before. Considering that combo (or BeachComber) will settle for a twinkle if the rare is already combed, those sections of the beach tend to be all rough sand or combed sand, so it's difficult to either verify a rare tile or discover a new uncommon tile.

A couple of days ago, a tile was added to tiles.rare.errors.json - which contains tiles that are in the list of "known" rares, but when we saw it as a twinkle and visited it, it turned out to be an uncommon.

Code: 
> Looking for rare tile at (4569,6,2)

[38796] Wandering 4569 minutes down the beach
Encounter: Comb the Beach (4569 minutes down the beach)
> 6 squares in beach 4569 contain combed sand
> 83 squares in beach 4569 contain rough sand
> 1 squares in beach 4569 contain rough sand with a twinkle
> 170 rare tiles have already been verified
> 334 unverified rare tiles are candidates for combing
> Combing the square at coordinates (4569,6,9) which contains rough sand with a twinkle
Combing square 6,9 (4569 minutes down the beach)
You acquire an item: dull fish scale
> 170 rare tiles have already been verified
> 333 unverified rare tiles are candidates for combing
> (4569,6,9) is an 'uncommon' tile.

This section of the beach supposedly has two rare tiles on it.
We went looking for one of them, but noticed that, although that one was combed, the other was a twinkle.
We combed it and discovered it was an uncommon tile.
We recorded the error in tiles.rare.errors.jso, but didn't say anything special in the log.

1) We should explicitly log that this was a rare "error" event.
2) It's probably not necessary to log the "already been verified" and "candidates for combing", although it makes it explicit that we are on a "verified" hunt - and it's satisfying to see verified go up and candidates go down. (candidates also goes down if we expected/hoped for a twinkle (or rough, if you don't have TwinkleVision™) and see a combed tile.
 
I

Irrat

Member
Oh hey, I didn't realize you were already doing this too.

I started on approx the 2nd of Sep.

So I wrote my own script from scratch and the goal of that was to be self sustaining with it sitting in 2crs for a cheap diet.
It's written as a standalone nodejs script which is half janky as it was made to work, not to be seen.

I did some minor work prior to that to just look at how often rares are resetting exactly, by refreshing every minute on a known rare spot.
I haven't looked at the data recently, but normal sand resets after 2-4 hours as known, but will reset 2-4 hours exactly after being combed. So it'll reset at 5:55 if you combed it at 2:55.

Similar to you, I didn't entirely trust the rare tiles list and we also know there's spots being kept secret. I'm skipping the known spots to focus instead on undiscovered and hoarded areas, though my overall goal is to generate a dataset telling us exactly what each tile is. My personal aim is to make beach combing as boring as possible by manipulating it with a crowdrun script that puts everyone not using said script; at a distinct disadvantage because we know where each rare is, when it should reset and communicate on when we comb it and what areas to comb.

So instead of checking known rares about 10 times a day, we'd check it at least 50 times a day at the least because we're skipping the ones that definitely are not ready.

That is a dream though, with the goal to try force TPTB to randomize beach tiles because of how much we'll make it suck.

Now I'll admit that I didn't fully examine your dataset so I'll do a brief writeup on how mine works.

In each minute I'm storing every tile, from the normal sand to the sparkles. Its effectively the same as the mafia state. Then I store the results of my combs (debugging purposes), the timestamps of combed tiles (so that as I progress it builds a dataset on tiles that haven't reset.)

So if a tile should have reset but I'm finding it combed, it's either being hogged or is a rare tile. I have 319 of those tiles already that has been over 2 weeks and multiple visits, but didn't reset.

It visits new beach areas when there's no rows covered, otherwise it tries to examine unknown sparkles or visit combs that should have reset by now. So even if someone successfully hides a bunch of rares, I'll still ultimately be made aware that this area has an abnormal amount of visits. I have a list of 52 of those areas already, which I can say is likely hiding a bunch of rares. More areas could be candidates, but with less confidence.

It's very recursive, the focus is mostly on discovering hoarded combs.

I expect someone abusing multis could hoard rares easily, or constantly comb a few random areas to obscure the data. I expect if it became obvious then we could get TPTB to investigate, it should be easy enough to track abnormal amounts of combs.

Code: 
9/23/2023, 12:00:09 PM    1695427210    SAND: 231560
9/23/2023, 12:00:09 PM    1695427210    UNCOMMON_SPARKLE: 4026
9/23/2023, 12:00:09 PM    1695427210    UNKNOWN_SPARKLE: 5830
9/23/2023, 12:00:09 PM    1695427210    COMBED: 2720
9/23/2023, 12:00:09 PM    1695427210    HEAD: 4
9/23/2023, 12:00:09 PM    1695427210    CASTLE: 78
9/23/2023, 12:00:09 PM    1695427210    RARE_SPARKLE: 2
9/23/2023, 12:00:09 PM    1695427210    We have data on 24.42% of total tiles, we know exactly what 96.5% of the tiles we have are, with 2720 being combed, and thus suspicious. We have 5830 sparkles to test that we know. We have 319 dodgy tiles that have been consistantly combed, but may just be RNG. And 52 really dodgy beaches..
9/23/2023, 12:00:09 PM    1695427210    Dodgy areas: 1, 4, 15, 16, 19, 24, 34, 49, 57, 74, 75, 79, 89, 100, 127, 134, 135, 149, 152, 160, 184, 196, 200, 204, 215, 220, 235, 271, 280, 286, 301, 320, 323, 325, 356, 393, 395, 396, 404, 410, 431, 438, 439, 465, 483, 500, 525, 528, 571, 580, 585, 615
9/23/2023, 12:00:09 PM    1695427210    We've visited 2655 of 10k beaches (26.55%), and of all visited beaches we have fully uncovered the secret of 54 (2.03%) of them. 8869 (88.69%) of beaches need to be revisited when there are no waves

Given the different approaches in code and data storage, and that I explicitly want to store each minute with every single tile. I'm not sure the two projects are mutually compatible. I'm also using a slightly different format for the tiles than mafia uses, in that it's using different characters. Which is somewhat annoying at times. I'm also using a giant json file, which I need to optimize.

My own project aims on recursively checking tiles, constantly rechecking to build datasets instead of focusing on exploring unless its a 0 rows covered day. I'm also pretty sure my weights on minutes to checked is bugged in some way but it's helpful in building a list of tiles that should've been reset so idm really.

I wouldn't say my approach is better than yours. The only notable differences would be that it's standalone, which results in possible performant gains. But slower as I'm not optimizing file storage.

It's also using a hardcoded diet and tiny daily activities (ode), and although it does support multiple accounts, it'd have to be setup the exact same way in 2crs.

Although adding another account or two to my combing would be a good idea, ascending multis is a pain and using more than one account feels like I'm asking for trouble.

I'm also trying to build up the dataset myself without crowdsourcing it, so unless you wanted to look at my own raw data, I don't feel a need to github it. I don't feel like it'd assist much as you've been running this for far longer than I have, and with more than a single account. And I recursively check areas; aka burn turns.

Also, a consideration is if someone is acting in bad faith and the data is public, then when they know we found their rare; They could just focus on other rares by hitting them more often. Multi abusing would make it pretty much impossible for us to find stuff over a decent timeline.

As far I have two rares, one would be completely unknown as it is at the first row. The other was definitely being hoarded. I intend to make a PR to combo at some point, just don't care to add only 1-2 rares at a time.

Anyways, my final closing thought is that we should be able to find a lot of potentially hoarded rares with the simple fact that I believe they're combing every sparkle in that area. So as a method to quickly narrow this down, you'd visit an area twice. The first to see what's combed, the second a week or two later to see if it's still being combed and wasn't just a one-off.

Possibly could be an opt-in for combo to check those areas to poke for rares.

If we're looking for unknown rares, then visiting the first few rows that are normally covered is where I believe they'd be found. There's probably only a few rares that are not automatically hoarded.
 
Veracity

Veracity

Developer
Staff member
Whoah. I just got up. Need coffee. I just skimmed your note, but will study in more detail, soon.

There are 10,000 beaches with 100 tiles per - 1,000,000 total, with from 0-4 rows underwater at any given time.
That's a lot of data to save - especially with timestamps!
But you could potentially discover how long it takes to regenerate a tile after its been combed.

My focus as well, has been to discover hoarded tiles.

I have 5 multis which are simply visiting the beaches from left to right. It takes that small "army" about 5 days to go through all 10,000 beaches and comb not-yet-visited twinkles. My main has been going only to not-previously-verified rare tiles from combo's rare list.

Results as of today:

Uncommon twinkles: 45,402
Verified rares from combo: 199
Erroneous rares from combo: 1

Previously unknown (hoarded) rares: 23.
Rows: 1:3 2:1 3:0 4:3 5:0 6:3 7:4 8:5 9:1 10:3
Most of those are rows that are always visible.

I wonder when combed tiles regenerate? If it's at rollover, then bad-faith hoarders can simply run turns right then and check their private tiles.
The Wiki indicates that tiles regenerate over a variable period of days. If so, KoL knows when each tile was combed and decides when each regenerates, and they might be regenerating at random times throughout the day - in which case there'd be no advantage in combing right after rollover; anybody combing any time during the day would have a chance of seeing a freshly regenerated tile.

I hope it is that way.

More later, after coffee. Thanks for sharing!
 
I

Irrat

Member
Because I had some power cuts, this isn't as reliable as I like. I didn't auto restart the bot. But firstly, tiles are not reset at rollover.
I also believe that waves don't reset them, though I never verified it.

The uncommon sparkles are confirmed to have reset on the minute mark. I didn't test for the exact second, not sure it matters either.
I observe the ranges where it resets from combed to sparkles as

Sand: 1 to 4 hours

Uncommon: 94 to 165 hours.
or
3 days 22 hours to 6 days, 21 hours

The rares tiles however, because of mentioned power cuts I'm not sure on the accuracy of these. Of the three rares I saw reset, only one of them was on the minute. And I'm sure I had power cuts between them, and don't care to investigate further.
The only thing I'm 100% confident on is that the 419 range is accurate. The lower bound could be smaller than 212.

Rare: 212 to 419 hours
or
8 days, 20 hours to 17 days, 11 hours

This was done on beach minute 3136 and looking at the claimed time period (12 to 21) for rares, I can't help but doubt if we're actually getting true rares.

Row 10,9 reset after 15 days, 10 hoursish
Row 7,2 reset after exactly 17 days, 11 hours. So it's a definite rare.
Row 8,4 reset after 8 days, 20ish hours.

I think combo needs an update to log if it's a false rare. I think in the above three rares, two of them may be fake and there's only one rare. With the two fake ones being combed while the bot was offline.
 
Veracity

Veracity

Developer
Staff member
If you see rough sand but not a twinkle, it is a common - assuming you can see twinkles.

KoLmafia doesn’t actually have a setting to record that. I’ll add one and set it to true if you visit Grandpa and ask for the right story.

I was planning on making my script require twinkle sight for spading, but not necessarily for just rare harvesting.
 
You must log in or register to reply here.
Top