BeachComber - fast and efficient beach combing

[fewyn]

Are spaded locations automatically submitted? or do you want the files here?

Are *.new.json files updated/truncated/merged if you have added content and I have updated from Git? or should they be periodically deleted?

No new rare tiles I'm afraid but some new uncommon and some verified rare tiles (I think)

The forum doesn't allow .json uploads so let me know if you want the contents of tiles.uncommon.new.json and tiles.rare.seen.json

(looks like we allow .js and other extensions so no real reason not to allow .json, probably something for @fewyn to take a look at)

json is now allowed

 

[Veracity]

Git update will replace tiles.rare.json and tiles.uncommon.json but will leave tiles.rare.new.json and tiles.uncommon.new.json untouched.

“BeachComber merge” is a command for me - do not use it! - to merge the .new tiles into the released tiles and zero the .new tiles.

I then release the merged tiles and everybody gets them.

If you have tiles in your .new files that are now public, no harm; both files are loaded and duplicates are discarded.

I will make a “BeachComber prune” command which will delete tiles in your .new files that are already released. There will no operational differences, since duplicates are ignored - but it will trim what you need to submit.

I am up to 51 tiles in tiles.rare.seen.json - rare tiles from combo that I have verified. I have also discovered 2 new rare tiles. I have not published those yet - or submitted them back to combo yet. I’m hoping to collect more, first.

There is not currently a distinction between rare tiles that are officially verified and those that other users have verified. I should do that, and make tiles.rare.verified.json the “published” one and tiles.rare.seen.json the “new” verified tiles.

Then “merge” and “prune” would deal with those, too.

Thanks for you comments! I’m not set up for integrating user contributions, yet, but it will be simple enough - a prune command, and rare.verified - to be good to go for that.

 

[Veracity]

By the way, the resources I am committing to this project:

I have 5 characters who are doing the following each day:

use 1 piece of driftwood
(unless they have a Beach Comb)
vmf nofarm synthesize=none
(do all my daily stuff - including Spacegate, which uses some turns)
BeachComber spade all
vmf
(do end of day stuff, including drinking a nightcap)
BeachComber spade all

That spades all (new) twinkles on about 500 sections a beach per day. More, if the tides are high, fewer, if they are low. Of course, if the tides are high, we'll just need to revisit the beach section at lower tide in order to learn about more twinkles.

(VMF calls BeachComber with its default arguments - BeachComber free rare - which uses all "free" turns looking for rares - so that is (potentially) a bit of spading and also the chance for a jackpot.)

And my main does this:

vmf nofarm synthesize=none
BeachComber all rare tidal twinkles
(look for rares, visit all unknown twinkles while doing so, and start with the row closest to the water,
which might have revealed new unvisited twinkles, now that the tides have retreated.)
vmf

This is how I have verified 51 rare tiles in combo's data set so far.

That leaves me extra (overdrunk) turns, in case I need to experiment with something else today. If not, they are available tomorrow.

 

[Veracity]

Revision 13 does the following:

1) Save sand castles in tiles.castle.json and tiles.castle.new.json

This is for fun; sand castles were part of a puzzle which has been solved, but why not preserve their locations, for posterity?
To my surprise, there were far fewer than I expected. Perhaps people have been combing them? One would hope they would regenerate.

2) Released "verified" tiles are now in tile.rare.verified.json

When you encounter a rare tile, it either goes into tiles.rare.new.json (if its not in combo's data) or tiles.rare.seen.json (if it is.)
(Hmm. I shouldn't bother with that, if it is in the new verified list.)

3) "BeachComber prune" will remove "released" tiles from "new" tiles for the following lists:

tiles.rare.json -> tiles.rare.new.json
tiles.rare.verified.json -> tiles.rare.seen.json
tiles.castle.json -> tiles.castle.new.json
tiles.uncommon.json -> tiles.uncommon.new.json

After doing so, the following files are suitable for submission as user-spaded data

tiles.rare.new.json
tiles.rare.seen.json
tiles.castle.new.json
tiles.uncommon.new.json

I can take those files into my own "beach" directory and do "BeachComber merge" to add them to the files that I release.

I'm up to 61 known "verified" rare tiles - and 3 "new" rare tiles.
I am holding off on releasing those until I feel I'm more-or-less done and ready to submit them to combo.
That's because once I merge tiles.rare.new.json into tiles.rare.json, as coded, the "merge" command clears out the .new file - and I think that is the file that combo will want: just the new tiles, rather than all 800+ existing tiles and the others intermingled.

Upcoming:

1) Don't add a tile to tiles.rare.seen.json if it is already in tiles.rare.verified.json
2) Add a new "unverified" keyword to make spading & verifying known rares more efficient: rather than picking from all known rares, pick from unverified. (If you run out, you can go to verified ones and maybe get lucky - or, at least, spade some more uncommons.)

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 3
Verified rare tiles: 61
Not previously verified rare tiles: 0
Known uncommon tiles: 20272
New uncommon tiles: 0
Known sand castles: 37
New sand castles: 0
Last minutes down the beach spaded: 6037

Beaches with rare tiles: 809
Beaches with uncommon tiles: 5782
Beaches with sand castles: 12
Beaches with unvisited twinkles: 0

 

[Veracity]

Speaking of looking at saved HTML, KoLmafia revision 27556 allows this:

> test showhtml beach/html/Veracity_4067_3_9_20230823234110454.html

Read 33,410 bytes into a 33,410 character string

> test showhtml beach/html/Veracity_4067_3_9_20230823234110636.html

Read 17,273 bytes into a 17,273 character string

> test showhtml beach/html/Veracity_50_8_3_20230823234227817.html

Read 18,733 bytes into a 18,733 character string

> test showhtml beach/html/Veracity_7971_10_2_20230821073030851.html

Read 17,260 bytes into a 17,260 character string

loads four of my saved results.

- Seeing a whale on the beach
- Combing a whale
- Combing a pirate treasure trove
- Finding a message in a bottle.

KoLmafia loads them in a (single) mini-browser.
The "location" field is the file name.
There are back and forward arrows to go backwards and forwards in the history list.
And the HTML is rendered below, complete with images that KoLmafia downloaded from KoL.

Here's what a whale looks like on the beach. (It would look better in a real browser - the beach columns would be adjacent, and perhaps there would be waves lapping at the bottom - but you'd find it harder to have the browser load the images, I think.)


Yeah. The bottom row is 10 columns, all adjacent.
The previous rows have spaces between the columns, such that 8 columns fit in the width of 10 columns with no spacing.
That is a bug in the mini browser. We’ll see if I care enough to try to fix it.

 

[Veracity]

Revision 14 does this:

BeachComber all rare unverified twinkles

will visited known rares that have not yet been verified and comb all not-yet-visted twinkles, either verifying rares (or finding errors!) or discovering new uncommons.

and also submits more tile data:

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 4
Verified rare tiles: 67
Not previously verified rare tiles: 0
Known uncommon tiles: 22295
New uncommon tiles: 0
Known sand castles: 66
New sand castles: 0
Last minutes down the beach spaded: 6604

Beaches with rare tiles: 810
Beaches with uncommon tiles: 6293
Beaches with sand castles: 28
Beaches with unvisited twinkles: 0

We may be to the point that I'll have nothing more than new spading data to submit here.

 

[Veracity]

Revision 15 fixes a bug in the "prune" command and submits more data I have spaded.
Another 3 days, and I should have visited all 10,000 beaches and have wrapped around to 1 again.
I am sure that new uncommon - and hopefully rare - tiles will continue to be found, as previously combed ones get renewed.
And there HAVE to be more sand castles.

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 5
Verified rare tiles: 73
Not previously verified rare tiles: 0
Known uncommon tiles: 28622
New uncommon tiles: 0
Known sand castles: 66
New sand castles: 0
Last minutes down the beach spaded: 8875

Beaches with rare tiles: 811
Beaches with uncommon tiles: 8229
Beaches with sand castles: 28
Beaches with unvisited twinkles: 0

 

[Veracity]

Revision 16 adds another script which, hopefully, none of us will need.

Yesterday, I ran my BeachComber turns using a buggy (under development) version of KoLmafia. Although it did all my combings, it trashed all my "new" files, as if I had learned nothing.

I wrote a program to scrape the session logs of my characters and, voila! It recovered everything.

I generalized the program and am releasing it for your amusement.

> BeachComberFix help

BeachComberFix DATE [PARAM...]
    DATE is the date of the first session log to inspect
    (We will inspect all session logs after that up to present)
    player=name - look at session log for a single player
    (if omitted, take players from beach/players.txt)
    (if empty, use current player, if logged in)
    save - after scraping tiles and pruning, save results.

After sending all 6 of my characters to comb the beach today, I ended with this:

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 9
Verified rare tiles: 73
Not previously verified rare tiles: 9
Known uncommon tiles: 28622
New uncommon tiles: 2022
Known sand castles: 66
New sand castles: 31
Last minutes down the beach spaded: 9575

Beaches with rare tiles: 814
Beaches with uncommon tiles: 8837
Beaches with sand castles: 45
Beaches with unvisited twinkles: 0

I then ran BeachComberFix to try to recreate that by scraping the session logs of all my characters (who are listed in beach/players.txt).
The way it works:

1) Scrape all the session logs and collect 'rare' and 'uncommon' tiles in lists.
2) Clear "new rare" and load it with all the rare tiles we've seen.
Clear "seen rare" and load it with all the rare tiles we've seen.
Clear "new uncommon" and load it with all the uncommon tiles we've seen.
3) "prune" the tiles - as in "BeachComber prune", which will do the following:
Remove all "known rare" tiles from "new rare"
Remove all "verified rare" tiles from "seen rare"
Remove all "known uncommon" tiles from "new uncommon"

The end result should have those last lists exactly correct: having only the unpublished data you've collected.

> BeachComberFix 20230801

Initial tile data

rare tiles: 849
new rare tiles: 9
verified_rare_tiles: 73
seen_rare_tiles: 9
uncommon_tiles: 28622
new uncommon_tiles: 2022

Processing logs for CHARACTER1
Processing logs for CHARACTER2
Processing logs for CHARACTER3
Processing logs for CHARACTER4
Processing logs for CHARACTER5
Processing logs for CHARACTER6

Tiles processed

common: 5241
uncommon: 30643
rare: 90
unknown: 4

Processed tile data

rare tiles: 849
new rare tiles: 90
verified_rare_tiles: 73
seen_rare_tiles: 90
uncommon_tiles: 28622
new uncommon_tiles: 30643

Pruned tile data

rare tiles: 849
new rare tiles: 9
verified_rare_tiles: 73
seen_rare_tiles: 9
uncommon_tiles: 28622
new uncommon_tiles: 2022

As you can see, the "Initial tile data" is identical to the "Pruned tile data".

 

[Veracity]

I completed my first cycle of 10,000 beaches and have wrapped around to minute 550.
I discovered 32418 "uncommon" tiles - an average of 3+ per beach.
I also discovered 93 + 9 = 102 rare tiles - a bit more than 10% of combo's "known" rare tiles and 9 new ones.
I intend to keep doing this cycle for the next few months; my multis have nothing better to do, and, aside from 11 "smol" runs, neither does my main.
What will I find?

- Uncommon and rare tiles I did not see previously because they were washed by the tides
- Uncommon tiles that people combed because their rare tile was unavailable, but which have renewed.
- Rare tiles that have renewed that combo knows - verifed rares
- Rare tiles that have renewed that combo does not know - new rares
- More sand castles

Progress should be a lot faster; since this skips known twinkles - unless one is a rare or there are no unknown twinkles - rather than averaging 3+ combs per square, we'll be a lot closer to 1 comb per square.

I'll continue publishing new data as I discover it, but, perhaps, not every day.

Revision 17:

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 9
Verified rare tiles: 93
Not previously verified rare tiles: 0
Known uncommon tiles: 32418
New uncommon tiles: 0
Known sand castles: 101
New sand castles: 0
Last minutes down the beach spaded: 550

Beaches with rare tiles: 814
Beaches with uncommon tiles: 9251
Beaches with sand castles: 47
Beaches with unvisited twinkles: 0

 

[Veracity]

Revision 18 adds three ways to filter rares:

- closest to the tides
- not yet published
- not yet verified.

I like BeachComber all rare unverified to focus on verifying that combo's rare list is correct.

I also submitted more tile data.

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 10
Verified rare tiles: 118
Not previously verified rare tiles: 0
Known uncommon tiles: 35656
New uncommon tiles: 0
Known sand castles: 226
New sand castles: 0
Last minutes down the beach spaded: 3096

Beaches with rare tiles: 815
Beaches with uncommon tiles: 9466
Beaches with sand castles: 101
Beaches with unvisited twinkles: 0

 

[Veracity]

Revision 19 fixes a bug in rare filtering and adds one more day of spading.
Even on my second time through the beaches, I continue to find lots of new uncommon tiles.

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 10
Verified rare tiles: 127
Not previously verified rare tiles: 0
Known uncommon tiles: 37391
New uncommon tiles: 0
Known sand castles: 264
New sand castles: 0
Last minutes down the beach spaded: 4315

Beaches with rare tiles: 815
Beaches with uncommon tiles: 9540
Beaches with sand castles: 123
Beaches with unvisited twinkles: 0

 

[Veracity]

Revision 20 completes my second pass through all 10,000 beaches.
I found a lot more uncommon tiles, due either to tides being lower than last time through, or having regenerated on beaches with rare tiles, where rare seekers had previously combed them in disappointment at finding their rare already combed.
Speaking of which, I also found 3 more previously unknown rare tiles.

I am not sure how best to share them, yet, since the intent is to let combo take them in, too.

1) Simply merge them into tiles.rare.json and let combo take that file to replace theirs. FWIW, it also fixes a few bugs: there was a duplicated entry, and some of the tiles were not sorted correctly.
2) Release my tiles.rare.new.json and let BeachComber users (and combo) do what they wish with it.
The former could simply install it in their beach directory - or, if they have discovered new rares themselves, merge it with what they have.
The latter would have to merge it with their existing tile database.

I am hoping and expecting to find more previously not public rares as I make additional passes over the 10,000 beaches.
Tomorrow, I start again with beach 1 - and the tides will have ebbed a row.

> BeachComber data

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 13
Verified rare tiles: 133
Not previously verified rare tiles: 0
Known uncommon tiles: 39524
New uncommon tiles: 0
Known sand castles: 327
New sand castles: 0
Last minutes down the beach spaded: 0

Beaches with rare tiles: 818
Beaches with uncommon tiles: 9743
Beaches with sand castles: 163
Beaches with unvisited twinkles: 0

 

[Veracity]

Revision 21 has tile data from yet another complete pass over the 10,000 beaches.

Additionally, adjusts what I save about sand castles: instead of saving every tile (I have seen up to 5 on a particular section of the beach), save only the beach #, since that is what encodes the secret message.

I added a new program - BeachComberCastles.ash - which decodes the castle beaches as recorded in beaches.castle.json.
There are a lot of castles which are not part of the message - including interspersed with the castles that DO encode the message.
BeachComberCastle uses heuristics to determine which ones are actually part of the message. Mainly, the interval between beach numbers must be a multiple of 11.

> BeachComberCastles

Unvisited twinkle tiles: 0
Known rare tiles: 849
Erroneous rare tiles: 0
New rare tiles: 15
Verified rare tiles: 160
Not previously verified rare tiles: 0
Known uncommon tiles: 43908
New uncommon tiles: 0
Known sand castle beaches: 173
New sand castle beaches: 0
Last minutes down the beach spaded: 2957

Beaches with rare tiles: 820
Beaches with verified rare tiles: 158
Beaches with uncommon tiles: 9872
Beaches with sand castles: 173
Beaches with unvisited twinkles: 0

There are 173 beaches with sand castles
There are 181 castles encoding the puzzle hint
Encoded message = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'
Decoded message = 'EVERYTWENTYSECONDLETTEROFTHEBOTTLEMESSAGELOOPEDTHRICE'
There are 163 hint castles that we have not seen yet
Decoded message from observed castles = 'EVERY'

Note that I can encode the message form the Wiki into a set of sand castle beaches (which I publish as beaches.castle.wiki) and I can decode it just fine.

Decoding from the set of castles my spading has observed is missing the bulk of the castles. Anything before beach #8902, in particular.
I look forward to picking up the rest of the message, over time.

 

[Veracity]

Something I'm wondering is if the "last" castle varies by character.
I have seen 8902 8935 8968 8979 9012 9067 9078 9111 9122 9177 9188 9243 9276 9287 9298 9309 9364 9375 as the last 18 castles on the beach - and that gives me EVERY - but what if the sequence starts at a different beach for other characters?
9375 9364 9309 encodes the letter "E".

What if you could start on beach 9365-9374 or 9376-9385 and go backwards from there?
There could be 11 different starting beaches all mixed in together. and not interfering with each other.

I'll modify BeachComberCastle to let it start looking at a specific beach and see if we have later parts of the message encoded in beaches I have collected.

 

[Veracity]

Ha ha ha.

> BeachComberCastles 18

There are 173 beaches with sand castles
There are 181 castles encoding the puzzle hint
Encoded message = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'
Decoded message = 'EVERYTWENTYSECONDLETTEROFTHEBOTTLEMESSAGELOOPEDTHRICE'
There are 163 hint castles that we have not seen yet
8317-8251=66
Decoded message from observed castles = 'T?NTY'

> BeachComberCastles 35

There are 173 beaches with sand castles
There are 181 castles encoding the puzzle hint
Encoded message = 'EVERY TWENTY SECOND LETTER OF THE BOTTLE MESSAGE LOOPED THRICE'
Decoded message = 'EVERYTWENTYSECONDLETTEROFTHEBOTTLEMESSAGELOOPEDTHRICE'
There are 163 hint castles that we have not seen yet
Decoded message from observed castles = 'SECOND'

I think so. I can decode message sequences from castles that we have observed that are not in my "derived" set of castles - even though we have actually observed (some of) those "derived" castles.

I don't REALLY feel like making castle data character specific. Lets see what happens when other characters end up visiting the "last castle" territory.

 

[Rinn]

If you make a combo PR someone will get it merged https://github.com/loathers/combo

 

[ckb]

Unknown variable 'int_array' (BeachComberData.ash, line 100, char 5 to char 14)
Expected ;, found array (BeachComberData.ash, line 100, char 15 to char 20)
Unknown variable 'array' (BeachComberData.ash, line 100, char 15 to char 20)
Function 'parse_json_int_array( buffer )' undefined. This script may require a more recent version of KoLmafia and/or its supporting scripts. (BeachComberData.ash, line 100, char 23 to char 43)
Unknown variable 'array' (BeachComberData.ash, line 101, char 25 to char 30)

 

[Veracity]

Oops. BeachComberJSON was updated too….

Revision 22 fixes that.

 

[matt.chugg]

Whenever I use >beachcomber help or >beachcomber data, I also get a Stack trace: printed in the CLI, sometimes it has a line number and sometimes it doesn't



I had a look and it seems to me it's caused by the exit; statements when outputting the data summary or the help text to the CLI preventing continuation of the script when help or data parameters are present, according to the wiki->control structures section on the KoLmafia wiki exit will exit the script and it makes sense to have the stack trace as I imagine it's intended to be used like the PHP die() function. Since these function are in void main() would return; work in the same way but prevent the stack trace?

I'm certain @Veracity has their own reasons for doing this, mostly just curious because I didn't know exit; existed!

 

[Veracity]

"exit" is not supposed to cause a stack trace.
I have never seen an ASH stack trace.
Looking at ASHRuntime:

  @Override
  public void setState(final ScriptRuntime.State state) {
    super.setState(state);

    if (state == ScriptRuntime.State.EXIT && Preferences.getBoolean("printStackOnAbort")) {
      this.printStackTrace();
    }
  }

1) It depends on a setting - "printStackOnAbort" - which I have never heard of.
2) Whoever added that decided that any "exit" is an "abort".

That is clearly wrong. It is an "abort" if you call the abort() function - which puts KoL into the ABORT state and does an exit.

  private static Value abort(ScriptRuntime controller, final String string) {
    KoLmafia.updateDisplay(MafiaState.ABORT, string);
    controller.setState(ScriptRuntime.State.EXIT);
    return DataTypes.VOID_VALUE;
  }

Whoever implemented "printStackTraceOnAbort" neglected to check for MafiaState.ABORT.
I don't run with that setting (although it seems like a good idea), but I'll have to fix that.