Feature - Implemented Prefs file backup/restore Draft PR

On the theory that until we can consistently reproduce the bug, we are not going to be able to fix it, I spent some time this morning trying to write a test to reproduce the behavior, with no luck. My experience of the bug is some version of running a script in Mafia, closing my laptop without thinking to stop the script, and then opening it much later on different wifi, which triggers a timein. Somehow multiple timeins fire simultaneously (from different threads?) and cause a race which leads to the pref reset.

I'd really like to find and fix the bug and/or modify the prefs logic to never write out an empty prefs file, rather than implementing some backup solution on top of everything.

MCroft said:

I don't think the backup PR inside KoLmafia is the best approach.

It doesn't address the underlying problem, which if we can solve it would mean all of the backup style solutions would be unnecessary.

@fronobulax, is there anything anyone could do to help you with the concurrency branch testing?

Click to expand...

Do you mean https://github.com/kolmafia/kolmafia/pull/1889 ?

That code almost always triggers a deadlock when a draw is made from the deck of every card. @heeheehee looked at the stack traces and had some ideas and I have been lazy enough to let them handle it while I get distracted by test coverage, test failures now believed to be caused by cleanup (but not Cleanups), the search for a case where the maximizer failing to equip a weapon is demonstrably the wrong choice and not just a vague user belief that any weapon is always better than one, and multitool. If it heats up again I'll assist.

ikzann said:

On the theory that until we can consistently reproduce the bug, we are not going to be able to fix it, I spent some time this morning trying to write a test to reproduce the behavior, with no luck. My experience of the bug is some version of running a script in Mafia, closing my laptop without thinking to stop the script, and then opening it much later on different wifi, which triggers a timein. Somehow multiple timeins fire simultaneously (from different threads?) and cause a race which leads to the pref reset.

I'd really like to find and fix the bug and/or modify the prefs logic to never write out an empty prefs file, rather than implementing some backup solution on top of everything.

Click to expand...

It would be great if you could get that, although it sounds like @fronobulax has at least one 'almost always' case.

timein can be called by itself, and it might be possible to see if different threads timing in caused it, like a relay script and a gCLI script. I think threaded chat may be also help with creating threads, and also could be a candidate for investigation for things that might kick off a pref change via chat.

On the fix side of things, we've discussed some of the post-Java 8 locking mechanisms, but I don't know which of them we've experimented with. I found a nice cheat sheet here (of things that came out after I last worked with locks), but I'd like more real-world examples of what works for others. I also don't see a ReadLock/WriteLock difference on that page, and I'm not sure if that would help or not.

I've been using a Windows based laptop for several decades. I developed several habits, which are not necessarily needed today, but those habits prevent some of the issues that cause preference problems.

Sleep and suspend were unreliable and the only fixes were a reboot and that may or may not lose something. So I avoid putting the top down especially when something is running because there is no guarantee that it will continue in the same state. I also disable power management functions whenever something is running for similar reasons. Indeed, I don't leave anything running "unattended". I often reboot when switching wireless networks but some of that is related to not always needing a VPN.

So that means people who close their laptops while mafia is running or change networks while mafia is running might be getting what they deserve and my arrogantly unhelpful suggestions is "Don't Do That".

That aside I do think there is a fundamental design or implementation flaw in how mafia persists preferences. Sometimes they should be read unconditionally, sometimes they should be written before rereading, sometimes they should really be some kind of checking - compare disk and memory before reading for example and see if some type of corruption can be detected.

We have a setting that always writes preferences to disk. Maybe that is part of the problem, especially when disk operations can be buffered outside of KoLmafia's knowledge or control? Maybe we transition to reading the file once at Login, writing it once at Logout, make sure Timeout and Timein Logout and Login appropriately. Maybe we accept that preferences are in memory and if KoLmafia does not exit cleanly data may be lost. Maybe we add a Band-Aid and write preferences every five minutes or 50 turns or after 100 changes?

A local backup with automated restore is probably not a good solution. A tool that was run when KoLmafia was not running and let a user manually merge a backup and the preferences would probably be better.

There is a robustness to database technology that is appealing, especially since the unrecoverable loss is likely to be only one preference but there are a lot of benefits to being able to manipulate preferences with text based tools.

I'm losing focus and should probably stop.

MCroft said:

It would be great if you could get that, although it sounds like @fronobulax has at least one 'almost always' case.

Click to expand...

To be clear the using the Deck often causes a deadlock when running the code in the PR. That may, or may not, set up conditions where the preferences file is corrupted since the only way I know to get out of the deadlock is to abort the JVM.