Feature - Implemented Prefs file backup/restore Draft PR

MCroft

MCroft

Developer
Staff member
Based on some chatter elsewhere I thought we might try to help users who are losing their prefs files by creating a backup after we successfully load prefs. Not sure if it would help, but it might.

There are a lot of possible approaches and the code is working for basic use cases (prefs don't load, prefs do load, etc), and I'd like comments on my draft PR if anyone wants to help out/pre-review it, even if you run screaming in horror from the very idea. Especially if you run screaming in horror from the very idea.

All my notes are in the PR, but I'm happy to discuss it in either place.

github.com

automatic backups for <user>_prefs.txt by BadHorseMonkey · Pull Request #1713 · kolmafia/kolmafia

So, as a safety net for users who accidentally do things that delete their prefs, I want to put in some automatic backup/restore functionality as a resiliency measure. I've got a working bare b...
github.com github.com
 
You can't fix stupid.

Do you know the situations and operating systems where this happens? Does it seem to be happening more frequently compared to a year ago or are users just getting more vocal? Is this merely a bandaid for a core problem with threaded IO as implemented or access via multiple instances of KoLmafia?

Thinking outside of the box, is it time to replace preferences with a database that can be configured to handle a whole bunch of situations that would make storage more robust? We would lose user editable files but from a safety perspective that would be a good thing.

Are the issues empty files or individual preferences that are somehow trashed?

As a user, if the original file is gone - completely missing - then I want you to tell me but just swap in the backup without asking. But if the original is there then you should ask in case I believe there might be salvageable information that is not backed up.

If I had this problem I would probably create a script that made backup copies and then started KoLmafia. So I'm already prepared to have a backup that might be out of date.

My quick glance at the PR suggests it does what you have said you want it to.
 
I agree with frono. I'm not against this, just that it might not be the super solution to this problem. What happens if the user does not notice prefs missing right away, and the backup is also messed up?
Better to suggest a user use something like dropbox backup that stores multiple version backups and they can go retrieve them if something goes wrong.
 
In my experience, when my Windows computer crashes while mafia is in operation, my user prefs file gets a very large number of \u0000s at the top of it; if I remember to delete these before opening mafia, my preferences are often right as rain, but if I don't, a lot of preferences get read incorrectly and reset.

I know of a few other people who have ended up in a similar scenario, although I doubt that this encompasses all cases where someone's preferences get borked.
 
My experience agrees with the above. I've only seen it happen when a sudden power outage occurs....can't remember the last time I had an actual crash. But cleaning the tops of the prefs files allows everything to work fine.
 
Both OS crashes and power loss often lead to preference data loss. I would say the frequency I've observed other users losing preference data is more frequent than a few years ago. I wrote a utility called kolfix in response to that.
 
phreddrickk said:
In my experience, when my Windows computer crashes while mafia is in operation, my user prefs file gets a very large number of \u0000s at the top of it; if I remember to delete these before opening mafia, my preferences are often right as rain, but if I don't, a lot of preferences get read incorrectly and reset.

I know of a few other people who have ended up in a similar scenario, although I doubt that this encompasses all cases where someone's preferences get borked.
Click to expand...

At the risk of undercutting my own ambivalence about the feature, I have seen a bunch of \u0000s at the top and the top of a preference file and editing them out also worked to restore the file. But the cause was not a Windows crash while running mafia.

I cannot ever recall a Windows crash while running KoLmafia. I tend to have one Windows crash every five years or so and they have all been traced to malfunctioning hardware. I run on a laptop with a working battery so power loss crashes just don't happen.

The anecdotes so far suggest this is a Windows problem that is not happening on Mac or Linux.
 
I was thinking basically along the lines of "Hey, someone is sad about their preferences getting nuked. I know that MS Word used to make a .bak file on launch to protect you from catastrophic crashes". I don't know if this would be useful and it sounds like the automated part isn't addressing the \u0000 problem, but the .bak would still be there and could be manually installed if the user noticed a problem. Having a backup of a correctly read in prefs file seems like a bandaid for a lot of possible problems, some of which might be malfunctioning hardware and some of which may be the java.io is crusty and some of which may be in our code.

If we put the prefs in a DB (and we have a fun discussion of sqlite vs Postgres vs MariaDB), I still like the idea of doing some sort of periodic dump to save state. But it sort of assumes that the persist after every change thing is fine to permanently leave on. That's a good topic, but may be a different topic from this one.

And I don't mind helping out our Windows brethren. They have enough to worry about already. :)

It may be that we should revamp Preferences.loadPreferences() to ignore any line that starts with \u0000 (or throw up a warning like Hey, these prefs look wrong! Did you crash?). I'd love to find a prefs file in that state to inspect and compare to a known good backup.
 
And yes, I've never seen it on my Mac, but I am also on a laptop. I also wonder if this is happening more to users with their prefs on Box or Google Drive or a NAS who forget not to open it from two different computers.
 
MCroft said:
And yes, I've never seen it on my Mac, but I am also on a laptop. I also wonder if this is happening more to users with their prefs on Box or Google Drive or a NAS who forget not to open it from two different computers.
Click to expand...

Interesting possibility. I've been using Dropbox for a very long time but never ran mafia on more than one computer at a time.
 
fronobulax said:
If I had this problem I would probably create a script that made backup copies and then started KoLmafia. So I'm already prepared to have a backup that might be out of date.
Click to expand...
It's funny one should mention to make a script to backup preferences and such, because I had done that. My preferences wiping happened to me very frequently when I was on a bad Internet connection and using a computer with failing hardware. And I did make a perl script at the time to not only cleanse the pref files of those \u0000s before running mafia, but also back those (cleansed) files up before and after running mafia.

But apparently cleansing and backing up the preference files was not enough.

I have very recently had my user preferences wiped while mafia was running. And I didn't know it until it stopped because some obtuse preference was no longer set. But by that point my daily preferences were way out of whack, and the backup couldn't have help with that because the backup was from an entirely different ascension. Though the backups did help to set the automation script preferences and other general mafia settings back to what they were.

I'm not sure exactly how it came about, but I do know a few things:
1. I was idle enough to have been logged out of kol without mafia knowing
2. When I came back, I noticed the last script finished a community service run, so I decided to run my post-run script via the CLI.
3. When I did that, I saw mafia started running my login script (which includes breakfast and other such things) at the start of the emptying Hangk's part of the post-run script. At which point a bunch of preferences were being explicitly ravaged according to the log.
4. After the login script was finished, the post-run script continued from the point after Hangk's is emptied. But the post-run script stopped pretty quickly when it got to trying to use multiple universal seasonings since there's an obtuse preference mafia uses to tell how many universal seasonings can be used that was no longer set properly.

I'm not sure how useful the log would be without debug, but I've attached it anyway. The log shows a bunch of preferences being explicitly reset right at Emptying storage, where normally mafia is paused, but as it recall mafia was very much not paused. Though it omits a ton of preferences that were also changed, since it was basically a full preference wipe as far as I could tell when trying to fix it.
 

Attachments

Last edited:
Zdrvst said:
It's funny one should mention to make a script to backup preferences and such, because I had done that. My preferences wiping happened to me very frequently when I was on a bad Internet connection and using a computer with failing hardware. And I did make a perl script at the time to not only cleanse the pref files of those \u0000s before running mafia, but also back those (cleansed) files up before and after running mafia.

But apparently cleansing and backing up the preference files was not enough.

I have very recently had my user preferences wiped while mafia was running. And I didn't know it until it stopped because some obtuse preference was no longer set. But by that point my daily preferences were way out of whack, and the backup couldn't have help with that because the backup was from an entirely different ascension. Though the backups did help to set the automation script preferences and other general mafia settings back to what they were.

I'm not sure exactly how it came about, but I do know a few things:
1. I was idle enough to have been logged out of kol without mafia knowing
2. When I came back, I noticed the last script finished a community service run, so I decided to run my post-run script via the CLI.
3. When I did that, I saw mafia started running my login script (which includes breakfast and other such things) at the start of the emptying Hangk's part of the post-run script. At which point a bunch of preferences were being explicitly ravaged according to the log.
4. After the login script was finished, the post-run script continued from the point after Hangk's is emptied. But the post-run script stopped pretty quickly when it got to trying to use multiple universal seasonings since there's an obtuse preference mafia uses to tell how many universal seasonings can be used that was no longer set properly.

I'm not sure how useful the log would be without debug, but I've attached it anyway. The log shows a bunch of preferences being explicitly reset right at Emptying storage, where normally mafia is paused, but as it recall mafia was very much not paused. Though it omits a ton of preferences that were also changed, since it was basically a full preference wipe as far as I could tell when trying to fix it.
Click to expand...

Thank you. Assuming there was only one instance of mafia running this seems to tell the story of a mafia flaw and not some kind of interaction between instances of mafia or mafia and the operating system. The proposed change definitely would not be expected to solve the problem, just reduce the consequences under certain contitions,
 
Hmm. Wondering if something is trying to adjust inventory while hangk is being emptied. Maybe chat or pvp theft? I don’t use either frequently, but they are on different threads.
 
I can't offer any help as to why it might have happened, but I suffered a preference loss recently that prompted me to add a little bit to my mafia-launching script (Mac, zsh).

Code: 
# On Monday, backup preferences and moods, and delete versions older than 30 days
if [ $(date +%u) = "1" ]; then
    theDate=$(date +%y-%m-%d)
    rsync -aP "~/Library/Application Support/KoLmafia/settings/username_prefs.txt" "~/Library/Application Support/KoLmafia/settings/backups/username_prefs_$theDate.txt"
    rsync -aP "~/Library/Application Support/KoLmafia/settings/username_moods.txt" "~/Library/Application Support/KoLmafia/settings/backups/username_moods_$theDate.txt"
    find "~/Library/Application Support/KoLmafia/settings/backups/" -type f -mtime +30 -exec rm -f {} \;
fi

I am not good at zsh scripting, so beware: it includes an rm command, and is most likely terribly inefficient.
 
mandybrigwell said:
I can't offer any help as to why it might have happened, but I suffered a preference loss recently that prompted me to add a little bit to my mafia-launching script (Mac, zsh).
Click to expand...

Thank you. A data point that it is not just a Windows issue.

MCroft said:
Hmm. Wondering if something is trying to adjust inventory while hangk is being emptied. Maybe chat or pvp theft? I don’t use either frequently, but they are on different threads.
Click to expand...

I always empty Hagnk's when I can. I get a comment from something (mafia? script? relay spoiler?) advising me to chill until it is unloaded. I suspect that comment was triggered by something unpleasant that happened, in the past, and there was no feasible fix. I do recall a time when I tried something before Hagnks finished emptying and was not happy. That is why I empty Hagnk's and just walk away for a few minutes. If we are going to explore this I don't chat or pvp so if there is something running concurrently with emptying it is probably something mafia is doing. As a hypothetical, if acquiring an item sets off a preference update I can construct a lot of things that might happen we really don't want. Life was so much simpler when there was only a single "thread" :-)
 
For reference, I have had this as a part of my daily backup for years:


Bash: 
#!/bin/sh

cd /home/xkiv/.kolmafia
cp -r data planting relay scripts settings ~/take/bak/mafia
echo "kolmafia git backup"
cd ~/take/bak/mafia
/usr/bin/git add -A .
/usr/bin/git commit -a -m 'auto commit'

I believe that if we already have a git library, using it to keep versioned backups might not be entirely stupid idea.
Unless the problem is such that *any* files withing mafia's structure can get clobbered.
 
I had my preferences get completely reset a few days ago while mafia was left running. I was running headless cli mode and in a combat in the relay browser and left for long enough to get logged out and when I opened the relay browser again all my preferences got reset to default when mafia attempted to log back in.
 
xKiv said:
I believe that if we already have a git library, using it to keep versioned backups might not be entirely stupid idea.
Unless the problem is such that *any* files withing mafia's structure can get clobbered.
Click to expand...
If we're going that far, I think I'd like Frono's idea of a SQL database better. downside can't edit with a text editor, upside, can edit with a SQL client. Upside is not as good for people who don't use sql. Imagine Transactions! in the prefs...
Rinn said:
I had my preferences get completely reset a few days ago while mafia was left running. I was running headless cli mode and in a combat in the relay browser and left for long enough to get logged out and when I opened the relay browser again all my preferences got reset to default when mafia attempted to log back in.
Click to expand...
Do you happen to have the damaged prefs file?
 
No, but it wasn't bad characters at the beginning. It was just all the settings were reset back to default and any properties not set by mafia were completely deleted.
 
You must log in or register to reply here.
Back
Top