Bug Issue with updating scripts from Github?

[fewyn]

So I've actually run into a really weird issue with updating scripts on that are stored on Github and for the life of me can't figure out where the problem might be.

Checking all SVN projects...
batbrain is at HEAD (r141)
batman-re is at HEAD (r62)
cdrock-TourGuide-branches-Release is at HEAD (r988)
bestbetweenbattle is at HEAD (r43)
clilinks is at HEAD (r6)
drag-n-drop-inventories is at HEAD (r6)
Ezandora-Bastille-branches-Release is at HEAD (r13)
Ezandora-Briefcase-branches-Release is at HEAD (r83)
Ezandora-Choice-Override-branches-Release is at HEAD (r36)
Ezandora-Comb-Beach-trunk-Release is at HEAD (r10)
Ezandora-Detective-Solver-branches-Release is at HEAD (r19)
Ezandora-Far-Future-branches-Release is at HEAD (r40)
Ezandora-Voting-Booth-trunk-Release is at HEAD (r8)
formhtml is at HEAD (r1)
gausie-excavator-trunk-RELEASE is at HEAD (r179)
kolmafiascripts-mafiarecovery is at HEAD (r6)
kolmafiascripts-cfstat is at HEAD (r4)
Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE needs updating from (r3482) to (r3490)
kolmafiascripts-shop is at HEAD (r10)
Loathing-Associates-Scripting-Society-Helix-Fossil-branches-Release is at HEAD (r23)
Loathing-Associates-Scripting-Society-newlife2-trunk-release is at HEAD (r170)
Loathing-Associates-Scripting-Society-philter-trunk-release needs updating from (r684) to (r690)
mafiachit is at HEAD (r702)
mercenarymood is at HEAD (r33)
reference-plus is at HEAD (r6)
smartstasis is at HEAD (r48)
soolar-CONSUME.ash-trunk-RELEASE is at HEAD (r78)
therazekolmafia-canadv is at HEAD (r110)
veracity0-arrrbor-day is at HEAD (r294)
veracity0-beach is at HEAD (r294)
veracity0-garden is at HEAD (r294)
veracity0-gingerbread is at HEAD (r294)
veracity0-meat-farm is at HEAD (r294)
veracity0-vcon is at HEAD (r294)
veracity0-vprops is at HEAD (r294)
zlib is at HEAD (r51)
winterbay-mafia-wham is at HEAD (r47)
Updating all SVN projects...
Validating repo...
Repo validated.
Updating Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE...
e:\OneDrive\KM\svn\Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE
At revision 3490
Done.
Validating repo...
Repo validated.
Updating Loathing-Associates-Scripting-Society-philter-trunk-release...
e:\OneDrive\KM\svn\Loathing-Associates-Scripting-Society-philter-trunk-release
At revision 690
Done.
Checking for working copy modifications...
No modifications.
Requests complete.

7454 prices updated from https://kolmafia.us/scripts/updateprices.php?action=getmap

It shows that the repo as a higher version but doesn't actually update the local copy. I can delete the script and reinstall it via the svn checkout command and it'll update to the latest commit. If I use tortoisesvn to check the log it shows I'm still on the older commit and when I update it it does update it but mafia will not update anything.

Note I have completely wiped out my settings recently so this is a fairly fresh install of mafia and running the latest build.

I do have all the SVN settings enabled.




I'm really at a loss of ideas

 

[fronobulax]

I note the two examples are from LASS which from where I am sitting, is a collection of scripters who are pushing the envelope on how mafia manages scripts. So it may be they have done something mafia wasn't expecting or mafia didn't document wasn't allowed.

Can you make a distinction between mafia not updating the local repository from GIT and mafia not updating /scripts etc. from the local repository?

The former might point to a change in the structure of the project or the layer that lets git respond to svn commands. The latter could be the result of some recent changes that allowed two scripts to use the same file name provided they installed things in different subdirectories.

I feel like this is mine to fix but I'm not sure where to start.

 

[AlbinoRhino]

I think I've noticed this behavior when a scripter has multiple scripts in the same repo. When they update a script you don't have installed you will get the messages like above but nothing actually changes locally unless there has been an update to the script which you DO have installed. I think...I could be wrong.

 

[fronobulax]

I think I've noticed this behavior when a scripter has multiple scripts in the same repo. When they update a script you don't have installed you will get the messages like above but nothing actually changes locally unless there has been an update to the script which you DO have installed. I think...I could be wrong.

It would not surprise me at all if multiple scripts in the same repo were an issue.

I'm going to tag @philmasterplus.

I note that the check out command for philter is https://github.com/Loathing-Associates-Scripting-Society/philter/trunk/release/ but mafia's local SVN repo is Loathing-Associates-Scripting-Society-philter-trunk-release so mafia's decision to collapse the tree might not be behaving as hoped.

 

[philmasterplus]

I observed this happening when you push a commit to GitHub, but the commits don't touch files that are actually installed.

For example, here's a popular project layout:

release/
  scripts/
    foo.ash
    bar.ash
README.md
LICENSE
.gitignore

Users are given an install URL that only downloads files under release/. This allows the developer to keep files that should not be distributed in the same repo.

IIRC Subversion creates a new revision (rNNN) whenever a file changes. But what if the developer modifies a file that is not under release/? The client (KoLmafia) sees a new revision, but since the install URL points to release/, the client sees no file changes. So KoLmafia merely updates the revision number and moves on.

In some rare cases, a file change does exist but is dropped by KoLmafia. I didn't investigate this because it's really hard to reproduce. But I suspect it's caused by network hiccups or GitHub server glitches.

 

[fewyn]

It did it again this morning.

For more information it looks like the following files were modified ("commits" 3491, 3499, 3530)







I'm going to clone this install directory if you want me to gather any additional information I will be able to.

 

[fronobulax]

To confirm, the issue seems to be between KoLmafia's local repository and the remote git repository and not between Kolmafia's local repository and Kolmafia's scripts (and other) directories, correct?

 

[fewyn]

So it looked like the repo in the SVN directory was not up to date... running the update command via tortoisesvn didn't work either. I'm starting to think this might be a github interaction with SVN?

I told it to manually update to the latest commit and it said this:

 

[philmasterplus]

Does the problem occur exclusively with autoscend? If so, it probably has to do with the recent branch reorganization.

Autoscend recently removed the beta branch. The way this was done was unorthodox: instead of merging beta into master, the maintainers deleted master and renamed beta to the current master. (This is only a guess. @Malibu Stacey could you please confirm this statement?) (edit: this is not true, sorry about the false information.)

It seems that this confused GitHub's infrastructure; reports of install/update problems have spiked since this incident.

I personally had trouble installing autoscend from the old URL (https://github.com/Loathing-Associates-Scripting-Society/autoscend/trunk/release). The first attempt resulted in no files being created, but the second attempt succeeded...and upgrades have been working ever since. Perhaps you could try installing from the old URL instead of the one that ends with "/branches/master/release"?

Edit: Then again, it could also be a problem with OneDrive. I have never used KoLmafia on OneDrive, but I've heard it occasionally causes problems.

 

[fewyn]

Does the problem occur exclusively with autoscend? If so, it probably has to do with the recent branch reorganization.

Autoscend recently removed the beta branch. The way this was done was unorthodox: instead of merging beta into master, the maintainers deleted master and renamed beta to the current master. (This is only a guess. @Malibu Stacey could you please confirm this statement?)

It seems that this confused GitHub's infrastructure; reports of install/update problems have spiked since this incident.

I personally had trouble installing autoscend from the old URL (https://github.com/Loathing-Associates-Scripting-Society/autoscend/trunk/release). The first attempt resulted in no files being created, but the second attempt succeeded...and upgrades have been working ever since. Perhaps you could try installing from the old URL instead of the one that ends with "/branches/master/release"?

Edit: Then again, it could also be a problem with OneDrive. I have never used KoLmafia on OneDrive, but I've heard it occasionally causes problems.

That's actually super strange, I've /never/ had an issue with OneDrive before today, used it and before that Google Drive and Dropbox. But I did for giggles I moved it to my desktop and it updated fine. I'll verify the next time it happens but this might have just been chalked up to weirdness with OneDrive?

 

[Malibu Stacey]

I think I've noticed this behavior when a scripter has multiple scripts in the same repo. When they update a script you don't have installed you will get the messages like above but nothing actually changes locally unless there has been an update to the script which you DO have installed. I think...I could be wrong.

All Loathing-Associates-Scripting-Society scripts are in their own repos. We have a team (the aforementioned Loathing-Associates-Scripting-Society) but the team itself has multiple public repos. See https://github.com/Loathing-Associates-Scripting-Society

That's actually super strange, I've /never/ had an issue with OneDrive before today, used it and before that Google Drive and Dropbox. But I did for giggles I moved it to my desktop and it updated fine. I'll verify the next time it happens but this might have just been chalked up to weirdness with OneDrive?

That is not what happened. I merged beta into master as we often do for releases. Beta was then deleted because we had collectively decided it was not benefiting anyone to have all new development happening on that branch as almost all users were using it anyway. Master exists as it has since autoscend was forked from sl_ascend back in October 2019.

 

[philmasterplus]

That is not what happened. I merged beta into master as we often do for releases. Beta was then deleted because we had collectively decided it was not benefiting anyone to have all new development happening on that branch as almost all users were using it anyway. Master exists as it has since autoscend was forked from sl_ascend back in October 2019.

Thanks for clarifying. I apologize for my misguided assumption.

 

[fewyn]

Looks like moving it out of the OneDrive didn't completely resolve this as it has happened again this morning.

Installing default certificate validation...
Sending login request...
Initializing session for boxwhore...
Loading character status...
Refreshing session data...
Synchronizing moon data...
Retrieving character data...
Updating inventory...
Seeing what's still unrestricted today...
Done checking allowed items.
Examining Meat in closet...
Updating closet...
Retrieving quest data...
Retrieving familiar data...
Familiar data retrieved.
Retrieving campground data...
Examining Meat and pulls in storage...
Updating storage...
You are currently a member of Bonus Adventures from Hell
Seeing what's still unrestricted today...
Done checking allowed items.
Session data refreshed.
2 days until Feast of Boris, Mysticism bonus today (not tomorrow).
Using data override: data/mallprices.txt
Checking all SVN projects...
batman-re is at HEAD (r62)
batbrain is at HEAD (r141)
cdrock-TourGuide-branches-Release is at HEAD (r988)
bestbetweenbattle is at HEAD (r43)
clilinks is at HEAD (r6)
drag-n-drop-inventories is at HEAD (r6)
Ezandora-Briefcase-branches-Release is at HEAD (r83)
Ezandora-Bastille-branches-Release is at HEAD (r13)
Ezandora-Choice-Override-branches-Release is at HEAD (r36)
Ezandora-Comb-Beach-trunk-Release is at HEAD (r10)
Ezandora-Detective-Solver-branches-Release is at HEAD (r19)
Ezandora-Far-Future-branches-Release is at HEAD (r40)
Ezandora-Voting-Booth-trunk-Release is at HEAD (r8)
formhtml is at HEAD (r1)
gausie-excavator-trunk-RELEASE is at HEAD (r179)
kolmafiascripts-cfstat is at HEAD (r4)
kolmafiascripts-mafiarecovery is at HEAD (r6)
kolmafiascripts-shop is at HEAD (r10)
Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE needs updating from (r3559) to (r3573)
Loathing-Associates-Scripting-Society-Helix-Fossil-branches-Release is at HEAD (r23)
Loathing-Associates-Scripting-Society-newlife2-trunk-release is at HEAD (r170)
Loathing-Associates-Scripting-Society-philter-trunk-release needs updating from (r690) to (r701)
mafiachit is at HEAD (r702)
mercenarymood is at HEAD (r33)
reference-plus is at HEAD (r6)
smartstasis is at HEAD (r48)
soolar-CONSUME.ash-trunk-RELEASE is at HEAD (r78)
therazekolmafia-canadv is at HEAD (r110)
veracity0-arrrbor-day is at HEAD (r294)
veracity0-beach is at HEAD (r294)
veracity0-garden is at HEAD (r294)
veracity0-gingerbread is at HEAD (r294)
veracity0-meat-farm is at HEAD (r294)
veracity0-vcon is at HEAD (r294)
veracity0-vprops is at HEAD (r294)
winterbay-mafia-wham is at HEAD (r47)
zlib is at HEAD (r51)
Updating all SVN projects...
Validating repo...
Repo validated.
Updating Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE...
C:\Users\fewyn\Desktop\KM\svn\Loathing-Associates-Scripting-Society-autoscend-branches-master-RELEASE
At revision 3573
Done.
Validating repo...
Repo validated.
Updating Loathing-Associates-Scripting-Society-philter-trunk-release...
C:\Users\fewyn\Desktop\KM\svn\Loathing-Associates-Scripting-Society-philter-trunk-release
At revision 701
Done.
Checking for working copy modifications...
No modifications.
Requests complete.

7454 prices updated from https://kolmafia.us/scripts/updateprices.php?action=getmap

I went and checked the files in my SVN directory compared to the latest commit and the changes are not in there.

Attempting to update via tortoisesvn also doesn't seem to be updating it either.

 

[fewyn]

I'm really starting to think this is a Github issue with how SVN interacts with it. Just trying to pull up the commit log with tortoisesvn only shows the latest commit being from May 15th. Which happens to be the same day autoscend beta was merged into master.

 

[MCroft]

So, I haven't seen the problem, but maybe if we think it's an SVN vs GitHub issue, we can have someone who does have the issue execute all update commands outside of mafia via a shell script or bat file. Basically duplcating the svn update/copy to scripts/relay logic.

And probably duplicate their user folder and run the whole thing again in KoLmafia and then diff the two trees, if it fails in one and works in the other.

This will at least lead us towards a resolution to the "which component/combination of components is misbehaving?" question.

 

[fronobulax]

So, I haven't seen the problem, but maybe if we think it's an SVN vs GitHub issue, we can have someone who does have the issue execute all update commands outside of mafia via a shell script or bat file. Basically duplcating the svn update/copy to scripts/relay logic.

And probably duplicate their user folder and run the whole thing again in KoLmafia and then diff the two trees, if it fails in one and works in the other.

This will at least lead us towards a resolution to the "which component/combination of components is misbehaving?" question.

I thought we had already established that the mafia step of copying to /scripts (or whatever) was not a problem.

So the issue is clearly mafia's SVN commands do not get the expected updates from github. What we don't know is whether this is a mafia problem or whether other SVN tools have a similar problem which would make it a problem with using SVN to access github.

 

[MCroft]

I thought we had already established that the mafia step of copying to /scripts (or whatever) was not a problem.

So the issue is clearly mafia's SVN commands do not get the expected updates from github. What we don't know is whether this is a mafia problem or whether other SVN tools have a similar problem which would make it a problem with using SVN to access github.

Yep, but I want whoever takes this bold, experimental step to be able to use their scripts.

 

[philmasterplus]

I tried to check out autoscend using multiple variations of URLs, using SlikSVN 1.12.0 on Windows 10 x64:

• https://github.com/Loathing-Associates-Scripting-Society/autoscend/: Success.
  Note that /branches/ only contained two subdirectories: combat/ and superposition/.
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/: Success, same note as above
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/combat/: Success
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/combat/RELEASE/: Success
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/master/: E170000 URL doesn't exist.
  I attempted a second checkout but didn't work, either.
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/master/RELEASE/: E235000: file '..\..\..\subversion\libsvn_wc\wc_db.c' at line 1648: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/master/BUILD/: E235000: file '..\..\..\subversion\libsvn_wc\wc_db.c' at line 1648: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/trunk/: Success
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/trunk/RELEASE/: E170000 URL doesn't exist
• https://github.com/Loathing-Associates-Scripting-Society/autoscend/trunk/release/: E170000 URL doesn't exist
• https://github.com/Loathing-Associates-Scripting-Society/philter/: Success.
  Note that /branches/ contains several subdirectories, but neither master/ nor main/ exist.
• https://github.com/Loathing-Associates-Scripting-Society/philter/branches/master/: E170000 URL doesn't exist
• https://github.com/Loathing-Associates-Scripting-Society/philter/branches/main/: E170000 URL doesn't exist
• https://github.com/Loathing-Associates-Scripting-Society/philter/trunk/release/: Success
• https://github.com/pastelmind/svn-checkout-test/: Success
  Note that /branches/ is empty (main is the only branch).
  (This is a throwaway repo I created right now to check if new repositories are affected)
• https://github.com/pastelmind/svn-checkout-test/trunk/foo/: Success

The tests involving Philter and svn-checkout-test suggest that two problems are at play.

• <org>/<repo>/branches/master/ and <org>/<repo>/branches/main/ are no longer accessible for all repositories
• <org>/<repo>/trunk/ is usually accessible, but the autoscend repo is broken

Edit: Added some more tests which changed my hasty conclusion

 

[philmasterplus]

Here's a crazy thing. I already have autoscend checked out under <kolmafia dir>/svn/Loathing-Associates-Scripting-Society-autoscend-trunk-RELEASE, sometime before all this fiasco started.

Today, I ran svn update and it detected a new revision, but failed to update:

Spoiler: KoLmafia gCLI

 > svn update
Checking all SVN projects...
AllenTuring-KoL-MineVolcano-trunk is at HEAD (r61)
bale-counterchecker is at HEAD (r23)
cdrock-TourGuide-branches-Release is at HEAD (r988)
Ezandora-Comb-Beach-trunk-Release is at HEAD (r10)
Ezandora-Detective-Solver-branches-Release is at HEAD (r19)
Ezandora-Far-Future-branches-Release is at HEAD (r40)
Ezandora-Voting-Booth-trunk-Release is at HEAD (r8)
gausie-excavator-trunk-RELEASE is at HEAD (r179)
Loathing-Associates-Scripting-Society-autoscend-trunk-RELEASE needs updating from (r3600) to (r3639)
Loathing-Associates-Scripting-Society-display3-branches-release is at HEAD (r113)
Loathing-Associates-Scripting-Society-Helix-Fossil-branches-Release is at HEAD (r23)
Loathing-Associates-Scripting-Society-newlife2-trunk-release is at HEAD (r170)
pastelmind-awesome-menu-lib-trunk-release is at HEAD (r16)
Loathing-Associates-Scripting-Society-philter-trunk-release is at HEAD (r719)
pastelmind-BetterTrainer-trunk-release is at HEAD (r254)
pastelmind-theme-randomizer-trunk-release is at HEAD (r4)
RESPRiT-bountiful-branches-release is at HEAD (r26)
plasticlobster-kol-skills-branches-Release is at HEAD (r20)
rlbond86-mafia-scripts-auto_mushroom-trunk is at HEAD (r38)
slyz-nemesis is at HEAD (r12)
therazekolmafia-eatdrink is at HEAD (r56)
therazekolmafia-canadv is at HEAD (r110)
zlib is at HEAD (r51)
Updating all SVN projects...
Validating repo...
The requested repo failed validation.  Complain to the script's author.
Done.

But when I ran svn update directly from the system terminal, it updated just fine:

Spoiler: terminal output

$ svn update
Updating '.':
UU   scripts\autoscend\auto_consume.ash
UU   scripts\autoscend\auto_util.ash
UU   scripts\autoscend\quests\level_12.ash
Updated to revision 3639.

Of course, I had to manually copy the new files over to my scripts/ dir.

$ cp -r svn/Loathing-Associates-Scripting-Society-autoscend-trunk-RELEASE/scripts/* scripts/

 

[MCroft]

So, I tried to check out autoscend into a new, blank directory from the command line and failed interestingly.

Michaels-MBP:gittesting mcroft$ svn checkout https://github.com/Loathing-Associates-Scripting-Society/autoscend/branches/master/RELEASE/
svn: E235000: In file 'subversion/libsvn_wc/wc_db.c' line 1648: assertion failed (SVN_IS_VALID_REVNUM(changed_rev))
Abort trap: 6

Michaels-MBP:rel mcroft$ svn --version
svn, version 1.14.0 (r1876290)
   compiled Oct 23 2020, 13:19:41 on x86_64-apple-darwin19.6.0

So I googled that error and it's not new: Subversion Users Mailing list, 2018 and 2014 and 2012
This is what a SVN committer thought the problem was in 2014:

github.com is sending the committed-rev entry property without a
revision:

<S:set-prop name="svn:entry:committed-rev"></S:set-prop>\n

This is a problem with github's svn proxy

There are other comments about different versions of SVN either also failing or claiming success, but not checking things out. It's probably wrong for it to work anywhere.

Still don't know how to fix it, but it sure looks like PM+'s theory is correct.