ZLib -- Zarqon's useful function library

[MCroft]

It is idiomatic around here that "latest build" and "up to date" refer to the builds available from https://ci.kolmafia.us/ and not what is available at SourceForge.

There is a mechanism for script writers to indicate that a particular daily build (or newer) is required, the since directive. As noted above zarqon failed to update that when a zlib update was pushed, but that error has since been corrected.

Users who wish to protect themselves from similar oversights by script authors tend to do so by always using the "latest build" as defined above. The days when the daily builds were buggy and unstable are long gone and since 5 months have passed since the SourceForge build was last updated the commitment to keeping SourceForge up to date might be wavering.

Your steps to try and resolve the issue were perfectly reasonable although ultimately incorrect

While bugs are possible (and I can point to several, including mine), KoLmafia daily builds are likely to do a better job keeping up with KoL changes. Anything introduced by KoL since KoLmafia 20.7 (July 2020) is not going to be there. Especially at Crimbo time, there are KoL changes that will affect play.

We've done some work to make releases more efficient so we can get on a better cadence, but the process is still a bottleneck.

And sourceforge has a number of things that are out of date on it, especially in the documentation. It still lists Java 1.5 as the required version.

 

[scudunculus]

Your steps to try and resolve the issue were perfectly reasonable although ultimately incorrect

Thanks for confirming I'm not crazy (at least not in that regard!).

It is idiomatic around here that "latest build" and "up to date" refer to the builds available from https://ci.kolmafia.us/ and not what is available at SourceForge.

...

Users who wish to protect themselves from similar oversights by script authors tend to do so by always using the "latest build" as defined above. The days when the daily builds were buggy and unstable are long gone and since 5 months have passed since the SourceForge build was last updated the commitment to keeping SourceForge up to date might be wavering.

I suppose then the real problems then are:
> that the version information that gets displayed is not precise enough (20.7 is not enough, it needs build number to be displayed somewhere)
> and that all indications point to sourceforge as the source of the latest builds. From the "version information" and the "kolmafia thread" link, you get directed to sourceforge. The "kolmafia forums" link does get you here but you might just see that the highest numbered thread title is 20.7 and not click through to see daily builds. From the wiki landing page, you probably would look first at the 'installation' page, where the first link you see is to sourceforge.

In any case, this seems more of a problem with mafia itself, not specifically zlib. Sorry if I'm coming across as accusatory, I just wanted to point out a thing that would avoid confusion for other people down the line. The problem as I saw it was just that I was under the mistaken impression that I already had the latest version... until I looked closer.

 

[fronobulax]

Thanks for confirming I'm not crazy (at least not in that regard!).



I suppose then the real problems then are:
> that the version information that gets displayed is not precise enough (20.7 is not enough, it needs build number to be displayed somewhere)
> and that all indications point to sourceforge as the source of the latest builds. From the "version information" and the "kolmafia thread" link, you get directed to sourceforge. The "kolmafia forums" link does get you here but you might just see that the highest numbered thread title is 20.7 and not click through to see daily builds. From the wiki landing page, you probably would look first at the 'installation' page, where the first link you see is to sourceforge.

In any case, this seems more of a problem with mafia itself, not specifically zlib. Sorry if I'm coming across as accusatory, I just wanted to point out a thing that would avoid confusion for other people down the line. The problem as I saw it was just that I was under the mistaken impression that I already had the latest version... until I looked closer.

The daily build clearly shows the revision number on the top of the login page and when displaying Help->Copyright Notice->Version Info. So this is really a documentation and user education problem, exposed by a script author's omission.

So....

You might consider two things. One is that you can edit the wiki https://wiki.kolmafia.us/index.php/Main_Page so that there is new or corrected information. Maybe the next person who doesn't understand the distribution model, as it is actually practiced, will not not get so confused. If you don't have permission to edit I think @fewyn can grant it.

Second, creating a Bug Report that identifies incorrect and confusing information on SourceForge might lead to some changes. There are no guarantees that anyone will take action but it is also true that politely squeaky wheels are sometimes greased.

Finally, remember that Kolmafia and scripts are maintained by volunteers who are doing it for reasons of their own and you are already getting your money's worth

 

[MCroft]

The daily build clearly shows the revision number on the top of the login page and when displaying Help->Copyright Notice->Version Info. So this is really a documentation and user education problem, exposed by a script author's omission.

So....

You might consider two things. One is that you can edit the wiki https://wiki.kolmafia.us/index.php/Main_Page so that there is new or corrected information. Maybe the next person who doesn't understand the distribution model, as it is actually practiced, will not not get so confused. If you don't have permission to edit I think @fewyn can grant it.

I'd like to include the pretty graphic from Jenkins on the wiki homepage. Probably also with a less-pretty link to sourceforge...

 

[zarqon]

I've learned my lesson!

And now I've learned it twice!

 

[philmasterplus]

So! In r47 I've adjusted URL-based version checking to use the new forum URL.

This is not working. I keep getting warnings like this:

Checking for updates (running auto_mushroom ver. 2.2)...
Server returned response code 404 (Not Found) for https://kolmafia.us/threads/t=9854
Unable to load current version info.

The current version (r48) checks https://kolmafia.us/threads/t=<Thread ID>. However, the correct URL is https://kolmafia.us/threads/<Thread ID>.

I'm submitting a patch for this that also fixes the "latest post" URL (line 226).

P.S. It seems that https://kolmafia.us/showthread.php?t=<Thread ID> also works.

 

[fronobulax]

This is not working. I keep getting warnings like this:

Checking for updates (running auto_mushroom ver. 2.2)...
Server returned response code 404 (Not Found) for https://kolmafia.us/threads/t=9854
Unable to load current version info.

The current version (r48) checks https://kolmafia.us/threads/t=<Thread ID>. However, the correct URL is https://kolmafia.us/threads/<Thread ID>.

I'm submitting a patch for this that also fixes the "latest post" URL (line 226).

P.S. It seems that https://kolmafia.us/showthread.php?t=<Thread ID> also works.

I'm going to push back on this and suggest you check your environment.

From today

AUTO_MUSHROOM already run today!
Next event in 8 days: eight gloomy black mushrooms picked
Checking for updates (running auto_mushroom ver. 2.2)...
Running auto_mushroom version: 2.2 (current)

and none of the other scripts that had been giving update errors has done so for me today.

You might also consider deleting or editing data/zversions.txt or confirming that you really did update zlib.

 

[zarqon]

@frono: I corrected this earlier today in r49. You're both right!

 

[3BH]

We have had multiple users reporting errors to the effect of: "Stack overflow during ASH script: (zlib.ash, line 375)". on discord. It seems to be tied to there being no vars_defaults.txt or vars_<user>.txt

Did anything change related to that recently?

Editing to add actual useful context because my above explanation sucked:
The issue is when zlib tries to vprint but are unable to because getvar("verbosity") sends them into a loop with vprint, since they cant find any variables because vars_defaults doesnt exist. The actual line number will vary depending on when mafia actually gets fed up with the getvar()->vprint() loop its found itself in.

 

[zarqon]

Not with ZLib, no. I believe there have been some changes to KoLmafia regarding opening data files though. If you can find a way to replicate the issue it's probably worth posting a bug report.

 

[MCroft]

The smallest use-case I saw in the online discussion was

>zlib vars

Stack Overflow during ASH script: (zlib.ash, line 379)

I thought it might be a problem creating vars_defaults.txt, but when I removed mine, it created a new one.

 

[zarqon]

Editing to add actual useful context because my above explanation sucked:
The issue is when zlib tries to vprint but are unable to because getvar("verbosity") sends them into a loop with vprint, since they cant find any variables because vars_defaults doesnt exist. The actual line number will vary depending on when mafia actually gets fed up with the getvar()->vprint() loop its found itself in.

Just saw your edit. From my viewing of the code I'm still at the "that can't possibly happen" stage of debugging -- ZLib calls setvar("verbosity") as its first top-level command of any relevance, which updates the relevant maps before calling vprint(). And I'm not seeing how vprint() could be called before that. I guess changing line 379 to a regular print() statement would fix that, but I'm confused as to why it would be necessary.

 

[3BH]

Just saw your edit. From my viewing of the code I'm still at the "that can't possibly happen" stage of debugging -- ZLib calls setvar("verbosity") as its first top-level command of any relevance, which updates the relevant maps before calling vprint(). And I'm not seeing how vprint() could be called before that. I guess changing line 379 to a regular print() statement would fix that, but I'm confused as to why it would be necessary.

I agree with your assessment, I was super confused on this when two different people posted about it in discord, took a while to narrow down what I thought was happening, and I could still be wrong. You can reproduce the error (even though I understand that this isnt really a real use case) by removing verbosity from vars_defaults.txt and trying to run zlib vars (provided you dont have an entry in vars_<user>.txt) . I do not think your script is the ultimate culprit here because like croft, when I deleted vars_default and going through these same steps it created a vars_default.txt file for me and worked as intended.

 

[fronobulax]

I cannot reproduce the error using zlib vars and various combinations of vars_<user> and vars_<default> being present or absent. Could the people who have it provide line counts of the various files and and view the files in a text editor and confirm there is nothing but plain text?

 

[matt.chugg]

Ran into this issue myself, and have some futher information.

I was starting a nerw "install" of KoLmafia, whole new directory so I could run a multi side by side without any risk of conflict, I installed a few basic scripts, of which zlib is a dependancy. Out of the box any script with zlib as a dependancy or "zlib vars" in the CLI gives error:

Stack overflow during ASH script: (zlib.ash, line 379)

Reading this thread and since it's a new "install" of KoLMafia it made sense that there was an issue with the vars_defaults.txt or vars_who.txt file,
I checked the data directory and they hadn't been created. Adding vars_defaults.txt solved the problem. zlib vars now worked correctly.

but...

I then deleted the created files and ran zlib vars again, but this time it created the files just fine. so I ran some futher tests:

Test #1

I renamed the whole data directory and let KoLmafia recreate it and ran zlib vars again,

Same error:

Stack overflow during ASH script: (zlib.ash, line 379)

creating the vars_defaults.txt file manually again fixed it.

Test #2


I renamed the whole data directory and let KoLmafia recreate it and ran zlib vars again,

Same error:

Stack overflow during ASH script: (zlib.ash, line 379)

I manually created a random filename called fu.bar,txt in the data directory and rand zlib vars again

FIXED!

Concolusions

1: This is likely to be a windows-only issue

2: This isn't necesarily a zlib issue although it might be possible to add errorhandling for the files not being created or being missing.

3: (conjecture) KoLMafia creates the data directory with some kind of filesystem flag that prevents it being written to by it's own process, this flag is then somehow cleared by interacting with the directory manually allowing KoLMafia to work with the file correctly.

Notes

Using windows 10,full admin - Java 1.8.0_271-b09

 

[matt.chugg]

I realised I normally run from dropbox which could be relevant, but was able to use the following steps to reproduce:

1: create new empty directory
2: copy latest kolmafia .jar release to directory and run
3: install zlib from SVN
4: execute "zlib vars" in CLI

 

[MCroft]

I realised I normally run from dropbox which could be relevant, but was able to use the following steps to reproduce:

1: create new empty directory
2: copy latest kolmafia .jar release to directory and run
3: install zlib from SVN
4: execute "zlib vars" in CLI

I was able to reproduce on my MacOS device as follows:

1. rename data to data.temp
2. launch kolmafia in debug mode
3. execute "zlib vars" in CLI

lather, rinse, repeat.

What we're seeing is a stack overflow because vprint attempts to access "verbosity". If there's no verbosity, it throws an error and tries to vprint it.

I made some changes to it and it gave me this a lot of times: attempt to access missing script setting "verbosity".

When I removed the verbosity check completely (hardcoded to '24'), I got this:

Updating use_for_items.txt to '2019-07-19T21:02:42+00:00'...
...use_for_items.txt updated.
New default value for ZLib int setting: verbosity => 3
New default value for ZLib boolean setting: automcd => true
New default value for ZLib int setting: threshold => 4
New default value for ZLib int setting: unknown_ml => 170
New default value for ZLib familiar setting: is_100_run => none
New default value for ZLib string setting: defaultoutfit => current
Copy/paste/modify/enter any of the following lines in the CLI to edit settings:

So, I don't know why it *does* work sometimes. It sometimes fails in getvars and sometimes in vprint because they depend on each other.

I know that you need a default that doesn't depend on something that might fail in something that's part of your error handling routines.

 

[MCroft]

What I tried:

string getvar(string varname) {   // this should replace using a direct vars[] lookup
   if (vars contains varname) return vars[varname];
   if (vardefaults contains varname) return vardefaults[varname].val;
   if (! (vars contains "verbosity") ){
      setvar("verbosity",3);
   }
   vprint("Attempt to access missing script setting '"+varname+"'!","purple",4);

   return "";
}

 

[zarqon]

Now that multiple people have reported this issue, I've changed the vprint() line to a regular print(), which should prevent the stack overflow. However, this does not fix the underlying issue, which I don't yet fully understand. ZLib has changed very little in the last few years and this always used to work.

I have an unverified suspicion that it may be related to this bug report. Perhaps ZLib needs to explicitly specify the data directory?

 

[MCroft]

So here's the logic I see:

1: something calls getvar("x")
2: getvar("x") checks for "x" in vars and vardefaults
3: if not found, it calls vprint("error message")
4: vprint calls getvar("verbosity")
5: getvar("verbosity") checks for "verbosity" in vars and vardefaults
6: if not found, it calls vprint("error message")
7: goto 4, until the stack runs out of room.

calling print instead of vprint works, but you lose the advantage of log levels.

My very tentative theory is that it's a race condition, and the stack doesn't fill up unless the data directory needs to be created before the bare-bones default file is created. I don't think it has anything to do with that file-to-map bug or it would never work to create a file.