Make your scripts automatically update their data files!
- Thread starter zarqon
- Start date
Banana Lord
Member
Reposting a summary of this post here. If a minion feels like tidying things up by deleting/copying/merging/voodoo magic that's fine by me.
I can't seem to upload a particular file to the map manager site, even though it's a plaintext .txt file. Other files work fine, but not this one, so I'm ever so slightly confused. Theraze confirmed that she can't upload the file either, but that she can (like me) upload other files without a problem. eegee gave some sort of confirmation saying "I've tried different file encodings, EOLs, and browsers. I wasn't able to upload a .txt.", but I'm not sure if he meant he wasn't able to upload ANY .txt files, or just HAR_Options.txt. If the former then something weird's going on, if the latter then something weird's going on with that file. Either way I can't work out what the issue is so any help would be much appreciated (I don't want to have to type it out by hand! :O).
Re the previous (and ancient) post: But it'd run on Bootcamp
And I thought musicians were keen on Macs? I'm genuinely curious, I have over 10 years of musical experience but I've never had cause to use a computer (I favour a more... traditional approach to music, y'know, one that doesn't involve whining synthesizers and junky adaptors . I've also never had to use something along the lines of Sibelius, so I guess I'm either doing pretty well or pretty poorly, depending on how you look at it).
I can't seem to upload a particular file to the map manager site, even though it's a plaintext .txt file. Other files work fine, but not this one, so I'm ever so slightly confused. Theraze confirmed that she can't upload the file either, but that she can (like me) upload other files without a problem. eegee gave some sort of confirmation saying "I've tried different file encodings, EOLs, and browsers. I wasn't able to upload a .txt.", but I'm not sure if he meant he wasn't able to upload ANY .txt files, or just HAR_Options.txt. If the former then something weird's going on, if the latter then something weird's going on with that file. Either way I can't work out what the issue is so any help would be much appreciated (I don't want to have to type it out by hand! :O).
Re the previous (and ancient) post: But it'd run on Bootcamp
And I thought musicians were keen on Macs? I'm genuinely curious, I have over 10 years of musical experience but I've never had cause to use a computer (I favour a more... traditional approach to music, y'know, one that doesn't involve whining synthesizers and junky adaptors . I've also never had to use something along the lines of Sibelius, so I guess I'm either doing pretty well or pretty poorly, depending on how you look at it).
Last edited:
zarqon
Well-known member
Fixed! I figured it out -- somehow this file was being submitted with a mime type of "text/x-lisp" which wasn't allowed. In fact, I'd only been allowing "text/plain", and I noticed a few other data files with substandard mime types as well, so whoever's scripts are using those files should be happy now too.
Also, thanks to this problem I fixed another bug -- failure to accept a data file still wasn't deleting the temp file. You may have seen the preponderance of strange data file names that included timestamps, or maybe not. Anyway, that's fixed too.
Also Banana, with no computer how do you record music? I don't use a computer to perform, but I use the crap out of it for recording/mixing. Sonar is a fabulous, powerful, and intuitive audio production suite, but I need it to make full use of my hardware -- running it inside a virtual machine would just add more latency to the system. Hence, no Mac.
Also, thanks to this problem I fixed another bug -- failure to accept a data file still wasn't deleting the temp file. You may have seen the preponderance of strange data file names that included timestamps, or maybe not. Anyway, that's fixed too.
Also Banana, with no computer how do you record music? I don't use a computer to perform, but I use the crap out of it for recording/mixing. Sonar is a fabulous, powerful, and intuitive audio production suite, but I need it to make full use of my hardware -- running it inside a virtual machine would just add more latency to the system. Hence, no Mac.
Banana Lord
Member
Sweet! Thanks! 
The only times I've had to record myself playing someone's been available with a mic and all the other necessary equipment (ie: someone else's always handled that end, and I don't usually record). Bootcamp isn't a virtual machine. It helps you partition your hard drive and configure the new partition so that it's ready for you to install windows on it. The only hitch is that, once you have windows up and running on one partition and OSX up and running on the other, you have to restart to switch between OSs.
The only times I've had to record myself playing someone's been available with a mic and all the other necessary equipment (ie: someone else's always handled that end, and I don't usually record). Bootcamp isn't a virtual machine. It helps you partition your hard drive and configure the new partition so that it's ready for you to install windows on it. The only hitch is that, once you have windows up and running on one partition and OSX up and running on the other, you have to restart to switch between OSs.
zarqon
Well-known member
Just FYI: Yesterday I changed nameservers for my domain, so there may be a few hiccups as the change works its way around the internet. Everything is already working fine for me here in Korea, but I did notice one or two people noting hangs on version checking in other threads. If it isn't already resolved, the change should be fully resolved within a day or so.
Bale
Minion
Zarqon, according to some users on the KoL forum, the map manager is refusing Russian IP addresses. Sounds like your provider got proactive with spammers again and blocked too large an IP range.
Discussion starts here.
Could you get him to fix it?
Discussion starts here.
Could you get him to fix it?
zarqon
Well-known member
As mentioned in that thread, since a huge majority of hacking attempts come from Russian IP's and none of his clients care about their sites being accessible in Russia, he was hoping he could just block Russia and breathe easier. I'm evidently his most international hostee.
We've re-opened the domain up to Russia, for now. The Map Manager may be moving to a VPS if Russian hacking efforts continue apace.
We've re-opened the domain up to Russia, for now. The Map Manager may be moving to a VPS if Russian hacking efforts continue apace.
https://certbot.eff.org/ if you aren't already using this it's a good idea. (I'm going to be enabling this soon on kolmafia.us)
zarqon
Well-known member
Map Manager Announcement
I am sorry to announce that -- as has happened in the past but was later undone -- my host has again implemented a country block. I'm not sure exactly which countries are being blocked, but it's at least Russia. I received a note apologetically informing me of the decision and citing as a reason that the "ransomware problem is getting ridiculous these days." I'm his only client with an international userbase so I understand and support his decision to better protect his paying clients.
I'm also sorry but I'm not willing to pay for additional hosting just for a few people to use my scripts for free, when I already have free hosting that works for me and most others. So this leaves us with several options:
1) I believe users who find themselves blocked can use proxy/VPN? This puts the onus on users to figure it out though and it won't work "out of the box" for those users. Pretty sure Opera has a built-in VPN, which is another great "browser first" from them (I'd thought they were done with those). If anyone tries it or another solution, please let me know if it works to get around the block.
b) Move the Map Manager to kolmafia.us or some other volunteer hosting solution. In the past it was suggested to move it to kolmafia.us but I wasn't keen on giving up my access to editing the relevant scripts. However, now that some users are blocked I'm willing to see it moved where it can best serve the most people. I haven't actually edited the Map Manager code in years anyway.
III) Dissolve the Map Manager and move all relevant data files to the SVN repositories associated with the scripts that use them. This would be a bit of a project and would break old scripts the moment the files were removed from the current location. It would also remove the collaborative nature of these files, which used to be a big deal but these days I don't see many people updating data files there.
All of these solutions involve parties besides myself. So, what do you think, other parties?
I am sorry to announce that -- as has happened in the past but was later undone -- my host has again implemented a country block. I'm not sure exactly which countries are being blocked, but it's at least Russia. I received a note apologetically informing me of the decision and citing as a reason that the "ransomware problem is getting ridiculous these days." I'm his only client with an international userbase so I understand and support his decision to better protect his paying clients.
I'm also sorry but I'm not willing to pay for additional hosting just for a few people to use my scripts for free, when I already have free hosting that works for me and most others. So this leaves us with several options:
1) I believe users who find themselves blocked can use proxy/VPN? This puts the onus on users to figure it out though and it won't work "out of the box" for those users. Pretty sure Opera has a built-in VPN, which is another great "browser first" from them (I'd thought they were done with those). If anyone tries it or another solution, please let me know if it works to get around the block.
b) Move the Map Manager to kolmafia.us or some other volunteer hosting solution. In the past it was suggested to move it to kolmafia.us but I wasn't keen on giving up my access to editing the relevant scripts. However, now that some users are blocked I'm willing to see it moved where it can best serve the most people. I haven't actually edited the Map Manager code in years anyway.
III) Dissolve the Map Manager and move all relevant data files to the SVN repositories associated with the scripts that use them. This would be a bit of a project and would break old scripts the moment the files were removed from the current location. It would also remove the collaborative nature of these files, which used to be a big deal but these days I don't see many people updating data files there.
All of these solutions involve parties besides myself. So, what do you think, other parties?
Last edited:
the dictator
Member
I'm in the Netherlands and I'm blocked, so it might be entirety of Europe. That said, I wasn't using any of the maps anyway (as far as I know), so it's not a big loss.
So, while all of the above options sound good, like I said, I'm not using the maps (at least not in my own scripting), so I don't really care if I can access them or not.
What I would like is if the map manager call was made optional, maybe contingent on actually trying to access one of these maps. Right now, when I import zlib to use say, it's kmail functions or its variable management, the script halts for half a minute waiting for the timeout, which is slowing down my automation, AND confusing me into thinking I didn't actually call my script yet. I *think* it's because of the static declaration on line 542, which is not included in the has_goal() function call that follows it.
Edit: yes, moving that static block inside the function remove the slowdown, but I don't know why it wasn't inside in the first place, so it might have mucked up other stuff somewhere. This is just what I managed to find after a quick text search.
So, while all of the above options sound good, like I said, I'm not using the maps (at least not in my own scripting), so I don't really care if I can access them or not.
What I would like is if the map manager call was made optional, maybe contingent on actually trying to access one of these maps. Right now, when I import zlib to use say, it's kmail functions or its variable management, the script halts for half a minute waiting for the timeout, which is slowing down my automation, AND confusing me into thinking I didn't actually call my script yet. I *think* it's because of the static declaration on line 542, which is not included in the has_goal() function call that follows it.
Edit: yes, moving that static block inside the function remove the slowdown, but I don't know why it wasn't inside in the first place, so it might have mucked up other stuff somewhere. This is just what I managed to find after a quick text search.
Last edited:
My experience using free VPNs from within the US is that the free VPNs won't give me a different country of origin. That is what they offer to entice customers to pay. So a free VPN might not help. I have also found that some of the free VPNs don't allow certain protocols which may not be relevant now but could be a concern down the road.
fewyn has been pretty accommodating with files hosted at kolmafia.us so perhaps that is the easiest path to follow.
While I appreciate the work others have done I have never actually edited and updated data files at zachbardon.com. Perhaps a script has done so on my behalf but it wasn't me.
Similarly I have not seen much evidence that multiple scripts are sharing common data files. So making data files into svn dependencies solves the problem at the cost of reducing the pool of maintainers. (As a tangent if a data file is an SVN dependency and a script is in SVN I would really want to see the concept of checking for updates deprecated at the script level. If a user chooses not to update via svn then the script (and author) should not try and second guess that decision by trying to be helpful and checking anyway).
I can pretty much cope with any choice. I have no sense of how many currently unmaintained scripts would break but we managed to rehost UR so we have that precedent to follow if necessary.
fewyn has been pretty accommodating with files hosted at kolmafia.us so perhaps that is the easiest path to follow.
While I appreciate the work others have done I have never actually edited and updated data files at zachbardon.com. Perhaps a script has done so on my behalf but it wasn't me.
Similarly I have not seen much evidence that multiple scripts are sharing common data files. So making data files into svn dependencies solves the problem at the cost of reducing the pool of maintainers. (As a tangent if a data file is an SVN dependency and a script is in SVN I would really want to see the concept of checking for updates deprecated at the script level. If a user chooses not to update via svn then the script (and author) should not try and second guess that decision by trying to be helpful and checking anyway).
I can pretty much cope with any choice. I have no sense of how many currently unmaintained scripts would break but we managed to rehost UR so we have that precedent to follow if necessary.
xKiv
Active member
FWIW, another european country, also likely blocked.
I, of course, don't have all the existing scripts installed, only about 290 ash files, but the only place that directly references the server URL in them is in zlib.ash, as it should be.
Anything that had its own hardwired url probably already broke with the switch from http to https.
I, of course, don't have all the existing scripts installed, only about 290 ash files, but the only place that directly references the server URL in them is in zlib.ash, as it should be.
Anything that had its own hardwired url probably already broke with the switch from http to https.
Duh. I have three references to the site: zlib, Registry and BatMan_RE all of which are Zarqon's
zarqon
Well-known member
Other scripts that import ZLib can and do access that data file, so I'm not moving the declaration inside a function.
What I have just done, however, is make load_current_map() check a given map only once per day regardless of success or failure, rather than only stopping attempts upon success. So you should see the slowdown only once per map per day (not per character; it's a data file shared by all characters) rather than every single time a script tries to load that map (which could make for quite a lot of waiting for users with combat scripts and/or mood scripts and/or beforeBattle scripts that import ZLib). You could, of course, edit the script to remove checking altogether.
@frono: Yes, I think I'll shoot fewyn a message about this. He's been quite accommodating in the past in these matters and would probably be willing provided he doesn't have to get into the nuts and bolts of it himself.
Bale used to add new items and skills to batfactors.txt fairly frequently. But...
Also, BatMan RE's reference doesn't really count because it's just adding a clickable link to see bats!
Rinn
Developer
Maybe you could use cloudflare as dns and use their proxy. It might be easier to convince your host to whitelist the cloudflare ip ranges that are in Europe if they're blocked https://www.cloudflare.com/ips/. I use cloudflare's free services for dns and to proxy all my server web traffic and I like it a lot.