Data entry: would it be possible to make it internal data?

E

efilnikufecin

Member
A few of the scripters hanging around here have written monstrous data maps to incorporate into their scripts to give info like fullness, stat gains, and adventure gains from food...inebriety, stat gains, and adventure gains from booze...and even more like data about equipment.

A lot of information became available to users of kolmafia when location stats was added to the main interface. I'm betting this was a data entry nightmare at first...even *if* the data was obtained from another source.

In my recent thoughts about this I found my way to the following page: http://kolmafia.sourceforge.net/credits.html which is a list of people who helped out in various ways with the development of kolmafia. Some of these people provided data. For all the data contained in kolmafia the list does not show there being many contributors.

[quote author=holatuwol link=topic=263.msg1129#msg1129 date=1150568263]
My ever-short thank-you list of data contributors is conveniently stored on the KoLmafia Sourceforge website, and I'd feel rather silly copy-pasting that list here. [/quote]

Looks like holatuwol might just be of the same opinion.

[quote author=holatuwol link=topic=250.msg1250#msg1250 date=1151636018]
A lot of people seem to be writing ASH libraries which equate to access to data that KoLmafia manages anyway through the course of doing what it does.  So, I decided I'd create this thread to address this -- why waste time writing giant libraries and maps for data KoLmafia already has?

If you have the suspicion that there's information that KoLmafia uses in order to do its internal data management, post the kind of information you think KoLmafia has access to, give a name for the ASH version of the method and give justification for why this is a good name for that method.  My hope is that anyone can READ ASH scripts to find out what they're doing and I prefer highly "literate" function names.[/quote]

If a person reads that last quote, and knows where I am going with this, then they could possibly read that partial post I quoted in more than one way.

It could be saying "I am not willing to make anything else internal data unless it needs to be internal data." I prefer the notion that it is saying "I'm just tired of doing all this excess data entry but I will give you access to the data that's already their and any future data that gets entered." In the latter case there is an opening for someone to step up and do some data entry.

another quote: This time off sight so I can't seem to get the full functionality of the quote tags. Here's the link: http://kolmafia.sourceforge.net/credits.html
PoolAce (#7977): I was told recently by Elazar that PoolAce was the person who created the most wonderful recipe table ever. KoLmafia's original item creation capabilities are owed almost completely to KoL Helper. So, if you'd like to send a thank you for KoLmafia's item creation engine, I recommend sending a thank you to PoolAce.
Click to expand...

There's some data someone else provided. I believe Holatuwol had to do the actual data entry in kolmafia readable format, but the data was collected by someone else at a minimum.

I'm going to quote myself here, but the message I am quoting never got posted so I figure it should be alright to do so...
[quote author=Veracity link=topic=322.msg1735#msg1735 date=1154325685]
You're going to have to make your own "fullness" map. And keep it up to date as new food items are added. And make sure you don't eat outside of the script that keeps track of the counter. :)
Click to expand...

OK so keeping track of the counter is simple enough. I have an issue with monstrous data maps though. I try to have way too much info in them, and my computer ends up bulking down really bad. On the other hand some of the data maps I have written, or obtained from other scripters to use have been implemented. That on top of the access to internal data thread http://kolmafia.us/index.php/topic,250.msg1250.html#msg1250 and the fact that someone said at one point "Programming I don't mind, I just don't like to do data entry" (or the equivelent) makes me wonder...

If one of us offered to do the data entry in a format specified by you or holatuwol would it increase the likelyhood of having the fullness data, and other potentially usefull data implemented into kolmafia? It's a big project, but some of us script writers would tackle and complete it without even thinking twice. It's been done for ash maps, and I see no reason why myself or another scripter couldn't do it for an internal file. [/quote]

I really hope you managed to read all the way down to this point :)
 
Hahah well with all my ASH maps all over the place if anyone wants to tell me some specific format you would like the maps converted to tab delimited (or whatever you call them tab separating ones... I'm self taught so technical jargon I wouldn't know >>) I'll be happy to convert it. Although as you know my maps are generally wiki leeched so yeah. ^^;;
 
Initial data has to come from somewhere. Once the data is entered errors will be found...typical of programming, but should be easy to fix. Anyway that makes 2 offers to enter data :)
 
Or, we could just give ASH scripters the ability to write their own tab-delimited data files and load them into aggregates/maps via script. This reduces initialization delay currently experienced, makes scripts more readable, and avoids the, "You need to come up with a nice, friendly GUI way of utilizing and/or representing that data" question I'd likely pose (and annoy everyone with) for every single data file that gets proposed.

What do you think?
 
[quote author=holatuwol link=topic=324.msg1746#msg1746 date=1154345542]
Or, we could just give ASH scripters the ability to write their own tab-delimited data files and load them into aggregates/maps via script.  This reduces initialization delay currently experienced, makes scripts more readable,
continued next quote
[/quote]

my thoughts on the cause of the slow execution time had to do with string lists. Normally a seek or search function in a string list starts at the top of the list. If my understanding is correct the import command adds the entire contents of the imported script where the import function is called. (May be wrong I am assuming). Every time a jump is executed (function call or function end) the native computer code searches from the first string in the list to find the line that it is jumping too. Crap why bore everyone with details. You know all this. Simply put it's way too long of a string list with all the initialization in it, and yes the idea mentioned above should solve that problem...on the other hand...

[quote author=holatuwol link=topic=324.msg1746#msg1746 date=1154345542]
and avoids the, "You need to come up with a nice, friendly GUI way of utilizing and/or representing that data" question I'd likely pose (and annoy everyone with) for every single data file that gets proposed.

What do you think?
[/quote]

I still love the location details pane! A lot of people do. I was thinking that an item details pane in the kol encyclopedia would be almost as good of an addition to kolmafia. It might be even more liked. More information than kol itself's built in item description database, and reduce the server hits to the item description database because more info is available. Ehh copy the entire item description database to kolmafia? Maybe, but it might be too much. Enter all sorts of great info about each item into a database inside kolmafia except the full text descriptions? Sounds like a great plan to me!
 
[quote author=efilnikufecin link=topic=324.msg1747#msg1747 date=1154350036]
my thoughts on the cause of the slow execution time had to do with string lists. Normally a seek or search function in a string list starts at the top of the list. If my understanding is correct the import command adds the entire contents of the imported script where the import function is called. (May be wrong I am assuming). Every time a jump is executed (function call or function end) the native computer code searches from the first string in the list to find the line that it is jumping too.[/quote]
No.

ASH parses your script into a "parse tree". It only reads the actual text of your script once. After that it executes the parse tree rather than rescanning strings of script.

Calling user functions -  and finding variables - does require looking them up in a symbol table, which means, currently, doing a linear search of function names as strings in an array. (Symbol tables used to be sorted lists, which have faster lookup, but when polymorphic functions were added, they were changed.)

But this is done only at parse time; when it is executing the parse tree, the function body of a user-defined function (or the internal method of a built-in function) is right there ready to be executed.

What makes for the slow startup would, in fact, be reading thousands of lines of text, parsing it, and doing all the data initialization. That is what the "read data from a tab delimited file and populate maps" suggestion addresses: it eliminates building a parse tree and executing assignment statements for filling the maps with data.
 
[quote author=Alexander Daychilde link=topic=324.msg1755#msg1755 date=1154387233]
I think that sounds rather spiffy!
[/quote]I concur.
 
So I've added the following two functions to the ASH, which will be available in the next release (which will probably happen shortly after I do another revamp to the buff purchasing frame):

void file_to_map( string file_to_load, map map_to_fill )
void map_to_file( map map_to_save, string file_to_write )

The first is designed to implement the proposal I suggested earlier in this thread and the second is designed to help people transition away from massive map loading by simply adding a simple function call to do the conversion.
 
[quote author=holatuwol link=topic=324.msg1770#msg1770 date=1154499452]
So I've added the following two functions to the ASH, which will be available in the next release (which will probably happen shortly after I do another revamp to the buff purchasing frame):

void file_to_map( string file_to_load, map map_to_fill )
void map_to_file( map map_to_save, string file_to_write )

The first is designed to implement the proposal I suggested earlier in this thread and the second is designed to help people transition away from massive map loading by simply adding a simple function call to do the conversion.
[/quote]

sweet!
 
[quote author=holatuwol link=topic=324.msg1770#msg1770 date=1154499452]
So I've added the following two functions to the ASH, which will be available in the next release (which will probably happen shortly after I do another revamp to the buff purchasing frame):

void file_to_map( string file_to_load, map map_to_fill )
void map_to_file( map map_to_save, string file_to_write )[/quote]

Would it be possible to have these functions return a boolean about whether or not it succeeded? (Or is there some other way to determine success programmatically?)
 
I think I found a bug with loading certain kinds of maps in a function. It seems that in some cases, map values don't fall out of scope like they should. I've attached a self-explanatory repro case. For what it's worth, I tried reproducing this with a float[int] map and it didn't work. Perhaps it's only an issue with keys being items...
 

Attachments

[quote author=picklish link=topic=324.msg1865#msg1865 date=1155409017]
Would it be possible to have these functions (file_to_map and map_to_file) return a boolean about whether or not it succeeded?[/quote]

Done.

[quote author=picklish link=topic=324.msg1873#msg1873 date=1155434575]
I think I found a bug with loading certain kinds of maps in a function.  It seems that in some cases, map values don't fall out of scope like they should.[/quote]

Interesting.

Code: 
void inner( int i )
{
    int[int] map;

    print("Entering function with local map");
    print("map has " + count(map) + " keys in it." );
    foreach key in map
        print( key + "->" + map[key] );

    print("Adding key " + i );
    map[i] = 2 * i;

    print("map has " + count(map) + " keys in it." );
    foreach key in map
        print( key + "->" + map[key] );
    print("Leaving function.");
}

inner(5);
inner(10);

yields:

> test.ash
Entering function with local map
map has 0 keys in it.
Adding key 5
map has 1 keys in it.
5->10
Leaving function.
Entering function with local map
map has 1 keys in it.
5->10
Adding key 10
map has 2 keys in it.
5->10
10->20
Leaving function.
Script succeeded!
Click to expand...

Entering a function with local variables does reinitialize all of the variables to the default value (or to the initializer the user specified), but for aggregates, that "default value" is not re-calculated each time you enter the function.

OK, I fixed this.

 >test.ash
Entering function with local map
map has 0 keys in it.
Adding key 5
map has 1 keys in it.
5->10
Leaving function.
Entering function with local map
map has 0 keys in it.
Adding key 10
map has 1 keys in it.
10->20
Leaving function.
Script succeeded!
Click to expand...

and

>mapError.ash
This should be 0
booze.map[chorizo taco] = 0
This should be 2
food.map[chorizo taco] = 2
his should be 0...but isn't
booze.map[chorizo taco] = 0
This should be 0
booze.map[chorizo taco] = 0
Script succeeded!
Click to expand...
 
Awesome, some have even started posting the needed files. Now I ponder the idea and wonder what everyone elses oppinion of URL_to_map(string URL) would be?

This idea would allow for working together as a group to create the map files, and keep them up to date as they need updating, and everyone would benefit when something new is added.

In turn I can see a bad side in that if someone meeses up the files, it will mess them up for everyone using them so I am iffy on the idea. ??? We'd also need reliable webspace to keep the data.
 
The wiki is probably not a good place for it, given that it's all HTML.  However, in the event that you guys ever find another use for it, file_to_map will now recognize URLs as valid file names.  As a side note, because of the way things are written, it is theoretically possible to read internal data files and transform them into maps by using "data/filename.dat" as your filename.  Admittedly, however, most data files weren't written to be usable in that way. :)

Update: In order to avoid mixing up KoLmafia's internal data files, character settings with player-made data maps, the next release of KoLmafia will ask that all paths to maps are relative to the JAR file, rather than relative to the data directory. If necessary, KoLmafia will create intermediate directories while writing files.
 
It doesn't seem quite friendly to put files on the wiki that aren't human-readable. Also, each file would be its own page, and that could get long in a hurry.

My recommendation would be that we continue to attach files here and then people can use the URL of that attachment to access the file. If there is a concern about global maintainability, then there can be some sort of understanding that mods can come in and freshen a file if they determine that it is warranted.
 
Ooh, attachments. Didn't think of that. Given that we have code samples on the wiki, it might not be a bad idea to have "map" examples that are actually attachments. Might me better on mafia's loading capabilities to not have to deal with redirects that may (or may not) require logins, since some attachments aren't viewable unless logged in.
 
You must log in or register to reply here.
Back
Top