Thoughts on organization of a KMail Parser

[Potatoman]

I've been working on a bot script for my clan for some time now, and I keep getting stuck on KMail parsing - not on how I can parse it, but on what would be the most optimal/efficent way to go about it.

Right now, it is built with recursion, so it will parse 10 KMails at the same time (or however many are in the inbox). They're returned into 10 Maps.

The problem is, I would like to build a functionally-similar parser that can be called from a "library", and I was wondering how I should go about it.

Can you program a function to return a map? If so, how should I organize it to pull in multiple KMails? Also (I haven't worked with Javascript), how can I delete messages?

Right now, the code is:

# kmail_message1[1] = "This is the Player ID#";
# kmail_message1[2] = "this is the Username";
# kmail_message1[3] = "This is the timestamp";
# kmail_message1[4] = "This is the message";

void kmail_extractor()
{
	messages_html = visit_url( "messages.php" );

	# Now we do the do-once parsing of the pwd value
	sewerpackbot_pwd_start = index_of( messages_html , "name=pwd value=\"" ) + 16;
	sewerpackbot_pwd_end = index_of( messages_html , "\"><input type=hidden", sewerpackbot_pwd_start );
	print( sewerpackbot_pwd );
	
	kmail_recursion_counter = 1;
	
	while( kmail_extraction_termination != 1 )
	{
		# Extraction of the selected message's data follows. MESSAGE_SKIP is a value that is ajusted to enable the recursion.
		kmail_playerid_start = index_of( messages_html , "showplayer.php?who=" , message_skip ) + 19;
		kmail_playerid_end = index_of( messages_html , ">" , kmail_playerid_start ) - 1;
		
		if( kmail_playerid_start >= kmail_playerid_end )
		{
			print( "The KMail extractor has failed while attempting to extract the PlayerID" );
			#FIGURE OUT WHAT TO DO HERE TO RECTIFY PROBLEM
		}
		
		kmail_playerid = substring( messages_html , kmail_playerid_start , kmail_playerid_end );

		print( kmail_playerid );
		
		kmail_username_start = kmail_playerid_end + 1;
		kmail_username_end = index_of( messages_html , "</a>" , kmail_username_start );
		kmail_username = substring( messages_html , kmail_username_start , kmail_username_end );
		
		print( kmail_username );
		
		kmail_timestamp_start = index_of( messages_html , "</b><!--" , message_skip ) + 8;
		kmail_timestamp_end = index_of( messages_html , "--><br><blockquote>" , kmail_timestamp_start );
		kmail_timestamp = substring( messages_html , kmail_timestamp_start , kmail_timestamp_end );
		
		print( kmail_timestamp );
		
		kmail_message_start = index_of( messages_html, "--><br><blockquote>" , message_skip ) + 19;
		kmail_message_end = index_of( messages_html, "</blockquote></td", kmail_message_start );
		kmail_message = substring( messages_html, kmail_message_start, kmail_message_end );
		
		print( kmail_message );
		
		message_skip = index_of( messages_html , kmail_message_end , "<input type=checkbox name=" );
		#SOME SORT OF ERROR PROTECTION NECESSARY HERE!
		
		if ( kmail_recursion_counter == 1 )
		{
			kmail_message1[1] = kmail_playerid;
			kmail_message1[2] = kmail_username
			kmail_message1[3] = kmail_timestamp;
			kmail_message1[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 2 )
		{
			kmail_message2[1] = kmail_playerid;
			kmail_message2[2] = kmail_username
			kmail_message2[3] = kmail_timestamp;
			kmail_message2[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 3 )
		{
			kmail_message3[1] = kmail_playerid;
			kmail_message3[2] = kmail_username
			kmail_message3[3] = kmail_timestamp;
			kmail_message3[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 4 )
		{
			kmail_message4[1] = kmail_playerid;
			kmail_message4[2] = kmail_username
			kmail_message4[3] = kmail_timestamp;
			kmail_message4[4] = kmail_message;
		}		
		if ( kmail_recursion_counter == 5 )
		{
			kmail_message5[1] = kmail_playerid;
			kmail_message5[2] = kmail_username
			kmail_message5[3] = kmail_timestamp;
			kmail_message5[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 6 )
		{
			kmail_message6[1] = kmail_playerid;
			kmail_message6[2] = kmail_username
			kmail_message6[3] = kmail_timestamp;
			kmail_message6[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 7 )
		{
			kmail_message7[1] = kmail_playerid;
			kmail_message7[2] = kmail_username
			kmail_message7[3] = kmail_timestamp;
			kmail_message7[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 8 )
		{
			kmail_message8[1] = kmail_playerid;
			kmail_message8[2] = kmail_username
			kmail_message8[3] = kmail_timestamp;
			kmail_message8[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 9 )
		{
			kmail_message9[1] = kmail_playerid;
			kmail_message9[2] = kmail_username
			kmail_message9[3] = kmail_timestamp;
			kmail_message9[4] = kmail_message;
		}
		if ( kmail_recursion_counter == 10 )
		{
			kmail_message10[1] = kmail_playerid;
			kmail_message10[2] = kmail_username
			kmail_message10[3] = kmail_timestamp;
			kmail_message10[4] = kmail_message;
		}
		
		
		kmail_recursion_counter = kmail_recursion_counter + 1;
		
		if( kmail_recursion_counter == 11 )
		{
			kmail_extraction_termination = 1;
		}
	}
}

But that seems unnecessarily messy. So, I'll be working on revising it. Just wondering what the most efficent way to organize it would be...

 

[Camber]

Have you looked at the kmail functions in zlib written by zarqon? He has written a kmail parser and other functions. They should help you.

http://kolmafia.us/showthread.php?t=2072

 

[Bale]

You might also get some good ideas from zarqon's registry script. It parses kmails and returns them as a map, much as you envision. This is the format it converts kmail into:

record message {
   string sender;
   string date;
   int id;
   string text;
};
   message[int] mail = parse_mail();

 

[Potatoman]

You might also get some good ideas from zarqon's registry script. It parses kmails and returns them as a map, much as you envision.

I looked at that originally (and just now), and couldn't help but be confused out of my mind.

I suppose my (real) question would be better phrased as:

I am making a kmail parser for a clan bot. I would like it to grab all 10 KMails in the inbox at once, but be a separate (library) function. How should I structure the KMail data so it can be carried over from the called function to the main script with ease? How would you go about the storing of parsed KMails, evaluation of access privileges, and execution of commands?

Right now what I have written works (mostly), but it is messy. Mostly because as my ASH skills get better, I see more and more of my folly in my (twice re-written) script.

 

[Bale]

Is your problem, a lack of understand about how records work?

message[int] mail; // Is how the data structure is defined
mail[1].sender     // is the sender of a mail
mail[1].text       // is that mail's text

And so on...

 

[Potatoman]

Is your problem, a lack of understand about how records work?

message[int] mail; // Is how the data structure is defined
mail[1].sender     // is the sender of a mail
mail[1].text       // is that mail's text

And so on...

Wow, now that all makes so much more sense! Thanks! Never realized you could define maps/records like that before. Back to work now, I guess...

Thanks for the suggestion of the registry script, it will be probably be a big help!

 

[Potatoman]

Quick Questions:
1) Is there any way to cycle or search the "record" results of a map? What if it is a record[string]?
2) Is there a way to delete a key (and all associated results) from a map?

EDIT: Oh, and thanks a ton Bale, using them has simplified my life a hundred-fold!

 

[Bale]

1) foreach key in mail
This will traverse a map checking each item in that map. You can then refer to: mail[key].text, mail[key].sender
It doesn't matter if the index to the map is an int, string, item or whatnot.
I'm assuming that this is what you meant. If I'm wrong, then please explain in more detail.

2) remove mail[key];
Does exactly what you're asking. It will remove that entry from the map.

 

[zarqon]

The foreach loop can also work like this:

foreach key, msg in mail

Then you can just use msg.text, msg.sender.

I'm sure you probably also want to parse items/meat, which the Registry script doesn't. I think mredge posted a modified version somewhere that also parses items and meat as separate record fields, although it doesn't check for fakes.

Another way would be to parse that out from the text when you process the message. I use the same parsing functions as the Registry script in my clan's StashBot, but the items/meat are parsed from the message text in the message-processing function. Here are the relevant bits for getting items/meat:

boolean process_message(message m) {
//   ...unrelated stuff removed
   int inmeat;
   int[item] inthings;
   if (contains_text(m.text,"<center><table>")) {
      inthings = extract_items(substring(m.text,index_of(m.text,"<center><table>")));
      inmeat = extract_meat(substring(m.text,index_of(m.text,"<center><table>")));
   }
   if (inthings contains $item[none]) return error("Unknown item found!");
// ...

Note that this requires a bit of KoL-added HTML which appears before the items/meat appear.

 

[DaMaster0]

This is all zarqon's work from various scripts. I take none of the credit. I just grouped them together. Will not work if you include zlib in your program though... Zarqon, tell me if you want this removed for any reason.

(oh and by the way, i'm still working on delete_mail)

 

[Potatoman]

1) foreach key in mail
This will traverse a map checking each item in that map. You can then refer to: mail[key].text, mail[key].sender
It doesn't matter if the index to the map is an int, string, item or whatnot.
I'm assuming that this is what you meant. If I'm wrong, then please explain in more detail.

I feel stupid, but what I meant was searching like this:

record cabbage {
int patch;
string happy;
}

cabbage[string] magic;
magic[sally].patch = 5;
magic[sally].happy = "Mooo!";
magic[betsy].patch = 78;
magic[betsy].happy = "Mooo!";

Is there any way I can run a 'search' that will return me the string (keys) that link to the happy value "Mooo!"?

 

[Veracity]

record cabbage {
  int patch;
  string happy;
};

cabbage [string] magic;
magic[ "sally" ] = new cabbage( 5, "Mooo!" );
magic[ "betsy" ] = new cabbage( 78, "Mooo!" );

foreach key, rec in magic {
  if ( rec.happy == "Mooo!" ) {
    print( "magic key " + key + " says Mooo!" );
  }
}

 

[zarqon]

How are you planning to remotely alter the pack contents? Mafia's item type should still work fine for you, given that there are plenty of ways to convert between items/strings/item id numbers:

int to_int( item )
item to_item( int )
string to_string ( item ) <-- often unneccessary
item to_item( string )

Given this, I'd say at least you have no need to store both the item id and the item name in your record.

I'm glad my mail-parsing work has been even more helpful to fellow scripters than just the Registry script. As far as I'm concerned, once I've shared a script it's public domain, so no worries about "ripping" me off, DaMaster. All that I share I share in the spirit of collaboration. Feel free to modify, improve, repurpose, reverse engineer... anything except saying you wrote it or trying to sell it, I suppose.

 

[Potatoman]

I'm glad my mail-parsing work has been even more helpful to fellow scripters than just the Registry script. As far as I'm concerned, once I've shared a script it's public domain, so no worries about "ripping" me off, DaMaster. All that I share I share in the spirit of collaboration. Feel free to modify, improve, repurpose, reverse engineer... anything except saying you wrote it or trying to sell it, I suppose.

You guys are credited in it, and honestly, all of your scripts were a great help (and inspiration) in writing it.

Right now (besides editing my posts) I'm changing over to a straight and narrow, run-of-the-mill ClanBot from the absurd SPB project I had been working on. So I'll be a lot happier, and my clannies won't be getting a bunch of overly-complicated features that they'll neither use nor fund.

All the advice (and teaching) has been an amazing help to me; I'd hope to be able to release my own small library of parsing functions (for others benefits) soon, although they'd probably be rather simple in loop or nature.