Bug - Fixed Irregular handling of floats in modifier_eval

[Theraze]

> ash modifier_eval(to_string(to_float(.01)))

Returned: 0.01

> ash modifier_eval(to_string(to_float(.00001)))

Expression syntax error for 'modifier_eval()': expected end, found E-5
Returned: 1.0

> ash to_string(to_float(.00001))

Returned: 1.0E-5

Basically, even though modifier_eval outputs a float, if you give it a float that has been transformed into scientific notation due to being really big or small, modifier_eval takes it not as a number but as a mixed number/text string. If it stays as a number, it just outputs the float. It would be very nice if these floats actually returned the same number that I put into it consistently.

I'm currently mostly seeing this affect scripts that employ BatBrain. Since monsters' chance to hit can drop VERY low at times, it occasionally goes to scientific notation and the computation fails...

 

[Catch-22]

I'm currently mostly seeing this affect scripts that employ BatBrain. Since monsters' chance to hit can drop VERY low at times, it occasionally goes to scientific notation and the computation fails...

I've seen this too, and whilst I don't have a patch, I can offer you an undesirable workaround...

> ash modifier_eval(to_float(to_string(to_float(0.00001))))

Returned: 0.0

I say undesirable, because you lose precision but if something has a 0.00001% chance to hit, it might be easier to just treat that as 0%.

 

[Theraze]

Yeah... that causes other problems... like this:

> ash (to_float(to_string(to_float(1000000000))))

Returned: 0.0

as mafia uses scientific notation both for very large and very small numbers... anything under .001 or over 9,999,999 gets wiped out with that. Not exactly a valid choice if numbers matter.

 

[Winterbay]

And also, if you try to divide by it you'll get wonderful "division by zero"-errors...

 

[Catch-22]

Due to the behaviour of Java, the string representation of a float will be notated, and due to a behaviour of KoLmafia expressions, the uppercase E is currently reserved for effects.

Rather than rewriting a bunch of stuff, wouldn't it be easier to implement a to_decimal function in ASH and then use modifier_eval(to_decimal("0.00001"))?

Just a thought...

 

[Catch-22]

So.. After reading how modifier_eval actually works, I realized it's pretty trivial to fix

Not broken, just a little misunderstood, haha.

> ash modifier_eval(replace_string(to_string(to_float(.00001)), "E", "*10^"))

Returned: 1.0E-5

> ash modifier_eval(replace_string(to_string(to_float(11111111)), "E", "*10^"))

Returned: 1.111111E7

 

[Theraze]

It's trivial to fix, if you know that you're just dealing with a straight float and you aren't checking on effects or actually doing anything complicated. It's not trivial to fix when you're dealing with an unknown amount of calculations and so forth...

I'd just like the function that returns a float to be able to handle receiving a float string as input without having it choke.

 

[Catch-22]

I'd just like the function that returns a float to be able to handle receiving a float string as input without having it choke.

The trouble with that is there's no such thing as a "float string" only a string that could be interpreted as a float. Java has some pretty specific guidelines as to what a string which can be interpreted as a float should look like. Unfortunately, KoLmafia's internal Expression interpreter already reserves "E" and "e" for specific purposes, unrelated to the Exponent symbol used in scientific notation. Unless I am over-complicating things (which, admittedly, I have a habit of doing), then implementing what you're asking is going to break backwards compatibility with other valid KoLmafia expressions.

I've offered a work-around because the "bug" (more like a feature request) you've outlined may potentially prove quite difficult to implement. Take note that you shouldn't replace all instances of "E" in your modifier with "^10*", only those which are from strings that should be interpreted as floats, that way you're not breaking anything that should be interpreted as a literal "E" for active effects. If you wanted to be extra sure, you could always enclose it in parenthesis, like so:

_myFloatString = "(" + replace_string(to_string(_myFloat), "E", "*10^") + ")";

If you think of this as "escaping" your exponent symbol, it makes much more sense

 

[Theraze]

I'm more thinking of the current implementation in zlib, which does this:

float eval(string expr, float[string] values) {
   buffer b;
   matcher m = create_matcher("\\b[a-z_][a-zA-Z0-9_]*\\b", expr);
   while (m.find()) {
      string var = m.group(0);
      if (values contains var) m.append_replacement(b, values[var].to_string());
   }
   m.append_tail(b);
   m = create_matcher("[a-z]",b);
   vprint("Evaluating '"+b.to_string()+"'...",10+(m.find() ? 0 : 1+to_int(is_integer(b))));
   return modifier_eval(b.to_string());
}

This gets used repeatedly in BatBrain to work through arrays of expressions, floats, and things that need to run through modifier_eval. Some of which may include effects. Many of which may involve small or large numbers.

 

[roippi]

This is indeed a quirk of the toString() method of java floats, which I particularly dislike. Scientific notation should never be forced upon the user, as it is nontrivial to parse, especially when arbitrarily mixed with other notations.

There are methods for dealing with this. The DecimalFormat class can construct a formatter to output in whatever format we want. I won't personally be implementing that though - I have zero familiarity with the ash back-end and I have enough on my plate right now.

 

[jasonharper]

Modifier expressions don't handle scientific notation, solely because I was under the impression that mafia never output floats in that format - did that change at some point, or was I just mistaken? In any case, there's no possible ambiguity between 'E' in a float literal, and 'E' as an effect count - they are never both valid at the same time. Think about it - if there was an actual problem here, then every single programming language that supports floats in scientific notation would have to disallow 'E' as a variable name.

Replacing 'E' with '*10^' is far more broken than just the possibility of changing an 'E' that's not an exponent - if the number was itself used on either side of an exponentiation, there would be operator precedence issues.

>>> 2 ** 1E3
1.0715086071862673e+301

>>> 2 ** 1*10**3
2000

>>> 1E3 ** 2
1000000.0

>>> 1*10**3 ** 2
1000000000

You'd have to wrap the entire number in parentheses to avoid that.

 

[Catch-22]

I'm currently mostly seeing this affect scripts that employ BatBrain. Since monsters' chance to hit can drop VERY low at times, it occasionally goes to scientific notation and the computation fails...

Okay, so I'm sick of looking at BatBrain, but there's floating point precision problems throughout this library. Whilst your issue with the handling of floats in expressions may be a valid one, it's arising out of accuracy issues deep within BatBrain and/or Zlib, KoLmafia handling these floats correctly isn't going to fix that.

The issue you're experiencing specifically comes out of monster_stat("hp") in BatBrain.ash, this line here:

      case "hp": return adj.dmg[$element[none]] + max(1.0, (m == last_monster() ? monster_hp() : monster_hp(m)));

For whatever whack reason, adj.dmg[$element[none]] seems to return a number with "too much precision", by that I mean the number has too many decimal places. From what I can tell, this comes out of some float accuracy errors somewhere, so instead of getting a value like -1 for adj.dmg[$element[none]], you get -0.99999994. To make things simple, let's say I'm fighting a monster with 1hp, the adjusted HP value becomes (1.0-0.99999994) which you might think is 0.00000006, but due to our old friend float, this actually becomes 5.9604645E-8. Much later down the track, modifier_eval() eventually gets called on this number, guess what happens?

> ash modifier_eval("5.9604645E-8")

Expression syntax error for 'modifier_eval()': expected end, found E-8
Returned: 5.9604645

So now you've gone from a number which for all intents and purposes should equal 0, to a number which is closer to 6. Henceforth, all the calculations done on this number are going to be wrong. I stopped reading through the logs shortly after this point, but 5.9604645 appears later on down the track in reference to "mp".

[NORMAL] <- 0.0
   Eval: fact
   Set: 0.33
   AREF: record advevent
      Key #1: "mp"
      [NORMAL] <- "mp"
   AREF <- 5.9604645
      Operator: *
         Operand 1: 5.9604645
         [NORMAL] <- 5.9604645
         Operand 2: 0.33
         [NORMAL] <- 0.33
      <- 1.9669534
   ASET: 1.9669534
[NORMAL] <- 1.9669534

If you want to have some fun, get on a decent level account, go to Noob Cave, turn debugging on and try to kill a crate with a combat script that uses BatBrain.

 

[Theraze]

Believe the number you're giving is probably due to fumble/crit chances. If you use attack, and you aren't using something like song of battle, you're looking at a chance your 'normal' damage will miss, regardless of how much it would do if you actually hit.

In other words, that's accurate... mafia's handling of floats in modifier_eval still isn't. As jason said.

 

[Catch-22]

Believe the number you're giving is probably due to fumble/crit chances. If you use attack, and you aren't using something like song of battle, you're looking at a chance your 'normal' damage will miss, regardless of how much it would do if you actually hit.

In other words, that's accurate... mafia's handling of floats in modifier_eval still isn't. As jason said.

Look, all I'm trying to say is that there's problems with BatBrain that won't become apparent until you fix this "bug", the errors have been obscured because eval(); hasn't been working with scientific notation in floats formatted as strings. The errors in BatBrain relate to the accuracy problems inherent in using floats for lots of the calculations in the first place.

As I can't seem to convince you, I went ahead and fixed the eval(); function in ZLib.ash to prove it to you.

float eval(string expr, float[string] values) {
   buffer b;
   matcher m = create_matcher("\\b[a-z_][a-zA-Z0-9_]*\\b", expr);
   while (m.find()) {
      string var = m.group(0);
      if (values contains var) m.append_replacement(b, values[var].to_string());
   }
   m.append_tail(b);
   m = create_matcher("[a-z]",b);
   vprint("Evaluating '"+b.to_string()+"'...",10+(m.find() ? 0 : 1+to_int(is_integer(b))));
   expr = b.to_string();
   buffer c;
   m = create_matcher("-{0,1}[0-9]\.[0-9]{7}E-{0,1}[0-9]{1,3}", expr);
   while (m.find()) {
      m.append_replacement(c, "("+m.group().replace_string("E", "*10^")+")");
   }
   m.append_tail(c);
   return modifier_eval(c.to_string());
}

Drop that over the top of eval() in ZLib.ash (starts line 162 in my file), now you can use scientific notation in eval(); (that is the custom function in ZLib that calls modifier_eval();) to your hearts content. Enjoy!

 

[Theraze]

I see what you're saying about the tiny numbers... unlike you, I see them as there on purpose. I'm saying that those "problems" are actually there due to attack not having a 100% chance to avoid fumbling. Which means that an adjusted damage chance does have that tiny chance to fail, causing tiny little numbers suggesting that using attack isn't as good as a 0 cost killing skill.

And while we can patch that function and hope never to need to evaluate something with an effect in BatBrain, first time that happens, your function modification fails. Hard.

Anyways, jason has said it's a mafia bug, I'm going to stop arguing with you about it for now and distracting the issue, since we apparently can't reach agreement whether or not attack has a fumble chance.

 

[Catch-22]

And while we can patch that function and hope never to need to evaluate something with an effect in BatBrain, first time that happens, your function modification fails. Hard.

Okay, you didn't even try the patch or you would've seen that effects get parsed properly and BatBrain.ash is still broken.

Do yourself a favour, try 0.06*0.02 in a calculator, now try "ash (0.02*0.06)" in the CLI. If your feature request ever gets honored before BatBrain gets fixed maybe you'll finally believe me.

 

[StDoodle]

[voco type='moderator']
Let's keep it civil, everybody. This isn't Gameplay-Discussion.
[/voco]

 

[jasonharper]

The problems with floats that get output in scientific notation go far beyond modifier expressions; you wouldn't be able to store them in a map file, and you wouldn't be able to let the user edit them in a relay script, because NOTHING in mafia will accept a float in scientific notation. Accordingly, r11042 causes all ASH floats to print in plain decimal format, no matter how many digits that takes.

This will have to be revisited if we ever switch from Java floats to doubles, since a double in plain decimal format can take over 300 digits.

 

[Catch-22]

Accordingly, r11042 causes all ASH floats to print in plain decimal format, no matter how many digits that takes.

Thanks Jason. So now the question is: Why are we even using floats?

> ash to_float("0.1")

Returned: 0.10000000149011612

 

[Winterbay]

Thanks Jason. So now the question is: Why are we even using floats?

> ash to_float("0.1")

Returned: 0.10000000149011612

What would you suggest instead for numbers with decimals?