Bug null pointer error while parsing skills

[Crowther]

I got a null pointer error today. The code it happened on is parsing my character's skills.

Two characters work fine in r20539. One character works, one fails in r20544. Debug log attached.

The skills on the character where things fail:

Spoiler

Liver of Steel
Powers of Observatiogn
Gnefarious Pickpocketing
Torso Awaregness (P)
Gnomish Hardigness
Cosmic Ugnderstanding
CLEESH
Summon Snowcone
Summon Hilarious Objects
Summon Candy Heart
Really Expensive Jewelrycrafting
Awesome Balls of Fire
Conjure Relaxing Campfire
Snowclone
Maximum Chill
Eggsplosion
Mudbath
Grease Lightning
Inappropriate Backrub
Raise Backup Dancer
Creepy Lullaby
Summon Crimbo Candy
Volcanometeor Showeruption
Wassail
Fashionably Late
Executive Narcolepsy
Lunch Break
Offensive Joke
Managerial Manipulation
Natural Born Skeleton Killer
Summon "Boner Battalion"
Pinch Ghost
Chip on your Shoulder (HP)
Frigidalmatian
Silent Slice
Splattersmash
Grab a Cold One
Summon Annoyance (HP)
Shrap
Psychokinetic Hug
Alien Source Code
Hollow Leg
Pirate Bellow (HP)
Summon Holiday Fun! (HP)
Rapid Prototyping
Mathematical Precision
Ruthless Efficiency
Summon Carrot (HP)
Garbage Nova
Bear Essence (HP)
Summon Kokomo Resort Pass (HP)
Healing Salve
Dark Ritual
Lightning Bolt
Giant Growth
Ancestral Recall
Calculate the Universe (HP)
Communism!
Steely-Eyed Squint
Eldritch Intellect
Stack Lumps (HP)
Drinking to Drink (HP)
Experience Safari (HP)
Drippy Eye-Beetle (HP)
Tongue of the Walrus (P)
Batter Up! (P)
Pulverize (P)
Ghostly Shell (P)
Reptilian Fortitude (P)
Empathy of the Newt (P)
Tenacity of the Snapper (P)
Wisdom of the Elder Tortoises (P)
Astral Shell (P)
Amphibian Sympathy (P)
Pastamastery (HP)
Leash of Linguini (P)
Transcendental Noodlecraft (HP)
Advanced Saucecrafting (HP)
Elemental Saucesphere (P)
Jalapeño Saucesphere (P)
Master Saucier (P)
The Way of Sauce (P)
Disco Face Stab (P)
Advanced Cocktailcrafting (HP)
Superhuman Cocktailcrafting (P)
Moxie of the Mariachi
Aloysius' Antiphon of Aptitude
The Moxious Madrigal
Cletus's Canticle of Celerity
The Polka of Plenty
The Magical Mojomuscular Melody
The Power Ballad of the Arrowsmith
Brawnee's Anthem of Absorption
Fat Leon's Phat Loot Lyric (P)
The Psalm of Pointiness
Jackasses' Symphony of Destruction
Stevedave's Shanty of Superiority
The Ode to Booze
The Sonata of Sneakiness
Carlweather's Cantata of Confrontation
Ur-Kel's Aria of Annoyance
Dirge of Dreadfulness
The Ballad of Richie Thingfinder
Benetton's Medley of Diversity
Elron's Explosive Etude
Chorale of Companionship
Prelude of Precision
Sing
Donho's Bubbly Ballad
Cringle's Curative Carol
Inigo's Incantation of Inspiration
Dissonant Riff
Cadenza
Crab Claw Technique
Accordion Bash
Accordion Appreciation
Concerto de los Muertos
Five Finger Discount
Suspicious Gaze
Bawdy Refrain
Thief Among the Honorable
Sticky Fingers
Cone of Zydeco
Master Accordion Master Thief
Knowing Smile
Mariachi Memory
Transcendent Olfaction (HP)

 

[gausie]

I got a null pointer error today. The code it happened on is parsing my character's skills.

Two characters work fine in r20539. One character works, one fails in r20544. Debug log attached.

The skills on the character where things fail:

Spoiler

Liver of Steel
Powers of Observatiogn
Gnefarious Pickpocketing
Torso Awaregness (P)
Gnomish Hardigness
Cosmic Ugnderstanding
CLEESH
Summon Snowcone
Summon Hilarious Objects
Summon Candy Heart
Really Expensive Jewelrycrafting
Awesome Balls of Fire
Conjure Relaxing Campfire
Snowclone
Maximum Chill
Eggsplosion
Mudbath
Grease Lightning
Inappropriate Backrub
Raise Backup Dancer
Creepy Lullaby
Summon Crimbo Candy
Volcanometeor Showeruption
Wassail
Fashionably Late
Executive Narcolepsy
Lunch Break
Offensive Joke
Managerial Manipulation
Natural Born Skeleton Killer
Summon "Boner Battalion"
Pinch Ghost
Chip on your Shoulder (HP)
Frigidalmatian
Silent Slice
Splattersmash
Grab a Cold One
Summon Annoyance (HP)
Shrap
Psychokinetic Hug
Alien Source Code
Hollow Leg
Pirate Bellow (HP)
Summon Holiday Fun! (HP)
Rapid Prototyping
Mathematical Precision
Ruthless Efficiency
Summon Carrot (HP)
Garbage Nova
Bear Essence (HP)
Summon Kokomo Resort Pass (HP)
Healing Salve
Dark Ritual
Lightning Bolt
Giant Growth
Ancestral Recall
Calculate the Universe (HP)
Communism!
Steely-Eyed Squint
Eldritch Intellect
Stack Lumps (HP)
Drinking to Drink (HP)
Experience Safari (HP)
Drippy Eye-Beetle (HP)
Tongue of the Walrus (P)
Batter Up! (P)
Pulverize (P)
Ghostly Shell (P)
Reptilian Fortitude (P)
Empathy of the Newt (P)
Tenacity of the Snapper (P)
Wisdom of the Elder Tortoises (P)
Astral Shell (P)
Amphibian Sympathy (P)
Pastamastery (HP)
Leash of Linguini (P)
Transcendental Noodlecraft (HP)
Advanced Saucecrafting (HP)
Elemental Saucesphere (P)
Jalapeño Saucesphere (P)
Master Saucier (P)
The Way of Sauce (P)
Disco Face Stab (P)
Advanced Cocktailcrafting (HP)
Superhuman Cocktailcrafting (P)
Moxie of the Mariachi
Aloysius' Antiphon of Aptitude
The Moxious Madrigal
Cletus's Canticle of Celerity
The Polka of Plenty
The Magical Mojomuscular Melody
The Power Ballad of the Arrowsmith
Brawnee's Anthem of Absorption
Fat Leon's Phat Loot Lyric (P)
The Psalm of Pointiness
Jackasses' Symphony of Destruction
Stevedave's Shanty of Superiority
The Ode to Booze
The Sonata of Sneakiness
Carlweather's Cantata of Confrontation
Ur-Kel's Aria of Annoyance
Dirge of Dreadfulness
The Ballad of Richie Thingfinder
Benetton's Medley of Diversity
Elron's Explosive Etude
Chorale of Companionship
Prelude of Precision
Sing
Donho's Bubbly Ballad
Cringle's Curative Carol
Inigo's Incantation of Inspiration
Dissonant Riff
Cadenza
Crab Claw Technique
Accordion Bash
Accordion Appreciation
Concerto de los Muertos
Five Finger Discount
Suspicious Gaze
Bawdy Refrain
Thief Among the Honorable
Sticky Fingers
Cone of Zydeco
Master Accordion Master Thief
Knowing Smile
Mariachi Memory
Transcendent Olfaction (HP)

Can you supply the HTML for this page instead?

 

[fronobulax]

I had skill errors which I commented on in the RSS feed for 20541. In my case the character with the errors was the never ascended character and I can believe the charpane is different and the code did not account for that.

 

[gausie]

Can you supply the HTML for the page frono

 

[fronobulax]

Can you supply the HTML for the page frono

This?

For some reason my track record for getting HTML on request is not good.

On login it couldn't find Acquire Rhinestones and Prevent Scurvy and Sobriety.

The gCLI actually displayed
Could not find a known, usable skill of yours uniquely matching "1 Acquire Rhinestones"
Could not find a known, usable skill of yours uniquely matching "1 Prevent Scurvy and Sobriety"

both were triggered by a call to use_skill, such as

ok = use_skill(1,$skill[Acquire Rhinestones]);

Probably cannot rule out the upgrade to htmlcleaner. Moving from source to jar shouldn't have made a difference but going from 2.20(?) to 2.24 might have.

 

[philmasterplus]

Unforeseen exceptions!

First, the charsheet.php provided by Fronobulax has this line:

<a onclick='javascript:poop("desc_skill.php?whichskill=17&amp;self=true","skill", 350, 300)'>Summon Hilarious Objects</a>

Where skill ID 17 was formerly used by Summon Hilarious Objects. Apparently skill IDs can move , and some charsheets may show skill IDs that are no longer valid.
(KoL wiki also says that Summon Hilarious Objects is using both skill ID 7226 and 8200, but desc_skill.php?whichskill=8200 returns "No skill found", so I assume it's outdated info).

Secondly, this line:

<a onclick="javascript:skill(19)">Transcendent Olfaction</a> (<b>HP</b>)

Where the game uses a completely different onclick handler. Makes me wonder what the source code for charsheet.php looks like...

The NullPointerException occurred because KoLmafia failed to identify a skill by ID. Either of the above could have been the cause.
I'll need to update the code to handle such edge cases.

 

[philmasterplus]

Here's a patch I tested on r20544 with 3 characters. @fronobulax, could you please test this on your characters as well?

 

[fronobulax]

No debug generated. I don't think there were any skill errors but since they had daily limits and had been cast already I can't be certain whether things were fixed or just never tried. But if I had to make a choice I'd lean towards fixed. We can let this percolate until I can run tomorrow or if someone else has confidence they are welcome to check it in, instead of waiting for me to do it.

 

[Xande1]

Just thought I'd chip in here:

TO isn't showing up in my mafia combat buttons. This is almost certainly a symptom of the weird TO HTML mentioned in post #6, and will be fixed when that parsing is fixed.

I am (or was, it hasn't happened today) getting the could not find message for Triple Size: "Could not find a known, usable skill of yours uniquely matching "1 CHEAT CODE: Triple Size". It was uneven, sometimes logging out and back in would fix the problem. It was happeneing even if I manually equipped the glove, and was happening both with ASH use_skill(), and with CLI cast.

 

[philmasterplus]

Just thought I'd chip in here:

TO isn't showing up in my mafia combat buttons. This is almost certainly a symptom of the weird TO HTML mentioned in post #6, and will be fixed when that parsing is fixed.

I am (or was, it hasn't happened today) getting the could not find message for Triple Size: "Could not find a known, usable skill of yours uniquely matching "1 CHEAT CODE: Triple Size". It was uneven, sometimes logging out and back in would fix the problem. It was happeneing even if I manually equipped the glove, and was happening both with ASH use_skill(), and with CLI cast.

If you don't mind, could you please post the HTML source of your charsheet.php as well? It would help me check whether there are other edge cases I don't know about.

 

[fronobulax]

Here's a patch I tested on r20544 with 3 characters. @fronobulax, could you please test this on your characters as well?

Worked. I won't be able to commit until later today so will check back to see whether I should wait for something new or go ahead.

 

[Crowther]

Sorry, I got busy decorating. Thanks for looking into this. I'm still getting errors with r20547. Here's my character sheet. While this account has ascended, it was a long time ago.

 

[fronobulax]

Sorry, I got busy decorating. Thanks for looking into this. I'm still getting errors with r20547. Here's my character sheet. While this account has ascended, it was a long time ago.

I'm going to interpret this as meaning you had not tried the patch in post 7 above. So I committed it as r20548. I think there is more testing and a couple more cases to be fixed but I guess multiple commits is the way to go, rather than have folks run something thinking there might have been a fix.

So some things have been fixed but it is still open.

 

[philmasterplus]

Sorry, I got busy decorating. Thanks for looking into this. I'm still getting errors with r20547. Here's my character sheet. While this account has ascended, it was a long time ago.

Thank you. Your character sheet contained three "old" skill IDs -- Summon Snowcone (16), Summon Hilarious Snowcone (17), and Summon Candy Heart (18). I expect that these would be handled in the patch I submitted above (reply #7).

Unfortunately, your post revealed to me another problem that is not fixed by reply #7: Softcore permed skills don't follow the pattern I expected from hardcore skills.

<a onclick='javascript:poop("desc_skill.php?whichskill=1010&amp;self=true","skill", 350, 300)'>Tongue of the Walrus</a> (P)<br>

Since the "(P)" text is not wrapped in a <b></b> tag, it won't be picked up by my XPath selector. If I can't come up with a new selector that covers this case, I'll have to resort to using regex...grrr!

Thankfully, this remaining issue only affects the "Skills You Haven't Permed" list shown when you ascend.

 

[philmasterplus]

This patch fixes the issue discussed on reply #14. It should parse softcore-permed skills properly again, so that you won't see them in the "Skills You Haven't Permed Yet" list.

I resorted to manually looping through next sibling nodes and checking the text content. Not ideal, but it should work.

Tested on r20548 (the revision which applies #7 but not this one) with 3 characters of mine. Unfortunately, only one of them can access the pre-ascension page right now, and he doesn't have any permed skills, so I can't really test if the modified code works as intended.

Sorry about this mess, and hope it gets resolved quick.

 

[Xande1]

If you don't mind, could you please post the HTML source of your charsheet.php as well? It would help me check whether there are other edge cases I don't know about.

Your wish is my command. Sorry I didn't see your request last night. Specifically did this with the Powerful Glove equipped, and noticed that even with the glove equipped the CHEAT CODE skills don't appear in the skills list, which might be part of that problem.

I can also post the charsheets of my multis, but due to RL scheduling, that won't happen until late tonight (but they don't have Powerful Gloves), but if you want more data points, let me know.

 

[Crowther]

Thanks for all your work. Yes, I read too fast and didn't try the patch. I was able to log on just now without an error.

 

[fronobulax]

Not done, I think. See https://kolmafia.us/threads/issue-with-hard-drinker-in-recent-daily-builds.25716/

Judgement call whether we want to merge that thread with this one or watch both places.

 

[fronobulax]

Charsheet html at https://kolmafia.us/threads/issue-with-hard-drinker-in-recent-daily-builds.25716/

I'm wondering if this was released too soon? It was done to deal with different skills that have the same name so it has some benefit but it broke a lot, too.

@philmasterplus are you on top of this? Should we revert and buy some time to explore edge cases? Recruit some help? (Not me - XPath is not in my useful skill set and I run from all the regexes I can).

 

[philmasterplus]

There are multiple places in KoLmafia where, in the past, we resorted to checking the character class to work around duplicate skill names. However, most of it didn't solve the issue of have_skill() thinking that the character had the wrong skill.

This bug is caused by one such case. In net/sourceforge/kolmafia/KoLCharacter.java, line 872, we check for the existence of Hard Drinker by skill name, which makes KoLmafia check for the Cow Puncher version of the skill. Previously, this worked because CharSheetRequest.java also parsed Hard Drinker as a Cow Puncher skill. It only surfaced because the new CharSheetRequest.java correctly parsed Hard Drinker by ID--though I can't be 100% certain since @discdeath has yet to provide the HTML for their charsheet.php, and I don't have Hard Drinker.

I maintain that it's better for KoLmafia to parse a correct list of owned skills, rather than parsing a wrong list (due to duplicate names) and working around them.

Currently, the following skills have the same names:

• [7094]Static Shock (none) and [7156]Static Shock (none)
• [138]Lightning Bolt (none) and [16025]Lightning Bolt (none)
• [15025]Hard Drinker (Avatar of Sneaky Pete) and [18008]Hard Drinker (Cow Puncher)
• [11002]Ferocity (Avatar of Boris) and [24017]Ferocity (Vampyre)
• [7329]Hammer Throw (none) and [25001]Hammer Throw (Plumber)
• [7330]Ultra Smash (none) and [25002]Ultra Smash (Plumber)
• [7332]Juggle Fireballs (none) and [25003]Juggle Fireballs (Plumber)
• [7333]Fireball Barrage (none) and [25004]Fireball Barrage (Plumber)
• [7335]Spin Jump (none) and [25005]Spin Jump (Plumber)
• [7336]Multi-Bounce (none) and [25006]Multi-Bounce (Plumber)

I think I'll look up where KoLmafia checks for these skills by name and replace them with skill ID checks.

Edit: See reply #21 for an attempted fix.