Has AI gotten any better at reading since 2019?

 …2019 being the year that Gary Marcus and Ernest Davis published Rebooting AI and discussed the inability of AI to handle passages like this one, from Laura Ingalls Wilder’s Farmer Boy:

Almanzo turned to Mr. Thompson and asked, “Did you lose a pocketbook?” 

Mr. Thompson jumped. He slapped a hand to his pocket and fairly shouted. “Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?” 

“Is this it?” Alamanzo asked. 

“Yes, yes, that’s it!” Mr. Thompson said, snatching the pocketbook. 

He opened it and hurriedly counted the money. He counted the bills over twice… 

Then he breathed a long sigh of relief and said, “Well this durn boy didn’t steal any of it.”

They find that AI fails to answer questions like:

• Why did Mr. Thompson slap his pocket with his hand?
• Before Almanzo spoke, did Mr. Thompson realize that he had lost his wallet?
• What is Almanzo referring to when he asks “Is this it?”
• Who almost lost $15,00?
• Was all of the money still in the wallet?

So I fed Gemini the same passage and asked it the same questions.

Gemini did well on the first question:

The passage describes Mr. Thompson being worried about his lost pocketbook. After Almanzo asked him if he had lost a pocketbook, the text says that “Mr. Thompson jumped. He slapped a hand to his pocket and fairly shouted.” This suggests that Mr. Thompson slapped his pocket to check if his pocketbook was still there. In other words, he was likely worried that he had lost it.

It didn’t do so well on the second: 

The passage doesn’t explicitly tell us if Mr. Thompson realized his wallet was missing before Almanzo spoke. Here’s why: The passage starts with Almanzo asking about a lost “pocketbook,” not mentioning Mr. Thompson’s awareness. Mr. Thompson’s reaction (jumping and slapping his pocket) could be due to either suddenly realizing it’s missing or simply confirming a nagging suspicion. There isn’t enough information to definitively say whether Mr. Thompson knew beforehand.

In its answer to the third question, it’s slightly off, identifying “it,” rather than “this,” as the item that refers to the wallet in Almanzo’s hand (as opposed to the wallet that was lost):

In the passage, when Almanzo asks “Is this it?”, he’s most likely referring to the pocketbook he holds (or possibly points to) that belongs to Mr. Thompson. 

Here’s why: 

The conversation revolves around a lost “pocketbook” that Almanzo mentions first. Mr. Thompson reacts strongly to the question, suggesting he recognizes the object as his missing belonging. He then snatches the “it” in question and confirms it’s his lost pocketbook. Therefore, “it” refers to the physical pocketbook Almanzo has presented to Mr. Thompson.

In its answer to the fourth and fifth questions, Gemini gets back on track:

From the passage, it was Mr. Thompson who almost lost $1,500. The text says he had that amount in the “pocketbook” and reacted with relief upon finding it.

...

The passage suggests that all the money was still in the wallet. Here’s why: Mr. Thompson snatches the wallet back and hurriedly counts the money. He counts the bills “over twice,” implying a thorough check. Finally, he breathes a sigh of relief and says, “Well this durn boy didn’t steal any of it.” This sequence suggests Mr. Thompson found all the money he was expecting to find, leading to his relief. There’s no mention of missing money.

But when I asked it a fifth question that doesn’t appear in Rebooting AI, things fell apart:

Unfortunately, the passage doesn’t give us a clear answer about who Mr. Thompson refers to as “this durn boy.” 

Here’s why: 

A general statement of relief without a specific boy in mind. There’s no mention of any other boys in the scene. 

It’s unclear if Mr. Thompson suspected someone (possibly a boy) of stealing the wallet before Almanzo found it. His relieved statement, “Well this durn boy didn’t steal any of it,” could be: 

• Directed at himself for possibly misplacing the wallet and fearing a boy might have found it.
• A general statement of relief without a specific boy.