On Chalk & Talk--Math and Autism with Anna Stokke

 In which we discuss Direct Instruction, Precision Teaching and--of course!--math instruction meets facilitated communication:

https://chalkandtalkpodcast.podbean.com/

Has AI gotten any better at reading since 2019?

 …2019 being the year that Gary Marcus and Ernest Davis published Rebooting AI and discussed the inability of AI to handle passages like this one, from Laura Ingalls Wilder’s Farmer Boy:

Almanzo turned to Mr. Thompson and asked, “Did you lose a pocketbook?” 

Mr. Thompson jumped. He slapped a hand to his pocket and fairly shouted. “Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?” 

“Is this it?” Alamanzo asked. 

“Yes, yes, that’s it!” Mr. Thompson said, snatching the pocketbook. 

He opened it and hurriedly counted the money. He counted the bills over twice… 

Then he breathed a long sigh of relief and said, “Well this durn boy didn’t steal any of it.”

They find that AI fails to answer questions like:

• Why did Mr. Thompson slap his pocket with his hand?
• Before Almanzo spoke, did Mr. Thompson realize that he had lost his wallet?
• What is Almanzo referring to when he asks “Is this it?”
• Who almost lost $15,00?
• Was all of the money still in the wallet?

So I fed Gemini the same passage and asked it the same questions.

Gemini did well on the first question:

The passage describes Mr. Thompson being worried about his lost pocketbook. After Almanzo asked him if he had lost a pocketbook, the text says that “Mr. Thompson jumped. He slapped a hand to his pocket and fairly shouted.” This suggests that Mr. Thompson slapped his pocket to check if his pocketbook was still there. In other words, he was likely worried that he had lost it.

It didn’t do so well on the second: 

The passage doesn’t explicitly tell us if Mr. Thompson realized his wallet was missing before Almanzo spoke. Here’s why: The passage starts with Almanzo asking about a lost “pocketbook,” not mentioning Mr. Thompson’s awareness. Mr. Thompson’s reaction (jumping and slapping his pocket) could be due to either suddenly realizing it’s missing or simply confirming a nagging suspicion. There isn’t enough information to definitively say whether Mr. Thompson knew beforehand.

In its answer to the third question, it’s slightly off, identifying “it,” rather than “this,” as the item that refers to the wallet in Almanzo’s hand (as opposed to the wallet that was lost):

In the passage, when Almanzo asks “Is this it?”, he’s most likely referring to the pocketbook he holds (or possibly points to) that belongs to Mr. Thompson. 

Here’s why: 

The conversation revolves around a lost “pocketbook” that Almanzo mentions first. Mr. Thompson reacts strongly to the question, suggesting he recognizes the object as his missing belonging. He then snatches the “it” in question and confirms it’s his lost pocketbook. Therefore, “it” refers to the physical pocketbook Almanzo has presented to Mr. Thompson.

In its answer to the fourth and fifth questions, Gemini gets back on track:

From the passage, it was Mr. Thompson who almost lost $1,500. The text says he had that amount in the “pocketbook” and reacted with relief upon finding it.

...

The passage suggests that all the money was still in the wallet. Here’s why: Mr. Thompson snatches the wallet back and hurriedly counts the money. He counts the bills “over twice,” implying a thorough check. Finally, he breathes a sigh of relief and says, “Well this durn boy didn’t steal any of it.” This sequence suggests Mr. Thompson found all the money he was expecting to find, leading to his relief. There’s no mention of missing money.

But when I asked it a fifth question that doesn’t appear in Rebooting AI, things fell apart:

Unfortunately, the passage doesn’t give us a clear answer about who Mr. Thompson refers to as “this durn boy.” 

Here’s why: 

A general statement of relief without a specific boy in mind. There’s no mention of any other boys in the scene. 

It’s unclear if Mr. Thompson suspected someone (possibly a boy) of stealing the wallet before Almanzo found it. His relieved statement, “Well this durn boy didn’t steal any of it,” could be: 

• Directed at himself for possibly misplacing the wallet and fearing a boy might have found it.
• A general statement of relief without a specific boy.

Some possible revelations on what goes on in facilitators’ heads when they facilitate big projects

(Cross-posted at FacilitatedCommunication.org.)

In this post, I return to one of the news stories I included in my FC news roundup from two weeks ago: a story on KQED’s California Report Magazine about a non-speaking 19-year-old who, after a breakthrough at age 16, defied impressions that, as reporter Sasha Khokha puts it, “he didn’t have the motor skills to point to the right answers in school.” The boy, who now goes by Jacob, was, purportedly, suddenly able to slowly type out full sentences with everything spelled correctly.

In one of the accompanying photographs, we can see Jacob engaging in the typical RPM/S2C-style of letter-selection, with someone else holding up a device to his extended index finger. We can also see a reference in the caption to a neurologist who “confirms that even though his father holds the device for him, Jacob types independently.” That neurologist, later identified as Dr. Margaret Bauman, has a history of defending FC, and it’s unclear on what basis she confirmed Jacob’s independence. In fact, the only way to establish independence is through a rigorous message-passing test in which the facilitator is reliably blinded. Bauman has been involved in only one published message-passing test, back in 1996. As we discuss here, that test was very far from rigorous.

Here comes the part I didn’t get to in my earlier post. According to the article, which is titled “Non-Verbal Teen to 'Take on the World' With a Symphony Written in His Head”:

Six months after learning to type, Jacob surprised his parents again. He told them he had a 70-minute symphony in his head… It’s called Unforgettable Sunrise.

Since Jacob can neither sing nor play an instrument (nor, apparently, read music), one of Jacob’s father’s collaborators, composer Rob Laufer, was brought in to translate Jacob’s verbal descriptions of the 6-movement, 70-minute symphony into a musical score on paper. Laufer, purportedly, “was amazed at the precision of Jacob’s vision for the symphony.”

But what, exactly, did it take for Laufer to translate Jacob’s vision onto paper? In the next dozen paragraphs I’m going to embark on what might seem like an off-topic excursion into symphonic composition and the translation of mental music into musical scores. But my objective here is to explore a broader question:

What is going on in the minds of parents and other facilitators when they assist minimally verbal individuals with nonverbal tasks and large projects that go beyond a few pages worth of typed output?

And my goal, once I emerge from this musical excursion towards the end of this post, is to formulate some possible answers that came to me only as I read and listened to this fascinating—if irresponsible—KQED report.

An excursion into classical music composition and translating mental music into notes on paper

People who attend classical music concerts regularly see multi-paragraph-long descriptions of the pieces being performed. Here’s an excerpt from the program notes of a recent Philadelphia Orchestra concert describing the first movement of Bruckner’s 4th symphony:

After a nocturnal string tremolo that opens the first movement (Bewegt, nicht zu schnell), the horn announces the sunrise and the dawn of a new day for hunting. The theme is eventually taken up by the rest of the orchestra and developed using Bruckner’s favorite rhythm of two beats followed by a triplet. A lighthearted second theme appears first in the strings, evoking the gentle folk-dance flavor that Mahler would later allude to in the Ländler movements of some of his symphonies. The main “horn call” motif then opens the development section, which ebbs and flows around a brief treatment of the second theme and further development of the “Bruckner rhythm,” culminating in a majestic brass chorale garlanded with string tremolos. This sets up the recapitulation, where the first theme is embroidered with an added flute solo, and the second theme is harmonically enriched with unusual modulations. An extended coda prepares for a triumphant return to E-flat at the conclusion.

—Luke Howard

Imagine, first, how long it would take someone extending an index finger to a held-up keyboard to type all this out.

Imagine, second, whether anyone reading this passage who had never heard Bruckner’s 4th symphony before, even if he is a skilled composer, could possibly translate it into the actual melodies, chords, rhythm, and orchestration of Bruckner’s 4th symphony. As pianist and conductor Daniel Barenboim writes in a recent New York Times Op-Ed, “when we try to describe music with words, all we can do is articulate our reactions to it, and not grasp music itself.”

Perhaps a better way to grok what it takes for Person B to transcribe music from Person A’s head into notes on paper, with little other than Person A’s words to guide them, is to watch this scene from the movie Amadeus. Here Salieri sits at the foot of Mozart’s deathbed and attempts to transcribe, from Mozart’s verbal directions and humming, a short passage of the requiem that Mozart has inside his pillow-propped head. While this scene gives us some idea of what’s involved, though, even it strains credulity. Amadeus takes many liberties not just with Mozart’s life, but also with what Salieri can realistically glean from what comes out of Mozart’s mouth.

What’s crucial here, however, is that the Mozart character uses not just ordinary words, but musical terms, and, in addition, that he frequently illustrates melodies by humming them. Finally, since he can read music, he can check Salieri’s transcription. Here’s an excerpt from the scene:

Strings in unison. Ostinato, on A. Like this: [HUMS] Next measures rising. Do you have it? Show me.

In contrast, Jacob’s directions consist of words only: words about specific instruments, specific actions or emotions, and the number of iterations and/or for how long. For example:

It starts with piano bangs. Five bursts of cacophony.

In response to this particular prompt, Laufer had to guess the specific chords. He reports hypothesizing that Jacob might have in mind “this chord that I used to think about when I was young that I associated with the end of the world.” So Laufer played a specific chord he used to think about for Jacob, who apparently confirmed it as exactly what he had in mind—presumably by poking out some sort of affirmation on the keyboard held up by his father. In contrast to Mozart checking Salieri’s transcription, Jacob can only type “yes” or “no.“ But conveniently—because otherwise there could have been an endless back and forth—of all the tens of thousands of possible sets of five bangs within the 88 keys of the piano, Laufer guessed the correct ones.

Some of Jacob’s directions are more emotional:

I want triumph and happiness followed by terrified laughter and grief over so many years of painful silence.

Others more poetic:

The violins are demanding sleep and the horns are demanding pain. They battle for 3 minutes of call and response until the horns realize that they are defeated (Every manic horn met by soaring violin).

But while a skilled composer might translate these prompts into music that evokes them, countless translations are possible, and the chance of any particular one matching the music in someone else’s head is close to nil.

At this point, it might be interesting to contrast a sequence of prompts attributed to Jacob to a sequence of prompts attributed to Mozart in Amadeus.

It’s hard to imagine how Laufer could have translated a 70-minute symphony from thousands of prompts like the ones on the left unless most of the time he guessed correctly. But Laufer suggests this is more or less what happened: Jacob’s directions “made sense” because they were “coherent to the story he wanted to tell.”

Jacob, for his part, purportedly says:

I was unbelievably damn floored by Rob’s ability to tap into my emotions. I can only say that he is my great collaborator and he reads my musical mind. He can always feel what I want and turn it into amazing notes.

But there’s another possibility—assuming we take everything we read here on faith. Perhaps what was in Jacob’s head wasn’t really a symphony per se, but merely a bunch of verbal ideas for what kinds of emotions and interactions a symphony might express.

But that hardly counts as having an actual symphony in your head.

Regardless of what actually happened, there are a number of things at play here that suggest some broader possibilities about facilitation and translation in minimally verbal autism.

What might be going on in the minds of facilitators when they assist minimally verbal individuals with large projects and nonverbal tasks?

A number of facilitated individuals have accomplishments attributed to them that amount to far more than producing short passages of typed output. Many attend college and presumably produce long research papers, answer math problems, balance chemistry equations, and/or participate in labs; some direct movies; some compose; some co-author peer-reviewed journal articles. Even output that’s purely verbal must often go through multiple drafts, from large-scale reorganization to sentence- and paragraph-level revisions to word-level copy-edits to footnoting and formatting of bibliographies. Much of this goes beyond what can reasonably be accomplished by holding up keyboards to extended index fingers. So what happens behind closed doors?

Here’s a possibility. Perhaps, just like Laufer with Jacob, the facilitator goes beyond facilitation to “translation.” Perhaps the facilitator, like Laufer, makes very specific guesses at what their client wants to produce—without thinking too hard about how many actual possibilities are out there—and then seeks confirmation on the keyboard, and then finally (in cases where, unlike with Laufer and Jacob’s dad, the facilitator and translator are one and the same person) facilitates out affirmative responses from them. Perhaps the facilitator, like Laufer, presumes that the person is competent enough to produce something that “makes sense” and is “coherent,” and also that they know the person so well that they can guess what the person would produce on their own if they could: what kind of essay revision; what kind of movie direction; what kind of symphonic passage. And as long as they believe that they have obtained affirmation from their client on whether these are the melodies and chords, or the scene and camera angle, or the thesis statement and supporting paragraphs, or the steps in the mathematical solution or lab report, that their client had in mind, they can also believe that the resulting product is, in essence, the creation of their client.

As for the affirmation process, assuming the story of Jacob and the 70-minute symphony is representative, it seems to go swiftly: the facilitated answer to the suggested melodies and chords, or suggested scene and camera angle, or suggested thesis statement and sentence revisions, or suggested steps in the mathematical solution or lab report, is quite likely to be consistently “yes.”

In the majority of the cases where only a facilitator is involved (and not, say, a composer or cinematographer or mathematician), what all this means is that the facilitators may unwittingly produce not just verbal messages, but entire projects.

Anecdotally, we’ve heard anonymous behind-the-scenes accounts that precisely this sort of thing has happened: a facilitated college student has to write a long essay, the facilitator holds up the keyboard and extracts some starting words or sentences, the student wanders off after a while, and the facilitator writes the rest of the essay, checking in with the student later about whether it’s what she had in mind—or, worse, simply assuming that that’s what the student would have written, if only she could have.

Epilogue

Returning to the KQED piece, we learn that Jacob’s next composition is a Mozart-influenced opera. Will Laufer be up to the challenge? Time will tell