Over the past couple of years, S2C proponent Vikram Jaswal, Professor of Psychology at the University of Virginia and the father of an S2C user, and Diwakar Krishnamurthy, Professor of Electrical and Computer Engineering at the University of Calgary and the father of an RPM user have co-authored several papers on the development of virtual tools that enable S2C users to virtually select virtual letters rather than pointing to letters on letterboards. Like Jaswal’s other recent papers, each of these begins with purported justifications for S2C as a valid communication method, and each reports instances of allegedly independent communication by S2C users. Like Jaswal’s other papers, therefore, these papers are worth reviewing in detail. In my last three posts, I discussed Alabood et al. (2024), a study involving a virtual letterboard, or HoloBoard, and Alabood et al. (2025), a study involving the use of virtual letter cubes to spell words.
In my next two posts, I’ll discuss two
additional studies in Jaswal’s 2023-2025 virtual reality oeuvre that are
relevant to us here at facilitatedcommunication.org: Nazari et al. (2024), a
study involving the “HoloGaze,” and Shahidi et al. (2023), a study involving
the “HoloLens.” What makes these studies relevant is that, like all the other
Jaswal studies we’ve reviewed:
· They involve language and literacy (some of
Jaswal et al.’s other studies instead involve visual games)
· They appear to show levels of language
comprehension that are generally not found in minimally speaking autism,
particularly in those with significant motor impairments (see Chen et al., 2024).
(Minimal speakers with autism and
significant motor impairments being representative, per Jaswal et al., of the
population that their studies are targeting.)
In this post, I’ll focus on the HoloGaze
study (Nazari et al., 2024), a paper in which Krishnamurthy and Jaswal are
listed, respectively, as the second and third authors. This paper focuses on a
virtual reality technology, complete with headsets, that allows users to select
letters and spell words by shifting their gaze to different letters from a
virtual display. Users can select a particular letter either by sustaining
their gaze on that letter, or by pushing a button while looking at it. The HoloGaze
study differs from the HoloBoard and LetterBox studies in one important way:
for the most part, it doesn’t to have its non-speaking autistic participants do
anything other than select letters and spell words—as opposed to tasks that
require actual comprehension of language. (It’s worth noting that a significant
proportion of autistic individuals are hyperlexic and able to spell words that
they don’t understand).
Like the other two papers, the HoloGaze
paper opens with the same unsubstantiated/false claims about minimal speakers
with autism, beginning with the purported attentional, sensory, and motor
issues:
[N}onspeaking
autistic people have significant attentional, sensory, and motor challenges..,
making it difficult for them to write by hand or type inconventional ways. For
example, many nonspeaking autistic people have difficulty with the fine motor
control needed to use a pen or pencil.
As in the LetterBox paper, their
authors’ sources for these claims are (1) and entire book
attributed to someone who has been subjected to facilitated communication
(Higashida, 2013), and (2) a meta-analysis of 41 studies on motor coordination
in autism, none of which included the skills involved in pointing to letters on
a letterboard (Fournier, 2010). To give Jaswal et al. the benefit of the doubt,
when they mention typing in “conventional ways,” they may mean ten-finger
typing. Ten-finger typing, as compared with typing with an index finger, does
involve significant motor coordination. But, to the extent that the authors’
goal here is to motivate an examination of eye-gaze technology, they haven’t
managed to explain why, for minimal speakers with autism, eye pointing would be
superior to index-finger pointing.
Furthermore, the authors make no
mention of the one specific motor issue, sometimes alleged in non-speaking
autism (see Handley, 2022), that might be relevant here: ocular
apraxia—or the ability to move one’s eyes in
a desired direction. Ocular apraxia would have been relevant to their HoloGaze
study because it reports involved a training phase in which non-speaking
autistics learned how to select letters via eye gaze. If there’s no eye gaze
impairment in autism, and if the non-speakers already know how to pick out
letters on demand, then why do the participants in this study require anything more
than a brief explanation in how to use the system?
Beyond their claims about attentional,
sensory, and motor challenges in autism, the authors claim, once again citing
the memoir attributed to Higashida (Higashida, 2013), that the tendency of non-speaking autistics
to have trouble sitting still while being prompted by their facilitators to
point to letters on the letterboard is the result of “regulatory” issues:
They
may also be in constant motion (which seems to serve a regulatory function...
making training to use a keyboard while remaining seated difficult [)].
Left out of this discussion is the
more likely possibility: boredom with a task that, due to those significant
comprehension problems in non-speaking autism that are particularly severe in
those with significant motor impairments (see Chen et al., 2024), these
individuals probably find meaningless.
Next, the authors claim that:
Some
nonspeaking autistic people have learned to communicate by typing, which has
allowed them to graduate from college and to write award-winning poetry.
Their sources are a National Public Radio piece about RPM user Elizabeth Bonker’s valedictory
speech at her graduation from Rollins College (in which she stood at the podium
while a pre-recorded speech, attributed to her, was broadcast), and an autobiographical piece attributed to David James Savarese, better
known as Deej.
Despite the purported communicative
successes of those who “communicate by typing,” the authors note that
the
process by which they learned to type was lengthy and expensive, and often
requires the ongoing support of another person.
The most likely explanation for these
hurdles is that those subjected to “communication by typing,” aka Spelling to
Communicate (S2C), depend on facilitator prompts and cues to determine which
letters to select; for Jaswal et al., these hurdles are instead a reason to
develop virtual reality technologies like the HoloGaze.
Importantly, however, the HoloGaze
allows
a caregiver to join an AR session to train an autistic individual in gaze-based
interactions as appropriate.
(“Caregiver” here is used in this
paper to denote the facilitator, or what S2C-proponents call a “Communication
and Regulations Partner.)
Those familiar with evidence-based AAC
devices might wonder, given that there already exist AAC devices that allow
eye-gaze selections or “eye typing,” what the point is of this new tool. But
the authors, acknowledging that such technologies already exist, claim that VR
tools offer “unique additional benefits.” They cite:
· The “wider context” in which these devices can
be used: “e.g., not just at a special education classroom but also for personal
use cases.”
· “mobility”
· The “3-dimensional environment shared between
educators and students,” which “can facilitate the training process for those
who require extensive training.”
These strike me as pretty weak
arguments. Standard AAC devices are mobile and can be used in a broad set of contexts.
And why is “extensive training” necessary? Believing, as Jaswal does, that his
non-speaking participants already know how to identify letters and point to
them to spell words, and that non-speaking autistic individuals, like people in
general, look at the letters that they point to (Jaswal, 2020), it’s unclear
why these participants would need “extensive training” in using selecting
virtual letters via the HoloGaze.
So who are these participants? In all
the other studies we’ve reviewed so far, Jaswal et al.’s participants are
explicitly described as individuals point to letters on letterboards with the
support of communication partners (the hallmarks of Spelling to Communicate and
Rapid Prompting Method). This study differs: its inclusion criteria do not
mention communication partners; only “experience in communicating using a
physical letterboard.” Participants therefore could have included those who
communicate independently: without someone hovering, holding up the
letterboard, prompting, or otherwise cueing them and controlling their letter
selections. And yet the identities of every single person who is thanked in the
paper’s acknowledgments suggest otherwise. In order of mention, they are: a neuropsychologist who believes in S2C and is
doing her own experiments on S2C-ed individuals, an RPM user,
the mother of an S2C user, an S2
practitioner, another S2C practitioner, and S2C
promoter and practitioner Elizabeth Vosseller.
Let’s turn, now, to the actual
experiment. After an initial “tolerance and calibration” phase, participants
underwent a “training phase” in which they first learned to select flashing
tiles and then flashing letters from arrays of tiles/letters in which only the
target item flashed. If they selected the flashing item, it turned green and was
surrounded by a “bounding box” that indicated
“successful eye gaze engagement.” Also
providing cues were the “caregivers”:
The
person assisting the user could also observe the tile’s
colour (because they were also wearing a device), enabling them to provide
verbal prompts to guide the user’s attention if necessary.
If the participants really had the
language comprehension skills that Jaswal et al. regularly attribute to them,
why couldn’t they just be told, verbally, how the system worked? All they
needed to know, in order to use the system correctly, was this: “Direct your
eyes to the flashing letter. Then either look at it for one second, or push
this button while you look at it.” One has to wonder whether, at some level of
consciousness or sub-consciousness, Jaswal suspected that his participants
didn’t have the kinds of comprehension skills that he has long attributed to
the broader non-speaking population to which they belong.
The flashing letters and the
facilitators’ “attentional prompts and cues” continued into the “Assisted
Spelling” part of the testing phase, where the participants had to spell actual
words by selecting letters from a virtual letterboard. The researchers dictated
six three-letter words (JET, DRY, EVE, FAN, GUM, RUG, IVY),
and the target letters flashed one by one in order as they were selected.
Besides the flashing, prompting, and facilitator cueing, participants received one
additional cue:
To increase their visual load gradually, participants did not see
the full letterboard at the beginning of the testing phase. Instead, only the
letters in the first word were presented. After the first word (and after all
subsequent words), the additional letters required to spell the next word were
added.
To justify
this, the authors cite feedback that was almost certainly generated by S2C:
This design was suggested by our nonspeaking autistic consultant
to reduce visual clutter initially as the participant learned the affordances
of a new interface.
This feedback was almost certainly generated by the
facilitators/CRPs rather than by the non-speakers themselves.
Following the “Assisted Spelling
Phase” was the “Unassisted Spelling Phase.” This involved six four-letter words
(ARCH, BALL, DUCK, EARL, FALL, GIFT, HOPE),
without the target letter flashing. It’s unclear whether the facilitators were still
allowed to prompt and cue, but in any case it’s a lot harder to detect and cue
people’s eye gaze than their extended index fingers.
Curiously, there was a fair amount of
attrition at each stage, from the “tolerance and calibration” phase to the
“training” phases to the “testing” phases:
Twelve of the 13 participants who tolerated the device attempted
the testing phases that involved spelling... Half of those who tried the
testing phases (6 of 12) completed both the phase where the letters flashed in
sequence ("assisted") and the phase where the letters did not flash
in sequence ("unassisted").
In other
words, less than half the participants made it through the whole study—short
though it was. And yet, the authors professed to be impressed:
This is a remarkable number given this was their first experience using eye
gaze interactions using a head-mounted AR device.
As for the actual results
of those who completed at least one of the testing phases, the author report
two factors: correct interactions per minute (which presumably means “correct
letter selections per minute”) and error rate. Mean interactions per minute
decreased over time from 13.49 to 10.53, which the author claim reflects
increased complexity (more letters to choose from; the shift from assisted to
unassisted spelling; the shift from three-letter words to four-letter words).
Thus, the unassisted spelling averaged between 5 and 6 seconds per correct
letter—a surprisingly low rate for anyone who actually knows how to spell the
given words.
Meanwhile, the error rate,
“surprisingly” according to the authors,” improved from 0.42 (range: 0.07
- 0.79) to 0.39 (range: 0.08 - 0.63).” In other
words, participants only selected the correct letters, on average, about 3/5 of
the time, ranging (across the 6 participants) from near-complete success to an
improvement, at the lowest end, from selecting the correct letter only about
1/5 times to selecting it just under 1/3 of the time.
While
the official training only involved spelling three and four-letter words, some
participants, “if they had time and interest,” were asked to engage in one
activity that actually required comprehension: “answer[ing] five questions with
one word answers on the virtual board.” However, the authors tell us:
This data is not reported here because these
tasks were completed by only a subset of participants and because of space
limitations.
We can only wonder what the excluded
data would have suggested to their readers about language and literacy skills
in non-speaking autism.
REFERENCES
Alabood, L., Nazari, A., Dow, T., Alabood, S., Jaswal,
V.K., Krishnamurthy, D. Grab-and-Release
Spelling in XR: A Feasibility Study for Nonspeaking Autistic People Using
Video-Passthrough Devices. DIS '25: Proceedings of the 2025 ACM
Designing Interactive Systems Conference. Pages 81 – 102
https://doi.org/10.1145/3715336.3735719
Alabood, L., Dow, T., Feeley, K. B.,
Jaswal, V.K., Krishnamurthy, D. From Letterboards to Holograms: Advancing
Assistive Technology for Nonspeaking Autistic Individuals with the HoloBoard.
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing
Systems Article No.: 71, Pages 1 - 18 https://doi.org/10.1145/3613904.3642626
Chen, Y., Siles, B., &
Tager-Flusberg, H. (2024). Receptive language and receptive-expressive
discrepancy in minimally verbal autistic children and adolescents. Autism
research : official journal of the International Society for Autism Research, 17(2),
381–394. https://doi.org/10.1002/aur.3079
Fournier,
K. A., Hass, C. J., Naik, S. K., Lodha, N., & Cauraugh, J. H. (2010). Motor
coordination in autism spectrum disorders: a synthesis and meta-analysis. Journal
of autism and developmental disorders, 40(10), 1227–1240.
https://doi.org/10.1007/s10803-010-0981-3
Handley, J. B., & Handley, J.
(2021). Underestimated: An autism miracle.
Skyhorse
Higashida,
N. (2013). The
reason I jump: The inner voice of a thirteen-year-old boy with autism.
Knopf Canada.
Jaswal,
V. K., Wayne, A., & Golino, H. (2020). Eye-tracking reveals agency in
assisted autistic communication. Scientific reports, 10(1),
7882. https://doi.org/10.1038/s41598-020-64553-9
Nazari, A., Krishnamurthy, D., Jaswal,
V. K., Rathbun, M. K., & Alabood, L. (2024). Evaluating Gaze Interactions
within AR for Nonspeaking Autistic Users. 1–11. https://doi.org/10.1145/3641825.3687743
Shahidi, A., Alabood, L., Kaufman, K.
M., Jaswal, V. K., Krishnamurthy, D., & Wang, M. (2023). AR-based
educational software for nonspeaking autistic people – A feasibility
study. 2023 IEEE International Symposium on Mixed and Augmented Reality
Posts
No comments:
Post a Comment