How valuable might a brain chip be to those who cannot move
Even today, the development of Motor BCIs is primarily evaluated on engineering feats rather than clinical outcomes. This should not come as a surprise given the numerous technical challenges. Moore’s Law seeps its way into every technological crevice, and our delicate biology does dictate the need to reiterate smaller and superior devices. Nevertheless, metrics such as electrode impedance, signal-to-noise ratio, or even decoded words per minute, time to grasp and transport objects, or word error rate, etc., may not represent the genuine needs of the Users.
Since the BCI community has not yet produced a framework centered around patient outcomes, the industry must look elsewhere.
Functional Assessments
To measure patients’ function (e.g. after suffering a stroke), occupational therapists often contrast their ability with the everyday activities that allow an individual to live independently in a community.
For instance, the widely used Occupational Therapy Practice Framework: Domain and Process (2020), identifies a broad range of such activities from eating to social participation, including basic and instrumental Activities of Daily Living (”ADL”).
Basic ADLs comprise tasks that allow individuals to care for themselves, such as feeding, toileting, or dressing. Instrumental ADLs, for example, those of the Lawton-Brody scale, assess the individual’s cognitive and physical abilities to live independently.
Another approach is to consider the bodily function itself. For instance, the Action Research Arm Test and Fugl-Meyer Motor Assessment are common measures to assess the patient’s ability to perform gestures, coordination, and range of motion tasks, to name a few.
Even some BCI studies have used these frameworks to measure the participant’s extent of commanding a robot (Collinger et al., 2013, Frolov et al., 2017) or of recovering function when coupled with functional electrical stimulation (Biasiucci et al., 2018, Bockbrader et al., 2018)
Other notable frameworks include Quadriplegia Index of Function (QIF), Spinal Cord Independence Measure version 3 (SCIM-III), Graded Redefined Assessment of Strength, Sensibility, and Prehension (GRASSP), Capabilities of Upper Extremity Test (CUE-T), Quadriplegia Index of Function-Short Form (QIF-SF), Six-Minute Walk Test (6MWT), ALS Functional Rating Scale-Revised (ALSFRS-R).
🦾 Considering the abilities granted through the envisioned Motor BCI applications, the industry may estimate the solution’s functional impact on the User’s life as well as any cost of care reductions (e.g. requiring less attendant care.)
A limitation of using established evaluation tools for the assessment of functional independence is that they are often out of date. Increasingly, many of the defined activities are nowadays performed digitally, especially by people with paralysis, yet the assessments do not reflect it (e.g. ordering groceries online, socializing digitally, etc.).
Quality of Life (”QOL”)
Another major limitation of relying on a therapist’s functional assessment to determine clinical benefit is the lack of incorporating the User’s subjective opinions (Patient Reported Outcomes “PROs”), especially regarding their well-being.
In other words, even if much of the User’s functioning is restored, they may not feel any better. For example, in an epilepsy medication trial, reducing monthly seizures compared to another drug may not be clinically meaningful if the frequency of seizures still disrupts work and personal safety.
📄 PROs play a crucial role not only in good product design but also in securing FDA approval for a clinical Motor BCI. The FDA’s guidelines emphasize the importance of incorporating useful, targeted, and understandable PRO assessments into clinical trials, leveraging existing frameworks where possible (FDA, 2022).
Several instruments seem particularly suitable for determining the Users’ QOL, starting with Quality of Life in Neurological Disorders (”Neuro-QOL”) which evaluates the physical, mental, and social effects experienced by those with neurological conditions.
Neuro-QOL instruments are not disease-specific measures and it is the responsibility of the researchers to shortlist the appropriate components of self-reported health (”domains”) and questions (“bank items”) for their study and participants. For instance, omitting detailed Sexual Function items, or the domain altogether, may be appropriate for Users with LIS.
Tailored QOL to a Neurological Disorder
Some researchers even went so far as to extend the Neuro-QOL into a framework of their own, targeting a specific neurological condition. In this case, Tulsky et al. (2011) developed the Spinal Cord Injury Quality of Life “SCI-QOL”) measurement system with SCI-specific subdomains and items not addressed by the Neuro-QOL instrument, in particular emotional factors that someone who has sustained a traumatic injury experiences.
Similarly, Felgoise et al. (2011) developed the Amyotrophic Lateral Sclerosis Specific Quality of Life Instrument-Revised (”ALSSQOL-R”) based on the McGill Quality of Life Questionnaire (”MQOL”), assessing features unique to ALS, such as the inevitability of death, and the rapidly progressive series of functional losses.
Like Neuro-QOL, MQOL is a well-established instrument but focuses more on measuring the quality of life in life-threatening illnesses. Cohen et al. (2019) published its most recent revision “MQOL-Expanded”.
👥 While tailoring the QOL instruments to specific patient populations improves its utility, a possible downside of introducing many ‘flavors’ of instruments is the growing difficulty in comparing the results across the conditions even though they share symptoms — participants with different etiologies may use one Motor BCI.
In the same vein, personalization of QOL instruments by including specific activities that impact the individual’s QOL can also lead to the invalidation of the instrument, as personal preferences differ. For example, some may identify their ability to play golf as an important contributor to their QOL, while someone else might find going to the cinema more appropriate; who defines what healthy behaviors outside of ADLs should be?
Some advocate for the development of instruments containing recreational or advanced ADLs. While some preferences are shared in a homogenous patient sample, inherent differences in interests prevail (Linden et al., 2009). However, some such instruments can be tremendously useful QOL indicators showing a high correlation with basic and instrumental ADLs (De Vriendt et. al, 2013). De Vriendt et al. (2013) circumvented the problem of personalization by first asking the respondents to indicate which of the 49 proposed ADLs are relevant to them, scoring only personally meaningful ADLs.
A simpler approach, that of SCI-QOL, is to ask the respondent to comment on their ability to perform unspecified activities of their interest. For instance, items like “I am able to do my hobbies or leisure activities.” or “I can keep up with my work responsibilities.” assess the respondent’s ability to engage in a personally meaningful activity without inquiring about its precise nature.
Tailored QOL to a Motor BCI
To overcome the limitation of comparing PROs across incompatible QOL instruments, one may consider applying a QOL instrument that measures the User’s perceived benefit of using assistive technology. In this case, a Motor BCI.
Indeed, several Motor BCI studies have adopted the Psychological Impact of Assistive Devices Scale to measure its impact on User’s perceived competence, adaptability and self-esteem (Holz et al., 2015, Vansteensel et al., 2016).
Additional instruments for assessing the impact of assistive technology include Quebec User Evaluation of Satisfaction with Assistive Technology (”QUEST 2.0”) which is preferred for its succinctness and contains items such as repairs, product dimensions, and more, and the novel Assistive Technology Usability Questionnaire for people with Neurological diseases questionnaire, aimed at individuals with neurological conditions.
Other researchers directly address reporting on QOL related to Motor BCIs and develop bespoke approaches such as Dobkin’s (2003) domains relevant to BCI trials or Fry’s et al. (2022) call for a Digital ADL scale necessary for an increasingly digital population.
Users with LIS face perhaps the greatest challenges in reporting their needs due to imposed communication difficulties, resulting in poor insight into their preferences. In a rare study, Branco et al. (2021) asked 28 individuals with LIS to rank their preferred Motor BCI applications, which may provide clues about abilities that would strongly impact their QOL. Overwhelmingly, self-expression (in the form of “Direct personal communication” and “Private conversation & writing”) dominated the remaining four options.
Figure 6; Branco et al., 2021: Preferred applications of people in LIS following progressive neuromuscular diseases (”NMD” like ALS) or sudden onset disorders (”SO” like SCI). Reprinted from Figure 4A.
Similarly, Collinger et al. (2013) asked 56 veterans with tetraplegia (20) and paraplegia (36) after SCI about their priorities related to up-and-coming Motor BCIs. Two particularly relevant findings are the top priorities for restoration for impact on QOL (Fig. 7) and Motor BCI design characteristics (Fig. 8), contextualizing the solution as assistive technology.
Figure 7; Collinger et al., 2013: Reprinted from Figure 4.
Figure 8; Collinger et al., 2013: Reprinted from Figure 6.
Considering the leading works measuring clinical outcomes for the envisioned Motor BCI Users, one can combine them into a set of a priori interim “Success Criteria” that a Motor BCI application should strive to amplify. With this goal in mind, I set out to develop them.
Success Criteria for Clinical Motor BCI — User Outcomes
Considering the leading works measuring clinical outcomes for the envisioned Motor BCI Users, one can combine them into a set of a priori interim “Success Criteria” that a Motor BCI application should strive to amplify. With this goal in mind, I set out to develop them.
Development
The following Success Criteria are based on the PROs most relevant to the Users, given the possible impact of first-generation Motor BCIs. For example, a Motor BCI could restore attempted movement, but not reduce pain, improve bowel function, or restore the sense of touch, excluding some PROs from the Success Criteria.
Users with SCI represent the largest and yet most diversely impaired group of Users. The effects of the injury may range from the inability to perform certain fine movements to complete paralysis below the level of injury. The majority of the success criteria will be based on SCI-QOL (Tulsky et al., 2015, Jette et al., 2012) for its most comprehensive assessment of the physical, emotional, social, and functional aspects of living with severe paralysis. The instrument also builds upon the gold standard Neuro-QOL and PROMIS measurement systems. In addition, it followed a rigorous development methodology, which included 42 expert clinicians and 65 individuals with SCI who define domains and the item bank, and 877 more who field-tested it (Fig. 9).
Figure 9; Tulsky et al., 2015: “SCI-QOL Development and Calibration: Phases and Goals”. Reprinted from Figure 1.
The Physical-Medical domain is omitted due to the poor Motor BCI influence over its outcomes. Success criteria include all short-form items from each bank along with a few additional items relevant to Users.
Figure 10; Tulsky et al., 2015: Overview of the scope of the SCI-QOL measurement system. Reprinted from Table 1.
The Success Criteria also include some items from the ALSSQOL-R item bank to account for the specific needs and experiences of individuals with ALS, the most prevalent cause of LIS and likely the largest initial User base. ALSSQOL-R was developed with similar rigor, revising the original ALSQOL instrument (validated with 342 individuals with ALS) developed by Simmons et al. (2006), with an additional 389 individuals with ALS.
Even though the ALSSQOL-R scale is incompatible with SCI-QOL for scoring purposes, together they provide a window into widely accepted measures of the Users’ QOL.
Lastly, the Success Criteria are also extended by QUEST 2.0 to account for the Motor BCI’s impact on the User as an assistive technology.
The 31 Success Criteria, span 440 items relevant to measure Users’ QOL: with 12 items contributed by QUEST 2.0, 24 ALSSQOL-R items, and 403 SCIQOL items (of which 82 were included in short-form versions).
Success Criteria
The following Success Criteria can be used to compare the impact of Motor BCIs, as an assistive technology, on User needs in a clinically meaningful way. For an Excel version and an extended breakdown of other QOL instruments see Supplementary Data 1.
(For the full 440 items see the footnote link) Table 1; uCat, 2024: QOL Items for 31 Clinical Motor BCI Success Criteria
Interpretation
The needs and interests of the Users differ tremendously from person to person. Some of the Success Criteria were designed to exemplify and encompass functionally equivalent items even though they may be performed for a different purpose.
For example, the Success Criterium “I can take care of a plant” includes functionally equivalent activities performed for hygiene (e.g. ”Are you able to insert and remove a tampon?”), nutritional (e.g. ”How much difficulty do you currently have chopping or slicing vegetables?”), general (”Are you able to pick up a small object?”) or even medical (e.g. ”Are you able to carry out your bowel program on the toilet?”) purposes. In such cases, the reasons for each item’s equivalency are described in Table 1.
Some people could not care less whether they can care for a plant, and yet the ability to do so allows them to perform various other basic, instrumental, and even recreational/advanced ADLs. While basic and instrumental ADLs are somewhat standard, and included in the SCI-QOL instrument, recreational/advanced ADLs differ based on individual preferences.
Recreational/Advanced ADLs
A list of such recreational/advanced ADLs — which the User may consider when assessing Success Criterium #28, 29, and 30 — are combined in the table below the works of Linden et al., (2009), De Vriendt et. al, (2013), Linden (2018), Zahl et al., (2022) who developed these categories primarily for mental and cognitive health assessments.
(For full 78 items see footnote link) Table 2; uCat, 2024: 78 Recreational and Advanced ADLs combining items from Linden et al., (2009), De Vriendt et. al, (2013), Linden (2018), Zahl et al., (2022)
Practical Implications
The high variability between impairments, and therefore needs, experienced by the Users prevents further reduction of the included items or assigning relative importance to each Success Criterium. For instance, 77% and 64% of individuals with ALS who can still speak have no problem swallowing and breathing, compared to just 14% and 29% who have lost their speech, respectively (Zizzi et al., 2022).
✅ These custom Success Criteria represent dimensions of how the User should function and feel, as they use their Motor BCI. To what extent they are met widely depends on the individual User, their Motor BCI, and applications that connect the two.
Rather than representing a clinical evaluation framework, these Success Criteria merely outline User needs grounded in well-established measures of clinical outcomes
.
Depending on the participant demographics and device specifications, the Motor BCI manufacturers engaged in pivotal clinical trials may omit many of the items listed here or use a different set of instruments altogether.
Specifying and optimizing for these clinical outcomes plays a central role in securing vital health insurance coverage. Clinical outcomes achieved by a Motor BCI impact many aspects of the cost-effectiveness/utility analysis, such as productivity gain (e.g. recovered wages reduce the cost of therapy), independence (e.g. reduced cost of care, increased QOL), and willingness to pay (better PRO allow greater cost of therapy).
📈 To scale Motor BCIs through reimbursement, their manufacturers need to shift focus to measurements of clinical outcomes over engineering benchmarks.
In the US, the Motor BCIs that already received the FDA Breakthrough Designation (Neuralink, Blackrock Neurotech, Paradromics, ONWARD, Synchron, and Precision Neuroscience — most of whom have already implanted their device in humans) likely follow the novel Transitional Coverage for Emerging Technologies (TCET) pathway. Here, the value for money of their solutions is put head-to-head against other data sources for prosthetics (American Association of Neurological Surgeons, 2023), eye tracking (Vansteensel et al., 2016), and other assistive technologies.
Part 3 of a series of unedited excerpts from uCat: Transcend the Limits of Body, Time, and Space by Sam Hosovsky*, Oliver Shetler, Luke Turner, and Cai Kinnaird. First published on Feb 29th, 2024, and licensed under CC BY-NC-SA 4.0.
uCat is a community of entrepreneurs, transhumanists, techno-optimists, and many others who recognize the alignment of the technological frontiers described in this work. Join us!
*Sam was the primary author of this excerpt.