Mind Is Matter: Part 1
Neuralink and the new era of brain computer interfaces
When I started my PhD in neuroscience, I read an article by a non-neuroscientist, on a blog called Wait But Why, that reaffirmed why I was putting myself through this.1 He wrote about the delicious paradox that if the brain were simple enough for us to easily understand, we would be so simple that we couldn’t.2
The topic of the post was Neuralink, a company trying to build a brain computer interface (BCI) that would at first be a medical device, but eventually be used by healthy people to talk to computers, and other people, with their thoughts. Building a safe and effective interface to the brain is a massive engineering challenge, and happens to be step one in understanding how it works.3
As word spread and the hype grew, several of my colleagues expressed skepticism about Neuralink. The CEO (Elon Musk) went nuts. Plus, six years in academia beat the excitement out of me and I became disillusioned by everything wrong with neuroscience research. My fascination with BCIs hid behind a cloud of caveats.
I moved on, started this blog, and forgot about Neuralink—until two months ago, when they released a video of their first human patient playing online chess using his brain activity, while chatting with the person recording. I want to set aside the caveats for a minute, and revel in the coolness of what is happening here.
Wait, But What is a brain computer interface?
As you read this sentence, your eyes move across the screen because neurons in your brain are telling your eye muscles what to do. If you get bored, your brain might signal your arm to fling your phone away. Thanks for not doing it, though.
Your brain knows what signals to produce to make muscles move in specific ways, and the muscles know what to do when they receive those signals. Some of these brain-muscle mappings are built-in, others are learnt. This baby’s brain is learning what neural activity will make its leg muscles contract and relax in the right combinations for walking.
Imagine using a desktop computer. Your brain generates signals that make the muscles in your arm and hand contract and relax, to operate the mouse and keyboard. Key presses, mouse movement and clicks are read out by the computer.
Noland Arbaugh, the patient in the video above, is a quadriplegic, meaning that he cannot control his arms and legs, due to damage to his spine. Neuralink performed a surgery to place an implant in his brain, which measures the activity of nearby neurons and sends it to an app, which converts the neural activity into computer commands using a pre-defined mapping. An adult can learn to use this implant to operate a computer, just like a baby learning to use its legs to walk.
The implant is placed in the motor cortex, because humans (including quadriplegics) can control its activity by trying to move their limbs. First, the patient imagines moving their arm in different directions, making a fist, tapping their finger and so on, and the software learns which patterns of neural activity correspond to these imagined movements. Then, researchers assign a computer command to each movement. The patient trains their brain to imagine the right movement whenever they want to perform a computer action, like a musician fine-tuning their muscle memory to perform a new piece.
Try playing this game, where you click on the blue square in a grid, and earn a bits-per-second (bps) score based on number of correct clicks per minute. My best score was 10.5 bps (on my phone). Arbaugh scored 8.01 bps using the Neuralink implant to control the cursor. The previous world record with a BCI was 4.16 bps (note that this is an average over two days, whereas 8.01 bps is Arbaugh’s peak performance).4
Better than what came before
Neuralink didn’t invent BCIs. In 1969, Eberhard Fetz recorded the activity of a single neuron in a monkey’s motor cortex, and gave the monkey a reward if that neuron fired. In a few training sessions, the monkey learned to make that neuron fire five times more than its normal firing rate.
Since then, both the number of neurons that could be recorded at once, and the duration of stable recordings, increased. We can now record several thousand neurons simultaneously in rodents. The more neurons you record, the larger is the set of possible activity patterns. You need lots of different activity patterns for someone to be able to use a computer, to assign to different computer commands (moving the cursor in all directions; left, right and double clicks; and key presses).
The Neuralink implant has ten times more recording sites than the BCI that held the previous bits-per-second record.5 But Neuralink’s big achievement is that their patient uses his BCI at home, on his own, no researchers present, to livestream on Twitter and play music on Spotify. Here’s how they made this happen.6
The implant is wireless. It sends neural activity data to the computer via bluetooth, using very little energy and dissipating very little heat. Quick and reliable wireless charging happens while the patient sleeps. The implant is small enough to fit into the part of the skull that was removed—the skin can be closed after the surgery.
The BCI is user friendly. When a neuron fires, the information is sent to the computer in less than ten milliseconds, enabling smooth cursor control (how would it feel if you tried to move your hand and there was a noticeable delay before it moved?). Mouse clicks generate visual feedback to replace the tactile feedback from a normal mouse, and let the user know the click worked.
The BCI works day after day. The algorithm accounts for day-to-day changes in the statistics of neural activity, and the patient can access a calibration tool (watch from 6:00 to 9:00 of this livestream) on his own, to fine-tune the algorithm if needed. Neuralink limit-tested the implant’s durability by putting it in artificial environments that do the same damage as brain tissue, but four times faster. Two of their monkeys have had implants for more than a year.
The Neuralink app, which receives data from the implant, connects seamlessly with all other applications on a computer that a person might want to use.
I recommend watching this 2022 Show and Tell to learn more about all of this. If the whole thing is too long for you, watch these sections to learn about energy efficiency, charging, day-to-day reliability, and accelerated lifetime testing.
A retraction
On 8th May, Neuralink announced that some of the threads in Arbaugh’s implant had retracted out of his brain, and could no longer record neural activity. These threads, and the robot that inserts them into the brain, are Neuralink’s lifeblood. The threads’ flexibility, and the robot’s precision, mean that they can avoid even the tiniest blood vessels on the brain’s surface.7 Their thin diameter and biocompatible materials fool the brain into not noticing that they’re there. The density of recording sites gives the algorithm access to rich neural activity data.
So why did they retract? Imagine inserting 64 strands of hair into a slab of tofu kept in a plastic container and walking around with it for a month. Would you expect them to stay put? It’s not a perfect analogy, and I’m not saying it’s OK for a brain implant to come out in two weeks, but it helps to understand the scale of the engineering problem, especially if you’re not used to the squishiness of brains.
Neuralink did not disclose how many of the threads retracted or how many recording sites remained functional, but they showed that Arbaugh’s daily peak bps score reduced as a result of the retraction.8 They were able to rescue his performance by adjusting the decoding algorithm, so there must be enough threads still embedded to measure similarly varied neural activity patterns as before.
Neuralink performed extensive testing with animals to prevent something like this, but it’s difficult to predict everything that will happen with humans (that’s why we have clinical trials with strict oversight). Could Neuralink have done more testing before enrolling their first patient? Yes. Should they have done more testing? I’m not so sure. The implant seems to be doing OK two months after the retraction.
Arbaugh says the implant has completely changed his life.
There’s a lot more to say about BCIs. In part 2, I’ll discuss some of the caveats I set aside here, so I could jump around in excitement unencumbered. I’ll talk about other companies making BCIs for various disabilities, why that’s a good thing for the future of BCIs, and why you need to care about BCIs—you might get one.
Wait But Why is the best blog on the internet—I will die on this hill. My favourites are the series on procrastination, these three posts about taking the long view of life and making the most of it, and the Q&A posts that made me snort several times as I re-read them right now. Linking to Wait But Why posts is dangerous because I inevitably pause and spend an hour reading instead of writing.
This is a great thought experiment to spend some time on. Is it possible to build a system that can understand itself? If it’s too simple, it may not be capable of ‘understanding’, and if it’s complex enough to be capable of understanding, it may be too complex to understand.
Understanding how the brain works is not one of Neuralink’s stated goals (their short term goal is to help quadriplegics operate computers and their long term goal, believe it or not, is to make sure AI doesn’t take over the world). For me, and I assume other neuroscientists, the advance in the quality and amount of human brain activity data is one of the most exciting aspects of their recent progress, but we need to be very careful not to push for scientific progress at the expense of people’s health and wellbeing—science does not have a good track record here. More on this in part 2.
The values reported by Neuralink are peak performance in a day, so they’re less informative than the average performance, and I wonder why Neuralink is choosing to publicise the peak, rather than the average, values. In any case, it is beyond a doubt that Arbaugh can control the cursor well enough to operate a computer normally, which is phenomenal for a BCI. You can see him playing Webgrid in Figure 3 of this blog post, and his daily peak BPS scores in Figure 4. You can watch a video of the previous world record BPS using a BCI here (remember, it’s a much smaller grid), and compare.
Each recording site picks up the activity of several nearby neurons, and a single neuron’s signals are picked up by multiple recording sites. Researchers triangulate signals measured across different sites to infer the activity of individual neurons. So while the number of neurons is likely to be of the same order of magnitude as the number of sites, it’s never exactly the same. Further, the electrodes move around in the brain, leading to different neurons being recorded each day.
Some of these things have also been achieved by other BCI companies, which I will cover in part 2 of this post. Taken together, it does seem that Neuralink is ahead of the curve in creating an out-of-the-box device that enables quadriplegics to control computers.
As opposed to the rigid electrodes historically used for BCIs. More on this in Part 2.
Again, these are peak scores, and daily or hourly averages would be more meaningful.
Excellent piece! Have heard the name neuralink but didn’t know about BCI. The mind boggles.