How our subconscious processes mislead us on the nature of human manipulation, by managing it brilliantly and quietly.
The Inner Robot
Can robotics offer insights into the human mind? One goal of robotics is to explore the nature of intelligence. Whatever we learn might apply to humans as well as to robots. Has that happened? I think it has. Fifteen years ago, I learned that I share my brain with an “inner robot,” which quietly takes care of all the complex details of manipulation. I will explain.
In 2009 I was trying to find a way to study human manipulation “in the wild”— just as it occurs naturally, unencumbered by experimental protocols or preconceived notions. As a first attempt, I spent one morning watching my hands, shooting photos, and dictating notes, as I went about my usual routine. What did I learn? Nothing momentous. The bathrobe donning process is intriguing. I discovered that a lot of my manipulation was operating simple mechanisms: the toilet lever, the sink knobs, cabinet doors, and so on. (I’m sparing you the more personal stuff.)
But I wanted more. I wanted video. I hit on an easy way to get it: cooking shows. Each episode includes several minutes of skilled manipulation, nicely photographed. I started with Julia Child’s The French Chef, episode 22 — “The Potato Show.” I slotted the DVD, and watched intently, and 30 minutes later … nothing. I learned nothing.
Perhaps that is not surprising. I had watched my mom slice potatoes many times. I had watched myself slice potatoes many times. I had even had a potato-slicing lesson in a Chinese cooking class. (I love the trick of curling your fingers and using your knuckles to guide the knife.) So, what could I possibly learn by watching one more demo of potato slicing?
I was not ready to give up. I started over at the beginning, but after two minutes, I had to stop. It was too depressing; I wasn’t learning a thing.
I started over one more time, but this time I hit pause when the camera went to Julia Child’s hands, and I advanced the clip one frame at a time. A universe opened up. I saw things I had never seen before.
What did I see? So many things. She never picks anything up! She manipulates over 40 items, between the knife, the whole potato, the two half-potatoes, and the slices. The knife is in her right hand the whole time, moving back and forth between two grasps. Instead of picking things up, she pushes and rolls them around, using either the knife or her hands. There is more: how the slices fall neatly instead of rolling around, how she adjusts the knife’s position and angle, and so much more.
All of that, in just 18 seconds. A universe.
The most surprising thing is that she never puts the knife down. She parks it in her hand when she wants to use her thumb and fingers for something else. That sounds crazy, doesn’t it? She parks it in her hand? But that is exactly what she is doing. It is faster than parking it on the counter. There is a name for this: the ulnar grasp. The ulna is one of your two forearm bones, the one that is closer to your pinky. So the fingers closer to the ulna are the ulnar fingers, and if you hold something between your palm and your ulnar fingers, that is an ulnar grasp.
Julia Child’s use of the ulnar grasp surprised me, because I had never seen or heard of an ulnar grasp. It was completely new to me. Or, to be more accurate, I thought it was new to me, even though I now realized I had been using it for years.
You might imagine that the ulnar grasp is rare or obscure. You would not be alone in that opinion — the ulnar grasp does not appear in any of the grasp taxonomies I have examined. But it is actually pretty common. Suppose you are removing a nut from a bolt. You use a wrench to loosen it and then you switch to your fingers. If it is not loose enough, you switch back to the wrench and then back to your fingers. The ulnar grasp is perfect for that situation.
Or, suppose you are taking silverware or plasticware from a bin, while holding a tray of food in one hand. You might pull out a knife, park the knife in an ulnar grasp, then pull out a fork, add that to the ulnar grasp, then pull out a spoon.
Or, suppose you are playing speed chess. To capture a piece, you might first pick up the opponent’s piece, park it in an ulnar grasp, then use your free fingers to move your piece and punch the time clock.
So then, I had never seen an ulnar grasp? Clearly wrong. As soon as I learned of its existence, I began to see it everywhere, including in my own behavior.
The only explanation is that there was a separate part of my mind that manages ulnar grasps and all similar behaviors, and I did not even know it existed. Put it this way. There are several mental processes involved when you slice a potato. Some are conscious, and some are subconscious. Let’s call the subconscious processes the inner robot. So earlier when I said, “I had never seen an ulnar grasp,” that was my conscious self, the part of me that I identify as me, the part that reasons, talks, and writes blog posts. And then there is this other part of me, which quite obviously knows the ulnar grasp well: my inner robot.
My inner robot and I are not completely separated. I can jump into the driver’s seat and direct the action at a high level, while the inner robot continues to operate at the lower levels. I can also constrain or encourage certain behaviors, like trying to keep my wrist cocked during a squash swing. But I cannot achieve the same speed and dexterity demonstrated by the inner robot acting alone. The squash swing depends on the inner robot absorbing the idea and adjusting its behavior.
My inner robot is quiet. It does not keep me informed of its activities. So, I was oblivious to it, and to its pivotal role. This is a particularly insidious type of ignorance. An unknown unknown. I was deceived; I thought that I knew what I was doing. The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge, as the saying goes. I did not see the manipulation behaviors happening right in front of my eyes, because I thought I knew what I was doing.
I suspect that this blindness is not mine alone. Roboticists and motor psychology experts have published numerous excellent studies of manipulative behaviors, but I see approximately zero references to the ulnar grasp. I say “approximately” because there is one paper, co-authored by myself, with the plasticware example, referring to it as a “storage grasp.” If you google “ulnar grasp,” you will find it mentioned in the context of infant development, or in the context of loss of thumb function, or I found one paper re use of the ulnar grasp by monkeys.
This blindness has implications for robotic manipulation research. For example, consider the question: How is the field progressing? How long until robots can do all the things humans can do?
Nobody knows! Some people think robots are closing in, but all those people are ignorant, just like you and I. We all share the same blindness. We all have inner robots quietly doing the real work, so we all share the illusion that we know what behaviors are employed. We don’t! When you see a video of a robot doing something cool, keep your expectations in check. The people who taught it to do that are just as clueless as the rest of us. Sometimes those robots are taught using human demonstrations, but the protocols and instruments may limit the demonstrators’ inner robots. The inner robots should be complaining, but that is not their way.
How many different manipulative behaviors will we likely find in a typical human’s repertoire? Again, nobody knows. Will it be just the few dozen previously identified, plus the ulnar grasp, plus just a few more? Or dozens more? Could it even be hundreds or thousands more? Again, nobody knows.
There is hope. New technology can give us the tools to do better studies of human manipulation. Some such studies have been published, and I am sure there will be more. If designed and conducted properly, avoiding the ignorance and prejudices of our cognitive selves, then we can observe our unfettered inner robots, and tap into that expertise hidden away in our own minds.
Notes
If you are interested in details and related stuff:
- Some discussions favor the terms physical intelligence versus cognitive intelligence. See, for example, Sangbae Kim’s TEDxMIT talk on physical intelligence. Marc Raibert, the founder of Boston Dynamics and the Robotics and AI Institute, uses the terms the scholar and the athlete.
This is all related to Moravec’s paradox, saying the things we expected to be easy turned out to be hard, and the things we expected to be hard turned out to be easy. It makes a good point, it is pithy, but of course neither the cognitive intelligence nor the physical intelligence has turned out to be easy. Regardless, you could view the inner robot and the resulting blindness we all share, as an explanation of why we expected the physical intelligence to be easy.
The Julia Child potato-slicing clip is here.
The ulnar grasp of the wrench, not previously published, is courtesy of Professor Nancy Pollard of the Carnegie Mellon University Robotics Institute.
The ulnar grasp of plasticware is from a project I worked on with Nancy Pollard. It appears in one publication described as a “storage grasp.” Yuzuko C. Nakamura, Daniel M. Troniak, Alberto Rodriguez, Matthew T. Mason, and Nancy S. Pollard. “The Complexities of Grasping in the Wild.” IEEE-RAS International Conference on Humanoid Robots. 2017.
The ulnar grasp of a chess piece is from a YouTube video: LACF NOJ Dubrovniks II (Closeup + Blitz game.) There are about 20 ulnar grasps in that 10-minute video.
I say we don’t know what behaviors are employed in manipulation. But of course there have been numerous studies of human manipulation, describing and cataloging many of those behaviors. We know a lot, but I am suggesting that we need to know a lot more. If you are curious, you might start with the Feix grasp taxonomy, and then look at the numerous papers citing it (Feix, Thomas, Javier Romero, Heinz-Bodo Schmiedmayer, Aaron M. Dollar, and Danica Kragic. “The GRASP Taxonomy of Human Grasp Types.” IEEE Transactions on Human-Machine Systems 46, no. 1 (2016): 66–77.) Then, for some work that I find inspiring, I would suggest the Byrne study of gorilla manipulation behaviors: (Byrne, R.W., N. Corp, and J.M. Byrne. “Manual dexterity in the gorilla: bimanual and digit role differentiation in a natural task.” Animal Cognition 4, no. 3 (2001): 347–361.)
Here’s a paper mentioning an ulnar grasp used by monkeys: Representation of reaching and grasping in the monkey postcentral gyrus. Iwamura Y, Tanaka M. Neurosci Lett. 1996 Aug 23;214(2-3):147-50.
Thanks to Marc Raibert, Nancy Pollard, Merrick Furst, and Cynthia Gelb for their generous help.
I think the list of physical actions people mentally bank might be endless. Is this essentially the Malcolm Gladwell 10,000 hour rule – we practice something enough (either knowingly or unwittingly) such that it gets banked in our subconscious? It makes your head spin when you start to think about how this is essentially a type of language developed and refined over millennia and handed down through training/observation.
We had an architecture reminiscent of this for the production version (not the research version people are more familiar with) of the Rethink Baxter robot. We had an AI “brain” which maps to consciousness action which was in charge of deciding at the high level what the goal of the robot was (pick up the part and put it over there) and a then a ever-growing series of “robust controllers” which were lower level physical coordination mapping of sensors<->actions (i.e. visual or haptic servoing). The brain didn’t track or know any of the details. i.e. subsumption architecture stuff. At the time all those low-level policies were created by human programmers …… a great space for current AI research.
This is really wonderful. It is a compelling demonstration of how many interesting and important things are right under our noses, but we look right through them until we adopt a method that focuses our attention on the right details.
I had my own experience of this when looking to see how experts do causal reasoning about mechanisms, so I studied language and reasoning rather than vision and manipulation. I wanted to apply the kind of verbal protocol analysis that Newell & Simon used in their 1972 book, “Human Problem Solving”, but applied to explanations about physiology created by expert physicians.
I expected to find causal networks, but the data showed objects and relationships that led to qualitative reasoning about dynamical systems (and QSIM), which are really quite different. You can read about what I found at https://web.eecs.umich.edu/~kuipers/research/pubs/Kuipers-cogsci-84.html. Johan deKleer and Ken Forbus were already working in this space, and a productive community developed around these approaches.
This is just one more example of the great methodology you describe.
The mighty of human hand is one of the first things I learned when I began doing research on robotic manipulation. The more I tried to make my robots work, the more I appreciated the capability of human hand. So many skills that would cost an engineering team multiple months to build, if possible at all, are so trivial for a human hand. This article reminds me of those experiences with great explanations.
A lot of researchers I talked with recently hold a pessimistic view of manipulation research: they think manipulation is solved (sounds familiar?) and the only thing left to do is to scale up. This is why I really like this article, which shows a few potentially very large gap that most people have ignored. The field (robotics, embodied AI, however you call them) is rapidly progressing, and I believe seeing robots doing more tasks will also make it more clear that what they cannot do, and let people better understand and appreciate the missing capabilities.
About the scale up part, I think there are still lots to do rather than data collection and pre-training. One main part is the more general robot arm / hand / body. The sensitive of human body is not ever replaceable by today’s robot.
When doing the Ulnar grasp aforementioned , the hand need to get the sense of the knife, the handle and so on, it need a more general body.
Without out such general body, maybe the scale up of robot manipulation dataset will fall in stuck.
A few years back I was diagnosed with Progressive Ataxia. PA is the progressive decay of the Cerebellum (the inner robot of your brain that also controls autonomic systems). Fortunately for me, the progression has been very slow.
The Cerebellum is highly repetitive, and compared to the Cerebrum, relatively simple. In CS/Robotics terms, the Cerebellum is a huge collection of GPUs while the Cerebrum is more like a smaller collection of CPUs. As Matt mentioned, concentrating on a particular mechanical motion of a task may force that behavior, but may also impact the “natural” dexterity one has in doing the task. The muscle memory if you will. I now concentrate on standing up and doing stairs, with an obvious reduction in spontaneity and fluidity. A couple years ago I used to have to concentrate on walking, however after forcing myself on enough long walks, I have recovered or formed new “muscle memory”. Plasticity is a wonderful thing and its own topic.
My current theory (hammer) on PA, based on CS/robotics (nail) is that everything can be based on the GPU vs CPU analogy. When disease takes away your inner robot, you lose much of your GPU stash, and you replace some functionality through the use of general purpose CPUs. This is much less energy efficient (which explains why chronic fatigue often accompanies PS). It is less spontaneous and fluid (as are many of my actions). It has slower performance, which can explain reaction delays.
Should I examine my use of ulnar grips? Is this still working, or is its failure why my outer robot occasionally drops things it thought it was holding?
Interesting article and thanks for the shout out in the notes. I think part of the ‘omission’ of the ulnar grasp and probably a bunch of others is that most grasp taxonomies focus on manipulating a single object. And as you said, the main function of the ulnar grasp is to hold and object and free the ‘important digits’ to do something else.
During my research and review of taxonomies I was operating under the assumption that one object is held with the hand. Though, I just re-read what I wrote back then and probably could make this point clearer.
And fully agree on all your points, we all use our hands effectively every day, but really don’t understand how.
Reading this post reminded me of Every Tool’s a Hammer by Adam Savage, where he describes his tendency to use anything at hand as a hammer. This made me wonder: if our “inner robot” can seamlessly adapt to new tasks and repurpose tools in ways we aren’t even consciously aware of, how can we develop robots with a similar level of flexibility? Beyond dexterous manipulation and predefined tool use, achieving true generalization—where robots can creatively apply tools beyond their intended function—seems crucial. Perhaps studying our own “inner robots” more closely could offer insights into unlocking such emergent capabilities in robotic systems.
Another great post! These insights were particularly poignant to be reading on my phone as I slowly realized my own inner robot used an ulnar grasp to scroll through this story. It’s also interesting to see the individual differences people have performing similar scrolling actions with their own hands on their own phones.
Thank you, Matt Mason for this interesting article and to all of those who supplied comments. I too have to agree that by studying Robotics, I have come to appreciate, what Sang Bae Kim calls, the superpowers we have. I do often wonder how the study of robotics can help us understand biology. In reading Matt’s article, I am that much more disinclined to believe that studying and blindly following biology will help us make better robots. I would be curious to hear other people’s thoughts on that.
Another aspect of Matt’s article is that he speaks at many levels. One of my favorite quotes from this article, and it’s one that I may put on a piece of paper with an image of Matt Mason thinking, is that “The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge, as the saying goes.” While Matt does not take credit for coining this phrase, I give him credit for saying to me for the first time. I believe we live in a time where we all have a lot of illusionary knowledge.
however, I digress. I would like to hear about biological vs robot inspiration.
Thank you very much Matt for this very stimulating text. It reminds me of the following citation from Paul Valéry’s ‘Discours aux chirurgiens’ (1938), which we used in our 2008 book on underactuated hands. I think that this citation captures the very nature of the inner robot:
‘Mais que ne fait point la main? […] cet organe extraordinaire en quoi réside presque toute la puissance de l’humanité, et par quoi elle s’oppose si curieusement à la nature, de laquelle cependant elle procède.’
A loose (perhaps not as striking) translation could be:
‘But what does the hand not do? […] this extraordinary organ in which resides almost all the power of humanity, and through which human kind so peculiarly opposes nature, from which it nevertheless proceeds.’
Opposing nature is the conscious action. The inner robot does not oppose nature.
Very interesting concept the ulnar grasp. I will get two Inspire hands very soon, I wonder whether they can do that…
Fascinating. I am wondering if there is a part of the inner robot that is shared between most/all of us and a part of the inner robot that is unique to each of us. I have big hands so I am always holding my phone with one hand in ulnar grasp and can still text with the thumb on the same hand, but my friend has smaller hands and she always holding her phone with two hands and typing with both of her thumbs…
And I feel like I can not unseen some of my inner robot behaviors now… Fascinating!
I wonder if there’s a good analogy here to the different AI tools we need to use when learning to control robots. We’ve been able to do RL optimization in sim for very focused tasks, carefully designing cost functions, and the dynamics get really good, really nuanced. The trick is trying to do broader tasks that require a lot more context; that seems to require learning-by-demonstration, either from teleop data, mocap, extraction from video, or just instructions from an engineer. The results are way less dynamic, way less confident, at least so far. And then this reminds me of how people learn to play the violin: a teacher shows them how, and they can do it poorly; but then it’s 100% necessary to practice, to learn the very dexterous details of force and impedance and uniqueness of your own body. I think that last step is a lot like RL, and also sounds to me a lot like your “inner robot.”