A subjective attempt at an objective look at manipulation
Manipulation is such a big deal that it takes a broad perspective — the broadest possible perspective — to appreciate it properly. So, let’s consider all of space and all of time, and ask a question.
Question 1: What is the most amazing thing ever in the universe?
Life! Life is amazing! It began with a bunch of stuff, randomly zinging around, obeying the laws of physics, and then after maybe ten billion years, just by chance, some of that stuff organized into life. It is quite a simple recipe — dump absolutely everything into the pot and stir.
Question 2: What is the most amazing thing about life?
Minds. Brains. After life appears, just keep waiting, and brains appear, enabling hunting, foraging, mating, writing, and other behaviors.
Question 3: Which animals have the most amazing brains?
Humans? Is it that simple? We might someday discover that the greatest challenge for a brain is to use eight tentacles lined with suckers, and hence that octopuses (not octopi — I looked it up) have the most amazing brains. But for now let’s assume the answer is humans.
Question 4: What is the human brain’s most amazing feat?
I say, manipulation. But perhaps you will disagree, favoring science, mathematics, poetry, or music. All of those are beautiful accomplishments of the human brain, and I might favor one of them myself, if we were judging based on beauty. But, the question is, which feat is most amazing, which could mean the most challenging or impressive. How could we answer that? One possible approach: Artificial Intelligence. We could try to construct a machine that mimics the human brain, and see which of the brain’s feats is the hardest to reproduce. That is where Moravec’s paradox comes in.
Moravec's paradox
Early AI researchers developed an interesting perspective when comparing physical intelligence with cognitive intelligence:
The “hard” things are easy; the “easy” things are hard.
Or, less succinctly, the things we expect to be hard turn out to be easy, and the things we expect to be easy turn out to be hard. Specifically, the early AI wizards expected cognitive intelligence to be hard, and physical intelligence to be easy. But the opposite is true. Physical intelligence is harder than cognitive intelligence.
Take chess, for example. Good chess play was a prominent goal of early AI research. Chess became an important measure of AI’s success, and there was considerable debate over whether and when an AI might play championship-level chess.
The answer: 1997. Here’s a photo of IBM’s Deep Blue playing chess with world champion Garry Kasparov. Deep Blue won the match, but Deep Blue addressed only the cognitive part: choosing the moves. Murray Campbell, on the right in the photo, handled the physical part — actually moving the pieces.
And while there are robots that move chess pieces, there is still no robot that exhibits human-level skill in manipulating the pieces. For example, humans use one hand to simultaneously place their piece and remove a captured piece. Here’s a picture, borrowed from an earlier blog post “The Inner Robot.”
On top of awesome manual skills, humans have a breadth of scope and adaptability far beyond any robot’s abilities. Humans can play chess with almost any imaginable style of board: from small travel sets where the pieces have pegs that fit into holes, to large sets — even as large as a human body. You could even use bits of paper, each with a piece’s name written on it, and a human could manage just fine.
So, the early AI folks thought the hard part — the physical intelligence — would be easy. Here’s the key question: Why does the hard part seem easy? One answer: the inner robot. Take all the subconscious processes involved in manipulation, and call them the “inner robot.” Your inner robot is taking care of all the hard stuff, like how to move chess pieces, while you ponder the easy part — choosing the next move. The inner robot manages its job brilliantly but quietly. It seems easy because for you — the cognitive self — it is easy. (See that earlier post “The Inner Robot” for more.)
But to return to the main point. Manipulation is the most amazing feat, by the most amazing brain, which is the most amazing thing about life, which is the most amazing thing ever in the universe. Manipulation is most-amazing to the fourth power. That is pretty amazing. Let’s go for one more most-amazing.
Question 5: What is the most amazing feat of manipulation?
I don’t know how to answer this. I suspect there is no single answer. There are so many different aspects of manipulation, and no single metric seems right.
On the other hand, perhaps the AI approach will serve once again. If we can build a robot with human-level performance, then perhaps one behavior will stand out as the most difficult to reproduce. And if the answer is surprising, somebody will get a paradox named after them.
One candidate would be throwing. Throwing played a prominent role in the evolution of apes into humans. Darwin himself noted that no animal comes close to the human ability to throw with speed and accuracy. Humans exhibit a special fondness for throwing, and celebrate the most accomplished throwers, such as baseball pitchers, football quarterbacks, and basketball players. For speed, perhaps a baseball pitcher wins. Such throwing involves the whole body, using the lower body’s motion to help cock the arm, so that the final throwing motion can result in a velocity of over 100 mph. It happens so quickly that it is a revelation to watch a slow motion video of it. There is a video of Drew Storen pitching, shot at 1000 frames per second, although it cannot be displayed here. You will have to watch it on YouTube. If you examine it carefully, and do a little math, you will find that the ball has a peak acceleration in excess of 100 gravities.
For accuracy, it is hard to compare a football pass versus a basketball shot versus a baseball throw versus a dart throw. But in all cases the accuracy can be truly stunning.
We humans do love manipulation, as the basis for many sports and hobbies, and it can even make our work satisfying. My granddad seemed to be very fond of sweeping his drive, but I don’t have video, so let’s look at some dazzling yo-yo tricks.
Without a doubt, the hand is quicker than the eye.
Almost all of us can handle a deck of cards — gathering the cards, straightening the deck, shuffling, and dealing. Even if it is common, it is remarkable: handling 52 objects simultaneously, then fashioning one hand into a dispenser, feeding cards to the second hand which throws them accurately across the table. Some people have taken “cardistry” a bit further.
There is an endless supply of amazing human manipulation videos. Search on “amazing street food compilation,” or just “juggling” and you will find more.
But enough about how awesome humans can be. Let’s flip the script and ask one more question.
Question 6: What is the dumbest human manipulation failure?
The inner robot does many things well, but some motion planning problems are too tough for either the inner robot or the cognitive self. My favorite is the stuck head problem. Some kids are skinny enough to squeeze through a narrow passage, except for their heads. If you see such a stuck kid, without seeing how it happened, it’s unlikely you would figure it out. Here’s a video:
You would think that any human would be familiar enough with the human body to solve this little puzzle. But if you search on “head stuck” you will find many more examples.
There are other physical problems that humans are surprisingly bad at. Humans find the visualization of spatial rotations challenging, and of course there are many challenging mechanical puzzles — the horseshoe puzzle being one of my favorites. Can we gain any insights into the nature of physical and cognitive intelligence from such puzzles?
Isn’t it interesting, that the universe’s most accomplished manipulationists have such notable weaknesses? There is plenty of room at the top! Will robots be stumped by the stuck-head puzzle? Will the threat to human pre-eminence inspire humans to even greater heights? Should we team up with the robots? What new feats of juggling, cardistry, or yo-yo might we reach, working together with robots? Perhaps we are entering a new era founded on a deeper appreciation of the glory of manipulation.
Notes
Moravec’s paradox is named for Hans Moravec, who described it in his 1988 book. Highly recommended:
Moravec, Hans.
Mind children: The future of robot and human intelligence.
Harvard University Press, 1988
Yup, manipulation. The most important frontier in robotics.
It feels like a fundamental challenge in manipulation is that we often do not know how to write down the objective. And because our “inner robot” handles so much of the complexity for us, we are often not consciously aware of what objective, constraints, or intermediate representations we are actually using. Do you think this abstraction away from the lower levels of manipulation is partly what leads to some of the human weaknesses you describe?
On a side note, may I suggest rope tricking to your list of manipulation feats: https://flying-knots.github.io/
Interesting question. Seems like there would be some trouble, where the conscious self wants one thing, but inconsistent with inner robot’s objective or constraints. Can we think of examples? Re rope tricks — I saw that paper! Good stuff. I am tempted to create a gigantic gallery of cool manipulation, but it would be a big time sink.
We achieved a dynamic running gait on Cassie that I think approximates the same physics we see from animal running, and we did it by writing cost functions for a reinforcement learning algorithm to find a good policy. Lots of groups are now doing much the same thing, sometimes seeded by a trajectory (i.e. DeepMimic). I do not believe that is feasible for manipulation. Walking and running is simple enough that we can write equations to tell the RL what’s better or worse; but for manipulation? It’s so complex, and involves so much estimation of the environment (and/or robustness to variation in the environment).
I am still not sure what the right technical path will be to achieve capable manipulation, though we’re seeing a lot of new ideas to try. I think locomotion is essentially solved.
I wonder if there might be some more forms of locomotion that also share the same challenges as manipulation, where the goals are hard to quantify. Potentially, really contact-rich interactions. Sort of like treating the legs as manipulators, “manipulating the ground”. I don’t know if I can classify this as locomotion though: https://www.youtube.com/watch?v=HqhM6uCaINE
agree totally on the octopus brain. My vote for question 4 is extrinsic dexterity via tool use! Taking a high dof dextrous system and projecting it onto a simple manipulator. And doing so nearly zero-shot on novel simple manipulators all the time. (example: children pick up a stick to reach things on the top shelf, using the shelf as a fulcrum and the stick as the lever).
Yeah, interesting. We fashion a mechanism, a machine, comprising our own bodies along with one or more elements in the environment. I wonder what the limits are, the range of such devices we can construct so quickly. I have a blog post started related to this, but don’t know when it will be ready, if ever.
Would love to read it someday (or pre-read) : -)
You could make the argument that music (playing the violin) IS an exquisite manipulation problem.
great read as always
Yes!!! I did not get into music performance, because it is a bit complicated, combining both manipulation prowess and other more artistic things. But, maybe it would provide the best examples. When I think of the speed and complexity of hand movement, I think of pianos. But violins … way over my head.
The inner robot is fundamental but the “outer body” has an essential role…
Thanks for the note, Giulio! I agree, but I doubt I can handle the point well. When I read about embodied cognition or artificial cognition, part of me wants to agree, and part of me wants to argue. Maybe I should try harder. What’s the right paper to start? This one? Sandini G, Sciutti A and Morasso P (2024) Artificial cognition vs. artificial intelligence for next-generation autonomous robotic agents. Front. Comput. Neurosci.
Agreed 100%! It’s counter-intuitive, but launching a rocket into space is much easier than getting a robot to tie a sneaker.
As robots advance to the point of being able to do some of the really complex manipulation tasks that we can do, I bet, that at least for a while, people will start doing manipulation tasks that the only reason to perform then will be to show up the robots! Then, of course, the robots will learn those and out-perform people, because they will have egos too. It’ll be interesting to see where this “arms” race will go. 🙂
I tried to think of Question 6, dumb human manipulation failures.
– If my arm or fingers were a bit longer, I could reach behind my back more easily.
– If my last two fingers (ring and little) were stronger, I could carry more in one hand. Say a cup of water plus a small plate.
I can believe there are good reasons for why things are not so. Those might still not be exact candidates for the kind of manipulation puzzles or illusions as you asked for (which could have properties like simple to state, universal, surprising, some combination of brain and hands, among others).
So now that manipulation is the last frontier where AI has not reached close to human levels as compared to tasks that are mainly done on a computer (programming, accounting,… ) I wonder what that means for the next generation in the workforce. A lot of the now ‘good’ jobs (programmers, lawyers, doctors, even researchers?) have a much greater risk of AI replacement compared to less prestigious jobs (plumber, electrician, construction worker). Will we see this hierarchy reverse in 20 years?