Tools
We in robotics often regard humans as the gold standard, but perhaps that is not quite right. The gold standard is not the naked human, but the human with tools. From either an evolutionary perspective or a robotics perspective, our vision of humans should include tools.
Evolution
Let’s start with the evolutionary perspective. The human species is defined, in part, by our tools. Here is the story:
- We went bipedal. Humans are the only habitually bipedal mammal. This freed up our hands for the next step.
- Our hands, arms, and even our hips evolved for throwing. Humans can throw missiles — rocks, clubs, spears — with power and accuracy unmatched by any other animals. And let me pause here to point out that a missile is a tool.
- Our hands and wrists evolved further to support tool manufacture.
Here is a great example illustrating the advances in human hand anatomy. Some scientists tried to teach a bonobo (also known as a pygmy chimp) named Kanzi to use rock knapping to make stone tools, as humans have been doing for eons. But non-human apes like Kanzi have less wrist mobility, and also lack certain grips. Kanzi never approached human-level knapping skills. If you are skeptical, you may wonder if Kanzi was smart enough or motivated enough. Well, Kanzi’s humans put a treat in a box and closed the box with twine, as motivation. Kanzi invented a different way to produce the necessary tool. He took one of the rocks, slammed it down on the concrete floor, and took a shard to cut the twine. When they put down a carpet and tried again, Kanzi found the carpet seam, pulled it up to expose the concrete floor, and repeated the performance. Definitely motivated, and definitely smart enough — he outsmarted the scientists.
Here is another one. Two scientists raised a chimp named Viki as if she were their own child. Among the many interesting things they tried with her, they taught her to use a hammer.
As you can see in the video above she tries several awkward-looking grasps, and never finds the “right” grasp. Chimpanzees lack the palmar opposition mobility to grasp a hammer in the oblique grasp used by humans. The humans’ hammer is the wrong design for a chimp. We should design some tools for chimps!
The evolution story in short: nature designed our hands to fit naturally-occurring tools, and then refined our hand design to manufacture tools. Who gets the credit for designing the early manufactured stone tools? Early humans or evolution? Archeologists are not in agreement, and I wouldn’t want to put forward an opinion, but I do love one paper’s title: “The Acheulean handaxe: More like a bird’s song than a Beatles’ tune?” Isn’t it amazing to think that our DNA could include tools in the definition of our species?
The word tool
I am using the word tool broadly. I include everything used by the agent (robot, human, or chimp) that is not part of the agent, to get the job done. That includes an anvil, a saw whether handsaw or table-saw, as well as a workbench or countertop, and so on.
Sometimes it isn’t clear whether to call something a tool. Here’s a tricky case. You can’t crack a walnut by squeezing it with just one hand, but if you grasp two walnuts and squeeze, one may crack. The one that doesn’t crack is the tool, but you won’t know which one is the tool until afterward.
Robot tools
One goal of AI and robotics is to better understand ourselves, and that should include our relationship to our tools. So, what can we learn from robot tools?
Conventional industrial robot tools are different from human tools. When you buy an industrial robot arm, it comes without a gripper or any other “end effector” (EE) or “end of arm tooling” (EOAT) as they sometimes call tools. Instead, the arm ends in a flange, and the customer buys the tool separately and bolts it on. It could be a welding torch, paint spray gun, glue gun, grinder, polisher … it could even be a gripper.
I don’t recall ever seeing an industrial gripper grasp another tool. Instead, they will switch the gripper out for the other tool, using a tool change station. Some robots even have a turret at the end of the arm, enabling fast switching of tools, like a motorized Swiss army knife.
Robots have advantages in tool use. The tools are mounted rigidly and precisely. They can have pneumatic, hydraulic, and electrical connections, so that the tool can be smart, sensorful, and powered.
We humans have to cope with disadvantages. We cannot dismount our hands, and cannot mount tools directly at the wrist. Our nervous system does not extend into the tool. A human with tools is an awkward mix of technology — part engineering, and part biology. A robot is 100% engineered technology, giving them advantages in tool design and tool interfacing.
Where is the robot?
There is a deeper difference between robots and humans that compounds the confusion, and that is how to define a robot’s physical extent. Where is the robot’s body? Most industrial robots have just one tool attached, which we might or might not consider to be part of the robot’s body. If we have two robot arms mounted close together, we might consider them to be a single bimanual robot, which means the bench on which they are mounted should be considered part of the body as well. In fact, it is often observed that if you have several arms rigidly mounted in a factory, they could be regarded as a single robot with many arms. The definition of a robot’s body is ambiguous.
Where is the human?
The situation could hardly be more different for a human. We know exactly where to draw the line. The body is one contiguous mass of well-known shape. Furthermore, every part of the body bears a label identifying the individual — the DNA. The extent of a human’s body seems to be well defined.
But, perhaps we should not let our DNA define us. That sounds crazy, but I am a bit jealous of robots. I love my tools, but my relationship to my tools lacks the intimacy that a robot has. A robot’s relationship to its tools is so intimate that sometimes you cannot say where the robot ends and the tool begins. I wonder, should we humans also look at our tools that way, as extensions of our bodies, despite their lack of the DNA labels? Perhaps we should broaden our view of the body to include the tool.
(There is some neuroscientific work suggesting that the brain does incorporate tools into its view of the body during tool use. See the notes for a few details.)
Or, we could go the other way, and narrow our view of the human body. Think of yourself as a brain, with a body attached to give the brain access to the physical world. The body is not necessarily permanently attached. Body parts sometimes get lost or switched, either by transplants or prosthetics. A self-centered brain might envision itself as just the brain (including the inner robot, blog post here), viewing the body as a proximal tool possibly attached to distal tools.
A robot is itself a tool
There is one other point to consider. Isn’t a robot a tool? We design and build tools to help us manage physical stuff. Over the centuries our tools have become more and more sophisticated. We started with the simplest tools — rocks and clubs — and have progressed all the way to robots.
Here is a concrete example. Consider a computer-controlled milling machine. I think we can all agree it is a tool, in the broad sense of the word. Well, how about a milling machine that shapes the surface of a human thigh bone, as part of surgical hip replacement? That tool, commercialized as “RoboDoc,” was an important step in the beginning of medical robotics. Some robots are definitely tools.
And what about humans?
Robots are tools? Here is one last point. Let’s imagine that AI and robotics have progressed to the point that robots have their own individual intentions. Now, suppose you are working together with a robot, sharing the task as equals. Which is the master? Which is the tool? Perhaps the master is the one with the initial intent. Let’s imagine that a robot asks you for some help, perhaps to remove a splinter from its finger. Doesn’t that make you the robot’s tool? If so, we need to revise the tools story by adding one more step:
- We went bipedal, freeing the hands for manipulation.
- We evolved to use natural tools like rocks and sticks more effectively.
- We evolved to manufacture new tools.
- Our tool designs advanced in sophistication, including robots.
- And perhaps one day, when robots rival humans, we might occasionally serve as tools ourselves — our tools’ tools.
But don’t worry. If that day comes, the robot won’t say, “Hey, tool! Get to work on my splinter!” Instead, judging from today’s chatbots, the robot will say, “Oh, Great and Powerful Master! I, your humble servant, have a splinter and am too feeble-minded to deal with it.”
Notes and references
Here’s a favorite book, from which I adapted some figures:
- Napier, John, and Russell H. Tuttle. Hands. Princeton University Press, 1993.
The story of Kanzi is in a fun book:
- Savage-Rumbaugh, E Sue, and Roger Lewin. Kanzi: The ape at the brink of the human mind. Wiley, 1994.
You can see Kanzi’s rock knapping in this video:
I have not found any video of Kanzi slamming the rock onto the floor.
Here is an early paper about Viki:
- Hayes, K.J., and C. Hayes. “The intellectual development of a home-raised chimpanzee.” Proceedings of the American Philosophical Society 95, no. 2 (1951): 105–109.
Numerous Viki videos have been posted on the internet, but many have disappeared. This site contains an archive of Viki videos, but I have only been able to download a few videos.
For the issue of whether evolution designed early tools, see:
- Corbey, Raymond, Adam Jagich, Krist Vaesen, and Mark Collard. “The Acheulean handaxe: More like a bird’s song than a Beatles’ tune?” Evolutionary Anthropology: Issues, News, and Reviews 25, no. 1 (2016): 6–19.
My discussion of evolution is largely condensed from my earlier paper, which provides some additional detail.
- Mason, Matthew T. “Toward Robotic Manipulation.” Annual Review of Control, Robotics, and Autonomous Systems 1, no. 1 (2018): 1–28.
Here are some sources I drew from:
- Marzke M. 1983. Joint functions and grips of the Australopithecus afarensis hand, with special reference to the region of the capitate. J. Hum. Evol. 12: 197–211.
- Marzke MW. 1996. Evolution of the hand and bipedality. In Handbook of Human Symbolic Evolution, ed. A Lock, CR Peters, pp. 126–47. Oxford, UK: Clarendon.
- Richmond BG, Roach NT, Ostrofsky KR. 2016. Evolution of the early hominin hand. In The Evolution of the Primate Hand: Anatomical, Developmental, Functional, and Paleontological Evidence, ed. TL Kivell, P Lemelin, BG Richmond, D Schmitt, pp. 515–43. New York: Springer.
- Williams-Hatala EM. 2016. Biomechanics of the human hand: from stone tools to computer keyboards. In The Evolution of the Primate Hand: Anatomical, Developmental, Functional, and Paleontological Evidence, ed. TL Kivell, P Lemelin, BG Richmond, D Schmitt, pp. 285–312. New York: Springer.
- Johanson DC, Taieb M. 1976. Plio-pleistocene hominid discoveries in Hadar, Ethiopia. Nature 260: 293–97
- Roach NT, Venkadesan M, Rainbow MJ, Lieberman DE. 2013. Elastic energy storage in the shoulder and the evolution of high-speed throwing in homo. Nature 498:_ 483–86.
For details on the experiments showing that the brain might be expanding its body image during tool use, see:
- Iriki, Atsushi, Michio Tanaka, and Yoshiaki Iwamura. “Coding of modified body schema during tool use by macaque postcentral neurones.” Neuroreport 7, no. 14 (1996): 2325–2330.
For more about Robodoc, and about surgical robotics generally, see:
- Taylor, R.H., and D. Stoianovici. “Medical robotics in computer-integrated surgery.” IEEE Transactions on Robotics and Automation 19, no. 5 (2003): 765–781.
Wonderful essay!
The “body schema” concept is worth a whole essay of its own. If you are using a tool like a screwdriver, the tool tip clearly becomes part of your body schema. I think this is closely related to the child developmentally learning to distinguish “self” from “other”. Initially, the child’s hand is something they can move, but not actually control according to their intentions, but within a month or two, they can, and the hand becomes part of “self”.
For an adult, or even an older child, a laser pointer becomes part of “self”, with the spot of laser light being the “tool tip” that you can control to directly reflect your intention. Once upon a time, a friend of mine who was an expert pistol shot said that shooting his pistol felt like reaching out with “a very long finger” and touching an intended point on the target far away. (And, of course, if you reach out with, say, a 10-foot-long pole and try to touch a specific point, you would quickly discover that it is not part of yourself. Probably that would change with sufficient practice and expertise.)
Moving on to your final point about humans as tools. Michael Tomasello, in his 2019 book, “Becoming Human”, discusses two important steps in the evolution of humans that took place about 400K and 100K years ago. First, he says that humans evolved the ability for two people to create a third, virtual person, “we”, who was able to accomplish certain tasks (like capturing an antelope) that neither of the two people could do alone. Each of the two physically existing (non-virtual) people is part of “we”, and they work together to accomplish their goal. This was the start of making humans into “super-cooperators”. Tomasello called this “joint intentionality”. A while later, about 100K years ago, we evolved “collective intentionality”, generalizing from a pair of people to a larger group.
I think this perspective on cooperation and the virtual person “we” may well be an essential piece of the puzzle for humans, AIs, and robots being able to live together safely and successfully.
The “virtual person” thing. Argh! Another book to read! But it does seem that collaboration, social stuff, culture, is one of our best things. Ants are pretty good collaborators, too, but I guess the mechanism, the foundation, is quite different.
A colleague of mine pointed out that this is very much reminiscent of Dawkins’ Extended Phenotype. We can further explain the phenomenological observation that humans have tools as a selection mechanism for how we became what we are.
Beyond the philosophy, you are describing in concrete terms what inspired a lot of my thinking around the recent Koala work that we published: https://koalagripper.rai-inst.com/. It was built for robots to wield human tools in mind, allowing for a power grasp that was challenging for parallel jaw mechanisms but not covering the entire human grasp spectrum. For a grasp spectrum we might look to my advisor, Peter Allen’s work on Eigengrasp. I think part of the challenge with grasping tools is determining how to program the complex behavior. In a 22 DoF gripper, it is hard to model both the position and forces in response to high dimensional stimuli.
Thank you for the excellent read!
Thanks for the pointers. Re Koala gripper — I hadn’t seen it. Very cool. I don’t think I have seen any other bot handling human tools so well. So either the gripper is awesome, or my memory is pitiful. Or both! And, re Dawkins — damn now I have to read another book!
I was enamored of animal documentaries from an early age and watched thousands of episodes of Wild Kingdom, Wild America, Jacques Cousteau and many others. I was always fascinating by the many examples of animals and tools. Some like the cephalopods seemed even more intelligent than others in solving problems but using a form of manipulation that we puny humans don’t seem capable of. You made a point of the body as a tool when grasping and holding or climbing and descending.
At one point, perhaps in middle school, I remembered an assertion that humans differed from animals because we use tools and they don’t. But there are hundreds, maybe thousands of examples of animals using tools. Otters smash mussels with a rock while floating on their back, crows seems preternaturally intelligent, even scary in figuring things out. We’ve all witnessed dogs and cats doing something that is tool like in your definition. Open cabinets, doors, packages, etc. Sometimes through brute strength but occasionally through cleverness. Loved your examples.
Others include dolphins protecting their snout while going after food, Crows fashioning tools, Chimpanzees making all kinds of tools (Observed by Jane Goodall. Just ask Claude to cite tool use amongst animals and there are many many examples.
You said that only humans are habitually bipedal but most flightless birds are the same way too.
Kevin
It is so strange to think that people used to say only humans use tools. You gave lots of great examples. You mention crows making tools — that’s the one I find most surprising: a crow bending a wire into a hook to pull a treat out of a tube. (25. Auersperg AMI, von Bayern AMP, Gajdon GK, Huber L, Kacelnik A. 2011. Flexibility in problem solving and tool use of kea and New Caledonian crows in a multi access box paradigm. PLOS ONE 6:e20231 and 26. Klump BC, Sugasawa S, St Clair JJH, Rutz C. 2015. Hook tool manufacture in New Caledonian crows: behavioural variation and the influence of raw materials. BMC Biol. 13:1)
Re bipedalism: I had to go back and check to make sure I got it right — the statement is that humans are the only habitually bipedal *mammal*. Sort of arbitrary, though, to exclude birds, isn’t it. I wonder if flightless birds will evolve to have hands instead of wings. Bats kept their thumbs — they have wings and just a little bit of a hand.
I just saw an interesting demo posted today by a robot company. They attach their dual-arm robot to a industrial robotic arm, and robot employs different kinds of tools and materials with its grippers and dexterous hands to complete various tasks.
The video is called “Together with ZINOVA’s Tool Intelligence, TRON 2 takes on complex construction workflows.”
https://www.youtube.com/watch?v=8KDxnT-4zl0
Thanks for this amazing philosophical groundwork. Lot’s of discussions to bootstrap off of it. A key takeaway for me is that the abstraction layers / boundaries can change depending on what is most useful for the task at hand and it’s worth acknowledging this from the get-go.
On the learning side, my fascination remains how human brains (and likely robots in the near future) can continually and smoothly adapt their behaviors to “adopt” a tool as an extension of what is controllable. Even sensitizing the “built-in” sensors like touch and proprioception to estimate what is happening at the distal end of the acquired tool.
What comes to mind is picking up a hockey stick to reach an object that fell behind the fridge (maybe for the first time). There’s some sequence like:
1. See hockey stick, associate it with ‘useful for task’.
2. Adopt it as part of your body with a firm grasp.
3. Slide it under the fridge and start touching the wall at the back.
4. Realize what it feels like to sense “through” the stick using your built in sensors.
5. Refine control dexterity slowly and implicitly build a “low-fi” model of what is happening back there behind the fridge based on what you feel through the stick.
6. Hook the object and pull it forward, but it gets stuck.
7. Collect a bit more “blind data” with your sensor model (which is also improving).
8. Retry with your new guess of what it’s stuck on and loosen grasp.
9. Hook object, pull and succeed!
All of this realistically happening within a 2 minute window… 🤯
Let’s not even get started with people who have prosthetics that adopt the tools over much longer “permanent” timeframes.
I’m not sure why that emoji ended up the size of the whole page… sorry about that, haha.
Liberate ourselves from tradition. Become a tool / slave of the liberal class.
Liberate ourselves from family. Become a tool / slave of the liberal class.
Liberate ourselves from our sexuality. Become a tool of the liberal class.
Liberate ourselves from our sex. Become a tool / slave of the liberal class.
Liberate ourselves from our bodies. Become a tool / slave of the liberal class.
Liberate ourselves from our thoughts. Become a tool / slave of the liberal class.
Is being a tool really that new? A robot functions at a high level, and will function at higher levels moving ahead. But haven’t human systems already been using us as tools for some time? Are they not also functioning at a level higher than us, using us as tools for some time? I’d write more but I gotta pick up my Ubereats kthxbye!
I would like to give Matt Mason a hand for such an interesting and perhaps introspective article. Thank you
Thanks for the glimpse into some very interesting ideas! Darn, now I need to go read your paper for more of the evolution discussion. I also liked the illustrations, both your sketches and the figures from Napier and Tuttle.