Sensorless Manipulation

How do you know where things are? Sensors? No, sensors are only part of the story. Prediction, based on past actions, also plays a role. It is even possible to do interesting things with no sensing at all.

Sensing and predicting

What is a robot? Some would say, “the intelligent connection of perception to action.” Here is a diagram capturing that view:

A glance at that figure suggests that knowledge of the world state comes only through sensors.

But, the sense-think-act thing can be misleading, since who knows what is going on in that “think” box? There could be a loop hidden in the box, where the actions are being used to make predictions about the state of the world. Let’s pull that “prediction” loop out into the open:

And if we want to explore the significance of prediction, and at the same time annoy some sensor nuts, let’s get rid of that “sense” arrow!

Now we have a robot with no sensors, yet it can still gain knowledge of the world state. The intelligent connection of action to action. How far can it go? Can we build a sensorless robot that does something interesting? This brings me to one of my favorite projects: tray-tilting.

Tray-tilting

 In 1984, Mike Erdmann and I wanted to explore the question of how robots might know where things are, and we specifically wanted to address the common misconception that sensing is required.

Mike was a PhD student at the MIT AI Lab, advised by Tomás Lozano-Pérez.  He was visiting my lab at CMU for the summer.  So, we talked about these issues for about six weeks, and came up with an idea that Mike could implement and test in just six more weeks, before the end of the summer.

Here’s the idea:  a tilting tray.  Suppose you drop an object into a rectangular tray with walls.  If you tilt the tray steeply enough, the object will slide into a wall, or into a corner.  Already you have reduced the uncertainty in the object’s location, without sensors.  Can we take this further?  Can you go all the way?  Is there a sequence of tilts that will eliminate all uncertainty?  It would take some work to answer that question.

So Mike adapted some software from his MS thesis, and built a planner.  You give the planner the object’s shape, center of mass, and coefficient of friction.  Then the planner searches for a sequence of tilts that moves the object to a predictable pose, regardless of its original pose, thus reducing the uncertainty to zero.

Six weeks later, the system was ready to test.  We had an industrial robot arm holding the tray up in the air, using its wrist motions to tilt the tray.  Mike’s planner was running on a state-of-the-art Lisp Machine attached to the robot’s controller.

Mike dropped an Allen wrench into the tray, gave it a spin, and hit Go.  The tray tilted this way and that, the wrench slid around, and then it stopped.  Here’s the sequence we saw.  You can also check out the video here.

Tilt 1 aligned the wrench with the bottom wall. Tilts 2 and 3 moved it to the left corner, then right corner. Tilt 4 did nothing. Tilt 5 was the most interesting: the wrench slid upward while rotating backward. Then, tilt 6 moved the wrench back to the bottom right corner, and then two more tilts followed that did nothing.

But, when I say that a tilt “did nothing,” I mean it did nothing for Mike and me. We knew where the wrench was. For the sensorless robot, these do-nothing tilts were eliminating possibilities, reducing the set of possible world states.

So, when all was done, the wrench was in the bottom right, in that upside-down-L orientation. But, was that where the robot expected it to be? Did the robot know where the wrench was?

Mike and I looked at the wrench. Then we looked at each other. I said, “Did it work?” Mike said, “I don’t know!”

We laughed. Mike took a seat at the computer terminal for a while, did some work with the planner, and then looked at me, smiled, and said, “It worked!”

What an interesting phenomenon. Usually you tell a motion planner where you wish the stuff — blocks or whatever — to be. And it is easy to see whether the planner worked or not. Is the stuff in the right place, or not?

But in this case, the goal was not for the wrench to be in a particular place. The goal was for the robot to know where the wrench was. Mike had not bothered to write the code for the planner to tell us what it knew, or where it expected the wrench to be. But later he did. Here’s the sequence of the robot’s belief states.

Each panel shows the possible initial wrench poses, and a range of tilt angles that will take us to the next panel.

A few notes:

• First, the tilt direction can vary within that angular range without affecting the outcome. The robot used the middle of the range to get the most robust tilt.
• Second, when the wrench is drawn in the middle of a wall, it might be anywhere along the wall. So position 8 appears to be completely determined, but it’s not. Position 9, in the corner, is completely determined.
• Third, there seems to be a bug in panel 6, which shows a single angle instead of a range, suggesting the tray should be tilted straight downward. That would not be robust. The tilt should be inclined to the right, to maintain contact with the right wall. (This figure was originally published forty years ago, so fixing it now seems pointless.)

One other noteworthy thing. Neither Mike nor I could say whether the wrench’s position was completely determined based just on prediction. And we could not have produced such a plan, except maybe Mike could have with a paper and pencil and a lot of time. Humans do sometimes use prediction to address uncertainty, with or without sensors, but it seems we are unable to plan a sequence of funnels in the way that Erdmann’s tilting-tray planner did. (Funnels?  Did I say funnels?  What is a funnel?)

Funnels

Prediction has been well-established in estimation theory for decades. Prediction figures prominently in Bayes filters, Kalman filters, and POMDPs, for example. Generally they assume probabilistic models: Gaussians, mixtures of Gaussians, and so forth.

But the tray-tilting planner did not use a probabilistic model. It modeled uncertainty as a set of possibilities. Sometimes we referred to it as a possibilistic model.

There is a word for an action that moves one set of possibilities to another set of possibilities: a funnel. A grain of sand can start anywhere in the mouth of the funnel, and will land anywhere in the end of the funnel.

Funnels and possibilistic models are sometimes the right concepts for manipulation in the face of uncertainty. Consider a mug sitting on your desk. If you use your eyes to estimate its pose, without prediction, you would know that the mug’s bottom is close to the desktop, perhaps modeled as a probability distribution centered half a millimeter above the desk. But using prediction, with the mug motionless in a gravity field, you can say that the mug’s bottom coincides with the desktop. What is the separation? Zero. Prediction and sensation work together, but we can credit prediction, especially in the presence of contact, for extraordinary precision.

This is not an argument for using possibilistic models in general. Turning to the mug example again, how would you describe the possible horizontal variations? With no contact limiting horizontal motion, we might settle for a less precise probabilistic model.

So then, the tray-tilting system eliminated all uncertainty by using a sequence of funnels, with each funnel’s tail covered by the next funnel’s mouth. The first funnel’s mouth covers all wrench poses in the tray, and the last funnel’s tail is a single possibility: the upside-down L shape, in the lower right corner.

Why do roboticists forget prediction?

There is one question I have puzzled over for my whole career: Why do we roboticists forget the role of prediction, and give all the credit to sensors? Many robots depend on prediction. Yet, when some roboticist says you have to have sensors to get rid of uncertainty, everybody just nods.

Maybe our own subjective experience as humans is misleading. Doesn’t it seem like your eyes are giving you a complete and detailed picture? And that after you shut your eyes, you can produce a complete and detailed visualization in your head? It is not so. Psychological experiments revealing the limits of sensing can be quite surprising.

Here’s another possibility. Maybe the neglect of prediction occurs because sensing is just simpler to understand, and so readily apparent. But prediction almost always plays a role too, especially in manipulation, since contact produces such dramatic decreases in uncertainty. Next time you hear somebody say that uncertainty requires sensors, don’t just nod. Take them to task! Call them out! Prediction! Funnels!

Mike Erdmann. In memoriam.

Mike Erdmann passed away a few months ago, on March 11, 2026. We were good friends for over forty years. We co-directed the Carnegie Mellon Manipulation Lab, and occasionally collaborated on research projects, including the tray-tilting project described above. Mike had a joint appointment between Robotics and Computer Science.

Mike’s work was deep, and as time went on, Mike went deeper. He loved math, and he continued to pursue mathematical approaches even after many other mathematicians moved on from robotics.

Mike was the least self-promotional academic I ever knew. When others were publishing many papers per year, Mike was spending multiple years per paper. He usually published in journals without explicit page limits, so that he could give each idea its due. Later in his career, when his papers were too lengthy for any journal, he published on arXiv. A 2019 paper was 75 pages long. A 2017 paper was twice as long: 150 pages.

Mike was also an excellent teacher. He created two courses for the CMU Robotics PhD program and taught them for years. One was a geometry course, addressing both computational and differential geometry. When the department head told him he would only be teaching it for one more year, I signed up to take it. I can confirm from first-hand experience that Mike was an excellent teacher. The second course was on the mathematical foundations of robotics, a required course for Robotics Institute graduate students, and attended by many others. Mike never published his course notes, but others copied them for use at other universities. He also taught an undergraduate CS course for many years, and won the School of Computer Science teaching award for it.

You may wonder how someone like Mike managed to survive as a Professor. “Publish or perish” was not Mike’s way. I recall one of his promotions well. The process began several months ahead of the promotion committee meeting, with one of the department heads looking at Mike’s publication record, fretting over it, and even wondering if it would be possible to postpone the promotion for a year or two. But his worries were misplaced. The process ended with a set of recommendation letters describing Mike as one of the great minds in the field. Mike’s case sailed through. I am proud that Carnegie Mellon could look beyond paper counts and citation counts and recognize Mike’s excellence.

Finally, Mike was a great human being. We sometimes traveled to professional meetings together, so I spent a lot of time with him. On our free time we toured Venice and some tourist sites in Japan, and spent six weeks together in Trento, Italy. The only problem was that Mike liked to start late in the day. One time we decided to visit Kinkaku-ji, the Temple of the Golden Pavilion, in Kyoto. Starting from Osaka, it is a day trip. Our Japanese friends told us exactly which train to take, early in the morning. But we started at noon, stopped at a noodle shop, stopped to let some school kids take pictures with us, and made our way casually through several neighborhoods. The temple was closed by the time we arrived, but so what? It was a fun and memorable day.

Mike was gentle but strong. Even when he was angry he managed it well. He always tried to do the right thing, both personally and professionally. He was fun to be with. I miss him.

Notes on sensorless manipulation

The tray-tilting work is described in two papers: Erdmann, Michael A., and Matthew T. Mason. “An Exploration of Sensorless Manipulation.” In IEEE International Conference on Robotics and Automation. San Francisco, CA, 1986, and a subsequent paper in the IEEE Journal of Robotics and automation, same authors and title, 1988.

The definition of robotics as “the intelligent connection of perception to action,” is due to Mike Brady. (Brady, Michael. “Artificial intelligence and robotics.” Artificial Intelligence 26, no. 1 (1985): 79–121.)

There is a considerable history of work addressing uncertainty that I suppressed in the interests of readability. The below is a subset. Apologies to those I leave out.

My own early career seemed to revolve around uncertainty. My PhD thesis focused on the use of contact to reduce uncertainty, particularly in grasping operations. (Mason, Matthew T. “Mechanics and Planning of Manipulator Pushing Operations.” International Journal of Robotics Research 5, no. 3 (Fall 1986): 53–71.) It was inspired by several examples in industry and in labs around the world — fodder for another post, perhaps. Randy Brost later explored some of the issues more fully and systematically. (Brost, R. C. “Analysis and Planning of Planar Manipulation Tasks.” CMU-CS-TR-91-149. PhD diss., The Robotics Institute, Carnegie Mellon University, January 1991.) Then Ken Goldberg followed up and took it to a new level, showing, for example, that any planar polygonal shape can be oriented, up to symmetries, by a sequence of squeezes or pushes. (Goldberg, K.Y. “Orienting polygonal parts without sensors.” Algorithmica 10, no. 2 (1993): 201–225.)

And just as I was finishing my thesis, I got involved in a project with Lozano-Pérez and Taylor, developing a general planning technique focused on uncertainty. The technique generated a bit of buzz at the time, and was called LMT. (Lozano-Pérez, Tomás, Matthew T. Mason, and Russell H. Taylor. “Automatic Synthesis of Fine-Motion Strategies for Robots.” International Journal of Robotics Research 3, no. 1 (1984).)

Uncertainty is a central concern of industrial automation. Yet, roboticists routinely dismiss factory automation, saying that the uncertainty has been engineered out. Well, first off, they should be asking how was the uncertainty engineered out, and learning from it. And, anyway, there remains a great deal of uncertainty, which industrial robots deal with using both sensing and prediction. One paper that addresses processes for reducing uncertainty is by Art Sanderson. Notably he also introduces use of information theory and entropy. (Sanderson, A.C. “Parts entropy methods for robotic assembly system design.” In Proceedings of the 1984 IEEE International Conference on Robotics and Automation, 600–608.)

Later Sanderson’s student Michael Peshkin produced a project similar to tray-tilting in some ways, but using fences crossing a conveyor belt at various angles. (Peshkin, Michael A., and Arthur C. Sanderson. “Planning Robotic Manipulation Strategies for Workpieces That Slide.” 4, no. 5 (October 1988): 524–531.)

Sometimes automated factories use a tilted tray to reduce uncertainty. Typically the tray is set at a fixed angle, and vibration might be used to eliminate friction. Erdmann’s and my tilting-tray project was inspired by some previous work by Grossman and Blasgen using such a tilted tray. (Grossman, D.D., and M.W. Blasgen. “Orienting mechanical parts by computer-controlled manipulator.” IEEE Tran. on Systems, Man, and Cybernetics 5, no. 5 (1975): 561–565.)

I introduced the word “funnel” in a 1985 ICRA paper. Originally I used it in an NSF proposal, at the suggestion of Marc Raibert. But Raibert doesn’t remember, so I think I’ll just keep quiet and take all the credit. (Mason, Matthew T. “The Mechanics of Manipulation.” In IEEE International Conference on Robotics and Automation, 544–548. 1985.)