What are we learning in the warehouse? Clutter matters.
What is clutter, and why does it matter?
Of course you already know what clutter is. If you live with a lazy roommate or a stereotypical teenager, their stuff might be tossed all over the place, making it difficult to walk through a room or even find a place to sit. That is clutter.
However, in robotics we use the term differently. To us, clutter means anything that is in the way. If a robot wants to grasp block A, and block B is on top of it, block B is clutter.
The “Grasp A” task is an example from the “blocks world,” moving simple shapes around on a tabletop. Some tasks are as simple as “Grasp A,” but others involve a little more clutter. Let’s make a clutter spectrum, showing different levels of clutter, ordered left to right.
The first two points in the spectrum are blocks world tasks. The third one is “random bin picking,” which gained attention when some researchers turned their attention to automated manufacturing.
The clutter spectrum above captures my view of clutter, as of about ten years ago. The blocks world was the extreme low for clutter, and random bin picking was the extreme high. Then I got involved in logistics and warehouse automation and learned a few things.
Logistics and warehouse automation
What is logistics? It is a big ride-sharing scheme. The goal is to get all the stuff from the farms and factories to the consumers — to us. But how?
Here is a stupid way. Suppose you want a stick of gum. You call the gum factory, and a worker tosses it into a car and drives it to your house. They would have to charge you hundreds of dollars for a stick of gum.
Here’s the smart way: ride sharing. No stick of gum travels alone. Each stick shares a box with other sticks, and each box shares a truck with other boxes. The cost of transport is amortized over all the stuff in the truck. To minimize cost per item, they maximize items per truck. They pack the trucks to the gills. All stuff, no air. In logistics, they love to say, “We don’t ship air.”
There are implications for clutter. Let’s look at that box of chewing gum. You have seen them near the grocery checkout stand, on display with boxes of candy and other stuff. The top of the box would be ripped off, so the customer can grab a pack of gum. The packs of gum fit the box perfectly. When the box is full, there is no room for fingers. This is what we in logistics refer to as “tiling.” Tiling occurs with the top layer, and perhaps again with the lower layers.
Being a human, the most dexterous creature in the universe, you can deal with a tiled bin. But for robots, a tiled bin can be a challenge. It deserves a spot on our clutter spectrum.
(I cannot let this pass without noting: tiling is not such a challenge for Berkshire Grey. And, full disclosure, I am the Chief Scientist at Berkshire Grey. We pick from tiled bins with ease. End of crass commercial note.)
There is another issue in bin picking. Some bins are homogeneous, and some are heterogeneous. In the figure above, the random bin is heterogeneous and the tiled bin is homogeneous. A heterogeneous bin can be easy, if you don’t care which item you pick. But picking an item at the bottom of a heterogeneous bin can be quite a challenge.
The crazy thing is that once alerted to the clutter issue, you will see it everywhere. If you watch carefully, you will even see that your own behavior is attuned to the clutter problem.
You are a link in the supply chain
The supply chain is a sequence of operations, moving stuff from the manufacturer to the consumer. Do you imagine yourself outside of that chain? You are not. You are doing a lot of the work. Let’s return to the grocery store. It looks quite a bit like a warehouse or a distribution center. Instead of a worker pushing a cart along the warehouse shelves, fulfilling orders, you are pushing that cart yourself. You pick items from a shelf, or from a box, and place them in the cart. Later you pick them from the cart and place them on a conveyor. Maybe a bagger packs bags or boxes for you, but then you pack them in your car, and transport them to your kitchen, and pick the items again and put them on your shelves. If we count all the operations in the supply chain, including all the picks, puts, and transports, you perform quite a few of them. You are not just one link, but several links, in the supply chain.
Efficient storage does matter in a retail store, just as it does in a warehouse. There is a bit of a compromise, though, because retailers want the customer to have a good view of the product. Still, clutter is an issue, and you have techniques for dealing with clutter, whether you have noticed them or not. In a high-clutter situation, such as taking a bottle out of a refrigerator display case, your grasp might proceed in two phases. First, you tip the bottle out just a bit, providing some clearance from the neighboring bottles. Second, you wrap your fingers firmly around the bottle to lift it out. The photo below captures the instant the firm grasp is obtained. (The whole video is available here.)
Here’s another example — pivoting a pack of gum up, lifting it away from the clutter to get a better grasp.
Here’s a similar example with a pizza box.
And another example with a salad container.
Perhaps these skills seem mundane to you, but that is because, I say again, you are one of the universe’s most skilled manipulators, or because your inner robot is managing all the complexity while you ponder other matters.
The home
Let’s return to the basic principle. Efficient storage or transport implies dense clutter. That applies to your home, not just to warehouses. Look at your drawers, cabinets, and bookshelves. Those storage areas might be densely packed — dense enough that clutter is an issue. Before you lift a hanger off a clothes rack, you might shove the neighboring hangers away. To get a book off a bookshelf, you might first tip it out, to get a firmer grasp.
Are your kitchen drawers homogeneous, or heterogeneous? Mine are mostly heterogeneous, and I often must do some digging. But my silverware is sorted into homogeneous compartments. I can grasp the desired number of forks, carry them to the table, and dispense them one at a time. Digging through clutter, deliberately grasping several items, dispensing items — these behaviors are mostly, but not entirely, beyond robots.
If we are coping with clutter all the time, in our own lives, why would we roboticists have a blind spot regarding clutter?
Let’s divide the handling process into three phases: taking out, using, and putting away. Or, to use the robotics lingo: pick, use, put.
Now, when roboticists think about the automation of cooking, for example, we tend to focus on the use phase. The pick and put phases might not even enter our minds.
Warehouses are different: there is no use phase. It is all picking and putting. We are forced to focus on picking and putting, and therefore on efficient storage and transport, and therefore on clutter. Warehouse automation has changed our perspective on manipulation, and that new perspective informs our understanding of manipulation more broadly.
One other thing we have learned from the warehouse. If anybody complains that your closet is too crowded, or your kitchen drawers are too cluttered, you can sneer at them and say, “I don’t store air.”
Notes and references
The front illustration is adapted from a figure taken from Terry Winograd’s PhD thesis of 1971. His thesis was focused on natural language, using a simulated blocks world as the domain of discourse. Here is the original figure.
Here is a funny thing about Winograd’s thesis. How could his robot avoid the clutter? In the figure, you can immediately notice an empty spot where the red block would fit. But how would a robot find that spot? Winograd found a simple way: randomly generate points on the table and then check if they would lead to a collision. The funny thing is that back in the 1970s we all thought that was a cheap trick, and not the right way to approach the problem in general. Twenty years later, Kavraki, Latombe, and others would revolutionize the field using an approach similar to Winograd’s.
The blocks world was also pursued at Stanford, SRI, Hitachi, and Edinburgh. The Edinburgh work dealt with some non-trivial clutter. Here is a photo of their robot on the left, and a typical scene on the right.
For early work on the random bin picking problem, motivated by factory automation, there was a notable project at the University of Rhode Island. Here’s a photo of the business end of their robot.
The pictures of picking out of clutter in the convenience store are taken from one of my favorite papers: Nakamura, Yuzuko C., et al. “The Complexities of Grasping in the Wild.” IEEE-RAS International Conference on Humanoid Robots. 2017. Here, again, is the link to the video.
Another brilliant post Matt! Picking from tiled bins is tricky even for us master manipulators: getting that first stick of gum out of the package usually requires some deformation, at least for me ;).
Also:
1) Another category of clutter is when parts are deformable:
https://www.newscientist.com/article/2404048-ai-cleaning-robot-can-tidy-up-clothes-in-a-messy-bedroom/
2) Squeezing the “air” out of pallets is NP-Hard, even in the plane:
https://www.thenewwarehouse.com/2025/08/04/613-reinventing-warehouse-stacking-with-ambi-robotics/
Hey, Ken. Thanks!
Yeah, I did not discuss deformables. Clothes: good example. Or, even a little deformation matters for cigars in a cigar box, or books on a bookshelf. Sometimes we work our fingers in there by compressing the surrounding clutter. And then I did not discuss the fact that containers are initially not open, so it’s clutter all around. Or, the hierarchy of containers — recursive clutter? So, there are some vague and unstated assumptions, and then when you try to go beyond those assumptions to the general case … Yikes!
Thank you for the great post Matt! Really a great read.
The insertion process of the fingers between the clutter and the target objects followed by a firm grasp highlights that the barrier between prehensile and non-prehensile manipulation is rather vague. This insertion process is critically assisted both by the compliance of the human hand as well as of the storage environment.
I’d like to help expand on the applications: we are tackling the problem of separating gangue from ore.
Hey Matt! I like the way your mind works. I had not thought about the complexities of clutter and automation. When you create an android that can clean up the clutter in a couple of rooms in my house, I’d be happy to beta test it for you. Oh yeah. My garage would be a good test bed, too. Thanks for you posts. I always find them interesting!
Sorry didn’t get this in the right place. Meant for Matt, not Weiwei.
Thanks Mike!
I thought people reading this blog might be interested in the fact that a video of the work at Edinburgh University that Matt references can be found here. Not only does the system cope with clutter but it also performs force guided assembly.
Hi Prof. Mason, this is a fascinating perspective on tiling! It reminds me that when ‘air’ is removed, standard pick-and-place often fails because there is no finger clearance.
In my current research on assembling chiral 3D tetris blocks (specifically reorienting an ‘L’ shape to a ‘J’ shape), I’m finding that standard grasping is impossible without a re-grasping station. I’ve been revisiting your work on Extrinsic Dexterity to solve this, possibly using the table itself to flip the object rather than a second arm.
Do you view ‘clutter’ as merely an obstacle, or as a source of constraints that we can exploit for these kinds of reorientations?
Good point. We call it “clutter” when it is an obstacle, but the same item can be a useful tool, a source of constraints as you put it. The table is a good example — you cannot pick up the L, flip the hand 180 degrees to get the J, and put it down. The table is an obstacle — clutter. But you can put a finger on the L, and roll it on the table to get the J.