How CNET Tests Robot Vacuums

Since CNET started testing robot vacuums in the early days of the first Roomba, we’ve worked hard to craft comprehensive testing procedures to evaluate every robot vacuum for the things that matter for buyers. Our lab experts have tested over 100 robot vacuums and counting, evaluating them for cleaning performance, navigation ability, object avoidance, noise levels and more. Price and special features also play a role in our overall rating and buying advice. 

Below, we’ve laid out our comprehensive set of lab tests that we run on each and every robot vacuum that gets sent to the lab. Our tests are also always being tweaked and refined to generate the most consistent and reliable results, but the current testing procedures are accurate as of this writing.

Awarding the highest performers

We weigh the results of our rigorous lab testing for cleaning performance, navigation and obstacle avoidance with our decades of editorial expertise reviewing robot vacuums. Then we evaluate the total consumer quality by factoring retail price and extra features to assess the overall value of each robot vacuum.

There are a number of awards we may bestow on top performers. Editors’ Choice represents the best of the best products we’ve tested. These robot vacuums represent our top-tier choices for most people. We may offer an Editors’ Choice for a premium model and a different award for the best budget model.

We may also designate a Lab Award, which acknowledges a single product that had the highest performance in a specific and repeatable metric or benchmark that’s meaningful to consumers. For example, the Dreame X50 won the Lab Award for best cleaning coverage. Finally, we will spotlight top-performing products in our best lists. 

Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.

Our test room in Louisville lab

The first trial is to figure out how well a robot covers the floor while cleaning and how well it works to avoid obstacles. We built an industry-standard testing room, as specified by the International Electrotechnical Commission, just for this purpose. The IEC is an international standards body responsible for managing robot vacuum testing procedures, among other things, for vacuum manufacturers. 

Inside this room are objects designed to simulate typical obstacles a robot vac encounters for navigation as it cleans. These obstacles include wall edges, table and chair legs, couches, loose cords, simulated animal poop, other furniture and so on, plus bare tile and hardwood floors, as well as carpet. 

Each robot vacuum we consider for recommendation is put through its paces in our test lab in Louisville, Kentucky. In addition to test floors where we run our controlled pickup tests, we monitor each robot vacuum in a special test room filled with mock furniture to gauge how well it navigates common obstacles. Beyond that, we evaluate each robot vacuum’s ability to collect pet hair without getting clogged or leaving loose strands behind. We also consider its mopping capabilities and how well it navigates (fake) pet messes.

We’ll dive into more of each of the sections below.

Robot vacuum pickup/cleaning power

When it comes to vacuuming prowess, we want to know how effective each robot is at cleaning common crumbs and other debris, and also how it fares against much smaller particles like dust, dirt and sand on all flooring types. 

Sand and pet hair

To find out, we use play sand as an analog for finer particles while pet hair we get from a groomer. In each case, we scatter a controlled amount across three test floors patches (the taped boxes you can see in the picture above).

For pet hair, we don’t weigh the pet hair after the test, but we take before and after pictures of our test floor patches for subjective visual evaluation. The robot vacuums that remove the most pet hair from the carpeting and hard flooring are the ones we recommend for pet owners.

Floor types we test on

To test each robot vacuums ability on different floor types, we place sand and pet hair on three different floor types:

1. Low-pile carpet: Typically shorter fibers and less plush than mid- or high-pile carpet. Robot vacuums have an easier time picking up from it (though not always). Low-pile and midpile carpet tend to be more challenging and our scores can vary from a high of 60% or more (excellent) to as low as 10%. 
2. Midpile carpet: Softer and more plush with taller fibers. It tends to be more challenging for robot vacuums (though again, not always). To ensure the sand is properly distributed, we use a brush to spread the sand out evenly. Robot vacuums especially start to differentiate themselves in midpile carpet performance, with our current best-scoring robot vacuum, the Mova Z60 Ultra Roller Complete, getting 47.54%. 
3. Hardwood floors: We use hardwood floors that can replicate any flat surface, such as tile or vinyl flooring. Generally, most robot vacuums score well on the hard flooring, with many of our top picks scoring above 80%.

How we measure pickup performance success

Before each test, we take the robot vacuum, thoroughly empty its dust bin, send it to clean the affected area and finally measure the weight of whatever it managed to pick up. That gives us a pickup percentage of the full amount. 

From there, we repeat each four more times for a total of five tests and average the results. If there’s an obvious outlier, such as a test with an unusually high or low pickup percentage, we rerun the test cycles. We then calculate the individual average pickup for each flooring type and get an overall average.

This gives us a frame of reference for how effective a robot vacuum is at cleaning. 

Our overall average pickup scores run the gamut, with the bottom performing robot vacuums scoring as low as 20%, to the best ones scoring above 60%.

Robot vacuum navigation skills: Heat mapping

Your robot vacuum will only clean your home as thoroughly as it’s capable of navigating it. The ideal cleaner will make easy work of finding its way from room to room and automatically avoiding obstacles along the way, all of which makes for proper, low-maintenance automated cleaning.

To test navigation efficiency, we record the robot vacuum in our test room with an overhead camera and use video analysis software to assign the robot vacuum an average coverage percentage based on the total portion of the accessible room it’s observed to visit during three standard test runs. The heat map that’s generated is then analyzed to provide the result. The test room includes dummy furniture, low-hanging furniture, different flooring types (tile, hardwood and carpet), lamps, cables and other obstacles.

We also have boxes with differently sized holes cut out of them, between 3 and 4 inches wide, to simulate low-lying furniture. This helps us see if the robot vacuum is capable of getting under this furniture to clean. The ones that are able to achieve it are robot vacuums with thinner profiles or the ability to retract their sensor.

Heat map color legend:

• Blue/cyan: Very few passes — light coverage
• Green: Occasional passes — low to moderate coverage
• Yellow/orange: Frequent passes — high coverage

Our two best performers can be seen below, the Mova Z60 and Dreame X50 Ultra. Both offer a high percentage of cleaning coverage with very few missed spots. There’s some variation in how frequently they pass over certain parts of the room. The X50 Ultra focused heavily on the center by the simulated table arrangement, while the Mova Z60 distributed its attention a bit more evenly. Notably, both were able to get under most pieces of furniture.

In contrast, our worst-scoring robot vacuum in this category was the Eufy E28, with numerous missed spots — especially around edges and corners — and light coverage in the outer parts of the room. It also missed both pieces of low clearance furniture. For you, that means this robot vacuum will provide less comprehensive cleaning coverage.

Object avoidance

We test object avoidance at the very end of our procedure by using six distinct common items placed in the environment to see if the robot vacuum is able to recognize and avoid them on its cleaning route. These items include 360-degree pet waste, 180-degree pet waste, 90-degree pet waste, a lamp, pet toy and sock.

For the pet waste classifications, we distinguish between the fake pet waste based on the angle of the barriers around it that limit the possible approaches from the vacuum.

That means:

• The 90-degree trial is in the corner of the room, usually preventing the vacuum from accessing that portion of the room if it successfully avoids the obstacle.
• The 180-degree trial is along a wall, allowing more options to clean around it.
• The 360-degree trial is in the middle of the room and should be easy for the vacuum to clean around.

The more objects avoided, the better we consider the robot vacuum’s obstacle navigation. Currently, no robot vacuum has successfully avoided all six of our obstacles, with the best-performing ones avoiding five and the worst avoiding none of them. We also distinguish different failure states, noting that several robot vacuums suffer from a critical failure in obstacle avoidance, by sucking in the sock, as with the Roomba DustCompactor Combo 205, or running right over the pet waste, as with the Narwal Flow.

Noise level

We measure the noise level of each robot vacuum using a sound level meter to get LAeq, which represents the average noise over time, with an adjustment for human hearing. The data above reflects average noise levels in decibels, recorded throughout the entire test while the unit is operating. A lower score indicates a quieter vacuum, which is generally preferable. However, noise doesn’t weigh heavily in our scoring at this time, as all the robot vacuums fall within a similar range to the human ear. As we continue our testing of more vacuums, we’ll reevaluate our noise level testing.

Price, features and other factors

Price and features don’t require objective testing, but we do consider them in our overall evaluation and rating. Robot vacuum models can cost anywhere from $200 to nearly $2,000. We try to evaluate a robot vacuum according to its price range by comparing it to other models that cost about the same. That way, we can provide recommendations for a budget model and a high-end model with all the bells and whistles, as well as everything in between.

Features like mopping, self-emptying base stations and the usability of the app and software are all other things we consider. Several of these features, like base stations and mopping, are becoming more common across all prices. We’re currently in the process of crafting a test to evaluate, and we’re planning on implementing this test in the near future.