Explaining the metrics that laser-scanning experts use to determine the quality of a mobile point cloud, and the process they use to audit data sets.
You’ve heard about the benefits of mobile mapping systems for building documentation, and you know what they can do for your business, so you decided to purchase one for yourself. You asked your colleagues for advice, did the reading, and looked at all the specs for all the tools on the market. After a lot of careful thought, you narrowed the list down to a few scanners that seem right for you and your business.
Now you’ve run into a problem: It’s time to download sample point clouds for the short list of candidates and check out the data quality. But you don’t have a lot of experience with mobile scanning, so you’re not sure how to proceed. How do you evaluate whether a mobile scanner’s point cloud is good enough for your needs?
This guide will help. We’ll explain the metrics that laser-scanning experts use to determine the quality of a mobile point cloud, and then show you the simple process that they use to audit a data set – all without the need for complex analysis tools.
What metrics do pros use to judge a mobile point cloud?
First, we’ll explore the qualities that you should look for in a mobile point cloud and explain why these qualities are important.
Coloring/texture
What is it? How realistic the coloring and texturing of the point cloud looks. Put another way: How much the point cloud looks like a photo from a high-quality digital camera.
Why is it important? Good coloring and texture help you identify small details in the data set, for instance trees, doors, and even the text on signs. This has tremendous value during the modeling process, when the ability to interpret the scan and discern objects is crucial.
By making the point cloud look like the real-world environment you captured, good coloring and texture also make it easier for experts and non-experts to understand what they are looking at, and navigate the data set.
Noise
What is it? The “fuzziness” of the point cloud. Noisy points show up in every raw data set, regardless of the scanner, because the physical behavior of the laser sensor creates random imperfections in the data. Every manufacturer has their own approach for removing this noise—some of which are more successful than others.
Why is it important? Too much noise can cause problems in modeling applications. For instance, if you are modeling a floor, a “line” that is 3 cm thick due to noise can make it difficult to discern where to start a line. The same problem comes up when you try to model fine details.
Excess noise also makes a point cloud larger than it needs to be to represent the geometry accurately, which causes bigger files and longer transfers. It can also mean slower handling on your computer and might even require you to shell out a lot of money to upgrade your machine.
Resolution of fine details
What is it? How crisp small details are, for instance 90° edges, or the shape of a door frame.
Why is it important? A lot of modeling applications require you to work at high levels of detail. If you are modeling a floorplan with a scale of 1:50, or doing visual inspection, then it’s likely you’ll need to see a lot of fine details to produce a good quality deliverable.
Absolute and relative accuracy
What is it? Though this is a very complex topic, we can give you a relatively simple working answer. Accuracy numbers represent how close a measurement is to the correct value.
In mobile scanning, we measure two kinds of accuracy. Absolute accuracy refers to the accuracy of your data on a global scale. This metric applies to a measurement taken across multiple scan locations, for instance from one corner of a building to another.
Relative accuracy, refers to the accuracy of the data set on a local scale. This metric applies to measurements between points in a single scan location, like a room.
[Check our glossary of terms for more precise definitions]
Why is it important? These metrics are important in determining the reliability of a scanner, because they can show how accurate the dimensions of a building are, and how accuracy the dimensions of room are – both of which are important considerations for applications like scan-to-BIM or generating floor plans.
[For more information on what metrics matter most for applications BIM modeling, floor-plan generation, and more, see our guide to modeling from mobile mapping data.]
5 steps for auditing point cloud quality
You’re probably aware that many experienced laser scanning professionals judge the quality of a point cloud using a benchmark data set and complex software. What you might not know is that many of them also use a simple step-by-step process to audit a point cloud by eye.
By looking for specific indicators, they can quickly judge how well any point cloud scores on the metrics we covered in the previous section. Here’s how Stefan Romberg, NavVis’ head of mapping and perception, does it.
If you want to follow along, you can find NavVis’ sample point cloud data sets here.
Step 1: Note your first visual impressions
Download the sample data provided by the device manufacturer and open it in your point cloud software. Explore the data and note your first impressions.
- How colorful is it? Does it look like a high-quality photograph, or is it dim?
- Does it seem crisp, or fuzzy?
A quick visual check sounds almost too subjective to be useful, but Romberg says it is a very important step. “Most of the time,” he explains, “the more beautiful a point cloud is, the better it is. Subjective beauty is not a proof of quality – but it is an indicator of hard, objective qualities that can be measured.”
This is because a beautiful point cloud requires realistic coloring and texture and noise reduction, and a point cloud must hit a baseline accuracy to prevent problems like warping and distortion.
Step 2: View the point cloud in an ortho view, and take a horizontal slice
Look at the point cloud from the top enables you to check for two more important indicators: straight lines and consistency.
- Are the walls straight where they’re supposed to be straight?
- Are the walls or beams the same thickness from one end to the other?
- Do you see warping in long hallways?
This will give you a better idea for the accuracy of the mobile mapping device. As Romberg explains, if the walls are warped rather than straight, if they get thicker or thinner from one end to the other, or if they split off into a false double, then your data is wrong. This might have happened because the SLAM could not cope with the environment.
[To learn why some mobile solutions can produce problems like this, check out our explainer for SLAM, the technology that powers mobile mapping devices.]
Step 3: Zoom in to check fine details
Pull up a small section of the point cloud, for instance the area around a door. This will allow you to look for familiar details and see how well the point cloud resolves them.
- How accurately does the data represent corners, or other 90° angles?
- Are the walls actually curving (not straight) when you look at them closer?
- How crisp are details like door edges, which have a contour that can be difficult to capture?
This will help you double-check for accuracy. It also helps to determine how well the point cloud scores on resolution of fine details.
Step 4: Check the white papers
This is a big one. The steps we just described will give you a very good sense for its accuracy – and they’ll only take you a few minutes. But they aren’t a guarantee of accuracy. For that, you’ll need hard numbers, which means the simplest option is to put aside some time to review the manufacturer’s white papers.
You might say, why do this if I’ve already looked at the spec sheet? A spec sheet for a mobile mapper has limited use. It indicates the accuracy of the mobile scanner in ideal, controlled circumstances. But the real world is not ideal. The accuracy of a mobile scanner depends on a number of factors, like how well its on-board processing handles its environment, and how effective its correction functions are.
That’s why you should review the manufacturer’s accuracy white papers, which demonstrate the accuracy of the scanner in a variety of applications. These papers will show you how the scanner performs in the kinds of environments that matter to you.
Step 5: Demo and repeat
Now you’re ready to call the manufacturer and set up a demo. This will enable you to try the device yourself, feel how it operates, and most importantly, see its data quality in an environment you’re familiar with.
Once you have the demo point cloud in hand, repeat steps 1 – 3 to audit the point cloud in the same way as you did the sample point cloud. Bonus: Now you can be sure the manufacturer hasn’t cherry-picked their sample data to make their point clouds look better than they really are.
If you feel like moving on to a more complex form of evaluation, you can capture the same environment with a terrestrial laser scanner and use cloud-to-cloud analysis to see how the mobile data stacks up.
Wrapping up
Anyone can verify the quality of a point cloud, with or without complex analysis tools. By performing a careful visual audit, you can get a very good idea for how it scores on a variety of important metrics. Finish up by reading the accuracy white papers – and maybe going for a demo and comparing to TLS data – and you should know whether the scanner’s data is right for you. You’ll be in a great position to make an informed choice for your business.
Sean Higgins is an independent technology writer, former trade publication editor, and outdoors enthusiast. He believes that clear, buzzword-free writing about 3D technologies is a public service.