Verifying the accuracy specifications of NavVis VLX 3
In this document, NavVis demonstrates a rigorous, reliable, and survey-based method for verifying the accuracy of point clouds produced by a dynamic laser scanning system.
The challenge of estimating and verifying accuracy for a dynamic laser scanning system
For surveyors or other professionals who rely on high-accuracy instruments, it is crucial to estimate and verify the accuracy of the instrument’s real-world measurements.
With a dynamic laser scanning system, this task is challenging. Estimating the accuracy of a device in a specific environment requires significant experience, and verifying the result is much more complex than with other devices.
This section will demonstrate the extent of this challenge.
To do so, we will explain how to estimate and verify the measurement accuracy of three instruments, in order of increasing difficulty: total station, static terrestrial laser scanner (TLS), and dynamic laser scanning system.
1. Total Station
Estimating the accuracy of the total station’s measurements is trivial because we can consult the datasheet to find the accuracy of the hardware components. Using this information, we calculate the expected accuracy of the results.
Verifying the accuracy is equally simple: we measure a point in the environment with the total station, then measure that same point with a reference measurement tool. Now we compare the two measurements to make an accuracy statement for the total station’s results.
2. Terrestrial Laser Scanner (TLS)
Estimating the accuracy of a single point measurement is simple. Again, we can find the accuracy of the components and use those numbers to estimate the accuracy of a single point in the point cloud.
Verifying the accuracy of this point measurement, however, is more complex. The problem is that a TLS does not measure repeatable points. It measures points spread randomly around the scan area. This makes it virtually impossible to locate the same point in both the point cloud and the real world, and then compare them.
We solve this by placing targets in the scan area (for example, checkerboard markers or spheres).
This is more labor intensive, but it enables us to extract the target’s coordinates from the point cloud, measure the point with a reference instrument, and compare the two measurements. Now we can make an accuracy statement about the results of the TLS.
3. Dynamic Laser Scanning System
As with the total station and the TLS, the hardware components affect the accuracy of the final point cloud. However, there are many other factors at play, so in this case, estimating the accuracy of a single point measurement is challenging.
The biggest is the sophisticated algorithms that enable the operator to capture as they walk. Known as simultaneous localization and mapping (SLAM), these algorithms have a significant effect on the accuracy of the final point cloud. And the extent of this effect is unknown unless you have access to the original algorithms themselves.
Estimating the accuracy of a point measurement would require us to perform calculations that account for the effects of the SLAM — a prohibitively challenging task.
Not only is estimating the accuracy of the point measurement difficult, but verifying it is as well. Again, this challenge is due to the SLAM processing.
To explain: SLAM works by tracking the scanner's trajectory through the scan. When the scanning workflow is complete, it passes information to the processing software, which generates a final point cloud.
This software runs the data through further algorithms to produce the best-possible point cloud. During this step the link between the trajectory and the single point is broken.
As a result, we can use the scanner’s trajectory to find its location at any time during the scan. But we cannot find the point measurements that the scanner captured from that location.
This makes it impossible to perform the same verification workflow as we would use for a TLS. If we do not know the scanner's location when a target was captured — also known as the measurement’s origin — we cannot calculate the angular and distance accuracy numbers for that point measurement. And that prevents us from making a final statement about an instrument’s accuracy.
RGB colorized point cloud of testing scenario with checkerboard targets
Figure 2: sticky targets from a common laser scanning equipment provider
Figure 3: target on DIBOND® plate mounted on the wall
Figure 5: RGB colored B&W target |
Figure 6: intensity multiHue colored B&W target |
Figure 7: intensity grayscale colored B&W target |
Eager to learn more about how to integrate dynamic laser scanning into your existing reality capture workflows?
Figure 8: top view - lines
Figure 9: 3D view - lines
Figure 11: API tracking system radian plus 50
Figure 12: vProbe for API tracking system
Figure 13: target extraction failed for P34
Figure 14: re-fitted target P34
Figure 15: 3D-deviation per NavVis VLX 3
The graph has three peaks, corresponding to targets P08, P28, and P38.
Examining the P08 target more closely, we found an obvious misplacement of the center in the rgb colorized point cloud. Due to insufficient intensity contrast, the algorithm was not able to find a proper target center. See Figure 16: Target P08.
Target P28 was placed in a location where the scanners had a line of sight to the target for only a short time. Therefore, the point density and intensity in that area made target identification challenging. See Figure 17: Target P28.
Target P38 has a slightly asymmetric intensity distribution in the point cloud. A visual inspection verifies that the target’s center was incorrect. See Figure 18: Target P38.
 
Figure 16: target P08 |
Figure 17: target P28 |
Figure 18: target P38 |
When it became clear that these three outliers were incorrect measurements, we removed them from further calculations. Though some extracted targets did not show ideal intensity information, from a visual point of view they are still correct, so we kept them for further investigation.
Results for relative approach (no GCPs)
The following histogram shows how far each measured point-to-point distance deviates from ground truth. Error distribution is organized in 0.5 mm “buckets.” The yellow indicates the percentile line.
To illustrate how to read the results: The red lines show that at 68% the error is 3.1 mm, which means that 68% of the point cloud distance measurements deviate from ground truth by 3.1 mm or less.
Figure 19: relative deviation histogram
The green lines indicate the 5 mm accuracy as stated in the NavVis VLX 3 specification sheet. Looking at the line shows that 87% of the distance measurements extracted from the point cloud deviate from ground truth by 5 mm or less.
Results for absolute approach (GCPs)
As previously stated, for the absolute approach we processed all point cloud data using NavVis VLX 3 ground control point functionality. It’s important to note that we did not consider the accuracy and precision with which the operators manually tapped the GCPs.
As mentioned in the chapter How to verify the accuracy of a dynamic laser scanning point cloud, the absolute approach could be used to benchmark accuracy in a number of different ways.
1. Point to point comparison: absolute points vs. reference points
The graph below illustrates how far the target centers in the point clouds deviate from ground truth. All outliers have been removed.
For the 478 observations (each corresponding to a single point), the average deviation is 2.8 mm and the median is 2.5 mm. 
Figure 20: 3D-deviation per NavVis VLX 3 - outliers reduced NavVis VLX 3
2. Distance comparison absolute data vs. reference data
The following histogram shows how far the point-to-point distance measurements in the point clouds deviate from ground truth.
The error distribution is in 0.5 mm “buckets.” The yellow line indicates the percentile line. To illustrate how to read the results: At 68% (as mentioned before) the error is 2.1 mm. This means that 68% of the distance measurements in the point clouds deviate from ground truth by 2.1 mm or less.
Figure 21: absolute deviation histogram
The green line indicates the 5 mm accuracy number as stated in the NavVis VLX 3 specification sheet. Looking at the line shows that 97% of the distance measurements extracted from the point clouds deviate from ground truth by 5 mm or less.