Jul 18, 2023
Version 3
iPhone LiDAR tutorial V.3
- Gregor Luetzenburg1,2
- 1Geological Survey of Denmark and Greenland;
- 2University of Copenhagen
Protocol Citation: Gregor Luetzenburg 2023. iPhone LiDAR tutorial . protocols.io https://dx.doi.org/10.17504/protocols.io.yxmvm21w9g3p/v3Version created by Gregor Luetzenburg
Manuscript citation:
Luetzenburg, G., Kroon, A., Kjeldsen, K.K. et al. High-resolution topographic surveying and change detection with the iPhone LiDAR. Nat Protoc (2024). https://doi.org/10.1038/s41596-024-01024-9
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: July 18, 2023
Last Modified: July 18, 2023
Protocol Integer ID: 85162
Keywords: LiDAR, iPhone, Apple, Point cloud, M3C2, iphone lidar data, analyzing iphone lidar data, iphone lidar tutorial, data transfer from the iphone, lidar scanner, 3d scanner app, apple iphone, iphone, lidar sensor, icloud, medium scale with the lidar sensor, cloud, available cloudcompare software, models of the earth surface, cloud service, selfie stick, scanner, device, earth surface, scanning, apple app store, scanning area, dropbox, data transfer
Abstract
Recording, exporting and analyzing iPhone LiDAR data
The aim of this tutorial is to provide a guide on how to create models of the earth surface on a small to medium scale with the LiDAR sensor built in the Apple iPhones, edit and export those models to a computer, and measure differences between models.
This protocol requires an Apple iPhone with a built in LiDAR scanner and the free ‘3d Scanner App’ from the Apple App Store, as well as a computer with the freely available CloudCompare software installed. Optionally, a selfie stick can be used to increase the extent of the scanning area. For data transfer from the iPhone to a non-apple device (e.g. windows pc) login credentials to a cloud service (e.g. Dropbox, Google Drive, OneDrive, iCloud) are required.
Guidelines
CloudCompare is a 3D point cloud and triangular mesh processing software. It has been developed to compare two 3D point clouds or a point cloud and a triangular mesh.
LiDAR is a laser-based method to measure distances between a laser emitter and an object by measuring the time the light of the laser travels between the emitter to an object and back to a receiver.
M3C2 stands for multiscale model to model cloud comparison. The M3C2 plugin in CloudCompare computes distances directly between two point clouds.
A triangular mesh is created by drawing lines (edges) between points (vertices) in a 3D environment. In contrast to a point cloud, a mesh has a surface consisting of triangles (faces).
Normals are vectors perpendicular to a surface plane.
A point cloud is a set of points in space. Every point has an X, Y and Z coordinate. Point clouds are created with LiDAR measurements.
A raster is a two-dimensional grid of square pixels. The value of each cell represents a property such as elevation. A point cloud can be converted into a raster.
Texture adds information like color to points of a point cloud or surfaces of a triangular mesh.
Troubleshooting
Before start
The LiDAR sensor is built in Apple iPhones and iPads from the Pro line that were released from 2020 onwards. For simplicity, this protocol will focus on the iPhone, but the same workflow can be achieved with an eligible iPad.
Several apps with different functionalities exist to record LiDAR data with the iPhone. Here we use the freely available 3d Scanner App developed by Laan Labs (https://3dscannerapp.com/). All available apps, including the 3d Scanner App, are constantly being updated and the functionality as well as the layout might change.
CloudCompare is a freely available 3D point cloud and mesh processing software. The software is available for Windows, macOS and Linux and can be downloaded from https://www.danielgm.net/cc/
See the guidlines of this protocol for a short glossary of the most important terms used here.
Recording data
3d Scanner App basic functionality
To record data with the iPhone LiDAR scanner, unlock the iPhone and open the 3d Scanner App. Make sure nothing obstructs the LiDAR scanner at the back of the iPhone. By default, the app opens the ‘camera interface’ which is used for scanning 3D models. While scanning, the App takes photos with the iPhone’s main camera. Those photos are used to add texture to the 3D models. A scan will be initiated by pressing the red button in the apps' default interface. Follow the instructions on the screen and extend the scan by moving the iPhone around.
3d Scanner App
The ‘Scans’ interface, accessible at the lower left corner of the main camera interface, will open a list of previous projects. Scan modes can be changed in the lower right corner of the main camera interface. In this manual, we will introduce the ‘LiDAR’ (normal) scan mode and the ‘Point Cloud’ scan mode. The 3d Scanner App offers several scanning modes which can be accessed through the ‘Scan Mode’ interface. A help button in the scan mode interface offers a short overview over the different scanning modes.
3d Scanner App with the default camera interface in the background and the ‘Scan mode’ interface extended from the lower right corner. Pressing the red button will initiate a scan, the blue ‘Scans’ button will lead to a list of previous projects. The ‘Scan mode’ interface shows the different scan options. Here we focus on the default ‘LiDAR’ mode and the ‘Point Cloud’ mode.
First time settings
When opening the app for the first time after installation, access to the iPhone’s camera is requested which must be granted.
From the default ‘Camera interface’, navigate to the ‘Ccans’ interface and press the ‘Cettings’ gear symbol in the lower left corner. Adapt the settings according to the screenshot on the right. Only enable 4K LiDAR images when you are familiar with the app and after checking the phones available storage.
LiDAR scan mode
LiDAR (normal) is the default Scan mode in the 3d Scanner App. When opening the app and pressing the 'red button', a LiDAR scan is initiated. While scanning in the LiDAR mode, a mesh is created which is a triangulated surface model. The raw point cloud data cannot be accessed in the LiDAR mode.
In comparison to the point cloud mode, models in the LiDAR mode can have a larger spatial extend and surfaces are smoothed, resulting in a smaller number of outliers. There is no set limit on scan time, but very large scans may not process well, so it's best to take smaller scans and work up to larger one to see the optimal size.
While recording the LiDAR data, the iPhones’ main camera is taking pictures. Those pictures are used to add texture to the model after the recording. When moving the phone too fast during scanning, the photos become blurry and the texture will look fuzzy.
To finish a scan, press the 'red button' again. After completing a LiDAR scan, a pop-up window will open to process the scan. Select 'HD' and press 'Start'. Leave the app open during processing. When processing is finished, the 3D model will be textured.
Point cloud scan mode
Point cloud mode records the raw point cloud data instead of triangulating the points to a mesh. The spatial extend of scans in the point cloud scan mode is smaller compared to the LiDAR scan mode. Point cloud scans are automatically textured and post-processing options on the iPhone are limited.
Tips and Tricks
- It's best to move slowly and pan the device around slowly while taking a scan. Quick and jagged movements will affect the quality of the scan.
- Plan your scan trajectory before starting the scan and avoid re-scanning the same area during one scan.
- It can take a few passes to figure the best technique for the object or scene you are trying to capture. To start a scan, press the red button once and again to stop.
- When a scan is complete a texture will be automatically applied if the object is small, but for large objects they must be textured manually from the scan interface.
Limitations
- The range of the iPhone LiDAR sensor is limited to around 5 m. Depending on the reflecting surface, the range might be smaller.
- Model extent is limited and there is no indication when the maximum extent is reached. Not exceeding the maximum model extent, relies on the users’ experience.
- Surfaces with a high roughness (many small details) take considerably more time to scan and reduce scanning extent
- Reflecting and transparent surfaces such as water and glass lead to model distortions and should be avoided
- GPS data can be stored during scans, but the error is >1 m and GPS information might not be included in data export.
- Compared to more advanced LiDAR scanners, the iPhone does not distinguish between different return signals and points clouds cannot be filtered for example for vegetation.
Viewing and editing models in the app
To view recorded 3D models, navigate from the default 'Camera' interface to the ‘Scans’ interface and select a previous recorded model. There are different options, depending on the recorded scan mode. The ‘Share’ option is discussed in the next section.
LiDAR scan mode
- ‘More’ offers a rich set of editing capabilities. Models can be crop and transformed, triangulated meshes can be simplified, noise can be reduced, and the mesh resolution can be refined. Most of those options can also be performed on a computer after exporting the model.
- ‘Process’ offers the possibility to texture the triangulated mesh with different resolutions. Texture should be added to a model before export.
- ‘Measure’ enables distance and volume measurements of a model.
- ‘Share’ offers different export options.
Point cloud scan mode
- ‘More’ offers extra information about the recorded data such as the camera trajectory.
- ‘Share’ offers different export options.
Exporting data
Select a LiDAR model from the 'Scans' interface and press ‘Share’ to export the recorded data. Several options are available to export the LiDAR models from the iPhone to another device. Here we will focus on two different data formats: OBJ and LAS. Toggle 'Zip Files' on before export.
Export LiDAR scans (OBJ)
To export a triangulated mesh created in the LiDAR scan mode, select 'OBJ'. Scanning in the LiDAR Scan mode is beneficial for larger scans, but for detecting small scale changes in a 3D environment, point cloud distance calculations are advantageous over difference calculations between triangulated surfaces.
Instead of exporting the LiDAR scan as a triangulated mesh you can also project a point cloud on the mesh surface. For exporting a mesh as a point cloud select point cloud and follow the steps in the next chapter.
The export of point clouds is preferred over mesh export here, but note, a point cloud that was recorded in the LiDAR scan mode differs from a point cloud that is projected on a mesh that was recorded in the Point Cloud Scan mode.
Share interface with different export format options. Enable 'Zip Files' and press 'OBJ' or 'Point Cloud'.
Export point clouds (LAS)
To export a point cloud, select 'High Density', toggle 'Zip Files' and 'Z axis up' on and press 'LAS'.
Models recorded in the LiDAR Scan mode can be exported as point clouds as well by selecting Point Cloud in the Share menu.
Selecting 'LAS Geo-Referenced' includes the global coordinates of the scan in the LAS file. The coordinates are stored as decimal degrees in lat and long. CloudCompare cannot read decimal degrees. Therefore, the coordinates must be changed to meters (e.g. UTM) before importing Geo-Referenced LAS files into CloudCompare.
Point Cloud Export interface. Select 'High Density', enable 'Zip Files' and 'Z axis up'. Press 'LAS' or 'LAS Geo-Referenced'.
Data transfer
Data is transferred from the iPhone to another Apple product (e.g. MacBook) via AirDrop or to any other device through a cloud service (e.g. Dropbox, Google Drive, OneDrive, iCloud). If you want to transfer data with a cable via iTunes, please visit the 3d Scanner App FAQ (https://docs.3dscannerapp.com/).
For AirDrop press 'AirDrop' after selecting the preferred export format (OBJ or LAS) and select the device you want the data to be transferred to. A notification will pop up on the pinged device and data transfer starts after approval.
For data transfer through a cloud service press 'Save to Files' after selecting the preferred export format (OBJ or LAS). Select the location where you want to save your data and press 'Save'. Data transfer through a cloud service requires the availability of login credentials for a cloud service provider and the installation of the respective iPhone app through the Apple App Store on the iPhone.
If the preferred cloud service cannot be selected in the Files app, navigate to the Apple App Store, and download the respective app from the preferred cloud service (Dropbox, Google Drive, OneDrive) or go to the Settings app on the iPhone and sign-in to the iCloud. Login with your user credentials to the preferred cloud service in the downloaded app. Afterwards, open the Files app on the iPhone and navigate to the 'Browse' interface (lower right corner).
If the preferred cloud service is not listed, press the three dots in the upper right corner and select edit. Activate the preferred cloud service and press done. To check if you successfully linked the cloud service with the Files app, select the cloud service in the Browse menu of the Files app and check if you can access your data. Now you can go back to the 3d Scanner App, select a scan, press share, select the export format, press 'Save to Files' and navigate to the preferred cloud service in the Files app. Press 'Save' when you have located the destination folder.
Analyzing data
The following chapter requires the desktop application CloudCompare. Note that CloudCompare does not save specific projects. After closing and restarting the application, all desired files must to be reloaded in CloudCompare manually. It is the user’s own responsibility to save data products frequently and separately. It is recommendable to save data products either in the original file format (e.g. OBJ or LAS) or in the CloudCompare .bin format.
Understanding the differences between a point cloud, a triangulated mesh and a rasterized DEM is beneficial. If the glossary is too short, use an internet-based search engine or an AI chatbot for more detailed explanations.
An extensive documentation about CloudCompare is available under the following link:
Loading data in CloudCompare
Locate the LiDAR data you transferred from the iPhone to your computer and unzip it.
Both triangulated meshes in the OBJ format and point clouds in LAS format can be dragged and dropped in CloudCompare. For OBJ import, select the '.obj' file from the unzipped folder and make sure .jpg and .mtl files are in the same folder. For LAS import, select the '.las' file.
A window will pop up in CloudCompare after dropping the file, press 'apply all'. After loading the file, the 3D model will appear in the central viewing pane. When loading several files, they can be selected individually in the DB tree to the left.
DB tree with point cloud and triangulated mesh in CloudCompare
Sample points in mesh (optional)
This chapter is only relevant in case you exported a 3D model as a triangulated mesh in OBJ and you want to work with point clouds in CloudCompare. Exporting a mesh as point cloud from the 3d Scanner App or sampling a point cloud on an exported mesh in CloudCompare results in the same product: a point cloud.
Select 'Edit'->'Mesh'->‘Sample points’ in CloudCompare, choose the desired number of points or point density and press ‘OK’. A point cloud will be projected on top of your mesh and the file will appear in the DB tree on the left. Save the new file.
Point sampling on mesh tool in CloudCompare
Subsample a point cloud (optional)
Subsample a point cloud to reduce the point density or distribute points more evenly throughout your point cloud. Select 'Edit'->‘Subsample’ and method ‘space’. Choose a min space between points (e.g. 0.01) and press ‘OK’. Save the new file.
Cloud subsampling tool in CloudCompare
Segment point cloud (optional)
This Segment tool is useful to discard areas of the 3D model that should not be part of the final analysis. It can be found at the 'Edit' -> 'Segment' menu and enables users to perform interactive segmentation of chosen entities by defining a 2D polygon or rectangle on the screen. The process can be repeated multiple times, with the entities' orientation adjusted each time, to accurately segment the entities in 3D.
Clean point cloud (optional)
The Statistical Outlier Removal (SOR) tool removes outliers from a point cloud and can be found at 'Tools' -> 'Clean' -> 'SOR filter'. The user is required to select the number of neighbors to be utilized in calculating the "distance to neighbors" for each point. Additionally, the user needs to specify the multiplier for the number of standard deviations (standard settings usually work well).
Align point clouds
3D LiDAR models from the iPhone are scaled and orientated to the real world, but the geographical location is not recorded precisely. Therefore, when measuring changes between two models of the same object or area, the two models must be aligned (registered) to each other. The first step in the model registration is the rough alignment. The Align tool lets the user align two models by picking at least three equivalent point pairs in both models.
When working with LAS Geo-Referenced files, the point clouds are already roughly aligned to each other and the point pair picking can be skipped.
Select two point clouds in the DB tree to the left and go to 'Tools'->'Registration'->'Align (point pairs picking)'. The first step is the role assignment. One of the models is the reference (won’t move) model and the other model will be aligned to the reference model. After designating the roles, a new dialog will appear in the upper right corner and the user can pick point pairs in the ‘to align’ and the ‘reference’ cloud.
Pick the same set of at least three remarkable points in both models. During this process, you can hide either the 'reference' or the 'aligned' cloud. You can also freely rotate and translate the viewpoint, enlarge points, and zoom in or out. Each time you pick a point, a new entry appears in the corresponding table and a marker in the 3D view.
As soon as you have selected at least three or more pairs, CloudCompare will display the resulting RMS and you can preview the result with the 'align' button. You can add new points to both sets anytime to get a more reliable result.
Validate the alignment with the green V. A report will pop up. Write down the final RMS and press ‘OK’.
Align tool in CloudCompare
Fine registration
Following the alignment, the fine registration will enhance the model registration. CloudCompare uses an iterative closest point (ICP) algorithm to minimize the differences between two subsets of two point clouds.
Select two models and go to Tools'->'Registration'->'Fine registration (ICP)'. A new window will open, and the first task is role assignment again. Assign the ‘to be aligned’ role to the model that you want to align and the ‘reference’ role to the reference model.
It is very important to determine a final overlap (in %) based on the overlap between the two models. The higher the final overlap the higher the RMS. Try to estimate the real world overlap between the two models as good as possible.
The ICP is performed on a random subset of points. To increase the number of points of the subset, navigate to ‘Research’ parameters, and increase the random sampling limit (default is 50,000). This will increase the processing time but also decrease the final RMS between the two models.
Press ‘OK’ and wait until a window with the registration info pops up. Write down the RMS and press ‘OK’ again.
Fien registration tool in CloudCompare
M3C2 distance calculation
With the M3C2 plugin in CloudCompare distances between two point clouds can be computed (Lague et al. 2013). Select two point clouds, go to ‘Plugins’->‘M3C2 Distance’. Use the ‘reference’ cloud as cloud #1 and the ‘to align’ cloud as cloud #2.
Under ‘Main parameters’ and ‘Scales’ set the normal and projection diameter. You can use the ‘Guess params’ button in the lower right corner of the M3C2 distance window to guess the normal and projection diameter. For iPhone LiDAR point clouds, the chosen value is usually somewhere around 0.5.
Under ‘Core points’ select ‘use cloud #1’. Activate the ‘Registration error’ and enter the RMS from the fine registration.
For more information see: https://www.cloudcompare.org/doc/wiki/index.php/M3C2_(plugin)
M3C2 distance tool in CloudCompare
To visualize the distances between the two point clouds, the display paramteres of the scalar fields need to be adapted after the M3C2 calculation. Select the M3C2 point cloud and scroll down to 'Color Scale' and 'SF display params'. Tick off the 'visible' box to make the color scale visible and adapt the 'displayed range' (displayed) to an average expected change in m. Set the 'absolute saturation' (abs. sat.) to approximatley half of the displayed range. Go to 'Paramteres' and deactivate 'show Nan/out of range values in grey'.
The displayed point cloud is the final product of this tutorial. Green points represent areas of little change and blue and red points represent areas of higher, positive and negative change over time. Color patterns in the M3C2 point cloud can reveal geomorphological processes in a landscape on a small to medium spatial scale.
M3C2 point cloud properties with adapted visualisation settings in CloudCompare
Convert a cloud to a raster
If you want to compare an iPhone LiDAR 3D model with another 3D representation of a landform or object, you might want to transform your point cloud to a raster. In CloudCompare you can rasterize a point cloud to create a DEM or hillshade mode with the 'Rasterize' tool. Select a point cloud, go to ‘Tools’->‘Projection’->‘Rasterize’.
The ‘step’ determines the spatial resolution of the raster. A step of 1 will result in a raster with 1x1 m cells. Select the 'Height grid values' as the active layer, 'Z' as the projection direction and the ‘average height’ as the cell heightRasterize tool. Press 'Raster' to save the elevation raster.
Rasterize tool in CloudCompare
Protocol references
CloudCompare. (2023). (Version 2.13) GPL software. Retrieved from http://www.cloudcompare.org/.
Lague, D., Brodu, N., & Leroux, J. (2013, Aug). Accurate 3D comparison of complex topography with terrestrial laser scanner: Application to the Rangitikei canyon (N-Z). ISPRS Journal of Photogrammetry and Remote Sensing, 82, 10-26. https://doi.org/10.1016/j.isprsjprs.2013.04.009
Luetzenburg, G., Kroon, A., & Bjork, A. A. (2021, Nov 15). Evaluation of the Apple iPhone 12 Pro LiDAR for an Application in Geosciences. Sci Rep, 11(1), 22221. https://doi.org/10.1038/s41598-021-01763-9