Jul 17, 2020
Image capture and pre-filtering for 3D photogrammetry of coral colonies
- 1University of Southern California
- Coral Bleaching RCN protocols
Protocol Citation: Wyatt C Million, Carly Kenkel 2020. Image capture and pre-filtering for 3D photogrammetry of coral colonies. protocols.io https://dx.doi.org/10.17504/protocols.io.bgdcjs2w
License: This is an open access protocol distributed under the terms of the Creative Commons Attribution License,  which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Protocol status: Working
We use this protocol and it's working
Created: May 14, 2020
Last Modified: July 17, 2020
Protocol Integer ID: 36996
Keywords: 3D, photogrammetry, coral, 3D models,
Abstract
This is a protocol for generating images to be used in 3D model building via Agisoft Metashape for coral photogrametry. This will cover underwater, field-based methods and tips to collect photographs and pre-processing of photos to improve model building.
Image capture is the most important part of 3D photogrammetry because the photos taken at this point will be all that you'll have to build models and collect data. As such, you want to ensure you have enough photos to work with in the future so, in general, more is better. That being said, too many blurry or out of focus pictures will hamper model building. You can optimize your time in the field by taking enough photos from the appropriate angles, however efficiency will come with practice. This is the protocol developed and used by the Kenkel lab to phenotype Acropora cervicornis colonies as part of field operations in the Florida Keys. We incorporate Agisoft Metashape markers in this workflow to scale models and improved model building. The scaling objects used by the Kenkel lab are custom-made, adjustable PVC arrays that include unique markers and bleaching color cards, affectionately called the "Tomahawk". Specs for building a Tomahawk are included in this protocol. Filtering and pre-processing of photos is not always necessary but can be used to salvage 3D models that would be otherwise blurry or incomplete. Here, we describe photo editing in Adobe Lightroom to adjust several characteristics of hundreds of images simultaneously.
For a walkthrough and scripts to run Agisoft Metashape on the command line, see https://github.com/wyattmillion/Coral3DPhotogram. For directions to phenotype coral from 3D models see our Phenotyping in MeshLab protocol.
These protocols, while created for branching coral, can be applied to 3D models of any coral morphology or any object really. Our goal is to make easy-to-use protocols using accessible softwares in the hopes of creating a standardized method for 3D photogrammetry in coral biology.
The Scaling System
The Scaling System
Building the Tomahawk
The Kenkel lab uses High Density Polyethylene (HDPE) 1/4 inch thick sheets from https://www.usplastic.com/search/?it=item&keyword=hdpe%20sheet to build our 3D photogrammetry scaling system, aka the Tomahawk. HDPE is sturdy and lightweight yet heavy enough to rest on the reef substrate without additional weight. In high wave energy conditions, dive weights can be attached or placed on top to further stabilize the system.
For a 4-sided Tomahawk, you will need:
4 - 9.75" x 1.25" x 0.25" HDPE bars (see figure below)
3 - 0.25" stainless steel carriage bolts (1" length)
3 - 0.25" stainless steel washers
3 - 0.25" stainless steel wing nuts
Figure 1: Schematic for building a Tomahawk scaling system
Once you have 4 HDPE bars, drill two centered holes 7/8" from either end of the bar with a 1/4" drill bit. Before assembling, it is useful to square off the backside of each hole to create a pocket roughly a 1/8" deep. This will allow the carriage bolt to sit into the plastic, preventing it from turning as you tighten the wing nut. This can be done with an appropriately-sized chisel or flathead screwdriver.
Front side of hole
Back side of hole with squared off edges
HDPE bars can then be connected with a carriage bolt secured by a washer and wing nut after being pushed through the overlapping bars (see Figure 1c). Wing nuts are especially useful for adjusting the Tomahawk underwater or when wearing gloves.
Metashape Markers and Color Cards
Agisoft Metashape provides printable markers to be included in the image capture process that we have found to improve model building and scaling. Because each marker is unique, the model building software can more easily determine overlapping points in photo sets which include markers. These markers are free and can be accessed through the Metashape GUI interface of Metashape Standard/Professional or from the free trial version.
To find the markers, open Metashape on your desktop (again, you do not need a license to access these).
Go to Tools > Markers > Print Markers
At this window, you can select the size of the markers you wish to make as well as the number you want to print on a page. The target size should be based on the object you're planning to photograph. Large reef-scapes will need larger targets. For individual coral colonies, I use twenty 12 bit, 2mm markers. I have found this number of markers is sufficient to surround the coral and the small size allows for more markers to be included in more photos.
Print the markers directly on waterproof paper in a desired arrangement. Be sure that your waterproof paper is printable (some types will melt in the printer). I rearrange the markers on the original Metashape PDF to increase the number of markers per page to save paper. See 
PStargets2mm_dense.pdf
PStargets2mm_dense.pdf The Kenkel lab also includes a CoralWatch Coral Health Chart color reference card on each of the Tomahawks to allow for downstream analyses of bleaching. Each of the 4 scales on the reference card is cut out and attached to the 4 faces of the Tomahawk.
Cut out and attach the Metashape markers onto the Tomahawk with glue or electrical tape. I find that electrical tape rather than glue is helpful when replacing worn out markers. See below for example of set up.
Single Tomahawk face with 4 Metashape markers and 1 side of a CoralWatch Coral Health Chart. White electrical tape holds down targets and color card while ensuring all markers and colored sections are visible.
The Tomahawk is designed to be adjustable to fit the size and shape of the area/object that you are photographing. By loosening the wing nuts, the PVC bars can pivot to enclose the focal object. Once the wing nuts are tightened, the Tomahawk will maintain its shape and position despite wave energy, thus ensuring a stable reference point for later model building. Small weights can be attached to the back or placed on the corners of the Tomahawk to further reduce movement of the scaling object in high wave energy environments.
We estimate the materials to build a Tomahawk to cost about $12, however, buying the supplies in bulk will lower the cost per system.
Completed Tomahawk with 4 faces that incorporate 16 Metashape markers and all 4 CoralWatch Coral Health Chart color scales.
Image Capture
Image Capture
Overview
Creating 3D models from 2D images requires collecting photos of your subject from numerous angles so that the software can use overlap from those photos to recreate the dimensions of your subject. So in general, more photos with a lot of overlap between photos is better.
We recommend 80% overlap between photos to ensure that Agisoft Metashape can accurately generate the coral model. The number of photos is dependent on the size and complexity of the coral. When in doubt, err in favor of more photos.
Here are the number of photos we typically take for Acropora cervicornis colonies of varying sizes:
5-20 cm TLE : 40 -75 photos
15 - 40 cm TLE: 75 - 125 photos
40-70 cm TLE: 125 - 175 photos
90-125 cm TLE: 175 - 300 photos
An 8 cm TLE coral that required 52 photos
A 29 cm TLE coral that required 97 photos
A 60 cm TLE coral that required 143 photos
A 114 cm TLE coral that required 276 photos
Angles
The complex structure of some corals requires photographs from multiple angles to provide a full picture of the coral for 3D model building. To do this, we recommend starting with the camera 0 degrees off the reef substrate about 5 - 10 inches away from the coral. (The closer the camera is to the coral, the more photos required for full coverage at 80% overlap).
Rotate in a circle taking photos as you move around the focal object. Then, move the camera to about 45 degrees off the substrate looking downward at the coral about 5-10 inches away and rotate around the coral as you take photos (see diagram below).
Optimal camera paths and angles to photograph a single coral colony for 3D photogrammetry
Additional photos should be taken of areas where complex branch intersections or overlaping portions of the coral will require increased resolution to accurately build. Take photos about 3-6 inches away from these sections, making sure the coral is in focus. This step tends to make up most of the difference between the number of photos needed for smaller versus larger corals.
Finally, take 1 or 2 top down photos that include the coral and the entire scaling object in the frame.
Tips from our experience
- Practice before getting in the water. The Kenkel lab practices with dead coral skeletons or sticks to ensure good models can be generated before entering the water.
- DO NOT MOVE the scaling object. If it moves during your photo series, you should restart. If wave energy is too high, a dive weight can be place on top to weigh down scaling object.
- Interval mode on the camera is helpful but can be slow and sometimes cannot focus on the coral with each shot. Rapid or burst mode allows you to collect photos quickly but often produces too many photos per coral. Instead, we most often will manually press the shutter-release for each photo so we can ensure the coral is in frame and in focus.
- White balancing or using "Underwater mode" helps improve the resolution and clarity of the final photos. We use the Olympus TG Tough 4/5 models which have interval, burst, and underwater mode and white balancing capabilities.
- Avoid creating shadows over the coral as you move. If possible, move your arm around to take photos without adjusting body position.
- Take photos of something unrelated to the coral (the sky or your hand) to indicate if you are restarting image capture for a coral or are switching to another colonyl. It will be hard to tell the difference later on when you organize the photos from multiple colonies taken in one dive.
- Hold the camera in one orientation the entire time. Do not switch between landscape and portrait. Do not use the zoom feature.
Pre-processing/filtering
Pre-processing/filtering
Organization and filtering
To build a model in Metashape, you must input photos that are specific to that model so it is best to organize photos into photosets for each coral to be individually modelled. This process can be time consuming, but while you organize photosets, you can remove blurry, out of focus images, or images where the coral is not present.
Removing blurry photos will speed up model building although Metashape will often remove unusable photos as it builds.
Color Adjustments
There are a few reasons to adjust the color of your photos. One case is to white balance to remove the blue/green tint from underwater photos. This may improve contrast and help the coral stand out from its background. Coral can have bright coloration that reduces the ability of Metashape to reconstruct that area. In the case of Acropora cervicornis, bright white apical tips of branches can be a problem area when definition of the branch tip is reduced. This can be improved with color correction, i.e. changing contrast and sharpness.
The Kenkel lab uses Adobe Lightroom to adjust the color of hundreds of photos simultaneously. Lightroom can be purchased and downloaded here . The following steps are specific to Lightroom but the goal of color correction can be applied using other softwares.
In Lightroom
Add photos with the + Icon on the left or from File > Add photos. Select the photos you want to edit (can be done as a batch) and select review for import. Add the photos to Lightroom.
Select a photo and adjust the effects using the Edit tool bar on the right or by hitting "e" on your keyboard. Here you can change Exposure, Contrast, Sharpening, etc., as well as white balance the image. The goal here is to make the coral stand out from the background and give definition to branches, especially those that overlap.
You can manually adjust each photo or apply settings generated for one photo to all the others. Because a single photoset will have roughly the same lighting throughout, you can get away with editting one or two photos and applying those setting to the rest of the photoset.
To do this, select the newly edited image then go to Photo > Copy Edit Settings. Then select the photos you wish to apply those settings to. Next, click Photo > Paste Edit Settings. View the newly edited photos to make sure the photos look the way you want.
Save the edited photos to your computer with File > Save to
The Kenkel lab uses Agisoft Metashape to build coral models. This software can be run through an interactive GUI window which is straighforward but is limited by the processing power of the computer it is ran on. Instead, the Kenkel lab utilizes a high power computing system to increase the speed at which Metashape builds models. Using Metashape on the command line is less straightforward so we have provided a walkthrough with the appropriate scripts to do this at https://github.com/wyattmillion/Coral3DPhotogram. For directions on how to analyze your 3D models, check out out our Phenotyping in MeshLab protocol.