A 2D fully convolutional neural network for nearshore and surfzone bathymetry inversion from synthetic imagery of the surfzone using the wave model Celeris Adam Collins,1 Katherine L. Brodie,2 Spicer Bak,2 Tyler Hesser, Matthew W. Farthing,3 Douglas W. Gamble,1 Joseph W. Long,1 3 1 University of North Carolina at Wilmington, Earth and Ocean Sciences, Wilmington, NC 2 U.S. Army Engineer Research and Development Center, Coastal and Hydraulics Laboratory, Duck, NC 3 U.S. Army Engineer Research and Development Center, Coastal and Hydraulics Laboratory, Vicksburg, MS 1 {amc3496, gambled, longjw}@uncw.edu,2 {katherine.l.brodie, spicer.bak}@erdc.dren.mil, 3 {tyler.hesser, matthew.w.farthing}@erdc.dren.mil Abstract processes, with water depth and slope being principal parts of the governing wave equations in the nearshore. Currently Bathymetry has a first order impact on nearshore and surfzone hydrodynamics. Typical survey techniques are expensive and the most accurate methods for determining bathymetry are time-consuming, require specialized equipment, and are not in-situ observations involving physical contact with the bot- feasible in a variety of situations (e.g. limited manpower tom, or acoustic hydrographic surveys from vessels (Moul- and/or site access). However, the emergence of nearshore ton, Elgar, and Raubenheimer 2014b). Both approaches are remote sensing platforms (e.g. Unmanned Aircraft Systems limited by the requirement of a physical presence at the (UAS), towers, and satellites) from which high-resolution im- site, which complicates their use in isolated environments agery of the sea-surface can be collected at frequent inter- or during unsafe water conditions (Birkemeier and Mason vals, has created the potential for accurate bathymetric esti- 1984). In addition, the surfzone bathymetry is constantly mation from wave-inversion techniques without in-situ mea- changing, and can vary considerably day-to-day making surements. While a variety of physics-based algorithms have consistent measurement impractical using traditional meth- been applied to nearshore and surfzone bathymetric inversion problems, the commonly used approaches do not account for ods (Moulton, Elgar, and Raubenheimer 2014a). An alter- non-linear hydrodynamics that are prevalent during breaking native approach to estimate bathymetry that would over- waves. Models for estimating non-linear wave dynamics are come some of these limitations is using remotely sensed slow and often require large amounts of computational power data sources, which don’t require a physical presence in the which make them unfeasible for rapid estimations of depth. water at a site. A number of remote sensing approaches to Fully convolutional neural networks (FCNs) are a branch of estimate bathymetry have been developed including direct artificial intelligence algorithms that have proven effective at (e.g. bathymetric LiDAR, multi and hyper-spectral imagery) computer vision tasks in semantic segmentation and regres- and inferred approaches (e.g. image or radar-derived obser- sion problems. In this work, we consider the use of FCNs for vations of wave-kinematics and breaking)(Holland, Palm- inferring bathymetry from video-derived imagery. The FCN sten, and others 2018). Visible band imagery offers a low- model presented shows the feasibility of using an AI sys- tem to perform bathymetric inversion on time-averaged im- cost approach which exploits the visible surface signature ages (timex) of realistic-looking, synthetically generated sur- of shoaling and breaking waves in the nearshore – wave fzone imagery from the hydrodynamic wave model Celeris transformation processes that are largely controlled by wa- (Tavakkol and Lynett 2017). Ongoing work includes extend- ter depth. Images record the location of wave breaking or ing the FCN to incorporate synthetic video frames as input as speeds of wave propagation, which can be related to wa- well as testing with actual tower and satellite imagery. ter depth using a bathymetry inversion algorithm (Holman, Lalejini, and Holland 2016; Van Dongeren et al. 2008). The use of different remote sensing platforms, such as satel- Introduction lites and unmanned aerial vehicles (UAVs) (Holland et al. Accurate knowledge of nearshore and surfzone water depths 2010; Holman, Brodie, and Spore 2017; Brodie et al. 2019; is important for a wide range of applications, ranging from Almar et al. 2019; Bergsma, Almar, and Maisongrande enhancing the personal safety of beach-goers, to industrial 2019), to collect this imagery offers opportunities to esti- and military applications such as identifying navigable ar- mate bathymetry in areas that would normally be difficult eas for ships or other landing craft (Avera et al. 2002). The or costly to assess with traditional methods, increasing data bottom boundary condition is one of the most important in- availability and reducing costs (both financial and temporal) puts for numerical simulations of nearshore and surfzone compared to in-situ observation methods (Gao 2009). Copyright c 2020, for this paper by its authors. Use permitted While analyzing sequences of coastal video imagery with under Creative Commons License Attribution 4.0 International traditional signal processing and computer vision algorithms (CC BY 4.0). to estimate bathymetry holds promise, the inherent complex- training data sets using real imagery are likely too small to ity of the nearshore and surfzone, which includes many non- find proper parameters from randomly initialized DCNN pa- linear processes, will always lead to errors in any bathymet- rameters, which would likely lead to over-fitting (Kemker, ric inversion model that simplifies the effects of these pro- Luu, and Kanan 2018). This study seeks to both utilize the cesses through a linear approach. Machine learning algo- non-linear prediction powers of a deep neural network and rithms, particularly deep neural networks, have previously explore the use of synthetic data to approach the bathymetry demonstrated the ability to identify and classify pixels in inversion problem through the development of a deep learn- complex images far beyond the quality of traditional hand- ing network using synthetic surfzone imagery derived from written algorithms (Simonyan and Zisserman 2014). Apply- a photorealistic visualization of the nearshore wave model, ing machine learning for classification of remote sensing im- Celeris (Tavakkol and Lynett 2017). ages on a pixel-wise basis is referred to as semantic segmen- tation and has increasingly been utilized in remote sensing Background over the past decade. The combination of high-resolution data and faster computer processing has made this possible Parametric equations have a long history of use to approxi- by allowing for the parallel processing of millions of param- mate beach slopes and are based on the model eters, which is required to process the increasing resolutions h = Ax2/3 from remote sensing technologies, such as UASs and/or HD camera systems (Christophe et al. 2011). where h is the water depth, A is a constant, and x is dis- Traditional low-resolution algorithms used to analyze re- tance in the cross-shore direction (Bruun 1954). While para- mote sensing imagery do not maintain their effectiveness at metric beach models are good for quantifying large-scale these higher resolutions of present-day interest, while the trends, such as regional inundation due to sea level rise; in abundance of parameters in the high spatial and spectral res- smaller regions of interest, surfzone and nearshore variations olution data make a traditional analytical algorithm more in bathymetry, like sandbars, are not accounted for. To ad- difficult to develop when classifying complex features (Zhu dress the limitations of parametric bathymetry models, the et al. 2017). Image processing algorithms to simplify these location of the shoreline and sandbars can be added to para- datasets are often time consuming to run and require sub- metric models using time-averaged imagery (timex) (Hol- stantial investment in powerful computer hardware. How- man et al. 2014). Sandbars are identified by time-averaging ever, the performance of semantic segmentation of high- sequences of video imagery of the surfzone, to generate a resolution scenes has increased rapidly since 2012, which timex image (Lippmann and Holman 1989). Timex images was the beginning of the domination of supervised deep are used to identify regions of persistent wave breaking. learning with the introduction of the deep convolutional neu- Waves break in areas with reduced water depth over sand- ral network (DCNN) AlexNet (Krizhevsky, Sutskever, and bars, when the water depth decreases to be between 0.4 and Hinton 2012; Alom et al. 2018). In addition, deep neural 0.8 of their wave height (Komar and Gaughan 1973). Per- networks have the advantage of being extremely fast to com- sistent regions of wave breaking appear as a white-band that pute targets once trained, yielding portability to run on a can then be manually digitized from timex images to iden- relatively modest processor. For example, deep neural net- tify the position of the surf zone sandbars. Exposure times works allow for near real-time semantic segmentation on to generate the time-averaged images can range from a min- board a UAS or sea-based vessel to aid autonomous navi- imum of 10 minutes to full day exposures, using a variety of gation (Tian et al. 2018). video capture techniques (Guedes et al. 2011). However, the downside of typical supervised training with Parametric bathymetry models can also be used in two- deep neural networks is the requirement for extremely large dimensions (2D) to generate more complex bathymetry labeled datasets. Because of this, many classifiers and seg- (Holman, Lalejini, and Holland 2016). Their parametric mentation networks start with pre-trained parameters as op- beach tool requires twelve parameters to create a shore- posed to the typical machine learning approach where the line morphology, however eight of the parameters are eval- parameters start out as random values. These pre-trained uated to constants in practical implementation. The remain- parameters are then transferred to the current task, chang- ing variable inputs are the climatological beach slope at the ing only a small subset of them with the training data for shoreline, the depth and bottom slope at some location sea- the new problem looking to be solved. Parameters that have ward of the active bar zone, and the cross-shore location of been pre-trained on large image datasets will be able to iden- the sand bar crest. In the 2D implementation, a mean shore- tify vague features, such as edges in an image. These vague line is input by the user, and normal transects from the shore- features are then used as inputs into the final layers, which line are calculated. The distance from the shoreline to the are the layers whose parameters will be adjusted by the new sandbar, using expert identification with time lapsed images, training dataset. This application of a trained network be- is also input into the model, along with an estimated offshore ing adjusted and then applied to another task is commonly depth and beach slope. This inversion model was tested by referred to as transfer learning (Huh, Agrawal, and Efros (Holman, Lalejini, and Holland 2016) and showed to have a 2016). mean bias and RMSE error of 0.27 m and 0.49 m, respec- Oceanographic data sets of coastal imagery coincident tively, over the study area at Duck, NC. to highly accurate bathymetric measurements are extremely Beyond simple parametric representations, a number of rare, and generally occur only during small waves. Available efforts have been made to directly measure surf-zone pa- rameters of interest in order to estimate bathymetry. Di- Time-lapse images from video rect inversion techniques have focused on measuring wave speeds from image sequences and estimating bathymetry using linear wave theory (Stockdon and Holman 2000; Plant, Holland, and Haller 2008; Holman, Plant, and Hol- land 2013; Bergsma and Almar 2018; Bergsma, Almar, and Maisongrande 2019), whereas other inversion schemes have utilized data assimilation techniques which combine the re- motely sensed parameters with numerical models. Data as- similation techniques have ranged from classical variational Figure 1: A single frame of the Celeris visualization (a), methods and Kalman Filters (Holman, Plant, and Holland 8000 frames are averaged together to form the time- 2013; Wilson and Berezhnoy 2018) to ensemble approaches averaged image (timex) (b), with brighter areas showing (Wilson, Özkan Haller, and Holman 2010) and more recent where waves break more often. The far offshore part of the nonlinear extensions of the Kalman Filter (Ghorbanidehno visualization in (a), which is the deepest section, is not in- et al. 2019). The types of surface observations that have been cluded in the final timex (b) due to the lack of breaking explored includes wave speeds as well as wave heights, cur- waves. This produces a timex image (b) of 1795 m in the rents (Holman, Plant, and Holland 2013; Wilson et al. 2014; alongshore direction, and 860 m in the cross-shore direction. Moghimi et al. 2016) and estimates of wave energy dis- Finally, the timex is then cropped to be (512, 512) pixels and sipation from timex images (Van Dongeren et al. 2008; centered on the alongshore of the image (c). The shape is for Aarninkhof, Ruessink, and Roelvink 2005). In general, ap- ease of input to the ML model, while the position is due to proaches combining modern inversion techniques with high edge effects from wave generation. fidelity models of nearshore hydrodynamics have shown the potential to provide higher accuracy estimates under a wider set of hydrodynamic regimes. However, this accuracy intro- Bathymetry Selection duces added complexity and computational expense, which are potential barriers to fielding these approaches for real- While there are multiple parameters to adjust in the Celeris time application in limited resource environments like mo- wave model, two inputs have the largest effect on the gen- bile platforms. eration of synthetic video imagery of surfzone processes: In this effort we explore the ability of machine learn- bathymetry and offshore wave boundary conditions. A set ing algorithms to learn the relationship between locations of 100 statistically driven bathymetries were generated us- of persistent wave breaking in timex images and surfzone ing an Empirical Orthogonal Function approach on 40 bathymetry, removing the need for manual digitization of the years of in-situ bathymetry surveys collected at the U.S. sandbar location (e.g. (Holman, Lalejini, and Holland 2016)) Army Corps of Engineers Field Research Facility (Braud or a numerical wave dissipation model (e.g. (Van Dongeren and Obled 1991). This set of bathymetries were then di- et al. 2008)). vided into separate sets of training (80 bathymetries), val- idation (10 bathymetries), and testing (10 bathymetries). Methodology These bathymetries extend 1795 m in the alongshore direc- tion (parallel to the beach), and 970 m in the cross-shore di- Wave Modeling Software Selection rection (perpendicular to the beach), with an average shore- Celeris is an open source Bousinessq wave model that runs line position of about 220 m in the cross-shore direction. on a GPU cluster and creates visually realistic simulations The cross-shore distance is chosen due to its correspon- of nearshore and surfzone waves in near real-time on a typ- dence with the location of the FRF’s 8 m water depth pres- ical desktop computer (Tavakkol and Lynett 2017). Celeris sure sensor array. In addition, 100 synthetically generated generates and visualizes different wave interactions, such as bathymetries were created, using parametric beach slopes, shoaling, refraction, reflection, and breaking. These are the with sandbars, troughs, and depressions created by perturb- relevant processes influencing the visual expression of wave ing the slope at random locations, frequencies, and inten- propagation in the nearshore, and therefore the wave model sities (Figure 2). These bathymetries are introduced to al- results provide a relevant corollary to observations collected low the ML model to learn different breaking patterns, such by remote video platforms. This wave model was selected as multiple sandbars, and their correspondence with water not only for its efficient run time, but also its pseudo-realistic depth that are not usually visible in the bathymetries statis- visualizations of wave transformation and breaking, which tically driven from the observed data set from Duck, NC. can be used as a proxy for coastal video imagery. Through It also serves as a preventative measure against over-fitting, video capture of the wave model results, a 20 minute video intending to generalize the ML model’s ability to accurately is created after an initial 10 minute spinup time. These video assess water depths for breaking wave intensities from im- files of the Celeris visualization are then averaged in time to agery, by using a wider range of inputs and depths at differ- produce a timex image (similar to the timex images typically ent cross-shore locations beyond that of the statistics from created by nearshore video monitoring stations (Holman and the historical dataset (Figure 3). Perturbations from the mean Stanley 2007)) for that bathymetry and wave condition (Fig- profile were generated and added between 200 m from the ure 1). shoreward domain edge and 200 m from the offshore bound- Random bathymetry generator Cross-shore profile hexbins Figure 2: The different parts of the random bathymetry gen- erator. This script was created to model different beaches and slopes than typically found in Duck, NC, such as multi- sandbar beaches. First a parametric profile is created, from that trough, sandbar, and spot variations are combined onto Figure 3: Density plot of the range of water depths at each the original slope. This is then blurred to create the final location that were derived within a standard deviation of bathymetry. historical Duck, NC bathymetries. A similar density plot of the randomly generated bathymetries. Red areas are where nearly all of the 100 bathymetries had the same average ary. Between 0-25 bar-trough (alongshore uniform) features, cross-shore depth at that point. of random amplitudes and spacing, are generated and ap- plied to the mean profile. At least 50 and up to 100 along- shore non-uniform, circular features of various radii and am- Wave height & direction rose plitude (positive and negative) are applied to the same por- tion of the profile area. The bathymetry is smoothed and stretched with a length scale of up to 20, and then the en- tire profile is shifted so that the average depth at the offshore boundary is at the desired depth. Bathymetries were con- ditioned to be centered at 8 m water depth at the offshore boundary since Celeris was setup to force with wave obser- vations observed in 8 m depth. This set of bathymetries was similarly divided into separates set for training (80), valida- tion (10), and testing (10). Wave Condition Selection To force the wave model, we selected the most highly proba- ble wave conditions that were measured at the FRF’s phased array of pressure sensors in 8 m water depth (about 950m from the shoreline) (Long and Oltman-Shay 1991). The wave rose, (Figure 4), bins the historical wave conditions by significant wave height and direction over the course of 10 years. Individual simulations were performed using the most frequent wave conditions observed in Duck, NC. While additional conditions were initialized by using the probabilistic wave conditions as boundary conditions for a Latin hypercube sub-sampling of the data plotted in Fig- ure 3. The wave height affects where in the domain the Figure 4: Binned histogram of observed wave conditions at waves break, whereas the wave frequency will affect how the FRF’s 8 m offshore array in Duck, NC from 1/1/2010 to often the waves break (and resultant image intensity). The 12/31/2019. The radial direction shows the incoming direc- wave direction also affects the final timex image by vary- tion of waves. The outward axis shows the conditions binned ing the direction waves travel toward the shore, and thus by their wave height. For example you can see that Duck, the direction in which breaking occurs. These ranges are: NC, where normally incident waves travel from 71.8◦ , expe- wave heights between 0.7 m and 2.5 m, peak frequencies rienced waves of a 0.5 - 1 m average significant wave height between 0.09 Hz and .2 Hz, and peak wave direction be- from the 67.5◦ to 90◦ interval around 6% of the time during tween 45◦ and 112.5◦ True North). Wave directions outside the queried interval. of these ranges only occur 14.7% of the time over the sam- pled time period. While wave conditions with wave heights for slope is written to the last channel. The RGB channels smaller than 0.7 m are quite common (greater than 80% of are normalized across the training set. The slope is calcu- the time), they are not considered in this study due to the lated by finding the physical slope from the alongshore av- very small observable surfzone features produced by low eraged shoreline elevation to the alongshore averaged end of energy wave conditions and lack of wave breaking. These image depth for each bathymety. Estimated offshore beach three conditional inputs were used as inputs to the TMA slope is also an input to the latest parametric beach model equation to generate a 2D wave spectra (Bouws et al. 1985; (Holman, Lalejini, and Holland 2016). With all the inputs Hughes 1984) that is used as the input wave condition to the into the model the role of the trained FCN model is to es- Celeris wave model. timate the existence and extent of perturbations from the parametric slopes by examining the breaking wave pattern Network Architecture observable through the timex imagery. The need to directly convert the visual signal of breaking waves in an image to water depth from the visual input fea- Training tures partially motivates the usage of a 2D fully convolu- Training was performed using the timex images from 80 ran- tional neural network (FCN), which has proven to be effec- domly generated bathymetries and 10 of the most highly tive in pixel-wise regression and semantic segmentation ap- probable wave conditions measured at Duck, NC, yielding plications in other remote sensing fields (Wu et al. 2019). 800 training samples, of which there are 80 unique targets. Another motivation for this network selection is the poten- The training was done with Tensorflow 2.0’s train on batch tial for transfer learning. By attempting to generate and use function and random images were selected using the modi- synthetic data as visually close to measured optical data as fied PyTorch Dataset class for a mini-batch size of 15, which possible, the potential exists for transfer learning, where the was chosen because it was the largest mini-batch size that found bathymetric inversion ML model can be only slightly could fit into GPU memory on current local hardware. Dur- modified with the smaller subset of true coastal imagery ing training, mini-batches were randomly selected from the data that exists, compared to the near limitless availability training dataset until the end of the epoch. The validation of synthetic data. The FCN setup has been demonstrated as dataset was created similarly to the training dataset but con- a particularly apt network architecture for transfer learning, sists of 10 different bathymetries ran over the same 10 wave with examples of it being successful for semantic segmen- conditions used in the training set. At the end of each of tation with similar remotely sensing data (Kemker, Salvag- these randomly sampled epochs validation was ran over 50 gio, and Kanan 2018; Kim et al. 2018; Sakurai et al. 2018; images randomly selected from the validation dataset. Wurm et al. 2019). The type of FCN model chosen is a mod- ified U-net (Ronneberger, Fischer, and Brox 2015) architec- The optimizer that found the best convergence was ture, which can be easily modified to do pixel-wise regres- NAdam with all parameters at default settings except the sion as well as its original use for semantic segmentation starting learning rate is modified to .00008. In addition, a (Yao et al. 2018). The current architecture is modified to custom learning rate decay is introduced where the learn- accept images of (512, 512, 4) size by adding two layers. ing rate is reduced by 10 percent after the validation loss The traditional dropout rates and upsampling methods used has not decreased for 8 straight epochs. Convergence with originally are also modified for better generalization to our these parameters takes around 12 hours of training time on domain. the hardware described above. Numerical Experiments Testing The FCN model was trained, validated, and tested using Py- Testing was done by using timex images from 10 bathyme- Torch, Tensorflow, scipy, numpy, cv2, and tifffile libraries. tries and 10 wave conditions selected using Latin hypercube The PyTorch.Dataset class was overwritten to perform si- sampling within the realistic boundary conditions measured multaneous loading and augmenting of the dataset to in- in Duck, NC, yielding 100 testing samples, where there were clude an additional channel with information on the offshore 10 unique targets. The bathymetries used for the test set were beach slope. The FCN model and training/validating/testing generated with the same bathymetry generation code used to functions were implemented in Tensorflow 2.0, due to the make the random training bathymetry sets, but differed visu- smaller memory imprint than when using a PyTorch model. ally from the training and validation samples, and were not The Celeris model simulations were ran on a Dell Precision used during those processes. The wave conditions were also 5820 with 64GB of RAM and a NVIDIA RTX 2080. The unique to the test set. FCN model was trained on a custom built PC with 64GB of The testing was done with Tensorflow 2.0’s predict func- RAM and a NVIDIA RTX Titan V with 24GB of VRAM. tion, with visualization done with matplotlib.pyplot. Exam- The final timex image used for training is a subset of the ple outputs are shown in Figure 5. In Figure 5a and 5b, the entire Celeris wave model domain (Figure 1). This timex im- largest RMSE and a significant amount of Bias error oc- age is stored with 3 red, green, and blue (RGB) channels as curs offshore of the sandbar/breaking wave visual signature a (512, 512, 3) tiff file. When these files are loaded during (right side of images). These areas will only occasionally training an additional channel is added to provide additional see breaking waves and in turn the estimates are biased by input features (slope) for a more accurate prediction, result- the algorithm as a result. Additionally, errors grow in the ing in a final image size of (512, 512, 4). The constant value trough between the sandbar and the shoreline, where the waves have dissipated enough energy to stop breaking be- fore re-breaking near the shoreline, and thus little informa- Example outputs & analysis tion about depth is observable in this region (Figure 5c). Over the entire test set of 10 unique bathymetries and 10 unique wave conditions the mean bias and RMSE of water depth were 0.449 m and 0.390 m (Figure 5d). In most in- stances across the test set, the prediction was too shallow (negative bias), exceptions to this rule are commonly seen in nearshore troughs and the seaward side of the sandbar when there are no breaking waves and the resulting prediction is often too deep (positive bias). Conclusions (a) Initial results show promise in the ability of the trained FCNs to estimate nearshore water depths from synthetic wave breaking signatures expressed in timex imagery, generated with the wave model Celeris. The FCN model shows a clear ability to identify the differences between deeper and shal- lower areas, identifying the location of sandbars, troughs, and depressions not seen in the original training dataset, and that they are directly related the amount of breaking waves in that particular location. For this study, the bias (0.449 m) and RMSE (0.390 m) over the test set is encouraging, and com- parable to other remotely sensed inversion techniques. Some (b) error is inherent as the timex images extend up to 660m off- shore, where waves are generally not breaking. We chose to use unique wave conditions during the testing phase to deter- mine if the FCN model could show understanding that timex images that varied greatly depending on different wave pat- terns can still point to the same target water depth. Testing with conditions not seen during training is also important because it would be impossible to train for all possible com- bination of wave conditions that could be seen at a given lo- cation due to the wide ranges in wave heights, frequencies, and directions. (c) Future Work Modifications to the model are currently in development. (d) The most promising is the inclusion of wave condition fea- tures as inputs to the U-net architecture by including their Figure 5: Three example input features and outputs are values along with slope in the additional channel. These en- shown above (a;b;c). Each prediction has six frames of data vironmental parameters are hypothesized to help with the shown here. The first frame shows the input RGB channels algorithm because they directly impact the resultant (RGB) of the timex image. The second frame is the ground truth timex image generated by the wave model. Development elevation. The third frame is the predicted elevation. The of these input features would also be advantageous for the fourth frame shows two average offshore cross-shore tran- transfer to real datasets, as they are available at most loca- sects, with the black line being the predicted transect, and tions worldwide from global wave and tide models. the cyan line being ground truth. The fifth frame shows the Additional future work will improve the FCN model and average offshore cross-shore RMSE error. The sixth frame analysis by 1) comparing the bias and RMSE of the FCN shows the offshore Bias with areas predicted too shallow in model on the test set to predictions made by the paramet- red, and areas predicted too deep in blue. The tables (d) give ric beach tool introduced by (Holman, Lalejini, and Holland summary statistics on each particular input: Mean Absolute 2016); 2) expand the synthetic training and testing dataset Error, Mean Square Error, Mean Bias, Median Absolute Er- in the form of more bathymetries and wave conditions to ror, Greatest Pixel Error, and percentile error, such that ’95% significantly increase the ranges of slopes and wave condi- of the pixels have less error than this value’ over the offshore tions seen during training/testing; 3) introduce video frame portion of the test set. data as a feature input into the FCN model, likely improv- ing the accuracy in areas with little to no breaking waves by allowing the algorithm to utilize observations of wave speed in these areas (which is proportional to water depth) in addi- phy and bathymetry from a lightweight multicamera uas. tion to wave breaking; and 4) curate and compile a real timex IEEE Transactions on Geoscience and Remote Sensing and bathymetry dataset that can be similarly represented by 57(9):6844–6864. Celeris to then test the ability of the FCN model to transfer Bruun, P. 1954. Coast erosion and the development of beach to real datasets. profiles, volume 44. US Beach Erosion Board. Christophe, E.; Michel, J.; Inglada, J. J. I. J. o. S. T. i. A. Acknowledgement E. O.; and Sensing, R. 2011. Remote sensing processing: We would like to thank the U.S. Army Corps of Engineers From multicore to gpu. 4(3):643–652. for providing the main source of funding for the research Gao, J. J. P. i. P. G. 2009. Bathymetric mapping by through the Deputy Assistant Secretary of the Army for Re- means of remote sensing: methods, accuracy and limitations. search and Technology under ERDC’s Military Engineer- 33(1):103–116. ing research program titled “Force Projection Entry Opera- tions”, as well as the Geological Society of America for their Ghorbanidehno, H.; Lee, J.; Farthing, M.; Hesser, T.; Kitani- funding support. We would also like to thank the many fac- dis, P. K.; and Darve, E. F. 2019. Novel data assimilation ulty and graduate students at University of North Carolina - algorithm for nearshore bathymetry. Journal of Atmospheric Wilmington that provided their input and feedback on the re- and Oceanic Technology 36(4):699–715. search, as well as Dr. Jonghyun Harry Lee at the University Guedes, R. M.; Calliari, L. J.; Holland, K. T.; Plant, N. G.; of Hawaii at Manoa for his support. Pereira, P. S.; and Alves, F. N. 2011. Short-term sandbar variability based on video imagery: Comparison between References time–average and time–variance techniques. Marine Geol- ogy 289(1-4):122–134. Aarninkhof, S.; Ruessink, B.; and Roelvink, J. 2005. Nearshore subtidal bathymetry from time-exposure video Holland, K. T.; Lalejini, D. M.; Spansel, S. D.; and Holman, images. Journal of Geophysical Research: Oceans 110(C6). R. A. 2010. Littoral environmental reconnaissance using tactical imagery from unmanned aircraft systems. In Ocean Almar, R.; Bergsma, E. W.; Maisongrande, P.; and Sensing and Monitoring II, volume 7678, 767806. Interna- de Almeida, L. P. M. 2019. Wave-derived coastal tional Society for Optics and Photonics. bathymetry from satellite video imagery: A showcase with pleiades persistent mode. Remote Sensing of Environment Holland, K. T.; Palmsten, M. L.; et al. 2018. Remote sens- 231:111263. ing applications and bathymetric mapping in coastal envi- ronments. Advances in Coastal Hydraulics 375–411. Alom, M. Z.; Taha, T. M.; Yakopcic, C.; Westberg, S.; Sidike, P.; Nasrin, M. S.; Van Esesn, B. C.; Awwal, A. Holman, R. A., and Stanley, J. 2007. The history and techni- A. S.; and Asari, V. K. J. a. p. a. 2018. The history be- cal capabilities of argus. Coastal engineering 54(6-7):477– gan from alexnet: A comprehensive survey on deep learning 491. approaches. Holman, R. A.; Lalejini, D. M.; Edwards, K.; and Veera- Avera, W. E.; Harris, M. M.; Bibee, L. D.; Lingsch, S.; and mony, J. 2014. A parametric model for barred equilibrium Sample, J. T. 2002. Multibeam bathymetry from a mine- beach profiles. Coastal engineering 90:85–94. hunting military sonar. Report, NAVAL RESEARCH LAB Holman, R. A.; Brodie, K. L.; and Spore, N. J. 2017. Surf STENNIS SPACE CENTER MS. zone characterization using a small quadcopter: Technical Bergsma, E. W.; Almar, R.; and Maisongrande, P. 2019. issues and procedures. IEEE Transactions on Geoscience Radon-augmented sentinel-2 satellite imagery to derive and Remote Sensing 55(4). wave-patterns and regional bathymetry. Remote Sensing Holman, R. A.; Lalejini, D. M.; and Holland, T. J. C. E. 11(16):1918. 2016. A parametric model for barred equilibrium beach pro- Bergsma, E. W., and Almar, R. 2018. Video-based depth files: Two-dimensional implementation. 117:166–175. inversion techniques, a method comparison with synthetic Holman, R.; Plant, N.; and Holland, T. J. J. o. G. R. O. cases. Coastal Engineering 138:199–209. 2013. cbathy: A robust algorithm for estimating nearshore Birkemeier, W. A., and Mason, C. 1984. The crab: A unique bathymetry. 118(5):2595–2609. nearshore surveying vehicle. Journal of Surveying Engineer- Hughes, S. A. 1984. The tma shallow-water spectrum de- ing 110(1):1–7. scription and applications. Technical report, COASTAL EN- Bouws, E.; Günther, H.; Rosenthal, W.; and Vincent, C. GINEERING RESEARCH CENTER VICKSBURG MS. 1985. Similarity of the wind wave spectrum in finite depth Huh, M.; Agrawal, P.; and Efros, A. A. J. a. p. a. 2016. What water: 1. spectral form. Journal of Geophysical Research: makes imagenet good for transfer learning? Oceans 90(C1):975–986. Kemker, R.; Luu, R.; and Kanan, C. 2018. Low-shot learn- Braud, I., and Obled, C. 1991. On the use of empirical or- ing for the semantic segmentation of remote sensing im- thogonal function (eof) analysis in the simulation of random agery. Ieee Transactions on Geoscience and Remote Sensing fields. Stochastic Hydrology and Hydraulics 5(2):125–134. 56(10):6214–6223. Brodie, K. L.; Bruder, B. L.; Slocum, R. K.; and Spore, Kemker, R.; Salvaggio, C.; and Kanan, C. 2018. Algorithms N. J. 2019. Simultaneous mapping of coastal topogra- for semantic segmentation of multispectral remote sensing imagery using deep learning. ISPRS Journal of Photogram- Van Dongeren, A.; Plant, N.; Cohen, A.; Roelvink, D.; metry and Remote Sensing. Haller, M. C.; and Catalán, P. 2008. Beach wizard: Kim, H.; Kim, H.; Hong, Y. W.; and Byun, H. 2018. Detect- Nearshore bathymetry estimation through assimilation of ing construction equipment using a region-based fully con- model computations and remote observations. Coastal en- volutional network and transfer learning. Journal of com- gineering 55(12):1016–1027. puting in Civil Engineering 32(2):04017082. Wilson, G., and Berezhnoy, S. 2018. Surfzone state esti- Komar, P. D., and Gaughan, M. K. 1973. Airy wave theory mation, with applications to quadcopter-based remote sens- and breaker height prediction. In Coastal Engineering 1972. ing data. Journal of Atmospheric and Oceanic Technology 405–418. 35(10):1881–1896. Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012. Wilson, G. W.; Özkan Haller, H. T.; Holman, R. A.; Haller, Imagenet classification with deep convolutional neural net- M. C.; Honegger, D. A.; and Chickadel, C. C. 2014. Surf works. In Advances in neural information processing sys- zone bathymetry and circulation predictions via data assim- tems, 1097–1105. ilation of remote sensing observations. Journal of Geophys- ical Research: Oceans 119(3):1993–2016. Lippmann, T. C., and Holman, R. A. 1989. Quantification of sand bar morphology: A video technique based on wave Wilson, G. W.; Özkan Haller, H. T.; and Holman, R. A. dissipation. Journal of Geophysical Research 94(C1):995. 2010. Data assimilation and bathymetric inversion in a two- dimensional horizontal surf zone model. Journal of Geo- Long, C. E., and Oltman-Shay, J. M. 1991. Directional physical Research: Oceans 115(C12). characteristics of waves in shallow water. Technical re- port, COASTAL ENGINEERING RESEARCH CENTER Wu, G.; Guo, Y.; Song, X.; Guo, Z.; Zhang, H.; Shi, X.; VICKSBURG MS. Shibasaki, R.; and Shao, X. 2019. A stacked fully convolu- tional networks with feature alignment framework for multi- Moghimi, S.; Ozkan-Haller, H. T.; Wilson, G. W.; and Ku- label land-cover segmentation. Remote Sensing 11(9):1051. rapov, A. 2016. Data assimilation for bathymetry estimation at a tidal inlet. Journal of Atmospheric and Oceanic Tech- Wurm, M.; Stark, T.; Zhu, X. X.; Weigand, M.; and nology 33(10):2145–2163. Taubenböck, H. 2019. Semantic segmentation of slums in satellite images using transfer learning on fully convo- Moulton, M.; Elgar, S.; and Raubenheimer, B. 2014a. Im- lutional neural networks. ISPRS journal of photogrammetry proving the time resolution of surfzone bathymetry using in and remote sensing 150:59–69. situ altimeters. Ocean Dynamics 64(5):755–770. Yao, W.; Zeng, Z.; Lian, C.; and Tang, H. 2018. Pixel-wise Moulton, M.; Elgar, S.; and Raubenheimer, B. J. G. R. L. regression using u-net and its application on pansharpening. 2014b. A surfzone morphological diffusivity estimated from Neurocomputing 312:364–371. the evolution of excavated holes. 41(13):4628–4636. Zhu, X. X.; Tuia, D.; Mou, L. C.; Xia, G. S.; Zhang, L. P.; Plant, N. G.; Holland, K. T.; and Haller, M. C. 2008. Ocean Xu, F.; and Fraundorfer, F. 2017. Deep learning in remote wavenumber estimation from wave-resolving time series im- sensing. Ieee Geoscience and Remote Sensing Magazine agery. IEEE Transactions on Geoscience and Remote Sens- 5(4):8–36. ing 46(9):2644–2658. Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 234–241. Springer. Sakurai, S.; Uchiyama, H.; Shimada, A.; Arita, D.; and Taniguchi, R.-i. 2018. Two-step transfer learning for se- mantic plant segmentation. In ICPRAM, 332–339. Simonyan, K., and Zisserman, A. 2014. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Stockdon, H. F., and Holman, R. A. J. J. o. G. R. O. 2000. Estimation of wave phase speed and nearshore bathymetry from video imagery. 105(C9):22015–22033. Tavakkol, S., and Lynett, P. 2017. Celeris: A GPU- accelerated open source software with a Boussinesq-type wave solver for real-time interactive simulation and visual- ization. Computer Physics Communications 217:117–127. Tian, Y.; Pei, K.; Jana, S.; and Ray, B. 2018. Deeptest: Au- tomated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering, 303–314. ACM.