Reducing methane emissions is essential for mitigating global warming. To attribute methane emissions to their sources, a comprehensive dataset of methane source infrastructure is necessary. Recent advancements with deep learning on remotely sensed imagery have the potential to identify the locations and characteristics of methane sources, but there is a substantial lack of publicly available data to enable machine learning researchers and practitioners to build automated mapping approaches. To help fill this gap, we construct a multi-sensor dataset called METER-ML containing 86,599 georeferenced NAIP, Sentinel-1, and Sentinel-2 images in the U.S. labeled for the presence or absence of methane source facilities including concentrated animal feeding operations, coal mines, landfills, natural gas processing plants, oil refineries and petroleum terminals, and wastewater treatment plants. We experiment with a variety of models that leverage diferent spatial resolutions, spatial footprints, image products, and spectral bands. We find that our best model achieves an area under the precision recall curve of 0.915 for identifying concentrated animal feeding operations and 0.821 for oil refineries and petroleum terminals on an expert-labeled test set, suggesting the potential for large-scale mapping. We make METER-ML freely available at this link to support future work on automated methane source mapping.
map infrastructure [4, 5, 6, 7]. Methods for mapping Table 1 methane source infrastructure have been emerging as Counts and proportions of each category in METER-ML. well, including well pads in the Denver basin [8], oil re- The labels on the training set are obtained from public data ifneries and concentrated animal feed operations in the whereas the labels on validation and test sets are obtained U.S. [9, 10], and wastewater treatment plants in Germany from a consensus of two methane source identification experts. [11]. Each of these works depended on the curation of The individual category counts do not add up to the overall large, labeled datasets to develop the machine learning tarraei nla/vbaelliedd/twesitthcomuonrtes athsasnomonee(0m.8e%t)hoafntehseopuorcsieticvaeteexgaomryp.les models, but there is a lack of publicly available, labeled Earth observation data, specifically on methane emitting Category Train (%) Valid (%) Test (%) Total infrastructure, which prohibits researchers and practi- CAFOs 24957 (29.3%) 47 (9.1%) 92 (9.0%) 25096 tioners from building automated mapping approaches. Landfills 4085 (4.8%) 46 (8.9%) 111 (10.9%) 4242 vatIinonthidsawtaosrekt,fwore mcoentshtarnuectsaoumrucletii-nsefrnassotrruEcatruthreoibdseenr-- CProoaRcl&PMTlaisnnetss 114970071062 (((224...217%%%))) 534980(((1771..48.5%%%))) 11070728 (((171.001..56%%%) )) 214081487589 tification called METER-ML. In support of a new ini- WWTPs 14519 (17.1%) 46 (8.9%) 129 (12.7%) 14694 tiative to build a global database of methane emitting Negatives 34195 (40.2%) 249 (48.3%) 426 (41.8%) 34870 infrastructure called the MEthane Tracking Emissions Total 85066 515 1018 86599 Reference (METER) [12], we develop METER-ML to allow the machine learning community to experiment with multi-view/multi-modal modeling approaches to auto- visible diferentiating features which make them feasible matically identify this infrastructure in remotely sensed to identify in high resolution remotely sensed imagery. imagery. METER-ML includes georeferenced imagery The locations are obtained from 18 diferent publicly from three remotely sensed image products, specifically available datasets, all of which have licenses that allow for 19 spectral bands in total from NAIP, Sentinel-1, and redistribution (see Table 6 in the Appendix). As various Sentinel-2, capturing 51,729 sources of methane from six datasets may contain the same locations of infrastructure, diferent classes as well as 34,870 negative examples (Fig- we deduplicate by considering locations within 500m of ure 1). The dataset includes expert-reviewed validation each other identical. In total we include 51,729 unique and test sets for robustly evaluating the performance of locations of methane source infrastructure in the dataset, derived models. Using the dataset, we experiment with a which we refer to as positive examples. variety of convolutional neural network models which leverage diferent spatial resolutions, spatial footprints, image products, and spectral bands. The dataset is freely 2.2. Negative locations available1 in order to encourage further work on developing and validating methane source mapping approaches.
which capture none of the six methane emitting facilities.
To do this, we define around 50 classes (see Appendix) 2. Methods of diferent facilities and landscapes and select characteristic examples of each class. Then we collect locations containing similar facilities and landscapes using the 2.1. Methane source locations Descartes Labs GeoVisual Search [13], providing up to We collect locations of methane emitting infrastructure 1000 similar locations per example. A sample of the simiin the U.S. from a variety of public datasets. We focus on lar locations were manually vetted in each case to ensure the U.S. in this study due to the high availability of pub- no locations obtained actually corresponded to the six licly accessible infrastructure data and remotely sensed methane source categories. In total we include 34,870 imagery. The infrastructure categories we include are locations of facilities and landscapes which are not any concentrated animal feeding operations (CAFOs), coal of the six infrastructure categories, and refer to these as mines (Mines), landfills (Landfills), natural gas processing negative examples. The counts and proportions of the plants (Proc Plants), oil refineries and petroleum termi- positive and negative classes in the dataset are shown in nals (including crude oil and liquified natural gas termi- Table 1. nals), and wastewater treatment plants (WWTPs). We group oil refineries and petroleum terminals together 2.3. Remotely sensed imagery due to their high similarity in appearance, and refer to that category as “Refineries & Terminals” (R&Ts). These We pair all of the locations in the dataset with three infrastructure categories were chosen based on their po- publicly available remotely sensed image sources. Speciftential for emitting methane along with their consistent, ically we include aerial imagery from the USDA National Agriculture Imagery Program (NAIP) as well as satellite 1https://stanfordmlgroup.github.io/projects/meter-ml imagery captured by Sentinel-1 (S1) and Sentinel-2 (S2).
imagery both have global coverage. For NAIP we use 1m resolution imagery, for Sentinel-2 we use the L1C product at 10m resolution, and for Sentinel-1 we use the Sigma Nought Backscatter product at 10m resolution. We use all spectral bands from each product. Specifically, we use the three visible (RGB) and single near-infrared (NIR) bands from NAIP and S2, the single coastal aerosol (CA) band, four red-edge (RE1-4) bands, single water vapor (WV) band, single cirrus (C) band, and the two shortwave infrared (SWIR1-2) bands from S2, and the V-transmit (VH and VV) bands from S1. We include S1 and S2 in the dataset in order to enable experimenting with coarser resolution satellite imagery which is globally available, unlike NAIP. The details of each imagery product and band are shown in Table 2.
In order to construct images containing each location in the dataset, we consider a 720m x 720m footprint centered around the location. This footprint was chosen to balance the size of the images with the contextual information, but we investigate this choice in the experiments. Due to the geographic coordinate noise in the publicly available datasets, we chose to center the imagery at the locations which increases the likelihood the facilities are captured in the imagery, but still has natural variation in the locations of the facilities in the imagery. We construct a mosaic of the most recently captured pixels in a time range for each image product, where we consider NAIP images captured between 2017 and 2021 and Sentinel1 and Sentinel-2 images between May and September 2021, where Sentinel-2 images are selected based on lowest cloud cover. We use the Descartes Labs platform to download all of the imagery [14].
The total dataset contains 86,599 images capturing ten spectral bands across the three imagery products. Information about the remotely sensed image products and bands included in the dataset are provided in Table 2 and characteristic examples for each methane source category are shown in Figure 2 in the Appendix. Two Stanford University postdoctoral researchers with expertise in methane emissions and related infrastructure individually reviewed 1,533 examples to compose the held-out validation and test sets. To determine which examples to include in these held-out sets, we randomly sampled 150 images from each of the six positive classes as well as a random sample of 33 images which have multiple labels, constituting 933 positive examples according to the original public dataset labels. We additionally sampled 12 images from each of the 50 negative categories resulting in 600 negative examples. The experts both manually reviewed these examples and identified the presence or absence of the six methane source categories by using a combination of NAIP imagery as well as Google Maps imagery, which often had finer spatial resolution as well as place names. The facility had to be captured by the NAIP image for the corresponding label to be assigned. If the expert identified no clearly visible methane source categories in the image, the example was labeled “negative”, and if the expert was uncertain about any label, the example was labeled “uncertain”. The two labels per example were then resolved as follows: 1. If the experts agreed and neither was uncertain, the agreed upon label was taken as the final label. 2. If the experts disagreed, and one was uncertain but the other was not, the expert’s certain label was taken as the final label. 3. If the experts disagreed, but one agreed with the original label, the original label was taken as the ifnal label. 4. In all other scenarios, the example was reviewed jointly by the experts and a final label was assigned.
Only 76 examples out of the 1,533 went to another round of review. The resulting datasets have 858 positive examples and 675 negative examples. We split the 1,533 examples into 515 for the validation set and 1,018 for the test set. The label counts on the validation and test sets are shown in Table 1.
on the curated dataset. In all of our experiments, we use a DenseNet-121 convolutional neural network architecture [15]. Preliminary experiments on the dataset explored various ResNet and DenseNet architectures and found that DenseNets outperformed all ResNet variants [16].
We use a linear layer which outputs six values indicating the likelihood that each of the six methane source categories are present in the input image, which outper- all spectral bands from S1 and S2 together (representing formed individual models across all classes in our prelimi- the model closest to public global transferability due to nary experiments. Although the model does not explicitly the global coverage of S1 and S2), and all spectral bands produce a value indicating the likelihood that the image from the three products together. is negative, a low value assigned to all classes indicates a The best model according to macro-average AUPRC negative prediction. The loss function is the mean of six is the one which uses NAIP with all bands (the unweighted binary cross entropy losses, where the label three visible bands and NIR band), achieving an overis 1 if the class if present in the image and 0 otherwise. all AUPRC of 0.548 and the highest performance on All six labels in the negative examples are 0. The network CAFOs, Landfills, Proc Plants, R&Ts, and WWTPs comweights are initialized with weights from a network pre- pared to all other tested product and band combinatrained on ImageNet [17]. Before inputting the images tions. Notably, it achieves very high performance on into the networks, we upscale the Sentinel-1 and Sentinel- CAFOs (AUPRC=0.945) and high performance on R&Ts 2 images to match the size of NAIP images using bilinear (AUPRC=0.857). The second best model is the joint resampling and normalize the values by the display range NAIP+S2+S1 model, achieving an overall AUPRC of 0.517 of the bands (see Table 7 in the Appendix). When using and the highest performance on Mines (AUPRC=0.473) inputs with less than or more than 3 channels, we re- compared to all other tested product and band combinaplace the first convolutional neural network layer with tions. one which accepts the corresponding number of chan- S1 alone underperforms all other combinations of prodnels. Each model is trained for 5 epochs with a batch size ucts and bands, followed by S2 and S1 jointly, which of 4. For each model we use the checkpoint saved after performed similarly overall to S2 with only the visian epoch which led to the lowest validation loss. We use ble bands and all spectral bands. Importantly, the S2 an Adam optimizer with standard parameters [18] and and S1 joint model still achieves high performance on a learning rate of 0.02. All models are trained using a CAFOs (AUPRC=0.923), although the performance is GeForce GTX 1070 GPU. lower than performance on CAFOs using NAIP imagery
The baseline setting for all experiments uses images (AUPRC=0.947). There is a significant drop in perforcapturing a footprint of 720m x 720m with 1m spatial res- mance on all classes when moving from NAIP to S2, olution (720 x 720 image dimensions). After the models highlighting the benefit of using high spatial resolution are trained, each of the six values output by the model are imagery. fed through an element-wise sigmoid function to produce The inclusion of the non-visible information substana probability for each of the six categories. To evaluate tially improves overall AUPRC for NAIP (AUPRC=0.480 the performance of the models, we compute the per-class → 0.548) but not for Sentinel 2 (AUPRC=0.450 → 0.448). area under the precision recall curve (AUPRC) and sum- For NAIP, the improvement is observed for all classes, marize the performance over all classes by taking the with substantial gains on CAFOs, Mines, Proc Plants, macro-average of the per-class AUPRCs. and WWTPs. For Sentinel 2, the inclusion of non-visible bands substantially improves performance on Mines but 3.1. Impact of using diferent imaging substantially degrades performance on Landfills. For both products, minimal change on R&Ts performance is products and bands observed when including the non-visible bands.
treatment plants are surrounded by industrial buildings and other infrastructure, so cropping out this infrastructure improves the model’s ability to identify the salient features of the wastewater treatment facilities.
We further find that the highest spatial resolution achieves the best overall performance (AUPRC=0.548), outperforming the coarser resolution models on CAFOs, Landfills, and R&Ts. The 1.5m resolution model closely follows with an overall AUPRC of 0.541 and outperforms the 1m resolution model on Mines. The 3m resolution model also closely follows the 1.5m resolution model achieving an overall performance of 0.531, and substantially outperforms both the higher resolution models on Proc Plants. This result suggests that models developed at 1.5m and even 3m resolution have the potential to perform almost as well as 1m resolution models, which has implications on global applicability as Airbus SPOT and PlanetScope are globally (privately) available at 1.5m and 3m resolution respectively. As image footprint (i.e. the amount of area on the ground captured by the image) and spatial resolution likely impact model performance due to the variation in the sizes 3.3. Per-class expert model test set results of the methane-emitting facilities and equipment, we conduct experiments to test these efects (Table 4). To For each methane source category, we select the exinvestigate the impact of footprint, we center crop the perimental configuration (product/band/footprint/reso720 x 720 1m images to obtain 480 x 480 and 240 x 240 lution) that achieved the highest validation AUPRC for 1m images corresponding to 480m x 480m and 240m x that class to serve as the “class expert”. We refer to the 240m footprints respectively. Note that this reduces the combination of the diferent class experts as the per-class area on the ground with spatial resolution held constant. expert model.
To investigate the impact of spatial resolution, we use We evaluate the per-class expert model on the hold-out cubic spline interpolation [19] to downsample the 720 x test set using a variety of metrics including AUPRC and 720 images to 480 x 480 (1.5m resolution, corresponding area under the receiver operating characteristic curve to Airbus SPOT imagery) and 240 x 240 (3m resolution, (AUROCC) as well as precision, recall, and F1 at the corresponding to PlanetScope imagery). Note that this re- threshold which achieves the highest F1 on the validaduces the spatial resolution without modifying the image tion set. The results are shown in Table 5. The per-class footprint. In all experiments, we up-sample the images expert model obtains a macro-average AUPRC of 0.558. back to 720 x 720 to avoid any diferences in performance The model does especially well on CAFOs (AUPRC=0.915) due to varying image size. We use NAIP with RGB + NIR and R&Ts (AUPRC=0.821), possibly because these sources bands for these experiments as this setting produced the have very distinctive features (e.g., long barns in CAFOs best overall performance compared to the other combi- and storage tank farms in R&Ts). It performs more poorly nations of products and bands. on the other sources, especially landfills which do not
We find that the largest tested image footprint achieves have many clear distinctive features visible at 1m resothe highest overall performance (0.548) and substantially lution. Notably it achieves the lowest performance on outperforms both smaller spatial footprints across all the categories with the least number of examples in the classes except for WWTPs. This may be explained by dataset, excluding R&Ts which may be simpler to identify the fact that a significant number of smaller wastewater due to their homogeneity and discernible features.
dinal information has the potential to provide additional signal to help diferentiate certain facilities, e.g. waste The experiments suggest that the choice of imaging prod- pile evolution at landfills. Third, we use a DenseNet121 uct, spectral band, image footprint, and spatial resolution model that is pre-trained on ImageNet, but the shape and can lead to substantial diferences in model performance, number of channels of remote sensing imagery can be with the efect often depending on the methane source significantly diferent from ImageNet. It would be worthcategory. In particular, this suggests that there is sig- while to train a network from scratch on METER-ML, nificant room to explore approaches which leverage the and compare its performance against a network that is multi-sensor and multi-spectral aspects of METER-ML. pre-trained on ImageNet and fine-tuned on METER-ML. For example, the NAIP & S2 & S1 model underperformed Fourth, our approach to combine the multi-sensor data the model which used NAIP alone, and using all 13 spec- may not be optimal as the products and spectral bands tral bands in the S2 model did not lead to substantial per- have diferent spatial resolutions and sensor types (e.g. formance diferences compared to the S2 model which active vs. passive sensors). One alternate approach may only used the three visible bands. We also do not leverage be to dedicate diferent network branches for the inputs the geographic information explicitly in the models, but and combine the representations from each branch. this has been shown to improve performance on other Earth observation tasks [20, 21]. Furthermore, there is potential to augment the dataset with other sources of 5. Conclusion imagery and information available at the provided geographic locations. We hope to help create new versions In this work, we curate a large georeferenced multiof METER-ML which may include other sources of input sensor dataset called METER-ML to test automated data and methane emitting infrastructure categories. methane source identification approaches. We conduct
The best model from our experiments achieves high a variety of experiments investigating the impact of reperformance on identifying CAFOs and R&Ts, suggesting motely sensed image product, spectral bands, image footthe potential to map these facilities with NAIP imagery print, and spatial resolution on model performance meain the U.S. which aligns with findings from prior studies sured against a consensus of expert labels. We find that a [9, 10]. The performance for identifying CAFOs remains model which leverages NAIP with all four bands achieves high when using S1 and S2, which are globally and pub- the highest overall performance across the tested image licly available. This suggests the potential to use these product and spectral band combinations, followed closely lower spatial resolution imagery sources to map CAFOs by a joint NAIP, Sentinel-2, and Sentinel-1 model. We also in other countries besides the U.S., but future work should ifnd that the highest spatial resolution and footprint leads investigate whether these findings generalize to other to the best overall performance, although performance regions. There is still a large gap to achieving high per- can depend on the methane source category. Finally we formance for each of the other methane source categories show that the best model achieves high performance in and further improve performance on the high performing identifying concentrated animal feeding operations and categories, so METER-ML is a challenging benchmark to oil refineries and petroleum terminals, suggesting the test new infrastructure identification approaches. potential to map them at scale, but substantially lower
There are many other publicly available remote sens- performance on the other four categories with notably ing datasets for classification, with some of the most com- lower performance identifying processing plants and mon being UC Merced [22], SAT-4 and SAT-6 [23], AID landfills. We make METER-ML freely available in or[24], NWPU-RESISC45 [25], EuroSAT [26], and BigEarth- der to encourage and support future work on developing Net [27]. Few of these datasets have georeferenced multi- Earth observation models for mitigating climate change. sensor images, which limits their utility for new modeling approaches and downstream use. The OGNet dataset [9] Acknowledgments is the most similar publicly available dataset to METERML and is essentially a subset of it, containing NAIP This work was supported by the High Tide Foundation imagery of refineries in the contiguous U.S. to construct the METER database. We acknowledge Rose
We identify four limitations of this work. First, we Rustowicz and Kyle Story for their support of this work, limit the geographic scope of METER-ML to the U.S. due as well as the Descartes Labs Platform API and tools to the availability of disseminatable infrastructure data for downloading and processing the remotely sensed imand publicly available, high resolution imagery. Future agery. We also thank Ritesh Gautam and Mark Omara work should include data in other regions worldwide. for their help working with the oil and gas infrastrucSecond, we do not include longitudinal imagery in the ture data, Evan Sherwin for his advice on the dataset and dataset to reduce the size and complexity of the dataset as methane source categories, and Victor Maus for providmost infrastructure is static over time. However, longitu- ing the coal mines data. tion, IEEE Transactions on Geoscience and Remote 2020. URL: https://maps.indiana.edu/metadata/ Sensing 55 (2017) 3965–3981. Environment/Agribusiness_Confined_Feeding_ [25] G. Cheng, J. Han, X. Lu, Remote sensing image Operations.html#Identification_Information, scene classification: Benchmark and state of the art, accessed: 2022-02-01.
Proceedings of the IEEE 105 (2017) 1865–1883. [38] U.S. Environmental Protection Agency,
[26] P. Helber, B. Bischke, A. Dengel, D. Borth, Eurosat: Landfill methane outreach program,
A novel dataset and dee
and Remote Sensing 12 (2019) 2217–2226. [39] A. J. Marchese, T. L. Vaughn, D. J. Zimmerle, D. M. [27] G. Sumbul, M. Charfuelan, B. Demir, V. Markl, Martinez, L. L. Williams, A. L. Robinson, A. L.
Bigearthnet: A large-scale benchmark archive for Mitchell, R. Subramanian, D. S. Tkacik, J. R. Roscioli,
remote sensing image understanding, in: IGARSS et al., Methane emissions fro
02-01. [41] Metropolitan Council, Wastewater treatment
[29] CA State Water Resources Control Board, 2022. plants, 2020. URL: https://gisdata.mn.gov/dataset/
URL: https://www.waterboards.ca.gov/resources/ us-mn-state-metc-util-wastewater-treatment-plants,
data_databases/site_ma
01. [42] MN Pollution Control Agency, Feedlots in
min[30] D. McKernan, 2012 factory farms in michi- nesota, 2016. URL: https://gisdata.mn.gov/dataset/
gan, 2012. URL: https://data.world/dmckernan/ env-feedlots, accessed: 2022-02-01.
2012-factory-farms-in-michigan, accessed: 2022- [43] ORNL DAAC, Sources of methane emissions
02-01. (vista-ca), state of california, usa, 2019. URL:
[31] O. Tsubiks, Concentrated animal https://daac.ornl.gov/NACP/guides/NACP_Vista_
feeding operations (cafo), us, 2017. CA_CH4_Inventory.html, accessed: 2022-02-01.
URL: https://data.world/dataforacause/ [44] Sierra Club Michigan Chapter, Michigan cafo
mapconcentrated-animal-feeding-operations-cafo, ping report, 2017. URL: htt
ergy mapping system, 2021. URL: https://www.eia. [45] Stanford Regulation, Evaluation, and Governance
gov/state/ma
J. DiGiulio, K. Jones, T. Jones, D. Justman, R. Miller, et al., Global Oil & Gas Features Database, Technical
Report, 2018.
[35] U.S. D of Homeland Security, Homeland
infrastructure foundation-level data (hifld), 2021. URL:
https://hifld-geoplatform.o
A. Limtong, R. Shakya, Distribution and characteristics of wastewater treatment plants within the global river network, Earth System Science Data 14 (2022) 559–577. [37] Indiana D of Environmental Management,
Confined feeding operation facilities: Indiana,
A.1.1. Coal Mines Data The mines data from [40] were subsetted to coal mines in order to capture the mines responsible for the vast majority of methane emissions. To do this, the polygons and coal mine coordinates obtained from S&P Global Commodity Insights were matched to determine which polygons were spatially related to coal mine coordinates. Then a visual check and hand cleaning was performed on the polygons assigned a coal mining label to ensure correctness.
We identify a variety of infrastructure which are not any of the six infrastructure categories to use as negatives in the dataset. Specifically we include football fields, marinas, solar panels, large bodies of water, parking lots, windmills, baseball fields, airport runways, clouds, neighborhoods, golf courses, roundabouts, mountainous terrain, trees, boats, islands, rocks, rivers, roads, bridges, ripples in water, snow, canyon formations, sparse forests, suburban neighborhoods, beaches, clear water, swimming pools, sand, corn farms, soy farms, trees on mountainside, farm houses, grass, airplanes, turning roads, intersections, multifamily residential facilities, rapids, docks, highway loops, mowed grass, container yards, soccer fields, greenhouses, crops, personal watercrafts, pivot irrigation systems, and concrete plants. Characteristic examples of each type were selected and a variety of similar examples per type were obtained using the Descartes Labs GeoVisual Similarity tool [13].