This work uses a big data set platform, AgriSenze™, to predict daily soil temperatures in Norway. The big dataset is primarily collected from the Norwegian University of Life Sciences (NMBU) to build an ensemble stacking regressor machine learning algorithm to predict daily soil temperatures at six diferent depths (2 cm, 5 cm, 10 cm, 20 cm, 50 cm, and 100 cm). This study has successfully developed a model-based soil temperature prediction with performance results comparable to most existing research and even better performance at some soil depths Modern agriculture faces two primary challenges: meeting rising food demand while simultaneously addressing growing environmental concerns. Smart Farming (SF) emerges as a potential solution to tackle these dual challenges in modern agricultural practices [1]. SF is a technological innovation that utilizes information and communication technology to integrate computing with the physical farming processes [2]. For the SF to make precise decisions and provide accurate information to farmers, the SF needs to understand the complex environmental ecosystems [1] which requires collecting various environmental data such as soil, weather, crop, fertilizer, farming practices, supply-chain and market analysis data. Precision agriculture (PA), a component of the SF process, employs specialized sensors and algorithms to ensure crops receive precisely what they need to maximize yield and promote sustainability [3]. PA involves gathering data on weather patterns, soil conditions, and crop data through Internet of Things sensors installed in the fields or remotely interconnected with the help of Wireless Sensor Networks. With PA, a farmer may determine which parameters are required for a healthy crop, when, and how much are required at any given time. The agricultural parameters that impact the production yield and sustainability should be collected from diferent sources and analysed by the PA system to produce agronomic recommendations [3].
LGOBE (V. G. Sanchez) https://www.usn.no/english/about/contact-us/employees/veralia-gabriela-sanchez (V. G. Sanchez)
CEUR
ceur-ws.org 1.1. Aim This work aims to use the agricultural data from AgriSenze™ platform and apply artificial intelligence to predict daily soil temperatures at six diferent depths (2 cm, 5 cm, 10 cm, 20 cm, 50 cm, and 100 cm).
The desire for using artificial fertilizers to increase plant yield has grown since they were first introduced.
However, this heightened consumption has led to concerns about overfertilization, and its environmental
impacts attributed to nitrogen and phosphorus, prompting the implementation of fertilization policies
since the 1970s to regulate the amount and application rates [
Overfertilization can result in runof, causing environmental issues such as greenhouse gas emissions,
eutrophication, and groundwater contamination. Conversely, under-fertilization can lead to decreased
plant yields and nutrient deficiencies.In addition to the meteorological factors, various soil parameters
such as soil temperature, soil moisture, soil pH, soil organic matter, cation exchange capacity, etc can
impact the soil nitrate-nitrogen concentration. Depending on the soil depth, soil temperature impacts
the dissolved organic matter, humification and nitrate concentration [
There was a limited number of research studies that employed data-driven approaches to predict soil
temperature. Elsayed S. et al. [
Kisi O. et al. [11] compared models including multi-layer perceptron (MLP), radial basis neural networks (RBNN), generalized regression neural networks (GRNN), and multiple linear regression (MLR) to predict monthly soil temperatures at depths of 5 cm, 10 cm, 50 cm, and 100 cm at Mersin Station, Turkey. Other works have used ANN such as Taheri M. et al. [12]. Zare Abyaneh H. et al [13] developed an ANN and Co-active neuro-fuzzy inference system (CANFIS). Farhangmehr V. et al. [14], have manipulated CNN model to predict daily maximum soil temperature under normal, hot and cold weather conditions at Ottawa area of Canada (45.250 N latitude, 75.500 W longitude). Bilgili M. et al. [15] evaluated ANFIS network with fuzzy c-means (ANFIS-FCM), subtractive clustering (ANFIS-SC), feed-forward neural network (FNN), grid partition (ANFIS-GP), Elman neural network (ENN), and LSTM models for predicting one-day ahead soil temperature at three depths (5, 50 and 100 cm) from the previously measured time-series daily soil temperature data from 2010-2020 in the Central Anatolia Region of Turkey. Ebtehaj Isa, et al.[16] have demonstrated emotional neural network using both meteorological variable and time-series based modelling to predict daily soil temperature at diferent depths (10 and 20 cm) for Springfield and Champaign stations, in Illinois, USA.
Xing L. et al. [17] have proposed an SVM and combined SVM models to predict annual average and daily average soil temperatures at 5, 10, 20, 50 and 100cm using air temperature, rainfall, wind speed, solar radiation and relative humidity input data collected for 130 years from 16 diferent sites located in the USA with dry, warm and snowy climates. Wang X. et al. [18] investigated an integrated network prediction model utilizing gated recurrent unit (GRU) to predict soil temperature at varying depths (5, 10, and 15 cm) and diferent time intervals (6, 12, and 24 hours). They utilized time-series soil temperature data gathered from two meteorological stations (Laegern and Fluehli) in Switzerland spanning from 2006 to 2014.
The range of machine learning models proposed by various researchers suggest that the best performing model is highly contingent upon the specific dataset and climatic region under consideration. Therefore, there is no single soil temperature prediction model that has been universally adopted, not even at the country level. Besides, it becomes even more dificult when developing models for cold climates at national level. Imanian H. et al.[19] recommended that machine learning models should consider the cold climate’s unique features, such as snow depth or other during exceptionally cold conditions, for better prediction accuracy.
Therefore, this study contributes to the research in soil temperature prediction for agriculture in cold climates, specifically using data available from Norway.
The data used in this study is gathered and pre-processed with help of a web API platform integrated to the Zimmer and Peacock (ZP) AgriSenze™ solution. The data was collected manually from NMBU and MET Norway using the data-gathering web API. The web API platform is integrated to the AgriSenze solution. The Meteorological data for ÅS - BIOKLIM collected by NMBU from January 01, 2000, to April 01, 2024, was used as the basis for building the agricultural dataset in this research.
ÅS - BIOKLIM research site is located on Søråsjordet in Ås i Viken weather station with geographic coordinates 59 39’ 37’’ N, 10 46’ 54’’ E, 93.3 m above sea level. The NMBU’s data collection process is discussed in detail in the oficial website and scientific report [ 20, 21]. This dataset was used to build a machine learning model to predict the daily soil temperatures at 2cm, 5cm, 10cm, 20cm, 50cm and 100cm soil depths. The data contains diferent meteorological parameters recorded on a daily basis, including date, minimum temperature (°C), maximum temperature (°C), mean temperature (°C), relative humidity (%), precipitation (mm), evaporation (mm), daily air pressure (mbar), wind speed (m/s), wind direction (degrees), snow depth (cm), global radiation (W/m²), reflected radiation (W/m²), radiation balance (W/m²), difuse radiation (W/m²), photosynthetic active radiation – PAR (mE/m²) and soil temperature at various depths (2cm, 5cm, 10cm, 20cm, 50cm, 100cm in °C).
The data parameters pertaining to the nitrate-nitrogen sensors of AgriSenze™ are obtained through the DjuliTM IoT cloud service API developed by Zimmer and Peacock AS. Although this API provides nitrate concentration data, it has not yet been deployed on the ÅS - BIOKLIM field station; hence, the data gathered is not specific to this location.
The Jupyter Notebook and Google Colab were used for data processing and analytics. The AgriSenze™ dataset was loaded from CSV file and analysed for duplicates, missing values, anomalies, and nonnumeric values. The dataset records are organized on daily values for the diferent parameters and duplicate date records were removed. Features from the original dataset that contained numerous missing values and were deemed to have minimal impact on daily soil temperature were removed from the dataset. Also, in the original snow depth data, there were instances where both manually recorded and automatically measured values were available for the same dates. In these cases, the manually recorded values were selected for their better reliability.
There were a large number of missing values for the snow and evaporation in the winter season (November to March). To input the missing values, time series data analysis was used to determine whether to impute the missing values or exclude the parameter from the dataset. Since 8 years of data was missing, the mean the average value for the same day across multiple years within the same month was calculated and inputted, providing a more representative estimate based on historical observations for that specific day and month.
Following this process, outlier were identified and removed using Z-score normalization. Then, Isolation Forest algorithm was used to identify anomalous data points. Finally, correlation analysis was used to identify highly correlated features aiming to avoid multicollinearity efect on the models. mean air temperature, min air temperature, max air temperature, relative humidity, air pressure, precipitation mm, evaporation mm, earth heat flux, radiation balance, photosynthetic active radiation, albedo, snow depth, ST2, ST5, ST10, ST20, ST50, month, day date, mean air temperature, min air temperature, max air temperature, relative humidity, air pressure, precipitation, evaporation, earth heat flux, radiation balance, photosynthetic active radiation, albedo, snow depth, ST2, ST5, ST10, ST20, ST50
The initial step in developing the machine learning model involves shufling and splitting the preprocessed dataset into training, validation, and test subsets. The dataset with 8252 sample size after Z-score outlier filtering was divided into 70% training, 15% validation, and 15% test sets. This 70/30 splitting proportion proved to be more performant compared to the 80% for training and 20% for testing considered as common split as discussed in Géron A. [22]. The shufling process ensures that each subset (training, validation, and test) is representative of the overall dataset. This preventive measure mitigates potential biases that might arise from the original order of the data, ultimately leading to reliable cross-validation results in subsequent steps [22].
The next step involves organizing the features, targets, and evaluation metrics to be utilized for the daily soil temperature prediction model. The targets are the daily soil temperatures at various depths, specifically 2 cm, 5 cm, 10 cm, 20 cm, 50 cm, and 100 cm. The features were initially used as independent input variables, a correlation and feature importance analysis were subsequently conducted to identify the most influential features for each target variable. Since the problem requires a regression model to predict daily soil temperature, the main evaluation metrics adopted for the prediction model includes the coeficient of determination (R²), mean absolute error (MAE), mean squared error (MSE) and root mean squared error (RMSE).
The regression models tested and compared consist of Stacking Regressor, CatBoost Regressor, HistGradientBoosting Regressor, RandomForest Regressor, AdaBoost Regressor, Ridge regressor, Support Vector Regression with both linear and non-linear kernels, Lasso Regressor, and ElasticNet Regressor. This comprehensive set of models were manipulated for a comparative analysis to identify the most suitable algorithm for the daily soil temperature prediction task. The models were fitted with the best params. Then cross-validation was performed followed by residual analysis. Residuals analysis was used as a valuable tool to evaluate the model’s fit to the data. Finally, feature importance and backward feature elimination technique were employed.
There are fourteen basic features (predictor variables) and six target (soil temperature) variables. To generate an optimized dataset for predicting soil temperature at six diferent depths, various preprocessing methods were employed. Table 1 summarizes the two main cases and the resulting datasets produced after applying the preprocessing steps. The dataset cases have normalized and denormalized versions and the performance of the predictions for each dataset case was evaluated. The dataset cases are generally categorized into time-independent shufled meteorological datasets (Case 1) and time-series meteorological datasets (Case 6). The prediction performance analysis was done for the all datasets cases to identify which dataset case fits best for the models. Other dataset cases were also tested and found less performative. Among the dataset cases, Case 1 and Case 6 were identified as having superior performance.
ST2 ST5 ST10 ST20 ST50 ST100
Z-score outlier filtered dataset (Case 1) results: Upon comprehensive performance evaluation of the time-independent datasets, the Z-score outlier filtered dataset (Case 1) emerged as the top performer out of the time-independent dataset cases. R 2, MAE and RMSE metrics were used as evaluation metrics. For a best model, the MAE and RMSE should be ideally the same. The performance results of the four base models (RF-R, XGB-R, CATB-R, HGB-R) and the meta-model (STACK-R) for the Case 1 dataset are listed in Table 2 for all targets.
Time-series dataset (Case 6) results: The dataset was divided into training, validation, and test sets sequentially in proportions of 70%, 15%, and 15%, respectively. As presented in Table 3, the performance results from the 10-Fold cross-validation (CV), 5-Fold CV, and test set evaluation were within a comparable range, except for ST50. In the case of ST50, there was a notable diference between the 10-Fold CV and 5-Fold CV RMSE values. Similarly, the test RMSE value significantly deviates from the CV RMSE value. For predicting soil temperatures at depths beyond 2 cm, historical soil temperatures data was used alongside the meteorological features.
Observed versus predicted values analysis: Scatter plots were employed to visualize the models’ performance by comparing observed values against predicted values, examining the outliers between these two sets for each target variable. The scatter plot of the observed values against the STACK-R predicted values for soil temperatures at 2 cm, 5 cm, 10 cm, 20 cm and 100 cm depths are depicted in 0.9956 0.9957 0.9962 0.9961 0.9964 0.9994 0.9991 0.9983 0.9991 0.9993 0.9991 0.9988 0.9983 0.9989 0.9990 0.9979 0.9974 0.9962 0.9977 0.9978 0.9651 0.9580 0.9708 0.9751 0.9750 0.9992 0.9992 0.9964 0.9992 0.9992
The data from AgriSenze™ solution was manipulated and valuable information was extracted to enhance decision-making. In this study, soil temperature was the primary focus of data analytics due to its major influence on plant growth and the available rich data source collected from NMBU. The soil temperature data analytics involved model-driven soil temperature predictions at six diferent depths: 2 cm, 5 cm, 10 cm, 20 cm, 50 cm, and 100 cm. The soil temperature prediction modelling process tested approximately 10 diferent models, including linear models (Lasso, ElasticNet, RidgeCV), non-linear model (SVR), tree-based models (RF-R, XGB-R, CATB-R, HGB-R), and meta-estimator (STACK-R). The study did not discuss the linear and SVR models in detail because in the preliminary testing they were found to be comparatively less eficient for the specific dataset used in this study.
According to the diferent pre-processing stages, various dataset cases were generated as shown previously in Table 1, and based on the preliminary assessment the candidate models (RF-R, XGBR, CATB-R, HGB-R, STACK-R) were tested for each case. Case 1 proved to be superior to all other pre-processed datasets except the Case 6, time-series dataset which may need further validation.
For the time-independent modelling, STACK-R demonstrated superior performance across all evaluation phases. This is because the stacking regressor leverages the strengths of the base models by learning from their performance. The other four base models (RF-R, CATB-R, XGB-R, HGB-R) have nearly similar performance except the RF-R has higher errors than the other models for almost all targets 2. The cause of the minimum error occurrence in diferent split sets is due to the nature of the data in the split sets. The prediction error for ST2, soil temperature at 2 cm, (RMSE = 0.9286) is higher compared to other soil depths. This is because the model for ST2 relies solely on meteorological variables, while the model for ST5 also incorporates the highly correlated soil temperature data at 2 cm. Similarly, the model ST10 considers the temperatures at the two shallower depths above it. This pattern continues for all higher depth soil temperature predictions.
The prediction accuracies were checked by dropping the historical soil temperature features from dataset resulting in weaker performance, for example for target ST5, when ST2 feature is dropped out, the prediction error increased from RMSE of 0.1286 to 0.9334. This is close to the prediction error of ST2 which was solely predicted from meteorological parameters only. The study by Bilgili M. et al. [15] supports this result that incorporating historical soil temperature data in the input helps to increase the prediction accuracy.
The model for ST2 is robust for outliers and has good generalization capabilities across wide range of data points according to the residual analysis- While the most stable prediction was achieved for ST50 with mean RMSE of 0.4015 and standard deviation of 2.6% between the cross-validation errors, the most accurate prediction was achieved for ST5 with mean RMSE of 0.1234 and standard deviation of 4%. Models can have good generalization capabilities across their wider range of data points but less stable (has higher variance in its prediction error) and less accurate (has higher mean prediction error) in the overall prediction error.
The time-dependent modelling with the time-series dataset (Case 6) exhibited a significant improvement in performance for some of the targets compared to the time-independent modelling. However, further validation is necessary to ensure that the prediction is consistent for any time-series data. While an exhaustive validation could not be performed due to time constraints, very promising and interesting results were observed. The modelling results are illustrated in Table 3. With the exception of the targets ST5 and ST50, all other targets demonstrated relative improvements in performance. The most crucial improvement in this time-series modelling was observed for ST2 with almost 50% decrease in RMSE, as all other soil temperatures depend on the impact of ST2 either directly or indirectly. This represents a promising result that warrants further consideration in the future work, involving more extensive validation and analysis on time-series modelling with more advanced deep learning models. This validation is highly required because a recent study by Imanian H. et al. [19] had suggested that time-series machine learning models face challenges in accurately predicting soil temperature especially for cold climates. The evaluation of the time-series modelling on the ST50 test set yielded somewhat strange results, necessitating further validation to determine whether the test set data is causing the extreme change in RMSE or if the model is not robust for all types of time-series data. This aspect will be considered in future work.
The other essential analysis done was the feature importances and backward feature elimination process. The prediction of soil temperature at 2 cm is fully dependent on the meteorological parameters and air temperature is the most crucial parameter with overall weight 56.61% (mean + max + min air temperatures) followed by the month, evaporation and snow depth at 14.68%, 9.68% and 3.74% respectively. The backward feature elimination showed that the ST2 prediction performance remained relatively similar regardless of whether evaporation and snow depth data were included or excluded in the Case 1 dataset. This could be due to some reasons: 1) it could be because majority of the snow depth data was missing in the original dataset, hinting that the impact of snow depth could be more pronounced if the actual snow depth data were accurately measured, 2) it could be because in cold climates the topsoil surface temperature is highly influenced by ice layer temperature during the winter season and that information is not well captured in snow depth dataset, 3) the impact of snow depth on topsoil temperature is relatively negligible. Similarly, the efect of evaporation, which was filled
by using the daily average data across all the years and zero for the cold season, could be increased if the actual measured data was accurate in the dataset. The feature importance analysis revealed that mean air temperature has a major influence on predicting soil temperatures at shallower depths, and its influence reduces as the soil depth increases. Conversely, precipitation had the least impact on soil temperature predictions across all depths. The feature sensitivity analysis conducted by Imanian H. et al. [19], on soil temperature predictions, for cold climates at 0-7cm depth strengthened similar ifndings. The evaporation, earth heat flux and radiation have considerable impact on the prediction of deeper soil depths, supporting the assertions by Farhangmehr V. et al. [14] that features should not be disregarded without a thorough analysis of feature importance if prediction accuracy is the primary concern. The month and mean air temperature emerged as significant features for most target variables. The influence of the month feature increased as the soil depth increased, potentially due to the soil temperature at deeper depths taking longer to change. As a result, monthly variations in meteorological parameters are more pronounced than daily fluctuations at these deeper soil levels. The soil temperatures at greater depths are afected by the long-term averaged efects of meteorological parameters.
Even though direct one-to-one quantitative comparison is not recommended due to many research contexts, contextually the performance results in table 4 indicate that the modelling in this study has close performance metrics to the recent models especially on cold weather climates by Imanian H. et al. [19, 23] and for some of the soil depths even better performances are recorded in this study.
The dataset consisted of fourteen independent meteorological variables and six dependent soil temperature variables at six soil depths. Considering the soil temperature variables at shallower depths as inputs for predicting temperatures at deeper depths helped increase the prediction accuracy. The soil temperature prediction models yielded comparably good results, with the potential for future improvements expected to enhance the accuracy of certain models, such as ST2 and ST50.
Research on soil temperature prediction for cold and snowy climates is rare and relatively challenging compared to ordinary and hot climates due to the less impact of air temperature on soil temperature during colder periods. Nevertheless, this study has successfully developed a model-based soil temperature prediction with performance results comparable to most existing research and even better performance at some soil depths. Although it is dificult to compare the prediction performance results directly with the results from other research without contextualizing the dataset, location, season, depth of soil and scaling factors used by the research, some results taken from the recent research are presented here for contextual analysis.
The authors would like to thank Zimmer and Peacok for their involvement and guidance in the project.
The author(s) have used Claude 3.7 Sonnet to generate the Latex code for the tables. [10] L. Mampitiya, K. Rozumbetov, N. Rathnayake, V. Erkudov, A. Esimbetov, S. Arachchi, K. Kantamaneni, Y. Hoshino, U. Rathnayake, Artificial intelligence to predict soil temperatures by development of novel model, Scientific Reports 14 (2024). doi: 10.1038/s41598- 024- 60549- x, publisher: Nature Research. [11] O. Kisi, M. Tombul, M. Z. Kermani, Modeling soil temperatures at diferent depths by using three diferent neural computing techniques, Theoretical and Applied Climatology 121 (2015) 377–387. doi:10.1007/s00704- 014- 1232- x, publisher: Springer-Verlag Wien. [12] M. Taheri, H. K. Schreiner, A. Mohammadian, H. Shirkhani, P. Payeur, H. Imanian, J. H. Cobo, A Review of Machine Learning Approaches to Soil Temperature Estimation, Sustainability (Switzerland) 15 (2023). doi:10.3390/su15097677, publisher: MDPI. [13] H. Zare Abyaneh, M. Bayat Varkeshi, G. Golmohammadi, K. Mohammadi, Soil temperature estimation using an artificial neural network and co-active neuro-fuzzy inference system in two diferent climates, Arabian Journal of Geosciences 9 (2016). doi: 10.1007/s12517- 016- 2388- 8, publisher: Springer Verlag. [14] V. Farhangmehr, J. H. Cobo, A. Mohammadian, P. Payeur, H. Shirkhani, H. Imanian, A Convolutional Neural Network Model for Soil Temperature Prediction under Ordinary and Hot Weather Conditions: Comparison with a Multilayer Perceptron Model, Sustainability (Switzerland) 15 (2023). doi:10.3390/su15107897, publisher: MDPI. [15] M. Bilgili, S. Ünal, A. Şekertekin, C. Gürlek, Machine Learning Approaches for One-Day Ahead Soil Temperature Forecasting, Tarim Bilimleri Dergisi 29 (2023) 221–238. doi:10.15832/ankutbd. 997567, publisher: Ankara University. [16] I. Ebtehaj, H. Bonakdari, P. Samui, B. Gharabaghi, Multi-depth daily soil temperature modeling: meteorological variables or time series?, Theoretical and Applied Climatology 151 (2023) 989–1012. doi:10.1007/s00704- 022- 04314- y, publisher: Springer. [17] L. Xing, L. Li, J. Gong, C. Ren, J. Liu, H. Chen, Daily soil temperatures predictions for various climates in United States using data-driven model, Energy 160 (2018) 430–440. doi:10.1016/j. energy.2018.07.004, publisher: Elsevier Ltd. [18] X. Wang, W. Li, Q. Li, A New Embedded Estimation Model for Soil Temperature Prediction,
Scientific Programming 2021 (2021). doi: 10.1155/2021/5881018, publisher: Hindawi Limited. [19] H. Imanian, A. Mohammadian, V. Farhangmehr, P. Payeur, D. Goodarzi, J. Hiedra Cobo, H. Shirkhani, A comparative analysis of deep learning models for soil temperature prediction in cold climates, Theoretical and Applied Climatology 155 (2024) 2571–2587. doi:10.1007/ s00704- 023- 04781- x, publisher: Springer. [20] NBMU, Meteorological data for Ås - BIOKLIM, 2024. URL: https://www.nmbu.no/forskning/ grupper/meteorologiske-data. [21] Wolf. M, Grimenes A.A, Meteorologiske data for Ås, Scientific report, Norwegian University of
Life Sciences (NMBU) 2000-2024, Technical Report, NMBU, ???? [22] A. Géron, Hands-On Machine Learning with Scikit-Learn and TensorFlow, Technical Report, 2017. [23] H. Imanian, J. Hiedra Cobo, P. Payeur, H. Shirkhani, A. Mohammadian, A comprehensive study of artificial intelligence applications for soil temperature prediction (2022). URL: www.preprints.org. doi:10.20944/preprints202202.0101.v2. [24] H. Hao, F. Yu, Q. Li, Soil Temperature Prediction Using Convolutional Neural Network Based on Ensemble Empirical Mode Decomposition, IEEE Access 9 (2021) 4084–4096. doi:10.1109/ACCESS. 2020.3048028, publisher: Institute of Electrical and Electronics Engineers Inc.