The APSIM software has proven to be extremely valuable decision support tool when it comes to optimizing agricultural management practices in order to maximize yield. Conducting high amount of simulations naturally requires processing of large volumes of data, therefore the available time and hardware resources create a limit for the scale of production simulation modelling. If APSIM is to be effectively used at a local level utilizing preexisting hardware infrastructure, an assessment of available resources must be conducted in order to optimize the scale of the simulation. Another issue is the availability of personnel with enough information technology skills and experience to conduct the processing. The focus of this paper are software automation and other assistance tools that are therefore required for production modelling to be successfully utilized by small to medium enterprises.
Maximizing yields of agricultural production is one the critical issues for society
today. Growing worldwide population exacerbates the need for sufficient food
production while increasing climate anomalies such as droughts can pose a great risk
for crops. One of the approaches to maintain and improve agricultural yields is to
utilize information technology to optimize managerial strategies and genotype
selection by conducting multi-factor analysis in form of simulations or modelling
The development of information technology hardware provides increasingly more
technological resources to conduct simulation processing on larger scales, however as
shown by
Second issue is the lack of qualified employees. Especially in agriculture there is a
distinct lack of IT proficient workers as pointed out by
As shown in Figure 1, the main workflow of agricultural simulation processing can be divided into four main steps. Firstly, it is necessary to establish the correct scale of the processing. When adjusting the number of options for each factor, the total number of simulations that needs to be processed changes accordingly. Since the number of simulations is essentially a product of the number of varying options, restraining variability only to the most relevant settings is the main way to reduce the processing time.
Fig 1. Basic simulation workflow
The factors and options that are determined in this first stage can be for instance different soil types, plant genotypes, different weather condition scenarios, and settings involving various managerial practices such as time of sowing, time of harvest, irrigation, fertilization, number of plants per meter square etc. (see Figure 2). Fig 2. Basic concept of GxExM framework (genotype, environment, management)
Second step is the data preparation and preprocessing. Each simulation is stored as a single xml file. When APSIM is being deployed on dedicated hardware, is it possible to generate these files “on the fly” during the processing. However as pointed out by (Jarolimek et al., 2019) the hardware components, mainly processor and RAM, constitute a limit on to how many simulations at once can APSIM handle. Therefore, in order to maximize effectiveness, it is necessary to optimize simulation batch sizes on a per machine basis, that is why the data preparation and preprocessing step is unavoidable when trying to utilize sub-par preexisting hardware in smaller or medium enterprises.
The actual simulation processing itself should be conducted during downtimes such as nights, when the infrastructure is not required for other critical operations. This part is generally the most time consuming and therefore also the most likely to significantly improve the overall efficiency if automation and scheduling is utilized. The hardware dependence of the processing can result in bottlenecks if the previous two steps were not conducted properly.
The analysis of results is essentially a statistical data analysis and can therefore be conducted using tools like MS Excel or more specialized software such as SAS, STATISTICA etc. The most basic analysis would consist of simply taking the simulations that produced highest yields and finding commonalities in the simulation settings.
The number of total simulations that needs to the processed determines the scale of the processing. The calculation of the number of simulations is a simple multiplication of options for each simulation settings. The scale can be therefore decreased or increased by simply adjusting the number of options. When scaling up, it is possible to include more options or “what-if” scenarios and generally explore more combinations. When there is a need to scale down, the most likely best approach is to use hands on experience of local producers or company agronomists to narrow down the simulation option settings only to a few variations that were historically most effective or make most sense in terms of the common local managerial practices.
Therefore, the important question in step one is how many simulations we should aim for. In order to properly select the scale of simulation an estimate must be made, based on allotted time and the hardware available for the computation. And it is in this point where the absence of experienced IT employees creates first problems. Unlike bigger companies or corporations that might have dedicated IT departments, SMEs generally lack specialists that would be able to adequately estimate the capabilities of older hardware when it comes to processing large number of simulations.
The support tools for this step could be a simple web application where the user inputs basic hardware information of their machines including processing power and RAM and the application will estimate number of simulations that can be run per day or per hour.
The main goal of this step is to gather all input information for the APSIM software.
There is a possibility for integration between APSIM and existing agricultural software
to partially automate this process, similar to that outlined by
The other part of preprocessing consists of generating the APSIM simulation files beforehand and grouping them into various size batches optimized on a per machine basis. The degree of detail this task needs to be performed to depends on the sophistication of the software automation tools used in the following step. As shown by (Pavlik et al., 2019) it is possible to develop an application that can combine some of the work required in steps two and three and handle both the batch APSIM file generation and the processing scheduling and automation.
There are four basic approaches to automate the simulation processing: 1. Use built-in APSIM capabilities 2. Process simulations ex-situ – on the cloud 3. Use existing software tools for automation 4. Develop new software specifically designed to automate APSIM simulations As explained earlier, using existing APSIM options for automation might not be possible due to the hardware limitations. Processing on the cloud is in essence similar to purchasing dedicated hardware. It might be cheaper, but in this paper, we are focusing mainly on exploiting already existing hardware and infrastructure. This leaves us with options three and four.
There are many existing software tools to automate tasks. Whether it be task schedulers that already come with the operating system, or dedicated automation software, such as HTCondor. The advantage of using such tools is that they include many useful functions such as workload monitoring, virtualization, checkpointing and can also combine serialized batch processing with parallelization. Therefore, this option is preferable when used on hardware that also performs other day to day tasks. The main disadvantage is that the previous and following steps (data preprocessing and result analysis) will require more work since they cannot be incorporated into these existing tools, not even partially.
The last approach is to design and develop brand new tools specifically to automate APSIM simulation processing. A custom-built tool could potentially encompass more that just the processing automation. It can theoretically handle all four steps of the workflow, bridging the gaps between the parts. This could be very valuable considering the lack of experienced IT employees in smaller agricultural companies. However, it will come at a cost of lower sophistication of automation and it will be harder to combine the processing with other necessary tasks the hardware needs to perform on a daily basis. This approach is therefore better suited for situations when the company can set aside one or several computers, perhaps older or currently unused machines, and fully use them towards the simulation processing.
If the analysis of the results is to be conducted in a separate statistical software, it is necessary to convert the output simulation from the basic text format into .csv or .xsl. This can be achieved with an extraction program or a script that will parse the output files and siphon only the necessary data.
However, when simulating agricultural production on a local level, the only important outputs are the simulation settings associated with the highest yields. A complex statistical analysis may therefore not be necessary. In case of developing new custom software for the previous processing steps, the extraction of yield data, selection of top simulations and any eventual visualization of simulation settings can be added to it, resulting in an overarching support tools that can automate vast majority of the workflow processes.
The paper discussed two main approaches to conducting agricultural production simulations in smaller or medium companies. The first approach is to split the process into logical parts and optimize and automate them separately, utilizing either already existing software tools, or developing a smaller programs for individual tasks, such as format conversion tools, simulation generation tools, data parsers etc. The main advantages of this approach are higher optimization and better interoperability when utilizing existing hardware that cannot be fully dedicated to the task. However, such a solution would require employee skilled in IT, prompting additional need for financial resources in order to hire or train someone.
The second approach consists of developing an overarching support tool that would incorporate all the various processing tasks. If developed as a general purpose streamlined software with focus on ease of use, it could overcome the problem with lack of experienced IT workers in small to medium companies. The disadvantages of this solution are harder incorporation with existing agricultural software running parallel on the same machines and loss of modularity options such as in-depth data analysis or conducting simulations for goals other than maximizing yields. Acknowledgment: The results and knowledge included herein have been obtained owing to support from the following institutional grant. Internal Grant Agency of the Faculty of Economics and Management, Czech University of Life Sciences in Prague, grant no. 2019MEZ0005 – “Optimization of management practices in sorghum production under uncertain future weather conditions”.