Developing an assessment tool to evaluate the sustainability of sheep and goat farming systems in Europe

Abstract

Assessing the sustainability of small ruminant systems is crucial to ensure their long-term viability, to identify potential areas of improvement, to uncover trade-offs between different aspects of performance and to, potentially, demonstrate benefits of particular management strategies. However, there is scarce literature and guidance regarding optimal tool choice and effective practical use of such tools with regards to the evaluation of the sustainability of small ruminant systems. The overall aim of this work was to select the most appropriate indicators following the coverage of the sustainability criteria defined by the SAFA framework and to select or create if appropriate, a rapid, but effective, assessment tool. Ease of tool use and it's easiness of adaptation to include new indicators were key criteria. This paper describes the selection of the most appropriate indicators and their incorporation into an adapted version of the Public Goods Tool (PG Tool).

Keywords

• Sheep
• Goats
• sustainability assessments
• PG Tool
• indicators
• ruminants

1 Introduction

Sustainability assessment can be defined as an evaluation exercise that directs decision-making to ensure ongoing feasibility of the production system (Bond et al., 2012; Hugé et al., 2013; Pope, 2006) . Indicator-based sustainability assessment tools and frameworks can either guide or conduct sustainability assessments (Gasparatos and Scolobig, 2012; Ness et al., 2007) and vary widely in their scope (e.g. geographical and sector), target group (e.g. farmers or policy makers), selection of indicators, aggregation and weighting method and time required (Binder et al., 2010; Marchand et al., 2014; Schader et al., 2014) . Scientific evidence on farm *These authors equally contributed to this work. sustainability assessments, as well as the sustainability assessment tools available to support decision-making, are ever-expanding; however, these assessment tools can vary enormously in their scope and approach. (Binder et al., 2010; Bockstaller et al., 2009; Gasparatos and Scolobig, 2012; Marchand et al., 2014; Ness et al., 2007; Schindler et al., 2015) .

Assessing the sustainability of small ruminant systems is crucial as it will help to ensure their long-term viability, to identify potential areas of improvement and efficiencies, to uncover trade-offs between different aspects of performance and to, potentially, demonstrate benefits of particular types of management strategies and the sector as a whole. Such assessments require physical visits to farms and collection of data of different nature. However, questions arise on how to navigate between existing tools, what their key characteristics are and how can one select the most appropriate tool for this specific purpose. There is scarce literature and guidance regarding optimal tool choice and the effective and practical use of such tools and methodologies when it comes to small ruminant systems (De Ridder et al., 2007; Bernués et al., 2011; Gasparatos and Scolobig, 2012) . In addition, the typology of the sheep and goat sector in Europe consists of a great diversity of farm types in terms of production aims, farm size, breeds kept, and levels of intensification, including also organic, Protected Designation of Origin (PDO), Protected Geographical Indication (PGI), pluriactive farms or breed specific farms (Theodoridis et al 2016) . These farm types are specific to a wide range of environmental and socioeconomic conditions with local, regional, national or intra-national importance and can vary in sustainability and their needs for innovation. Therefore, the diversity of the existing farm types and their production aims across Europe imposes a great challenge for selecting appropriate indicators and tools to assess the sustainability of the sheep and goat production systems.

The aim of this work was therefore to identify a) the most suitable indicators and tools for assessing the sustainability of sheep and goat farms and b) to incorporate the identified indicators to the most appropriate sustainability tool selected for the assessments. To do this, we first conducted a thorough review of existing sustainability assessment tools and frameworks and evaluated the suitability of the identified tools with the involvement of a number of stakeholders across Europe. Concurrent to this, specific indicators of sustainability of particular relevance to sheep and goat systems, focusing on the often under-represented social, economic and animal health and welfare aspects (e.g. Schader et al., 2014) , were identified. The information collected were analysed in order to select and establish a final tool to be used for the assessments. The indicators already contained in it and the ease of adding additional indicators was also taken into account. The methodology followed is described in detail in the following sections. 2

Review of sustainability tools and indicators

Different terms are used in the literature to describe sustainability assessments, such as methods, methodological approaches, frameworks, and tools (Marchand et al., 2014; Schader et al., 2014; Schindler et al., 2015) . In this review, we focused on Tool

RISE SAFA IDEA

Full name Response Inducing Sustainability Evaluation Sustainability Assessment of Food and Agriculture Systems

Target group

Farmers Food and agricultural enterprises, organizations, governments PG Tool

Public Goods Tool

Farmers, policy-makers those sustainability assessments that have been developed into tools aimed at conducting ex-post assessments of the sustainability performance of farms using indicators. These are called indicator-based sustainability assessment tools.

A range of scientific papers on sustainability assessments relevant to agricultural systems were identified in the literature (e.g. Alrøe et al, 2016; Alrøe and Noe, 2016; Olde et al., 2016; Lewis et al., 2010; Marchand et al., 2014; Padel et al., 2015; Schader et al., 2014; Schader et al., 2016; Smith and Little, 2013) . The review of these and other studies resulted in a long but not exhaustive, list of 103 sustainability tools; these tools were in turn categorized based on the following criteria: i. the quantification of sustainability used (functional units; e.g. currency, carbon footprint, standardised units etc.), ii. farm, product or sector level (spatial scale), iii. whether the tool was designed for a specific country or region or is more widely applicable (transferability), iv. Whether it is sector specific (i.e. specific to dairy/crops/etc. or covers a range of farm types), v. time taken to complete the assessment and vi. software or platform used. Following this exercise, 21 tools (out of the 103 identified in the literature) were subsequently selected and prioritised based on their coverage in view of the FAO’s SAFA (Sustainability Assessment of Food and Agriculture systems) framework guidelines. A similar approach was conducted by Olde et al. (2016) for four of the tools included in our analysis (RISE, SAFA Tool1, PG Tool and IDEA). Their results are shown below, in Table 1. Indicateurs de Durabilité des Exploitations Agricoles

Farmers, policy-makers, Zahm et al. education (2008)

Reference

Häni et al. (2003) FAO (2012) Gerrard et al. (2012)

Origin

Switzerland (Bern University of Applied Sciences Multiple countries and institutes UK (The Organic Research Centre) France (multiple institutes)

The FAO’s SAFA guidelines define sustainability in in four domains (i.e. environmental integrity; economic resilience; social wellbeing and good governance) and each domain is further subdivided into themes and then sub-themes when appropriate. Indicator quality is assessed in SAFA based on whether it is outcomebased (e.g. actual measures of performances such as soil nitrogen surplus), related to individual farm practices or simply a farm target. Being a well-founded and widely accepted approach in the sector, SAFA provided a sensible framework to use as basis for the conduction of our own review of indicators. Therefore, any indicators additional to those suggested in the SAFA framework were identified and where these indicators did not fit within an existing theme or sub-theme, new classes were proposed.

2.1 Stakeholders survey on identifying appropriate indicators and sustainability tool

Following the literature review of the tools and indicators a list of the potential indicators and sustainability tools was presented by means of an on-line survey to a number of sheep and/or goat stakeholders representing farmers in Greece, UK, France, Italy, Spain, Finland and Turkey. These partners were asked to rank the indicators in view of their experience and knowledge, on the basis of the most “appropriate” and farmer/industry “commonly used” indicators. Participants were also asked to rank and give details of any sustainability assessment tools that they had used in the past or heard details of. The survey was developed using the Qualtrics survey platform (Qualtrics, 2016) and was web-based. A total of 35 responses were received from 69 potential participants (researches and stakeholders) while the 35 responses covered 95 % of the participating stakeholders. The results of the survey were analysed in view of the industry type i.e. sheep vs. goat organisations and were compared. In addition, preferences of research vs. industry partners were also evaluated. Interestingly, there was a high level of agreement between different stakeholder groups. The results of the survey were used to help compile a final list of sustainability indicators and identify a suitable assessment tool for the purposes of this work.

3 Selection of sustainability tool and indicators

In addition to the literature review and stakeholder survey, the development of the assessment approach was based also on workshop discussions that aimed to i. identify the most appropriate indicators in all dimensions (i.e. social, economic, environmental, governance), ii. select the best sustainability assessment tool in view of the European typology (see report by Theodoridis et al., 2016) of the sheep and goat farms and iii. ensure that all the key indicators identified in step (i) were included in the tool selected in step (ii) and if not, an additional list of indicators would be built to be incorporated in the selected tool.

3.1 Selection of the indicators

The decision on the most appropriate indicators (with starting point on survey results) was discussed with project partners through workshops and on-line discussions; these meetings helped also to identify additional indicators that were particularly relevant in more specialist types of production systems. The SAFA’s general criteria to the categorisation of indicators was applies and selected indicators fall within following three categories.

Animal health, welfare and livestock management indicators. The final selection of the specific animal welfare indicators considered the particularities of sheep and goat farming systems and their production purposes in view of the existing farm-type variation (e.g. intensive, extensive, PDO, organic, transhumance, etc.). in addition to the literature review and stakeholders’ feedback, the selection built upon recommendations from the AWIN project (AWIN, 2015), DairyCare COST Action and SOLID project experience, as well as latest EFSA recommendations (2014). The addition of new indicators was based on a multi-dimensional concept (mental and physical health in harmony with environment and the ability to cope with likely changes in the environment due to climate change) following the principles of good feeding, good housing, good health and appropriate behaviour, and considered both extensive and intensive systems. In the selected indicators, resource-based and management-based indicators have been combined with questions associated with direct observations of animals.

Socioeconomic indicators. As previously, the identification of socioeconomic indicators considered the specificities of sheep and goat farming systems across Europe (Theodoridis et al., 2016) . For the economic indicators, preference was given to the “gross margin” approach (goat or sheep income, less variable costs), which is already used by several farmers and requires less time to evaluate the economic efficiency of livestock enterprises than the “costs of production” approach. The selected indicators covered a range of socioeconomic aspects such as product quality, food safety, product certification, integration in the local economy, vulnerability and long-term profitability, investments farm succession in addition to indicators related to the on-farm working conditions, health and safety and interaction of the farm with the wider social environment.

Environmental indicators. These indicators were mostly selected based on the SAFA themes of atmosphere, water, landscape, soil quality, biodiversity, and materials and energy. Individual indicators associated with each area were drawn from a range of sources including Government guidelines and codes of best practice (e.g. Defra, 2006, Environment Agency, 2004, Natural England, 2007) , industryfacing guidelines (e.g. ADAS and The Organic Research Centre, 2002) and the SAFA framework itself (FAO, 2012). New indicators were selected in relation to the use of supplementary feed, on-farm energy efficiency and biodiversity management.

3.2 Selection and adaptation of the sustainability tool

The decision on the most appropriate tool was based in principle on its compliance with the FAO SAFA guidelines, the time taken for the completion of the assessment, . the scientific rigour of the farm assessment process and the need to adapt the assessment framework through the inclusion of new indicators identified within the literature review, stakeholder survey and workshops. It was also apparent such a tool should be capable of assessing the sustainability of sheep and goat farms across a range of farm types and will serve the basis for developing a toolbox of assessment tools and indicators that can be used in future assessments. Additional selection criteria included the easiness to adapt the tool to include new indicators identified and the ease of tool use i.e. the expertise and time taken to carry out the assessment. Following the characterisation of tools according to above criteria it was concluded that concluded that the Public Goods Tool (PG Tool) provides the best option for a rapid yet comprehensive framework for assessing the sustainability of the sheep and goat sector. Amongst other advantages the readily of the PG Tool to be adapted, its modular structure to follow FAO’s SAFA framework and that fact the tool has been used successfully in multi-national sustainability assessments within a range of European projects made it clearly the best choice in view of the aims of the project.

The development of the PG Tool for the specific project involved a) the inclusion of the indicators identified and b) a comprehensive approach and analysis with regards to the formulation of the questions to asked, what will be the potential options for responses, and how each response will be scored. Some questions required five different answers, given scores 1 to 5, while others required only three and the scores assigned were 1, 3 and 5. Scores were attributed based on recommendations for best practice included within guideline documents (e.g. CALU, and ADAS, 2007) with the higher score representing the best sustainable practice while some indicators required more than one question in order to be addressed. Once indicators incorporated in the tool, they were analysed to finalise scoring criteria. A significant proportion of the new indicators added to the PG tool related to animal health and welfare, farm livestock management, social sustainability and governance (the latter area was previously missing from the PG tool). Animal welfare and social sustainability also tend to be under-represented in sustainability assessments in general and were therefore given a greater emphasis.

Following incorporation of the new indicators, the PG-Tool underwent systematic tests in UK, Spain, Greece, France and Italy as part of the WP1, assessing in sheep or goat farms in these counties, in order to identify possible flaws or errors. Feedback from farmers was recorded on the applicability and usefulness of the tool and the time it took to assessment. Results from these assessments suggest the tool provides a useful framework and identifies areas of poor/good performance. Following farmers’ feedback, considerable attention was paid on the time that the assessment is taking to be completed. The sustainability assessment using the final version of the tool takes about 2 to 3 h to be completed. This work showed that, despite the growing interest in livestock sustainability assessments, no specific tools that cover all sustainability domains and types of production systems were developed with focus on the small ruminant sector. Therefore, we put forward an assessment approach for selection of appropriate indicators and sustainability tools that lead in the creation of a rapid, but effective, assessment tool. We therefore consolidated information and data collected through the industry partner survey, workshop discussions and literature review and the most appropriate indicators in all dimensions (i.e. social, economic, environmental, governance) were identified in addition to the best tool for assessing sustainability of sheep and goat farms ensuring adaptability to a range of farm types. We concluded that the Public Goods Tool (PG Tool) was the most appropriate framework for adaptation as it was the first to fulfil all the key selection criteria (i.e. ease of tool use; the coverage of a range of sustainability criteria as defined within the SAFA framework and; the possibility and ease of adapting the tool to include new indicators). The results of the on-line survey highlighted the importance of keeping the process of the sustainability assessments relatively short (i.e. between two and three hours) in order to attack farmer’s interest and keep them engaged through the process. Thus, during the adaptation of the tool considerable attention was payed to the time that the assessment is going to take on farm.

Acknowledgments. This work was undertaken as part of the iSAGE Project (http://www.isage.eu/), with financial support from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 679302. We also thank professor Raffaele Zanoli for kindly providing access to the Qualtrics survey platform (Qualtrics, 2016) , which was used for the development of the on-live survey used in this work. 4. AWIN, 2015. AWIN welfare 10.13130/AWIN_GOATS_2015.

assessment protocol for goats.

DOI: 13. EFSA recommendations on animal https://www.efsa.europa.eu/en/topics/topic/animalwelfare welfare (2014): 14. FAO (2012). Sustainability (SAFA). FAO, Rome.

Assessment of Food and

Agriculture systems

References

1. ADAS and Organic Research Centre ( 2002 ), Managing Manure on Organic Farms. London: Defra.

2. Alrøe , H. and Noe , E. , 2016 . Sustainability assessment and complementarity . Ecology and Society 21

3. Alrøe , H. , Moller , H. , Laessøe , J. , Noe , E. , 2016 . Opportunities and challenges for multicriteria assessment of food system sustainability . Ecology and Society 21 .

5. Bernués , A. , Ruiz , R. , Olaizola , A. , Villalba , D. , Casasús , I. , 2011 . Sustainability of pasture-based livestock farming systems in the European Mediterranean context: Synergies and trade-offs . Livestock Science 139 , 44 - 57 .

6. Binder , C.R. , Feola , G. , Steinberger , J.K. ( 2010 ) Considering the normative, systemic and procedural dimensions in indicator-based sustainability assessments in agriculture . Environmental Impact Assessment Review . 30 , 71 - 81 .

7. Bockstaller , C. , Guichard , L. , Keichinger , O. , Girardin , P. , Galan , M.- B. , Gaillard , G. , 2009 . Comparison of methods to assess the sustainability of agricultural systems: a review . Sustainable Agriculture . Springer, pp. 769 - 784 .

8. Bond , A. ; Morrison-Saunders , A. ; Pope , J. ( 2012 ) Sustainability assessment: The state of the art . Impact Assess. Proj. Apprais . 30 , 53 - 62 .

9. CALU and ADAS ( 2007 ) Managing Energy and Carbon: The farmer's guide to energy audits . ADAS Wales.

10. De Ridder , W. , Turnpenny , J. , Nilsson , M. , Von Raggamby , A. , 2007 . A framework for tool selection and use in integrated assessment for sustainable development . Journal of environmental assessment policy and management 9 , 423 - 441 .

11. Defra ( 2014 ) Application of the public goods tool on conventional farms . Final report to Defra .

12. Dumanski , J. , Eugene Terry , Derek Byerlee, Christian Pieri. ( 1998 ) Performance Indicators for Sustainable Agriculture (Discussion Note) Policies of the World Bank .

15. Gasparatos , A. , Scolobig , A. , 2012 . Choosing the most appropriate sustainability assessment tool . Ecological Economics 80 , 1 - 7 .

16. Gerrard , C.L. , Smith , L.G. , Pearce , B. , Padel , S. , Hitchings , R. , Measures , M. ( 2012 ) The OCIS PG tool: A tool for assessing the Public Goods provision of a farm, Sustainable Agriculture Reviews . pp 1 - 22 .

17. Häni , F. , Braga , F. , Stämpfli , A. , Keller, T., Fischer , M. , and Porsche , H. ( 2003 ). RISE, a tool for holistic sustainability assessment at the farm level . International Food and Agribusiness Management Review 6 , 78 - 90 .

18. Hugé , J. , Waas , T. , Dahdouh-Guebas , F. , Koedam , N. , Block , T. , 2013 . A discourse-analytical perspective on sustainability assessment: interpreting sustainable development in practice . Sustainability science 8 , 187 - 198 .

19. Lewis K.A . et al. ( 2010 ) The climate change mitigation potential of an EU farm: towards a farm-based integrated assessment . Deliverable 5 .1, Final Report , ENV.B.1/ETU/ 2009 /0052.

20. Marchand , F. , L. Debruyne , L. Triste , C. Gerrard , S. Padel , and L. Lauwers ( 2014 ) Key characteristics for tool choice in indicator-based sustainability assessment at farm level . Ecology and Society 19 , 46 .

21. Natural England ( 2007 ). The Heather and Grass Burning Code. London: Defra.

22. Ness , B. , Urbel-Piirsalu , E. , Anderberg , S. , Olsson , L. , 2007 . Categorising tools for sustainability assessment . Ecological economics 60 , 498 - 508 .

23. Olde , E.M. , Oudshoorn , F.W. , Sørensen , C.A. , Bokkers , E.A. , De Boer , I.J. ( 2016 ). Assessing sustainability at farm-level: Lessons learned from a comparison of tools in practice . Ecological Indicators . 66 , 391 - 404 .

24. Padel , S. , Gerrard , C. , Smith , L. , Schader , C. , Baumgart , L. , Stolze , M. and Pearce , B. , ( 2015 ) Further Development of Methodologies for Sustainability Assessment and Monitoring in Organic/Ecological Agriculture . ORC reports, no. 2015 /1. Organic Research Centre, Newbury.

25. Pope , J. , Grace , W. , 2006 . Sustainability assessment in context: Issues of process, policy and governance . Journal of Environmental Assessment Policy and Management 8 , 373 - 398

26. Qualtrics ( 2016 ), Version 3 . Copyright © 2017 Qualtrics. Qualtrics and all other Qualtrics product or service names are registered trademarks or trademarks of Qualtrics, Provo , UT , USA. http://www.qualtrics.com.

27. Schader , C. , Baumgart , L. , Landert , J. , Muller , A. , Ssebunya , B. , Blockeel , J. , Weisshaidinger , R. , Petrasek , R. , Mészáros , D. , Padel , S. , 2016 . Using the sustainability monitoring and assessment routine (smart) for the systematic analysis of trade-offs and synergies between sustainability dimensions and themes at farm level . Sustainability 8 , 274 .

28. Schader , C. , Grenz , J. , Meier , M.S. , Stolze , M. ( 2014 ) Scope and precision of sustainability assessment approaches to food systems . Ecology and Society 19 .

29. Schindler , J. , Graef , F. , König , H.J. , 2015 . Methods to assess farming sustainability in developing countries. A review . Agronomy for sustainable development 35 , 1043 - 1057

30. Smith , L and Litte T ( 2013 ) Environmental footprinting for farm businesses Organic Research Centre, Elm Farm Tony Little .

31. Theodoridis , A. Ragkos , K. Zaralis , S. Mullender and G. Arsenos, ( 2016 ) iSAGE Deliverable No: 1.1 Report on new farm typologies for sheep and goat systems within the EU , Greece: Aristotle University of Thessaloniki.

32. Zahm , F. , P. Viaux , L. Vilain , P. Girardin , and C. Mouchet . 2008 . Assessing farm sustainability with the IDEA method from the concept of agriculture sustainability to case studies on farms . Sustainable Development 16 : 271 - 281 .