OpenNESS Synthesis Paper
Ulrich Heink (UFZ, Germany); Christoph Görg (UNI-KLU, Austria); Kurt Jax (UFZ, Germany)
Introduction and rationale
The ultimate goal for managing ecosystem services is contributing to human well-being. To analyse how this goal can effectively be achieved is a major task in OpenNESS. In general, effectiveness can be regarded as the extent to which an entity to be evaluated produces desired or intended outcomes (Davidson, 2005: 122). Policy or management activities can deliver outputs which are the direct products and services produced by the activities, and lead to outcomes, i.e., the consequences of those outputs for individuals or (parts of) our society and their well-being. Outputs are results that can largely be controlled, while the outcomes aimed at are influenced by factors outside policy or management. In OpenNESS, for example the ES concept itself, a policy affecting ES, or ecosystem management can be evaluated in terms of effectiveness. The definition given above, though very broad, has some important implications (Mickwitz, 2003; Davidson, 2005): First, as effectiveness is a measure for an instrumental relationship towards a goal, it does not refer to the acceptability or relevance of the goal itself. Second, it is possible for something to be effective towards a well-accepted goal but at the same time produce serious detrimental, if unintended, effects. Further, a measure can be effective, but very costly.
Assessing effectiveness faces some general problems. The prediction of activities on ultimate outcomes is quite difficult, and when intended outcomes occur it is quite uncertain that they have been affected by the activities conducted. Further, even when outputs or immediate outcomes are achieved, activities might not be very effective in terms of ultimate outcomes. For example, the cultivation of energy plants (output) can lead to the immediate outcome of enhancing the share of renewable resources for road transport fuels. However, land use change compromises the intended ultimate outcome of reducing overall greenhouse gas emissions. For these reasons evaluating effectiveness is a very complex task, and for one and the same intervention the evaluation result can be very different depending on the perspective taken.
Effectiveness expresses a relation between an outcome of a policy or management and a (group of) person(s): something appears effective to somebody. Hence, properties of a concept or policy itself (like foundation on sound information, consistency) as well as processes of communication and social learning can trigger effectiveness. And last, what is judged as effective by one person can be regarded as ineffective by somebody else. Therefore, effectiveness also has something to do with personal or cultural disposition. Effectiveness and hence its assessment are thus social constructs. What effects are referred to and which values underlie an evaluation of effectiveness ideally should be negotiated as part of an ES management process.
There are different conceptual approaches for determining effectiveness. Most notably, in the context of evaluations of scientific advice in general, and environmental assessments in particular, credibility, relevance and legitimacy are regarded as determinants of effectiveness (Cash et al., 2003, Heink et al., 2015). Credibility reflects the perceived validity of information, methods and procedures provided, relevance reflects the extent to which a concept or activity is responsive to the needs of problem-solving, and legitimacy reflects the fairness, balance and political acceptability of processes and their outcomes. The effectiveness of interventions does not only depend on an outcome achieved but also on practitioners’ understanding of credibility, relevance and legitimacy in a specific and often changing policy context.
But how can effective activities be planned and their effectiveness evaluated? In this respect, “program theories”, i.e., explicit theories how activities cause intended or observed outcomes by modeling intermediate steps and linkages, can be useful (Chen, 2005). On the one hand, such theories can be used to establish the intended effects of instruments and the policy fields where these instruments are applied (Mickwitz, 2003); further, they can be used to determine which outputs and outcomes need to be measured and how and what linkages between these can be assumed Finally, program theories can help to interpret and evaluate the effectiveness of policy interventions.
Significance referring to "operationalisation of the concept of Ecosystem Services"
What constitutes “effectiveness”, and how it is measured, can vary considerably between fields of application and knowledge users. For example, the effectiveness of an intervention (e.g., based on ES or NC) that promotes health can be much different than for keeping a favourable conservation status of a protected area. Different evaluation approaches and languages amongst disciplines will be a big challenge when providing operational guidance by the WPs that is relevant to multiple sectors. We cannot detail on every field of application and knowledge user here but highlight three fields which are especially relevant to OpenNESS, namely (1) effectiveness of the ecosystem service concept in its potential to advance knowledge and add a new perspective for policy-making, (2) effectiveness of governance of ecosystem services and (3) effectiveness of ecosystem management.
Firstly, effectiveness can be related to the use of the ecosystem service concept as a tool for advancing knowledge and framing policies: does the ecosystem service concept help to recognize values, i.e. to express values people assign to ecosystems or to successfully change thinking about the value of ecosystems? Further, is it helpful for decision-making, e.g., by enhancing acceptance of decisions?
The effectiveness in using the ES concept for advancing knowledge and framing policies depends on the purpose of knowledge utilization. There are three main types of knowledge utilization (Hertin et al., 2009):
- Conceptual knowledge utilization: when knowledge `enlightens' policy makers by slowly feeding new information, ideas, and perspectives into the policy system, challenging existing beliefs and opening up new opportunities for policy change.
- Instrumental knowledge utilization: when knowledge directly informs concrete decisions by providing specific information on the design of policies, e.g. information to technically implement decisions or information reducing uncertainty on outcomes from policy decisions.
- Political knowledge utilization: when knowledge is put forward to attain political objectives. For example, it may justify decisions already taken, disarm opponents, or postpone a decision. In contrast to conceptual knowledge utilization, knowledge is used here strategically to justify a policy which is already decided on.
The effectiveness of the ES concept can be quite different according to the type of knowledge utilization. For example, it might be conceptually highly effective in framing a policy in terms of human well-being but instrumentally ineffective when there are barriers to operationalize the concept. Or it might be politically effective if ES arguments actually “count” (e.g., Kareiva & Marvier, 2012). Unfortunately, there is little literature on the effectiveness of the ecosystem service concept (as an exception Fisher et al., 2013).
Secondly, effectiveness is related to the governance of ecosystem service. Several aims from environmental policy and nature conservation, but also from agriculture and other policy fields (and potential trade-offs between these aims) may be involved. Thus, as mentioned before, the questions of negative side-effects, on potential trade-offs and policy integration is very important (see SP on Good Governance). Here the question arises, how a multi-level and multi sectorial governance system can effectively sustain ES delivery? And how can this be improved by the process of stakeholder participation in decision-making?
Effectiveness of governance generally refers to the adequacy of instruments to bring about an intended outcome. On the one hand this directs attention to policy instruments. Generally speaking, there are three types of policy instruments used in governance (Vedung, 1998): economic, regulatory, and informative instruments, or carrots, sticks, and sermons for short. Here, the underlying question is: What is the most suitable policy instrument to achieve a certain outcome? On the other hand, it is important that policy instruments are applied coherently so that they do not neutralize themselves with regard to their effects. This effectiveness of policy integration and coherence is central to the OpenNESS project. The underlying question here is: How can environmental objectives be incorporated in decision-making, within but also beyond environmental sectors, and how can contradictions between environmental and other policies be minimized (cf. Lafferty & Hovden, 2003)?
Policy integration is mainly tackled from an institutional perspective, but can also be highlighted from a political perspective concerning political ‘will’ and ‘leadership’ and a cognitive perspective assuming that policy interests are often embedded in a frame of reference or set of ideas which pre-structures the thinking within a policy sector (Jordan & Lenschow, 2010). So far, it remains open to what extent OpenNESS addresses these different perspectives that influence the effectiveness of policy on ecosystem services.
Governance not necessarily needs to be related instrumentally to substantiate decision-making, but the governance process may serve to enhance fairness or to fulfil democratic principles. Stakeholder involvement is here not only a means to broaden the information basis but an end in itself. The effect which is aimed at is therefore legitimacy rather than policy impact (e.g., Heink et al., 2015).
Thirdly, effectiveness can be related to ecosystem management: do management practices provide for a sustainable flow of and meet the demand for ecosystem services?
Compared to the two aforementioned fields of effectiveness, conceptualizing effectiveness of ecosystem management is a seemingly simple task: effectiveness of ecosystem management is the achievement of a management goal. However, there are many practical obstacles for effective ecosystem management. Often there is a failure to clearly define management endpoints (Failing & Gregory, 2003). Therefore, in OpenNESS management endpoints for target ecosystems should be explicitly defined. Ecosystem services, in this context, can be seen as a tool to support formulating and communicating such endpoints. A further obstacle is a lack of implementing management goals (Koontz & Bodine, 2008). Problems in implementation resemble the problems in policy integration although their importance might be different. Further, lack of knowledge, uncertainty and complexity of (socio-)ecological systems affect the effectiveness of ecosystem management. And last, failure to appropriately identify and engage with stakeholders and resource limitations hinder effective ecosystem management.
Remaining Problems / Issues
- From the three fields of effectiveness that can be referred to, which are most important ones when dealing with the operationalization of the ecosystem service concept?
- What importance has effectiveness in overall policy appraisal and evaluation of ecosystem management?
- What are adequate methods for evaluating effectiveness? What aspects of effectiveness are being tested in the project and can be asked to be tested in the case studies?
- How can program theories for the case studies be developed and in which way do these differ from each other?
- What is the role of effectiveness in different evaluation procedures (e.g., cost-effectiveness analysis)?
- How does the importance of factors influencing effectiveness differ between different policy levels, sectors and between policy and ecosystem management?
Relationship to four challenges
Enhancement of human well-being is the final outcome of managing ecosystem services. The extent to which goals related to human well-being are achieved is therefore a fundamental measure of effectiveness.
Sustainable Ecosystem Management:
Ecosystem management aims at sustainably delivering ecosystem services. Thus, when ecosystem services are provided sustainably ecosystem management is effective.
Policy coherence and integration as well as stakeholder integration can be tackled from the perspective of effectiveness: Are ecosystem services mainstreamed in policy and do participatory processes enhance legitimacy?
Competitiveness is a double-edged sword with regard to effectiveness. On the one hand efficiency (and thus indirectly effectiveness) certainly enhances competitiveness by saving costs for inefficient activities, on the other hand competition sometimes leads to ineffectiveness and cooperation would rather lead to intended outcomes.
Effectiveness can relate to different activities and concepts. For any study using the concept of effectiveness it is crucial to check which of these activities should be analysed from the perspective of effectiveness and if or how this can be done in the case studies.
Suggested four ‘Must read papers’
Jordan, A. and Lenschow, A. (2010): Environmental Policy Integration: a State of the Art Review. Environmental Policy and Governance 20: 147-158.
Mickwitz, P. (2003): A framework for evaluating environmental policy instruments. Context and key concepts. Evaluation 9: 415–436.
Koontz, T. M. & Bodine, J. (2008): Implementing ecosystem management in public agencies: Lessons from the US Bureau of Land Management and the Forest Service. Conservation Biology 22: 60-69.
Stem, C., Margoluis, R., Salafsky, N., & Brown, M. (2005). Monitoring and evaluation in conservation: a review of trends and approaches. Conservation Biology, 19(2), 295-309.
Further cited papers in this text
Chen, H.-T. (2005): Program theory. In: Mathison, S. (ed.) Encyclopedia of evaluation. Thousand Oaks, London, New Delhi: Sage Publications, 340-342.
Davidson, J. (2005): Effectiveness. In: Mathison, S. (ed.) Encyclopedia of evaluation. Thousand Oaks, London, New Delhi: Sage Publications, 122.
Failing, L. and Gregory, R. (2003): Ten common mistakes in designing biodiversity indicators for forest policy. Journal of Environmental Management 68: 121.
Fisher, J. A., Patenaude, G., Meir, P., et al. (2013): Strengthening conceptual foundations: Analysing frameworks for ecosystem services and poverty alleviation research. Global Environmental Change-Human and Policy Dimensions, 23, 1098-1111.
Hertin, J., Turnpenny, J., Jordan, A., et al. (2009): Rationalising the policy mess? Ex ante policy assessment and the utilisation of knowledge in the policy process. Environment and Planning A 41: 1185-1200.
Jordan, A. and Lenschow, A. (2010): Environmental Policy Integration: a State of the Art Review. Environmental Policy and Governance 20: 147-158.
Heink, U., Marquard, E., Heubach, K., et al. (2015): Conceptualizing credibility, relevance and legitimacy for evaluating the effectiveness of science-policy interfaces: challenges and opportunities. Science and Public Policy, Advance Access, doi:10.1093/scipol/scu082
Kareiva, P. and Marvier, M. (2012): What Is Conservation Science? Bioscience 62: 962-969.
Lafferty, W. M. and E. Hovden (2003): Environmental Policy Integration: Towards an Analytical Framework. Environmental Politics 12(3): 1 – 22.
Vedung, E. (1998): Policy Instruments: Typologies and Theories. In: Bemelmans-Videc, M.-L. et al. (eds): Carrots, Sticks, & Sermons: Policy Instruments & Their Evaluation, pp. 21–58. New Brunswick, NJ: Transaction Publ.
Review editor: Marion Potschin (UNOTT)
Suggested Citation: Heink, U.; Görg, C. and K. Jax (2016): Effectiveness. In: Potschin, M. and K. Jax (eds): OpenNESS Ecosystem Services Reference Book. EC FP7 Grant Agreement no. 308428. Available via: www.openness-project.eu/library/reference-book
Acknowledgements: The following OpenNESS partners have further contributed to the SP: Jan Dick (CEH), Hans Keune (INBO), Pam Berry (UOXF), Bálint Czúcz (MTA ÖK), Ben Delbaere (ECNC) , Raktima Mukhopadhyay (IBRAD), Angheluta Vadineanu (UB-DSES), Heli Saarikoski (SYKE), Sanna-Riikka Saarela (SYKE), Marina García-Llorente (UAM), Conor Kretsch (UNOTT), Rob Budgter (ALTERRA), György Pataki (ESSRG), Francis Turkelboom (INBO).
Disclaimer: This document is the final version of the Synthesis Paper on the topic within the OpenNESS project. It has been consulted on formally within the consortium in 2015 and updated in 2016.