Measuring Accountability: The politics of indicators
By Barbara Adams, Roberto Bissio and Karen Judd
The visible commitment of the UN system and its Member States to the universal and inclusive 2030 Agenda for Sustainable Development risks being undermined by a less visible debate about the indicators by which to assess progress on each of its goals.
Although produced following an extensive series of consultations - with statisticians, academic and civil society experts - the SDG indicator framework submitted to the Statistical Commission of the UN in March 2016 continues a process of narrowing the Agenda and limiting its universality. While in a few cases they contribute to the interconnectedness of elements needed to meet the targets, for the most part the indicators fail to address the complexity of the targets, at times distort their meaning and in a few cases serve to legitimize totally inadequate targets.
Despite the universal framing of the 2030 Agenda, the responsibilities of the rich, including extraterritorial responsibilities, remain largely outside the indicator framework. How is it possible to measure vulnerability to global power dynamics vs. power to shape them? Some countries are extremely vulnerable to the consequences of rules on debt or trade for example with little or no power to shape these rules. The same is true of global tax rules. How can progress by middle-income countries be measured without addressing this dynamic? Is there scope to correct this at national and regional levels?
This chapter looks at the two-stage process by which the indicators not already agreed upon in the first indicator framework were determined. Chapter 3.2 presents alternative measures that cover the breadth of the targets and indicator framework as a whole.
The politics of indicators
On 11 March 2016 the UN Statistical Commission1 approved “as a practical starting point” an initial set of global indicators submitted by the Interagency and Expert Group on SDGs (IAEG-SDGs), which had been charged with developing a global indicator framework.2 More than 80 UN Member States expressed dissatisfaction with this framework, raising concerns about their failure to adequately address the targets.
The Group of 77 (G-77) and China, for example, stated that the framework should encompass all of the 17 SDGs and 169 targets in a balanced and integrated manner, including Goal 17 on means of implementation (MoI) and stressed that the “indicators should be faithful and relevant to the 2030 Agenda and should not re-interpret targets”.3 In fact, the indicators for Goal 17 have proved to be the most difficult to identify throughout the IAEG-SDGs process (see below).
Several countries stressed the need for more disaggregated data. Given that neglected groups and areas tend to disappear in national averages, the fact that the framework now includes less data disaggregation rather than more is a failure to conform to the ambition of the 2030 Agenda, particularly regarding its overarching commitment to leave no one behind. How can we reach “the furthest behind first” if we don’t know who they are?
Other concerns went to specific goals. Least developed countries (LDCs) noted that the indicator on proportion of population using the internet fails to adequately capture the target on technology transfer or to measure the operationalization of the LDC Technology Bank set up for that purpose. The Community of Latin American and Caribbean States (CELAC) noted the inadequacy of the indicator to measure inequality, which as many civil society organizations have pointed out is limited to income per capita among the bottom 40 percent of the population, saying nothing about the top.
India raised concerns about the use of perception surveys or opinion polls as indicators for some of the targets, noting that they lack internationally accepted standards or guidelines and cautioning that they could be “overly subjective, imprecise and also prone to misuse”.4
The UN Statistical Commission requested the IAEG-SDGs to consider these and other reservations and continue to refine the framework. At the same time it emphasized the importance of “guaranteeing international comparability”.5 However, the UN Statistics Division (UNSD) has stated repeatedly that the global indicators are intended for global follow-up and review and are not necessarily applicable to all national and regional contexts. This reflects the recognition that there is widespread distrust on part of many developing countries that, notwithstanding such assurances, the emphasis on comparability will pressure countries to use the global framework as the starting point, thereby running the risk of multiplying its weaknesses (without securing its strengths).
The IAEG-SDGs, following a meeting in Mexico in March 2016 to assess availability of data for each of the indicators, passed the ball back to governments, saying in its report that “specific proposals for refinement of indicators mentioned by Member States” and “possibly reviewing those indicators that are determined to not completely cover the full scope of the target” is a job that “will not commence until after the indicator framework is adopted (and possibly a mandate for refinements/revisions is given) by ECOSOC and the General Assembly”.6 This leaves agreement on a revised indicator framework open for at least another year.
What challenges need to be addressed?
While the MDGs had a total of 21 targets and 60 indicators, but in practice focused primarily on a single target, the SDGs may confront a different problem. MDG 1 on poverty was considered achieved when the World Bank-monitored target of halving the number of people living on US$1.25/day was reached, even when the poverty profile of most developing countries remained much more nuanced. Similarly the goal of reducing gender equality was considered advanced when the target of universal primary education was reached.
In the case of the SDGs, however, despite pressure to try to limit the goals and targets, the IAEG-SDGs was asked to identify at least one (frequently more) indicator for each of the 169 targets, which to date has resulted in a list of 230 indicators and might end up with some 300 indicators. The challenge is now how to avoid evaluating progress on each of these separately without considering the way in which they need to be coordinated.
To meet this challenge and finalize the indicators, Member States that approved the SDGs will have to explain the intent of paragraph 17 of the 2030 Agenda which states that “(...) there are deep interconnections and many cross-cutting elements across the new Goals and targets (…)” and this reflects the “(…)integrated approach that we have decided on (…).”7
The indicators to measure progress on gender equality and on decent work for all, for example, are cross-cutting throughout the goals, and include those to measure the right to paid employment and to rights at work; to equal pay for work of equal value; to recognize and value unpaid care and domestic work; to reduce inequalities in income and social protection coverage; to measure the right to economic resources and ownership and control of land and property. Target 10.3 focuses not only on equality of opportunity but also of outcome, offering scope for civil society monitoring. The selection of the indicator on people’s experiences of discrimination and harassment to measure this target may be a starting point through which to capture the promise of “no one will be left behind.” In so doing, these indicators stretch the envelope, especially from the perspective of rights.
Several other targets are even more comprehensive, requiring a multiplicity of cross-cutting policies and potential results that cannot be captured within one or two indicators. However, the decision to limit the number of indicators, for which data was (or could be) available, means that in many cases only one element of the target has an indicator, often one that distorts the overall meaning, directly or by omission.
Some targets lack indicators entirely. This is the case with Target 1.4, to ensure equal rights to “economic resources, as well as access to basic services, ownership and control over land and other forms of property, inheritance, natural resources, appropriate new technology and financial services, including microfinance.” An indicator has been proposed to measure the proportion of population living in households with access to basic services, but the IAEG-SDGs pointed out that “there is no established methodology for this indicator” and UNEP (the UN Environment Programme) offered to “contribute to the definition of basic services as this is within the scope of UNEP's existing work on SDG ontologies.”8 But even if an acceptable assessment tool for basic services is developed, this is only one of the issues covered by this target, and access to property, technology and finances will also need to be monitored.
This approach risks viewing each of the 17 goals as nothing more than the sum of its separate targets and indicators and can be measured accordingly. By counting the trees, therefore, this approach risks hiding the forest.
This risk is obvious under Goal 10, to “reduce inequality within and among countries”. This should be straightforward, since despite the failure of governments to identify a specific target, there is broad political agreement that current inequalities have reached the point where they are impeding development and need to be reduced.
Yet, while many of the targets address the problem of inequalities the specific target to do so is limited to improving the income growth of the bottom 40 percent, with no mention of the top 1 percent. This omission is also apparent in the indicators, which fail to measure this gap, either with the well-established Gini index (which measures the extent to which household income/or consumption deviates from perfect equality), or the “Palma ratio” (the ratio between the income of the top 10 percent and the bottom 40 percent) that is also widely accepted and easier to understand. Data for both measures are available for most countries and are used in other reports, which suggests that despite the fact that the selection of the indicators is meant to be a technical process only, it is indeed highly political. This conclusion is strongly reinforced by the fact that the framework completely ignores inequalities among countries.
As noted, the indicators on implementation, both for Goal 17 and for the MOI targets in all the other SDGs remain among the most difficult. Many of them are still being debated and many of those already agreed miss the point or limit/distort the intention of the target. For Goal 1 on poverty, for example, to “ensure significant mobilization of resources from a variety of sources, including through enhanced development cooperation, in order to provide adequate and predictable means for developing countries, in particular least developed countries, to implement programmes and policies to end poverty in all its dimensions,” the indicators measure only the percentage of government spending that goes to poverty reduction programmes and the provision of essential services (education, health and social protection), saying nothing about development cooperation.
The same is true with Target 17.1, to “strengthen domestic resource mobilization, including through international support to developing countries, to improve domestic capacity for tax and other revenue collection”. The indicator, “total tax revenue/GDP”, ignores international support, not only through development cooperation but more importantly though global agreements and measures to curb tax evasion and illicit financial flows.
Target 17.3, to “mobilize additional financial resources for developing countries from multiple sources” is to be measured only by “foreign direct investments as % of total FDI + ODA” and “additional volume of remittances (USD)/GDP.” These indicators might artificially inflate the accounted contribution of developed countries. On the one hand the OECD itself recognizes that “microeconomic or macroeconomic impacts of remittances are controversial and the extent to which these flows contribute to development is still not clear.”9 On the other hand not every FDI contributes to development and if capital inflow weights positively in the balance of payments, it is the net balance what counts and thus outflows (profits, royalties, illicit financial flows, etc.) should not be ignored.
Moreover, an indicator on the percentage of tax paid by multinational corporations within host countries, suggested by civil society, was not adopted in the final framework.
The main point of Target 17.6 is to “enhance cooperation on and access to science, technology and innovation and enhance knowledge sharing, not only through existing mechanisms but also through “a global technology facilitation mechanism.” The proposed indicator, rather than assessing whether or not these procedures and mechanisms have been set up, measures “access to the WIPO Patent Database and use of the international IP system.” This is adding insult to injury, because the current global intellectual property system is precisely one of the obstacles that this goal seeks to overcome.
Moreover, given Member States’ commitment “to developing broader measures of progress to complement gross domestic product” and the inclusion of a specific target (17.19) to meet this, it is alarming that there is no adequate measure included in the current list of SDG indicators. So far, the only proposed indicator is “countries conducting population and housing census and achieving 100% birth and 80% death registration.”
Options that could be explored are well-being indicators like those adopted in Italy (see Box XXX) and environmental “footprint” assessments to show where countries are positioned in terms of the ecological sustainability of their development. These could illustrate the continued relevance of the principle of common but differentiated responsibilities (CBDR), and point out that while countries pursue universal SDGs their pathways and priorities to reach them will be different.
Beyond GDP in Italy
In February 2015 a group of Parliament members presented a bill entitled "Provisions for the use of well-being indicators in public policy-making". The objective, according to the introduction, is “introducing indicators of well-being, environmental sustainability, gender equality and social quality with means provided for by national law in the elaboration, adoption and assessment of public policies, so that they can be effective in improving welfare conditions for the country as a whole.”
This proposal takes one step further the work on alternative indicators to GDP. Italy’s “Equitable and Sustainable Well-being”, or Benessere Equo e Sostenibile (BES) in Italian, was adopted in 2013. The analytical framework was used to extend the analysis to the provincial and municipal levels, and the BES is now the reference measuring Italian well-being at all levels, for policy-makers, scholars as well as for civil society. The process of selection and refinement has led to a set of tested indicators on the basis of which synthetic indices have also been proposed to facilitate effective communication of results.
Synthetic indexes are computed for health, education and training, cultural participation, employment, quality of employment, economic hardship, income and inequality, social relations, security, homicides and subjective well-being. Thus, it is possible to assess the impact of the recent economic crisis on all of these dimensions of wellbeing in Italy, indicating that all have shown some deterioration. The level of income and employment decreased as expected, yet a more intense impact is shown for other linked phenomena such as the rise of small-scale criminality and the fall of cultural activities, demonstrating the way in which the crisis has had a negative impact not only on the economic life but also on the social fabric of Italy.
Excerpted from the Social Watch Italy Report 2016, authored by Soana Tortora, Jason Nardi and Tommaso Rondinella
With regard to policy measures, under Target 17.9, to implement effective and targeted capacity building to support national plans for sustainable development, the measure to “implement a policy mix” to achieve the goals that includes the elements of reducing inequality has been omitted to focus only on the dollar value of financial and technical assistance.
Key demands like enhancing global macroeconomic stability (which is important everywhere and not just for developing countries), or to “enhance the global partnership (among countries) for sustainable development” still lack agreed indicators.
The commitment to “respect each country’s policy space and leadership to establish and implement policies for poverty eradication and sustainable development” is left without an agreed indicator. Developing countries suggested to use a simple count of the “numbers of constraints” (conditionalities) that are embodied in ODA or loan agreements as well as investment and trade agreements.
Under Target 17.15 to enhance policy coherence, the proposed indicator “Numbers of constraints that are embodied in official development assistance or loan agreements, international investment agreements, regional trade agreements” etc., was revised to omit reference to constraints and reads now: “extent of use of country owned results frameworks and planning tools by providers of development cooperation”.
Moreover, in a number of cases, proposed indicators were revised to eliminate key concepts of sustainable development and its measures: Under Target 17.9 on international support for capacity building, the proposed indicator was simplified to omit reference to “implementing a holistic policy mix that aims at sustainable development in 3 dimensions (including reducing inequality within a country and governance)”.
Under Target 17.14 to enhance policy coherence, the single indicator was simplified to omit references to “countries ratifying fundamental ILO conventions and recommendations” and instead only measures the number of countries with “mechanisms in place to enhance policy coherence of sustainable development.”
Follow-up and review
The 2030 Agenda states that the primary responsibility for follow-up and review lies with Governments, and that at the global level the High Level Political Forum on Sustainable Development (HLPF) will have the central role in overseeing the process, and will also “promote system-wide coherence and coordination of sustainable development policies. It should ensure that the Agenda remains relevant and ambitious and should focus on the assessment of progress, achievements and challenges faced by developed and developing countries as well as new and emerging issues.”10
It is also “mandated to conduct national reviews and thematic reviews of the implementation of the Agenda, with inputs from other intergovernmental bodies and forums, relevant UN entities, regional processes, major groups and other stakeholders.”11
The 2030 Agenda rejected the concept of “accountability” in favour of “follow-up and review”. Could the failure of Member States to agree on a universal reporting and accountability process be somewhat mitigated by the continuing work of the IAEG-SDGs? It will be essential to assess all of the indicators in terms of who benefits and who is accountable. The Statistical Commission has requested the IAEG-SDGs to take into account the specific proposals by Member States on refining the indicators, many of which address the need to capture disparities at the top of the income spectrum and not just the bottom. Will this be an opportunity to adopt an indicator to measure or monitor reducing extreme wealth? Will it open the discussion on monitoring extraterritorial obligations?
Enormous disparities of opportunity, wealth and power…
The 2030 Agenda could further development, peace, and sustainability to the extent that it can address the root causes of multi-dimensional violence and reach the most marginalized. The addition of an indicator that measures disparities within countries along income, residential location, gender or ethnic lines could be a step forward. This figure should be included alongside the average measure to support efforts not only to increase or decrease the national average but also to decrease the gaps. The data is available, and was utilized in several of the last MDG reports, to measure disparities in income and/or location for key issues, such as working poverty, hunger, education, health, and access to clean drinking water.
Will the IAEG-SDGs’ ongoing review and refinement process revisit the indicators under Goal 17 proposed by several civil society organizations in a joint statement during the consultation process including those focused on “goals for the rich”?12
Several of these seek to assess constraints to policy coherence, including trade and investment treaties and loan agreements as well corporate tax avoidance and drains on the public purse through outsourcing development to the corporate sector. While Target 3.b specifically refers to the TRIPS provisions that allow developing countries to produce generic medicines, bilateral, and regional trade agreements typically include binding arbitration provisions that make that impossible. An indicator that should be added to assess the impact of trade on sustainable development would be the number of disputes brought against countries through trade and investment dispute settlement processes.
Target 17.17 on partnerships is another case where indicators need improving. The promotion of “effective public, public-private and civil society partnerships” will be measured by the money “committed on public-private partnerships”, not assessing whether those funds were actually disbursed nor their real contribution and impact.
There are ways to do this, for instance by introducing an indicator to measure the existence of binding human rights/environmental protection frameworks to regulate partnerships, including periodic impact assessments. In addition to proposing an indicator on contributions to PPPs by source, there should be also indicators to assess the value of public‐private partnerships in terms of their contribution to sustainable development. These include:
- the number of public‐(for profit) private partnerships that deliver greater value for achieving the SDGs than public or private finance alone;
- the number of public‐(for profit) private partnerships that include full transparency of contracts, terms, and assessment results, and are subject to the highest international environmental and social safeguards.
Reverse the slippery slope
The fact that the proposed indicators framework has been sent back for refinement is an opportunity for the statisticians to take these and other recommendations from Member States and civil society on board. While much is made of the distinction between the technical work of the IAEG-SDGs and the political decisions by Member States, it is clear that the border is more fluid.
Acknowledging this, the UN Statistical Commission also set up the High-level Group for Partnership, Coordination and Capacity-Building for post-2015 monitoring, partnership and coordination (HLG). The HLG’s report of January 2016 notes that it can help shape the interaction between the technical and political aspects of the work on indicators, and that it will define mechanisms to make recommendations to the IAEG-SDGs on strategic issues at the country level, including the use and interpretation of indicators and means of implementation.13
The 2030 Agenda states that “data and information from existing reporting mechanisms should be used where possible”.14 Does this open an opportunity for other reports, including shadow reports? This has become an accepted part of the reporting process for treaty bodies such as UN Committee on the Elimination of Discrimination against Women (CEDAW) and the International Covenant on Economic, Social and Cultural Rights (ICESCR), and allows for the broader representation of views from civil society organizations. While the number of reporting countries will not be automatic or comprehensive, such a development could lead to a mechanism for NGO reporting as part of the official process.
Looking at the overall process of elaborating the goals, targets, and indicators, the progression reveals a downward trend: with some exceptions, the set of goals are more ambitious than the targets, and the targets are more ambitious than the indicators.
The HLPF must face its responsibility to reverse this slippery slope.
Adams, Barbara/Judd, Karen (2016): 2030 Agenda and the SDGs: indicator framework, monitoring and reporting. New York (Global Policy Watch Briefing #10).
Civil Society Reflection Group on Global Development Perspectives (2015): Goals for the Rich: Indispensable for a Universal Post-2015 Agenda. Discussion Paper. Berlin/Geneva: Friedrich-Ebert-Stiftung.
G77 (2016): Remarks on behalf of the Group of 77 and China by Mr. Nontawat Chandrtri, Chargé d'Affaires and Minister Counsellor, Permanent Mission of the Kingdom of Thailand to the United Nations, at the 47th session of the United Nations Statistical Commission (New York, 8 March 2016).
India, Government of (2016): Statement delivered by Dr. T.C.A. Anant, Chief Statistician of India and Secretary, Ministry of Statistics and Programme Implementation, Government of India, (New York, March 8, 2016).
OECD DAC Working Party on Development Finance Statistics (2013): External resource flows beyond ODA: Remittances in the post-2015 DAC statistical framework. Paris (DCD/DAC/STAT(2013)19).
UN Statistical Commission (2016a): Report on the forty-seventh session (8-11 March 2016). New York (UN Doc. E/2016/24-E/CN.3/2016/34)
UN Statistical Commission (2016b): Report on the Third Meeting of the High-level Group for Partnership, Coordination and Capacity-Building for post-2015 monitoring (HLG for post-2015 monitoring). New York.
UN Statistics Division (2016): Third Meeting of the Inter-Agency and Expert Group on the Sustainable Development Goal Indicators. Report. New York (UN Doc. ESA/STAT/AC.318/L.3).
United Nations (2016): Provisional Proposed Tiers for Global SDG Indicators, draft as of March 24, 2016. New York.
United Nations (2015a): Report of the Inter-Agency and Expert Group on Sustainable Development Goal Indicators. New York (UN Doc. E/CN.3/2016/2).
United Nations (2015b): Transforming our world: the 2030 Agenda for Sustainable Development. New York (UN Doc. A/RES/70/1).
2 Cf. United Nations (2015a).
3 Cf. G77 (2016).
4 Cf. India, Government of (2016).
5 UN Statistical Commission (2016a), p 9.
6 UN Statistics Division (2016), para. 19.
7 United Nations (2015b), para. 17.
8 Cf. United Nations (2016), p.3.
9 OECD DAC Working Party on Development Finance Statistics (2013), para, 24.
10 United Nations (2015b), para. 82.
12 Cf. Civil Society Reflection Group on Global Development Perspectives (2015).
13 Cf. UN Statistical Commission (2016b), p. 3.
14 Cf. United Nations (2015b), para. 48.