Measuring the SDGs: Who controls the process, who owns the results?

Statisticians from around the world, meeting at the UN Statistical Commission in March, will again take stock of progress in the world of data over the previous 12 months, largely driven by the 2030 Agenda for Sustainable Development. The official report on filling the gaps in the global indicator framework—a clear priority of the 2018 Commission—show that while some progress has been made much has stalled. Gaps and tensions continue over the selection and interpretation of indicators, the data to fill them, the selection of partners as well as control of the process and ownership of the results.

These struggles go back to the negotiations over the 2030 Agenda and its goals and targets, and have continued into the effort to define the global indicator framework.  A special edition of the Global Policy Journal details the complex power dynamics involved throughout this process. Contributors show that the selection of indicators does not depend purely on technical considerations but ultimately concerns political questions of competing priorities among a range of different players. One proposal, outlined below, argues that national statistical systems urgently must take charge of this process, and shows how they can do it.

Download UN Monitor #02 (pdf version).

Indicators and Tier Classification

Stepped-up methodological work on the global indicator framework since the 2018 Commission has enabled a total of 16 indicators to move up from Tier III to Tier II, so that data collection can begin; there are a total of 84 indicators now at Tier II, up from 77 in November 2018. Several indicators have also moved up from Tier II to Tier I, meaning there are a sufficient number of countries compiling this data to enable monitoring and reporting. There are a total of 101 Tier I indicators, up from 93 in November 2018. But, as of December 2018, a total of 41 indicators remained stalled in Tier III.

The Interagency and Expert Group –Sustainable Development Goals (IAEG-SDGs) report states: “Given the urgency of reclassifying the remaining Tier III indicators, the IAEG-SDGs will review additional tier reclassification requests at virtual meetings from December 2018 to February 2019” and at additional meetings throughout the year. An update on the reclassification of these indicators will be given verbally at the presentation of the framework to the 50th session of the UN Statistical Commission.

For the remaining Tier III indicators, proxies have been reviewed for those with a 2020 deadline, those that concern means of implementation, and those for Goals 12-14 for which more than half remain at Tier III.  The IAEG-SDGs has identified proxies for nine of these (see box): three concern climate change (Goal 13); three relate to marine ecosystems (Goal 14); one relates to poverty reduction (Goal 1); one relates to cities (Goal 11) and one relates to waste reduction (Goal 12).

Tier III indicators without proxies include those for six of the seven goals that will be reviewed at the 2019 High Level Political Forum. For Goal 8, on inclusive growth and decent work, there are still two Tier III indicators: 8.4.1, on material footprint per capita and per GDP, and 8.9.2, on sustainable tourist jobs. For Goal 10, on reducing inequalities, one, 10.3.1, on the proportion of different population groups experiencing discrimination or harassment, which is also an indicator under Goal 16, remains at Tier III.

For Goal 16, on peaceful and just societies, there are still five Tier III indicators, including 16.4.1 on the value of inward and outward illicit financial flows, 16.1.2 on conflict related deaths, and three on representation in or discrimination by various institutions. And for Goal 17, means of implementation, there are still six Tier III indicators, although one at least, on policy coherence, has moved to Tier II.

Additional indicators – maybe

Regarding the possible 37 additional indicators drawn up in 2017 to fill in gaps in the global framework, the IAEG-SDGs agreed that the 2020 comprehensive review would consider additional indicators “only in exceptional cases when a crucial aspect of a target is not being monitored by the current indicator(s) or to address a critical or emerging new issue that is not monitored by the existing indicators, or when a whole Goal has very few Tier I or Tier II indicators for the follow-up”. The objective is “to maintain the same number of indicators currently in the framework in order not to alter significantly the original framework, which is already being implemented in most countries and not to increase the reporting burden on national statistical systems” (para 29).

Big Data – Geospatial and Earth Observations

In the last year there has also been accelerated activity on Big Data, especially on geospatial and earth observations. The first UN World Geospatial Information Congress, held in China in November 2018, brought together academia, civil society and the private sector to explore the potential of big data to “locate, map, view, measure, analyse, model and monitor” global challenges. As such, it represents a conscious effort to link big data, including earth observation and satellite data, to policy measures—so-called ‘evidence-based policy making’.

Global Platform of trusted data, methods and learning for official statistics

Created in 2014 by the UN Global Working Group on Big Data for official statistics, the Global Platform has “evolved from a concept of the Global Working Group into a reality, with delivery of data, methods and learning” (para 6). It holds an increasing number of data sets, such as Landsat and Sentinel data, trial satellite data from Planet.com, AIS ship positioning data and ADSB aircraft positioning data and offers services such as various cloud servers, geospatial analytics services and Jupyter Notebook (para 8). Countries and agencies alike have drawn on these to measure SDG targets: the UK for indicator 9.1.1 on rural population access to all-season roads; the UNEP for indicator 6.6.1, on change in extent of the water related ecosystems over time, which has now moved from Tier III to Tier I.

The Working Group reports that the next task is “to more precisely define and agree on the concepts of its four basic pillars: trusted data, trusted methods, trusted partners and trusted learning. This implies agreement on the ownership of and access to the various large data sets on the Global Platform, whether data and algorithms need to be ‘open’ and how software, services and tools will be ‘Platform independent’“(para 6).  With reference to the use of mobile phone data for official statistics, Eurostat is seeking to clarify legal aspects and enable multi-mobile network operator processing and analysis (para 13).

Harnessing Big Data? – another proposal

The need to tackle how to integrate new sources of data into official statistics has grown increasingly urgent. Taking up this challenge is a working paper by Steve MacFeely and Bojan Nastav, ‘You say you want a [data] Revolution’: A proposal to use unofficial statistics for the SDG Indicator Framework. The paper underlines the urgency of establishing a framework agreement for getting control of the dynamic but essentially fragmented data “revolution”. The authors offer a number of proposals, designed to enable both national and global statistical bodies to exercise some control over the currently unequal landscape, which heavily favours private and contracted sources.

One of these would go beyond using existing unofficial data as inputs to derive SDG indictors, as is currently being done, to also adopt “already compiled unofficial indicators” (p. 7). Calling it a “risk management strategy”, the authors also propose the establishment of “an agreed recognized body, mandated by the Statistical Commission, to review unofficial statistics to determine whether they are ‘fit for purpose’ to populate the global indicator framework, provided they meet international standards and are widely available.  Acknowledging the risks that this may be regarded as an admission of failure on the part of NSOs, they point out that “a (cold) data war is already underway”, with “growing asymmetry in the resources available for public/official and private/unofficial statistics and indicators”. They argued that the risk of “reputation damage” to official statistics arising for certification of unofficial data must be balanced against that arising from official statistics failing to deliver on Agenda 2030 (p.13).

Statistics-policy nexus

Should the UN appoint a senior statistician (AS-G level) with a mandate to coordinate activities of statistical units in funds, programmes, specialized agencies, and secure a stronger voice for statistics in the UN system policy process. This option is presented by the High Level Group for Partnership, Coordination and Capacity Building for statistics for 2030 Agenda background paper: “Modernization of the United Nations Statistical System: A more effective and efficient UN statistical system in the era of the 2030 Agenda for Sustainable Development and beyond”.

First floated at 2018 UNSC meeting in 2018, this is one of a number of proposals to address the policy-evidence gap, at global and national levels. The background paper asserts:  "In the era of the 2030 Agenda, different entities are producing metrics and indexes for SDG monitoring that are not aligned to the official SDG indicator framework and are not supported by any intergovernmental or country-led process. This is contravening ECOSOC Resolution 2006/64 and leads to the under-utilization of official statistics, which is in turn a strong disincentive to capacity building efforts." 

The substantive and complex power dynamics involved in closing the evidence-policy gap are examined in a special edition of Global Policy Journal, Knowledge and Politics in Setting and Measuring SDGs, edited by Sakiko Fukuda-Parr and Desmond McNeill.  The 2030 Agenda was hammered out in the context of conflicting visions, ideas and interests between developed and developed countries, requiring negotiation and trade-offs.  Journal contributors explore how these differences did not end with the adoption of the SDGs but continued -- and are continuing -- in the way that the goals and targets are interpreted in the selection of the indicators to measure progress towards achieving the SDGs. Collectively they warn that “governance by data and indicators can alter meanings of social objectives, shift power relations, reorganize national and local priorities, create perverse incentives and create new narratives”.

Have the energy and struggles, to say nothing of resources, going into the selection, compilation, monitoring and reporting of data and statistics, deflected attention from the goals they are meant to measure? Among the most important are sustainable production and consumption, reducing inequalities (income and non-income), creating sustainable livelihoods, preventing conflict and halting and reversing the impact of climate change. Governments have recognized the urgency of tackling these enormous challenges, but UN deliberations continue to govern by “counting” dollars and data rather than reshaping the global frameworks – of taxation, job creation, investment, dispute resolution, and so on – that are essential for meeting commitments in the 2030 Agenda.

Download UN Monitor #02 (pdf version).

By Barbara Adams and Karen Judd

Source: UN Monitor #02: Measuring the SDGs: Who controls the process, who owns the results? Global Policy Watch.