Desperately Seeking Indicators: different players, different priorities

Three years into the implementation of the 2030 Agenda for Sustainable Development, concerns continue about stalled indicators, missing indicators and proliferating and potentially competing data sources, which makes it difficult to assess progress (see GPW Briefings #22: The Ups and Downs of Tiers: measuring SDG progress; #23: SDG Indicators-the forest is missing).

Initiatives abound in the shifting terrain of the generation, validation and use of data to satisfy the demands of a growing market of players. In addition to the work of the UN mandated Inter-agency and Expert Group on SDG Indicators (IAEG-SDGs), these concerns and challenges have drawn the attention of a number of official statisticians and practitioners.

Download this briefing (pdf version).

A new working paper by Steve MacFeely and Bojan Nastav, titled “You say you want a [data] Revolution’: A proposal to use unofficial statistics for the SDG Indicator Framework”, underlines the urgency of establishing a framework agreement for getting control of the dynamic but essentially fragmented data “revolution”. This timely proposal parallels the efforts of the UN Statistics Division, mandated by the UN Statistical Commission, to harness unofficial and open data, and integrate it into official statistics in a uniform way, subject to common principles.

Responding to the concerns about stalled indicators the IAEG-SDGs has made progress in three main areas:

  • advancing the methodological work needed to move some Tier III indicators to Tier II;1
  • identifying several proxy indicators by which to temporarily monitor those remaining in Tier III, prioritizing those with a deadline of 2020. The proxies will serve only to monitor the indicators until the methodology work is completed;
  • identifying gaps in 33 targets lacking sufficient indicators.

Further progress on all of these areas is expected to be made at its 8th meeting, to be held 5-8 November in Stockholm.2

The attention to speeding up progress on indicator measurement is no doubt driven by the enormous interest generated by the 2030 Agenda for Sustainable Development. However, it is difficult to assess what should be considered progress, who are the main players and how their (possibly competing) lines of measurement will play out.

Among the players are the proliferating big data initiatives, including the Global Partnership for Sustainable Development Data (GPSSD), Data2X (on gender), and the Digital Impact Alliance, all focused on maximizing the contribution of private sector data. The result has been an ad hoc use of cell phone or satellite data in various countries, so far without official certification and without an assessment of sustainability.

*************

A closer look at the IAEG-SDGs agenda

As of October 2018, the IAEG-SDGs had moved six indicators up to Tier II and identified custodian agencies (charged with monitoring and reporting) for all but three. There are now 93 Tier I indicators, 77 Tier II indicators, for which there is an agreed methodology but country coverage is insufficient and 57 Tier III indicators. (see the IAEG-SDGs website).

Tier movement

At its 8th meeting, the IAEG-SDGs will review 11 requests from custodian agencies to move indicators from Tier III to Tier II. These requests include three for Goal 10 (including the proportion of people living below 50% of median income, by sex, age and people with disabilities), two for Goal 4, and one each for Goals 2, 3, 12 (on mainstreaming sustainable development into educational curricula), 14, 16 (proportion of different population groups in national institutions) and 17 (economic dashboard).

Proxy indicators

The IAEG-SDGs will also endeavor to confirm a set of proxy indicators. The identification of proxies has been driven by the need to monitor progress on targets, on an interim basis while methodological work continues. The UN Statistics Division has made clear these will be temporary, and not a replacement for the official indicators.

While the list of proxy indicator candidates has not been announced, there are at least 40 indicators stuck in Tier III for which no agency request for reclassification has been received. These include two for Goal 10: the proportion of population who personally felt discriminated against or harassed in the previous 12 months (10.3.1) and ‘financial soundness indicators’ (10.5.1). While the custodian agency, IMF, has monitored financial soundness indicators at the country level for years, it reports that more work is needed on regional and global data aggregates. Given that the indicator has little to do with inequality, the use of a proxy indicator to measure the target – “improve the regulation and monitoring of global financial markets and institutions and strengthen the implementation of such regulations” – could be welcome.

Other indicators stuck in Tier III for which proxies could surely be considered might be the five indicators for Goal 13 on climate change, the five indicators on the related Goal 14 on sustainable use of oceans, seas and marine resources, the six indicators for Goal 16 on peaceful and secure societies and justice for all – among them total value of inward and outward illicit financial flows – and the seven indicators for Goal 17 on means of implementation, including the sole target on policy coherence. Other good candidates might be the three means of implementation indicators under Goal 1 on eliminating poverty, for which no custodian agency is yet established.

Additional indicators; filling in gaps

In addition to upward tier movement and the selection of temporary proxies, progress will be further measured by additional indicators, increasing the overall total. Recognizing gaps in the ability of the existing indicator framework to measure all of the targets, particularly those with different elements, the IAEG-SDGs drew up a list of 37 possible indicators for 33 targets and submitted it to the UN Statistical Commission in March 2017.  The IAEG-SDGs will finalize a select list of such indicators, giving particular consideration to those with an established methodology and some available data. The list of additional indicators will then be submitted to the UN Statistical Commission in 2020. Unlike the proxy indicators, which will be used temporarily instead of existing ones, these will add to the total list of indicators in the global framework.

Among the 37 additional indicators being considered are four for Goal 8 on employment and decent work and three for Goal 10 on inequality along with three for Goal 17 on Means of Implementation, including one on additional measures of progress to complement GDP, which was unfortunately dropped from an earlier indicator framework (for a full list of these see GPW Briefing #22: The Ups and Downs of Tiers”).

Integrating new data sources into official statistics

The need to tackle how to integrate new sources of data into official statistics has grown increasingly urgent. In 2018, the Big Data Project Inventory compiled by the UN Global Working Group on Big Data showed 109 separate big data projects. Both national statistical offices (NSOs) and international organizations are investigating a wide range of big data sources, from satellite imagery to mobile phone records. (see Big Data Project Inventory catalogue).

Illustrating this need is the fact the UN Sustainable Development Goals Report 2018 turned to a private sector report in order to measure corporate sustainability. Since the indicator for target 12.6 – to “encourage companies, especially large and transnational companies, to adopt sustainable practices” – remains at Tier III, the report states that the private auditing firm KPMG reports that “93 percent of the world’s 250 largest companies are now reporting on sustainability” (see GPW briefing #25: “UNSDG progress reports: how statistics play favorites”).

More common is the use of cell phone data which the UN Statistics Division reports is used in some countries as a supplement to various national surveys, such as poverty or disease patterns, which are expensive, labour intensive, and infrequent. The same is true for satellite data, which can capture the extent of deforestation for example in real time and avoid the exclusive reliance on costly and difficult-to-undertake surveys in remote areas.

The need to manage such data in a systematic way, subject to common principles and standards, has been on the agenda of the UN Statistical Commission since 2014, with the report, ‘A World That Counts: Mobilising the Data Revolution for Sustainable Development,’ by the UN Secretary-General’s Independent Expert Advisory Group on a Data Revolution for Sustainable Development. The UN Statistical Commission established a UN World Data Forum on Sustainable Development Data as the suitable platform for intensifying cooperation with various professional groups, such as NSOs, information technology and geospatial information managers, and data scientists among other representatives of government, intergovernmental organizations and civil society. The Forum held its first meeting in Cape Town in 2017 and its second in Dubai in October 2018 (see UN World Data Forum website).

In March 2018, the Statistical Commission also created a sub-group on Open Data, as part of the Friends of the Chair Group on the Fundamental Principles of Official Statistics, intended to provide guidance and support for integrating open data into official statistics at the country level. An Open Data Hub, to be created at Dubai, will include guidelines to address SDG data interoperability issues—thus bringing to the national level a challenge in terms of capacity and resources, especially regarding the necessary infrastructure. This group is also exploring partnerships to strengthen Open Data within the statistical system at country level.

As a complement the UN Statistics Division announced in March 2018 the creation of a “Federated System of SDG Data Hubs and Collaborative Platforms for Innovation”, based on the blueprint agreed under the Cape Town Global Action Plan. The UN World Data Forum will provide the space to review the implementation of this system of interconnected SDG data hubs.

In their detailed working paper, MacFeely and Nastav go a step further, saying (p. 7):

“Addressing the data gaps using only traditional approaches will realistically not achieve success. For this reason, we propose, not only using existing unofficial data as inputs to derive SDG indictors but also using already compiled unofficial indicators.”

Calling it a “risk management strategy” the authors develop a number of interesting proposals regarding the use and validation of additional (non-official) data sources:

  • to enable NSOs to certify the use of additional sources of data, as well as unofficial national statistics to compile official statistics to measure results for some indicators, and
  • to enable an agreed recognized body, mandated by the Statistical Commission, to review unofficial statistics to determine whether they are ‘fit for purpose’ to populate the global indicator framework, provided they meet international standards and are widely available.

They advocate these proposals as a way for both national and global statistical bodies to exercise some control over the currently unequal landscape, which heavily favours private and contracted sources.

The authors indicate some concerns regarding present practices – notably that by allowing some unofficial sources to be designated official, this may be the thin end of a wedge, ‘whereby the complication of official statistics is slowly outsourced or privatized and incrementally taken away from NSOs…’. They note the fact that initiatives such as the GPSSD and the UN Global Pulse are already competing with the UN, for funding and other resources, stating that: “a (cold) data war is already underway. There is a growing asymmetry in the resources available for the compilation of public/official and private/unofficial statistics and indicators.”

Global Policy Watch has described and commented with concern many times on these developments. The proposals would be a step forward in trying to eliminate such competition and tackle the imbalance between public and private resources. However, the question arises as to whether the statistics community, broadly defined, is increasingly identifying the implementation of the SDGs with the monitoring and reporting of data and statistics. In fact, four of the five indicators under the section of Goal 17 on “data, monitoring and accountability” focus on statistical capacity, including one (17.19.1) to measure the total amount of resources devoted to statistical capacity building in developing countries. There is no doubt that the need for capacity building for NSOs has increased exponentially, driven by the detailed and far-reaching SDG targets and indicators. However, the enormous attention to resource mobilization – both public and private – for this purpose raises the question of how funds are also being allocated to other development goals, and how closely data collection and reporting is linked to policy-making to correct lack of progress on these goals. Did Member States, in negotiating the 2030 Agenda, intend for such monitoring to be the main goal of domestic and global resource mobilization?

In the case of Tier II indicators, where data are not being systematically produced, MacFeely and Nastav propose that unofficial data be certified to “populate” these indicators. However, the lack of reporting on some indicators may be due to political considerations as well as technical ones, as in the case of Goal 16 indicators of corruption, bribery and institutional accountability in all regions, which the authors acknowledge are a gap. If governments and NSOs are reluctant to compile and report on these indicators why would they certify and legitimize proxies that do the same thing?

Facing the reality of new sources of data and statistics and their impact on measuring SDG progress, the working paper explores how to integrate them into official statistics at different levels in a uniform way. Without tackling the political dimension, does this proposal close one accountability gap while neglecting the important one between data and statistics and development progress in the implementation of the 2030 Agenda?

Notes:

1 Tier III indicators lack agreed methodology, those and Tier II are methodologically sound but lack sufficient country coverage, while only Tier I indicators meet both conditions and are thus reported as well as monitored.

2 As in previous meetings, this will comprise of a members-only session on the first day (5 November) followed by a plenary session during the next three days (6 - 8 November), to which “all countries, international and regional agencies and entities, and other stakeholders are invited to attend”.

Download this briefing (pdf version).

By Barbara Adams and Karen Judd.

Source: Global Policy Watch Briefing #27: Desperately Seeking Indicators: different players, different priorities Global Policy Watch.