The Ups and Downs of Tiers: Measuring SDG Progress

After two years of measuring for SDG implementation the emphasis has shifted from the pressure to develop a global indicator framework to the need for capacity development. This has generated a significant increase in interest in national statistical offices (NSOs) for data disaggregation, not only by income, gender and population group but also by municipal and neighborhood levels in an effort to ’leave no one behind’. The shift to implementation and capacity-building has also spawned a host of initiatives and partnerships, designed primarily to enable NSOs to integrate data from non-traditional sources, such as satellite imagery, mobile phones, and social media and scanning data.

Download this briefing (pdf version).

Member States at the 49th session of the UN Statistical Commission addressed the work of the Inter-agency and Expert Group on SDGs Indicators (IAEG-SDGs) and the High Level Group for Partnership, Coordination and Capacity-Building for Statistics for the 2030 Agenda (HLG-PCCB), along with a large number of other reports, ranging from household surveys and systems of national accounts to gender statistics, open data and big data for official statistics.1

Commenting on the IAEG-SDGs background document “Guidelines on Data Flows and Global Data Reporting for the Sustainable Development Goals”, Member States emphasized that consultations with NSOs should be regular and ongoing on each indicator and that more time was needed for feedback and revision; they urged that the Guidelines should come back to the Commission in 2019. They also drew attention to the large number of SDG indicators that lack agreed methodology for measurement (Tier III).

The UN Economic and Social Commission for Asia and the Pacific (UNESCAP) 2017 Statistical Yearbook points to the large number of data gaps in critical areas such as poverty, climate change, environment, gender, inequality, and governance. Overall only 50 of the 169 SDG targets are ready for progress assessment. Over half of the 230 indicators lack agreed measurement criteria (68) or sufficient data coverage (66) for regular monitoring or reporting or both.

Lack of progress on Tier III indicators was also a concern of civil society organizations (CSOs), who pointed to the lack of clarity on work plans for advancement to the next tier. Another concern was that many of the proposed indicators, particularly those under Goal 10 on inequality, dilute the aims of their targets.

These concerns were also shared by women’s rights advocates, who pointed out that the tweaking and refinements had weakened some of the indicators, particularly those related to sexual and reproductive health and rights. The 2018 UN Women (UNW) report “Turning Promises into Action: Gender equality in the 2030 Agenda for Sustainable Development”, points out that less than a third of the data needed for monitoring the gender-specific indicators are currently available, only 24 percent of the data available for gender-specific indicators are from 2010 or later, and only 17 percent have information for two or more points in time, allowing for trend analysis. The UNW report concludes that many of the gender-specific indicators rely on data collection mechanisms that are ad hoc or one-off exercises and are not integrated into national statistical plans and strategies. Further analysis of other goals might reveal similar problems.

While there is more openness to CSO and other stakeholder participation in the indicator refinement process, the volume and breadth of players and processes have made it difficult for CSOs to track and engage. CSOs that attended the meetings of the IAEG-SDGs during last year appreciated the opportunity to offer suggestions for revised indicators but noted that there was no feedback concerning decision-making about indicator refinement and revisions. Some Member States at the Commission meeting flagged concerns more broadly about the lack of transparency regarding decisions on the ups and downs of indicators between tiers.

As the focus of efforts to implement the 2030 Agenda shifts from identifying the global indicator framework to national-level capacity building, involving a multi-player set of initiatives, there is an apparent disconnect between the indicator measurement process and the high-level political forum (HLPF) review process now underway, a process that has garnered quantity and some quality engagement from Member States, CSOs and others.

Member States and CSOs alike are confronted with parallel processes whereby the HLPF has a multi-year programme to review clusters of the SDGs at the same time as indicators are being re-tiered and refined. This makes it unclear whether to press for better indicators such as those initially proposed by CSOs on public-private partnerships (PPPs) and inequalities. Would committing resources to improving relatively neglected or under-assessed indicators for Goals 10 and 12 be well spent?

Box 1: HLPF 2018 and 2019 SDGs for review

2018

SDG 6:    Ensure availability and sustainable management of water and sanitation for all

SDG 7:    Ensure access to affordable, reliable, sustainable and modern energy for all

SDG 11:  Make cities and human settlements inclusive, safe resilient and sustainable

SDG 12: Ensure sustainable consumption and production

SDG 15: Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss

SDG 17: Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development

2019

SDG 4:  Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all

SDG 8:  Promote sustained, inclusive & sustainable economic growth, full & productive employment & decent work for all

SDG 10: Reduce inequality within and among countries

SDG 13: Take urgent action to combat climate change and its impacts

SDG 16: Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels

SDG 17: Strengthen the means of implementation and revitalize the Global Partnership for Sustainable Development

This paper offers initial comments on the ups and downs of SDG indicators among the tiers with a focus on the substantial number of indicators lacking progress, including some that concern SDGs to be reviewed in 2018 and 2019. It also introduces some of the possible new indicators being considered. In addition to observations on the progress regarding the tiers and indicators, it looks briefly at some challenges for NSOs going forward, notably with regard to capacity building, engaging with the private sector and big data and linking indicators to policy-making. An assessment of the quality of the agreed Tier I indicators is offered in GPW Briefing 23, “SDG Indicators: the forest is missing”.

Tier classifications

As of December 2017, there are 93 Tier I indicators, classified as such because they are conceptually clear, with established methodology and standards and data regularly produced by 50 percent of countries and of population in every region where the indicator is relevant; 66 Tier II indicators, which are conceptually clear, with established methodology and standards, but data not regularly produced by countries; and 68 Tier III indicators, for which there are no internationally established methodology or standards yet available but are to be developed and tested. There are also five indicators with multiple tiers, owing to different components, while four indicators have been refined, breaking down targets with multiple components into separate parts, with indicators for each. Additionally, some 37 new indicators have been proposed for review.

Indicators stalled at Tier III

The large number of indicators stalled in Tier III includes 11 refined indicators adopted at the 48th Statistical Commission, which were newly classed into tiers by the IAEG-SDGs in March 2017 based on their methodological soundness and data availability.

A number of indicators stalled in Tier III are relevant for the 2018 HLPF review. In addition to those related to SDG 12 on consumption and production, which has by far the largest number of such indicators, these include water pollution and land degradation indicators (Goals 6 and 15), as well as access to energy (Goal 7), and those related to cities (Goal 11). Indicators that relate to SDGs under review at the 2018 and 2019 HLPFs are indicated in bold.

Goals that have over 50 percent of their indicators stalled in Tier III include:

  • Goal 12 on consumption and production, with 10 out of 13 indicators still in Tier III, including two on migrant workers’ rights, three on jobs and labour rights, one on global food loss index; one on material footprint, one on fossil fuel subsidies and two on recycling, both of which remain inadequate.
  • Goal 13 on climate change, with five out of 8 indicators still in Tier III, ranging from the adoption of strategic plans to address climate change to measures of their implementation and communication.
  • Goal 14 on marine ecosystems, with 7 out of 10 indicators still in Tier III, ranging from marine acidity and sustainable fisheries to floating plastic debris to expenditure on conservation and ecosystems. In addition, one of the stalled indicators, 14.b.1 relates to the recognition and protection of access rights for small-scale fisheries.
  • Goal 17 on means of implementation, with seven indicators still in Tier III, including 17.3.1, Macroeconomic Dashboard,2 for which the custodian agency is the World Bank; 17.14.1, policy coherence for sustainable development, for which the custodian agency is the UN Environment Programme, 17.17.1, financial commitments to civil society partnerships; and 17.18.1, proportion of national sustainable development indicators with full disaggregation when relevant to the target.

Indicators for SDGs scheduled for review at the 2019 HLPF stalled in Tier III include:

  • Goal 10 on inequality, with 5 out of 11 indicators still in Tier III, including the proportion of population living below 50 percent of median income, by sex, age and disabilities (10.2.1) for which the custodian agency is the World Bank; and Financial Soundness Indicators (10.5.1) for which the custodian agency, the IMF, reports that more work is needed on regional and global data aggregates. However, as civil society points out, the indicator has little to do with inequality; and
  • Goal 16 on inclusive institutions and access to justice, with six indicators still in Tier III, three of which focus on perceptions of public decision-making and public service delivery, and one on the value of inward and outward illicit financial flows. In addition, the revised indicator 16.4.2 addresses illicit arms seizure.

In two cases the tier classification reflects the fact that the indicator has been revised to be closer to the target, and therefore has been classified for the first time in Tier III:

  • SDG 8 Indicator 8.8.2 - a revised indicator on the level of national compliance with labour rights based on International Labour Organization  sources and national legislation, by sex and migrant status, has replaced the original indicator on the number of ILO Conventions ratified, by type of convention; and 
  • SDG 16 Indicator 16.4.2 - a revised indicator on the proportion of seized, found or surrendered arms whose illicit origin or context has been traded or established by a competent authority replaced an indicator that measured only  small arms and did not measure illicit nature. While this indicator is now closer to the target, for progress to be made there should be a complementary target on arms sales, legal and illegal, and from which country.

In addition, three Goal 1 indicators (1.a.1, 1.2.3 and 1.b.1) intended to assess progress on means of implementation on poverty eradication, still lack a custodian agency, leaving them to stagnate in Tier III. The IAEG-SDGs has indicated that if an indicator remains without a custodian agency, it may be refined or removed at the comprehensive review in 2020. This could result in failing to measure the multidimensional nature of poverty, adopted in the 2030 Agenda, thereby undermining the integrated nature of the SDGs and risking a slip back to the silo-specific orientation of the MDGs (see GPW Briefing 23, “SDG Indicators: the forest is missing”).

Up from Tier II to Tier I

In 2017, responding to concerns that some countries will use the lack of data coverage as a way to avoid reporting on an issue, the IAEG-SDGs identified those indicators whose development is quite advanced and could be ‘fast tracked’ into Tier I. Following a first-time review of global data availability, 19 Tier II indicators were moved up to Tier I, which means there is sufficiently wide coverage to begin reporting, while 18 remained in Tier II.

Tier I indicators reflect this review  process, along with previous work done in connection with the eight Millennium Development Goals (MDGs), which still predominate among Tier I indicators. However, the number of indicators to be monitored for the MDGs has increased greatly, reflecting the far greater breadth and ambition of the SDG targets.  A large number of Tier I indicators are also in new goals, such as access to energy and industrialization.

The dynamics of the indicator selection process and their classification into tiers, has been more in line with / responsive to national efforts to realize development, not just accomplish a few goals, and has benefited greatly from the active engagement of civil society. It is important that the process of upgrading the indicators speeds up in order to make sure that reporting is not done only on a limited set of Tier I indicators, which by no means measure up to the scope and ambition of the 2030 Agenda (see GPW Briefing 23, “SDG Indicators: the forest is missing”).

Indicators that moved up in 2017 include six indicators for Goal 3 on health and wellbeing, which has been championed by high-level initiatives and focus largely on mortality, along with health emergency preparedness, which has become a global concern. Another indicator that has moved to Tier I is the proportion of individuals who own mobile phones, by sex (5.b.1), which reflects the participation of mobile phone companies through data partnerships.

Also moving up to Tier I is indicator 15.6.1 on number of countries with legislative administrative and policy frameworks to ensure fair share of benefits from genetic resources. However, this indicator is focused only on domestic frameworks, and should include impact of global treaties, protections, redress, and so on.

Tier I indicators that lacked sufficient data coverage and were relegated to Tier II include one on social protection floors (1.3.1), two on food security (2.1.2 and 2.a.1), two on gender equality (5.3.1 on early marriage and 5.3.2 on female genital mutilation), two on water and sanitation (6.1.1 and 6.2.1) one on land ecosystems and biodiversity (15.9.1), and three on violence, trafficking and institutional corruption (16.2.1, 16.2.2 and 16.5.1). While in some cases the lack of adequate coverage may be due to insufficient technical capacity, it also may reflect the fact that some indicators, especially those on gender, are not high priorities for NSOs or line ministries.

Up from Tier III to Tier II

In addition to the 18 remaining in Tier II after the global review of data availability, 21 indicators moved up from Tier III to Tier II including two for Goal 1 on poverty eradication; one for Goal 2 on food security; five for Goal 5 on gender equality; two for Goal 15 on terrestrial ecosystems; one for Goal 16 on effective and accountable institutions; and one for Goal 17 on means of implementation, which is reviewed every year at the HLPF.

Proposed additional indicators

Of potentially greatest interest to CSOs, there are 37 proposed new indicators currently under review for 14 goals. The IAEG-SDGs will review these to determine those to be included in an open consultation to be held before the end of 2018. Following any changes or deletions, it will hold a second open consultation and by the end of 2019 prepare a final proposal for the Statistical Commission 2020, giving preference to indicators with an established methodology and available data.3 The final proposal will be posted on the UN Statistics Division (UNSD) website.

The possible new indicators include one for Goal 1 on poverty reduction, one for Goal 6 on water and sanitation, one for Goal 7 on access to energy, and three for Goal 16 on access to justice and inclusive institutions. There are also four proposed new indicators for Goal 8 on employment and decent work and four for Goal 10 on inequality, as well as three for Goal 17 on means of implementation.4 In addition, six new indicators are being reviewed for Goal 3 on health and well-being.

Some indicators that are of concern to human rights advocates and are pertinent to current and upcoming reports and debate are listed in Box 2.

Box 2: Proposed new indicators (For a complete list see E/CN.3/2017/2, Annex IV)

Target 1a on resource mobilization in order to implement programmes and policies to end poverty: to indicators on domestic resource mobilization, add indicator on ‘international cooperation for education, health and social protection’

Target 4.1 on completion of primary and secondary education, to an existing indicator with multi-components, some of which are stalled in Tier III, add two new indicators, including ‘the number of children not in school’ and ‘the number of guaranteed years of free and compulsory education mandated by law’

Target 6.4 on water-use efficiency add indicator on ‘the number of individuals who experience water stress or water shortages’. CSOs have pointed out indicators on perception such as this one this should be disaggregated by sex, age and income level at a minimum.

Target 7.1 on access to affordable and sustainable energy for all, to indicators on proportion of population with access to electricity, add an indicator on ‘affordability of electricity’

Target 8.5 on full employment and decent work: to unemployment data by sex, age and disabilities, add a new indicator on ‘additional aspects of decent work’

Target 8.10 on expanding access to financial services: to indicator on bank branches per 100,000 adults and ATM machines, add an indicator on ‘access to social insurance’

Target 10.2 on promoting social, economic and political inclusion: to indicator on proportion of population living below 50 percent of median income, add an indicator on ‘political inclusion of all’

Target 10.4 on fiscal, wage and social protection policies and the achievement of greater equality, to indicator on labour share of GDP comprising wages and social protection transfers, add an indicator on ‘economic inequality’

Target 10.7 on safe migration, to indicators on recruitment costs borne by employee and number of countries that have implemented well managed migration plans, add an indicator on ‘deaths, injuries and crimes committed against migrants’

Target 12.6 on encouraging companies, especially TNCs to adopt sustainable practices and include information in their reporting cycle, add indicator on ‘sustainable practices’

Target 16.3 to promote the rule of law at national and global levels and ensure equal access to justice, add indicator on ‘access to civil justice’

Target 16.6 on effective, accountable and transparent institutions, to share of population satisfied with their last experience of public services, add indicator on ‘trust in different public institutions’

Target 17.19 on initiatives to measure progress of sustainable development that complements GDP and support to statistical capacity building in developing countries, add indicator on ‘additional measures of progress on sustainable development that complement GDP’

Challenges of capacity, and demand and supply of data

The scope of capacity development ranges from data disaggregation to skills to negotiate contracts with private sector entities, like phone companies. The UNSD reports that the use of cell phone data is already used as a supplement to a myriad of surveys, like poverty, relocation, disease patterns. In many countries the NSOs are independent agencies but they are related in different ways to different line ministries, including Finance, Education, Agriculture and Health.

The capacity to negotiate contracts is also a factor in establishing data partnerships. Statistics Ireland, for example, partnered with its Ordnance Survey office to start building a ‘data hub’. The advantage, as the Statistics Ireland representative explained, is that the Ordnance Survey office had a preexisting relationship with Esri, an international private sector firm that supplies geographic information system (GIS) software, which greatly facilitated licensing agreements.

Esri, and its partners (Microsoft, Public Foundation, Voltaic Systems, Sharemeister, Computer Associates) are working in a number of pilot countries, including Ireland and Mexico, as well as several low-income countries. The UNSD considers this an ‘entry point’ with the private sector and is now working with Esri to set up a Federated Information System for the SDGs (FIS).

At the national level, the Division explains the main purpose of the FIS “is to strengthen interconnectedness of data providers, analysts, and users from multiple communities…to enable the use of high-quality data and information by making the data accessible to different audiences in a convenient and engaging manner, maintaining its credibility, and putting the needs of users at the center”.  It is also designed to enable national SDG data hubs “to scale and connect with each other and with regional and global data hubs”, allowing national data to be “fully visible, accessible, and usable also at the global level”.5

There is increased emphasis on efforts to communicate data and statistics to the media and the public and make them accessible through open data. Member States showed support for this at the 2018 Statistical Commission meetings, although different points were emphasized. The UK said: “Of course, we should be open, but we have a particular interest in ensuring that data held by others is also open”, while Tunisia, on behalf of the Africa Group stated: “Countries remain hugely dependent on donors’ support for delivering on open data initiatives. Sustainable capacities and skills are not being built in-house as expected.” And Suriname pointed out: “We have not heard anyone speak against open data. Still, we must make sure that openness does not share the same fate as coordination – while most of us believe that coordination is a good thing, few of us want to be coordinated.”

The emphasis on partnerships and communication strategies also raises the question of how much NSOs will be expected to take on, and how to prioritize. Are the NSOs being overstretched as statisticians – and facilitators, mediators, and message crafters? At the Statistical Commission meeting it was pointed out that “as the data ecosystem evolves, statisticians need to become information managers and not just producers of statistics”.

Different views on big data for development: who is winning and who is losing?

Greatly adding to the capacity building challenges is the need to integrate so-called ‘Big Data’ into the data collected at country level. As there is already a proliferation of partnerships and initiatives on Big Data, a Global Working Group on Big Data for Official Statistics (GWG) has been set up to identify the proliferation of partnerships and initiatives on Big Data and try to “harness their use in research and capacity building” for statistics production. The appeal of this data is widespread. CARICOM pointed out that given the region’s susceptibility to natural disasters, satellite imagery can be very helpful.

Presenting the report of the GWG, including progress made on setting up a GWG Platform under the auspices of the Commission, Denmark stated: “Several countries and private sector companies have committed to projects on data, projects which will be executed in the next 18 months as proof of data collaboration.”

The UK echoed this saying: “We will move forward to set up a range of collaborative, innovative projects”, and added: “Newly emerging research and development partnerships, focusing on use of big data for statistics, will allow us to explore the options of future business case, which are of benefit to all partners.” The USA, outlining its use of big data in its own statistics stated: “The platform would be a very useful tool to promote collaboration, innovation, and knowledge transfer.” The Netherlands, noting that surveys are not adequate methods to measure internet-based operations like Airbnb and Uber, stated: “We are supportive of exploring possible collaboration with tech companies in the international platform… It is extremely important to focus on delivering concrete results in the short run.”

A number of countries – notably the USA, the UK and the Netherlands – are very supportive; others – such as Switzerland and Malaysia – urged caution, noting the concerns around confidentiality as well as the costs of adding yet another task – data management—to the NSO roles and responsibilities. Switzerland said: “We strongly believe that the use of this platform should not be compulsory and national legislation regarding data protection should be given precedence.” Malaysia stressed challenges in managing, saying: “The data has not been managed and cannot be used for analytics. The ability to actually manage the data in a reasonable time frame is costly in terms of wasted time and wasted storage capacity.”

France stated its reservations on the platform to centralize data on the global level, stating: “It is not completely clear to us what is envisaged. It seems to us that the global centralization of data has risks.” Romania echoed this, saying: “We think that there is currently a paradox in official statistics. Most statistics are offered for free, but there are more problems for accessing metadata that could be used to produce statistics and to reduce the costs of official statistics. Concerning this paradox, the question is who is winning and who is losing.”

Data-policy interface

As the monitoring and review process continues, it is important to address the gaps and unevenness in the process as a whole – one of which concerns a disconnect between the review process of the HLPF and through the Voluntary National Reviews (VNRs), and the monitoring-by-indicators process.

To date most of the VNRs have consisted of government commitments and strategies for implementing the 2030 Agenda, rather than progress on implementation. The UN Committee for Development Policy (that provides advice and recommendations to ECOSOC on long-term development issues) examined the 22 VNRs submitted in 2016 and 38 of the VNRs submitted in 2017 to see whether they had a strategy to engage with the commitment to ‘leave no one behind’ and, in particular, how they defined the inequalities impeding this commitment – horizontally in terms of equal opportunities or vertically in terms of income and wealth.6 The Committee will continue this analysis on cross-cutting issues in order to highlight trends in the implementation process and to highlight best practices.

The High Level Forum on Official Statistics convened by the Statistics Division prior to the Statistical Commission meeting addressed ways to establish more systematic communication between statisticians and policy-makers. The Vice-President of the ECOSOC Bureau urged statisticians to “participate fully in political discussions at the global level and inter-governmental process to ensure that strong statistical systems and proper use of data are a top priority for the full realization of the 2030 Agenda”. It was emphasized that “statisticians can and should be engaged throughout all reviews at the HLPF, and the data statisticians report to the HLPF should be used to shape evidence-based policy recommendations”.

Stressing that “data produced must be policy-relevant and not exist for its own sake”, the session emphasized that NSOs should put a greater effort into data analysis and its application to policy-making. It was pointed out that while strong data generating capacity already exists in many countries, the capacity to analyse it is weak.

The existence of better data and statistics does not guarantee their use, including in the policy-making process. Engagement across sectors and professions as well as public awareness, transparency, and accountability are essential elements in closing the science-policy gap. The 2030 Agenda has provided momentum for capacity development and can be a driving force in bridging gaps in the indicator framework and its policy value, but the high expectations generated must show results to stave off disillusionment.

Notes:

2 The Macroeconomic Dashboard is a replacement for “current account surplus and deficit/GDP” in measuring macro-economic stability, including through policy coordination and policy coherence”. See:  https://unstats.un.org/sdgs/files/meetings/iaeg-sdgs-meeting-02/Statements/UNSSO%20statement_Goal%2017%20-%20Oct%202015.pdf

4 CSOs, while welcoming the additional indicators under Goal 10, have repeatedly noted that there is still no indicator to measure inequality between countries, proposing an indicator based on either the Gini coefficient or the Palma ratio which will not require additional data from countries but will provide a crucial guide to the effectiveness of the entire agenda. 

Download this briefing (pdf version).

By Barbara Adams and Karen Judd

Source: Global Policy Watch.