Methodology

Monitoring and preparation of reports  

The country partners with ECNL's support prepared the country reports in several key stages.

Data collection and analysis

All six partners conducted extensive desk research and reviewed numerous relevant legal acts, amendments, reports, and articles. In addition, they implemented other data collection methods, namely:

• Interviews with relevant stakeholders: The partners conducted a total of 22 interviews with mainly CSOs.
• Focus group discussions: The partners organised a total of 9 focus group discussions with 76 CSO participants coming from both the capitals and different regions in the countries.
• Brief consultative exchanges: The partners in Moldova conducted brief consultative exchanges with 13 CSO experts to clarify isolated aspects of certain issues.
• Request for access to information: Partners in Georgia submitted a total of 15 requests to access information on the distribution of state grants to CSOs and received responses from 7 different institutions. Partners in Armenia submitted and received a total of 3 requests to access information to different institutions.

Drafting reports and validation process

The partners prepared draft country narrative reports based on the data analysis under each Standard within Law and Practice of the 11 areas that are subject to monitoring.

• ECNL reviewed the draft country narrative reports in several rounds in terms of completeness, accuracy, logic of argumentation, and quality of findings. The researchers implemented ECNL’s suggestions and instructions and revised the report.
• After the review process, the country partners sent the narrative reports to the Advisory Board members in their countries for review. They also organised discussions with the Advisory Board members to receive individual feedback. The partners then finalised the reports incorporating recommendations received from the Advisory Board and shared them with ECNL.

Limitations

• The reports cover the period mainly from January to December 2023. The important developments for civil society that occurred between the period of data collection and the finalisation of the report were included in the executive summary of the report but were not considered when assigning scores.
• The reports do not explicitly cover the environment of CSOs in the breakaway regions (for example, the Transnistrian region of the Republic of Moldova). These territories are outside of the control of the constitutional authorities of the respective countries.
• The preparation of the report in countries where civil society is significantly limited and restricted, or operates in exile, required increased flexibility in data collection. For example, in Belarus, independent civil society researchers faced numerous challenges: safety concerns, lack of published information for the sector, etc. Working from different jurisdictions impacted their efforts to obtain reliable data and analysis on the developments.

Scoring process

Measurement tool for the scoring process

For the scoring process, a 7-point scale is used. The researchers and Advisory Board members assigned scores for each specific standard (separately for Law and for Practice). The score was assigned based on the research findings on the quality of legislation and the application of the specific standard in practice.

The extreme values of the scale are conceived as the extreme/ideal situation or environment. For example, (1) is an extremely unfavourable (authoritarian) environment, while (7) is an extremely favourable (ideal democratic) environment for CSOs. The scores in 2023 are compared against the 2022 scores in this regional report.

Stages of the scoring process

The scoring process was conducted in three main stages:

1. Country researchers assigned scores. In each country, country researchers provided the scores for each standard in each area of the narrative country report, taking into consideration the country context and the regional perspective.
2. Advisory Board members in each country assigned scores. 53 Advisory Board members assigned scores, and the country partners collected, checked for completeness, and organised the scores in an Excel template.
3. Preparation of final scores. ECNL, together with the methodology experts, applied the following calculation method:

• An arithmetic average is calculated from the Advisory Board members’ scores for each standard.
• The final value of each standard is then calculated according to a formula in which the reference score participates with 50 per cent, and the Advisory Board members’ average score with 50 per cent.
• The value of each area is then calculated as the average value of the final values of each standard

FAQ

1.    How is the CSO Meter scoring different from other indexes?

• Is tailor-made from the region, for the region;
• Has standardised methodology for monitoring and scoring;
• Provides a comprehensive analysis that considers global and regional trends and emerging issues ;
• Offers forward-looking evidence-based tool for advocacy;
• Comes with a highly participatory monitoring and scoring process;
• Provides a continuous monitoring and early warning system;
• Responds to civic space challenges with monitoring that is based on international standards.

2.    How does the scoring take place?

Check out the infographic describing the process.

3.    What do the scores mean?

For the scoring procedure a 7-point ordinal scale is used - 1 being the lowest and 7 being the highest score (which are defined separately for Law and Practice). The extreme values of the scale (anchors) are conceived as extreme/ideal situations or environment. For example, (1.0) is an extremely unfavourable (authoritarian) environment, while (7.0) is the extremely favourable (ideal typical democratic) environment for the CSOs. The scorers would normally not assign these end-values while scoring and they should have in mind that the scores pertaining to the countries in the region are measured not only against the countries of that region, but also against the ideal typical situations.

Explanation of how the CSO Meter standards should be interpreted and assessed against international standards and good practices, with concrete examples, is available in the CSO Meter Tool.

4.    What is the role of the Advisory Board?

The Advisory Board (AB) in each country is composed of 10 expert representatives of key local stakeholders. The Advisory Board helps ensure that the CSO Meter country reports and their findings, conclusions and recommendations reflect the overall vision and needs of the CSO sector. The Advisory Board members have two main tasks:

1. The first task is to review the country narrative reports, including the recommendations prepared by the Country Researchers.
2. The second task is to assign valid and substantiated scores for every Standard based on the research evidence presented in the narrative reports.

5.    How is the objectivity of the monitoring and scoring ensured?

We have designed the monitoring and scoring process in a way that maximises objectivity and is based on evidence. Namely, we describe and follow a standardised methodology. In addition, we will highlight the following aspects of the process:

• All participants are experts, and other stakeholders which work in some of the CSO Meter areas.
• Each evaluator (Country Researchers and AB members) assigns his/her scores individually and independently. There are 2 layers of the process:
  • Researchers conduct in-depth monitoring and based on that prepare a detailed report covering listed standards. In the end, they also provide their expert scores.
  • The AB members provide their scores based on the narrative report they receive.
• Verification of the quality of the narrative text and the scores is ensured throughout the different stages of the process, including data collection and analysis, country report drafting and scoring.
• The country partners host consultative events where they invite wider civil society, to provide additional suggestions.

6.    How are changes in the scores recorded?

Every year, the situation is assessed in relation and comparison to the previous year (both in terms of improvement and deterioration). In the annual CSO Meter country reports, any changes in the scores in relation to the previous year are highlighted, showcasing if the situation improved or deteriorated.

 Regional report

• ECNL is responsible for preparing the regional reports each year, once the process of country reports has concluded in all countries. Based on the findings, priorities and scores from the 6 country reports ECNL drafts the regional report. This includes regional findings, trends, assessments and comparison across countries and areas of the civil society environment, and most importantly, highlights the priorities for civil society across countries and also regionally.

• ECNL consults the country partners in the process of identifying regional findings and priorities.

 

To read more about the methodology underpinning the CSO Meter reports in 2022, click here.