Using the ‘Most Significant Change’ (MSC) Technique to measure the intangible

Download MSC guide

In development, it is essential to measure if programs are having the desired impact. However, intangible goals such as ’empowerment’, ‘leadership skills’ and ‘confidence’ are difficult to measure with normal quantitative monitoring tools. This is particularly true in social change programs where improvements in attitudes, behaviour and beliefs take place over a long time period. Unfortunately, many development programs are output focused with program staff struggling to measure the long term impact.

For these reasons, Rick Davies and Jess Dart developed a participatory form of monitoring and evaluation, the ‘most significant change (MSC) technique’, which provides a way to measure intangible qualitative indicators like ’empowerment’ or ‘confidence’ through the systematic collection and analysis of stories provided by program participants and stakeholders.

The MSC technique is participatory because many stakeholders are involved in collecting and analysing the data. It is a form of monitoring because it occurs throughout the program cycle, and it is also a form of evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program.

How does it work?

The MSC technique works by collecting stories of change brought about by the program and systematically analysing them for their significance. Stories are collected from the field with the most significant stories being filtered up the organisation as shown in the diagram below.

The full methodology contains 10 steps, though steps 4,5 and 6 really define the MSC process:

1) Starting and raising interest

For the MSC technique to be effective, it is important that those who are going to be using it understand its importance and are willing to put their time and effort into the method. It is therefore vital from the outset to have within the organisation at least one ‘champion’ of the method who understands how the method should work, can answer any questions from field staff and ensure that the process is implemented correctly.

2) Defining domains of change

In the second step, it is important to define the domains of change you wish to collect stories about. It is easiest to base these domains on project objectives. For example, if a project’s objective is ‘more women in target areas exercise their rights to participate in decentralisation processes’, then the domain could be ‘changes in women involved in local politics’. As well as having clearly defined domains linked to project objectives, it is advisable to have one domain open-ended, and also to consider having an ‘any other change’ domain to provide room to collect stories of unanticipated changes.

3) Defining the reporting period

Before you implement the MSC technique, you must clearly define the reporting period. This depends on the resources available in your organisation but in most cases, stories should be collected and analysed once per month or quarter, and collated annually.

4) Collecting significant change stories

Stories of change should be collected at the field level by field staff. Field staff should collect unsolicited stories of change related to any of the domains that they have witnessed in the communities in which they work, and then bring these stories to monthly team meetings to be discussed.

To facilitate the collection of stories, it may help to ask open-ended questions to field staff within the domains already identified. For example, ‘In the last quarter, what do you think was the most significant change in women taking part in local politics in target areas?’

For each change story, the following information should be collected:

• Title of the story
• Information on who was involved and when the change took place
• Information on what change took place
• Why is the story is significant (this question is particularly important)
• Any lessons learned/recommendations for the wider program

It is worth noting here that some time should be spent at the beginning of this process finding an appropriate translation for the term ‘significant’ in non-English speaking cultures. For example, ‘important’ was a better translation in the Cambodian context.

5) Selecting the most significant change stories

Once the stories are collected, they are brought by the field staff to team meetings. During these meetings the stories are discussed and the most significant ones are selected to be passed up to the management team. The management team then repeats this process, discussing each of the stories they receive and selecting the most significant ones.

The emphasis is on open debate during these sessions and not individual decision-making. Make sure you put aside enough time (usually about 2 hours per meeting), to go through each story, discuss its significance and how it links into program goals and to select the most significant stories.

While various processes can be used, the key ingredients to story selection are:

• Everybody reads the stories
• The group holds an in-depth conversation about which stories should be chosen
• The group decides which stories are felt to be most significant, typically through voting
• The reasons for the group’s choice(s) are documented

Note: Documenting reasons why the group thinks the story is most significant is incredibly important. The reasons for selecting a story as the most significant should be documented and attached to the story following the explanations given by people who initially documented the story.

6) Feeding back the results of the selection process

Feeding back which stories were selected by the senior management as most significant and why is an essential component of the MSC technique. This allows the field staff to readjust their focus on collecting stories in each reporting period. Feedback can be provided via an email newsletter or verbally at the next team meeting. The reason for selecting each story should be clearly explained to program staff, and ideally link into the program’s theory of change.

7) Verifying the stories

Once a story has been selected as ‘most significant’ you may want to verify some of the facts or get any missing information, particularly if this story is to be sent on to external stakeholders and funders. In addition, it can be useful to go back and conduct a more detailed interview with the participant or field staff to develop a case study with quotes which can be used for fundraising and communication purposes.

8) Quantifying

Although the emphasis of this tool is on collecting qualitative information, it can be useful to quantify important information such as how many people benefited, how many activities took place, etc.

9) Doing secondary analysis

If you have the resources, you may want to do some secondary analysis of stories of change over a period of time, to get a sense of the bigger picture and to inform strategic planning processes.

10) Revising the system

In every organisation, there will need to be some changes/adjustments to the MSC system based on what works and what doesn’t. It is a good idea to initially pilot the system to see what works and what doesn’t before rolling it out through your organisation.

What is it like to use in practice?

I was involved in training a local advocacy organisation in Cambodia on using the MSC technique to measure the impact of their women’s empowerment and violence against women programs. First of all I would say that in order to be successfully integrated into existing M&E systems, the method requires a significant amount of buy-in, resources and effort from all involved. The first step in the process was really crucial to the method being implemented properly.

I quickly learned that responsibility for implementing the MSC approach would need to be written into the job description of the ‘champion’, in this case the M&E Coordinator, to ensure that there was at least one person responsible for continuing the method. It was also really important that those facilitating the monthly team meetings (in this case the program managers) also understood the method and had the skills required to facilitate group discussions.

One problem we encountered was a reluctance of staff to write up stories to bring to regular team meetings, as they perceived this as adding to their already tight workload. To solve this problem we revised the method to include verbal discussion of stories at the team meetings, and only those stories voted as most significant were written up.

The stories discussed at the team meetings reflected the views of program staff on how change happens, which in some cases highlighted underlying problems that needed to be addressed. For example, some stories described how women were able to reduce their husband’s violence by changing their own behaviour. The field staff considered this to be a significant success story. However, it also highlighted a cultural perception that women are partly to blame for violence against them. This gave the organisation an opportunity to review its theory of change, and also showed the need for more training of field staff.

What do the results look like and how can they be used?

The MSC technique produces a series of stories in a predetermined format (see example below). These stories can then be used to assess whether the program is having the desired impact.

Title of story	Woman community facilitator voted as president of community network.
Who was involved and what took place?	A local woman who has received capacity building support and training on leadership skills, has been voted as president of the community facilitator network representing 5 villages.
What change took place?	A local woman is now representing her community as a leader in local development issues. She had the confidence to put herself forward and was voted by her community to play a leading role in the community’s development process.
Why is story significant?	The story shows that women in communities where the NGO works are playing an increased role in local politics and decision making as a result of capacity building and training from the NGO.
Lessons learned / recommendations for the wider program	Training and capacity building can help build women’s confidence to participate in local politics. If women are given the opportunity to become involved in local politics they will. The beneficiary can be a role model for other local women.

What are the limitations?

The MSC technique is useful for collecting qualitative data on changes that are occurring with program participants and stakeholders. However, it is not as good as other qualitative techniques (such as process tracing methodology) in showing whether the change was likely to be caused by a particular program.

Although the authors of the manual believe that MSC is easy to implement, in my experience it actually requires a relatively high level of skill in analytical thinking, in particular from program managers and those facilitating the method. It also required significant time and human resources. If these things are lacking then the stories collected may be too brief to be useful, or managers may end up selecting stories that are less relevant or only show one view of the program (for example, that the program is successful, without including any stories of unintended outcomes or failures).

The bottom line

The MSC technique is a useful qualitative M&E tool for collecting data on hard-to-measure indicators.

The MSC technique is a good choice when:

• You want to measure intangible indicators related to social change (e.g. empowerment, confidence, etc).
• The field team has lots of contact with program participants, allowing them to collect stories regularly.
• You need to collect data on impact rather than outputs.
• You want to find out about any unintended consequences of the program.

The MSC technique is NOT a good choice when:

• It is a simple program with easily measured quantitative outcomes.
• The field team is too busy to collect stories regularly, or the managers are too busy to discuss them.
• The managers do not have the analytical skills or experience required to select and interpret stories effectively.

Photo by treesftf