Monitoring, Evaluation and Learning : Testing A New Method

During the CARE West Midlands programme, we wanted to develop a method of measuring partners’ growing knowledge and confidence. Previously, we have used statistics such as training attendees, OISC assessment passes and levels raised. We didn’t feel this captured the impact of confidence and how knowledge can grow. It is also difficult to encapsulate organisational shifting power journeys with these types of statistics, as a lot of this work can be more conceptual and less well demonstrated by attendance at an event.

What did we do?

We took the model of the Warwick-Edinburgh Mental Wellbeing Scale, which is used to monitor wellbeing and mental health. We decided to ask partners to score themselves on a scale of 1-5 in response to statements. These statements covered confidence with OISC registration, shifting power and how connected they felt to other organisations in the region. They would do one at the start of the year, and once again towards the end. We hoped that the difference in responses would indicate a change in their skills and confidence.

For example:

What happened?

Unfortunately, this did not work as well as we had hoped.

We found however that without questions designed to prompt partners to consider their previous responses, and to think about change over time, many gave themselves similar scores. We were also conscious that they would not be likely to remember their previous scores from months earlier. We did not think that the responses matched up with the skills and knowledge they had gained through network sessions, immigration advice training and shifting power work we knew they were undertaking. See the examples below.

Year 1

Year 2

On a positive note, we were able to quickly gather responses from all partners. The questions were quick to answer and were presented on a simple Google form, a format they were all familiar with already. It was very easy to monitor who had completed the survey and compare results.

How did we change our approach and put learning into action?

At the start of Year 2, we wanted to refine this technique to focus more on change over time. Julie Mansfield, FIAP Partner Development Project Manager, recalled using the Health and Wellbeing Prism, and how clients would score themselves with their previous responses in front of them encouraging them to think about change.

We decided to be more explicit with what we were asking and developed a new version of the questionnaire. In Version 2, we asked partners to score themselves on a scale of 1-5 again, but this time we asked separately how they felt before and after participation in the project in three specific areas. For example:

We immediately received much more useful responses, which we felt captured developing knowledge and confidence much better than the two-questionnaire method we used in Year 1. These can be clearly displayed in graph format which is a useful visual aid in our reporting.

What did we learn?

This corroborates the existing statistics of attendance at training and events, and it goes a step further to demonstrate how useful these sessions have been. This was also simpler, as we only needed to monitor replies to one survey. The simplicity was a benefit to our partners, many of whom are small organisations working on the frontline with clients. We wanted to make sure that what we asked of them was quick and easy, as we understand the pressure they are under.

We look forward to adapting and developing this method, potentially for use on other projects in future.