Evaluating Advocacy Efforts: How Do You Measure Social Change?

So you read the last blog post in this series, about asking the right questions of your social change efforts, right? (You did, right? We know it’s been a minute.) If you did, you know that there is just as much strategy involved in deciding what data you need to collect as there is in carrying out the strategy itself!

Its one thing (and a very important thing) to know what you want to collect and why. But the next big question is. . .HOW? Engaging hearts and minds to actually care about something, breaking through the social media din to get people to actually act – thats hard enough. Once you think you’d done that, how do you go about measuring the indicators that those actions led to policy change?

Many methods for measuring policy change look a bit different than measurement methods you may have used in other evaluation contexts. This is because trying to attribute a change in policy or opinion to a campaign is hampered by the complexity and number of messages people receive as well as the influencers they interact with.

While some familiar methods, such as case studies and focus groups, are applicable to measuring policy change, here are some methods you may not know that could be more suitable:

Bellweather Method This method involves doing structured interviews with influential people, those who are in a position to see many aspects of a policy issue. When conducting a bellweather interview, its important that the interviewee does not know the topic in advance, so their answers arent pre-influenced.

Questions you might ask in a bellweather interview: Currently, what three issues do you think are at the top of the policy agenda? How familiar are you with [insert specific issue/policy or specifics about an issue]? What individuals, constituencies, and/or groups do you see as the main advocates for this policy? How likely do you think it is that [specific] policy will be adopted in the next 5 years?

Policymaker Ratings In this method, you find someone able to speak knowledgeably about the performance of a policy maker on a particular issue. This might be an advocate or other stakeholder who has a well informed perspective about a policy maker and their support for an issue. This person should also have some knowledge about whether a policy maker has influence over the issue. You should plan to talk to 3-5 people, and to look for consensus on a policy maker’s level of influence and whether their views have ever been altered on the issue of concern. You will also want to assess how much confidence the rater has in their own ratings.

Intense Period Debriefs When there is little time for data collection, and things are happening fast, you might consider intense period debriefs as your method. After a campaign concludes or a policy shifts, for this method the evaluator would speak with advocates, usually in a focus group setting. They would ask participants to discuss what happened, what strategies were used and which were effective, and what they would do differently.

System Mapping System mapping is similar to creating a theory of change but on a grander scale. A system map is not linear, allowing for a more complex and accurate rendering of how different actors and social forces influence change. Mapping out the components of the system, what changes are expected, and what the relationship is between the system elements is very helpful for advocacy and policy change work.

Stay tuned for the third and final installment of this series in 2019!_

Resources

Methods to Consider (Julia Coffman and HFRP)

Intense Period Debrief (Innonet)

System Mapping (Innonet)