Measuring the impact and value of our work

,

 

What is the value and impact of thousands of boreholes and pumps 10.000? And how can we measure the ‘unmeasurable’ such as quality of life and hope?

Purpose

DACAAR – Danish Committee for Aid to Afghan Refugees is a pioneer in Water Hygiene and Sanitation in Afghanistan. And DACAAR is an expert in counting the number of boreholes bored, pumps installed, and cubic meters of healthy water delivered to poor and vulnerable households.

But what is the value of fresh water to a household?  How and how much does water contribute to quality of life? And how does a sewing machine delivered by DACAAR contribute to change social norms in a community controlled by Taleban?

In 2019 StrategyHouse.dk worked with DACAAR staff to assess DACAAR’s ability to measure impact and value of its work, develop tools and approaches to measure qualitative changes and outcomes of water supply and other ‘quantifiable’ outputs and build the capacity of staff to carry on the measurements themselves.

The method

Many organizations spend too much time collecting too much data, which is of little value to their M&E goals: tracing and assessing outcomes, changes and value of their work. Yet to measure change, staff must have a clear understanding of what change is, the changes they contribute to and what they can do to measure the qualitative changes – which is often at the core of their work.

To this end, StrategyHouse.dk worked with DACAAR to:

  • Understand the qualitative changes it contributes to in people’s lives.
  • Review DACAAR’s procedures and significantly reduce baseline data that DACAAR collects.
  • Develop scales of measurements enabling DACAAR’s M&E team to assess incremental changes and qualitative results of its work
  • Introduce procedures and approaches embracing unexpected, positive and negative changes that DACAAR may contribute to
  • Reflect on findings and use them for reporting and internal learning.

Main results

Measuring and managing towards outcomes/changes takes that organizations adopt focus on change and outcomes rather than activities and outputs. Changing focus from activities to results is rarely done with a single training workshop only. As with any other learning process changing perspectives and approaches takes a bit of training and repetition.

Yet the combination of:

  • Workshop training
  • Co-creation of tools for data collection that makes sense to DACAARs staff
  • Joint data collection in the field and practice with StrategyHouse’s team

Has led to a stronger capability in DACAAR to document and reflect on outcomes – and to use findings and lessons learnt for organizational learning and improvement.

Occupancy Free of Charge

, ,

Nexus – between relief and development

Will Occupancy Free of Charge for 12 months strengthen Syrian refugees’ resilience and ability to develop positive coping mechanisms in Lebanon? And what is the impact on the housing market in Lebanon per se?
Norwegian Refugee Council has supported medium scale rehabilitation of water and sanitation infrastructure, such as water tanks and solid waste disposal sited (community support projects) and Occupancy Free of Charge (OFC) to Syrian refugees in Lebanese border areas since 2012.
The aim has been to contribute to:

a) An increased availability of minimum standard housing for vulnerable households, at affordable cost.
b) An improved security of tenure/lease for vulnerable refugee households and landlords and an enforcement of the legal rights of refugees.
c) A reduced strain on the already overstretched public infrastructure in Lebanese communities with a high refugee caseload.

Purpose

In 2018 StrategyHouse.dk worked with Norwegian refugee Council to assess the intervention’s impact on

  • Syrians households post their participation in the Occupancy Free of Charge Program
  • The availability of low-income housing in targeted areas in Lebanon
  • Host communities

The challenge

Evaluating the impact, (let alone effectiveness and sustainability) of an initiative aimed at contributing to long-term or (more) durable changes in households, markets or communities as was the case with NRC’s intervention in Lebanon, presents certain methodological challenges.

Evaluating the impact, (let alone effectiveness and sustainability) of an initiative aimed at contributing to long-term or (more) durable changes in households, markets or communities as was the case with NRC’s intervention in Lebanon, presents certain methodological challenges.
A first challenge is the role of external forces or conditions in promoting desired changes. The political context in Lebanon is complex and not in favor of any move stabilizing the refugees in the country

A second challenge is that change is typically stimulated by numerous concurrent factors and many actors. It may be difficult to determine the exact cause and effect between one specific organization’s interventions activities and impact identified.
A third challenge is the typical way that we may conceptualize programme outcomes or results. Outcomes are typically expressed as “forward progress” but outcomes might be positive also when they are “defensive” in nature, such as preventing further deterioration of public services.

Method and solutions

To overcome these challenges, and enable project staff to reflect on the viability of the intervention’s implicit theory of change the evaluation adopted a mixed evaluation approach consisting of the following, key elements:

a) Articulation of the intervention’s theory of change and the rationale between provision of occupancy free of charge and the expected changes related to household income, behaviour, attitudes and market conditions.
b) Identification of expected and unexpected outcomes and changes relevant to the interventions direct sphere of influence.
c) Validation of outcomes, assessment of their significance and the intervention’s contribution as well as the contribution of other factors and actors.
d) Comparison between the theory of change (point A) and ‘how change really happened’ according to outcomes harvested (point B and C).

This was done using a rage on participatory approaches and tools to ensure ownership and learning among staff members as well as beneficiaries. These included desk reviews, interviews, focus groups, and analysis, interpretation and discussion of findings with NRC staff and a participatory comparison of the intervention’s initial theory of change with outcomes identified during the evaluation process.

Main results

The evaluation found that the intervention contributed to stabilize households’ socio-economic situation during the 12 months that the intervention provided occupancy free of charge. This had a positive impact on household’s food consumption as households were able to buy meat and fresh vegetables. Households also reported that they could afford to buy medicine and non-food items such as soap or clothes. The OFC reduced the financial stress and enabled households to pay accumulated debts.

The evaluation also found that occupancy free of charge had a positive impact on refugees’ social capital as staying in the same location enabled refugees to build relationships with neighbours and landlords. This was instrumental in terms of accessing information, potential jobs, help to look after children or access credit.

Yet coping and resilience post the occupancy free of charge period was challenged by an exhaustion of refugee households’ human capital, including deteriorating physical and mental health, low levels of self-esteem, limited production skills and feelings of disempowerment. So while occupancy free of charge did provide refugee households with time and mental space’ to recover from the stress of homelessness, it did not enable them to develop more sustainable coping strategies and strengthen resilience beyond to period where housing was provided free of charge.

The findings therefor led to a revision of the intervention’s theory of change and understanding of the importance to – also – invest in refugees’ human capital to maximize the benefits and sustainability of an occupancy free of charge intervention.

 

Promoting Workers’ Rights through mobile internet

,

Can mobile phones and social media promote respect for workers’ rights in the garment industry?

Purpose

Since 2017, WageIndicator together with C&A foundation support a project which aims at strengthening transparency of working conditions and stimulating dialogue between employers and employees to improve and align working conditions with legal provisions. Interestingly, the tools to do so were online surveys and disclosure of results on a web-page, www.gajimu.com/garment
In 2019, StrategyHouse.dk worked with C&A Foundation and WageIndicator to assess the preliminary impact of the project, its effectiveness and relevance for workers in the garment industry in Indonesia.

The method

Evaluating the impact of a ‘cloud’ or web-based initiative necessitates a dual focus on the functionality of the social media platforms and their embeddedness with the ‘lived reality’ among garment workers, employers and brands.
To this end, StrategyHouse.dk adopted a two-pronged approach to the evaluation:

  • We collected data on the gajimu.com/garment websites, accessible in Google Analytics.
  • We collected data through semi-structured interviews and focus group discussions with intended users of the website, including employers, influencers in the sector and workers themselves.

Main results

The evaluation found that mobile surveying and disclosure through the internet had a strong potential to strengthen workers’ awareness about their rights, mobilise their unions to address issues of violations of the labour law and mobilise employers to comply with the labour law.

The main reasons to the change were that:

  • Questioning in itself ‘form reality’. When asked in an enabling space, questions draw the audience’s – here the workers’ – attention to the issue being asked. It stimulates reflection, curiosity and internal learning. Questioning is therefore a very powerful and simple tool to help set an agenda and strengthen awareness.
  • The disclosed survey results ‘set an agenda’ and helped raise expectations to workers’ trade unions that they addressed the disclosed issues. It also spoke to employers in that they rectified violations of the law and to buyers and brands, which honoured their social responsibility.

Another main finding was that ‘customization’ of surveys and social media platforms to workers’ internet behavior and the time they have available to use facebook or complete a survey, is of vital importance to the success of social media based campaigning. So is the importance to consider workers’ understanding of the questions raised when designing surveys and questionnaires.

Campaigning and surveying through mobile internet therefore could not ‘stand alone’ but should be introduced by facilitators with skills to create a platform for reflection and dialogue among workers completing the survey and with a mandate to assist trade unions communicate and follow-up on survey findings once disclosed on the internet.

These findings were consequently incorporated into the intervention’s second phase.

Strategic Planning with Theory of Change

,

How do you improve your ability to help cities collaborate effectively, share knowledge and drive meaningful, measurable and sustainable actions on climate change?

C40 Cities is a network of the world’s megacities that is committed to addressing the issues arising from climate change.

They work in cities as varied as Lagos and Copenhagen or Shanghai and Chicago. Their Planning and Measurement Team has adopted Theory of Change (ToC) as a key steering instrument to underpin and improve the impact of their work. They recently met with StrategyHouse.dk at a retreat designed to help them learn about ToC, and to plan how they will use it to ensure they get the best out of it.

Purpose

How do you build a common approach to the problem, but tailor the interventions to regional differences? The purpose of C40’s Planning retreat with StrategyHouse.dk was to get a shared framework of understanding, to establish a cross-cultural and cross-regional exchange of experiences and viewpoints on change. The team wanted to produce a Theory of Change (ToC) that could be communicated with, shared between and strengthened by other C40 teams and stakeholders.

Challenges

When you are a truly global organisation, spread across continents and operating at both practice and policy levels (and everything possible in between), getting all the stakeholders involved is part of the challenge. The greatest value of Theory of Change is that it offers is a joint reflection process. Ideally this ensures that all the relevant actors, in a programme, have a part of defining how to achieve their desired outcome(s). C40 Cities opted to gather the core planning team together to work on this and then to disseminate the results of their work to the rest of the organisation, in a new phase.

Method and solutions

A first step was to decide at which level to develop the ToC. This required specification and reflection about what C40 Cities wanted want their ToC for. Will it be used for discussing how change happens? Is it for detailed planning? For C40 Cities, what they needed most was a programme and project-level theory change. This task included working in teams to identifying the key domains of change within the C40 Climate Action Programme, taking into account the contextual and regional differences from Cape Town to Vancouver. From there a more detailed task was undertaken – to brainstorm the main preconditions for change and to sketch out pathways where the flow of change was analysed and described.

Output

Although the main value for the C40 Cities team was the working together, in a participatory process, to develop the theory of change, a physical output was necessary in order to engage other teams in the organisation, who could not be part of the workshop. Together with StrategyHouse.dk, the team co-developed pathways and theory of change diagrams and narratives. These emerged as strong opportunities for further detailed planning and prioritising. All in all, there was not only hot air in Middelfart!

Want to know more?

Contact Morten Ronnenberg mrm@strategyhouse.dk

p.s. – The hard work pays off

Having a theory of change at programme level helps to identify the main axes of intervention, to understand the overall flow of required actions and it provides a strong basis for prioritisation. However, if you want to go all the way, you need to develop detailed theories of change (all the way down to theories of action) at the local level. This helps individual offices and team members assess where the generic assumptions are wrong, which calls for a different approach. Most often, when something goes wrong, the fault lies in the details and in our assumptions. Testing these, all the way down to what you plan to do tomorrow is hard work – but it pays off.

Strategic planning and fundraising for youth

,

The African Youth Panel is a platform of youth leaders from different African countries. It is committed to advancing policy, political and economic actions to reduce youth unemployment and to enhancing governance and participation.

The AYP was initiated by the Danish NGO Forum, to provide input to the Danish Africa Commission (DAC) on what youth can do in Africa’s development as well in what development partners should do to ensure that tangible results are achieved with their support.

In its first years, the AYP achieved some remarkable breakthroughs and managed to build a strong political network.

Purpose of the assignment

StrategyHouse.dk was engaged to support the network with a strategy and fundraising development process that could assist the network to becomw financially and organisationally sustainable. The AYP had already received substantial start-up funding from Danida, but needed to diversify its funding and move from incubation to a mature stage.

Challenges

One of the major challenges for the AYP was the lack of paid staff and a secretariat, to ensure consistency in the work flow, constant delivery and implementation of the plans. The AYP’s structure comprises a panel of 60 young people in over 23 African countries. A Steering Committee is the governance arm of the AYP with representatives from all sub-regions of Africa. Given this set-up, an additional challenge were  how to enable the network to meet (very costly) and how to facilitate coordination across the continent (very time consuming), in order to ensure that the AYP could succeed in its lobbying ambitions.

Solutions

StrategyHouse worked with the AYP during training and planning sessions to solidify its strategic planning. This included a review of the ‘2015-2018 Strategic Plan’ to ensure the right balance between objectives, campaign activities and resources. It also included a classic fundraising training session to enable participants to pitch their cases andbe aware of the key steps and elements in professional fundraising. A theory of change training and planning workshop ensured a solid analysis was made in order to be able to establish a shared understanding of what activities would lead to the goals and  why. This provided the basis for a prioritized planning of campaigning and fundraising activities.

Want to know more?

Contact Morten Ronnenberg mrm@strategyhouse.dk