Relief Watch was a one-year collaboration between Sonder Design Collective and the Humanitarian Policy Group at the Overseas Development Institute, in partnership with Humanity United. It focussed on prototyping a new approach to accountability to affected populations within the Humanitarian sector.

Early in the Design process, Iraq was chosen as the context within which to research and prototype the service, allowing the team to co-create the service with affected populations and humanitarians living and working in Iraq.

The team visited Iraq three times over the course of 2019, visiting camps and urban areas where Internally Displaced People (IDPs), and refugees (mostly from Syria) are living. Each trip involved prototyping the service experientially with users, and ensuring that the touchpoints of the service were informed through consultations and input from the most vulnerable people in communities (older people, people with disabilities, ethnic minorities etc).

The Relief Watch concept consists of two pillars; a bottom-up mechanism that ensures communities have access to open, and accessible channels to give feedback on humanitarian services in their area; and a top down mechanism that incentivises organisations to listen and respond to community feedback on the platform.

The team tested many different input channels with communities in camps and out-of-camp areas across Iraq. They found that the service could not rely on one channel, and that a multitude of channels would be essential to ensure that the service would be accessible and intuitive for the diverse population.

In-person community forums are the foundation of Relief Watch. The forums are a space for communities to discuss their experiences of humanitarian organisations in a safe and non-judgemental environment. A key role within the service is the Community Voice Ambassador, who facilitate the forums. The ambassadors are trained socio-linguists from outside the humanitarian sector, with knowledge of the local language and customs.

The Community Voice Ambassadors are trained to ensure that the community themselves set the agenda of the forums, and try to understand what communities desired changes are. The Community Voice Ambassadors work in pairs, with one facilitating the forum, and another inputing verbatim quotes and experiences into the Relief Watch platform.

The forums are supplemented by digital channels on Facebook and WhatsApp (apps readily found on most people's smartphones) that allow users input their experiences directly to the Relief Watch website conversationally, and anonymously.

The top-down pillar of Relief Watch incentivises organisations to listen and engage with community feedback on the Relief Watch platform. This element of the service was co-created with humanitarians both in Iraq, and at a global level, with a Global Steering Committee consisting of representatives from international humanitarian organisations, affected communities, accountability experts, and donor agencies. The Organisation Index part of the Relief Watch prototype website allows users to see which organisations are responding to people's reviews. Organisations that are responding regularly are awarded a 'Top Listener Badge' to allow donors identify responsive humanitarian organisations.

Relief Watch is currently preparing for a pilot in various humanitarian contexts in 2020.

Relief Watch was developed in response to a perceived lack of a means of affected people to hold the humanitarian system to account. While having the means to hold those exercising power accountable for their actions is commonly considered a right across much of public life, it is not reflected in the current reporting mechanisms of humanitarian organisations. The design process yielded a number of key insights which informed the design of the Relief Watch concept:

Firstly, that in the interests of creating an inclusive service that a diverse population with differing needs could use, the means of collating feedback would need to be flexible and open-ended, utilising technology already in use in that particular context but also carefully considering the inequalities of accessing the internet and mobile phone-based tools.

Secondly, that such a service would need to be both independent of, but act as a complement to, the existing assessment and accountability apparatus currently working in the particular context of Iraq, exploiting the clear gap that exists for allowing affected people to voice their feedback and experiences while being informed by, and feeding into, the approaches humanitarians are already using to understand needs.

Thirdly, that any service aiming to be useful and sustainable for affected people would need to deliver a 'closed feedback loop' that ensures questions and complaints are listened and responded to. As a voluntary service, the ReliefWatch concept cannot ensure this, but can collate and present feedback in a manner as useful as possible to responders, ensuring replying is as easy as possible.

Relief Watch presents community feedback to organisations and the general public. Though anonymous, data on the post's general location can assist humanitarian organisations in assessing the impact of programmes and in planning future responses. The quality of such feedback would be an important asset for the service and offers the means for people to provide less restrictive qualitative data, comprising the kind of rich experiences and stories that have the power to highlight people's agency, priorities and wishes for the future. For many humanitarians who were interviewed and co-designed the concept, this type of information was lacking and very different to the 'tick-box' exercises that often comprise current accountability tools.

The most obvious consequence of not having an effective means of engaging with, and being held to account by, affected people is that assistance is often not aligned with what is needed. The 'appropriateness' of aid is a common source of criticism, and unnecessary non-food aid, low quality supplies and inadequate shelters continue to be a feature of almost all humanitarian contexts. From the perspective of affected people, responses often leave much to be desired, with 75% of surveyed people in seven crises reporting that the aid they receive does not meet their most pressing needs. The impact of inappropriate aid can also be demonstrated by the scale and frequency of affected people selling some or all of the assistance they receive, in order to buy more useful goods and services that constitute a sizeable black market that surrounds displacement camps in humanitarian contexts across the world.

The design of the concept was shaped by these experiences shared by affected people and humanitarians, as well as the wider research that considered the state of accountability in the sector. From these findings, the design team created two problem statements from the perspective of affected people and humanitarian organisations:

People affected by crisis do not consider feedback mechanisms as accessible, effective, confidential and/or safe. When provided, feedback is not always investigated, resolved and results fed back to the relevant persons promptly, or at all.

Humanitarian actors are held to account based on resource use to funders, and not based on the expressed needs and feedback of people affected by crisis.

Presenting Feedback Data

Once submitted, feedback data is categorised, geo-located and presented on the ReliefWatch site. The prototyping process revealed the necessity for a layer of aggregation of this qualitative data to allow users in managerial positions to see emerging themes, while for those working at an operational level, the details of feedback from one individual theme could be identified and responded to. Categories that affected people and service coordinators will submit can be sorted according to details including relevant area (in the case of Iraq, by governorate), gender, camp or non-camp context, theme (camp management, socio-economic support, reconstruction, cash assistance etc.), and 'type' of complaint, if relevant (quality of aid, staff misconduct, organisational corruption, government concerns etc.). Those submitting feedback can name specific humanitarian organisations in their messages, but this is optional and reflects the lack of knowledge of exactly which aid provider was responsible for what programme among those interviewed.

Prior to comments being publicly available online, a 'moderator' function performs a basic level of both categorising and limiting of submitted comments that are explicitly abusive or relate to especially sensitive issues. To some degree, this process can be automated with the moderator verifying rather than manually categorising feedback. Certain words would constitute tags which could then be presented to a moderator as comprising an emergent category to present on the site. A key issue for the design was the means by which the service would distinguish between complaints over services with especially egregious reports of abuse, including instances of staff misconduct or corruption. In such a situation, as with all complaints, the priority is to provide sufficient anonymity in order to not expose vulnerable populations to more risk and so no data that can be used to identify an individual is presented on the site. In cases where such a report is submitted, a member of ReliefWatch staff can elevate the claim and directly forward it to the relevant organisation, if that information is available. If not, staff will approach the claimant directly and seek further details before contacting the relevant organisation.

The feedback collated by the service would be geographically specific, and is presented as a 'heatmap' that plots the density of submitted data. These maps allow users to identify the specific locality and/or displacement camp where the feedback originated, but not to the level of an individual building, striking a balance between being sufficiently localised for the benefit of humanitarian staff while preserving the anonymity of those submitting feedback. Heatmaps fulfil a core function of the service in providing an accessible 'monitor' of the tone of particular areas and highlight key areas of need or neglect. It would also enables humanitarians to evaluate whether their current points of service are aligned with locations on the map where unmet needs are emerging.

While the concept draws inspiration from the private sector ratings platforms and appears superficially similar, its underlying ethos is based on the fundamental right of affected people to be heard and to increase accountability in the sector. In the process of designing this concept, the team found a growing awareness and frustration among both affected people and the staff of humanitarian organisations that progress on accountability and participation is slow, to the detriment of both groups. There was also considerable enthusiasm for an approach which would lead to people caught up in humanitarian crises having a more direct relationship with those that seek to assist them. In doing so, organisations that provide assistance and populations in crisis would be more informed and accountable, and could lead to both more effective responses carried out by a more accountable humanitarian system and more empowered affected people.

In summer 2019, the ReliefWatch concept was merged with a similar initiative, also in its early stages. Named 'Loop', the service would also enable aid recipients to put their experiences directly to humanitarian, development and local organisations. Many characteristics of ReliefWatch were shared by Loop, with both concepts committing to principles of open data and a two-way dialogue, and stressing the importance of users being able to categorise feedback by geographical or programme area. In addition, Loop proposed a governance model that reinforces its ambition for greater participation, with affected people and communities taking a key role in deciding how the service would operate and its priorities.

Loop replaced ReliefWatch as the name of this combined initiative. The name Loop has a number of advantages and is more reflective of the final design of the concept. Initial proposals for an independent AAP mechanism began during the HPG project 'Constructive Deconstruction', and originally took the form and title of an 'ombudsman' and later, a 'watchdog'. Such concepts suggest a form of accountability that carries with it a punitive function for transgressing individuals or organisations. However, it became apparent that developing a voluntary service with a robust enforcement element would currently be an unrealistic goal. Instead, the design focus shifted toward developing more of a two-way dialogue between affected people and humanitarian staff and organisations, and was seen by the design team as being of greater benefit for those the service sought to assist.

The Loop team are currently raising investment funding for pilots in a variety of humanitarian contexts in 2020.