Final Evaluation Report_MOTS Service for Community Health Workers in Bo and Kambia, Sierra Leone

EXECUTIVE SUMMARY

Between 2014 and 2016, West Africa was struck by the largest outbreak of Ebola in the history of humanity. The EBOLA vaccine Deployment, Acceptance & Compliance (EBODAC) project, a public-private partnership of London School of Hygiene and Tropical Medicine, Janssen Pharmaceutica N.V., World Vision and Grameen Foundation, was launched with the goal to develop strategies and tools to promote the acceptance and uptake of new candidate Ebola vaccines. In addition, EBODAC developed an Interactive Voice Response (IVR)-based Mobile Training and Support (MOTS) service as an innovative way to provide refresher trainings on vaccinations and disease surveillance to a large group of remotely located community health workers (CHWs), with a special focus on Ebola.

This report covers the results from a quasi-experimental, multi-method pre- and post-test assessment conducted with CHWs located in Kambia District, Sierra Leone to assess the effectiveness of MOTS. The curriculum used for MOTS was adapted from the Ministry of Health and Sanitation’s (MoHS) education on vaccinations and community outbreak responses. The Vaccination module focused on the importance of vaccines and receiving them at the correct time; the Outbreak Response module focused on identifying diseases such as Ebola and the protocols necessary to protect people and the community. In addition to IVR-based refresher messages, MOTS also assessed knowledge change with IVR-based pre- and post-test quizzes.

The MOTS assessment aimed to answer two key questions:

  • To what degree does MOTS improve knowledge and behavior of CHWs related to vaccinations and outbreak response (compared to a comparison group)? The hypothesis: MOTS improves knowledge/behaviors of CHWS compared to CHWs who receive no mobile training.
  • To what degree does the order of the refresher modules (Vaccination followed by the Outbreak Response module, or vice-versa) matter on knowledge and behavior change? The hypothesis: Participation in MOTS sessions back-to-back can lead to user fatigue, resulting in reduced effectiveness of the second refresher module in which a CHW participates. A prior evaluation in the Bo district hypothesized that this might be the case.1

To answer these two questions, the quasi-experimental study consisted of two treatment groups and one comparison group. One group was assigned the Vaccination refresher module first followed by the Outbreak Response refresher module (Treatment 1); another received the Outbreak Response refresher module first followed by the Vaccination refresher module (Treatment 2). The comparison group (Comparison) did not receive any of the modules until the endline assessment was completed. A total of 811 CHWs were enrolled for the refresher training and out of this, a sample of 375 CHWs was randomly selected and evenly divided among the two treatment and the one comparison group for the quasi-experimental study. These 375 CHWs were interviewed in person by enumerators using SurveyCTO (known hereafter as the outcomes survey). In addition, pre- and post-test IVR quiz data, which are part of the MOTS system (known hereafter as the MOTS quiz), were compared to the results of SurveyCTO results to evaluate consistency in the results for the two assessment methodologies. Qualitative interviews with CHWs, project staff, and health staff were also included.

The research suggests the following results and recommendations, according to the two key research questions:

To what degree does MOTS improve the knowledge and behavior of CHWs related to vaccinations and outbreaks (compared to a comparison group)?

There were slight improvements in knowledge for both modules, but improvements were not dramatic. For knowledge change, a target of 80 percent is used by Grameen Foundation. As a result of education, at least 80 percent of beneficiaries should “know” key information. As the results showed from the outcomes survey, all but one of the vaccination knowledge questions achieved the target, but knowledge levels were also relatively high at baseline. Approximately eight out of the 14 outbreak response questions were slightly or far below the 80 percent target and 7 of them experienced decreases (or no change) between baseline and endline, suggesting some confusion with module objectives or priorities. Both treatment groups outperformed the Comparison group regarding an increased frequency in communicating with communities regarding outbreak response. Only Treatment Group 2 outperformed the Comparison group on a higher frequency of communicating with communications on vaccinations.

To what degree does the order of the module (vaccination and outbreaks) matter on knowledge and behavior change?

The Outbreak Response module did not perform as well as the Vaccination module regardless of whether it was the first or second module for a CHW. In addition, baseline knowledge was much lower for the Outbreak Response than the Vaccination module, suggesting that prior in-person trainings conducted by the MoHS through classroom training may not have been effective at imparting knowledge or the lessons were not retained. The objectives of the Outbreak Response module should be reconsidered—both for MOTS and MoHS in-person trainings.

Recommendations

Overall, it is recommended that sessions designed for MOTS should aim to achieve fewer learning objectives that build on one another. Given the limited attention and digital literacy skills of CHWs, the modules should aim to change/refresh critical-to-know learning objectives that should be reinforced throughout the module. The Outbreak Response module, in particular, covers ten notifiable conditions; this may have resulted in too many module objectives, resulting in poor outcomes for the entire module, and Ebola knowledge outcomes in particular. Given not all CHWs completed the IVR trainings, incentives for completion should be considered, including those that help overcome technical difficulties such as lack of charging capabilities and incentives for active participation, such as recognition for correct responses to quizzes. Given the reasons for non-participation included not understanding how to navigate the system using one’s mobile phone, CHWs should receive training that requires them to personally practice using the system. While trainings were completed with CHWs on the system, not all CHWs may have personally practiced during the demonstrations.

In addition to these key findings and recommendations, the research also revealed that the MOTS quizzes and the outcomes survey data presented mixed findings. Results from the MOTS system suggested correct knowledge regarding most indicators was below 70 percent while results for the outcome survey often had scores above 70 percent for most of the questions at baseline and endline. This suggests that participation in the IVR quizzes did not accurately reflect real knowledge among the CHWs. Furthermore, the gender of the CHW was also found to be influential on whether the trainings were completed and whether knowledge improved—men were less likely to complete the trainings but more likely to have correct knowledge compared to women. Future iterations and functionality assessments of the MOTS trainings should consider how men and women CHWs engage with the MOTS system differently to ensure women have equal outcomes and benefits.

The costing assessment completed on MOTS estimated that the cost for maintaining a MOTS refresher training approach is approximately $12 per each of the 16,000 CHWs nationwide in Sierra Leone and costs approximately 50 percent less than in-person refresher trainings. While IVR should never fully replace in-person trainings, MOTS enables a dramatically more efficient and widespread community health outreach, builds engagement with decentralized health workers, and can provide a critical safeguard in the case of health emergencies.

In conclusion, the results from this study show marginal improvements in knowledge, with male CHWs being most associated with knowledge change. The results also point to specific and actionable areas for improvement, most notably the IVR quiz questions as well as the content that underpins the Outbreak Response module. These improvements are needed to ensure MOTS can serve as a cost-effective, alternative virtual training tool for reaching rural CHWs with critical information to support the health of their communities. For diseases like Ebola and the recent emergence of COVID-19, technologies like MOTS could be a potential alternative training and information-sharing methodology for CHWs that can be rapidly and virtually deployed to any CHW that has a mobile feature or smartphone.

Resource Type:
  • Publication

Publication Date: 05/21/2020
Countries:
  • Sub-Saharan Africa

Program Area:
  • Health

Download  
Together we can transform lives.
Give Now