Implementation strength assessment and Quality of Care

The Implementation Strength Assessment (ISA) measures a program’s rollout and readiness to deliver services.

The Quality of Care (QoC) tools determine whether correct preventive and curative services are being provided during health service contacts.

The RADAR ISA and QoC suite is a set of questionnaires, manuals, and other tools designed to assist program evaluators in planning and implementing evaluations.

Overview of survey process and tools

The tools are divided by program topic area:

Family planning | Child health | Intrapartum care

The tools in each topic area were developed and tested in sub-Saharan African contexts.

Family planning

 The FP tool is a set of questionnaires and resources that allows evaluators to capture and analyze data on FP program implementation strength and quality. This suite of tools provides a comprehensive assessment of facility structural and program readiness, provider knowledge, provider practice (routine and “best-case” quality), and client knowledge, satisfaction, and respectful care.  The FP tool was developed and tested during a country-wide program evaluation in Malawi, 2017 - 2018

Indicators measured include:

% of facilities equipped and staffed to provide FP services.
% of facilities with no contraceptive stockouts in the previous 3 months.
% of providers with complete counseling knowledge. 
% of clients correctly counseled on the method they received.
% of visits where contraceptive (IUD, injectables, implants) was administered by the provider using appropriate sterile technique.

 The questionnaires contain the following forms:

1. The Implementation Strength Assessment, which includes:

• The facility in-charge Survey,

• Health Facility Worker Survey, and

• Community Health Worker Survey.

2. Quality of Care Tools:

• The Health Worker In-Person Interview, used to assess knowledge.

• The Clinical Vignettes, which can be conducted both over the phone and in person.

• The Simulated Client Checklist to assess routine practice.

• The Direct Observation Checklist to assess “best-case” quality practices; and

• Client Exit Interview to assess client knowledges, satisfaction, and contraceptive intentions

 The package also includes manuals that describe, in detail, the steps necessary to plan, implement, and analyze data from FP-specific implementation strength and quality assessments: data collection tools, sample size calculators, budget templates, supervision tools, logistics guidelines, analysis plans, among others.

Worksheet 1: Survey sample size calculator
Worksheet 2: Facility selection sheet
Worksheet 3: Supply calculator
Worksheet 4: Budget template
Worksheet 5: Indicators
Worksheet 6: Supervisor tracking sheet
Worksheet 7: Facility listing sheet
Worksheet 8: Health worker listing sheet
Worksheet 9: Interviewer performance review checklist
Worksheet 10: Priority indicators summary table
Worksheet 11: Indicator review
Worksheet 12: Indicator performance factors
Worksheet 13: Actions for improving indicators

Tools

Manual Module 1: Family Planning - v.3Nov2020

Family Planning Manual Annexes - v3Nov2020

Family Planning Manual Worksheets - v.3Nov2020

ODK sheets - v.25Nov2021
Products

Results from Implementation Strength Assessment, Quality of Care Evaluation and Coordinator Survey Evaluating Malawi’s National Family Planning Program

Malawi Implementation Strength Assessment study

Quality of Care of Family Planning Services in Malawi

Top of page | Module 2 | Module 3

 Community-based Newborn, Child, and Nutrition Services

The Newborn, Child, and Nutrition Services tool is a set of questionnaires and tools and resources that allows evaluators to capture and analyze data on Newborn, Child, and Nutrition program implementation strength and quality. This suite of tools provides a comprehensive assessment of program readiness, provider knowledge, provider practice, and client knowledge, satisfaction, and care. This tool was developed and tested during two evaluations: in the Koulikoro and Sikasso Provinces of Mali in 2018, and in the Inhambane Province of Mozambique in 2019.

Indicators measured include:

% of CHWs with a working infant and child scale
% of CHWs with knowledge of at least 4 counseling messages on exclusive breastfeeding
% of CHWs with full stock of unexpired child health medications
% of children > 6 months of age having their MUAC checked by CHW
% of children with malnutrition correctly classified by CHW
% of post-partum women given at least 5 key messages on dietary diversity
% of newborn caregivers who recall at least 3 ways to maintain newborn’s body temperature

The questionnaires contain the following forms:

1.  The Community Health Worker (CHW) interview, which includes:

• A questionnaire on demographic information and CHW readiness to provide care, and

• A questionnaire with nutrition-specific knowledge questions

2. The Direct Observation Checklists, which include sections for sick child care, newborn care, and nutrition home care.

3. A Re-examination Checklist, included for completing after a sick child direct observation.

4. The Caregiver Exit interview, to assess client knowledge and practice.

The package also includes manuals that describe, in detail, the steps necessary to plan, implement, and analyze data from Community-based Newborn, Child, and Nutrition implementation strength and quality assessments: data collection tools, sample size calculators, budget templates, supervision tools, logistics guidelines, analysis plans, among others.

Worksheet 1: Survey sample size calculator
Worksheet 2: CHW selection sheet
Worksheet 3: Supply calculator
Worksheet 4: Budget template
Worksheet 5: Indicators
Worksheet 6: Supervisor tracking sheet
Worksheet 7: Health worker listing sheet
Worksheet 8: Interviewer performance review checklist
Worksheet 9: Priority indicators summary table
Worksheet 10: Indicator review
Worksheet 11: Indicator performance factors
Worksheet 12: Actions for improving indicators

Tools

Manual Module 2 : Community-Based Newborn, Child and Nutrition Services - v.11Nov2020
Community-Based Newborn, Child and Nutrition Services Manual Annexes - v.11Nov2020
Community-Based Newborn, Child and Nutrition Services Manual Worksheets - v.11Nov2020
Indicator .do files - v.25May2021

Products

Mozambique

Avaliação dos Serviços de Nutrição para Mães e Crianças no Sul de Moçambique
Relatorio sobre avaliaçao dos serviços de nutriçao para maes e crianças no sul de Moçambique (Projecto SANI - Inhambane)
Report on the evaluation of nutrition services for mothers and children in southern Mozambique
Feasibility and Acceptability of Innovative Data Collection Methods of Quality of Care in Southern Mozambique
SANI Evidence on implementation strength policy brief

Mali

Atelier sur l’évaluation du Programme de Prise en Charge Communautaire des enfants de 0 à 59 mois dans les Régions de Koulikoro et Sikasso au Mali en 2018
Évaluation du Programme de Prise en Charge Communautaire des enfants de 0 à 59 mois dans les Régions de Koulikoro et Sikasso au Mali en 2018
Evaluation of the Community Management Program in the Koulikoro and Sikasso Regions in Mali in 2018
Evaluation Of The Community Management Program For Children Aged 0-59 Months In The Koulikoro And Sikasso Regions In Mali In 2018 Dissemination presentation

Table 1: Community-based newborn, child and nutrition care survey questionnaires

Top of page | Module 1 | Module 3

 Intrapartum care

The IP Care tool is a set of questionnaires and tools and resources that allows evaluators to capture and analyze data on the implementation strength and quality of programs strengthening Basic Emergency Obstetric Care (BEmONC). This suite of tools provides a comprehensive assessment of facility structure and program readiness, provider knowledge, provider practice, and client care. The tool was developed and tested during an evaluation of a BEmONC (Basic emergency obstetric and newborn care) program in the Simiyu Region of Tanzania in 2019.

Indicators measured include:

% of health workers trained in BEmONC in the previous 2 years
% of facilities with no stockouts of essential medicines
% of facilities open 24/7 for delivery services
% of laboring women with a partograph correctly completed
% of newborns who receive immediate drying

The questionnaires contain the following forms:

1. The Implementation Strength Assessment, which involves a Facility Assessment and interviews with Health Facility Workers. 

2. Case Vignettes, which are scenarios that Health Facility Workers respond to as if they were real. Included are the following topics:

• “normal” delivery,

• Pre-eclampsia,

• Post-partum hemorrhage, and

• Neonatal resuscitation.

3. A direct observation tool, during which data collectors observe a delivery beginning at assessment and admission of a woman in labor.

The package also includes manuals that describe, in detail, the steps necessary to plan, implement, and analyze data from IP-specific implementation strength and quality assessments: data collection tools, sample size calculators, budget templates, supervision tools, logistics guidelines, analysis plans, among others.

Worksheet 1: Survey sample size calculator
Worksheet 2: Facility selection sheet
Worksheet 3: Supply calculator
Worksheet 4: Budget template
Worksheet 5: Indicators
Worksheet 6: Supervisor tracking sheet
Worksheet 7: Facility listing sheet
Worksheet 8: Health worker listing sheet
Worksheet 9: Interviewer performance review checklist
Worksheet 10: Priority indicators summary table
Worksheet 11: Indicator review
Worksheet 12: Indicator performance factors
Worksheet 13: Actions for improving indicators

Tools

Manual Module 3: Intrapartum Care - v.28Oct2020

Intrapartum Care Manual Annexes - v.28Oct2020

Intrapartum Care Manual Worksheets - v.28Oct2020

Products

Uzazi Uzima Mid-term Evaluation - v.25May2021

Top of page | Module 1 | Module 2