WHO 032865 orig

The primary objective of a vaccination coverage survey is to provide a coverage estimate for selected vaccines. This page provides a list of documents related to the design, implementation, analysis, and reporting of vaccination coverage surveys. The documents have been organized into the categories below and are provided here to assist in conducting an effective vaccination coverage survey.

This page was created and is maintained by the WHO Expanded Programme on Immunization (EPI) Strategic Information Group. Much of this material was brought together at the "WHO Tools and Guidance on Immunization Data Quality and Vaccination Coverage Survey" meeting, which took place in Istanbul, Turkey in December 2015. For more information on this meeting, click here. To access the meeting report, click here.

  

Survey forms and tools

This collection contains useful documents for carrying out a vaccination coverage survey, including sample surveys, sample log sheets, a project timeline, and equipment lists. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e40a6631c5a

Sample questionnaire - Bolivia

This document contains the vaccination coverage survey for Bolivia in 2013.

Link: http://www.technet-21.org/library/main/2410-survey-sample-questionnaire-bolivia

Sample household log sheet - Haiti

This document contains a template for a household log sheet, used in a vaccination coverage survey conducted in Haiti.

Link: http://www.technet-21.org/library/main/2411-survey-sample-household-log-sheet-haiti

Sample timeline (1) - Haiti

This document contains a timeline of when certain activities for a serosurvey and EPI survey should be undertaken, from submitting the protocol to sharing results with partners.

Link: http://www.technet-21.org/library/main/2412-survey-sample-timeline-1-haiti

Sample timeline (2) - Haiti

This document is a sample of a timeline for a vaccination coverage survey conducted in Haiti.

Link: http://www.technet-21.org/library/main/2418-survey-sample-timeline-2-haiti

Sample consent form - Haiti

This document contains an example of a verbal consent form to participate in a household survey.

Link: http://www.technet-21.org/library/main/2413-survey-sample-consent-form-haiti

Sample MR coverage log - Haiti

This document contains an example of a measles and rubella vaccination coverage log for children aged 1 to 9 years of age.

Link: http://www.technet-21.org/library/main/2414-survey-sample-mr-coverage-log-haiti

Sample questionnaire - Haiti

This document contains a sample questionnaire to be used in a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2416-survey-sample-questionnaire-haiti

Sample recruitment log - Haiti

This document contains an example of a recruitment log sheet for households approached for a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2417-survey-sample-recruitment-log-haiti

Sample Equipment List - DHS

This document contains a list of necessary items for conducting a DHS survey.

Link: http://www.technet-21.org/library/main/2419-survey-sample-equipment-list-dhs

 

Budget examples

This collection includes documents to assist with the development of a budget for a vaccination coverage survey. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e403d85a4b7

Budget example

This document outlines a sample budget template for a vaccination coverage survey performed in Haiti, including items for the project coordinator, training workshop, data collection, specimen collection and processing, data management, contract costs, data analysis, report development, and presentation of findings.

Link: http://www.technet-21.org/library/main/2402-survey-budget-example

DHS budget example

This document outlines a sample budget template for a vaccination coverage survey in any area, including items for administration, household listing, pretest, main training, survey field work, data processing, laboratory costs, and national seminar.

Link: http://www.technet-21.org/library/main/2403-survey-dhs-budget-example

WHO cluster survey budget template

This document contains a suvey budget template for a WHO vaccination coverage cluster survey (see Annex C)

Link: http://www.technet-21.org/library/main/2404-survey-list-of-budget-examples

DHS budget template

This document contains a template for a DHS budget (see Annex B)

Link: http://www.technet-21.org/library/main/2552-survey-dhs-budget-template

MICS budget template

This document contains a spreadsheet template for a MICS budget

Link: http://www.technet-21.org/library/main/2551-survey-mics-budget-template

 

Analytic plan

This collection includes documents to assist with the development of an analytic plan for vaccination coverage survey data. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e00e70eb407

Analytic definitions

This document outlines a list of important analytic definitions, such as those for "household" and "on time doses".

Link: http://www.technet-21.org/library/main/2389-survey-analytic-definitions

Suggested analyses

This document outlines a list of suggested analyses for a typical vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2391-survey-suggested-analyses

Definitions of timeliness

This document outlines the recommended age for Hep B, DTP, Hib, Polio, Measles and MMR doses, and definitions for what constitutes an early, on time, delayed, or late dose (PAHO 2014).

Link: http://www.technet-21.org/library/main/2400-survey-definitions-of-timeliness

 

Report outline

This collection contains an outline for developing a report for a vaccination coverage survey. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56f0f967d21f0

Vaccination coverage survey report outline

This is a document containing an outline of all the sections that should be included in a vaccination coverage survey report.

Link: http://www.technet-21.org/library/main/2494-survey-vaccination-coverage-survey-report-outline

 

Request for proposal templates

This collection contains several examples of request for proposals. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56f0f8c8efe49

Request for proposal - Confidentiality

This document is a template for a confidentiality undertaking.

Link: http://www.technet-21.org/library/main/2495-survey-request-for-proposal-confidentiality

Request for proposal template - 50k+

This document is a template for a formal request for proposal for a project over $50,000.00.

Link: http://www.technet-21.org/library/main/2496-survey-request-for-proposal-template-50k

Request for proposal template - less than 50k

This document is a template for a formal request for proposal for a project under $50,000.00.

Link: http://www.technet-21.org/library/main/2497-survey-request-for-proposal-template

 

Vaccination card picture management

This collection contains information on the protocol for collecting and managing vaccination card pictures. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56f0f9a15faa3

SOP - vaccination card picture

This document outlines the standard operating procedure for vaccination card picture management. It includes details about training, taking the pictures, downloading the pictures, and renaming the pictures.

Link: www.technet-21.org/en/library/explore/immunization-information-systems-coverage-monitoring/4646

 

Sample terms of reference and profiles

This collection contains several sample terms of reference for positions such as survey coordinator and interviewers. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e44737000a3

Sample personnel descriptions - Haiti

This document contains the primary responsibilities of each personnel member for a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2460-survey-sample-personnel-descriptions-haiti

ToR - Data entry clerk for MMR vaccination coverage survey

This document is an example of a ToR for a data entry clerk for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2459-survey-tor-data-entry-clerk-for-mmr-vaccination-coverage-survey

ToR - Coordinator for MMR vaccination coverage survey

This document is an example of a ToR for a coordinator for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2458-survey-tor-coordinator-for-mmr-vaccination-coverage-survey

ToR - PH Nurse and Nurse Practitioner for MMR vaccination coverage survey

This document is an example of a ToR for public health nursing supervisor and nurse practitioner for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2457-survey-tor-ph-nurse-and-np-for-mmr-vaccination-coverage-survey

ToR - Data management coordinator for MMR vaccination coverage survey

This document is an example of a ToR for a data management coordinator for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2456-survey-tor-data-management-coordinator-for-mmr-vaccination-coverage-survey

ToR - Data analysis and report writing for MMR vaccination coverage survey

This document is an example of a ToR for data analysis and report writing for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2455-survey-tor-data-analysis-and-report-writing-for-mmr-vaccination-coverage-survey

ToR - Interviewers for MMR vaccination coverage survey

This document is an example of a ToR for an interviewer for an MMR vaccination coverage survey in St. Lucia.

Link: http://www.technet-21.org/library/main/2454-survey-tor-interviewers-for-mmr-vaccination-coverage-survey

ToR - Survey coordinator for supplemental national coverage survey

This document is an example of a ToR for a survey coordinator for a supplemental national immunization coverage survey in Nigeria.

Link: http://www.technet-21.org/library/main/2453-survey-tor-survey-coordinator-for-supplemental-national-coverage-survey

ToR - Consultant for pilot test of post SIA survey

This document is an example of a ToR for a consultant position for a pilot test of post-SIA survey guidelines for Burkina Faso.

Link: http://www.technet-21.org/library/main/2451-survey-tor-consultant-for-pilot-test-of-post-sia-survey

ToR - Consultant for pilot test of post intensified immunization activity

This document is an example of a ToR for a consultant position for a post intensified immunization activity for measles and rubella coverage.

Link: http://www.technet-21.org/library/main/2452-survey-tor-consultant-for-pilot-test-of-post-intensified-immunization-activity

 

Current WHO reference manuals

This collection contains current WHO reference manuals on vaccination coverage surveys. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e436326f970

WHO vaccination coverage cluster surveys reference manual 2015

This document contains the WHO vaccination coverage cluster surveys reference manual working draft, last updated July 2015.

Link: http://www.technet-21.org/library/main/2444-survey-who-vaccination-coverage-cluster-surveys-reference-manual-2015

WHO vaccination coverage cluster surveys reference manual briefing note

This document outlines the main enhancements to the 2015 reference manual update.

Link: http://www.technet-21.org/library/main/2445-survey-who-vaccination-coverage-cluster-surveys-reference-manual-briefing-note

WHO vaccination coverage cluster surveys reference manual presentation

This presentation reviews the new content of the WHO vaccination coverage cluster survey reference manual 2015.

Link: http://www.technet-21.org/library/main/2510-survey-who-vaccination-coverage-cluster-surveys-reference-manual-presentation

 

Other WHO reference manuals

This collection contains various WHO reference manuals related to vaccination coverage surveys. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e43bd1a380d

WHO immunization coverage cluster survey reference manual 2005
WHO vaccination coverage using clustered LQA field manual 2012

Assessing Vaccination Coverage Levels Using Clustered Lot Quality Assurance Sampling, edited for the global polio eradication initiative.

Link: http://www.technet-21.org/library/main/2449-survey-who-vaccination-coverage-using-clustered-lqa-field-manual-2012

WHO Hep B immunization surveys reference manual 2012

Sample design and procedures for Hepatitis B immunization surveys: A companion to the WHO cluster survey reference manual.

Link: http://www.technet-21.org/library/main/2448-survey-who-hep-b-immunization-surveys-reference-manual-2012

 

DHS training manuals

This collection contains documents to assist with the training of multiple members within a vaccination coverage survey team. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e405543e4c4

DHS manual for household listing - Liberia (2012-2013)

This document contains the DHS manual for the Liberia DHS conducted in 2012-2012. It outlines the specifics of determining household listings in preparation for the survey. The sections include responsibilities of listing staff, locating clusters, preparing the location map and the sketch map, listing of households, segmentation of large EA, quality control, GPS waypoint collection, and examples of forms for mapping and segmenting.

Link: http://www.technet-21.org/library/main/2448-survey-who-hep-b-immunization-surveys-reference-manual-2012

DHS manual for interviewers - General (2015)

This document contains a general training manual for DHS interviewers. It outlines items in the following sections: conducting an interview, fieldwork procedures, general procedures for completing the questionnaire, household questionnaire, woman`s questionnaire, and man`s questionnaire.

Link: http://www.technet-21.org/library/main/2406-survey-dhs-manual-for-interviewers-general-2015

DHS manual for supervisors and editors - General (2015)

This document contains a general training manual for DHS field supervisors and editors. It contains items in the following sections: introduction to the DHS survey, preparing for fieldwork, organizing and supervising fieldwork, maintaining fieldwork control sheets, monitoring interviewer performance, and editing questionnaires.

Link: http://www.technet-21.org/library/main/2407-survey-dhs-manual-for-supervisors-and-editors-general-2015

DHS manual for field staff- General (2009)

This document contains a general training manual for DHS field staff. It contains items in the following sections: Recruitment of field workers, administrative and logistical aspects of training, content of the training course, supervisor and field editor training, fieldwork supervision.

Link: http://www.technet-21.org/library/main/2408-survey-dhs-manual-for-field-staff-general-2009

Sample training agenda

This document contains an example of a potential training agenda, including agendas for regional and zonal supervisors, field supervisors, and enumerators.

Link: http://www.technet-21.org/library/main/2409-survey-sample-training-agenda

 

Introductory presentations

This collection contains a series of presentations outlining the basic steps of a vaccination coverage survey, as well as some presentations on commonly asked questions and variations on a coverage survey. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e44c55eed99

Steps 1-2-3 presentation

This presentation contains a summary of step 1 (assessing need for a survey), step 2 (creating a steering group), and step 3 (defining survey scope and budget) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2473-survey-steps-1-2-3-presentation

Steps 4-5-6 presentation

This presentation contains a summary of step 4 (setting a survey schedule), step 5 (developing a survey proposal), and step 6 (confirming that funding is in place) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2461-survey-steps-4-5-6-presentation

Steps 7-8-9 presentation

This presentation contains a summary of step 7 (deciding who will conduct the survey), step 8 (finalizing the survey protocol), and step 9 (getting ethical approval) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2462-survey-steps-7-8-9-presentation

Steps 10-11-12 presentation

This presentation contains a summary of step 10 (designing data collection tools), step 11 (hiring staff and coordinating logistics), and step 12 (selecting a sample) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2502-survey-steps-10-11-12-presentation

Step 13 presentation

This presentation contains a summary of step 13 (training staff) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2503-survey-step-13-presentation

Step 14 presentation

This presentation contains a summary of step 14 (conducting field work) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2504-survey-step-14-presentation

Step 15-16 presentation

This presentation contains a summary of step 15 and 16 (analyzing data) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2505-survey-step-15-16-presentation

Step 17 presentation

This presentation contains a summary of step 17 (interpreting and sharing results) of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2506-survey-step-17-presentation

FAQ presentation

This presentation contains a list of responses to questions that arose from Steps 1-3 of a vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2507-survey-faq-presentation

Coverage survey variations

This presentation contains a brief description of different variations of vaccination coverage survey.

Link: http://www.technet-21.org/library/main/2508-survey-coverage-survey-variations

Methods used for monitoring

This presentation contains a table of the characteristics, advantages, and limitations of the methodologies used for monitoring vaccination coverage.

Link: http://www.technet-21.org/library/main/2509-survey-methods-used-for-monitoring

WHO vaccination coverage cluster surveys reference manual presentation

This presentation reviews the new content of the WHO vaccination coverage cluster survey reference manual 2015.

Link: http://www.technet-21.org/library/main/2510-survey-who-vaccination-coverage-cluster-surveys-reference-manual-presentation

 

Coverage survey publications

This collection contains several publications on vaccination coverage surveys from various regions and countries. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56f29b073d461

Publication: Diaz-Ortega Salud Publica Mex 2013 (Mexico)

Publication abstract: Objective. To assess vaccination coverage of children and adolescents. Materials and methods. Study based on National Health and Nutrition Survey 2012. Results. Coverage in 80%. No health insurance and maternal or adolescent illiteracy were explanatory variables for incomplete schema. Conclusions. Results suggest it is necessary to strengthen information systems, health promotion, training, and daily vaccination without restrictive schedules, ensuring timely and adequate supply of vaccines..

Link: http://www.technet-21.org/library/main/2522-survey-diaz-ortega-salud-publica-mex-2013-mexico

Publication: Rainey Vaccine 2012 (Haiti)

Publication abstract: INTRODUCTION: Since 1977, vaccinations to protect against tuberculosis, diphtheria, tetanus, pertussis, polio, and measles (and rubella since 2009) have been offered to children in Haiti through the routine immunization program. From April to July 2009, a national vaccination coverage survey was conducted to assess the success of the routine immunization program at reaching children in Haiti. METHODS: A multi-stage cluster survey was conducted using a modified WHO method for household sampling. A standardized questionnaire was administered to collect vaccination histories, demographic information, and reasons for under-vaccination of children aged 12-23 months. A child who received the eight recommended routine vaccinations was considered fully vaccinated. The routine vaccination schedule was used to define valid doses and estimate the percentage of children vaccinated on time. RESULTS: Among 1345 children surveyed, 40.4% (95% CI: 36.6-44.2) of the 840 children with vaccination cards had received all eight recommended vaccinations. Coverage was highest for the Bacille Calmette-Guérin vaccine (87.3%), the first doses of the diphtheria-tetanus-pertussis vaccine (92.0%), and oral poliovirus vaccine (93.4%) and lowest for measles vaccine (46.9%). Timely vaccination rates were lower. Assuming similar coverage for the 505 children without cards, coverage with the complete vaccination series among all surveyed children 31.9%. Reasons for under-vaccination included not having enough time to reach the vaccination location (24.8%), having a child who was ill (13.8%), and not knowing when, or forgetting, to go for vaccination (12.8%). CONCLUSIONS AND RECOMMENDATIONS: Coverage for early-infant vaccines was high; however, most children did not complete the full vaccination series, and many children received vaccinations later than recommended. Efforts to improve the immunization program should include increasing the frequency of outreach services, training for vaccination staff to minimize missed opportunities, and better communicating the timing of vaccinations to encourage caregivers to bring their children for vaccinations at the recommended age. Efforts to promote the benefits of vaccination and card retention are also needed.

Link: http://www.technet-21.org/library/main/2511-survey-rainey-vaccine-2012-haiti

Publication: Tohme Tropical Med & Int Health 2014 (Haiti)

Publication abstract: OBJECTIVES: We conducted a nationwide survey to assess measles containing vaccine (MCV) coverage among children aged 1-9 years in Haiti and identify factors associated with vaccination before and during the 2012 nationwide supplementary immunisation activities (SIA). METHODS: Haiti was stratified into five geographic regions (Metropolitan Port-au-Prince, North, Centre, South and West), 40 clusters were randomly selected in each region, and 35 households were selected per cluster. RESULTS: Among the 7000 visited households, 75.8% had at least one child aged 1-9 years; of these, 5279 (99.5%) households consented to participate in the survey. Of 9883 children enrolled, 91% received MCV before and/or during the SIA; 31% received MR for the first time during the SIA, and 50.7% received two doses of MCV (one before and one during the 2012 SIA). Among the 1685 unvaccinated children during the SIA, the primary reason of non-vaccination was caregivers not being aware of the SIA (31.0%). Children aged 1-4 years had significantly lower MR SIA coverage than those aged 5-9 years (79.5% vs. 84.8%) (P < 0.0001). A higher proportion of children living in the West (12.3%) and Centre (11.2%) regions had never been vaccinated than in other regions (4.8-9.1%). Awareness, educational level of the mother and region were significantly associated with MR vaccination during and before the SIA (P < 0.001). CONCLUSIONS: The 2012 SIA successfully increased MR coverage; however, to maintain measles and rubella elimination, coverage needs to be further increased among children aged 1-4 years and in regions with lower coverage.

Link: http://www.technet-21.org/library/main/2514-survey-publication-tohme-tropical-med-int-health-2014-haiti

Publication: Suarez-Castaneda Vaccine 2014 (El Salvador)

Publication abstract: While assessing immunization programmes, not only vaccination coverage is important, but also timely receipt of vaccines. We estimated both vaccination coverage and timeliness, as well as reasons for non-vaccination, and identified predictors of delayed or missed vaccination, for vaccines of the first two years of age, in El Salvador. We conducted a cluster survey among children aged 23-59 months. Caregivers were interviewed about the child immunization status and their attitudes towards immunization. Vaccination dates were obtained from children immunization cards at home or at health facilities. We referred to the 2006 vaccination schedule for children below two years: one dose of BCG (Bacillus Calmette-Guérin) at birth; rotavirus at two and four months; three doses of pentavalent - DTP (diphtheria-tetanus-pertussis), hepatitis B, and Haemophilus influenzae type b (Hib) - and of oral poliomyelitis vaccine (polio) at two, four, and six months; first MMR (measles-mumps-rubella) at 12 months; and first boosters of DTP and OPV at 18 months. Timeliness was assessed with Kaplan-Meier analysis; Cox and logistic regression were used to identify predictors of vaccination. We surveyed 2550 children. Coverage was highest for BCG (991%; 95% CI: 98.8-99.5) and lowest for rotavirus, especially second dose (86.3%; 95% CI: 84.2-88.4). The first doses of MMR and DTP had 991% (95% CI: 98.5-99.6) and 977% (95% CI: 970-985), respectively. Overall coverage was 837% (95% CI: 81.4-86.0); 96.4% (95% CI: 95.4-97.5), excluding rotavirus. However, only 26.7% (95% CI: 24.7-28.8) were vaccinated within the age interval recommended by the Expanded Programme on Immunization. Being employed and using the bus for transport to the health facility were associated with age-inappropriate vaccinations; while living in households with only two residents and in the "Paracentral", "Occidental", and "Oriental" regions was associated with age-appropriate vaccinations. Vaccination coverage was high in El Salvador, but general timeliness and rotavirus uptake could be improved.

Link: http://www.technet-21.org/library/main/2513-survey-suarez-castaneda-vaccine-2014-el-salvador

Publication: PAHO Newsletter 2014 (Honduras)

Publication summary: Secondary analysis of DHS data in Honduras to analyse timeliness of vaccination receipt and trends in co-administration of vaccines.

Link: http://www.technet-21.org/library/main/2512-survey-paho-newsletter-2014-honduras

Publication: Gunthmann Vaccine 2012 (Brazil)

Publication abstract: We conducted a national cross-sectional survey to investigate vaccination coverage (VC) in health care personnel (HCP) working in clinics and hospitals in France. We used a two-stage stratified random sampling design to select 1127 persons from 35 health care settings. Data were collected by face-to-face interviews and completed using information gathered from the occupational health doctor. A total of 183 physicians, 110 nurses, 58 nurse-assistants and 101 midwives were included. VC for compulsory vaccinations was 91.7% for hepatitis B, 95.5% for the booster dose of diphtheria-tetanus-polio (DTP), 94.9% for BCG. For non-compulsory vaccinations, coverage was 11.4% for the 10 year booster of the DTP pertussis containing vaccine, 49.7% for at least one dose of measles, 29.9% for varicella and 25.6% for influenza. Hepatitis B VC did not differ neither between HCP working in surgery and HCP in other sectors, nor in surgeons and anaesthesiologists compared to physicians working in medicine. Young HCP were better vaccinated for pertussis and measles (p<0.01), and those working in an obstetric or a paediatric ward were better vaccinated for influenza and pertussis (p<0.01). HCP are overall well covered by compulsory vaccinations, whereas VC for non-compulsory vaccinations is very insufficient. The vaccination policy regarding these latter vaccinations should be reinforced in France.

Link: http://www.technet-21.org/library/main/2520-survey-gunthmann-vaccine-2012-brazil

Publication: Suarez-Castaneda Vaccine 2015 (El Salvador)

Publication abstract: Rotavirus vaccine was introduced in El Salvador in 2006 and is recommended to be given concomitantly with DTP-HepB-Haemophilus influenzae type b (pentavalent) vaccine at ages 2 months (upper age limit 15 weeks) and 4 months (upper age limit 8 months) of age. However, rotavirus vaccination coverage continues to lag behind that of pentavalent vaccine, even in years when national rotavirus vaccine stock-outs have not occurred. We analyzed factors associated with receipt of oral rotavirus vaccine among children who received at least 2 doses of pentavalent vaccine in a stratified cluster survey of children aged 24-59 months conducted in El Salvador in 2011. Vaccine doses included were documented on vaccination cards (94.4%) or in health facility records (5.6%). Logistic regression and survival analysis were used to assess factors associated with vaccination status and age at vaccination. Receipt of pentavalent vaccine by age 15 weeks was associated with rotavirus vaccination (OR: 5.1; 95% CI 2.7, 9.4), and receipt of the second pentavalent dose by age 32 weeks was associated with receipt of two rotavirus vaccine doses (OR: 5.0; 95% CI 2.1-12.3). Timely coverage with the first pentavalent vaccine dose was 88.2% in the 2007 cohort and 91.1% in the 2008 cohort (p=0.04). Children born in 2009, when a four-month national rotavirus vaccine stock-out occurred, had an older median age of receipt of rotavirus vaccine and were less likely to receive rotavirus on the same date as the same dose of pentavalent vaccine than children born in 2007 and 2008. Upper age limit recommendations for rotavirus vaccine administration contributed to suboptimal vaccination coverage. Survey data suggest that late rotavirus vaccination and co-administration with later doses of pentavalent vaccine among children born in 2009 helped increase rotavirus vaccine coverage following shortages.

Link: http://www.technet-21.org/library/main/2519-survey-suarez-castaneda-vaccine-2015-el-salvador

Publication: Sanchez BMC Public Health 2015 (Venezuela)

Publication abstract: BACKGROUND: Vaccination Week in the Americas (VWA) is an annual initiative in countries and territories of the Americas every April to highlight the work of national expanded programs on immunization (EPI) and increase access to vaccination services for high-risk population groups. In 2011, as part of VWA, Venezuela targeted children aged less than 6 years in 25 priority border municipalities using social mobilization to increase institution-based vaccination. Implementation of social communication activities was decentralized to the local level. We conducted a survey in one border municipality of Venezuela to evaluate the outcome of VWA 2011 and provide a snapshot of the overall performance of the routine EPI at that level. METHODS: We conducted a coverage survey, using stratified cluster sampling, in the Venezuelan municipality of Bolivar (bordering Colombia) in August 2011. We collected information for children aged 85%, with a few exceptions. CONCLUSION: Low levels of VWA awareness among caregivers probably contributed to the limited vaccination of eligible children during the VWA activities in Bolivar in 2011. However, vaccine coverage for most EPI vaccines was high. Additionally, high vaccination card availability and high participation in VWA among those caregivers aware of it in 2011 suggest public trust in the EPI program in the municipality. Health authorities have used survey findings to inform changes to the routine EPI and better VWA implementation in subsequent years.

Link: http://www.technet-21.org/library/main/2518-survey-sanchez-bmc-2015-venezuela

Publication: Kaiser WHO Bulletin 2015 (Africa)

Publication abstract: Objective To assess the methods used in the evaluation of measles vaccination coverage, identify quality concerns and provide recommendations for improvement. Methods We reviewed surveys that were conducted to evaluate supplementary measles immunization activities in eastern and southern Africa during 2012 and 2013. We investigated the organization(s) undertaking each survey, survey design, sample size, the numbers of study clusters and children per study cluster, recording of immunizations and methods of analysis. We documented sampling methods at the level of clusters, households and individual children. We also assessed the length of training for field teams at national and regional levels, the composition of teams and the supervision provided. Findings The surveys were conducted in Comoros, Eritrea, Ethiopia, Kenya, Lesotho, Malawi, Mozambique, Namibia, Rwanda, Swaziland, Uganda, Zambia and Zimbabwe. Of the 13 reports we reviewed, there were weaknesses in 10 of them for ethical clearance, 9 for sample size calculation, 6 for sampling methods, 12 for training structures, 13 for supervision structures and 11 for data analysis. Conclusion We recommend improvements in the documentation of routine and supplementary immunization, via home-based vaccination cards or other records. For surveys conducted after supplementary immunization, a standard protocol is required. Finally, we recommend that standards be developed for report templates and for the technical review of protocols and reports. This would ensure that the results of vaccination coverage surveys are accurate, comparable, reliable and valuable for programme improvement.

Link: http://www.technet-21.org/library/main/2517-survey-publication-kaiser-who-bulletin-2015-africa

Publication: Sheth Natl J Com Med 2012 (India)

Publication abstract: Objectives: To assess the immunization status of children of Gandhinagar (Rural) district and to compare it with the NFHS3/DLHS3 coverage results. Materials & Methods: A Multi-Indicator Cluster Survey (MICS) was planned and community-based cross-sectional survey was conducted in April 2008. The Study was conducted using 30 cluster technique. Proforma designed by UNICEF, modified by experts and approved for uniform use by department of health & family welfare, Government of Gujarat was used as a study tool. Statistical analysis used: Simple proportions and Chi-square test. Results: Coverage for BCG, OPV3, DPT3 & Measles were 92.04%, 85.23%, 83.71% & 82.20% respectively. BCG scar was seen in 83.95% of children out of those who received BCG. The proportions of fully immunized children were 79.55%. Unimmunized children were 4.16%. Dropout rate was 9.05% for BCG-DPT3, 10.69% for BCG-Measles & 7.53% for DPT1-DPT3. Compared to NFHS3 (2005-06) as well as DLHS3 (Gandhinagar district, 2007-08) the current survey shows higher coverage for all vaccines except measles which was higher in DLHS3 (87.3%). Gender wise difference in the coverage of different vaccines or various dropout rates was not statistically significant. Conclusions: Although the vaccination coverage shows higher coverage than previous studies, it is still below the minimum targets set as national goal.

Link: http://www.technet-21.org/library/main/2527-survey-publication-sheth-natl-j-com-med-2012-india

Publication: Hu Asia Pacific J Public Health 2015 (China)

Publication abstract:The study aimed to assess the determinants of immunization coverage in children born in 2008-2009, living in Zhejiang Province. The World Health Organization's cluster sampling technique was applied. Immunization coverage of 5 vaccines was assessed: BCG vaccine, diphtheria and tetanus toxoids and pertussis vaccine, poliomyelitis vaccine, hepatitis B vaccine, and measles-containing vaccine. Determinants for age-appropriate immunization coverage rates were explored using logistic regression models. Immunization coverage of 5 vaccines were all greater than 90%, but the age-appropriate immunization coverage rates for 3 months and for first dose of measles-containing vaccine was 41.3% and 64.5%, respectively. Siblings in household, mother's education level, household registration, socioeconomic level of resident areas, satisfaction with clinical immunization service, and convenient access to local immunization clinic were associated with age-appropriate coverage rates. Age-appropriate immunization coverage rates should be given more attention and should be considered as a benchmark to strive for in the future intervention.

Link: http://www.technet-21.org/library/main/2526-survey-publication-hu-asia-pacific-j-public-health-2015-china

Publication: Minetti Emerg Themes Epi 2012 (Mali)

Publication abstract: BACKGROUND: Estimation of vaccination coverage at the local level is essential to identify communities that may require additional support. Cluster surveys can be used in resource-poor settings, when population figures are inaccurate. To be feasible, cluster samples need to be small, without losing robustness of results. The clustered LQAS (CLQAS) approach has been proposed as an alternative, as smaller sample sizes are required. METHODS: We explored (i) the efficiency of cluster surveys of decreasing sample size through bootstrapping analysis and (ii) the performance of CLQAS under three alternative sampling plans to classify local VC, using data from a survey carried out in Mali after mass vaccination against meningococcal meningitis group A. RESULTS: VC estimates provided by a 10 × 15 cluster survey design were reasonably robust. We used them to classify health areas in three categories and guide mop-up activities: i) health areas not requiring supplemental activities; ii) health areas requiring additional vaccination; iii) health areas requiring further evaluation. As sample size decreased (from 10 × 15 to 10 × 3), standard error of VC and ICC estimates were increasingly unstable. Results of CLQAS simulations were not accurate for most health areas, with an overall risk of misclassification greater than 0.25 in one health area out of three. It was greater than 0.50 in one health area out of two under two of the three sampling plans. CONCLUSIONS: Small sample cluster surveys (10 × 15) are acceptably robust for classification of VC at local level. We do not recommend the CLQAS method as currently formulated for evaluating vaccination programmes.

Link: http://www.technet-21.org/library/main/2525-survey-publication-minetti-emerg-themes-epi-2012-mali

Publication: Caini Vaccine 2013 (Niger)

Publication abstract: MenAfriVac™ is a conjugate vaccine against meningitis A specifically designed for Africa. In Niger, the MenAfriVac™ vaccination campaign was conducted in people aged 1-29 years in three phases. The third phase was conducted in November/December 2011 targeting more than 7 million people. We estimated vaccination coverage for the third phase; classified the 31 target districts according to vaccination coverage levels; analysed the factors associated with being vaccinated; described the reasons for non-vaccination; and estimated coverage of the MenAfriVac™ introduction in Niger by aggregating data from all three phases. We classified the districts by clustered lot quality assurance sampling according to a 75% lower threshold and a 90% upper threshold. We estimated coverage using a minimum cluster-sample of 30 x 10 in each region. Two criteria were used to document vaccination status: presentation of vaccination card only or by card and/or verbal history of vaccination (card+history). We surveyed 2390 persons. After the third phase, estimated coverage was 68.8% (95% CI 64.9-72.8) by card only and 90.9% (95% CI 88.6-93.2) by card+history. Five districts were accepted for coverage above 75% based on card only, whereas 25 were accepted based on card+history. Factors positively associated with being vaccinated were younger age (

Link: http://www.technet-21.org/library/main/2524-survey-publication-caini-niger-vaccine-2013

Publication: Barata J Epidemiol Community Health 2012 (Brazil)

Publication abstract: BACKGROUND: Since 1988, Brazil's Unified Health System has sought to provide universal and equal access to immunisations. Inequalities in immunisation may be examined by contrasting vaccination coverage among children in the highest versus the lowest socioeconomic strata. The authors examined coverage with routine infant immunisations from a survey of Brazilian children according to socioeconomic stratum of residence census tract. METHODS: The authors conducted a household cluster survey in census tracts systematically selected from five socioeconomic strata, according to average household income and head of household education, in 26 Brazilian capitals and the federal district. The authors calculated coverage with recommended vaccinations among children until 18 months of age, according to socioeconomic quintile of residence census tract, and examined factors associated with incomplete vaccination. RESULTS: Among 17,295 children with immunisation cards, 14,538 (82.6%) had received all recommended vaccinations by 18 months of age. Among children residing in census tracts in the highest socioeconomic stratum, 77.2% were completely immunised by 18 months of age versus 81.2%-86.2% of children residing in the four census tract quintiles with lower socioeconomic indicators (p<0.01). Census tracts in the highest socioeconomic quintile had significantly lower coverage for bacille Calmette-Guérin, oral polio and hepatitis B vaccines than those with lower socioeconomic indicators. In multivariable analysis, higher birth order and residing in the highest socioeconomic quintile were associated with incomplete vaccination. After adjusting for interaction between socioeconomic strata of residence census tract and household wealth index, only birth order remained significant. CONCLUSIONS: Evidence from Brazilian capitals shows success in achieving high immunisation coverage among poorer children. Strategies are needed to reach children in wealthier areas.

Link: http://www.technet-21.org/library/main/2523-survey-barata-j-epidemiol-community-health-2012-brazil

Publication: Ozcirpici BMC Public Health 2014 (Turkey)

Publication abstract: Background Health care systems in many countries are changing for a variety of reasons. Monitoring of community-based services, especially vaccination coverage, is important during transition periods to ensure program effectiveness. In 2005, Turkey began a transformation from a “socialization of health services” system to a “family medicine” system. The family medicine system was implemented in the city of Gaziantep, in December, 2010. Methods Two descriptive, cross-sectional studies were conducted in Gaziantep city center; the first study was before the transition to the family medicine system and the second study was one year after the transition. The Lot Quality Technique methodology was used to determine the quality of vaccination services. The population studied was children aged 12–23 months. Data from the two studies were compared in terms of vaccination coverage and lot service quality to determine whether there were any changes in these parameters after the transition to a family service system. Results A total of 93.7% of children in Gaziantep were fully vaccinated before the transition. Vaccination rates decreased significantly to 84.0% (p <0.005) after the family medicine system was implemented. The number of unacceptable vaccine lots increased from 5 lots before the transition to 21 lots after the establishment of the family medicine system. Conclusions The number of first doses of vaccine given was higher after family medicine was implemented; however, the numbers of second, third, and booster doses, and the number of children fully vaccinated were lower than before transition. Acceptable and unacceptable lots were not the same before and after the transition. Different health care personnel were employed at the lots after family medicine was implemented. This result suggests that individual characteristics of the health care personnel working in a geographic area are as important as the socioeconomic and cultural characteristics of the community.

Link: http://www.technet-21.org/library/main/2521-survey-ozcirpici-bmc-2014-turkey

Publication: Henderson WHO Bulletin 1982 (Various countries)

Publication abstract: A simplified cluster sampling method, involving the random selection of 210 children in 30 clusters of 7 each, has been used by the Expanded Programme on Immunization to estimate immunization coverage. This paper analysis the performance of the method in 60 actual surveys and 1500 computer simulated surveys. Although the method gives a proportion of results with confidence limits exceeding the desired maximum of 10 absolute percentage points, it is concluded that is performs satisfactorily.

Link: http://www.technet-21.org/library/main/2531-survey-henderson-who-bulletin-1982-various-countries

Publication: Murray Lancet 2003 (Global)

Publication abstract: BACKGROUND: Monitoring and assessment of coverage rates in national health programmes is becoming increasingly important. We aimed to assess the accuracy of officially reported coverage rates of vaccination with diphtheria-tetanus-pertussis vaccine (DTP3), which is commonly used to monitor child health interventions. METHODS: We compared officially reported national data for DTP3 coverage with those from the household Demographic and Health Surveys (DHS) in 45 countries between 1990 and 2000. We adjusted survey data to reflect the number of valid vaccinations (ie, those administered in accordance with the schedule recommended by WHO) using a probit model with sample selection. The model predicted the probability of valid vaccinations for children, including those without documented vaccinations, after correcting for bias from differences between the children with and without documented information on vaccination. We then assessed the extent of survey bias and differences between officially reported data and those from DHS estimates. FINDINGS: Our results suggest that officially reported DTP3 coverage is higher than that reported from household surveys. This size of the difference increases with the rate of reported coverage of DTP3. Results of time-trend analysis show that changes in reported coverage are not correlated with changes reported from household surveys. INTERPRETATION: Although reported data might be the most widely available information for assessment of vaccination coverage, their validity for measuring changes in coverage over time is highly questionable. Household surveys can be used to validate data collected by service providers. Strategies for measurement of the coverage of all health interventions should be grounded in careful assessments of the validity of data derived from various sources.

Link: http://www.technet-21.org/library/main/2530-survey-murray-lancet-2003-global

Publication: Xu Hum Vaccin Immunother 2012 (China)

Publication abstract: Varicella vaccine has been licensed in China for decade to be used as single dose in children aged ≥ 12 mo of age in private sector. Little data were available on varicella uptake to date in China yet. A cross-sectional study was conducted in Shandong Province in May 2011 to examine varicella vaccination coverage among children aged 16-40 mo and examine factors associated with varicella vaccine uptake. The overall coverage among children eligible for varicella vaccine was 62% (range 16.7-94.7% by county), much lower than the coverage of the eight vaccines included in the national immunization program (all above 97%). Though proximity to immunization services (< 5 km) was linked with higher vaccine uptake (62.6 vs. 37.4%, p = 0.02), county-level economic development (77.8, 61.0 and 47.1% for developed, sub-developed and developing regions, respectively, p < 0.001) played an even more important role in varicella vaccination. Moreover, there was little variation in coverage of vaccines included in the national immunization program along with county-level economic development. Even though varicella vaccine uptake is relatively high for use on a private basis, the vaccination coverage is not high enough to prevent epidemiology shift to adolescents and adults who are more prone to develop severe outcomes to varicella. Further enhancement on varicella vaccination coverage is necessary and inclusion to national immunization program seems to be a promising option for achieving and maintaining high coverage.

Link: http://www.technet-21.org/library/main/2529-survey-publication-xu-hum-vaccin-immunother-2012-china

Publication: Bagonza BMC Public Health 2013 (Uganda)

Publication abstract: BACKGROUND: Following an outbreak of yellow fever in northern Uganda in December 2010, Ministry of Health conducted a massive emergency vaccination campaign in January 2011. The reported vaccination coverage in Pader District was 75.9%. Administrative coverage though timely, is affected by incorrect population estimates and over or under reporting of vaccination doses administered. This paper presents the validated yellow fever vaccination coverage following massive emergency immunization campaigns in Pader district. METHODS: A cross sectional cluster survey was carried out in May 2011 among communities in Pader district and 680 respondents were indentified using the modified World Health Organization (WHO) 40 × 17 cluster survey sampling methodology. Respondents were aged nine months and above. Interviewer administered questionnaires were used to collect data on demographic characteristics, vaccination status and reasons for none vaccination. Vaccination status was assessed using self reports and vaccination card evidence. Our main outcomes were measures of yellow fever vaccination coverage in each age-specific stratum, overall, and disaggregated by age and sex, adjusting for the clustered design and the size of the population in each stratum. RESULTS: Of the 680 survey respondents, 654 (96.1%, 95% CI 94.9 - 97.8) reported being vaccinated during the last campaign but only 353 (51.6%, 95% CI 47.2 - 56.1) had valid yellow fever vaccination cards. Of the 280 children below 5 years, 269 (96.1%, 95% CI 93.7 - 98.7) were vaccinated and nearly all males 299 (96.9%, 95% CI 94.3 - 99.5) were vaccinated. The main reasons for none vaccination were; having travelled out of Pader district during the campaign period (40.0%), lack of transport to immunization posts (28.0%) and, sickness at the time of vaccination (16.0%). CONCLUSIONS: Our results show that actual yellow fever vaccination coverage was high and satisfactory in Pader district since it was above the desired minimum threshold coverage of 80% according to World Health Organization. Massive emergency vaccination done following an outbreak of Yellow fever achieved high population coverage in Pader district. Active surveillance is necessary for early detection of yellow fever cases.

Link: http://www.technet-21.org/library/main/2528-survey-publication-bagonza-uganda-bmc-2013

Publication: Cotter Epi Infec 2003 (Zimbabwe)

Publication abstract: Neonatal tetanus (NT) elimination, < 1 case per 1,000 live births (LB), was assessed at district level in Zimbabwe using a combined lot quality assurance-cluster sampling survey (LQA-CS). Three of the highest risk districts were selected. NT was considered eliminated if fewer than a specified number of NT deaths (proxy for NT cases) were found in the sample determined using operating characteristic curves and tables. TT2 + vaccine coverage was measured in mothers who gave birth 1-13 months before the survey and women aged 15-49 years. NT was considered as eliminated, TT2+ coverage was 78% (95% CI 71-82%) in women aged 15-49 and 83% (95% CI 76-89%) in mothers. The survey cost 30,000 US dollars excluding costs of consultants. NT incidence was below the elimination threshold (< 1/1,000 LB) in the surveyed districts and probably in all districts. LQA-CS is a practical, relatively cost effective field method which can be applied in an African setting to assess NT elimination status.

Link: http://www.technet-21.org/library/main/2535-survey-publication-cotter-epi-infec-2003-zimbabwe

Publication: Pezzoli Trop Med Int Health 2009 (Boliva)

Publication abstract: OBJECTIVE: To estimate the yellow fever (YF) vaccine coverage for the endemic and non-endemic areas of Bolivia and to determine whether selected districts had acceptable levels of coverage (>70%). METHODS: We conducted two surveys of 600 individuals (25 x 12 clusters) to estimate coverage in the endemic and non-endemic areas. We assessed 11 districts using lot quality assurance sampling (LQAS). The lot (district) sample was 35 individuals with six as decision value (alpha error 6% if true coverage 70%; beta error 6% if true coverage 90%). To increase feasibility, we divided the lots into five clusters of seven individuals; to investigate the effect of clustering, we calculated alpha and beta by conducting simulations where each cluster's true coverage was sampled from a normal distribution with a mean of 70% or 90% and standard deviations of 5% or 10%. RESULTS: Estimated coverage was 84.3% (95% CI: 78.9-89.7) in endemic areas, 86.8% (82.5-91.0) in non-endemic and 86.0% (82.8-89.1) nationally. LQAS showed that four lots had unacceptable coverage levels. In six lots, results were inconsistent with the estimated administrative coverage. The simulations suggested that the effect of clustering the lots is unlikely to have significantly increased the risk of making incorrect accept/reject decisions. CONCLUSIONS: Estimated YF coverage was high. Discrepancies between administrative coverage and LQAS results may be due to incorrect population data. Even allowing for clustering in LQAS, the statistical errors would remain low. Catch-up campaigns are recommended in districts with unacceptable coverage.

Link: http://www.technet-21.org/library/main/2536-survey-publication-pezzoli-trop-med-int-health-2009-boliva

Publication: Jahn Trop Med Int Health 2008 (Malawi)

Publication abstract: OBJECTIVE: To assess factors related to recorded vaccine uptake, which may confound the evaluation of vaccine impact. METHODS: Analysis of documented vaccination histories of children under 5 years and demographic and socio-economic characteristics collected by a demographic surveillance system in Karonga District, Malawi. Associations between deviations from the standard vaccination schedule and characteristics that are likely to be associated with increased mortality were determined by multivariate logistic regression. RESULTS: Approximately 78% of children aged 6-23 months had a vaccination document, declining to <50% by 5 years of age. Living closer to an under-5 clinic, having a better educated father, and both parents being alive were associated with having a vaccination document. For a small percentage of children, vaccination records were incomplete and/or faulty. Vaccination uptake was high overall, but delayed among children living further from the nearest under-5 clinic or from poorer socio-economic backgrounds. Approximately 9% of children had received their last dose of DPT with or after measles vaccine. These children were from relatively less educated parents, and were more likely to have been born outside the health services. CONCLUSIONS: Though overall coverage in this community was high and variation in coverage according to child or parental characteristics small, there was strong evidence of more timely coverage among children from better socio-economic conditions and among those who lived closer to health facilities. These factors are likely to be strong confounders in the association of vaccinations with mortality, and may offer an alternative explanation for the non-specific mortality impact of vaccines described by other studies.

Link: http://www.technet-21.org/library/main/2538-survey-publication-jahn-trop-med-int-health-2008-malawi

Publication: Grais Emerg Themes Epi 2007 (Niger)

Publication abstract: In two-stage cluster surveys, the traditional method used in second-stage sampling (in which the first household in a cluster is selected) is time-consuming and may result in biased estimates of the indicator of interest. Firstly, a random direction from the center of the cluster is selected, usually by spinning a pen. The houses along that direction are then counted out to the boundary of the cluster, and one is then selected at random to be the first household surveyed. This process favors households towards the center of the cluster, but it could easily be improved. During a recent meningitis vaccination coverage survey in Maradi, Niger, we compared this method of first household selection to two alternatives in urban zones: 1) using a superimposed grid on the map of the cluster area and randomly selecting an intersection; and 2) drawing the perimeter of the cluster area using a Global Positioning System (GPS) and randomly selecting one point within the perimeter. Although we only compared a limited number of clusters using each method, we found the sampling grid method to be the fastest and easiest for field survey teams, although it does require a map of the area. Selecting a random GPS point was also found to be a good method, once adequate training can be provided. Spinning the pen and counting households to the boundary was the most complicated and time-consuming. The two methods tested here represent simpler, quicker and potentially more robust alternatives to spinning the pen for cluster surveys in urban areas. However, in rural areas, these alternatives would favor initial household selection from lower density (or even potentially empty) areas. Bearing in mind these limitations, as well as available resources and feasibility, investigators should choose the most appropriate method for their particular survey context.

Link: http://www.technet-21.org/library/main/2532-survey-publication-grais-emerg-themes-epi-2007-niger

Publication: Luman Int J Epi 2007 (Ethiopia)

Publication abstract: BACKGROUND: Measuring vaccination coverage permits evaluation and appropriate targeting of vaccination services. The cluster survey methodology developed by the World Health Organization, known as the 'Expanded Program on Immunization (EPI) methodology', has been used worldwide to assess vaccination coverage; however, the manner in which households are selected has been criticized by survey statisticians as lacking methodological rigor and introducing bias. METHODS: Thirty clusters were selected from an urban (Ambo) and a rural (Yaya-Gulelena D/Libanos) district of Ethiopia; vaccination coverage surveys were conducted using both EPI sampling and systematic random sampling (SystRS) of households. Chi-square tests were used to compare results from the two methodologies; relative feasibility of the sampling methodologies was assessed. RESULTS: Vaccination coverage from a recent measles campaign among children aged 6 months through 14 years was high: 95% in Ambo (both methodologies), 91 and 94% (SystRS and EPI sampling, respectively, P-value = 0.05) in Yaya-Gulelena D/Libanos. Coverage with routine vaccinations among children aged 12-23 months was <20% in both districts; in Ambo, EPI sampling produced consistently higher estimates of routine coverage than SystRS. Differences between the two methods were found in demographic characteristics and recent health histories. Average time required to complete a cluster was 16h for EPI sampling and 17 h for SystRS; total cost was equivalent. Interviewers reported slightly more difficulty conducting SystRS. CONCLUSIONS: Because of the methodological advantages and demonstrated feasibility, SystRS would be preferred to EPI sampling in most situations. Validating results in additional settings is recommended.

Link: http://www.technet-21.org/library/main/2533-survey-publication-luman-ethiopia-int-j-epi-2007

Publication: Milligan Int J Epi 2004 (Gambia)

Publication abstract: BACKGROUND: The Expanded Program for Immunization (EPI) random walk method has been widely used by the World Health Organization and others for rapid cluster sample surveys where an up-to-date household sampling frame is not available. However, it is not a probability sample, does not allow for population movement since the last census, and does not ensure objectivity in household selection or permit call-backs for non-response. Compact segment sampling avoids these problems and has been proposed as a slower but cleaner alternative. METHODS: We conducted two surveys, one using the EPI scheme and one using compact segment sampling, to estimate vaccination coverage in Western Region of The Gambia within 3 months of each other in 2000-2001. RESULTS: Point estimates for vaccination coverage from the two surveys rarely differed by more than 2%. Any differences were more likely to be due to household selection than to population movement. A simple mathematical model showed that even in extreme situations, ignoring population movement since the last census is unlikely to have any appreciable effect. Rates of homogeneity did not differ systematically between the surveys. CONCLUSIONS: In situations where quality of fieldwork can be guaranteed, the EPI random walk method can give accurate and precise results. However, compact segment sampling is generally to be preferred as it ensures objectivity in household selection and permits the estimation of population totals (such as those unvaccinated), which are helpful for planning service provision.

Link: http://www.technet-21.org/library/main/2534-survey-publication-milligan-gambia-int-j-epi-2004

Publication: Gareaballah WHO Bulletin 1989 (Sudan)

Publication abstract: Estimates of measles vaccination coverage in the Sudan vary on average by 23 percentage points, depending on whether or not information supplied by mothers who have lost their children's vaccination cards is included. To determine the accuracy of mother's reports, we collected data during four large coverage surveys in which illiterate mothers with vaccination cards were asked about their children's vaccination status and their answers were compared with the information given on the cards. Mothers' replies were very accurate. For example, for measles vaccination, the data supplied were both sensitive (87%) and specific (79%) compared with those on the vaccination cards. For both DPT and measles vaccination, accurate estimates of the true coverage rates could therefore be obtained by relying solely on mothers' reports. Within +/- 1 month, 78% of the women knew the age at which their children had received their first dose of poliovaccine. Ignoring mothers' reports of their children's vaccination status could therefore result in serious underestimates of the true vaccination coverage. A simple method of dealing with the problem posed by lost vaccination cards during coverage surveys is also suggested.

Link: http://www.technet-21.org/library/main/2539-survey-publication-gareaballah-who-bulletin-1989-sudan

Publication: Kim PloS One 2012 (Niger)

Publication abstract: MenAfriVac is a new conjugate vaccine against Neisseria meningitidis serogroup A developed for the African "meningitis belt". In Niger, the first two phases of the MenAfriVac introduction campaign were conducted targeting 3,135,942 individuals aged 1 to 29 years in the regions of Tillabéri, Niamey, and Dosso, in September and December 2010. We evaluated the campaign and determined which sub-populations or areas had low levels of vaccination coverage in the regions of Tillabéri and Niamey. After Phase I, conducted in the Filingué district, we estimated coverage using a 30×15 cluster-sampling survey and nested lot quality assurance (LQA) analysis in the clustered samples to identify which subpopulations (defined by age 1-14/15-29 and sex) had unacceptable vaccination coverage (<70%). After Phase II, we used Clustered Lot Quality Assurance Sampling (CLQAS) to assess if any of eight districts in Niamey and Tillabéri had unacceptable vaccination coverage (<75%) and estimated overall coverage. Estimated vaccination coverage was 77.4% (95%CI: 84.6-70.2) as documented by vaccination cards and 85.5% (95% CI: 79.7-91.2) considering verbal history of vaccination for Phase I; 81.5% (95%CI: 86.1-77.0) by card and 93.4% (95% CI: 91.0-95.9) by verbal history for Phase II. Based on vaccination cards, in Filingué, we identified both the male and female adult (age 15-29) subpopulations as not reaching 70% coverage; and we identified three (one in Tillabéri and two in Niamey) out of eight districts as not reaching 75% coverage confirmed by card. Combined use of LQA and cluster sampling was useful to estimate vaccination coverage and to identify pockets with unacceptable levels of coverage (adult population and three districts). Although overall vaccination coverage was satisfactory, we recommend continuing vaccination in the areas or sub-populations with low coverage and reinforcing the social mobilization of the adult population.

Link: http://www.technet-21.org/library/main/2537-survey-kim-plos-one-2012-niger

Publication: Barker Stat Med 2005 (United States)

Publication abstract: The National Immunization Survey (NIS) provides state-level estimates of preschool immunization coverage. These coverages are frequently presented in ranked lists, and ranks are frequently over-interpreted. In this paper, we highlight the difficulty in interpreting ranked point estimates. To demonstrate the uncertainty of ranks, parametric bootstrap methods were used to derive 90 per cent confidence intervals for ranks of state vaccination coverage levels among preschool children. We graphically compared states to a reference state. If NIS data are used to rank states, one should consider presenting confidence intervals for rank and the results of comparisons of one state with another graphically.

Link: http://www.technet-21.org/library/main/2548-survey-publication-barker-stat-med-2005-united-states

Publication: Luman BMC Public Health 2008 (Northern Mariana Islands)

Publication abstract: Background Lack of methodological rigor can cause survey error, leading to biased results and suboptimal public health response. This study focused on the potential impact of 3 methodological "shortcuts" pertaining to field surveys: relying on a single source for critical data, failing to repeatedly visit households to improve response rates, and excluding remote areas. Methods In a vaccination coverage survey of young children conducted in the Commonwealth of the Northern Mariana Islands in July 2005, 3 sources of vaccination information were used, multiple follow-up visits were made, and all inhabited areas were included in the sampling frame. Results are calculated with and without these strategies. Results Most children had at least 2 sources of data; vaccination coverage estimated from any single source was substantially lower than from all sources combined. Eligibility was ascertained for 79% of households after the initial visit and for 94% of households after follow-up visits; vaccination coverage rates were similar with and without follow-up. Coverage among children on remote islands differed substantially from that of their counterparts on the main island indicating a programmatic need for locality-specific information; excluding remote islands from the survey would have had little effect on overall estimates due to small populations and divergent results. Conclusion Strategies to reduce sources of survey error should be maximized in public health surveys. The impact of the 3 strategies illustrated here will vary depending on the primary outcomes of interest and local situations. Survey limitations such as potential for error should be well-documented, and the likely direction and magnitude of bias should be considered.

Link: http://www.technet-21.org/library/main/2547-survey-publication-luman-bmc-2008-northern-mariana-islands

Publication: Luman Vaccine 2008 (Northern Mariana Islands)

Publication abstract: Public health programs rely on household-survey estimates of vaccination coverage as a basis of programmatic and policy decisions; however, the validity of estimates derived from household-retained vaccination cards and parental recall has not been thoroughly evaluated. Using data from a vaccination coverage survey conducted in the Western Pacific's Northern Mariana Islands, we compared results from household data sources to medical record sources for the same children. We calculated the percentage of children aged 1, 2, and 6 years who received all vaccines recommended by age 12 months, 24 months, and for school entry, respectively. Coverage estimates based on vaccination cards ranged from 14% to 30% in the three age groups compared to 78-91% for the same children based on medical records. When cards were supplemented by parental recall, estimates were 51-53%. Concordance, sensitivity, specificity, positive and negative predictive values, and kappa statistics generally indicated poor agreement between household and medical record sources. Household-retained vaccination cards and parental recall were insufficient sources of information for estimating vaccination coverage in this population. This study emphasizes the importance of identifying reliable sources of vaccination history information and reinforces the need for awareness of the potential limitations of vaccination coverage estimated from surveys that rely on household-retained cards and/or parental recall.

Link: http://www.technet-21.org/library/main/2546-survey-publication-luman-vaccine-2008-northern-mariana-islands

Publication: Shimabukuro J Public Health Manag Pract 2007 (United States)

Publication abstract: OBJECTIVE: To evaluate potential age-appropriate up-to-date (UTD) vaccination coverage achievable in preschool children if missing vaccinations were administered during a well-child visit at 18 months of age. METHODS: Data from the 2004 National Immunization Survey were used in a series of simulations analyzing UTD coverage of the 4:3:1:3:3:1 (diphtheria, tetanus, pertussis/poliovirus/measles-containing vaccine/Haemophilus influenzae type b/hepatitis B/varicella) and 4:3:1:3:3:1 (+) pneumococcal conjugate vaccine (PCV) series. In the models, children not already up-to-date received up to four missing vaccinations during a simulated routine 18-month-old well-child visit. RESULTS: For the 4:3:1:3:3:1 series, UTD coverage increased from baseline 61 percent (95% confidence interval [CI] = 60-62) to simulated 87 percent (95% CI = 86-88). Among the baseline non-UTD children, 69 percent became up-to-date by simulation with the single visit, of which 44 percent required only one vaccination. For the 4:3:1:3:3:1 (+) PCV series, UTD coverage increased from baseline 38 percent (95% CI = 37-40) to simulated 74 percent (95% CI = 73-75). Among the baseline non-UTD children, 59 percent became up-to-date by simulation with the single visit, of which 47 percent required only one vaccination. UTD coverage increased substantially for all racial/ethnic groups and in all states. CONCLUSIONS: Taking full advantage of the recommended 18-month-old well-child visit to administer missing vaccines would be a strategically timed opportunity to achieve high age-appropriate UTD coverage in preschool children.

Link: http://www.technet-21.org/library/main/2545-survey-publication-shimabukuro-j-public-health-manag-pract-2007-united-states

Publication: Luman Am J Prev Med 2001 (United States)

Publication abstract: BACKGROUND: This study characterizes the healthcare visits at which children receive vaccinations, including the number of these visits and the number of vaccinations that are administered. METHODS: The 1999 National Immunization Survey (NIS) is a nationally representative sample of children aged 19 to 35 months, verified by provider records, that is conducted to obtain estimates of vaccination coverage rates. We describe the number of healthcare visits in which one or more vaccinations were given, the number of vaccinations given at these visits, and the number of visits and vaccinations needed for an underimmunized child to complete the recommended vaccination series. RESULTS: Of the children who did not receive all doses of the recommended vaccinations (4:3:1:3:3 vaccination series), three fourths had four or more immunization visits. Vaccination coverage increased as the number of visits increased, and children who had completed the series were more likely to receive multiple vaccinations than those who had not. Most children (70.7%) received a maximum of four vaccinations in any immunization visit. The majority of children (73.5%) who had not completed the 4:3:1:3:3 vaccination series needed only a single visit to complete the series. The majority (61.7%) of children who needed only one visit also needed only one additional vaccination. CONCLUSIONS: While estimated national coverage for all recommended vaccinations is considerably below the Healthy People 2000 and Healthy People 2010 goal of 90%, achieving this goal is in essence just one visit away. If all children who needed one more visit were to receive that final visit, the national coverage among children 19 to 35 months for all recommended vaccinations would be 93%.

Link: http://www.technet-21.org/library/main/2544-survey-publication-luman-am-j-prev-med-2001-united-states

Publication: Bennett Int J Epi 1994 (Uganda)

Publication abstract: BACKGROUND: Cluster sample surveys of health and nutrition in rural areas of developing countries frequently utilize the EPI (Expanded Programme on Immunization) method of selecting households where complete enumeration and systematic or simple random sampling (SRS) is considered impractical. The first household is selected by choosing a random direction from the centre of the community, counting the houses along that route, and picking one at random. Subsequent households are chosen by visiting that house which is nearest to the preceding one. METHODS: Using a computer, and data from a survey of all children in 30 villages in Uganda, we simulated the selection of samples of size 7, 15 and 30 children from each village using SRS, the EPI method, and four different modifications of the EPI method. RESULTS: The choice of sampling scheme for households had very little effect on the precision or bias of estimates of prevalence of malnutrition, or of recent morbidity, with EPI performing as well as SRS. However, the EPI scheme was inefficient and showed bias for variables relating to child care and for socioeconomic variables. Two of the modified EPI schemes (taking every fifth house and taking separate EPI samples in each quarter of the community) performed in general much better than EPI and almost as well as SRS. CONCLUSIONS: These results suggest that the unmodified EPI household sampling scheme may be adequate for rapid appraisal of morbidity prevalence or nutritional status of communities, but that it may not be appropriate for surveys which cover a wider range of topics such as health care, or seek to examine the association of health or nutrition with explanatory factors such as education and socioeconomic status. Other factors such as cost and the ability to monitor interviewers' performance should also be taken into account.

Link: http://www.technet-21.org/library/main/2542-survey-publication-bennett-int-j-epi-1994-uganda

Publication: Valadez Am J Pub Health 1992 (Costa Rica)

Publication abstract: In the absence of vaccination card data, Expanded Program on Immunization (EPI) managers sometimes ask mothers for their children's vaccination histories. The magnitude of maternal recall error and its potential impact on public health policy has not been investigated. In this study of 1171 Costa Rican mothers, we compare mothers' recall with vaccination card data for their children younger than 3 years. Analyses of vaccination coverage distributions constructed with recall and vaccination-card data show that recall can be used to estimate population coverage. Although the two data sources are correlated (r = .71), the magnitude of their difference can affect the identification of the vaccination status of an individual child. Maternal recall error was greater than two doses 14% of the time. This error is negatively correlated with the number of doses recorded on the vaccination card (r = -.61) and is weakly correlated with the child's age (r = -.35). Mothers tended to remember accurately the vaccination status of children younger than 6 months, but with older children, the larger the number of doses actually received, the more the mother underestimated the number of doses. No other variables explained recall error. Therefore, reliance on maternal recall could lead to revaccinating children who are already protected, leaving a risk those most vulnerable to vaccine-preventable diseases.

Link: http://www.technet-21.org/library/main/2541-survey-publication-valadez-am-j-pub-health-1992-costa-rica

Publication: Langsten Soc Sci Med 1998 (Egypt)

Publication abstract: Estimates of immunization coverage in developing countries are typically made on a "card plus history" basis, combining information obtained from vaccination cards with information from mothers' reports, for children for whom such cards are not available. A recent survey in rural lower Egypt was able to test the accuracy of mothers' reports for a subset of children whose cards were not seen at round 1 of the survey but were seen a year later at round 3. Comparisons of the unsubstantiated reports at round 1 with information recorded from cards seen at round 3 indicate that mothers' reports are of very high quality; mothers' reports at round 1 were confirmed by card data at round 3 for between 83 and 93%, depending on vaccine, of children aged 12-23 months, and for 88 to 98% of children aged 24-35 months. Mothers of children who had not been vaccinated were more likely to give consistent responses than were mothers of vaccinated children. Thus, these "card plus history" estimates slightly understate true coverage levels. Most of the inconsistencies between round 1 and round 3 data apparently arose from interviewer or data processing error rather than from misreporting by mothers.

Link: http://www.technet-21.org/library/main/2540-survey-publication-langsten-soc-sci-med-1998-egypt

Publication: Bratton, Wallace and Danovaro; 2005 & 2011 Honduras DHS analysis of Vaccination Timeliness, Co-administration and Factors (Honduras)

This is a report of the secondary analysis of the 2011-2012 Honduras DHS. It describes analysis of timeliness, simultaneity of vaccination, and missed opportunities for vaccination.

Link: http://www.technet-21.org/library/main/3430

Book: Brown 2002 (Various countries)

Publication abstract: Background This study aims to assess of the quality of child immunization coverage estimates obtained in 101 national population-based surveys in mostly developing countries. Methods The Demographic and Health Surveys (DHS) and UNICEF's Multiple Indicator Cluster Sample (MICS) surveys provide national immunization coverage estimates for children aged 12-23 months once every three to five years in many developing countries. The data are collected by interview from a nationally representative sample of households. 83 DHS and 18 MICS surveys were included. Findings 85% of mothers reported that they had ever received a health card for their child. 81% still had the card at the time of the interview, and nearly two-thirds of these presented the card to the interviewer. Cards were therefore observed for 55% of children overall. Rural and less educated mothers were less likely to report receiving health cards. Recall of additional immunizations by mothers that presented a card ranged from 1 to 3%. Recall of immunizations by mothers who reported never receiving a card ranged from 9 to 32%. Coverage among those who did not show a card rarely exceeded coverage among those who did, and there was good correlation between DPT and OPV doses received according to health card and recall data. Conclusion Though maternal recall data are known to be less accurate than health card data, we found no major systematic weaknesses in recall and believe that inclusion of recall data yields more accurate coverage estimates.

Link: http://www.technet-21.org/library/main/2543-survey-book-brown-2002-various-countries

Rainey, 2011, Reasons related to non-vaccination and under-vaccination of children in low and middle income countries: Findings from a systematic review of the published literature, 1999-2009

Despite increases in routine vaccination coverage during the past three decades, the percent of children completing the recommended vaccination schedule remains below expected targets in many low and middle income countries. In 2008, the World Health Organization Strategic Advisory Group of Experts on Immunization requested more information on the reasons that children were under-vaccinated (receiving at least one but not all recommended vaccinations) or not vaccinated in order to develop effective strategies and interventions to reach these children. METHODS: A systematic review of the peer-reviewed literature published from 1999 to 2009 was conducted to aggregate information on reasons and factors related to the under-vaccination and non-vaccination of children. A standardized form was used to abstract information from relevant articles identified from eight different medical, behavioural and social science literature databases. FINDINGS: Among 202 relevant articles, we abstracted 838 reasons associated with under-vaccination; 379 (45%) were related to immunization systems, 220 (26%) to family characteristics, 181 (22%) to parental attitudes and knowledge, and 58 (7%) to limitations in immunization-related communication and information. Of the 19 reasons abstracted from 11 identified articles describing the non-vaccinated child, 6 (32%) were related to immunization systems, 8 (42%) to parental attitudes and knowledge, 4 (21%) to family characteristics, and 1 (5%) to communication and information. CONCLUSIONS: Multiple reasons for under-vaccination and non-vaccination were identified, indicating that a multi-faceted approach is needed to reach under-vaccinated and unvaccinated children. Immunization system issues can be addressed through improving outreach services, vaccine supply, and health worker training; however, under-vaccination and non-vaccination linked to parental attitudes and knowledge are more difficult to address and likely require local interventions.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3530

 

Methodological publications

This collection contains several technical and methodological publications on vaccination coverage surveys. The collection also can be found in the TechNet Resource Library:

http://www.technet-21.org/library/main/collection?cid=56e41182ec3f3

Publication: Dean J survey stats & methods 2015

Publication abstract: In survey settings, a variety of methods are available for constructing confidence intervals for proportions. These methods include the standard Wald method, a class of modified methods that replace the sample size with the survey effective sample size (Wilson, Clopper-Pearson, Jeffreys, and Agresti-Coull), and transformed methods (Logit and Arcsine). We describe these seven methods, two of which have not been previously evaluated in the literature (the modified Jeffreys and Agresti-Coull intervals). For each method, we describe two formulations, one with and one without adjustment for the design degrees of freedom. We suggest a definition of adjusted effective sample size that induces equivalency between different confidence interval expressions. We also expand on an existing framework for truncation that can be used when data appear to be more efficient than a simple random sample or when data have standard error equal to zero. We compare these methods using a simulation study modeled after the 30 × 7 design for immunization surveys. Our results confirmed the importance of adjusting for the design degrees of freedom. As expected, the Wald interval performed very poorly, frequently failing to achieve the nominal coverage level. For similar reasons, we do not recommend the use of the Arcsine interval. When the intracluster correlation coefficient is high and the prevalence, p, is less than 0.10 or greater than 0.90, the Agresti-Coull and Clopper-Pearson intervals perform best. In other settings, the Clopper-Pearson interval is unnecessarily wide. In general, the Logit, Wilson, Jeffreys, and Agresti-Coull intervals perform well, although the Logit interval may be intractable when the standard error is equal to zero.

Link: http://www.technet-21.org/library/main/2420-survey-publication-dean-j-survey-stats-methods-2015

Publication: Barker Am J Epi 2002

Publication abstract: Eliminating health disparities in vaccination coverage among various groups is a cornerstone of public health policy. However, the statistical tests traditionally used cannot prove that a state of no difference between groups exists. Instead of asking, "Has a disparity--or difference--in immunization coverage among population groups been eliminated ?," one can ask, "Has practical equivalence been achieved?" A method called equivalence testing can show that the difference between groups is smaller than a tolerably small amount. This paper demonstrates the method and introduces public health considerations that have an impact on defining tolerable levels of difference. Using data from the 2000 National Immunization Survey, the authors tested for statistically significant differences in rates of vaccination coverage between Whites and members of other racial/ethnic groups and for equivalencies among Whites and these same groups. For some minority groups and some vaccines, coverage was statistically significantly lower than was seen among Whites; however, for some of these groups and vaccines, equivalence testing revealed practical equivalence. To use equivalence testing to assess whether a disparity remains a threat to public health, researchers must understand when to use the method, how to establish assumptions about tolerably small differences, and how to interpret the test results.

Link: http://www.technet-21.org/library/main/2421-survey-barker-am-j-epi-2002

Publication: Minetti Emerg Themes Epi 2012

Publication abstract: BACKGROUND: Estimation of vaccination coverage at the local level is essential to identify communities that may require additional support. Cluster surveys can be used in resource-poor settings, when population figures are inaccurate. To be feasible, cluster samples need to be small, without losing robustness of results. The clustered LQAS (CLQAS) approach has been proposed as an alternative, as smaller sample sizes are required. METHODS: We explored (i) the efficiency of cluster surveys of decreasing sample size through bootstrapping analysis and (ii) the performance of CLQAS under three alternative sampling plans to classify local VC, using data from a survey carried out in Mali after mass vaccination against meningococcal meningitis group A. RESULTS: VC estimates provided by a 10 × 15 cluster survey design were reasonably robust. We used them to classify health areas in three categories and guide mop-up activities: i) health areas not requiring supplemental activities; ii) health areas requiring additional vaccination; iii) health areas requiring further evaluation. As sample size decreased (from 10 × 15 to 10 × 3), standard error of VC and ICC estimates were increasingly unstable. Results of CLQAS simulations were not accurate for most health areas, with an overall risk of misclassification greater than 0.25 in one health area out of three. It was greater than 0.50 in one health area out of two under two of the three sampling plans. CONCLUSIONS: Small sample cluster surveys (10 × 15) are acceptably robust for classification of VC at local level. We do not recommend the CLQAS method as currently formulated for evaluating vaccination programmes.

Link: http://www.technet-21.org/library/main/2422-survey-publication-minetti-emerg-themes-epi-2012

Publication: Murray Lancet 2003

Publication abstract: BACKGROUND: Monitoring and assessment of coverage rates in national health programmes is becoming increasingly important. We aimed to assess the accuracy of officially reported coverage rates of vaccination with diphtheria-tetanus-pertussis vaccine (DTP3), which is commonly used to monitor child health interventions. METHODS: We compared officially reported national data for DTP3 coverage with those from the household Demographic and Health Surveys (DHS) in 45 countries between 1990 and 2000. We adjusted survey data to reflect the number of valid vaccinations (ie, those administered in accordance with the schedule recommended by WHO) using a probit model with sample selection. The model predicted the probability of valid vaccinations for children, including those without documented vaccinations, after correcting for bias from differences between the children with and without documented information on vaccination. We then assessed the extent of survey bias and differences between officially reported data and those from DHS estimates. FINDINGS: Our results suggest that officially reported DTP3 coverage is higher than that reported from household surveys. This size of the difference increases with the rate of reported coverage of DTP3. Results of time-trend analysis show that changes in reported coverage are not correlated with changes reported from household surveys. INTERPRETATION: Although reported data might be the most widely available information for assessment of vaccination coverage, their validity for measuring changes in coverage over time is highly questionable. Household surveys can be used to validate data collected by service providers. Strategies for measurement of the coverage of all health interventions should be grounded in careful assessments of the validity of data derived from various sources.

Link: http://www.technet-21.org/library/main/2423-survey-publication-murray-lancet-2003

Publication: Burton Lancet 2009

Publication abstract: WHO and UNICEF welcome Stephen Lim and colleagues' contribution to the more accurate measurement of immunisation coverage, and fully concur with the recommendations to improve routine immunisation monitoring systems and the need to validate results of these systems periodically with surveys. We feel that such efforts are necessary not only to document progress towards international goals and to meet donor reporting requirements but, even more importantly, to improve immunisation service delivery at local and national levels.

Link: http://www.technet-21.org/library/main/2424-survey-publication-burton-methods-lancet-2009

Publication: Hancioglu PLoS Med 2013

Publication abstract: Household surveys are the primary data source of coverage indicators for children and women for most developing countries. Most of this information is generated by two global household survey programmes—the USAID-supported Demographic and Health Surveys (DHS) and the UNICEF-supported Multiple Indicator Cluster Surveys (MICS). In this review, we provide an overview of these two programmes, which cover a wide range of child and maternal health topics and provide estimates of many Millennium Development Goal indicators, as well as estimates of the indicators for the Countdown to 2015 initiative and the Commission on Information and Accountability for Women's and Children's Health. MICS and DHS collaborate closely and work through interagency processes to ensure that survey tools are harmonized and comparable as far as possible, but we highlight differences between DHS and MICS in the population covered and the reference periods used to measure coverage. These differences need to be considered when comparing estimates of reproductive, maternal, newborn, and child health indicators across countries and over time and we discuss the implications of these differences for coverage measurement. Finally, we discuss the need for survey planners and consumers of survey results to understand the strengths, limitations, and constraints of coverage measurements generated through household surveys, and address some technical issues surrounding sampling and quality control. We conclude that, although much effort has been made to improve coverage measurement in household surveys, continuing efforts are needed, including further research to improve and refine survey methods and analytical techniques.

Link: http://www.technet-21.org/library/main/2425-survey-publication-hancioglu-plos-med-2013

Publication: Henderson WHO Bulletin 1982

Publication abstract: A simplified cluster sampling method, involving the random selection of 210 children in 30 clusters of 7 each, has been used by the Expanded Programme on Immunization to estimate immunization coverage. This paper analysis the performance of the method in 60 actual surveys and 1500 computer simulated surveys. Although the method gives a proportion of results with confidence limits exceeding the desired maximum of 10 absolute percentage points, it is concluded that is performs satisfactorily.

Link: http://www.technet-21.org/library/main/2426-survey-publication-henderson-who-bulletin-1982

Publication: Luman Int J Epi 2007

Publication abstract: BACKGROUND: Measuring vaccination coverage permits evaluation and appropriate targeting of vaccination services. The cluster survey methodology developed by the World Health Organization, known as the 'Expanded Program on Immunization (EPI) methodology', has been used worldwide to assess vaccination coverage; however, the manner in which households are selected has been criticized by survey statisticians as lacking methodological rigor and introducing bias. METHODS: Thirty clusters were selected from an urban (Ambo) and a rural (Yaya-Gulelena D/Libanos) district of Ethiopia; vaccination coverage surveys were conducted using both EPI sampling and systematic random sampling (SystRS) of households. Chi-square tests were used to compare results from the two methodologies; relative feasibility of the sampling methodologies was assessed. RESULTS: Vaccination coverage from a recent measles campaign among children aged 6 months through 14 years was high: 95% in Ambo (both methodologies), 91 and 94% (SystRS and EPI sampling, respectively, P-value = 0.05) in Yaya-Gulelena D/Libanos. Coverage with routine vaccinations among children aged 12-23 months was <20% in both districts; in Ambo, EPI sampling produced consistently higher estimates of routine coverage than SystRS. Differences between the two methods were found in demographic characteristics and recent health histories. Average time required to complete a cluster was 16h for EPI sampling and 17 h for SystRS; total cost was equivalent. Interviewers reported slightly more difficulty conducting SystRS. CONCLUSIONS: Because of the methodological advantages and demonstrated feasibility, SystRS would be preferred to EPI sampling in most situations. Validating results in additional settings is recommended.

Link: http://www.technet-21.org/library/main/2428-survey-publication-luman-int-j-epi-2007

Publication: Grais Emerg Themes Epi 2007

Publication abstract: In two-stage cluster surveys, the traditional method used in second-stage sampling (in which the first household in a cluster is selected) is time-consuming and may result in biased estimates of the indicator of interest. Firstly, a random direction from the center of the cluster is selected, usually by spinning a pen. The houses along that direction are then counted out to the boundary of the cluster, and one is then selected at random to be the first household surveyed. This process favors households towards the center of the cluster, but it could easily be improved. During a recent meningitis vaccination coverage survey in Maradi, Niger, we compared this method of first household selection to two alternatives in urban zones: 1) using a superimposed grid on the map of the cluster area and randomly selecting an intersection; and 2) drawing the perimeter of the cluster area using a Global Positioning System (GPS) and randomly selecting one point within the perimeter. Although we only compared a limited number of clusters using each method, we found the sampling grid method to be the fastest and easiest for field survey teams, although it does require a map of the area. Selecting a random GPS point was also found to be a good method, once adequate training can be provided. Spinning the pen and counting households to the boundary was the most complicated and time-consuming. The two methods tested here represent simpler, quicker and potentially more robust alternatives to spinning the pen for cluster surveys in urban areas. However, in rural areas, these alternatives would favor initial household selection from lower density (or even potentially empty) areas. Bearing in mind these limitations, as well as available resources and feasibility, investigators should choose the most appropriate method for their particular survey context.

Link: http://www.technet-21.org/library/main/2427-survey-publication-grais-emerg-themes-epi-2007

Publication: Valadez Am J Pub Health 1992

Publication abstract: In the absence of vaccination card data, Expanded Program on Immunization (EPI) managers sometimes ask mothers for their children's vaccination histories. The magnitude of maternal recall error and its potential impact on public health policy has not been investigated. In this study of 1171 Costa Rican mothers, we compare mothers' recall with vaccination card data for their children younger than 3 years. Analyses of vaccination coverage distributions constructed with recall and vaccination-card data show that recall can be used to estimate population coverage. Although the two data sources are correlated (r = .71), the magnitude of their difference can affect the identification of the vaccination status of an individual child. Maternal recall error was greater than two doses 14% of the time. This error is negatively correlated with the number of doses recorded on the vaccination card (r = -.61) and is weakly correlated with the child's age (r = -.35). Mothers tended to remember accurately the vaccination status of children younger than 6 months, but with older children, the larger the number of doses actually received, the more the mother underestimated the number of doses. No other variables explained recall error. Therefore, reliance on maternal recall could lead to revaccinating children who are already protected, leaving a risk those most vulnerable to vaccine-preventable diseases.

Link: http://www.technet-21.org/library/main/2432-survey-publication-valadez-am-j-pub-health-1992

Publication: Langsten Soc Sci Med 1998

Publication abstract: Estimates of immunization coverage in developing countries are typically made on a "card plus history" basis, combining information obtained from vaccination cards with information from mothers' reports, for children for whom such cards are not available. A recent survey in rural lower Egypt was able to test the accuracy of mothers' reports for a subset of children whose cards were not seen at round 1 of the survey but were seen a year later at round 3. Comparisons of the unsubstantiated reports at round 1 with information recorded from cards seen at round 3 indicate that mothers' reports are of very high quality; mothers' reports at round 1 were confirmed by card data at round 3 for between 83 and 93%, depending on vaccine, of children aged 12-23 months, and for 88 to 98% of children aged 24-35 months. Mothers of children who had not been vaccinated were more likely to give consistent responses than were mothers of vaccinated children. Thus, these "card plus history" estimates slightly understate true coverage levels. Most of the inconsistencies between round 1 and round 3 data apparently arose from interviewer or data processing error rather than from misreporting by mothers.

Link: http://www.technet-21.org/library/main/2431-survey-publication-langsten-soc-sci-med-1998

Publication: Gareaballah WHO Bulletin 1989

Publication abstract: Estimates of measles vaccination coverage in the Sudan vary on average by 23 percentage points, depending on whether or not information supplied by mothers who have lost their children's vaccination cards is included. To determine the accuracy of mother's reports, we collected data during four large coverage surveys in which illiterate mothers with vaccination cards were asked about their children's vaccination status and their answers were compared with the information given on the cards. Mothers' replies were very accurate. For example, for measles vaccination, the data supplied were both sensitive (87%) and specific (79%) compared with those on the vaccination cards. For both DPT and measles vaccination, accurate estimates of the true coverage rates could therefore be obtained by relying solely on mothers' reports. Within +/- 1 month, 78% of the women knew the age at which their children had received their first dose of polio vaccine. Ignoring mothers' reports of their children's vaccination status could therefore result in serious underestimates of the true vaccination coverage. A simple method of dealing with the problem posed by lost vaccination cards during coverage surveys is also suggested.

Link: http://www.technet-21.org/library/main/2430-survey-publication-gareaballah-who-bulletin-1989

Publication: Turner Int J Epi 1996

Publication abstract: BACKGROUND: Although the Expanded Programme on Immunization (EPI) cluster survey methodology has been successfully used for assessing levels of immunization programme coverage in developing country settings, certain features of the methodology, as it is usually carried out, make it less-than-optimal choice for large, national surveys and/or surveys with multiple measurement objectives. What is needed is a 'middle ground' between rigorous cluster sampling methods, which are seen as unfeasible for routine use in many developing country settings, and the EPI cluster survey approach. METHODS: This article suggests some fairly straightforward modifications to the basic EPI cluster survey design that put it on a solid probability footing and render it easily adaptable to differing and/or multiple measurement objectives, without incurring prohibitive costs or adding appreciably to the complexity of survey operations. The proposed modifications concern primarily the manner in which households are chosen at the second stage of sample selection. CONCLUSIONS: Because the modified sampling strategy maintains the scientific rigor of conventional cluster sampling methods while retaining many of the desirable features of the EPI survey methodology, the methodology is likely to be a preferred 'middle ground' survey design, relevant for many applications, particularly surveys designed to monitor multiple health indicators over time. The fieldwork burden in the modified design is only marginally higher than in EPI cluster surveys, and considerably lower than in conventional cluster surveys.

Link: http://www.technet-21.org/library/main/2429-survey-publication-turner-int-j-epi-1996

Publication: Lemeshow Int J Epi 1985

Publication abstract: A Monte Carlo simulation study was designed to evaluate the sample survey technique currently used by the Expanded Programme on Immunization (EPI) of the World Health Organization. Of particular interest was how the EPI strategy compared to a more traditional sampling strategy with respect to bias and variability of estimates. It was also of interest to investigate whether the estimates of population vaccination coverage were accurate to within 10 percentage points of the actual levels. It was found that within particular clusters, the EPI method was particularly sensitive to pocketing of vaccinated individuals, but the more traditional method gave more accurate and less variable results under a variety of conditions. However, the stated goal of the EPI, of being able to produce population estimates accurate to within 10 percentage points of the true levels in the population, was satisfied in the artificially created populations studied.

Link: http://www.technet-21.org/library/main/2435-survey-publication-lemeshow-int-j-epi-1985

Publication: Bennett Int J Epi 1994

Publication abstract: BACKGROUND: Cluster sample surveys of health and nutrition in rural areas of developing countries frequently utilize the EPI (Expanded Programme on Immunization) method of selecting households where complete enumeration and systematic or simple random sampling (SRS) is considered impractical. The first household is selected by choosing a random direction from the centre of the community, counting the houses along that route, and picking one at random. Subsequent households are chosen by visiting that house which is nearest to the preceding one. METHODS: Using a computer, and data from a survey of all children in 30 villages in Uganda, we simulated the selection of samples of size 7, 15 and 30 children from each village using SRS, the EPI method, and four different modifications of the EPI method. RESULTS: The choice of sampling scheme for households had very little effect on the precision or bias of estimates of prevalence of malnutrition, or of recent morbidity, with EPI performing as well as SRS. However, the EPI scheme was inefficient and showed bias for variables relating to child care and for socioeconomic variables. Two of the modified EPI schemes (taking every fifth house and taking separate EPI samples in each quarter of the community) performed in general much better than EPI and almost as well as SRS. CONCLUSIONS: These results suggest that the unmodified EPI household sampling scheme may be adequate for rapid appraisal of morbidity prevalence or nutritional status of communities, but that it may not be appropriate for surveys which cover a wider range of topics such as health care, or seek to examine the association of health or nutrition with explanatory factors such as education and socioeconomic status. Other factors such as cost and the ability to monitor interviewers' performance should also be taken into account.

Link: http://www.technet-21.org/library/main/2434-survey-publication-bennett-int-j-epi-1994

Publication: Eisele PLoS Med 2013

Publication abstract: Nationally representative household surveys are increasingly relied upon to measure maternal, newborn, and child health (MNCH) intervention coverage at the population level in low- and middle-income countries. Surveys are the best tool we have for this purpose and are central to national and global decision making. However, all survey point estimates have a certain level of error (total survey error) comprising sampling and non-sampling error, both of which must be considered when interpreting survey results for decision making. In this review, we discuss the importance of considering these errors when interpreting MNCH intervention coverage estimates derived from household surveys, using relevant examples from national surveys to provide context. Sampling error is usually thought of as the precision of a point estimate and is represented by 95% confidence intervals, which are measurable. Confidence intervals can inform judgments about whether estimated parameters are likely to be different from the real value of a parameter. We recommend, therefore, that confidence intervals for key coverage indicators should always be provided in survey reports. By contrast, the direction and magnitude of non-sampling error is almost always unmeasurable, and therefore unknown. Information error and bias are the most common sources of non-sampling error in household survey estimates and we recommend that they should always be carefully considered when interpreting MNCH intervention coverage based on survey data. Overall, we recommend that future research on measuring MNCH intervention coverage should focus on refining and improving survey-based coverage estimates to develop a better understanding of how results should be interpreted and used.

Link: http://www.technet-21.org/library/main/2433-survey-publication-eisele-plos-med-2013

Publication: Luman BMC Public Health 2008

Publication abstract: Background Lack of methodological rigor can cause survey error, leading to biased results and suboptimal public health response. This study focused on the potential impact of 3 methodological "shortcuts" pertaining to field surveys: relying on a single source for critical data, failing to repeatedly visit households to improve response rates, and excluding remote areas. Methods In a vaccination coverage survey of young children conducted in the Commonwealth of the Northern Mariana Islands in July 2005, 3 sources of vaccination information were used, multiple follow-up visits were made, and all inhabited areas were included in the sampling frame. Results are calculated with and without these strategies. Results Most children had at least 2 sources of data; vaccination coverage estimated from any single source was substantially lower than from all sources combined. Eligibility was ascertained for 79% of households after the initial visit and for 94% of households after follow-up visits; vaccination coverage rates were similar with and without follow-up. Coverage among children on remote islands differed substantially from that of their counterparts on the main island indicating a programmatic need for locality-specific information; excluding remote islands from the survey would have had little effect on overall estimates due to small populations and divergent results. Conclusion Strategies to reduce sources of survey error should be maximized in public health surveys. The impact of the 3 strategies illustrated here will vary depending on the primary outcomes of interest and local situations. Survey limitations such as potential for error should be well-documented, and the likely direction and magnitude of bias should be considered.

Link: http://www.technet-21.org/library/main/2440-survey-publication-luman-bmc-2008

Publication: Miles Vaccine 2013

Publication abstract: Immunization programs frequently rely on household vaccination cards, parental recall, or both to calculate vaccination coverage. This information is used at both the global and national level for planning and allocating performance-based funds. However, the validity of household-derived coverage sources has not yet been widely assessed or discussed. To advance knowledge on the validity of different sources of immunization coverage, we undertook a global review of literature. We assessed concordance, sensitivity, specificity, positive and negative predictive value, and coverage percentage point difference when subtracting household vaccination source from a medical provider source. Median coverage difference per paper ranged from -61 to +1 percentage points between card versus provider sources and -58 to +45 percentage points between recall versus provider source. When card and recall sources were combined, median coverage difference ranged from -40 to +56 percentage points. Overall, concordance, sensitivity, specificity, positive and negative predictive value showed poor agreement, providing evidence that household vaccination information may not be reliable, and should be interpreted with care. While only 5 papers (11%) included in this review were from low-middle income countries, low-middle income countries often rely more heavily on household vaccination information for decision making. Recommended actions include strengthening quality of child-level data and increasing investments to improve vaccination card availability and card marking. There is also an urgent need for additional validation studies of vaccine coverage in low and middle income countries.

Link: http://www.technet-21.org/library/main/2439-survey-publication-miles-vaccine-2013

Publication: Luman Vaccine 2008

Publication abstract: Public health programs rely on household-survey estimates of vaccination coverage as a basis of programmatic and policy decisions; however, the validity of estimates derived from household-retained vaccination cards and parental recall has not been thoroughly evaluated. Using data from a vaccination coverage survey conducted in the Western Pacific's Northern Mariana Islands, we compared results from household data sources to medical record sources for the same children. We calculated the percentage of children aged 1, 2, and 6 years who received all vaccines recommended by age 12 months, 24 months, and for school entry, respectively. Coverage estimates based on vaccination cards ranged from 14% to 30% in the three age groups compared to 78-91% for the same children based on medical records. When cards were supplemented by parental recall, estimates were 51-53%. Concordance, sensitivity, specificity, positive and negative predictive values, and kappa statistics generally indicated poor agreement between household and medical record sources. Household-retained vaccination cards and parental recall were insufficient sources of information for estimating vaccination coverage in this population. This study emphasizes the importance of identifying reliable sources of vaccination history information and reinforces the need for awareness of the potential limitations of vaccination coverage estimated from surveys that rely on household-retained cards and/or parental recall.

Link: http://www.technet-21.org/library/main/2438-survey-luman-vaccine-2008

Book: Brown 2002

Publication abstract: Background This study aims to assess of the quality of child immunization coverage estimates obtained in 101 national population-based surveys in mostly developing countries. Methods The Demographic and Health Surveys (DHS) and UNICEF's Multiple Indicator Cluster Sample (MICS) surveys provide national immunization coverage estimates for children aged 12-23 months once every three to five years in many developing countries. The data are collected by interview from a nationally representative sample of households. 83 DHS and 18 MICS surveys were included. Findings 85% of mothers reported that they had ever received a health card for their child. 81% still had the card at the time of the interview, and nearly two-thirds of these presented the card to the interviewer. Cards were therefore observed for 55% of children overall. Rural and less educated mothers were less likely to report receiving health cards. Recall of additional immunizations by mothers that presented a card ranged from 1 to 3%. Recall of immunizations by mothers who reported never receiving a card ranged from 9 to 32%. Coverage among those who did not show a card rarely exceeded coverage among those who did, and there was good correlation between DPT and OPV doses received according to health card and recall data. Conclusion Though maternal recall data are known to be less accurate than health card data, we found no major systematic weaknesses in recall and believe that inclusion of recall data yields more accurate coverage estimates.

Link: http://www.technet-21.org/library/main/2436-survey-brown-global-book-2002

Bharti, 2016, Measuring populations to improve vaccination coverage

In low-income settings, vaccination campaigns supplement routine immunization but often fail to achieve coverage goals due to uncertainty about target population size and distribution. Accurate, updated estimates of target populations are rare but critical; short-term fluctuations can greatly impact population size and susceptibility. We use satellite imagery to quantify population fluctuations and the coverage achieved by a measles outbreak response vaccination campaign in urban Niger and compare campaign estimates to measurements from a post-campaign survey. Vaccine coverage was overestimated because the campaign underestimated resident numbers and seasonal migration further increased the target population. We combine satellite-derived measurements of fluctuations in population distribution with high-resolution measles case reports to develop a dynamic model that illustrates the potential improvement in vaccination campaign coverage if planners account for predictable population fluctuations. Satellite imagery can improve retrospective estimates of vaccination campaign impact and future campaign planning by synchronizing interventions with predictable population fluxes.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3519

Brown, 2014, Lot Quality Assurance Sampling to Monitor Supplemental Immunization Activity Quality: An Essential Tool for Improving Performance in Polio Endemic Countries

Monitoring the quality of supplementary immunization activities (SIAs) is a key tool for polio eradication. Regular monitoring data, however, are often unreliable, showing high coverage levels in virtually all areas, including those with ongoing virus circulation. To address this challenge, lot quality assurance sampling (LQAS) was introduced in 2009 as an additional tool to monitor SIA quality. Now used in 8 countries, LQAS provides a number of programmatic benefits: identifying areas of weak coverage quality with statistical reliability, differentiating areas of varying coverage with greater precision, and allowing for trend analysis of campaign quality. LQAS also accommodates changes to survey format, interpretation thresholds, evaluations of sample size, and data collection through mobile phones to improve timeliness of reporting and allow for visualization of campaign quality. LQAS becomes increasingly important to address remaining gaps in SIA quality

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3520

Cutts, 2013, Measuring Coverage in MNCH: Design, Implementation, and Interpretation Challenges Associated with Tracking Vaccination Coverage Using Household Surveys

Vaccination coverage is an important public health indicator that is measured using administrative reports and/or surveys. The measurement of vaccination coverage in low- and middle-income countries using surveys is susceptible to numerous challenges. These challenges include selection bias and information bias- which cannot be solved by increasing the sample size- and the precision of the coverage estimate- which is determined by the survey sample size and sampling method. Selection bias can result from an inaccurate sampling frame or inappropriate field procedures and- since populations likely to be missed in a vaccination coverage survey are also likely to be missed by vaccination teams- most often inflates coverage estimates. Importantly- the large multi-purpose household surveys that are often used to measure vaccination coverage have invested substantial effort to reduce selection bias. Information bias occurs when a child's vaccination status is misclassified due to mistakes on his or her vaccination record- in data transcription- in the way survey questions are presented- or in the guardian's recall of vaccination for children without a written record. There has been substantial reliance on the guardian's recall in recent surveys- and- worryingly- information bias may become more likely in the future as immunization schedules become more complex and variable. Finally- some surveys assess immunity directly using serological assays. Sero-surveys are important for assessing public health risk- but currently are unable to validate coverage estimates directly. To improve vaccination coverage estimates based on surveys- we recommend that recording tools and practices should be improved and that surveys should incorporate best practices for design- implementation- and analysis.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/511

Cutts, 2016, Monitoring vaccination coverage: Defining the role of surveys

Vaccination coverage is a widely used indicator of programme performance, measured by registries, routine administrative reports or household surveys. Because the population denominator and the reported number of vaccinations used in administrative estimates are often inaccurate, survey data are often considered to be more reliable. Many countries obtain survey data on vaccination coverage every 3-5years from large-scale multi-purpose survey programs. Additional surveys may be needed to evaluate coverage in Supplemental Immunization Activities such as measles or polio campaigns, or after major changes have occurred in the vaccination programme or its context. When a coverage survey is undertaken, rigorous statistical principles and field protocols should be followed to avoid selection bias and information bias. This requires substantial time, expertise and resources hence the role of vaccination coverage surveys in programme monitoring needs to be carefully defined. At times, programmatic monitoring may be more appropriate and provides data to guide program improvement. Practical field methods such as health facility-based assessments can evaluate multiple aspects of service provision, costs, coverage (among clinic attendees) and data quality. Similarly, purposeful sampling or censuses of specific populations can help local health workers evaluate their own performance and understand community attitudes, without trying to claim that the results are representative of the entire population. Administrative reports enable programme managers to do real-time monitoring, investigate potential problems and take timely remedial action, thus improvement of administrative estimates is of high priority. Most importantly, investment in collecting data needs to be complemented by investment in acting on results to improve performance.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3522

Cutts, 2016, Reply to comments on Monitoring vaccination coverage: Defining the role of surveys

Dear Editor, We thank Pond and Mounier-Jack for their comments on our paper, ‘‘Monitoring vaccination coverage: Defining the role of surveys” [1]. We agree that for many countries, administrative estimates of coverage are greatly inflated and misleading for programme planning purposes. The robustness of the WHO-UNICEF estimates of national immunization coverage (WUENIC) depends on the quality of the underlying data reviewed, which include administrative reports, as well as probability and non-probability sample surveys. In 2012, the Grade of Confidence (GoC) was introduced as a means of conveying uncertainty in WUENIC [2] and is low in the seven conflict-affected countries listed by Pond and Mounier-Jack. Table 1 shows that in five of these countries, vaccination cards were available for less than half the children surveyed; when card availability is low, it is particularly difficult to compare coverage trends. For example, in Nigeria, the proportion of children with DTP3 according to card was similar in surveys in 2010, 2011 and 2013, but in the EPI survey of 2010 a verbal history of vaccination was reported for 43% of children, more than double that of previous or subsequent surveys. Elsewhere, results from surveys did not always match expected trends (e.g. no apparent fall in coverage between surveys despite a 7 month stockout of DTP in one country), and some results were very unlikely (e.g. zero dropout between DTP1 and DTP3 in one Multiple Indicator Cluster Survey (MICS) (data from country reports at http://apps.who.int/ immunization_monitoring/globalsummary/wucoveragecountrylist. html)). The updated WHO guidelines on vaccination coverage surveys (http://www.who.int/immunization/monitoring_surveillance/ Vaccination_coverage_cluster_survey_with_annexes.pdf) discuss the challenges of using a new survey to compare with an older one, particularly an immunization coverage survey – these often lacked information on likely biases and confidence intervals were either not reported or not very meaningful from non-probability samples. The best way to compare results from different surveys is to plan a pair of surveys for such a purpose and work very hard to ensure standardised, well-documented and high quality data collection in both. Pond and Mounier-Jack suggest that two such surveys are feasible within each 5 years period. We would be reluctant to stipulate any particular interval as the usefulness of repeat surveys will depend in part on the likelihood of a change in coverage having occurred (which can be predicted from monitoring other indicators) [1] and the availability of accurate documentation of vaccination status on home-based or clinic records. Most of all, surveys should lead to action to strengthen programme performance and this is likely the weakest link in many countries, including those affected by conflict.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3521

Cutts, 2016, Seroepidemiology: an underused tool for designing and monitoring vaccination programmes in low- and middle-income countries

Seroepidemiology, the use of data on the prevalence of bio-markers of infection or vaccination, is a potentially powerful tool to understand the epidemiology of infection before vaccination and to monitor the effectiveness of vaccination programmes. Global and national burden of disease estimates for hepatitis B and rubella are based almost exclusively on serological data. Seroepidemiology has helped in the design of measles, poliomyelitis and rubella elimination programmes, by informing estimates of the required population immunity thresholds for elimination. It contributes to monitoring of these programmes by identifying population immunity gaps and evaluating the effectiveness of vaccination campaigns. Seroepidemiological data have also helped to identify contributing factors to resurgences of diphtheria, Haemophilus Influenzae type B and pertussis. When there is no confounding by antibodies induced by natural infection (as is the case for tetanus and hepatitis B vaccines), seroprevalence data provide a composite picture of vaccination coverage and effectiveness, although they cannot reliably indicate the number of doses of vaccine received. Despite these potential uses, technological, time and cost constraints have limited the widespread application of this tool in low-income countries. The use of venous blood samples makes it difficult to obtain high participation rates in surveys, but the performance of assays based on less invasive samples such as dried blood spots or oral fluid has varied greatly. Waning antibody levels after vaccination may mean that seroprevalence underestimates immunity. This, together with variation in assay sensitivity and specificity and the common need to take account of antibody induced by natural infection, means that relatively sophisticated statistical analysis of data is required. Nonetheless, advances in assays on minimally invasive samples may enhance the feasibility of including serology in large survey programmes in low-income countries. In this paper, we review the potential uses of seroepidemiology to improve vaccination policymaking and programme monitoring and discuss what is needed to broaden the use of this tool in low- and middle-income countries.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3523

Dietz, 2004, Assessing and monitoring vaccination coverage levels: lessons from the Americas
Hund, 2015, Choosing a Cluster Sampling Design for Lot Quality Assurance Sampling Surveys

Lot quality assurance sampling (LQAS) surveys are commonly used for monitoring and evaluation in resource-limited settings. Recently several methods have been proposed to combine LQAS with cluster sampling for more timely and cost-effective data collection. For some of these methods, the standard binomial model can be used for constructing decision rules as the clustering can be ignored. For other designs, considered here, clustering is accommodated in the design phase. In this paper, we compare these latter cluster LQAS methodologies and provide recommendations for choosing a cluster LQAS design. We compare technical differences in the three methods and determine situations in which the choice of method results in a substantively different design. We consider two different aspects of the methods: the distributional assumptions and the clustering parameterization. Further, we provide software tools for implementing each method and clarify misconceptions about these designs in the literature. We illustrate the differences in these methods using vaccination and nutrition cluster LQAS surveys as example designs. The cluster methods are not sensitive to the distributional assumptions but can result in substantially different designs (sample sizes) depending on the clustering parameterization. However, none of the clustering parameterizations used in the existing methods appears to be consistent with the observed data, and, consequently, choice between the cluster LQAS methods is not straightforward. Further research should attempt to characterize clustering patterns in specific applications and provide suggestions for best-practice cluster LQAS designs on a setting-specific basis.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/1475

Jandee, 2015, Effectiveness of Using Mobile Phone Image Capture for Collecting Secondary Data: A Case Study on Immunization History Data Among Children in Remote Areas of Thailand

Entering data onto paper-based forms, then digitizing them, is a traditional data-management method that might result in poor data quality, especially when the secondary data are incomplete, illegible, or missing. Transcription errors from source documents to case report forms (CRFs) are common, and subsequently the errors pass from the CRFs to the electronic database. OBJECTIVE: This study aimed to demonstrate the usefulness and to evaluate the effectiveness of mobile phone camera applications in capturing health-related data, aiming for data quality and completeness as compared to current routine practices exercised by government officials. METHODS: In this study, the concept of "data entry via phone image capture" (DEPIC) was introduced and developed to capture data directly from source documents. This case study was based on immunization history data recorded in a mother and child health (MCH) logbook. The MCH logbooks (kept by parents) were updated whenever parents brought their children to health care facilities for immunization. Traditionally, health providers are supposed to key in duplicate information of the immunization history of each child; both on the MCH logbook, which is returned to the parents, and on the individual immunization history card, which is kept at the health care unit to be subsequently entered into the electronic health care information system (HCIS). In this study, DEPIC utilized the photographic functionality of mobile phones to capture images of all immunization-history records on logbook pages and to transcribe these records directly into the database using a data-entry screen corresponding to logbook data records. DEPIC data were then compared with HCIS data-points for quality, completeness, and consistency. RESULTS: As a proof-of-concept, DEPIC captured immunization history records of 363 ethnic children living in remote areas from their MCH logbooks. Comparison of the 2 databases, DEPIC versus HCIS, revealed differences in the percentage of completeness and consistency of immunization history records. Comparing the records of each logbook in the DEPIC and HCIS databases, 17.3% (63/363) of children had complete immunization history records in the DEPIC database, but no complete records were reported in the HCIS database. Regarding the individual's actual vaccination dates, comparison of records taken from MCH logbook and those in the HCIS found that 24.2% (88/363) of the children's records were absolutely inconsistent. In addition, statistics derived from the DEPIC records showed a higher immunization coverage and much more compliance to immunization schedule by age group when compared to records derived from the HCIS database. CONCLUSIONS: DEPIC, or the concept of collecting data via image capture directly from their primary sources, has proven to be a useful data collection method in terms of completeness and consistency. In this study, DEPIC was implemented in data collection of a single survey. The DEPIC concept, however, can be easily applied in other types of survey research, for example, collecting data on changes or trends based on image evidence over time. With its image evidence and audit trail features, DEPIC has the potential for being used even in clinical studies since it could generate improved data integrity and more reliable statistics for use in both health care and research settings.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/1469

Lessler, 2011, Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis

The performance of routine and supplemental immunization activities is usually measured by the administrative method: dividing the number of doses distributed by the size of the target population. This method leads to coverage estimates that are sometimes impossible (e.g., vaccination of 102% of the target population), and are generally inconsistent with the proportion found to be vaccinated in Demographic and Health Surveys (DHS). We describe a method that estimates the fraction of the population accessible to vaccination activities, as well as within-campaign inefficiencies, thus providing a consistent estimate of vaccination coverage. Methods and Findings: We developed a likelihood framework for estimating the effective coverage of vaccination programs using cross-sectional surveys of vaccine coverage combined with administrative data. We applied our method to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone, using data from each country’s most recent DHS survey and administrative coverage data reported to the World Health Organization. We estimate that 93% (95% CI: 91, 94) of the population in Ghana was ever covered by any measles vaccination activity, 77% (95% CI: 78, 81) in Madagascar, and 69% (95% CI: 67, 70) in Sierra Leone. ‘‘Within-activity’’ inefficiencies were estimated to be low in Ghana, and higher in Sierra Leone and Madagascar. Our model successfully fits age-specific vaccination coverage levels seen in DHS data, which differ markedly from those predicted by naı¨ve extrapolation from country-reported and World Health Organization–adjusted vaccination coverage. Conclusions: Combining administrative data with survey data substantially improves estimates of vaccination coverage. Estimates of the inefficiency of past vaccination activities and the proportion not covered by any activity allow us to more accurately predict the results of future activities and provide insight into the ways in which vaccination programs are failing to meet their goals.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3546

Liu, 2013, Measuring Coverage in MNCH: A Validation Study Linking Population Survey Derived Coverage to Maternal, Newborn, and Child Health Care Records in Rural China

Accurate data on coverage of key maternal, newborn, and child health (MNCH) interventions are crucial for monitoring progress toward the Millennium Development Goals 4 and 5. Coverage estimates are primarily obtained from routine population surveys through self-reporting, the validity of which is not well understood. We aimed to examine the validity of the coverage of selected MNCH interventions in Gongcheng County, China. Method and Findings: We conducted a validation study by comparing women’s self-reported coverage of MNCH interventions relating to antenatal and postnatal care, mode of delivery, and child vaccinations in a community survey with their paper- and electronic-based health care records, treating the health care records as the reference standard. Of 936 women recruited, 914 (97.6%) completed the survey. Results show that self-reported coverage of these interventions had moderate to high sensitivity (0.57 [95% confidence interval (CI): 0.50–0.63] to 0.99 [95% CI: 0.98–1.00]) and low to high specificity (0 to 0.83 [95% CI: 0.80–0.86]). Despite varying overall validity, with the area under the receiver operating characteristic curve (AUC) ranging between 0.49 [95% CI: 0.39–0.57] and 0.90 [95% CI: 0.88–0.92], bias in the coverage estimates at the population level was small to moderate, with the test to actual positive (TAP) ratio ranging between 0.8 and 1.5 for 24 of the 28 indicators examined. Our ability to accurately estimate validity was affected by several caveats associated with the reference standard. Caution should be exercised when generalizing the results to other settings. Conclusions: The overall validity of self-reported coverage was moderate across selected MNCH indicators. However, at the population level, self-reported coverage appears to have small to moderate degree of bias. Accuracy of the coverage was particularly high for indicators with high recorded coverage or low recorded coverage but high specificity. The study provides insights into the accuracy of self-reports based on a population survey in low- and middle-income countries. Similar studies applying an improved reference standard are warranted in the future.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3524

Luman, 2007, Use and abuse of rapid monitoring to assess coverage during mass vaccination campaigns

This article describes the intended use of Rapid Coverage Monitoring tool, and some of the ways in which it has been misused for unintended purposes.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/760

MacNeil, 2014, Issues and considerations in the use of serologic biomarkers for classifying vaccination history in household surveys

Accurate estimates of vaccination coverage are crucial for assessing routine immunization program performance. Community based household surveys are frequently used to assess coverage within a country. In household surveys to assess routine immunization coverage, a child's vaccination history is classified on the basis of observation of the immunization card, parental recall of receipt of vaccination, or both; each of these methods has been shown to commonly be inaccurate. The use of serologic data as a biomarker of vaccination history is a potential additional approach to improve accuracy in classifying vaccination history. However, potential challenges, including the accuracy of serologic methods in classifying vaccination history, varying vaccine types and dosing schedules, and logistical and financial implications must be considered. We provide historic and scientific context for the potential use of serologic data to assess vaccination history and discuss in detail key areas of importance for consideration in the context of using serologic data for classifying vaccination history in household surveys. Further studies are needed to directly evaluate the performance of serologic data compared with use of immunization cards or parental recall for classification of vaccination history in household surveys, as well assess the impact of age at the time of sample collection on serologic titers, the predictive value of serology to identify a fully vaccinated child for multi-dose vaccines, and the cost impact and logistical issues on outcomes associated with different types of biological samples for serologic testing.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3531

Ngandu, 2016, Does adjusting for recall in trend analysis affect coverage estimates for maternal and child health indicators? An analysis of DHS and MICS survey data

The Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) are the major data sources in low- and middle-income countries (LMICs) for evaluating health service coverage. For certain maternal and child health (MCH) indicators, the two surveys use different recall periods: 5 years for DHS and 2 years for MICS. Objective: We explored whether the different recall periods for DHS and MICS affect coverage trend analyses as well as missing data and coverage estimates. Designs: We estimated coverage, using proportions with 95% confidence intervals, for four MCH indicators: intermittent preventive treatment of malaria in pregnancy, tetanus vaccination, early breastfeeding and postnatal care. Trends in coverage were compared using data from 1) standard 5-yearDHS and 2-year MICS recall periods (unmatched) and 2) DHS restricted to 2-year recall to match the MICS 2-year recall periods (matched). Linear regression was used to explore the relationship between length of recall, missing data and coverage estimates. Results: Differences in coverage trends were observed between matched and unmatched data in 7 of 18 (39%) comparisons performed. The differences were in the direction of the trend over time, the slope of the coverage change or the significance levels. Consistent trends were seen in 11 of the 18 (61%) comparisons. Proportion of missing data was inversely associated with coverage estimates in both short (2 years) and longer (5 years) recall of the DHS (r0.3, p0.02 and r0.4, p0.004, respectively). The amount of missing information was increased for longer recall compared with shorter recall for all indicators (significant odds ratios ranging between 1.44 and 7.43). Conclusions: In a context where most LMICs are dependent on population-based household surveys to derive coverage estimates, users of these types of data need to ensure that variability in recall periods and the proportion of missing data across data sources are appropriately accounted for when trend analyses are conducted.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3525

Okayasu, 2014, Cluster Lot Quality Assurance Sampling: Effect of Increasing the Number of Clusters on Classification Precision and Operational Feasibility

To assess the quality of supplementary immunization activities (SIAs), the Global Polio Eradication Initiative (GPEI) has used cluster lot quality assurance sampling (C-LQAS) methods since 2009. However, since the inception of C-LQAS, questions have been raised about the optimal balance between operational feasibility and precision of classification of lots to identify areas with low SIA quality that require corrective programmatic action. METHODS: To determine if an increased precision in classification would result in differential programmatic decision making, we conducted a pilot evaluation in 4 local government areas (LGAs) in Nigeria with an expanded LQAS sample size of 16 clusters (instead of the standard 6 clusters) of 10 subjects each. RESULTS: The results showed greater heterogeneity between clusters than the assumed standard deviation of 10%, ranging from 12% to 23%. Comparing the distribution of 4-outcome classifications obtained from all possible combinations of 6-cluster subsamples to the observed classification of the 16-cluster sample, we obtained an exact match in classification in 56% to 85% of instances. CONCLUSIONS: We concluded that the 6-cluster C-LQAS provides acceptable classification precision for programmatic action. Considering the greater resources required to implement an expanded C-LQAS, the improvement in precision was deemed insufficient to warrant the effort. Published by Oxford University Press on behalf of the Infectious Diseases Society of America 2014. This work is written by (a) US Government employee(s) and is in the public domain in the US.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3527

Pond, 2016, Comments on “Monitoring vaccination coverage: Defining the role of surveys”

Response to an article - Dear Editors, Felicity Cutts and co-authors [Vaccine 34 (2016) 4103–4109] provide a good overview of the role of household surveys in monitoring of immunization coverage. More should be said, however, about the optimal monitoring strategy for lower coverage countries. The most recent WHO/UNICEF estimates of national immunization coverage (WUENIC) suggest that for 26 (57%) of 46 countries with 2015 DTP3 coverage below 85%, the administrative data over-estimate coverage by from 10 percentage points to as much as 40 percentage points.1 Of course, as the article points out, coverage surveys provide an imperfect ‘‘gold standard” with which to assess even national (let alone sub-national) immunization coverage. However, when national coverage is low it is essential to periodically attempt to validate the administrative coverage estimate with an estimate from a high quality nationwide household survey.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3529

Rhoda, 2010, LQAS: User Beware

Researchers around the world are using Lot Quality Assurance Sampling (LQAS) techniques to assess public health parameters and evaluate program outcomes. In this paper, we report that there are actually two methods being called LQAS in the world today, and that one of them is badly flawed. Methods This paper reviews fundamental LQAS design principles, and compares and contrasts the two LQAS methods. We raise four concerns with the simply-written, freely-downloadable training materials associated with the second method. Results The first method is founded on sound statistical principles and is carefully designed to protect the vulnerable populations that it studies. The language used in the training materials for the second method is simple, but not at all clear, so the second method sounds very much like the first. On close inspection, however, the second method is found to promote study designs that are biased in favor of finding programmatic or intervention success, and therefore biased against the interests of the population being studied. Conclusion We outline several recommendations, and issue a call for a new high standard of clarity and face validity for those who design, conduct, and report LQAS studies.

Link: www.technet-21.org/en/library/main/explore/programme-management/3407

Travassos, 2016, Immunization Coverage Surveys and Linked Biomarker Serosurveys in Three Regions of Ethiopia
Weber, 2009, Consultancy services for conducting an evaluation of immunization coverage monitoring methodology and process

Good quality immunization data are crucial for an accurate monitoring of progress towards immunization related targets. The accuracy of immunization data has raised serious concerns. Immunisation coverage figures from various sources referring to the same similar geographical area or target group are often inconsistent. The survey aims at describing the perceptions and experience of selected immunisation stakeholders in relation to the use, quality and ways to improve immunisation coverage data. A web-based questionnaire was elaborated, piloted and sent out to around 250 institutions involved in immunisation programmes including funding and research agencies, health policy decision makers, technical experts, and managers of immunisation programmes in 80 countries. This report presents data from 55 responses, mainly from EPI managers and WHO / UNICEF offices at country level. Further information expected from global funding agencies, research institutions and technical organisations will be included in the final survey report. Findings have to be interpreted with caution because responses may not necessarily reflect true opinions or facts. Administrative data is the most common source of data used.

Link: www.technet-21.org/en/library/main/explore/immunization-information-systems-coverage-monitoring/3533