A generic theory of change-based framework with core indicators for monitoring the effectiveness of large-scale food fortification programs in low- and middle-income countries

Abstract

Large-scale food fortification (LSFF) programs are widely implemented in low- and middle-income countries (LMIC) to alleviate micronutrient deficiencies. However, these programs may not achieve the desired impact due to poor design or bottlenecks in program implementation. Monitoring and evaluation (M&E) frameworks and a set of agreed indicators can help to benchmark progress and to strengthen the evidence-base of effectiveness in a standardized way. We aimed to formulate recommendations towards core indicators for evaluating the effectiveness of LSFF programs with their associated metrics, methods, and tools (IMMT). For this, we used a multi-method iterative approach, including a mapping review of the literature, semi-structured interviews with international experts, compilation of a generic Theory of Change (ToC) framework for LSFF program delivery, and selection of IMMT for M&E of LSFF programs at key stages along the ToC delivery framework. Lastly, we conducted exploratory, qualitative interviews with key informants in Nigeria to explore experiences and perceptions related to the implementation of LSFF programs in Nigeria’s context, and their opinion towards the proposed set of core IMMT. The literature search resulted in 14 published and 15 grey literature documents, from which we extracted a total of 41 indicators. Based on the available literature and interviews with international experts, we mapped a ToC delivery framework and selected nine core indicators at the output, outcome and impact level for M&E of the effectiveness of LSFF programs. Key informants in Nigeria revealed that the main bottlenecks for implementation of the proposed IMMT are related to the lack of technical capacity, equipment, laboratory infrastructure, and financial resources. In conclusion, we propose a set of nine core indicators for enabling comprehensive M&E of the effectiveness of LSFF programs in LMIC. This proposed set of core indicators can be used for further evaluation, harmonization and integration in national and international protocols for M&E of LSFF programs.

1. Introduction

Large-scale food fortification (LSFF), or mass fortification, concerns the addition of one or more micronutrients to industrially processed staple foods or condiments (1, 2). LSFF programs attempt to address micronutrient deficiencies of public health concern, such as deficiencies of iodine, iron, vitamin A, and folic acid, which occur most profoundly in low- and middle-income countries (LMIC). According to the Global Fortification Data Exchange platform¹ and the Food Fortification Initiative Network,² 126 countries had implemented universal salt iodization programs, 92 countries had mandatory food fortification programs for at least one cereal grain, and 34 countries had mandated the fortification of oil by September 2022, with in addition an unknown number of countries that have implemented fortification of sugar. LSFF is one of the most cost-effective and scalable micronutrient interventions to address inadequacies at the population level (2–6). Nevertheless, LSFF programs may not achieve the desired impact due to poor design and bottlenecks in program implementation. With regard to the latter, for example, almost half of fortified foods in 20 national fortification programs implemented in 12 countries were found not to meet national fortification standards (7). Most LSFF programs, however, do not report any data on compliance, coverage, or impact, leaving major data gaps in monitoring their effectiveness and limiting the ability for corrective measures to be taken (2, 4, 5, 7, 8).

As also highlighted in the Arusha Statement on Food Fortification in 2015,³ effective LSFF programs require sound design; to have clear regulatory frameworks; to be continuously monitored for quality assurance (QA) and quality control (QC) procedures at the production site; to have a simple and cost-efficient enforcement system that monitors compliance with fortification standards at production and commercial levels; and to assess equitable coverage and consumption of fortified foods by the targeted population (4, 5). Although an initial generic monitoring and evaluation (M&E) system for food fortification programs was proposed by WHO/FAO in 2006 (1), there is currently no coherent and harmonized M&E framework with a standardized set of essential indicators.

Here, in a multi-method iterative study approach, we aimed to: (1) review published and grey literature for frameworks and commonly used indicators, metrics, methods, and tools (IMMT) to monitor and evaluate LSFF programs; (2) to contextualize the findings from the literature through interviews with international experts; (3) to develop a generic theory of change, followed by a proposed selection of a minimum set of IMMT that are key to track LSFF program effectiveness; and (4) to verify the applicability of this generic theory of change in a local food fortification context, including elucidating the perceptions and barriers towards the proposed set of IMMT by interviewing key informants in Nigeria.

2. Methodology

2.1. Literature review

To identify relevant sources from the published literature, we used the same search strategy as that used in a previous review by us on M&E indicators for biofortification programs (9). In short, we systematically searched for reviews with detailed descriptions of implementation processes, M&E activities, and IMMT for LSFF programs published between 2010 and 2019. The search was performed in MEDLINE, Cochrane Library, Web of Science, and Scopus (see Supplementary material). Additionally, key documents from the grey literature (e.g., program manuals, workshop reports, and indicator dashboards) were identified through an internet-based search and in the global databases GINA, WHOLIS, and SIGLE. The search was complemented with citation mining (10). In addition, documents provided to us by interviewed experts were added (see Section 2.2). Based on the inclusion and exclusion criteria (Table 1), published reviews and grey literature were selected for further use by two investigators independently (TCA and SRM). Any disagreements between the investigators were resolved through consultation with another researcher from the team (CL). To guide the identification of indicators for M&E from the published and grey literature, we adapted the WHO/CDC logic model for implementing micronutrient interventions in public health (12) to represent the underlying implementation processes of LSFF programs (Supplementary Figure S1). After pilot-testing by CL, two data charting forms were used to extract indicators categorized by the different components of the logic M&E framework (see Box 1 for definitions).

2.2. Semi-structured interviews with international experts

We then conducted semi-structured interviews (SSI) with senior international experts on LSFF programs to deepen our understanding of the literature. Based on their track record in the field, nine experts (Table 2) were recruited through purposeful and snowball sampling (14, 15) for an individual (online) interview on a voluntary basis. The purpose of the interview was explained to the participants in detail, both verbally and in writing. SSI participants were asked for verbal informed consent to participate and digitally record the interviews. We aimed to obtain a detailed description of LSFF delivery models, crucial success factors, M&E frameworks, methods, and tools for data collection, and in particular methods for assessing coverage and consumption of fortified foods. SSI guides were developed with guiding questions, including detailed probes on topics of inquiry (16). Data were collected over 5 weeks from March–June, 2021 until data saturation was reached among key themes (16, 17).

After verbatim transcription of the interview recordings by two research assistants, thematic analysis was conducted with the help of the data management software Dedoose version 8.3.47 (18). A codebook with 22 categories of information was developed by SRM and FMG, and cross-checked by SRK. The first coding cycle comprised the application of the initial codes to the transcripts, while the 22 categories were clustered and merged into eight pattern codes (i.e., thematic areas) during a second coding cycle. Further details on the study methodology can be found in Rodas-Moya et al. (9).

2.3. Compilation of a theory of change framework and selection of IMMT

Based on the implementation processes of LSFF programs encountered in the literature and as described by international experts, we used a Theory of Change (ToC) approach to create an initial draft framework to show the linkages between the activities, outputs, outcomes, and impact. We then compared this draft to: (1) a generic monitoring and evaluation framework for food fortification programs published by WHO in 2006 (1); (2) the WHO/CDC logic model which we had already used to select IMMT from the literature (Supplementary Figure S1); (3) a generic impact pathway for LSFF developed by GAIN as part of the Fortification Assessment Coverage Tool (FACT) kit (19); and (4) an impact pathway of an LSFF program using multiple food vehicles, i.e., fortifiable foods, from a basket of fortified foods from Costa Rica, which was based on interviews with LSFF experts, program documents, program monitoring data and representative national surveys data (20). The first three of these models represent generic LSFF delivery models constructed with multi-contextual evidence of LSFF programs from LMIC and may thus be transferable across similar settings. The delivery model from Costa Rica represents a rigorously constructed, plausible impact pathway of a fully mature LSFF program that shows the crucial success factors for program effectiveness.

We then compiled a final version of the ToC and employed it as an analysis framework for the selection of a set of core IMMT. Since our aim was for the IMMT to measure effectiveness of LSFF programs rather than their design and implementation (Figure 1), we limited our selection to indicators of output and outcome. We aimed to limit the number of indicators to a minimum in order to keep routine M&E practices as simple and cost-effective as possible.

Figure 1

2.4. Interviews with key informants on food fortification programs in Nigeria

As a final step, we aimed to verify the applicability of our M&E framework in a local context. We aimed to select an LMIC with mandatory fortification of at least two food vehicles. Nigeria was selected because it fortifies (and biofortifies) multiple food vehicles. Nigeria started its first LSFF program in 1993 with the iodization of salt. This fortification program was successful: the latest national surveys showed that 92% (2018) and 97% (2019) of households had iodized salt available (22, 23). The salt iodization program was followed by mandatory fortification of several other foods in 2000 and beyond, such as vegetable oil (vitamin A), wheat and maize flour (multiple nutrients), sugar (vitamin A), and margarine and butter (vitamin A). However, these programs have been relatively less successful. In a national survey carried out in 2013, only 12.2–33.0% of wheat flour samples were adequately fortified with vitamin A and 1.0–21.0% with iron, while just 14.9–20.2% of vegetable oil was adequately fortified (24). In addition, a sub-national study in the states of Lagos and Kano found that only 5.4–22.7% of households consumed fortified wheat flour and 7.2–7.6% consumed fortified vegetable oil (25, 26).

To reach our aim, we conducted semi-structured, in-depth interviews with key informants. We were specifically interested to hear opinions towards the potential opportunities and barriers when using our proposed IMMT in Nigeria. Participants were identified based on their role in food fortification programs (i.e., being program implementers), either as representatives of government, industry, non-governmental organizations (NGOs), or academia, followed by some snowball sampling as suggested by interviewed participants [(14, 15); Table 3]. The interviews were conducted in person by a team of local data collectors led by FS and AO. Data collectors were trained remotely by SRM and FMG. Interviews were conducted in English and recorded with digital recorders after gaining informed consent from the participants. The recordings were transcribed verbatim locally by professional transcribers and cleaned by SRM and FMG for subsequent analysis. Data were collected in January 2022. Participation in interviews was voluntary. All interviewees were informed that they could refuse to participate or withdraw from the study at any point without any consequences for them. Participants received a detailed explanation (both verbal and written) about the purpose of the interview. Data management and analysis followed similar procedures as described for SSI with international experts (see section 2.2).

3. Results

3.1. M&E frameworks and IMMT extracted from the literature

The literature search resulted in 29 documents: 14 published and 15 gray literature documents. The published literature included 11 reviews (2, 20, 27–35); an effectiveness evaluation study (36); a case study (37); and a review older than 10 years but with extensive information on indicators for M&E of flour fortification programs (38). Nine reviews included information on methods for data collection (20, 28, 30, 33–38); while five reviews (20, 33–35, 38) and two studies (36, 37) included information on tools.

From the grey literature, we retrieved 15 documents, including eight manuals for internal and external quality assurance and control (QA/QC) of fortified wheat flour, salt, sugar, and oil (39–46); a manual that described procedures to enforce the quality of fortified foods at importation sites (47); a manual for commercial inspection of fortified foods (48); a code of practice for micronutrient premix operations (49); a workshop report containing a critical assessment on the root causes of failure in fortification quality; a report of a technical consultation on M&E of LSFF programs (21); and an article on technological aspects of LSFF and public health nutrition (50). Furthermore, the Fortification Assessment Coverage Toolkit (FACT) manual was included to retrieve IMMT for assessing the quality of fortified foods, their coverage, and consumption at population level (19, 51, 52). We did not find any relevant documents containing IMMT for LSFF in the global databases GINA, WHOLIS, and SIGLE.

A total of 16 activity, 11 output, 9 outcome, and 2 impact indicators were extracted from the literature, as presented in Table 4. We did not encounter any specific input indicators in the literature. Activity indicators were centered around the existence of legislation, regulation, and national standards; quality assurance of fortificant premixes, and the presence of internal and external QA/QC processes in support of food fortification. Output indicators concerned the actual production of fortified foods, their market availability, and quantitative verification of adherence to the national standards. Outcome indicators provided information on consumer awareness, coverage and contribution of fortified foods to micronutrient intake (primary/direct outcome). Impact indicators, finally, served to evaluate the effect of a fortification program on nutritional (secondary) and clinical (tertiary) outcomes.

3.2. Insights shared by international experts

We conducted nine SSI with international experts in total (Table 2). Two relevant primary themes emerged from the SSI dataset for theory building, namely: (1) Feasibility to develop a generic M&E framework for LSFF programs; and (2) Methods to estimate coverage and consumption of fortified foods.

3.2.1. Feasibility to develop a generic M&E framework for LSFF program delivery

When exploring the feasibility of developing a generic M&E framework for LSFF programs for global use, centered around common elements of program implementation, most participants indicated that creating such a framework should be possible under certain conditions. One participant explained:

“There are many similarities between [fortification] programs across countries and regions. It should be possible to have a generic M&E framework regardless of how the programs are designed. Yet, the instruments for data collection and the questions to feed the M&E systems need to be designed and adapted for specific contexts to get relevant information. Also, the system should be flexible, not prescriptive or rigid to make contextual adaptability possible.”

                 Academic LSFF expert.

It was broadly acknowledged that any food fortification program cannot happen without participation of the government and support from the industry. Participants mentioned several crucial responsibilities from the side of the government, such as: (1) defining the need and conditions of a fortification program, based on evidence of deficiencies and food consumption patterns of the population; (2) identification of appropriate food vehicle(s); (3) determining the quantities and chemical form of the fortificants to be added and setting national standards (NS); (4) pass legislation; and (5) installing an efficient and reliable enforcement system to ensure compliance with NS. To enable this, designees should be trained that are responsible for enforcement of the legislation by QA/QC at production site and commercial level (e.g., wholesalers, retailers, bakeries). Lastly, governments should level the playing field for industries participating in mandatory fortification program(s). The majority of senior experts stressed that behavior change communication towards consumers does not need to be a priority, since, as stated, when implemented well, the fortified commodities will be consumed anyway.

When exploring the role of the industry in the fortification process, four primary responsibilities were described by participants: (1) Acquiring appropriate technology and inputs (e.g., premix and fortificant) for fortification; (2) Training technical personnel on fortification-related QA/QC processes internally; (3) Mainstreaming fortification QA/QC procedures into existing QA/QC protocols (i.e., not creating new or separate QA/QC programs for fortification) to warrant the production of high quality food products that meet the NS for fortification; and (4) Consistent fortification of the food vehicles at NS as mandated by law, transparent reporting of QA processes and QC results.

3.2.2. Methods to assess the coverage and consumption of fortified foods

Several participants stressed that fortification program effectiveness can only be evaluated when robust data on the quality of fortified foods are available, as well as data on coverage and consumption of fortified foods. Such data allow the estimation of program contribution to micronutrient intake and the extent to which the program contributes to filling nutrient gaps in target population groups. All participants indicated that program evaluation at the outcome level (coverage, consumption, contribution of fortified foods to micronutrient intake, and relief of micronutrient intake inadequacies) should only be carried out by specialists with experience in program evaluation, and only when there is evidence that the program has been implemented as planned and operates efficiently (e.g., the industry is fortifying according to standards). A participant explained:

“Countries should avoid conducting program evaluations focusing on coverage and consumption of fortified foods and changes in nutritional status attributable to fortification programs because the design of these evaluations is complex. Also, these evaluations should be conducted only for well-established programs, i.e., when fortification is mandatory, the national standards are enacted, enforcement works, and hence, there is evidence that the industry is fortifying according to standards, and coverage is extensive. Once these conditions are met, the effectiveness of the program can be evaluated by an experienced evaluation team.”

                 Senior nutritionist at an international NGO.

Participants indicated that the evaluation of coverage and consumption of fortified foods should be based on data from representative population samples, for instance surveys with cluster or multistage sampling. Some participants suggested to involve local universities to assess the coverage and consumption of fortified foods to strengthen local capacity in program evaluation.

One participant voiced concern about the proper conduct of surveys to assess coverage and consumption of fortified foods and indicated that survey design could be simplified:

“…too much complexity has been added to this level of monitoring. Some programs use large surveys with 3,000 households or more and the corresponding collection of samples that subsequently are sent to a laboratory for analysis. This is not needed. Coverage surveys have often a cluster design where data are collected in clusters of, for example, 30 households (= 1 cluster). A sample of the fortified food vehicle can be collected in each of the 30 houses to make a composite sample for the cluster. The composite samples of the clusters are then sent for quantitative analysis to the lab to get the average and standard deviation (variation) of the fortification levels. Those data can be used to estimate the contribution of the fortified food vehicle to micronutrient intake at the household level. This method is also logical because people don’t eat one sample one day, but a collection of samples over a time period. Hence the average of fortification levels collected from a cluster of households can give stronger estimates of average consumption over time than single samples.”

                 Senior LSFF expert at an international NGO.

The same participant suggested using Household Consumption and Expenditure Surveys (HCES) to estimate coverage and consumption of fortified foods. An advantage of these surveys is that they already work with large sample sizes that permit disaggregation of data at national, regional and sub-regional levels, as well as by household factors to assess equity of coverage. The participant explained:

“HCES uses large sample sizes, for example, 24,000 households are surveyed, and data can be disaggregated by region, type of residence (urban/rural), and economic quintile. These surveys have a food module that contains a list of more than 100 foods consumed at the household with reported weights [.....]. Purchase and sometimes consumption of the foods on the list are reported over seven or 15 days. It is also possible to check which of these foods are […] fortifiable or already fortified foods. HCES data allow to estimate consumption of foods at the household level using the average (fe)male equivalent method⁴.”

                 Senior LSFF expert at an international NGO.

We further enquired whether 24-h recalls (24hR) or Food Frequency Questionnaires (FFQ) should be used to provide estimates of coverage and consumption of fortified foods, rather than those calculated based on average (fe)male equivalents with HCES data. The majority of participants indicated that 24hR can provide more accurate estimates of food intake than HCES. However, all participants acknowledged that conducting 24hR in most LMIC program settings will be challenging, and that there is a need to simplify methods. A participant explained:

“Consumption assessment is the science of approximation. 24hR are also an approximation. Why do we want to be so precise when we are talking about program evaluation at the population level? A scientific mind could be simple or complicated. Some researchers want to do things so strictly that they become impractical. HCES gives you highly relevant data for programs because it allows you to see what segments of the population are covered and what aren’t. This has programmatic value because it helps crafting strategies to target uncovered population groups. You monitor and evaluate a program because you want to steer it to achieve its goals. Program evaluation is not an epidemiological study”

                 Senior LSFF expert at an international NGO.

All participants also indicated that the integration of a module to assess coverage and consumption of fortified foods in large cross-sectional surveys, such as Demographic and Healthy Surveys (DHS) or the UNICEF Multiple Indicator Cluster Survey (MICS), could be a good alternative for establishing a separate routine evaluation system for LSFF. When asking participants about the time frame for assessing coverage and consumption of fortified and fortifiable foods, all of them indicated that once in 5 years would be sufficient.

Finally, we explored the possibilities of using existing diet quality tools, such as the Minimum Dietary Diversity for women (MDD-W) (54) and the Global Diet Quality Score (GDQS) (55, 56), to monitor the coverage of LSFF programs. However, international experts unanimously discouraged the use of these tools for M&E of LSFF programs, because they would provide insufficient actionable information.

3.3. Compiling a generic ToC framework with selected IMMT for monitoring effectiveness of LSFF programs

The final version of the ToC framework can be found in Figure 2. Out of the indicators identified through the mapping review (Table 4), a set of nine core indicators (Table 5) was selected such that the most critical implementation stages of LSFF are covered, as indicated in Figure 2. The selection includes one indicator at the activity level (indicator #2.5: Proportion of premix/fortificant samples with micronutrient content specified in CoA meeting national standards), because it was deemed crucial by most of the international experts to verify that the LSFF program is properly implemented before performing an effectiveness evaluation.

Figure 2

Indicators #1 and #2 were extracted from the manuals prepared by the East, Central and Southern Africa Health Community (ECSA-HC) with support from partners in the Regional Food Fortification Initiative (39–48). These modules have been used already for long to monitor the quality of premix and fortified foods. Indicator #1 is used to verify that the premix complies with NS. It can be used by the industry to check the quality of premix before fortifying the food vehicles of interest (39–42). Indicator #2 tests whether the fortified food vehicles comply with the NS based on the random selection of 3–5 composite samples within a given month, to be repeated 1–2 times per year. It can be used to test the average content of micronutrient indicator/s of imported foods (per brand) at the site of import (47), and of locally produced foods (per brand) at the site of production by the industry (39–42) as well as by enforcement authorities (43–46). Monitoring compliance of premix and fortified foods with NS is crucial for ensuring that the fortified foods have the expected quality (1, 58).

Indicators #3–6 were extracted from the FACT (19, 51, 52). The FACT is a comprehensive toolkit designed to undertake thorough evaluations of LSFF programs. It assesses the availability of fortified foods and their quality at the market and/or household level; coverage, consumption, and the contribution of fortified staples and condiments (and targeted foods to specific population subgroups) to micronutrient intake (52). We only extracted those indicators from the FACT that assess the availability and quality of fortified foods at the market level (indicators #3 and #4, respectively), the availability of fortifiable food vehicles (i.e., centrally produced by the industry and thus amenable for fortification), and the consumption of fortified foods at the household level (indicators #5 and #6, respectively). The latter indicators are particularly useful to assess the actual reach of the LSFF program, and the potential reach and impact of the program if all the fortifiable food vehicles would be fortified. Hence, these indicators are crucial for decision-making on whether scaling fortification of specific food vehicles could help to close nutrient gaps at the population level.

Indicators #7–9 were recommended by a senior expert on LSFF who participated in the SSI. These indicators estimate the consumption of fortified foods (indicator #7), the reduction in the proportion of the target population at risk of micronutrient inadequacy due to the consumption of fortified foods (indicator #8), and the proportion of the population per strata of interest that remains at risk of micronutrient inadequacy despite consuming fortified foods (indicator #9). These indicators can be constructed with a mathematical modelling framework using HCES data and the FAO adult (fe)male equivalent formula to estimate apparent food consumption at the household level (53, 59). This methodology has already been applied to evaluate the contribution of fortified foods to micronutrient intake of women of reproductive age in Malawi (57), although some caution should be exercised when using HCES data for LSFF program design (60). HCES data can be used to model the potential contribution of fortified food vehicles, e.g., oil, flour, or sugar, to closing micronutrient gaps in the general population as well as in vulnerable population subgroups. Different fortification scenarios can be modeled, for instance, the contribution of a single fortified food vehicle vs. the combination of two or more food vehicles for reducing micronutrient inadequacies in the target groups, which elicits crucial information for program decision-making (57). Although conducting this kind of assessment also requires technical expertise, the analysis can be performed with secondary data sources, which simplifies the evaluation process and reduces costs.

3.4. Exploratory study on the delivery of LSFF programs and their M&E frameworks in Nigeria

Four primary themes, aligned with the study aims, emerged from the 11 key informant interviews in Nigeria for theory building, namely: (1) Delivery models of LSFF programs in Nigeria; (2) Perceived barriers to the implementation and M&E of LSFF programs; (3) Perceived opportunities to improve M&E of LSFF programs at the outcome and impact level; and (4) Perceptions toward the proposed set of core IMMT (Table 5).

3.4.1. Delivery framework of LSFF programs in Nigeria

Nigeria’s LSFF programs are firmly grounded in the broader spectrum of its nutrition policies, with a clear legislative framework. LSFF is implemented as a multi-sectoral effort between government and industries, with support from international NGOs. Two governmental institutes are put in place to enact LSFF programming and to enforce implementation: the Standards Organization of Nigeria (SON), whose mandate is to set and review the industrial standards for fortified foods, as well as monitor compliance at industry level; and the National Agency for Food and Drug Administration and Control (NAFDAC), which is responsible for Good Manufacturing Practices in general, and is expected to monitor food fortification levels at the commercial level (i.e., retail, wholesale, and bakeries). Coordination of the LSFF programs is organized through various structures. The National Advisory Council on Micronutrient deficiency control (or: MNDC Taskforce) is tasked with the coordination of all micronutrient interventions in general, including LSFF. Salt testing for iodine content is specifically coordinated between the Federal Ministry of Health and the Federal Ministry of Education. The Universal Salt Iodization/Iodine Deficiency Disorders Task force is a multistakeholder platform with its secretariat at SON, and coordinates activities specifically to control iodine deficiency. The National Fortification Alliance (NFA), with its secretariat at NAFDAC, is another multistakeholder platform that fosters (mandatory) food fortification, and represents regulators, food producers, and NGOs involved in food fortification. One of our key informants explained that the NFA is the main stakeholder that facilitates SON to conduct audits and inspections in the factories, and NAFDAC to monitor fortification at the commercial level. The NFA also monitors the periodic reporting by SON and NAFDAC. We summarized the descriptions of the delivery of LSFF programs by SSI participants in a ToC framework specifically for Nigeria (Figure 3).

Figure 3

3.4.2. Perceived barriers to the implementation and M&E of LSFF programs in Nigeria

Limited financial and human resources, as well as lack of adequate training for monitoring staff and limited adequate facilities, equipment and consumables to enable staff to perform their duties, were seen as important barriers to M&E of LSFF in Nigeria. Another important barrier mentioned by most interviewed stakeholders was the long-standing trust issues between the monitoring agencies and food manufacturers. A general lack of structure, harmonization, and coordination between stakeholders in monitoring activities was also mentioned, for example resulting in different compliance rates being reported for the same food vehicle. The current M&E system, which consists of deriving coverage data from sporadic spot checks only, both at the industry- and at the market-levels, was perceived to provide unreliable information on fortification quality and coverage of LSFF. The government stakeholders explained their reluctance to take on a more complex approach to monitor product quality because of the costs and the limited capacity available.

Also, the difficulty to monitor the informal market of some fortifiable food vehicles (e.g., sugar and vegetable oil, often sold in small, unmarked containers in informal markets) was mentioned to be a barrier. The participants explained that the informal market makes up a significant portion of the total market and is not regulated, which the larger food manufacturers perceive as an unfair playing field.

“They [the government regulatory agencies] know, so that is part of the inefficiencies, so we see a banned product, that majority of it may not be fortified, being sold everywhere. But the issue is - I stay in my factory every time, and they [SON] come to check on me, and I’ve been wondering what they are doing to the ones that are contraband, or the ones that are informally done and not being fortified.”

                 Executive from the Nigerian food industry

Further, it was mentioned that there is a lack of oversight over the regulatory agencies, which may allow poor practices and/or corruption to occur. Additionally, government officers in charge of monitoring may be reluctant to take actions or report instances of non-compliance because retaliatory episodes of violence have happened in the past (e.g., to a SON agent).

“Now if you have a regulatory officer in that situation that finds somebody that has contravened, maybe it’s a NAFDAC official that went to the market and discovered that thing, and he knows that if he says anything his life is at stake, his office is a public office, anybody can come to the office for revenge […]”

                 Senior Nigerian government officer

Limited infrastructure and technical capacity to collect and analyze data in Nigeria was also mentioned to be a barrier to effective monitoring:

“[…] the 2021 National Food Consumption and Micronutrient Intake Survey, and part of the issue that came up is that we are exporting the food samples and the blood samples that we are collecting, because we don’t have adequate labs to monitor the level of the micronutrients. So, to do a micronutrient survey now, we can’t do it locally. It can’t be done locally, because we can’t assess the level in the sample. Some samples are going to China, some are going to maybe Germany, and I am not sure - maybe some are going to US or South Africa. They are sending samples in different directions, because we don’t have the right infrastructure locally. So, all of this monitoring these things, there is no point collecting samples if there is no one that would […] analyze the samples and say, this is the level of what we are looking for, in the sample. So it is impossible to have an M&E framework without that. […] If we are going to have an effective monitoring system, it has to be affordable. If you are exporting your samples for testing, there is no way your M&E system is affordable.”

                 Academic.

3.4.3. Perceived opportunities to improve M&E of LSFF programs at the outcome and impact levels

Key informants mentioned two mechanisms that can be harnessed for M&E of LSFF programs at the outcome and impact levels. First, the “diet questionnaire” in the National Food Consumption and Micronutrient Survey conducted in 2021 (Federal Government of Nigeria/IITA, 2022), can be used to assess the consumption of fortifiable and fortified foods. According to some of the key informants, at the time of the interview the NFCMS was still finalizing the optimal way of interpreting the results related to fortification and deciding on the best approach to analyze the collected data. The quote below illustrates some of the challenges faced in interpreting the survey results when assessing fortified food intake:

“Are you going to base it on the national standard? That’s the assumption that it is fortified adequately. Or what do you want to base that on? What’s going to be your denominator? And even the numerator? So that one is still a puzzle that we cannot solve. […] But if it is food frequency, you do this: is it fortified, yes or no? Then if you collect samples, fine! You follow up with derived… okay this thing is fortified, if the person mentions the brand, look at fortification status of the product it is adequately fortified, you can say that. If it is not, you know if it’s to an extent, to what extent? One-quarter, two and half, three-quarter; it will complicate the whole thing. […] We are having that issue; I say where we have multiple vehicles with different status of fortification, how are we going to manage that? […] So how do you want to manage within the recipe?

                 Senior government officer

The second mechanism mentioned by some key informants is the periodic impact assessment of the salt iodization program conducted by UNICEF, where subsets of school children are asked to bring salt samples from home to school (cluster sampling design), which are then tested for iodine content.

3.4.4. Perception toward the proposed set of core IMMT

When exploring perceptions, barriers, and enablers for using the indicators as proposed in section 4, most Nigerian key informants found it quite challenging to give their feedback. They focused on identifying barriers, rather than opportunities. Table 6 summarizes the most salient perceptions that emerged for indicators deemed most relevant and applicable to the Nigerian context by the participants.

Indicators #2 (‘Proportion of monthly composite samples with micronutrient indicators meeting NS’), #6 (‘Proportion of households that consumes a fortified food vehicle’), and #8 (‘Reduction in the proportion of the target population at risk of micronutrient inadequacy due to fortified foods’) were all perceived as important; yet, important barriers mentioned were the required resources in terms of technical capacity, infrastructure, equipment, and finances (indicators #2 and #8), as well as the difficulty for consumers to know or remember if they consumed fortified foods (#6). In addition, the time-frame for monitoring of these three indicators was discussed, where our key informants felt that indicator #2 should not be limited by the word ‘monthly’, while for indicators #6 and #8 a timeframe of 5 and 10 years, respectively, was seen as appropriate.

Respondents were uncertain about the definition, usefulness, and applicability of indicators #3 (‘Proportion of food vehicle brands that are fortified to any extent’) and #4 (‘Nutrient content of food vehicle brand’). Specifically with regard to indicator #3, our key informants raised that the wording ‘to any extent’ could provide manufacturers with a free pass for inadequate fortification. For both indicators, it was mentioned that available resources would be a limitation for data collection.

We did not receive any clear feedback on indicators #1 (‘Proportion of premix or fortificant samples with micronutrient content specified in CoA meeting NS’), #5 (‘Proportion of households that consume a fortifiable food vehicle’), #7 (‘Estimated intake of “fortifiable” food vehicles per adult (fe)male equivalent at households’) or #9 (‘proportion of population per stratum that remains at risk of micronutrient inadequacy in spite of consuming fortified foods’).

4. Actionable recommendations

Monitoring of output indicators (indicator #1–4) requires standardized food sampling and measurement procedures. As per the exploratory findings from Nigeria, there is need for support to build local M&E capacity in some LMIC. This may entail the development (or strengthening) of technical capacity for data collection, processing, storage, and reporting; as well as the development (or strengthening) of laboratory capacity, including training of skilled personnel, ensuring good laboratory practices, and the availability of adequate equipment and inputs for food sampling and analysis of micronutrient content. Capacity building and strengthening local entities will facilitate the implementation and continuity of the proposed M&E framework, while disconnecting the system from its reliance on external technical capacities and donor funding.

As also shown here for Nigeria, building trust between government and industry is a crucial factor for the successful implementation of LSFF programs. Cooperation of the industry can be triggered by full involvement in all aspects of LSFF programming and ownership, rather than just providing a regulatory framework and enforcement. Activity and output indicators (indicators #1–4) require standardized sampling and measurement procedures, but also transparent reporting as a means to improve efforts where needed. The introduction of the Micronutrient Fortification Index (MFI)⁵ in Nigeria and other countries, pioneered by Technoserve, may bring positive change in industry compliance. At the same time, enforcement by the government will still be required to guarantee compliance and to level the playing field.

Concerning indicator #2, based on the exploratory interviews with local stakeholders in Nigeria, it should be clarified that ‘monthly’ means ‘in a given month’, rather than ‘every month’. Whereas factories are supposed to continuously monitor their own fortification performance by weekly sending out 3–5 composite samples (i.e., composed of samples collected hourly during an 8-h shift) to a laboratory, the enforcement authority is expected to pay an unannounced inspection visit once or twice per year to cross-check compliance. During such a visit, three randomly chosen composite samples may be selected from the last month’s production to be send to a certified laboratory. The average micronutrient content of the composite samples will be taken as approximation of the micronutrient content in a fortified food vehicle at that particular production site and be compared to NS. Regarding the suggestions of the local key informants to rephrase indicator #3 by replacing the words ‘fortified to any extent’ with ‘adequately fortified,’ we would like to emphasize that this indicator is meant to track coverage at the commercial level, and not to monitor compliance with NS at the factory level. Lastly, the concern of respondents related to indicator #6 that consumers may not know whether they are consuming fortified foods: this indicator is constructed based on information on consumption of food vehicle brands; hence, consumers do not need to reveal any direct information on fortification because this will be captured by indicator #3 and #4.

For monitoring of outcome and impact indicators (indicators #5–9), information on dietary intake is required. We encountered divergent views on the use of 24hR methods versus FFQs to capture the contribution of fortified foods to the daily dietary intake of consumers. Conducting 24hR is laborious and requires advanced technical expertise, while FFQ’s are easier to administer but require rigorous design and thorough validation before they can be used for a specific purpose. Although conducting 24hR at national scale has its challenges, they provide good quality data and are an important component of periodic nutrition and dietary surveillance systems. If such a surveillance system is in place, it can also be used as a design and monitoring tool for LSFF programs, as shown by a case study in Cameroon (61–63). In the future, collection of 24hR and post-collection data processing may be simplified by technological solutions (64, 65). However, consumption data at the household level, such as collected for HCES, provides sufficient and more cost-efficient information to monitor nutrient intake from fortified foods at the (adult) population level within an acceptable error margin, while allowing for subgroup analyses, and may therefore be preferred (59, 63, 64).

Periodic surveys (e.g., with intervals of 5 years) can ensure regular and sustainable data collection on the coverage and consumption of fortified foods, and their contribution to diet quality. For mature programs with high coverage, integration of indicators #5–9 into modules of existing representative national data collection systems, such as National Nutrition and Health Surveys, HCES, or DHS, will be the most cost-efficient means for routine M&E of LSFF programs. Such surveys are already implemented in many LMIC with typical update frequencies of once per 3–10 years, which was regarded as sufficient as indicated by the experts and the local key informants. Further work is required to determine the feasibility of incorporating data collection for the recommended indicators into such existing surveillance systems. When larger-scale national surveys are not available or when a new LSFF program is introduced, sentinel site surveys can also be useful. Sentinel site-based monitoring systems can generate timely actionable information to improve program implementation at a relatively low cost with proven effectiveness (36, 38, 66). The government of Costa Rica, for example, uses such surveys in strategic geographic circumscriptions to track changes in the prevalence of iron deficiency and anemia attributable to their LSFF programs (36). As reflected by the selected indicators, it was not deemed required to assess biochemical micronutrient status based on blood samples for M&E of LSFF programs. Collection of such data is costly and unlikely to be specific for the impact of food fortification. Nevertheless, monitoring of micronutrient deficiencies at the population level is an important component of periodic dietary and nutrition surveillance in general.

Previously, we have compiled a ToC framework with IMMT for biofortification programs, using very similar methodology as described here (9). Initially, we aimed to develop one ToC framework that would capture both biofortification and LSFF programs. However, early on in the process we discovered this to be cumbersome, because of the distinct differences in the implementation pathways between biofortification and LSFF programs. This mainly concerns differences in the type of stakeholders, where biofortification is a largely agriculture-based program, involving farmers, seed companies, and agricultural extension workers, while LSFF programs mainly involve producers of premixes and food manufacturers. Also, there are currently no national standards for biofortification programs, and biofortified foods have not yet penetrated markets at large scale. For these reasons, we decided to separate M&E frameworks and the selection of IMMT for biofortification and LSFF programs. Nevertheless, there is overlap between the frameworks when it comes to evaluating their effectiveness. Hence, it may be possible to integrate these two frameworks at the program outcome level (evaluation), while allowing them to differ at the input, activity and output level (monitoring).

We recommend further testing and revising the presented generic M&E framework with the nine indicators across LSFF programs within and between LMIC and implementing institutions, for subsequent harmonization of a global M&E framework for LSFF programs. A harmonized M&E framework will contribute to keeping track of fortification progress globally, exert positive pressure for effective LSFF implementation and advance the sharing of knowledge.

5. Discussion and conclusion

Based on this multi-method iterative study findings, we present here nine core indicators with their metrics, methods, and tools to be used for M&E of the effectiveness of LSFF programs in LMIC. The main strengths of the work presented here include (1) the use of a generic ToC framework for LSFF programming grounded in the available literature base, triangulated with tried-and-tested impact pathways of various LSFF programs implemented globally, and enriched with the perspectives of international experts; (2) identification of the most pertinent indicators used by LSFF programs; (3) selection of a core set of indicators with metrics, methods and tools, recommended for harmonization of M&E across LSFF programs and geographies; and (4) cross-checking of the compatibility of the generic ToC frameworks with their indicators in a local setting. The number of selected indicators has been kept to a minimum to facilitate efficient M&E practices in terms of cost, time and effort. We believe that this proposed set of core indicators provides the requisite information required for adaptive program management and results-oriented decision-making.

In the literature review, we limited ourselves to published reviews, assuming that they reflect the documented evidence to date. Also, our search strategy included literature published until 2020, thereby not considering potentially relevant literature published more recently. However, in an additional quick-scan of the literature, we did not find any reviews published up till August 2022 that would fulfill our inclusion criteria. Moreover, the grey literature was the main source of the core indicators finally selected. The discourse of the international experts aligned well with the key elements of a successful LSFF as described in our initial impact pathway (Supplementary Figure S1): (1) the selection of a food vehicle frequently consumed in sufficient quantities by large proportions of the population (in particular by vulnerable groups), so that fortification can make a meaningful contribution to micronutrient intake; (2) centralized production in a few large and well-establish industries; (3) sound internal QA/QC procedures mainstreamed into existing QA/QC protocols (i.e., not creating new QA/QC programs); (4) implementation of a simple and cost-effective enforcement system; and (5) effective, simplified, and sustainable assessment of coverage and consumption of fortified foods using HCES. However, the M&E framework proposed in this report has not yet been tested in context-specific programs, since we only explored perceptions towards the M&E framework, and particularly to the indicators, among program implementers in Nigeria and not in other settings. The relatively small number and diversity of key informants for LSFF programs from Nigeria (n = 11) was a limitation in this study, resulting in generalized perceptions that may not be shared by other program implementers in other contexts. Since participants represented different sectors and referred to different food vehicles, we were not able to extend our analysis to characterize specific LSFF programs, for example those of flour, oil, and salt. Also, some of the participants’ perceptions were based on a lack of familiarity with the indicators we presented to them and thus some responses lacked richness.

In conclusion, in this formative study, we formulated recommendations towards core indicators for comprehensive M&E of the performance and effectiveness of LSFF programs in LMIC. This work has resulted in a proposed set of nine core indicators and associated metrics, methods, and tools. This proposed set of core indicators can be used for further evaluation, harmonization, and integration in national and international protocols for M&E of LSFF program effectiveness.

Funding

Funding for the research was provided by Wageningen University & Research, as recipient of a Food Fortification Service (2FAS) grant from the European Union, Landell Mills, and the Global Alliance for Improved Nutrition (GAIN).

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

• 1.

  AllenLde BenoistBDaryOHurrellR. Guidelines on food fortification with micronutrients. Geneva, Switzerland: World Health Organization & Food and Agricultural Organization of the United Nations (2006). 341 p.

  • Google Scholar

• 2.

  KeatsECNeufeldLMGarrettGSMbuyaMNNBhuttaZA. Improved micronutrient status and health outcomes in low-and middle-income countries following large-scale fortification: evidence from a systematic review and meta-analysis. Am J Clin Nutr. (2019) 109:1696–708. doi: 10.1093/ajcn/nqz023

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3.

  DasJKSalamRAKumarRBhuttaZA. Micronutrient fortification of food and its impact on woman and child health: a systematic review. Syst Rev. (2013) 2:1–24. doi: 10.1186/2046-4053-2-67

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4.

  OsendarpSJMMartinezHGarrettGSNeufeldLMDe-RegilLMVossenaarMet al. Large-scale food fortification and biofortification in low- and middle-income countries: A review of programs, trends, challenges, and evidence gaps. Food Nutr Bull. (2018) 39:315–31. doi: 10.1177/0379572118774229

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5.

  MkambulaPMbuyaMNNRoweLASablahMFriesenVMChadhaMet al. Gorstein J. The unfinished agenda for food fortification in low- and middle-income countries: quantifying progress, gaps and potential opportunities. Nutrients. (2020) 12:1–19. doi: 10.3390/nu12020354

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6.

  HennessyÁWaltonJFlynnA. The impact of voluntary food fortification on micronutrient intakes and status in European countries: a review. Proc Nutr Soc. (2013) 72:433–40. doi: 10.1017/S002966511300339X

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 7.

  LuthringerCLRoweLAVossenaarMGarrettGS. Regulatory monitoring of fortified foods: identifying barriers and good practices. Glob Health Sci Pract. (2015) 3:446–61. doi: 10.9745/GHSP-D-15-00171

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8.

  NeufeldLMAaronGJGarrettGSBakerSKDaryOvan AmeringenM. Food fortification for impact: a data-driven approach. Bull World Health Organ. (2016) 94:631–2. doi: 10.2471/BLT.15.164756

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9.

  Rodas-MoyaSGiudiciFMMudyahotoBBirolEKodishSRLachatCet al. Feskens EJM critical review of indicators, metrics, methods, and tools for monitoring and evaluation of biofortification programs at scale. Front Nutr. (2022) 9:963748. doi: 10.3389/fnut.2022.963748

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 10.

  RamerSL. Site-ation pearl growing: methods and librarianship history and theory. J Med Libr Assoc. (2005) 93:397–400. PMID:

  • Pubmed Abstract
  • Google Scholar

• 11.

  World Bank. (2020). Available at:https://blogs.worldbank.org/opendata/new-world-bank-country-classifications-income-level-2020-2021

  • Google Scholar

• 12.

  De-RegilLMPeña-RosasJPFlores-AyalaRdel Socorro JefferdsME. Development and use of the generic WHO/CDC logic model for vitamin and mineral interventions in public health programmes. Public Health Nutr. (2014) 17:634–9. doi: 10.1017/S1368980013000554

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13.

  Oxford Languages. (2022). Available at:https://www.dhsprogram.com/pubs/pdf/FR359/FR359.pdf

  • Google Scholar

• 14.

  CoyneIT. Sampling in qualitative research. Purposeful and theoretical sampling; merging or clear boundaries?J Adv Nurs. (1997) 26:623–30. doi: 10.1046/j.1365-2648.1997.t01-25-00999.x

  • CrossRef
  • Google Scholar

• 15.

  MacDougallCFudgeE. Planning and recruiting the sample for focus groups and in-depth interviews. Qual Health Res. (2001) 11:117–26. doi: 10.1177/104973201129118975

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16.

  PattonMQ. Qualitative research & evaluation methods: integrating theory and practice. 4th ed. Thousand Oaks, CA: SAGE Publications Ltd (2002).

  • Google Scholar

• 17.

  MilesMBHubermanAMSaldanaJ. Qualitative data analysis: a methods sourcebook. 3rd ed. Thousand Oaks, CA: Sage Publications Ltd (2014). 408 p.

  • Google Scholar

• 18.

  AduP. A step-by-step guide to qualitative data coding. 1st ed. New York: Routledge (2019). 444 p.

  • Google Scholar

• 19.

  Global Alliance for Improved Nutrition (GAIN). Fortification assessment coverage toolkit (FACT). (2019). Available at:https://www.gainhealth.org/resources/reports-and-publications/fortification-assessment-coverage-toolkit-fact

  • Google Scholar

• 20.

  MartorellRde RomañaDL. Components of successful staple food fortification programs: lessons from Latin America. Food Nutr Bull. (2017) 38:384–404. doi: 10.1177/0379572117707890

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21.

  A2Z Project. Monitoring and Evaluating Food Fortification Programs: General overview, technical consultation July 7, 2006. (2008). Available at:https://www.spring-nutrition.org/sites/default/files/a2z_materials/508-food-fortification-me-july-2006-consultation-aed-final.pdf

  • Google Scholar

• 22.

  National Bureau of Statistics. National nutrition and health survey (NNHS) 2018. Abuja, Nigeria: Nigeria Federal Ministry of Health (2018).

  • Google Scholar

• 23.

  National Population Commission (NPC) / ICF. Nigeria demographic and health survey 2018. Abuja, Nigeria and Rockville, Maryland, USA; (2019) Available at:https://www.dhsprogram.com/pubs/pdf/FR359/FR359.pdf

  • Google Scholar

• 24.

  OgunmoyelaOAAdekoyeniOAminuFUmunnaLO. A critical evaluation of survey results of vitamin A and Fe levels in the mandatory fortified food vehicles and some selected processed foods in Nigeria. Nigerian Food J. (2013) 31:52–62. doi: 10.1016/S0189-7241(15)30077-1

  • CrossRef
  • Google Scholar

• 25.

  AaronGJFriesenVMJungjohannSGarrettGSNeufeldLMMyattM. Coverage of large-scale food fortification of edible oil, wheat flour, and maize flour varies greatly by vehicle and country but is consistently lower among the most vulnerable: results from coverage surveys in 8 countries. J Nutr. (2017) 147:984S–94S. doi: 10.3945/jn.116.245753

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 26.

  AnjorinOOkpalaOAdeyemiO. Coordinating Nigeria’s micronutrient deficiency control programs is necessary to prevent deficiencies and toxicity risks. Ann N Y Acad Sci. (2019) 1446:153–69. doi: 10.1111/nyas.14055

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  CodlingKQuangNVPhongLPhuongDHQuangNDBéginF. Mathisen R the rise and fall of universal salt iodization in Vietnam: lessons learned for designing sustainable food fortification programs with a public health impact. Food Nutr Bull. (2015) 36:441–54. doi: 10.1177/0379572115616039

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  Garcia-CasalMNPeña-RosasJPMcleanMDe-RegilLMZamoraGAguileraYet al. Fortification of condiments with micronutrients in public health: from proof of concept to scaling up. Ann N Y Acad Sci. (2016) 1379:38–47. doi: 10.1111/nyas.13185

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 29.

  SoekirmanJI. Food fortification in Indonesia. Mal. J Nutr. (2017) 23:1–7.

  • Google Scholar

• 30.

  Ramírez-LuzuriagaMJLarsonLMMannarVMartorellR. Impact of double-fortified salt with iron and iodine on hemoglobin, anemia, and iron deficiency anemia: A systematic review and meta-analysis. Adv Nutr. (2018) 9:207–18. doi: 10.1093/advances/nmy008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  TanumihardjoSAMokhtarNHaskellMJBrownKH. Assessing the safety of vitamin A delivered through large-scale intervention programs. Food Nutr Bull. (2016) 37:S63–74. doi: 10.1177/0379572116630480

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32.

  VictoraCGBarrosFCAssunçãoMCRestrepo-MendézMCMatijasevichAMartorellR. Scaling up maternal nutrition programs to improve birth outcomes: A review of implementation issues. Food Nutr Bull. (2012) 33:S6–S26. doi: 10.1177/15648265120332S102

  • CrossRef
  • Google Scholar

• 33.

  WirthJLaillouARohnerFNorthrop-ClewesCAMacdonaldBMoench-PfannerR. Lessons learned from national food fortification projects: experiences from Morocco, Uzbekistan, and Vietnam. Food Nutr Bull. (2012) 33:S281–92. doi: 10.1177/15648265120334S304

  • CrossRef
  • Google Scholar

• 34.

  NyumuahROHoangTCCAmoafulEFAgbleRMeyerMWirthJPet al. Implementing large-scale food fortification in Ghana: lessons learned. Food Nutr Bull. (2012) 33:S293–300. doi: 10.1177/15648265120334S305

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35.

  DijkhuizenMAWieringaFTSoekarjoDVanKTLaillouA. Legal framework for food fortification: examples from Vietnam and Indonesia. Food Nutr Bull. (2013) 34:S112–23. doi: 10.1177/15648265130342S113

  • CrossRef
  • Google Scholar

• 36.

  MartorellRAscencioMTacsanLAlfaroTYoungMFAddoOYet al. Flores-Ayala R effectiveness evaluation of the food fortification program of Costa Rica: impact on anemia prevalence and hemoglobin concentrations in women and children. Am J Clin Nutr. (2015) 101:210–7. doi: 10.3945/ajcn.114.097709

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 37.

  WirthJPNicholsEMasdHBarhamRJohnsonQWSerdulaM. External mill monitoring of wheat flour fortification programs: an approach for program managers using experiences from Jordan. Nutrients. (2013) 5:4741–59. doi: 10.3390/nu5114741

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 38.

  Peña-RosasJPParvantaIvan der HaarFChapelTJ. Monitoring and evaluation in flour fortification programs: design and implementation considerations. Nutr Rev. (2008) 66:148–62. doi: 10.1111/j.1753-4887.2008.00019.x

  • CrossRef
  • Google Scholar

• 39.

  MakhumulaPGuamuchMDaryO. Manual for internal monitoring of fortified wheat flour. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 40.

  MakhumulaPGuamuchMDaryO. Manual for internal monitoring of salt fortified with iodine. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 41.

  MakhumulaPGuamuchMDaryO. Manual for internal monitoring of oil with vitamin A. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007)

  • Google Scholar

• 42.

  MakhumulaPGuamuchMDaryO. Manual for internal monitoring of sugar fortified with vitamin A. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 43.

  MakhumulaPGuamuchMDaryO. Manual for external monitoring of fortified wheat flour. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 44.

  MakhumulaPGuamuchMDaryO. Manual for external monitoring of fortified salt. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 45.

  MakhumulaPGuamuchMDaryO. Manual for external monitoring of oil with vitamin A. Arusha, Tanzania: East Central and Southern Health odelled (ECSA-HC) (2007).

  • Google Scholar

• 46.

  MakhumulaPGuamuchMDaryO. Manual for external monitoring of sugar fortification with vitamin A at fortification sites. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 47.

  MakhumulaPGuamuchMAfidraRDaryO. Manual for inspection of fortified foods at importation sites. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 48.

  MakhumulaPGuamuchMDaryO. Manual for commercial inspection of fortified foods [internet]. Arusha, Tanzania: East Central and Southern Africa Health Community (ECSA-HC) (2007).

  • Google Scholar

• 49.

  Pan American Health Organization (PAHO). Code of practice for food premix operations. Washington, DC: Nutrition Unit (FCH/NU), PAHO (2005).

  • Google Scholar

• 50.

  DaryOMoraJ. Food fortification: technological aspects In: CaballeroB, editor. Encyclopedia of human nutrition. 3rd ed. Waltham, MA: Academic Press Inc. (2013). 306–14.

  • Google Scholar

• 51.

  FriesenVMAaronGJMyattMNeufeldLM. Assessing coverage of population-based and targeted fortification programs with the use of the fortification assessment coverage toolkit (FACT): background, toolkit development, and supplement overview. J Nutr. (2017) 147:981S–3S. doi: 10.3945/jn.116.242842

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52.

  FriesenVMbuyaMNeufeldL. Fortification assessment coverage toolkit (FACT): development of a methodology for assessing population coverage of fortification programs (P24-043-19). Curr Dev Nutr. (2019):3. doi: 10.1093/cdn/nzz044.P24-043-19

  • CrossRef
  • Google Scholar

• 53.

  WeisellRDopM. The adult male equivalent concept and its application to household consumption and expenditures surveys (HCES). Food Nutr Bull. (2012) 33:S157–62. doi: 10.1177/15648265120333S203

  • CrossRef
  • Google Scholar

• 54.

  FAO and FHI 360. Minimum dietary diversity for women: a guide to measurement. Rome: FAO/USAID/FANTA-III (2016).

  • Google Scholar

• 55.

  BromageSBatisCBhupathirajuSNFawziWWFungTTLiYet al. Willett WC development and validation of a novel food-based global diet quality score (GDQS). J Nutr. (2021) 151:75S–92S. doi: 10.1093/jn/nxab244

  • CrossRef
  • Google Scholar

• 56.

  MoursiMBromageSFungTTIsanakaSMatsuzakiMBatisCet al. Deitchler M There’s an app for that: development of an application to operationalize the global diet quality score. J Nutr. (2021) 151:176S–84S. doi: 10.1093/jn/nxab196

  • CrossRef
  • Google Scholar

• 57.

  TangKAdamsKPFergusonELWoldtMKalimbiraAALikosweBet al. Modeling food fortification contributions to micronutrient requirements in Malawi using household consumption and expenditure surveys. Ann N Y Acad Sci. (2022) 1508:105–22. doi: 10.1111/nyas.14697

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58.

  NeufeldLMBakerSGarrettGSHaddadL. Coverage and utilization in food fortification programs: critwical and neglected areas of evaluation. J Nutr. (2017) 147:1015S–9S. doi: 10.3945/jn.116.246157

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 59.

  CoatesJRogersBLBlauALauerJRobaA. Filling a dietary data gap? Validation of the adult male equivalent method of estimating individual nutrient intakes from household-level data in Ethiopia and Bangladesh. Food Policy. (2017) 72:27–42. doi: 10.1016/j.foodpol.2017.08.010

  • CrossRef
  • Google Scholar

• 60.

  AdamsKPVostiSAMbuyaMNNFriesenVMEngle-StoneR. Update on analytical methods and research gaps in the use of household consumption and expenditure survey data to inform the Design of Food-Fortification Programs. Adv Nutr. (2022) 13:953–69. doi: 10.1093/advances/nmac021

  • CrossRef
  • Google Scholar

• 61.

  Engle-StoneRNdjebayiAONankapMBrownKH. Consumption of potentially fortifiable foods by women and young children varies by ecological zone and socio-economic status in Cameroon. J Nutr. (2012) 142:555–65. doi: 10.3945/jn.111.148783

  • CrossRef
  • Google Scholar

• 62.

  Engle-StoneRNankapMNdjebayiAOVostiSABrownKH. Estimating the effective coverage of programs to control vitamin A deficiency and its consequences among women and Young children in Cameroon. Food Nutr Bull. (2015) 36:S149–71. doi: 10.1177/0379572115595888

  • CrossRef
  • Google Scholar

• 63.

  Engle-StoneRNankapMNdjebayiAOBrownKH. Simulations based on representative 24-h recall data predict region-specific differences in adequacy of vitamin A intake among Cameroonian women and young children following large-scale fortification of vegetable oil and other potential food vehicles. J Nutr. (2014) 144:1826–34. doi: 10.3945/jn.114.195354

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 64.

  CoatesJCColaiezziBABellWCharrondiereURLeclercqC. Overcoming dietary assessment challenges in low-income countries: technological solutions proposed by the international dietary data expansion (INDDEX) project. Nutrients. (2017) 9:1–15. doi: 10.3390/nu9030289

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 65.

  NguyenPHTranLMHoangNTTrươngDTTTranTHTHuynhPNet al. Gelli A relative validity of a mobile AI-technology-assisted dietary assessment in adolescent females in Vietnam. Am J Clin Nutr. (2022) 116:992–1001. doi: 10.1093/ajcn/nqac216

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 66.

  LevinsonFJHerforthA. A collaborative approach to monitoring and evaluating the impact of agricultural projects on nutrition. World Nutrition. (2022) 13:34–46. doi: 10.26596/wn.202213234-46

  • CrossRef
  • Google Scholar