In India, local health systems at the district level are generally organized in three tiers. The first point of contact between people and healthcare professionals is a network of Primary Health Centers (PHC), which depend on a hospital at the sub-district level (taluka) for secondary care. A district hospital is supposed to provide tertiary specialty care. On average, a district in India has a population ranging from 500,000 to a few million people. A sub-district (called taluka in Karnataka) would generally have a few hundred thousand inhabitants. The production of healthcare outputs of hospitals and health centers is complex; it is influenced by the internal dynamics of the workforce in these organizations, by the relationships among individual health workers, as well as by their external environment (25). Environmental factors include local socio-political, governance, and policy influences from the higher levels in the system. An overview of the organization and management of health services in Karnataka state and the policy context within which the intervention operated is summarized in File S2 in Supplementary Material.

We developed the program theory in a step-wise fashion based on guidance in literature (16, 17, 26). We first summarized prior theory and research on the subject, then collected data on contextual factors that could affect the expected outcomes of the intervention, and finally formulated the implicit theory of the intervention. Based on this program theory, we eventually developed CMO configurations helping us to understand how the intervention could have worked, for whom, and under what conditions. A realist evaluation of health management interventions builds upon such CMO configurations in order to generate plausible explanations on how this intervention could have worked (24, 27).

The steps followed are described below and summarized in Table 1. (1)

Understand the intervention (initial program theory, IPT): we started with eliciting the IPT of the intervention. These are the assumptions and hypotheses of the designers of the intervention and other stakeholders. To do so, we reviewed program documents (list of documents in File S3 in Supplementary Material) to identify the implementers’ main assumptions, to understand the perceptions of the key actors and to identify potential mechanisms – if any – as identified by the implementers. Implementers are the people and organizations who designed and are in charge of the implementation of the intervention. At this stage, we were looking for assumptions of the designers on how and why the program would bring about the expected outcomes. In a second step, we interviewed 16 actors using an interview guide (File S4 in Supplementary Material): 2 program designers, 2 policymakers, 10 participants (health managers), and 2 health services staff during the early phase of the implementation of the intervention. The interviews focused on the process of planning health services, the perceived scope for change given the current decentralization process, and the possible role of the intervention in this change. The program documents and interview transcripts were imported into NVivo 10 (QSR International Pvt. Ltd., Australia). Portions of text reflecting implementation assumptions, possible contextual factors (see step 3 below), and actors’ perceptions were coded and analyzed using thematic analysis. These themes were then summarized and the IPT was gradually constructed.

We used the multipolar performance framework to analyze and discuss the refined program theory (28). This framework is an adaptation of a conceptual framework designed for analyzing the performance of healthcare organizations proposed by Sicotte et al. (29). It integrates several theories on organizational change and has been applied to explain healthcare organizations’ performance in LMICs settings (1, 28–30).

The IPT is readily available – see the publication of the study protocol (15) – reproduced in Figure 2. The IPT is therein schematized as a linear representation of the intervention’s inputs (contact classes and mentoring of participants) connected to the intervention’s expected outcomes (improved annual action plans and supervision practices) through a set of intermediate steps (better problem and solution identification, better monitoring and more supportive supervision). The IPT considered the posting of non-medical management professionals at district and sub-district levels [as part National Rural Health Mission (NRHM) – see below for description of the NRHM] as a significant contextual factor, expected to influence the intervention’s outcomes. Based on this IPT, the intervention could be formulated as a human resources management intervention consisting of an in-service training and mentoring program to bring about organizational change in district health management, through improved preparation and implementation of annual action plans and supportive supervision. The IPT was represented as a logic model. However, from a realist evaluation perspective, it requires further elaboration by making explicit the assumptions of the program designers on the possible intermediary steps and by taking into consideration the contextual factors that could affect the implementation of the intervention. Also, potential mechanisms of change need to be identified.

Based on documentary review and thematic analysis of the interviews with the designers of the intervention, we identified three key assumptions that a priori guided the design and implementation of the intervention. These assumptions have had implications on how the program was structured (e.g., role of experienced teachers visiting the participants as mentors, whom to include in the program), its content (e.g., what type of content to include and how to present them), and the implementation (e.g., focus on improving the district and taluka health system and focusing on health management teams at these levels). These assumptions are the following:

Human resource management interventions can be important drivers of health service provision and thus good health outcomes. However, studies on HRM in health are few; they focus mainly on continuing education, supervision, payment of incentives, decentralization of HRM functions, or a combination of these (7, 34). An understanding of the mechanisms through which HRM interventions produce change in healthcare institutions is crucial for the design and delivery of such interventions in LMICs settings. However, the current evidence base for how positive organizational change could be achieved through capacity-building based HRM interventions in health services is scarce (7, 34, 35). Both systematic and realist reviews of studies in human resources for health note that the role of context in producing desirable outcomes in HRM interventions is under-explored; either contextual factors are neglected in designing effectiveness evaluations, or context descriptions are scarce, rendering the studies not amenable to realist reviews (7, 22).

Contrary to the evidence base on HRM in the public health literature, much has been written on the topic in management sciences, particularly in the corporate business industry (36). In a review focusing on evidence on achieving and maintaining good performance of health workers in LMICs, Rowe et al. (34) identify eight theories underlying most HRM interventions in health. These theories explain organizational improvement through change in health worker behavior and practices, which they place across several levels: the team, the institution, and the larger health system environment within which they work. The theories reviewed are summarized in Figure 4. They could be thought of as providing explanations of change seen at individual, institutional, and systems levels; however, Rowe et al. note that: “… little is known about how well the theories predict health worker practices or success of interventions.” Furthermore, most studies included in their review concern healthcare workers. Studies pertaining to local health system managers, mostly doctors in the case of Indian districts, are scarce.

Capacity-building programs inject new resources, i.e., knowledge, skills, and experiences, in organizational settings. A variety of individual, institutional, and environmental factors determine who benefits from such programs and who does not, and who applies what they learnt in terms of organizational change, and who does not. Among the frameworks proposed to evaluate the effect of training programs, the Kirkpatrick and Kirkpatrick framework is one of the most frequently used (37). The framework proposes four levels at which the training programs could be evaluated: reaction (to the training), learning (knowledge and skills), behavior (applying the new learning), and impact (changes brought in the organization). In a critical analysis of evaluation practice, Bates (38) summarizes the common assumption (in literature) of causal linkages between these four levels as follows: the four “… levels of criteria represent a causal chain such that positive reactions lead to greater learning, which produces greater transfer and subsequently more positive organizational results.” Bates further notes that several training evaluation studies and meta-analyses have failed to confirm such a linear causal pathway connecting training program inputs to outcomes through these four levels (38). Although the four levels of the Kirkpatrick and Kirkpatrick framework therefore cannot be assumed to represent an incremental four-level causal pathway, the framework provides possible sequential steps that an individual trainee might experience during and after training programs. It helps us by indicating where to look for contextual factors that could affect individual learning and its application within the participants’ organization.

One of the key components of the IPT of the implementers was the intent to bring about a can-do attitude among the participants. The implementers expected that mentoring by experienced public health professionals at the workplace of the participant would bring about such an attitudinal change. The initial program documents, however, did not elaborate on how the implementers expect such a change to take place in the individual participants. We had hypothesized that, in addition to organizational and environmental factors, such a positive organizational change could be linked to individual attributes of the participant, like the organizational commitment of the individual and the confidence that the individual places in his ability to produce such a change (15). The latter is related to perceived self-efficacy, identified as a mechanism that explains why some people feel able to take up some tasks while others do not, in spite of similar knowledge and skill levels (39). Similarly, organizational commitment and performance of an individual are closely related, as shown in several industrial and healthcare organizational settings (40–43).

In a local health system, where individual health managers work in small teams within organizations belonging to a broader network of healthcare institutions, the dynamic nature of the interactions at individual, team, institutional, and broader environmental levels contributes to whether participants apply what they learn and whether the expected organizational change manifests or not. Organizational frameworks therefore incorporate the role of such factors when analyzing healthcare organizational performance (29, 38). Workplace environment, nature of teams and teamwork, supervision received, attitude of state-level officials, and the needs and demands of the communities are all important factors that can affect organizational change after capacity-building. The various sub-units of a district health system, their interactions and influence on organizational performance are visualized in Figure 1.

Workplace environment in healthcare organizations has been identified as an important element explaining the application of learning from training programs in some settings, while not in others (44). In the conceptual framework of workplace learning proposed by Jacobs and Park, the inter-relationships between location of the training (learning occurs away from workplace), degree of planning (use of a systems approach), and an active role of the trainer were key variables in understanding workplace learning (45). Although not specific to healthcare organizations, this framework identifies important elements for developing the program theory of our intervention.

While the theoretical frameworks provide plausible pathways through which the intervention inputs and outcome could be related, for the outcomes to effectively occur, local conditions matter. From a realist evaluation perspective, these are contextual factors that facilitate (or hinder) the outcome – they are crucial in refining the program theory. Contextual factors have been shown to influence organizational change in healthcare settings (46–52). While it is unlikely that each and every possible contextual factor will be identified, a documentary and literature review may help identify the most important and more obvious ones, especially if a plausible causal chain can be used as an anchoring point. We also used the mapping of key events (Figure 5) as a guide to identify relevant contextual factors.

We identified three key themes from the contextual analysis: the local effects of the ongoing decentralization of planning to district level in India, the acceptance and role of newly introduced non-medical program managers, and the local actors’ conceptualization of the district health services as a system. Below, we discuss these in more detail and analyze implications for our evaluation.

Capacity-building of district health managers and its contribution to organizational change is influenced by relationships between actors and among the components of the district health system. In Figure 3, we represented the refined program theory of the intervention. As described above, we did this by critically examining the IPT and the assumptions of the implementers, in relation to existing literature on capacity-building and by drawing from a description of the most important aspects pertaining to the local context.

The program theory enables us to formulate a number of CMO configurations that can be subsequently used in guiding the analysis of data collected in course of the intervention (see Figure 7; Table 2). The CMO framework provides a lens through which to analyze empirical cases and build explanations for purposively chosen cases of positive and negative outcomes among the participants and teams. For example, the program theory points toward an important intermediate outcome, the intention to make positive organizational change after a training program. The contextual analysis and review of literature have also indicated important factors – individual mechanisms, institutional, and systemic factors (local context) – which could be mapped on a CMO framework. In this case, the CMO frame would start with positing possible contextual factors and mechanisms that could together bring about an intention to make positive change within a healthcare organization (outcome). We formulated three such CMO formulations based on the refined program theory. These formulations can be tested using a mix of qualitative and quantitative data to explain how positive organizational change can occur in response to such capacity-building programs (Table 2).

A program theory is a way of representing the expected relationships between the elements of an intervention implemented in a given context and its expected outcomes. Programs introduce new resources into a dynamic system. A program theory is a “set of propositions regarding what goes on in the black-box during the transformation of input into output; that is, how, via treatment inputs, a bad situation is transformed into a better one” (64, 65). A PT could be conceptualized as a logical and ordered description of the relationships between the various constitutive elements of an intervention, and the plausible pathways through which they interact with the elements of the system to produce the expected outcome. It draws upon the assumptions that the implementers have made in designing and implementing the intervention. It also incorporates the response of various actors within the system to the intervention and other contextual factors that could influence these actors and their responses to the intervention. A program theory is thus a pathway with interacting elements, showing how the inputs of an intervention could lead to the expected outcomes, taking into consideration contextual elements and the assumptions of the implementers on how they could achieve the objectives of the intervention.

In realist evaluation, Pawson and Tilley posit that programs are embedded in social systems. They stress the importance of understanding what works for whom, and under what conditions (66, 67). The realist evaluation approach focuses on the interaction between the mechanisms activated by the intervention and the context(s) in which it is implemented, specifically seeking to understand how this interaction in the various contexts produces changes that could lead to the outcomes (of interest to the evaluation). It is one of the several context-sensitive approaches to evaluate health programs at district level in low resource settings (23). In keeping with this, the program theory should identify intermediate steps in the pathway connecting the inputs of the intervention to the outcomes, the relationships between the steps and the conditions under which these occur. While existing theories provide plausible explanatory mechanisms through which inputs and outcomes could be related, the systemic factors unrelated to the intervention that could affect the outcome of the intervention (the context) are very important to understand how the intervention worked. Configurations of CMO based on program theory could be seen as plausible explanations of what worked for whom, and under what conditions. CMO configurations will enable us to collect data and test how the change occurred, in addition to whether the change occurred (or not). A critical reformulation of the IPT on the basis of empirical research, taking into consideration the conditions that could affect the outcome (like for instance other initiatives with similar outcomes and/or contextual conditions favorable or hindering the outcome locally) would eventually improve our understanding of the mechanisms of change, as well as enable a scientific evaluation.

A district health system in India consists of a network of government-owned healthcare organizations providing primary, secondary, and tertiary care in addition to private hospitals and providers. A district or a taluka, ideally, would be the meeting point of top-down resource allocation and planning with bottom-up planning driven by local needs and people’s demands. Both dynamics would need to be integrated in a manner that ensures a well-performing health service that is responsive and provides good quality and equitable healthcare.

Local health systems at taluka and district level can be conceptualized as complex adaptive systems, a network of inter-related and inter-dependent organizations, which are relatively similar, yet dynamically interacting with each other and their environment. From a complex adaptive systems perspective, a linear causal logic model cannot be applied to evaluate a local health system intervention. Local contextual factors contribute to determine differences in outcomes, even if the appropriate resources for change are introduced (68). For example, the lack of a supportive learning and working environment in a given taluka hospital will be a barrier to realize the expected outcome, despite the introduction of new knowledge and skills through a training program, while it would have the potential to achieve this outcome in yet another hospital, where such an environment would exist.

A healthcare organization is a dynamic entity constantly interacting with a continuously changing environment through an internal dynamic exchange between its different functions. Based on a synthesis of several, often competing models of organizational performance, Sicotte et al. proposed a conceptual framework for analyzing performance in healthcare organizations. The overall performance of a healthcare organization is seen as being “determined by the dynamic equilibrium resulting from continual interaction of, and interchange among (these) four functions (attributes or properties of an organization)”: attaining the organization’s goals and adaptation to its external environment on one hand and its internal environment (the organizational culture and values) and its production (healthcare outputs) on the other (see Figure 8). This framework has been the basis for designing the multipolar performance framework, a heuristic that has been used to analyze performance of complex public sector healthcare organizations (30).

The pushes and pulls of the four poles result in six alignments, and the resulting performance of a given healthcare organization could be conceptualized as the net result of how the management team deals with the pushes and pulls along these various axes. For example, in the case of our program theory, change proposals introduced by a given taluka management team need to be in line with the district’s vision on one hand (allocation alignment) and the expectations of local actors (PRI members, community, and other actors at the taluka level) on the other (strategic alignment). In such a scenario, capacity-building interventions could operate through triggering changes in alignments, for example, by decreasing tensions in the allocation alignment through better negotiation skills provided to the taluka. The Tumkur capacity-building intervention implemented within the existing policy context of NRHM aimed to improve both allocation alignment (improved allocation of financial, human, and technical resources for planning) and strategic alignment (strengthening role of districts in making their own action plans). However, our program theory shows that in the present Indian district context, improved organizational performance would also need better contextual alignment (perceived decision-spaces of health managers and BPMs) and operational alignment (decreasing the social distance between doctors as leaders of health management teams and their other members, improved teamwork among the members of the team including the young program managers).

Programs for capacity-building of health workers operate at and across several levels – individual, institution, and system – with the positive outcomes, in this case, improved management of health services vary from one institution to another, and across different healthcare delivery teams. In a hospital where the doctor would be able to decrease the perceived social distance between herself and the BPM, there is likely to be a better operational alignment and higher chances of the BPM’s improved knowledge and skills manifesting as improved organizational performance. These plausible explanatory CMO configurations need to be tested using program data to understand which of the many possible organizational change mechanisms is operating in the Indian district setting.