Escherichia coli cell extract was generated using our previously reported CFAI workflow (Levine et al., 2019b). A culture was prepared by inoculating a loopful of BL21^∗ DE3 cells into a 2 L baffled flask containing 1 L of Cell-free Autoinduction media (5.0 g of sodium chloride, 20.0 g of tryptone, 5.0 g of yeast, 14.0 g of potassium phosphate dibasic, 6.0 g of potassium phosphate monobasic, 6.0 mL of glycerol, 4.0 g of D-lactose, 0.5 g of D-glucose, and nanopure water to 1.0 L). The culture was incubated at 30°C and 200 rpm for approximately 15 h. Subsequently, the culture was centrifuged at 4°C and 5,000 g for 10 min. Harvested cells were resuspended in 30 mL of S30 buffer [10 mM Tris OAc, pH 8.2, 14 mM Mg(OAc)₂, 60 mM KOAc, 2 mM DTT] by vortexing, then spun down at 4°C and 5,000 g for 10 min. Supernatant was removed and cell pellets were flash frozen and stored at −80°C or used immediately for extract preparation. Cell pellets were resuspended in 1 mL of S30 buffer per 1 g of cells. 1.4 mL of resuspended cells were aliquoted into a 1.5 mL microfuge tube. The resuspension was sonicated using a Qsonica Q125 Sonicator with a 3.175 mm probe, with the cell resuspension surrounded by an ice water bath. Three pulses of 45 s on and 59 s off, at 50% amplitude were carried out. Immediately after sonication, 4.5 μL of 1.0 M DTT was spiked into the lysate and the tube was inverted several times to mix. Lysate was centrifuged at 4°C and 18,000 g for 10 min. The resulting supernatant is the cell extract. The mixture was flash frozen in liquid nitrogen and stored at −80°C until Genetic Code Kit preparation.

DNA template pJL1-sfGFP was purified from DH5α cells using an Invitrogen PureLink HiPure Plasmid Maxiprep Kit. DNA was eluted using warm molecular biology grade water instead of the provided TE buffer for compatibility with the CFPS system. DNA plasmid was diluted with molecular biology grade water to a concentration of 42.1 ng/μL, such that no additional water was needed to prepare 30 μL CFPS reactions with a final DNA concentration of 16 ng/μL. DNA was stored at −20°C until Genetic Code Kit preparation.

Solution A (containing cofactors and substrates) was prepared with the following specified concentrations of reagents: 8.14 mM ATP, 5.77 mM GTP, 5.77 mM UTP, 5.77 mM CTP, 153.8 mg/mL folinic acid, 771.9 mg/mL tRNA, 2.71 mM NAD, 1.81 mM CoA, 27.1 mM oxalic acid, 6.79 mM putrescine, 10.2 mM spermidine, 386.9 mM HEPES buffer. Solution B (containing the energy system) was prepared with the following specified concentrations of reagents: 71.6 mM magnesium glutamate, 71.6 mM ammonium glutamate, 930.8 mM potassium glutamate, 14.3 mM 20 amino acids, and 238.1 mM phosphoenolpyruvate. All reagents were dissolved in molecular biology grade water. Both solutions were stored at −80°C until Genetic Code Kit preparation, however, these solutions are also stable at −20°C for 3 months (Supplementary Figure 1).

Each kit contained the appropriate amount of pre-aliquoted reagents for the laboratory size and was stored at −20°C for up to 5 days until student use. Each pair of students was provided a strip of four PCR tubes, each containing 10 μL of extract. Each group of four students shared a set of PCR tubes containing molecular biology grade water, pJL1-sfGFP DNA plasmid, solution A, and solution B. Students added 11.4 μL of water, 4.4 μL of solution A, and 4.2 μL of solution B to two tubes as negative controls and 11.4 μL of DNA plasmid, 4.4 μL of solution A, and 4.2 μL of solution B to two tubes as positive controls. All reagents were kept on ice throughout reaction setup. The completed reactions were placed in a 37°C incubator and checked intermittently for green fluorescence. Necessary equipment includes a p20 pipette, pipette tips, an incubator, and a blue or black light. More details can be found in the laboratory manual (Supplementary Data Sheet 1).

The lab manual and worksheet (Supplementary Data Sheets 1, 2) for the Genetic Code Kit were developed with the following student learning objectives as a framework: (A) illustrate and describe the processes of transcription and translation; (B) identify the minimally necessary genetic components, enzymes, and reagents necessary for transcription and translation in vitro; (C) predict and visualize the outcomes of adding, or not adding, various components to CFPS reactions; (D) define CFPS and its advantages over in vivo protein synthesis; (E) paraphrase how energy metabolism sustains transcription and translation in a CFPS reaction. Background on CFPS, the processes of transcription and translation, including the necessary components for each of these processes, and the energy metabolism system operating in CFPS reactions was provided in the lab manual (Supplementary Data Sheet 1).

The student worksheet contained open-ended questions corresponding to each of the learning objectives; some questions also required students to draw a schematic to represent their understanding of a topic (Supplementary Data Sheet 2). For example, for learning objective B, students were asked to illustrate the templates for transcription and translation, including genetic elements like a promoter and ribosomal binding site, and their relative locations to one another on a DNA template. Students were asked to consider the outcome of the experiment if certain elements were missing, such as dNTPs or a particular amino acid, in order to address learning objective C. Questions related to learning objective E focused on steps that require energy input, and how the levels of high-energy molecules like ATP change throughout the CFPS reaction.

The student questionnaire contained 16 content-based questions and 12 attitudinal questions (Supplementary Data Sheet 4). All questions were multiple choice. The content-based section contained three baseline questions that tested knowledge independent of the intervention’s learning objectives and were not expected to be impacted by this laboratory exercise. They acted as a control for differences in baseline aptitudes between the pre- and post- questionnaires. Of the remaining 13 content-based questions, four questions tested transcription knowledge and nine tested translation knowledge. Transcription questions focused on key enzymes and required genetic elements on the DNA template for initiation and termination of transcription. Translation questions were focused on the basic mechanism of the ribosome, including how tRNA and mRNA interact, and the required genetic elements on the mRNA template for initiation and termination of translation. The 12 attitudinal questions asked students to rank their knowledge of transcription and translation vocabulary and comfort with research techniques.

The augmented reality activity utilized Augment¹, a smart phone application, to project the three-dimensional structure of sfGFP onto student benchtops for an exploration of protein structure, structure-function relationships, and the structural basis for fluorescence (Supplementary Data Sheet 3). However, our pre- and post- questionnaire did not assess student understanding of sfGFP structure or structure-function relationships, so the impacts of this activity on student learning cannot be reported here.

The Genetic Code Kit and relevant assessments were implemented in the laboratory component of our non-majors’ “Survey of Biochemistry and Biotechnology” course (CHEM 313) taught by biochemistry faculty. The prerequisite for enrollment was the completion of an introductory organic chemistry course. Our curriculum allows students to select either Organic Chemistry I (CHEM 216), which is the first quarter of a year-long organic chemistry sequence or Survey of Organic Chemistry (CHEM 312), which is a one-quarter survey of organic chemistry (Table 1). The students involved in this study represent a breadth of educational backgrounds, with diverse majors from four colleges at Cal Poly SLO (Table 1). All student data was used with written consent of the participants in the study, based on Institutional Review Board (IRB) approval obtained prior to execution.

Implementation occurred over a 3-week period, with each lab section meeting once a week for 3 h. As a “pre-questionnaire” in week 1, all students completed the questionnaire described above (Supplementary Data Sheet 4). In week 2, students in the intervention group used the Genetic Code Kit in their regularly scheduled lab section (Supplementary Data Sheets 1–3). The control group did not meet and did not perform the experiment or augmented reality activity due to a holiday. However, they were provided with the lab manual and completed the same post-lab worksheet. In week 3, all students repeated the same questionnaire administered in week 1, representing the “post-questionnaire.” A total of 69 students completed both pre- and post- laboratory questionnaires, with 15 in the control group and 54 in the intervention group.

Intervention group students performed the Genetic Code Kit lab module in a single 3 h lab period. They were provided the lab manual at least 3 days prior to performing the experiment. After a brief introduction to the experiment in the lab period, students were asked to follow the instructions for reaction setup described in the lab manual, commencing in vitro transcription and translation. Reaction tubes were then placed in a 37°C mini-incubator for 1–1.5 h (Supplementary Figure 2). During the incubation period, students completed the post-lab worksheet and augmented reality activity (Supplementary Data Sheets 2, 3) and listened to a short lecture from instructors on the basics of transcription and translation. This brief lecture reviewed information on transcription and translation that was also covered in the 4-h per week lecture portion of the course, and introduced the components of each of the solutions in the Genetic Code Kit that correspond to these processes. This information was also available to students in the control group in the form of the introduction in the lab manual, and in the course textbook. At the end of the incubation period, students visualized fluorescence with the naked eye, and enhanced visibility was achieved using a handheld blue light before the lab period was over.

Student responses were collected and all anonymized assessment scores and responses can be found in Supplementary Table 1. Content-based questions were divided into baseline (1, 5, 6) and transcription and translation (2–4, 7–16) categories based on each question topic. Each category was analyzed by comparing the pre- and post- questionnaire scores for the control and intervention groups, visualized via box and whisker plots generated in SigmaPlot. Paired t-tests were run for both groups using JMP, and p-values were recorded with a significance level of 0.05. These categories were also analyzed by calculating the normalized learning gain and effect size for both student groups to understand the magnitude of the effect of the Genetic Code Kit. Normalized gain enables the comparison of groups that start at different levels of performance, as it calculates the score increases with respect to the window of potential learning based on pre-questionnaire scores (Hake, 1998). Effect size provides an additional metric that accounts for the number of students tested and the variation in scores among the students (Cohen, 1988). Question-based normalized gain was calculated to determine student performance on each of the 16 questions individually. This metric uses the same equation as normalized gain, however, the average pre- and post- scores are replaced by the percentage of students who answered the question correctly on the pre- and post- questionnaires. Additionally, the content-based data was matched to student major and previous course completion data in the form of an Excel dashboard that allows the user to analyze trends that occur within these subgroups (Supplementary Table 2). The dashboard also allows for a statistical comparison of the control group relative to the intervention group. Due to the different sample sizes, the comparison was performed using the Fisher’s Z Test. Point biserial analysis was performed using the Akindi software².

Attitudinal questions were analyzed by comparing the trends in the percentage of students that selected each answer choice on the pre- and post- questionnaires. For statistical analysis, student answers were converted to numerical values, where A = 1 and E = 5. Paired t-tests comparing pre- and post- scores for each question were run using JMP and p-values were recorded with a significance level of 0.05.

The content-based section of the questionnaire contained 16 questions (3 baseline, 13 transcription and translation). For baseline questions unrelated to the learning objectives, there was a minimal increase in the mean percentage of correct answers; the control group’s mean score increased from 35.6 to 37.8% and the intervention group’s mean score increased from 34.0 to 41.3% (Figure 2A). However, a two-sided paired t-test showed that neither of these increases were significant (p-value > 0.05). Thus, we concluded that neither group became significantly better at answering the post-questionnaire as a result of previous exposure in the pre-questionnaire. On transcription and translation questions for the control group, we observed minimal increases in the mean score, from 41.5 to 48.7%. Comparatively, the intervention group had a larger increase in the average score on transcription and translation questions, from 49.6 to 63.8% (Figure 2B). One-sided paired t-tests within the control and intervention groups comparing pre- and post- student scores indicated no significant increase (p-value > 0.5) for the control group and a significant increase (p-value < 0.001) for the intervention group. This indicates that completing the hands-on Genetic Code Kit experiment significantly improves students’ ability to correctly answer questions regarding transcription and translation.

In addition to observing improvements in average assessment scores, we also wanted to better understand the magnitude of the effect of the intervention on student learning gains. Toward this goal, we evaluated both normalized learning gains and effect sizes, since both are commonly used metrics in STEM education. The extent of normalized learning gains is categorized as low (gain < 0.3), medium (0.7 > gain ≥ 0.3), and high (gain ≥ 0.7) (Hake, 1998). On baseline questions, the control and intervention groups demonstrated low gains of 0.03 and 0.11, respectively, as expected (Figure 3A). For the transcription and translation questions, the control group demonstrated a normalized gain of 0.12 while the intervention group demonstrated a gain of 0.28. Effect sizes were also calculated as an additional metric to understand the magnitude of learning gains, while accounting for the student sample size and variation. Effect sizes are categorized as small (effect = 0.2), medium (effect = 0.5), and large (effect = 0.8) (Cohen, 1988). For the baseline questions, we observed small effect sizes of 0.07 for the control group and 0.28 for the intervention group (Figure 3B). Effect sizes on the transcription and translation questions were 0.32 for the control (small-medium) and 0.60 for the intervention (medium-large). As with the normalized gain analysis, the intervention group’s ability to correctly answer questions related to transcription and translation after using the Genetic Code Kit module was much greater than the control group, who did not carry out the activity.

Lastly, we analyzed the question-based normalized gains for each of the 16 questions individually (Figure 3C). This analysis was intended to indicate student performance on individual questions, allowing us to identify questions that were poorly designed or not well-addressed by the Genetic Code Kit. The outcome of question-based normalized gain assessment was the identification of questions 7 and 11 as particularly challenging for the intervention group. In fact, the control group outperformed the intervention group on those two questions, and the normalized gain for the intervention group was negative for question 11. Quantitatively, the point-biserial correlation coefficient values for questions 7 and 11 were above 0.2, suggesting that they are “fair” questions. Qualitatively, it is possible that these questions were written ineffectively, were mismatched with our learning objectives, or that CFPS was not able to resolve student misconceptions regarding the macromolecular interactions involved in translation. In fact, non-covalent interactions involved in translation were not explicitly covered in the pre-lab lecture, worksheet, or lab manual.

Given that we observed meaningful normalized learning gains and effect sizes upon intervention despite questions 7 and 11, we remained curious about the learning gains observed in the remaining questions. In a follow-up analysis (Supplementary Tables 2, 3), we removed questions 7 and 11 from the group of transcription and translation questions and used this narrower scope to evaluate learning gains by student demographics. We observed that students who had previously taken Ochem I, the first quarter in a year-long series of organic chemistry, had significantly higher learning gains compared to the control group (p-value < 0.05), while those who had taken Survey of Ochem did not significantly benefit (p-value > 0.05) from the Genetic Code Kit intervention compared to the control group (Supplementary Table 3). The intervention group students that did not significantly benefit were mostly from the College of Agriculture, Food, and Environmental Science, who have historically underperformed in the Survey of Biochemistry and Biotechnology course. While this observation is only suggestive when we removed questions 7 and 11 from the analysis, it represents an intriguing starting point for using CFPS to consider preparation gaps and achievement gaps within our student populations. These results suggest that if question design can be improved and sample size can be increased, implementation of CFPS has the potential to explore the basis for preparation and achievement gaps in biochemical education. Regardless, these additional findings are contingent on solving the learning issues identified in questions 7 and 11, as these differences in prerequisite preparation only appear when they are removed from the analysis.

Overall, significant increases in the average scores on content-based questions (Figure 2), a normalized learning gain around 0.3, and an effect size of 0.6 for the intervention group (Figure 3) indicate that implementing the Genetic Code Kit improved students’ ability to comprehend and answer questions relating to transcription and translation. As no significant increase (p-value > 0.05) in the performance on baseline questions was observed, we propose that the observed increase in assessment scores for transcription and translation questions was a result of the Genetic Code Kit rather than repeated exposure to the questionnaire.

The pre- and post- questionnaires completed by both the control and intervention groups contained a total of 12 attitudinal questions. These questions prompted students to self-assess their recognition and knowledge of transcription and translation vocabulary, as well as their comfort with laboratory techniques used in CFPS. Prior work has documented students’ deficiency in metacognitive skills and found that active learning pedagogies can strengthen these skills (National Research Council, 2000). Our attitudinal-based questions allow us to examine how students’ perceptions of their learning correlate with their results on the content-based assessment (Supplementary Figure 3). We found that both the control and intervention groups showed positive correlations on pre- and post- questionnaires, with an increase in the slope from pre- to post- questionnaire. For the control group, the pre-questionnaire R²-value was 0.02 and post-questionnaire was 0.36. For the intervention, the pre-questionnaire R²-value was 0.10 and post-questionnaire was 0.30. The relatively low pre-questionnaire R² is noteworthy: it shows that students’ knowledge and attitudes are, effectively, uncorrelated. The increase in post-questionnaire R² indicates that knowledge and attitudes move in the same direction. Overall, this analysis indicates that students’ self-reported confidence correlated with their performance on content-based questions. As a result, we pursued more detailed analysis of the attitudinal-based questions.

We first considered the possibility that improvements in students’ self-assessment of their confidence were an outcome of their recognition of vocabulary terms through repeated exposure to the questionnaire rather than as a result of improved conceptual understanding of the terms. To address this concern, we chose to perform detailed per-question analysis for the attitudinal-based assessment on questions that involved comfort with CFPS as an indicator of how beneficial the activity was in introducing a novel biotechnology. For the intervention group, we observed significant increases (p-value < 0.05) between pre- and post- scores for questions 23 and 25–27 using a one-sided paired t-test (Figure 4). When prompted with “I know what CFPS is” (question 23) on the pre-questionnaire, over 50% of the intervention group students indicated that they had no idea what the term meant and ∼11% indicated that they knew what the term meant (Figure 4A). After conducting the experiment, this changed to less than 5% and greater than 50%, respectively. The control group saw a similar, but less extensive shift in the trend with almost 40% of students reporting that they knew what the term meant in the post-questionnaire (Supplementary Figure 4). The comparable shift in the control and intervention groups is likely due to the background information that they received on CFPS through the lab manual alone. For question 25 (Figure 4B), “I am comfortable conducting experiments with enzymes,” pre-questionnaire comfort was generally high for the intervention group, but only ∼11% of students indicated that they “strongly agreed.” However, the Genetic Code Kit increased intervention student confidence in working with enzymes, such that almost 30% of students said they “strongly agreed” on the post-questionnaire. This was a noteworthy observation, since the Genetic Code Kit was implemented at the end of the quarter, and students had worked with enzymes in numerous previous laboratory modules. The intervention group’s comfort with conducting experiments involving in vitro transcription and translation (questions 26 and 27; Figures 4C,D) also showed notable improvement, with the number of students answering “strongly agree” increasing to ∼25% from less than 2%. Comparatively, the control group had less than 8% of students say that they “strongly agreed” in response to questions 25–27 (Supplementary Figure 4). These data indicate that the intervention group’s hands-on exposure to the CFPS reaction improved their comfort with these laboratory skills over the control group.