We administered a paper-based questionnaire (Supplementary Figure 1) to all 10.433 participants of the 2018 TMS in all 48 test centers. Participation in the evaluation study was optional and took place before the actual test. TMS scores of all participants were gathered on the same day.

To match the TMS result with the answers in the voluntary evaluation questionnaire, we only included participants who provided their TMS-participant ID in the questionnaire and gave answers to all relevant questions. The TMS-participant ID was then used to match the TMS result as well as the participants’ sex and age with the results of the voluntary evaluation. The final sample consisted of 7903 participants (75.8% of the overall cohort), of which 30.2% were male. On average, the participants were 19.20 years old (SD = 1.97) and had an average high school GPA of 1.68 (SD = 0.46).

Questions and response scales were based on previous studies (e.g., Deter, 1982; Bartussek et al., 1984; Trost et al., 1998; Briggs, 2009) and were developed in a consensus process between expert groups comprised of test developers and members of the medical faculty of the University of Heidelberg involved in the selection process.

The questionnaire was kept as short as possible due to limited time and to reduce participant burden. Questions were tested in two pilot studies before being used in the current study. Piloting was aimed at resolving issues with understandability and scaling. Questions and response scales are described in detail in the following paragraphs.

To determine the relationship between time and money spent on preparatory activities and the total TMS score, we used a hierarchical multiple regression analysis. For analyses of the TMS-subtests, we conducted a hierarchical multivariate multiple regression to determine the global effect of all predictors (motivation, expenses, and preparatory activities) on all nine TMS-subtests simultaneously. Subsequent univariate multiple regression analyses were performed to determine the unique relationships between each of the predictor and criterion variables. To account for multiple testing in the subsequent univariate analyses, the alpha level was adjusted to 0.005 via Bonferroni correction. In the first step of every regression, we controlled for age, sex and high school GPA. Because participants could not be randomly assigned to the different conditions, this approach was chosen in accordance with Briggs (2004), as these variables have been demonstrated to be associated with performance on aptitude tests (e.g., Briggs, 2004; Buchmann et al., 2010) and therefore are potentially confounding variables. The multivariate multiple regression was performed with R (R Core Team, 2022) and all other analyses were conducted with SPSS 24.0.

Our results are reported as standardized and unstandardized regression coefficients with 95% confidence intervals. Standardized regression coefficients are used to make comparisons between predictors which are on different scales (e.g., preparation time and motivation), whereas unstandardized regression coefficients are used for comparisons of the preparatory activities. We report corrected R² determinants to adjust for potential overestimation bias due to a high number of predictors.

The results of our regression model testing the associations of motivation, preparatory activities, and expenses in addition to age, sex and high school GPA with the TMS total score are depicted in Table 3. 79.3% of all participants used the information booklet for preparation, 92.9% used books, 49.0% used the computer, 23.8% used learning groups, and 18.9% used training courses. 2.6% of all participants indicated that they had invested more than 80 hours in at least one of the preparatory activities.

The corresponding correlation matrix of all predictors and the outcome variable can be found in Table 4. We also calculated correlations between all predictors with the relative times spent on each preparatory activity (absolute time spent on preparatory activity x divided by the sum total of times spent on all preparatory activities) to facilitate interpretation. The results are depicted in Table 5.

In total, the model explains 20% of variance, where 9% of variance is explained by the control variables (high school GPA, age, and sex) and 11% is explained by the predictors motivation, time, and expenses incrementally over GPA, age, and sex.

Our results indicate that with an increase in self-reported motivation by one point on the respective scale, such as from low to medium or from medium to medium-high, the TMS score increases by 3.90 points (b = 3.90, CI [3.35, 4.46]) on a scale from 0 to 178 points, holding all other predictors constant.

Regarding the time spent on different preparatory activities, significant but small associations can be detected: their average magnitude equals less than a one point change of the 178 points in the TMS score.

When studying one hour longer and holding all other factors constant, using the information booklet is associated with a decrease of 0.211 (b = −0.211, CI [−0.267, −0.155]) points in the TMS score, using books with an increase of 0.118 (b = 0.118, CI [0.104, 0.132]) points. With an average across all participants of about 36 hours spent preparing with books, this factor accounts for an increase of about 4.2 points or 0.19 standard deviations on average (note that we used the standard deviation of 21.56 points from the total sample as reference). In contrast, using computer programs was associated with a decrease of 0.10 (b = −0.10, CI [−0.137, 0.063]) points and using training courses with an increase of 0.076 (b = 0.076, CI [0.038, 0.114]) per hour. With the average preparation time of roughly 5.5 h, preparation with training courses would account for an increase of 0.4 points or 0.02 standard deviations on the TMS total score. Spending more time learning in study groups is not significantly associated with the TMS score (b = 0.06, CI [0.01, 0.111]).

Expenses for preparatory activities show a significant association with the TMS score. Each Euro spent on preparatory activities is associated with an increase of 0.004 (b = 0.004, CI [0.002, 0.006]) points in the TMS-score. This means that – according to our model – an investment of 250 EUR would be associated with a TMS total score increase of one point out of 178 points. 136 EUR – the average amount spent on preparation – on the other hand, would result in a predicted TMS total score increase of about 0.6 points or 0.03 standard deviations. Additionally, test-takers who spend more money overall on preparation are significantly more likely to spend most of their time preparing with training courses suggesting that training courses are the main driver of preparation cost (r(7901) = 0.51, p < 0.001). Comparing the standardized regression coefficients, the association of motivation with the TMS score is 2.5 times stronger than the association of expenses with the TMS score.

Because the multivariate multiple regression yielded significant results for all predictors (Table 6), univariate multiple regressions were performed for each subtest to determine the unique relationships with each predictor (Tables 7, 8). The univariate regression analyses showed differences with respect to the incremental variance explained by motivation, preparatory activities, and expenses. When controlled for age, sex, and GPA the complex field-specific subtests TV (ΔR² = 0.012), QFP (ΔR² = 0.013), MNGV (ΔR² = 0.014), DUT (ΔR² = 0.008) are less affected by preparatory activities, expenses and motivation than the remaining subtests MUZ (ΔR² = 0.075), SF (ΔR² = 0.121), FIL (ΔR² = 0.097), FAL (ΔR² = 0.118), KONZ (ΔR² = 0.05).

Self-reported motivation showed a significant association with all subtest-scores. Regarding the complex subtests, a one-point increase in self-reported motivation was associated with a score increase of 0.211 points (DUT) to 0.306 points (MNGV). For the less complex subtests, a one-point increase in self-reported motivation was associated with a score increase of 0.445 (KONZ) to 0.759 (FAL) points.

Concerning the different preparatory activities, studying with the information booklet and books showed a significant association with all subtests. Preparation with the information booklet was associated with a decrease of points in all subtests, ranging from – 0.02 (TV) to – 0.026 (QFP) points per self-reported hour for the complex subtests and from – 0.016 (MUZ) to – 0.036 (FAL) points for the less complex subtests. For preparatory activities with books, one self-reported hour was associated with an increase in points ranging from 0.004 (DUT) to 0.007 (MNGV and QFP) for the complex subtests and from 0.011 (MUZ) to 0.025 (FAL) points for the less complex subtests. Preparatory activities on the computer were not associated with the complex subtests MNGV and QFP. For DUT and TV preparation on the computer was associated with a decrease of 0.009 and 0.01 points per self-reported hour respectively. For all less complex subtests preparation on the computer was associated with a decrease in points, ranging from 0.011 (SF) to 0.014 (MUZ and FAL) per self-reported hour.

Spending time preparing in study groups only showed significant associations with the less complex subtests MUZ and KONZ with an increase of 0.012 to 0.016 points per each self-reported hour, respectively. Time spent in training courses also had no significant association with the complex subtests. In contrast, the time spent in training courses was associated with a point increase on all less complex subtests. The increase of points per self-reported hour ranges from 0.011 (FIL) to 0.022 (KONZ) points.

Expenses for preparatory activities showed significant associations with the less complex subtests except for KONZ. The increase of points per self-reported hour was 0.001 for FIL, MUZ, SF, and FAL respectively. No associations of expenses were found with the complex subtests.

In the current study, we assessed the association of self-reported motivation, time spent on various preparatory activities, and financial investment with TMS results of a cohort of medical school applicants. Most notably, preparatory activities and money spent for preparation had only small associations with the TMS result, while, for example, self-reported motivation during preparatory activities had a 2.5-fold stronger association with the TMS score than the money spent on it.

As we controlled for preparation time, the aforementioned association of motivation with the TMS result cannot be explained by a mere difference in duration, but rather appears to point to a difference in the quality of time spent. Motivation might play a role in how a person “uses” the preparation time and also how he or she works on the TMS during the actual test-taking procedure. Highly motivated persons might take the TMS more seriously. Additionally, persons with higher motivation may perceive the TMS tasks as more joyful than others and have a higher interest in them, because motivation is associated with positive emotions such as interest and enjoyment (Deci and Ryan, 1985; Brandstätter et al., 2018).

Interestingly, motivation also appears to be a factor in the differences we found in the effectiveness of time spent on each preparatory activity. The higher the motivation, the more absolute time test-takers spent on either of the preparatory activities. It is not surprising that participants who are more motivated invest more time in preparation, but it is also noteworthy that they appear to invest more time in preparatory activities that are efficient (especially books). This may be the result of a combination of two different pathways: motivated test-takers may be drawn to the modes of preparation which are ‘per se’ more effective modes of preparation, and it is also conceivable that due to higher motivation, the overall quality of the time they spent on the more effective activities positively affected results.

On the other hand, we investigated the association of money spent on preparation with the TMS result. Previous studies on whether financial resources are a significant predictor of high-stakes admission test results have yielded contradictory findings (e. g. Lambe et al., 2012; Laurence et al., 2013; Wilkinson and Wilkinson, 2013). Our results show, that financial investments in preparatory activities play a statistically significant but a practically small role. Therefore, our findings are in line with studies in the context of other study admission tests showing the direct effects of money spent on preparation to be very small and, therefore, either non-significant or overrated (e.g., Powers and Rock, 1999; Sackett et al., 2008; Wilkinson and Wilkinson, 2013). All other factors (including motivation) being equal, mere time spent on the different preparatory activities – the information booklet, books, computer, study groups, and training – yielded only small positive and sometimes even small negative associations with the total TMS score and its subtests, respectively. This suggests that the mere quantity of time invested in preparation is not the decisive factor in achieving good test results. However, within these small associations, some relevant differences were detected.

Our results indicate, that of all preparatory activities, preparation with books descriptively yielded the highest positive association with the TMS score. Additionally, it is the only activity that showed a positive association with the performance in all subtests. Interestingly, the magnitude of these associations seems to be more in line with effects found for training courses and test retaking in previous meta-analyses (e.g., Messick and Jungeblut, 1981; Powers and Rock, 1999; Briggs, 2004, 2009; Hausknecht et al., 2007) than with usually smaller effects reported for preparation with books. For example, Briggs (2004) reported effects of 0.11 to 0.31 standard deviations for training courses, but no significant effects for preparation with books. Hausknecht et al. (2007) found an adjusted effect of d = 0.26 for retesting of different admission tests (i.e. taking a parallel or the same test form a second time). However, on a closer look, the relative effectiveness of preparation with books does not surprise in the context of the TMS. Books typically used for preparation contain original earlier TMS test versions, which means that participants who spend time preparing with those books would practice with original earlier tests. Thus, for the TMS, preparation with books can be seen as a form of retesting, which would explain why the associations found in our study are more comparable to the effects of retesting reported by Hausknecht et al. (2007). Practicing with books, therefore, is highly encouraged by the test administrators as the most effective and cost-efficient way of preparation.

Preparing with the information booklet does not appear to yield any additional training benefit beyond mere informational purposes and general advice about the style, format, and general requirements associated with the different subtests of the TMS. In addition, only a small number of practice items are provided, which may cause participants who spend too much time on the booklet to become too focused on this narrow subset of items, potentially providing them with a false sense of security.

Accordingly, time spent in study groups was found to have no association with the overall TMS score. One possible reason for this finding is, that study groups may vary considerably with regard to the structure and content discussed and therefore may not yield reliable positive results overall. Concerning training courses, we found a small positive association with the TMS total score. For 20 h of preparation with training courses, our model would predict a score increase of 0.07 standard deviations – considerably less than the training effects of 0.12 or 0.20 standard deviations reported by Messick and Jungeblut (1981) or the training effects reported by Briggs (2004) and Hausknecht et al. (2007). To achieve a similar effect for the TMS it would require about 34 to 80 h of preparation.

Analyses suggest, that the complex subtests may be more robust to preparation than the less complex subtests. At the subtests level, preparation with books appears to be the only activity that has a positive association with the performance in all subtests, while the score increases were higher for the less complex subtests. Training courses, on the other hand, appear to yield no association with the complex subtests. In contrast, training courses had positive associations with all less complex subtests, while the score gain per hour was somewhat higher for MUZ and KONZ compared to preparation with books. For all other less complex subtests, preparation with books seems to be more effective than preparation with training courses. On the subtest level, preparation on the computer and preparation in training courses do not appear to be effective types of preparation. Both activities yielded negative associations with most of the complex and all less complex subtests. Again, associations were predominantly higher for the less complex subtests than for the complex subtests. Even though preparation with study groups was not associated with the overall TMS score, at the subtest level, small positive associations were found for MUZ and SF. Suggesting that, at least for some of the less complex subtests, positive results can be obtained with this mode of preparation.

Due to time constraints during test-taking, we only assessed motivation with a single item directly and retrospectively asking for participants’ subjective degree of motivation during preparation, instead of a validated questionnaire designed to assess the entirety of the construct. This is associated with a number of additional sources of error and provides room for alternative explanations of our findings. For example, participants’ answers could have been affected by their degree of confidence with regard to the test material, which in turn might have been affected by their actual or perceived ability and their expectation to succeed in the test. Therefore, participants who believe that they will succeed in the test might indicate higher levels of motivation during preparation in retrospect than they actually experienced at the time. Also, participants who are more optimistic about their general chances to obtain a study place – for example due to very good GPA grades, a generally high self-efficacy, or high academic confidence – might indicate higher motivation at the time of assessment regardless of their actual motivation during preparation. All of these pathways are conceivable, and we cannot rule out the possibility that our measure of motivation is affected by other constructs such as self-efficacy, optimism, actual ability, or academic self-concept rather than measuring intrinsic motivation at the time of preparation. Therefore, results regarding the role of motivation should be further investigated with validated questionnaires and in a longitudinal design. Nevertheless, we do believe that our interpretation of the results is still valid, as motivation and constructs such as self-efficacy and ability itself are generally correlated (Goldberg, 1994; Richardson et al., 2012). We therefore believe that our results can provide a first indication that motivation may not only be relevant during university training and for a successful career in medicine (e.g., Lawler and Hall, 1970; Zapata-Phelan et al., 2009; Latham, 2012) but also already at the stage of application and admission.

Since the complex and less complex subtests seem to be differentially associated with preparatory activities, it should be mentioned that preparation effects do not necessarily harm the validity of a test. Preparation can even increase the validity by reducing the construct-irrelevant variance in participants’ test scores attributable to differences in participants’ familiarity with the test (cf. Briggs, 2009; Arendasy et al., 2016). Also, it may be possible that the underlying skills measured are trainable to a different extent and that the differing effects found for the complex and less complex subtests reflect differing actual gains in ability (cf. Arendasy et al., 2016). In this case, the training effects would not affect validity. To be able to assess the meaning and the differences of preparation effects in the context of the TMS and other admission tests, it would therefore also be important to investigate the influence of preparation on validity.

In selecting the different preparatory activities listed in our questionnaire, we followed prior relevant studies (e.g., Deter, 1982; Bartussek et al., 1984; Trost et al., 1998; Briggs, 2004, 2009). Nevertheless, it is possible that we did not include other relevant preparatory activities. Therefore, one possibility for future research would be to allow participants to specify additional preparatory activities by use of an open response format. However, based on the previous studies mentioned above, we assume that at least the most essential preparatory activities are included in our study.

Another important limitation is that the data was collected cross-sectionally. Therefore, no conclusive statements can be made about the causality of the associations found. However, it is also important to note that the questionnaire on preparatory activities was conducted shortly before the actual testing. Thus, the participants did not yet have an accurate impression of their test performance, which might then have affected the statements regarding their preparatory activities. However, a longitudinal study would be an important approach for future research to clearly answer the question of causal direction.

We also had no information on the extent to which the participants already had connections within the medical field (e.g., through parents, siblings, or friends). For example, it is conceivable that well-connected individuals have better knowledge of and access to the different types of preparation, which may moderate the observed effects to some extent and should therefore be taken into account in future studies.

Previous studies have also shown that SES can be associated with performance on high school aptitude tests (e.g., Briggs, 2004). Since we had no information on the participants’ SES, we could not control for this potentially confounding variable. For example, it may be the case that some effective preparatory activities were more available to applicants with higher SES. Around 93% of the participants in our study reported using books for preparation, which indicates that almost all participants had knowledge of and access to this type of preparation. Thus, the likelihood that access to preparation books is related to the participants’ background, seems to be quite low. Nevertheless, this possibility cannot be ruled out completely at this moment. The situation is different for training courses, however: only 19% of the participants stated that they used this type of preparation. Since training courses are usually expensive, they are most likely not available to all subgroups of participants. For future studies, it would be important to investigate this relationship in more detail. Nevertheless, since books are significantly less costly and more effective, applicants who attended costly training courses are not likely to have an advantage over participants that cannot afford this kind of preparation. As noted before, time can also be seen as a resource not everyone may have equal access to. Therefore, it may be the case that SES is related to the time spent for preparation. Since all preparation effects are small, this study provides an indication that the effects of such a relationship on the selection process would likely also be small. However, in order to answer the question conclusively and to rule out the possibility that some groups of participants are systematically disadvantaged, the aforementioned limitation must be taken into account in future studies.

An important implication for TMS participants is that the test scores are only associated to a comparatively small extent with any given preparatory activity, indicating little influence of preparatory activities on the test result. However, the results also suggest that a certain amount of preparation and familiarization with the test material is sensible. Based on our findings, preparation with books can be recommended for this purpose. In contrast, it is neither necessary nor advisable to spend a lot of money on expensive courses, especially because more effective and at the same time less expensive options such as books are available. For comparison, books with original TMS versions are available for the cost of approximately 13 euros, whereas training courses often range in cost from 150 to 500 euros.

The findings also have important implications in terms of equal opportunity. First, money spent on preparatory activities is only marginally associated with the test result indicating that participants with more money available to spend on preparation are not likely to be advantaged over other participants. Since there still is a small positive association, further research is needed to address the practical significance of this association regarding the allocation of study places. However, as already mentioned, the results so far indicate that money spent on preparation is not the decisive factor in context of the TMS. Second, of all preparatory activities included in the study, books seem to be the most effective and – because of the relatively low cost – most accessible type of preparation. This is also reflected in the fact that most participants prepared using books, which in turn minimizes the likelihood that certain subgroups of applicants are disadvantaged because they lack access to this type of preparation. However, we are aware that books still cost money and may not be equally accessible to everyone. To address this issue, the test developers have invited all test takers of 2021 to practice free of charge with original test items through an online-preparation platform as a first pilot. This project was continued in 2022 and is planned to be further expanded and evaluated in the future. Third, although preparation with books and training courses is associated with test results, the effects of these preparatory activities are modest. This again limits the likelihood that subgroups of applicants with more time to spend for preparation have a systematic advantage over other applicants. As mentioned earlier, this topic needs to be explored further in future research because the effects of other important variables, such as participants’ SES or any pre-existing connections to the medical field (such as having parents who are doctors), could not be addressed in this study. In summary, we are still confident that the results provide initial indications that important prerequisites for equal opportunities are met. However, as mentioned before, further research is necessary in order to be able to treat this topic conclusively.

Since the results suggest a linear relationship between preparatory activities and test results, this finding should not be misinterpreted in a way that the test score can be increased to any degree by excessive preparation. This is because our results apply primarily to preparation times up to a maximum of 80 hours, as only a small proportion of participants reported preparation times above 80 hours (mainly for books). Thus, for preparation times exceeding 80 hours, other associations (e.g., logarithmic or quadratic relationships) with the test results are still conceivable (e.g., Messick and Jungeblut, 1981; Hausknecht et al., 2007).

In addition, our results provide evidence, that the TMS-subtests are differently affected by preparatory activities. Although the associations of preparatory activities with the TMS were small for all subtests, mostly the complex and more field-specific TMS-subtests appear to be resistant to preparatory activities. On the other hand, performance on more unidimensional subtests such as concentration, mental rotation, and memory seems more likely to improve with preparation. This also has important implications for test construction. For example, a promising approach might be using a substantial number of more complex field-specific subtests to further minimize the trainability of admission tests and thus the impact of resources such as time and money.