^{*}

Edited by: Joseph W. Houpt, Wright State University, USA

Reviewed by: Richard Schweickert, Purdue University, USA; Noah H. Silbert, University of Cincinnati, USA

*Correspondence: Richard A. Chechile, Psychology Department, Tufts University, Psychology Building, 490 Boston Av., Medford, MA 02155, USA e-mail:

This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

In the area of memory research there have been two rival approaches for memory measurement—signal detection theory (SDT) and multinomial processing trees (MPT). Both approaches provide measures for the quality of the memory representation, and both approaches provide for corrections for response bias. In recent years there has been a strong case advanced for the MPT approach because of the finding of stochastic mixtures on both target-present and target-absent tests. In this paper a case is made that perceptual detection, like memory recognition, involves a mixture of processes that are readily represented as a MPT model. The Chechile (

The title of this special issue implies two very different questions. The first question is: how should perceptual decision-making be modeled? The second question is: how should individual differences be estimated? This paper addresses both of these questions from a perspective that has been informed by research in the area of model-based memory measurement. The recommendations from this perspective result in some novel techniques for examining perceptual detection data.

Signal detection theory (SDT) is the classic method for measuring the perceived strength of a stimulus (Tanner and Swets,

Egan (

MPT models are mixture models because with this approach it is assumed that there are possibly different knowledge states that have differential consequence for behavior. For example, sometimes there is enough information stored in memory that the individual can reproduce the target event entirely, provided that the information is accessible at the time of test. But for other tests, the requisite information is either incomplete or totally missing. In the Chechile (_{S}. The θ_{S} parameter is thus a mixture component. Similarly the other parameters in the 6P model are also probabilities and can be regarded as conditional mixture probabilities. Chechile (^{1}

While there is an ongoing debate about SDT and MPT models in the memory literature, there has not been a corresponding contemporary debate in perceptual psychology about the relative merits of SDT and MPT approaches. Yet the possibility of stochastic mixtures is quite plausible for perceptual detection studies, so there are reasons for considering MPT models for perceptual detection.

One rationale for suspecting that there are mixtures comes from the Stevens (

… it is interesting that some of the better known prothetic continua seem to be mediated by an

The Stevens distinction stresses the difference between changes in intensity on a single dimension and changes in qualities. A homogeneous process (as opposed to a mixture) is more likely when dealing with a prothetic continuum; although DeCarlo (

The second focus for this paper concerns the relative accuracy of various statistical procedures for modeling individual differences in terms of the key parameters of a perceptual detection MPT model. There is a widespread belief that the maximum likelihood estimates (MLE) of model parameters, done on an individual basis, is the optional method for obtaining estimates of individual differences. This belief is mistaken; there is now considerable evidence that the MLE can be non-optimal and biased for a number of important practical cases. Even in the case of the Gaussian model with more than two conditions, the MLE estimates are known to be biased and “inadmissible” due to the Stein paradox (Stein,

The Perceptual-Detection (PD) model is essentially the Chechile (_{d} parameter is the proportion of target-present tests when the operator clearly and confidently detects the target stimulus; this parameter corresponds to the sufficient storage parameter θ_{S} in the 6P model. The θ_{nt} parameter is the proportion of the target-absent trials when the operator can confidently identify a stimulus that is different than the target; this parameter corresponds to the knowledge-based foil rejection parameter θ_{k} in the 6P model.

The θ_{d} and 1 − θ_{d} parameters are mixing rates for target-present trials. When the target is not clearly detected, the observer can still decide that the stimulus is a target (with conditional probability θ_{g}) by a secondary process that is simply labeled as a guessing process. Similarly on target-absent tests, the operator (with probability 1 − θ_{nt}) fails to confidently identify a non-target but can still guess (with probability θ_{g′}) that the stimulus is more likely a non-target than a target. The two guessing parameters in the PD model are the same as the guessing parameters in the 6P model. Finally the θ_{h} parameter is a “nuisance” parameter because it is a conditional probability that is only important as a correction for overly confident guessing. This parameter corresponds to the θ_{1} parameter in the 6P model.

A great deal is known about the 6P model, and this information directly transfers to the PD model. For example, Chechile (^{2}^{3}^{4}

As an example of parameter estimation for the PD model, let us consider the actual case of the detection characteristics of a single radiologist who was assessing 109 CT scans in order to detect abnormal versus normal scans. Hanley and McNeil (^{5}^{6}

θ_{d} |
0.552 | 0.555 | 0.578 |

θ_{nt} |
0.496 | 0.507 | 0.523 |

θ_{g} |
0.734 | 0.711 | 0.721 |

θ_{g′} |
0.405 | 0.438 | 0.421 |

θ_{h} |
0.250 | 0.259 | 0.227 |

The PD model point estimates fit the multinomial frequencies very well as indicated by a non-significant goodness-of-fit difference between the observed and predicted frequencies, i.e., ^{2}(1) = 0.262. In addition to the point estimates, the two Bayesian methods have a posterior probability distribution for each model parameter, and these distributions provide a method for testing some important questions about the radiologist. One of the central ideas in the PD model is the concept that there is a mixture of states for both target-present cases (abnormals) and for target-absent cases (normals). From the posterior distribution of the θ_{d} parameter, it can be stated that the probability exceeds 0.95 that the θ_{d} parameter is at least 0.39, i.e., _{d} > 0.39) > 0.95. Similarly the posterior distribution for the θ_{nt} parameter results in the high probability statement that θ_{nt} is at least 0.37, i.e., _{nt} > 0.37) >0.95.

Using a standard SDT model analysis of the radiological data results in an estimate of ^{2}(1) = 0.220. However, the SDT model does not posit that there are mixtures, so the finding that the θ_{d} and θ_{nt} parameters are reliably different than zero demonstrates that the conventional signal detection model is missing an important feature exhibited by the radiologist. If there were an absence of mixtures, then the PD model would have estimated the θ_{d} and θ_{nt} parameters as approximately 0.

For MPT models, the mean of the Bayesian posterior distribution for a parameter is usually a different value than the MLE. Chechile (_{i} values in Figure ^{7}_{d} parameter.

_{d} parameter for both the PPM and MLE methods

10 | 0.129 | 0.198 | 0.090 | 0.174 |

20 | 0.102 | 0.143 | 0.076 | 0.135 |

30 | 0.090 | 0.124 | 0.070 | 0.123 |

40 | 0.082 | 0.112 | 0.064 | 0.115 |

50 | 0.075 | 0.099 | 0.061 | 0.104 |

100 | 0.059 | 0.071 | 0.050 | 0.074 |

300 | 0.039 | 0.043 | 0.035 | 0.047 |

600 | 0.029 | 0.030 | 0.027 | 0.031 |

1000 | 0.023 | 0.023 | 0.022 | 0.025 |

The Bayesian PPM estimates are more accurate for all the sample sizes. Although the MLE and PPM errors are approaching each other, the rate of approach is relatively slow. Notice that even for the case of

The θ_{g} and θ_{g′} parameters have actually been used in memory applications since the original storage-retrieval separation paper by Chechile and Meyer (

For the PD model, there is a counterpart to the educated guessing based on partial storage. For the perceptual detection task, there might be occasions where a stimulus is judged more likely a target than not but the quality of the perception is not good enough to constitute a confident classification. On other occasions, the stimulus might be judged more likely a particular “non-target” than a target, but again because the stimulus quality is degraded, the observer is uncertain. For both cases the stimulus is not in a clear detection state, but nonetheless, the person is still able to make informed decisions above a random guessing level.

An interesting special case is when the guessing in both target-present and target-absent conditions are purely response bias, i.e., when θ_{g} = 1 −θ_{g′}. However, if there is something like the partial storage found for some memory studies, then the stimulus is more likely to yield a yes response in the target-present condition than in the target-absent condition. Note that the radiologist measured with the PD model exhibited guessing better than pure response bias because θ_{g} = 0.734 > 1 − θ_{g′} = 0.595. These results are consistent with the interpretation that the radiologist was relatively conservative because the doctor guessed that the patient had an abnormality at a rate of 0.595 for the subset of difficult scans from healthy patients. Nonetheless for the subset of difficult scans from patients with an abnormality, the rate for deciding on the abnormal categorization increased to 0.734. Consequently on these more challenging CT scans the physician did have some differential tendency to use the abnormal classification when in fact the CT scan came from a patient with an abnormality.

The Receiver Operator Characteristic (ROC) in SDT is a curved plot of the hit rate versus the false alarm rate. In standard SDT, any point on the ROC is a possible operating point depending on the decision criterion used by the subject. Hence in standard SDT, the ROC is an iso-sensitivity curve. In standard SDT, the points (0, 0) and (1, 1) are on the ROC curve; these points are the extrema. If the subject had no ability to detect the target, and the data are identical in the target-absent and target-present conditions, then the ROC would be the line of slope 1 connecting the extrema. If there is some greater tendency to detect the target in the target-present condition, then in standard SDT the ROC is a smooth curve in the region of the unit square where

Empirical ROC plots have been used in numerous experimental papers as a method for comparing theories, but it is challenging to statistically discriminate between models based on only a few points on the empirical ROC. However, given the historical interest in the ROC in psychology, it is instructive to consider the theoretical ROC for the PD model. See Figure _{2}, _{3}, and _{4}. These three points and the two extreme points for the PD model, _{1} and _{5} are a function of the five parameters in the PD model. If 0 < θ_{d} < 1, 0 < θ_{nt} < 1, and θ_{g} > 1 − θ_{g′}, then the ROC path is along two linear segments. Note that the single-high threshold model discussed by Macmillan and Creelman (_{nt} = 0 and θ_{g} = 1 − θ_{g′}. The double-high threshold model also discussed in Macmillan and Creelman (_{nt} = θ_{d} and θ_{g} = 1 − θ_{g′}.

_{i}, _{1} = (0, θ_{d}) and _{5} = (1 − θ_{nt}, 1).

_{0} |
0 | 0 |

_{1} |
0 | θ_{d} |

_{2} |
(1 − θ_{nt})(1 − θ_{g′})θ_{h} |
θ_{d} + (1 − θ_{d})θ_{g}θ_{h} |

_{3} |
(1 − θ_{nt})(1 − θ_{g′}) |
θ_{d} +(1 − θ_{d})θ_{g} |

_{4} |
(1 − θ_{nt})(1 − θ_{g′}θ_{h}) |
θ_{d} +(1 − θ_{d})(1 − (1 − θ_{g})θ_{h}) |

_{5} |
1 − θ_{nt} |
1 |

_{6} |
1 | 1 |

_{2} corresponds to the case where a positive response is considered as a high confident yes, but for point P_{3} a positive is regarded as any yes response. For point P_{4} a positive is considered as any response that is not a high confident no

To better understand the PD ROC, consider points _{2} and _{3}. If we were to define an affirmative response as strictly a “yes” with high confidence, then the corresponding false alarm rate and hit rate would be illustrated by _{2} and have the values corresponding to the prediction equation shown in Table _{3} and the corresponding prediction equation in Table _{2} and _{3} is denoted as _{23} and is given as
_{1} and _{2} is also equal to _{23}. The linear path from points _{1} and _{3} can be described in terms of a hypothetical variable

The least risky point _{1} corresponds to when _{2} corresponds to the more risky case when _{h}. Point _{3} corresponds to the even more risky case of _{1} to _{3} are _{2} and _{3}. Interestingly the slope from _{3} to _{4} is in general different than the slope from _{1} to _{3}. Let us denote the slope from _{3} to _{4} as _{34}, and it is given as

It is also the case that the slope from _{4} to _{5} is also equal to _{34}. Moreover, the linear path from _{3} to _{5} can be described in terms of another hypothetical variable

The _{3} point corresponds to _{4} point corresponds to _{h} and _{5} corresponds to

Figure _{1} to _{3} line as stipulated by Equations (2, 3). Similarly as _{3} to _{5} line. Notice that θ_{h} determines the separation from each of the two extreme ends. This feature is a property of the PD model because there is a common parameter of incorrectly using the high confidence rating when guessing regardless if the guessing is done in either the target-present condition or the target-absent condition. Chechile (_{2}) versus over confidence when using the “no” response (θ_{1}). This model is the 7B model. Other than the difference in the handling of over confidence, the 7B and 6P models are identical, i.e., the 6P model is the special case of 7B where θ_{h} = θ_{1} = θ_{2}. Model 7B can also be applied to the perceptual detection task (lets denote that model as the PD^{*} model). In the PD^{*} model the θ_{2} parameter determines the location for the _{2} point, and the θ_{1} parameter determines the separation for the ^{*} model than the spacing shown in Figure

In general the slope from _{3} to _{5} is less than the slope from _{1} to _{3}. Given Equations (1), and (4) the ratio of the slopes can be written as

If there is some partial or degraded perception, then the tendency to respond “yes” is at least equal or greater in the target-present condition as it is in the target-absent condition. It follows that

It also follows from Equations (7, 8) that _{g} > 1 − θ_{g′}, then the slope from _{1} to _{3} is larger than the slope from _{3} to _{5}. The case where _{g} = 1 − θ_{g′} or when there is the same “yes” guessing in the target-present condition as in the target-absent condition. In this special case, there is no partial detection, and the ROC does not have two linear components, but there is instead a single line of slope _{1} and _{5}.

The area under the ROC has been used as a measure of sensitivity in standard SDT. It is straightforward to show that area _{c} between the _{1} − _{5} dashed line in Figure _{d} + θ_{nt} − θ_{d} θ_{nt})^{8}_{c} by 2, so that the area measure of certain detection is placed on a 0 to 1 scale. This measure is defined as a certain detection _{c}, and

The area of the _{1} _{3} _{5} triangle is a function of guessing. This area is denoted as _{g}, and it can be found from Heron's formula, i.e., _{g} = _{nt})(1 − θ_{d})[θ_{g} − (1 −θ_{g′})]. We can put this measure of effective guessing on a 0 to 1 scale by defining _{g} = 2_{g} or

Thus the total detection measure can be defined as twice the area between the ROC and the main diagonal; this metric is _{c} + _{g} or

As an example, let us compute these area-based metrics for the radiological data discussed in section _{d} and θ_{nt}, it follows from Equation (9) that _{c} = 0.774. The corresponding _{g} measure from Equation (10) is 0.031, so the overall

Although the detection measure _{nt} = 0.805 and θ_{d} = 0, then the resulting _{d} = 0 and θ_{nt} = 0.805 would be very good at recognizing a normal CT scan, but would not be capable of detecting an abnormal scan, which would be a rather serious problem for the diseased patients of that hypothetical radiologist! Consequently, the area-based _{c} and _{g}, is less informative as the original PD model parameters. The detection of the target increases with the value of the θ_{d} parameter, and the identification of a non-target increases with the value of the θ_{nt} parameter. Those two types of detection can be quite different. It is also informative to know how the observer does for the unclear cases where there is guessing. The

A fundamental issue that arises in mathematical psychology is the basis for fitting a model. One method is to fit the model separately for each individual and to average individual estimates for the group average. Another method is to aggregate the data across a group of individuals for a particular experimental condition and then fit the model once for that condition^{9}

There are several contexts that necessitate the fitting of the model on an individual basis. For example, if the model is a non-linear function of an independent variable, then many investigators have demonstrated that group-averaged data can result in biased fits (Estes,

There are also cases when pooling the data prior to the model fit is the preferred analysis (Cohen et al., _{g} simulated “subjects” with slightly different true values for the model parameters was constructed, and for each artificial subject there were _{r} “observations” that were randomly sampled from the appropriate multinomial likelihood distribution^{10}_{g} subjects, and these estimates were averaged to obtain an estimate for each model parameter. For an arbitrary model parameter, θ_{x}, the group average estimate is _{x i} is the parameter estimate for the _{x} and the true mean for that parameter _{g} and _{r}. Across these separate Monte Carlo runs the model parameters were varied, so the model was simulated over a vast set of configurations of the parameters. The overall error for the averaging method is the mean error across the 1000 Monte Carlo data sets for each combination of _{g} and _{r}. For the identical data as described above, a corresponding error was also found for the pooling method. For the pooling method the frequencies in each multinomial response category was summed across the _{g} subjects in a group, and the model was fit once with the pooled data. The estimate based on pooling for the _{x j}(^{11}_{g} and _{r}, the mean error for the pooling method was less than the corresponding error obtained for the averaging method^{12}

One of the models examined in Chechile (_{d} parameter in Table _{S} parameter in Model A; whereas θ_{g} and θ_{h}, respectively, correspond to the θ_{g} and θ_{1} parameters in Model A.

_{g}_{r}

_{g} n_{r} |
_{d} |
_{g} |
_{h} |
---|---|---|---|

20 20 | 0.069 | 0.076 | 0.078 |

20 50 | 0.045 | 0.050 | 0.051 |

20 100 | 0.034 | 0.035 | 0.034 |

20 400 | 0.015 | 0.013 | 0.013 |

40 20 | 0.078 | 0.086 | 0.087 |

40 50 | 0.054 | 0.059 | 0.059 |

40 100 | 0.037 | 0.040 | 0.040 |

40 400 | 0.017 | 0.015 | 0.015 |

80 20 | 0.087 | 0.096 | 0.098 |

80 50 | 0.059 | 0.065 | 0.064 |

80 100 | 0.043 | 0.043 | 0.045 |

80 400 | 0.020 | 0.016 | 0.016 |

The pooling advantage scores in Table _{r}), there is still a non-trivial advantage for pooling even when _{r} = 400. It is challenging to do an experiment with large values for _{r}. For example, a replication number of 50 is larger than all but two of the memory studies reported from my laboratory. Consequently, the idea of running a large number of replication trials per subject is not a practical option. Third, the size of the pooling advantage increases with group size _{g}. This effect is due to the fact that the error for the pooling method decreases rapidly with increasing group size; whereas the error for the averaging method slowly decreases with increasing _{g}, so the net effect is that the pooling advantage score increases with _{g}.

It might not seem intuitive as to why the pooling of data results in superior estimates for the group mean. This result is more reasonable when viewed from a Bayesian perspective. From Bayes theorem it does not matter if the data are examined in aggregate or one observation at a time, provided that the same starting prior probability is used. Suppose we use a uniform distribution as the prior distribution for each combination of the parameters (θ_{d}, θ_{g}, θ_{h}). Let us call this prior the “vague” prior. Furthermore suppose we examine the model parameters for the first individual in the group via Bayes theorem to yield a posterior distribution. The posterior distribution after the first individual should then be the prior distribution for examining the data for the second subject, i.e., it is no longer appropriate to maintain the vague prior after examining the first subject. Similarly the prior distribution for Subject 3 should be the posterior distribution after considering the first two subjects. This one-subject-at-a-time method eventually yields a posterior distribution that is the same as the posterior distribution achieved by pooling the multinomial categories and applying Bayes theorem once. Had the Bayesian analyst used a vague prior for each of the _{g} subjects and averaged the estimates, then the analysis would not be consistent in the application of Bayes theorem. The averaging of separate estimates is not an operation by which probability distributions are revised via Bayes theorem. In terms of this framework, the findings in Table

Despite the above demonstration of a pooling advantage for estimating the group mean, it is still an open question as to what should be the basis for estimating the model parameters for an individual. Two choices seem reasonable. One method is simply to use the data for just the individual, e.g., for the θ_{d} parameter it would be _{d i} for the _{d} parameter this estimate is denoted as ^{(a)}_{d i} and is defined as

Note that the two methods have estimates that are perfectly correlated because the adjusted estimate ^{(a)}_{d i} is a constant plus the individual estimate _{d i}. The constant correction term is equal to _{d}(_{d}. The correction makes the mean of the adjusted estimates equal to the pooling method estimate because

The estimate based on Equation (12) is similar in principle to a James–Stein estimator used for the linear model for Gaussian random variables because the estimate for the individual is shifted based on properties of the group.

Another Monte Carlo simulation was designed for a widely different group of simulated observers in order to assess the relative accuracy of the two methods for estimating the parameters for individuals. The group consisted of 10 observers for each of the 3 × 3 combinations of values for θ_{d} and θ_{nt}. The three values were 0.2, 0.5, and 0.8. For each of the 90 simulated observers the values for θ_{h} were randomly selected from a beta distribution with coefficients of 2 and 4, and the θ_{g} and θ_{g′} parameters were randomly selected from a beta distribution with coefficients of 28 and 14. Consequently true scores were established for each simulated observer. For each observer, 20 simulated observations were randomly sampled for the target-present condition, and another 20 observations were randomly sampled for the target-absent condition. These observations were based on the appropriate multinomial likelihood distribution for each subject. The PD model was then estimated by each method described above. Because θ_{d} and θ_{nt} are the two key parameters of interest in the PD model, the root mean square (rms) error was found between the true score point {θ_{d i}(_{nt i}(_{d i}, _{nt i}}. The rms error for the adjusted score point {^{(a)}_{d i}, ^{(a)}_{nt i}} was also found. The rms errors for the individual and the adjusted method are respectively 0.1671 and 0.1385. Thus, the adjusted estimates based on Equation (12) resulted in a 17% reduction in the rms error. This simulation illustrates the improvement in the accuracy of model estimation by the use of the adjusted score method.

In this paper the Chechile (_{d} and θ_{nt}) that measure the proportion of times that the observer confidently detects something that belongs to an identifiable category. The categories are different for targets and non-targets, but in both cases something is being identified. The measurement of these detection rates is an important part of the psychometric assessment of perceptual performance. The PD model also has three other parameters that come into play when the observer is unable to confidently classify the stimulus.

The PD model differs from standard SDT on the issue of stochastic mixtures. MPT models, like the PD model, are essentially probability mixture models. In contrast, SDT developed in the context of assuming separate but homogeneous distributions for target-present and target-absent conditions. The success of the PD model in accounting for the radiological judgments described earlier in this paper occurred because the PD model was sensitive to the fact the radiologist was able to know sometimes that a CT scan was normal and to know at other times that a CT scan revealed an identifiable abnormality. This attribute of categorical and sophisticated perception is not an isolated property of experts. More than 120 years ago William James discussed the importance of perceptual learning; in fact perception according to James differed from a pure sensation because of the information that the person associates and adds to the sensation (James,

It is noteworthy that the prototypic experiments in the early history of SDT used stimuli that were designed to be featureless and varied on only a single prothetic intensity dimension. For example the stimulus-absent stimulus for some experiments was white noise; whereas the target-present stimulus was a louder white noise (Tanner et al.,

The PD model is a minimalistic model that intentionally eschews delineating any specific cognitive representation of the stimulus. Like other MPT models, there are probability measures for specific states. The states for the PD model are: (1) a state of certain target recognition, which occurs on θ_{d} proportion of the target-present trials, and (2) the state of certain identification of something other than a target, which occurs on θ_{nt} proportion of the target-absent trials. These probability measures provide for a characterization of the observer's detection ability.

MPT models have many desirable statistical properties and can be estimated by a variety of methods. Monte Carlo simulations with large sample sizes demonstrated that the MLE and the Bayesian posterior mean for the PD model were very close, but the accuracy of these estimates differed more substantially for smaller sample sizes. When the estimates differ, the Bayesian mean was found to be more accurate. In addition, an improved estimate was found for the individual observer when the estimate based on the individual's data was adjusted. The adjustment was a fixed amount for all observers, and it equated the mean of the adjusted scores to the mean of the estimate based on pooled data. This adjustment was discussed as an analogous adjustment to the James–Stein shrinkage improvements to the MLE found for the multiple-group Gaussian model.

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

_{S}a suitable measure of recognition memory “strength”?

^{1}Some MPT models have been characterized as threshold models by the authors of the model (e.g., the two high-threshold model of Snodgrass and Corwin,

^{2}See Chechile (

^{3}The MCMC method is an implementation of the Metropolis–Hastings algorithm after an initial “burn in” period of 300,000 cycles for sampling each model parameter.

^{4}There is a difference in the prior distributions used for the MCMC method and for the PPM method. For the MCMC approach, a flat prior is assumed for each of the PD model parameters, i.e., the (θ_{d}, θ_{nt}, θ_{g}, θ_{g′}, θ_{h}) parameters. However, for the PPM method the prior is a flat distribution for the multinomial cell proportions shown in Figure _{i}) parameters. The joint posterior distribution for the (ϕ_{i}) parameters is a product of two Dirichlet distributions. With the PPM method, random samples of (ϕ_{i}) values are taken from the posterior distribution, and each vector of (ϕ_{i}) values is mapped to a corresponding vector of the PD model parameters.

^{5}There were six cases for the normals where the radiologist used another category called questionable. Three of these cases are assigned here to the second category (probably normal), and three cases were assigned here to the third category (probably abnormal).

^{6}There were two CT scans for the abnormals that the radiologist gave the response of questionable. One of these cases was assigned here to the second category, and one was assigned here to the third response category.

^{7}Given the values for _{1} = ϕ_{1}, _{2} = ϕ_{1} + ϕ_{2}, and _{3} = ϕ_{1} + ϕ_{2} + ϕ_{3} there are three decision points for randomly assigning a simulated “observation” to one of the four cells. For each simulated observation, a random score is sampled from a uniform distribution on the (0, 1) interval. If the random score is less than _{1}, then the observation is for cell 1. If the random score is in the [_{1}, _{2}) interval, then it is an observation for cell 2. If the random score is in the [_{2}, _{3}) interval, then the observation is for cell 3. If the random score is greater or equal to _{3}, then it is an observation in cell 4.

^{8}Note that the total area above the main diagonal is _{d})(1 − θ_{nt}), so _{c} can be determined by subtracting these quantities.

^{9}A third approach also exists for obtaining individual and group effects by means of a hierarchical Bayesian model similar to the analysis developed for MPT models by Klauer (

^{10}Each individual was within ±0.03 of the group mean.

^{11}This whole procedure of estimating the model with both the averaging and pooling method was done for both PPM and MLE estimates for each of the four typical MPT models.

^{12}Only eight cases out of 640 cases reported in Chechile (