Full Ability Estimates Report

From BEARWiki

The Full Ability Estimates Report lists each student’s proficiency estimate, the standard error of that estimate, an infit mean square statistic for the student, an associated t-statistic, and the student’s raw score on items he or she completed in that dimension. Summary statistics are displayed at the end of the report.

1. Select Full Ability Estimates Report from the menu.
2. Complete the Full Ability Estimates Report display options, illustrated in Figure 1.
   • Select a title for the report
   • Browse... to the desire folder and enter a filename for the text-based report
   • Select Yes for Student Detail if you wish to see data for each case, or select No if you only want to see summary data
   • Select Yes for Import Matrices if you wish to import scoring and design matrices. This would be necessary to compute proficiencies for a within-item multidimensional model. Otherwise, select No.
   • Select Yes for Show Estimate Type if you want the type to be displayed in the report heading, otherwise select No.
   • Click on OK to continue.

Figure 1. Full Ability Estimates Report display options dialog window.

The report will then be generated and exported as a text file and also displayed on the screen. As shown in Figure 2, the Full Ability Estimates report heading indicates whether EAP, DPV, or MLE estimates were computed if the user selected this option.


Figure 2. Excerpt from the Full Ability Estimates report for Example 2.

When Student Detail is requested by the user, the student data is presented in rows, one row per student. The first column shows the case system ID and the second column shows the respondent’s name or user-defined identifier. Columns three and four show the raw score the respondent achieved on that variable while the fourth column shows the total possible score from the items the respondent attempted.

The next column, labeled “Est” is the person proficiency estimate and the next column shows the standard error for that estimate. Standard errors for EAP and DPV estimates are computed from the variance of the estimate using ______, while standard errors for ML estimates are computed from the asymptotic variance, ______.

Person fit statistics are also shown including infit and outfit mean squares and their associated t-statistics. These can be interpreted in much the same way as for item fit statistics, which are discussed in the Infit and Outfit Mean Square section of the Item Estimates report.

As shown in this example, student “110104” has an infit, or weighted, mean square of 1.09 and a t-statistic of .35. The value of the infit mean square near 1 suggests that the person’s variance in responding to items is about what we would expect. On the other hand, student “110123” has a smaller mean square than we would expect (.70) and student “110409” has a larger mean square than we would expect (1.66). Higher mean squares suggest more randomness while lower mean squares suggest less randomness. Isolated values like these are merely indications of patterns we may want to explore further, for example in the Diagnostic Maps for these students, or item patterns in the Item Fit report, or in an analysis of differential item functioning.


Figure 3. Full Ability Estimates report summary.

The average raw score, maximum score, proficiency estimate, standard error, and fit statistics are shown in the row labeled "Average" at the end of the case proficiency detail. Below that, case count, the average and variance of the estimates, the (given) model variance, the person separation reliability index and Cronbach’s alpha are displayed. For EAP and DPV estimates, the maximum marginal likelihood (MML) reliability index is also shown.

An example of this summary information for the SEPUP project using MLE estimates is shown in Figure 4. The meanings of the summary statistics and associated equations are shown in the table below. Note that summary statistics are displayed at the end of the student detail report for each variable, and Cronbach’s Alpha, a total test statistic, is displayed at the end of the entire report.

Table 1. The meanings of the summary statistics and associated equations.

Statistic	Meaning
Student Count by Dimension	The number of students who had sufficient data to compute a proficiency estimate in that dimension.
Average Proficiency	Mean of the proficiency estimates on each dimension.
Proficiency Variances	Variance of the proficiency estimates on each dimension.
S.E. Mean	The standard error of the mean on the dimension. Computed as where N is the number of respondents.
Model Variance	The population variance.
Person Separation Reliability	This is an index of the consistency of the person measures (i.e., if the instrument were administered again); an IRT version of Cronbach’s alpha. It is computed as a ratio of the variance in “true” person abilities over the variance in the observed abilities. where is the variance of estimated abilites (), and is the average error variance (). While IRT person separation may be considered more exact than Cronbach’s alpha (which is a raw score index), sometimes Cronbach’s alpha produces a higher reliability index. This is usually due to differences in the error variance applied to extreme scores: small errors for Cronbach’s alpha and large errors for IRT.
MML Reliability	Maximum marginal likelihood reliability of the EAP estimates is computed as the variance of the current proficiency estimates over the model variance. This index is not computed for ML estimates. This approach, described in Mislevy, Beaton, Kaplan and Sheehan (1992), generates a ratio of the variance in EAP (or DPV) estimates from the sample over the estimated variance of the population.
Cronbach's alpha	This is an index of the variance in the true person scores over the variance of the observed raw scores. It is an indication of how much of the variance is due to person variance rather than to item variance; low alpha values (i.e., those below .70), may indicate that the items measure something other than the targeted latent variable. , or , where is the variance of the total test scores, is the variance of item i across all students, and I is the number of items in the model.

-

3. The Ability Estimates Reports may contain a large amount of data. In order to view the entire report, you may need to scroll down to the very bottom to view the summary values.
4. Since this report produces text file output as well as an output screen, you can print this report using Word or Notepad.
5. Close the map display by clicking on the close box, , in the upper right-hand corner.