# rcrtan

#### 2018-12-18

This package contains functions for criterion-referenced test and item analysis.

## Test Analysis: Dependability

The functions below can be used to examine the dependability of single administration CRT assessments.

### subkoviak

The subkoviak function follows Subkoviak’s (1988) single administration procedures for estimating $$\kappa$$ and po. To use the subkoviak function the data can come in one of two forms:

1. A dataframe of dichotomously scored items
2. A dataframe of total (raw) scores on the test. This also requires the analyst/researcher to know how many (dichotomous) items were on the test.

In both situations, the analyst/researcher needs to know the raw cut score of the test.

The procedure uses two pieces of information to estimate $$\kappa$$ and po. They are a reliability coefficient (estimated in this function using Kuder-Richardson 20 or 21) and a standardized cut-score, which is estimated following Subkoviak (1988). The estimated $$\alpha$$ coefficient and standardized cut-score are used to look up the values for $$\kappa$$ and po in look up tables, which the function will return if look_up = TRUE

$z = \frac{c - 0.5 - M}{S}$

where:

• z = The standardized cut-score
• c = The raw cut-score of the test
• M = The mean raw total score of the test
• S = The standard deviation of the raw total scores of the test

#### Examples: With item information


library(rcrtan)

# Some data
test_data <- tibble::tribble(
~ID, ~Q1, ~Q2, ~Q3, ~Q4, ~Q5, ~Q6, ~Q7, ~Q8, ~Q9, ~Q10, ~Total,
1441L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
1387L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
1994L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
1453L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
1679L,  1L,  0L,  1L,  0L,  0L,  1L,  1L,  1L,  1L,   1L,     7L,
1899L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
1631L,  1L,  1L,  1L,  1L,  1L,  1L,  0L,  1L,  1L,   1L,     9L,
1894L,  1L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,     9L,
1206L,  0L,  0L,  1L,  1L,  0L,  1L,  0L,  1L,  0L,   1L,     5L,
1163L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
1964L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
1050L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
1034L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
1826L,  1L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,     9L,
1973L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
1936L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   0L,     8L,
1114L,  1L,  0L,  0L,  1L,  1L,  0L,  0L,  0L,  1L,   1L,     5L,
1181L,  0L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,  1L,   1L,     7L,
1917L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L
)

# Analyze the dichomous data. Use look_up = TRUE if you want to see the look up tables from Subkoviak (1988).
sub_ex_one <- subkoviak(data = test_data, items = 2:11, raw_cut_score = 8)

The output from the analysis contains five variables.

1. z: The actual estimated standardized cut-score
2. z_rounded: The rounded standardized cut-score used in the table look up
3. KR_est: The estimated alpha coefficient using one the Kuder-Richardson methods
4. agree_coef.r_0.4: The estimated agreement (po) coefficient
5. kappa_coef.r_0.4: The estimated kappa ($$\kappa$$) coefficient
sub_ex_one
#>      z z_rounded    KR_est agree_coef.r_0.4 kappa_coef.r_0.4
#> 1 0.45       0.4 0.4250152             0.66             0.25

#### Examples: Without item information

When item-level information is not available the items argument takes the number of dichotomous items on the test, and the total argument takes the name of the column in the dataframe with the total score.

sub_ex_two <- subkoviak(data = test_data, items = 10, raw_cut_score = 8, total = "Total")

sub_ex_two
#>      z z_rounded    KR_est agree_coef.r_0.3 kappa_coef.r_0.3
#> 1 0.45       0.4 0.3309846             0.63             0.19

### phi_domain

The phi_domain function calculates Brown (1988) short-cut estimate of the generalizability coefficient ($$\phi$$ dependability) for absolute decisions.

# When item level information is known
phi_d_one <- phi_domain(data = test_data, items = 2:11)

# When only total scores and number of items on the test are known
phi_d_two <- phi_domain(data = test_data, items = 10, total = "Total")

phi_d_one
#>   Domain Phi
#> 1  0.3864201

phi_d_two
#>   Domain Phi
#> 1  0.3290602

### phi_lambda

The phi_lambda function uses Brennan (1984) formula to estimate the dependability of a test. One difference to note between this function and subkoviak is that the cut_score argument is expressed as a proportion.

phi_l_one <- rcrtan::phi_lambda(test_data, 2:11, cut_score = 0.80)

phi_l_two <- phi_lambda(test_data, 10, cut_score = 0.80, total = 'Total')

phi_l_one
#>   Phi Lambda
#> 1  0.2962963

phi_l_two
#>   Phi Lambda
#> 1  0.2962963

# References

Brennan, R. L. (1984). Estimating the dependability of the scores. In A guide to criterion-referenced test construction, (Ed.) Ronald A Berk. 231–66. Baltimore, MD: The Johns Hopkins University Press.

Brown, J. D. (1988). Short cut estimates of criterion-referenced test reliability. Language Testing, 7, 77–97.

Subkoviak, M. J. (1988). A practitioner’s guide to computation and interpretation of reliability indices for mastery tests. Journal of Educational Measurement, 25, 47–55.