GRASS methods companion: notation, calibration detail, and auxiliary diagnostics

Scope of this companion

This vignette accompanies the GRASS methods paper. It is a parallel reference: the paper is self-contained for review, while this companion carries the procedural detail and auxiliary diagnostics a practitioner running grassr may want at hand. Section numbering does not parallel the paper.

Content moved here from earlier paper appendices during the 2026-05-04 appendix consolidation. The paper does not cite this vignette; this is a free-standing artifact for grassr users.

1. Notation glossary

Symbols used throughout the package and the paper:

• Y — the \(N \times k\) binary rating matrix; rows are subjects, columns are raters.
• pi_hat — observed marginal positive rate.
• tau2_hat — between-subject variance of the per-subject mean call rate (sample-side, computed from Y).
• Se_j, Sp_j — per-rater sensitivity and specificity, inferred from the rating matrix, not against external truth.
• q — diagonal-DGP rater quality; \(\mathrm{Se}_j = \mathrm{Sp}_j = q\) for every rater under the diagonal case the surfaces are calibrated on.
• q_hat — internal scaffolding only. Used by position_on_surface() to select the sampling distribution from which the percentile is read; not exposed on the Report Card.
• delta_hat — cross-coefficient surface-percentile spread.

2. Recovering \(\hat q\) from an observed coefficient

For each agreement coefficient, position_on_surface() inverts the closed-form expectation curve to recover \(\hat q\) at the observed coefficient value.

• PABAK: algebraic inverse, \(\hat q = \tfrac{1}{2}(1 + \sqrt{\mathrm{PABAK}})\).
• Fleiss \(\kappa\), Krippendorff \(\alpha\), AC1, ICC: 501-point \(q\)-grid lookup with linear interpolation between the two nearest grid points where the curve crosses the observed value.
• Out-of-range observations: clamped to the boundary \(q\) value with a clamped = TRUE flag.

3. ICC and the subject-prevalence distribution

The Report Card prints ICC alongside PABAK, AC1, and Fleiss \(\kappa\), but with a [distribution-sensitive] marker, and ICC is excluded from the cross-coefficient stability signal \(\hat\delta\) by construction.

The agreement-family coefficients each have a closed-form expectation that depends only on the panel marginal \(\hat\pi_+\) and the rater quality \(q\). ICC depends additionally on the full distribution \(F\) of subject-level positive probabilities \(p_i\) across the rated subjects — the shape of how the underlying positive rate varies from one subject to the next. Two studies can both have an overall positive rate of 30% and yet differ on whether that 30% comes from a clear class of obvious-positives (high between-subject variance) or from a mass of borderline-50/50 cases (low between-subject variance). The agreement-family coefficients can’t tell those two studies apart at the panel level. ICC can, and so its reference surface must commit to a specific \(F\).

The bundled grassr reference covers a 48-point logit-normal grid in \((\mu, \tau^2)\) plus four discrete-mixture profiles (uniform, bimodal-low-high, skew-rare-positive, and similar shapes). A panel whose true \(F\) sits cleanly inside that grid reads ICC’s surface percentile faithfully. A panel whose \(F\) sits between grid points or under a different family carries a distribution-mismatch cost that the agreement-family percentiles do not. This is why ICC carries a [distribution-sensitive] marker on the Report Card and does not enter \(\hat\delta\).

Variant selection (ICC(1,1), ICC(2,1), ICC(3,1)) follows the Koo and Li (2016) decision tree on both the inter-rater and intra-rater axes. The package exposes each variant through the metric argument of position_on_surface(). The four-question decision tree (same-rater-set / rater sampling / type / definition) is reproduced with worked examples per branch in the package help files.

4. Coefficient recovery: the full diagnostic

The paper’s App C carries the lead recovery table plus the asymmetry-cost figures. This companion expands the same dataset across the full grid.

5. \(\hat\delta\) thresholds and operating characteristics

The paper’s Appendix~D carries the per-\((k, N)\) calibrated threshold table, the adopted-thresholds at the modal applied design, the TPR-by-asymmetry-magnitude tables for both caution and divergent flags, and the shared-bias vs split-bias detection comparison. This section populates the practitioner-facing detail that didn’t make the paper.

How the thresholds are derived. At each \((k, N)\) design point, the cuts are size-\(\alpha\) calibrated against the diagonal-symmetric null distribution of \(\hat\delta\) following the Neyman-Pearson construction: pick the smallest \(t\) such that \(P(\hat\delta > t \mid \text{symmetric panel at } (k, N)) \leq \alpha\) on the calibration grid, with \(\alpha = 0.05\) for caution and \(\alpha = 0.01\) for divergent. The thresholds are therefore a function of \((k, N)\), not a fixed pair. At the modal applied design (\(k = 5\), \(N = 200\)) the cuts are \((9.25, 10.75)\) pp; at smaller designs they shift upward (the snapped pair at \(k = 5\), \(N = 50\) is \((10.25, 16.00)\) pp).

Small-design behavior. At \(k = 2\) and at \(k = 3\) with \(N \leq 30\), the diagonal-null distribution of \(\hat\delta\) has heavy tails that no size-\(\alpha\) cut controls. The lookup snaps such panels to the nearest fully-calibrated cell on the \((k, \log_{10} N)\) grid and surfaces the snap on the Report Card; this is an optimistic lower bound rather than a calibrated cut at the user’s design. At \(k = 2\) the snap collapses to \(k = 3\) at the same \(N\), because the panel-level coefficient battery at \(k = 2\) (PABAK, AC1, Cohen’s \(\kappa\)) measures effectively the same agreement structure under both the symmetric null and the asymmetric alternatives. This is why the framework’s \(k = 2\) prescription routes to direct pairwise reporting and identifiable bounds on per-rater \((\Se_j, \Sp_j)\) rather than relying on \(\hat\delta\) to fire.

What \(\hat\delta\) targets. \(\hat\delta\) targets cross-coefficient surface-percentile inversion — the asymmetry pattern in which different chance-correction baselines respond differently to per-rater heterogeneity. Independently-drawn-sign asymmetry produces this inversion; uniform shared bias, conditional dependence, and other panel pathologies that move all coefficients together do not. Paper Appendix~D documents the gap: at \(A = 0.20\) the divergent flag’s TPR plateaus near \(0.40\)–\(0.46\) under split-bias asymmetry, while shared-bias panels move all three coefficients together and leave \(\hat\delta\) nearly unmoved; the percentile, not \(\hat\delta\), carries the shared-bias signal.

5b. Pairwise PABAK and pooled-reference recovery

When the divergent flag fires, the Report Card abandons the panel-aggregate summary and routes to a pairwise PABAK matrix plus per-rater pooled-reference \((\widetilde\Se_j, \widetilde\Sp_j)\).

Why PABAK at the pair level. PABAK is the framework’s pairwise coefficient because under symmetric raters its closed-form expectation reduces to \(E[\mathrm{PABAK}] = (2q-1)^2\), a function of rater quality \(q\) alone — independent of prevalence and of the subject-prevalence distribution \(F\). Pairwise reading needs that property because every rater pair has its own observed marginal \(\hat\pi_{+, ij}\), and those per-pair marginals can move substantially with rater asymmetry. Cohen’s \(\kappa\) at the pair level is prevalence-attenuated (the \(\kappa\) paradox of Byrt, Bishop, and Carlin (1993), the original motivation for PABAK); AC1’s chance term carries a milder but non-zero prevalence dependence; ICC at \(k = 2\) is degenerate because one pair of raters per subject leaves the variance decomposition under-identified.

Pooled-reference identifiability. The pooled-reference estimates are label-flip clean conditional on the panel-majority direction matching the truth direction: the reference is the observable panel majority rather than a latent class, so the estimates do not depend on which mode an EM solver lands in. They read each rater’s profile relative to whichever direction dominates the panel. Under a 2:3 panel where the minority direction happens to be the truth direction, the pooled-reference inverts and reads each rater’s profile backwards.

Empirical recovery (54-cell Monte Carlo). A sweep over \(k \in \{3, 5, 7\}\), \(\pi \in \{0.10, 0.30, 0.50\}\), asymmetry magnitude \(A \in \{0.10, 0.20, 0.30\}\), and both shared-bias and split-bias configurations at \(N = 200\) with \(500\) replications per cell:

• Pooled-reference recovers each rater’s true bias direction in \(94\%\) of split-bias in-majority-direction rater-replicates (rate improving in \(k\) from \(90\%\) at \(k = 3\) to \(97\%\) at \(k = 7\)).
• Correctly inverts the direction in \(98\%\) of minority-direction rater-replicates — exactly as the conditional predicts.
• Mean bias on \(\widetilde\Se_j\) is \(0.06\) and on \(\widetilde\Sp_j\) is \(0.03\) at \(A \geq 0.20\).

The pooled-reference summary is meaningful only at \(k \geq 3\) (no rest-of-panel to pool against at \(k = 2\)) and is noisier on subjects where the panel is split. The package reports the per-rater pool size \(\widetilde n_j\) alongside each pooled estimate so the reader can see how much information backs each one.

6. Worked code examples per Report Card branch

A small helper generates an N x k rating matrix under the paper’s diagonal DGP at a stated seed, k, N, per-rater Se/Sp, target prevalence pi, and logit-normal spread parameter F_sigma2:

library(grassr)

simulate_panel <- function(seed, k, N, Se, Sp, pi, F_sigma2 = 0.25) {
  stopifnot(length(Se) == k, length(Sp) == k)
  set.seed(seed)
  mu_logit <- qlogis(pi)
  p_i <- plogis(rnorm(N, mu_logit, sqrt(F_sigma2)))
  C   <- rbinom(N, 1L, p_i)
  Y   <- matrix(0L, N, k)
  for (j in seq_len(k)) {
    Y[, j] <- rbinom(N, 1L, ifelse(C == 1L, Se[j], 1 - Sp[j]))
  }
  Y
}

# Se = Sp = 0.85, k = 5, N = 1000, seed = 5
Y_aligned <- simulate_panel(
  seed = 5L, k = 5L, N = 1000L,
  Se = rep(0.85, 5), Sp = rep(0.85, 5),
  pi = 0.08
)
card_aligned <- grass_report(Y_aligned)
card_aligned
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.21
#>   PABAK        = 0.49  ->  46th percentile  (decisive)  <- primary
#>   AC1          = 0.62  ->  46th percentile
#>   Fleiss kappa = 0.23  ->  47th percentile
#>   ICC          = 0.33  ->  50th percentile  [distribution-sensitive]
#>   delta       =   1 pp (aligned)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=-1.716, tau2_hat=1.633 -> F_key tau2=1.0000, mu=-1.386.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=-1.386_tau2=1.0000, k=5, N=1000 (family=logit_normal, M1=0.239).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

# op_strong panel: R1,R3,R5 at Se=0.95/Sp=0.75; R2,R4 inverted
# k = 5, N = 1000, seed = 6, balanced prevalence
Y_divergent <- simulate_panel(
  seed = 6L, k = 5L, N = 1000L,
  Se = c(0.95, 0.75, 0.95, 0.75, 0.95),
  Sp = c(0.75, 0.95, 0.75, 0.95, 0.75),
  pi = 0.50
)
card_divergent <- grass_report(Y_divergent)
card_divergent
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.50
#>   PABAK         = 0.49  ->  40th percentile  <- primary
#>   AC1           = 0.49  ->  58th percentile
#>   Fleiss kappa  = 0.49  ->  41st percentile
#>   ICC           = 0.62  ->  50th percentile  [distribution-sensitive]
#>   panel-agg.  = suppressed (divergent)
#>   delta       =  18 pp (divergent)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   pairwise PABAK / surface percentile (lower / upper):
#>     R1     R2     R3     R4     R5    
#>   R1    --     25%    79%    43%    98% 
#>   R2    0.47   --     97%    90%    17% 
#>   R3    0.51   0.41   --     98%    96% 
#>   R4    0.48   0.52   0.42   --     62% 
#>   R5    0.55   0.46   0.54   0.50   --  
#> 
#>   per-rater vs panel-majority of OTHER raters (pooled-reference):
#>     R1   Se_tilde = 0.94  Sp_tilde = 0.77  (n_pos = 434, n_neg = 471, excl = 95)
#>     R2   Se_tilde = 0.71  Sp_tilde = 0.95  (n_pos = 466, n_neg = 439, excl = 95)
#>     R3   Se_tilde = 0.95  Sp_tilde = 0.71  (n_pos = 433, n_neg = 485, excl = 82)
#>     R4   Se_tilde = 0.74  Sp_tilde = 0.95  (n_pos = 468, n_neg = 441, excl = 91)
#>     R5   Se_tilde = 0.96  Sp_tilde = 0.78  (n_pos = 428, n_neg = 476, excl = 96)
#> 
#>   per-rater (latent-class fit; alongside pairwise):
#>     R1   Se = 0.95  (0.21, 0.97)   Sp = 0.78  (0.05, 0.82)
#>     R2   Se = 0.73  (0.05, 0.77)   Sp = 0.95  (0.27, 0.97)
#>     R3   Se = 0.96  (0.27, 0.98)   Sp = 0.72  (0.04, 0.76)
#>     R4   Se = 0.75  (0.05, 0.79)   Sp = 0.95  (0.25, 0.97)
#>     R5   Se = 0.96  (0.22, 0.98)   Sp = 0.78  (0.04, 0.82)
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=0.005, tau2_hat=5.278 -> F_key tau2=4.0000, mu=0.000.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=+0.000_tau2=4.0000, k=5, N=1000 (family=logit_normal, M1=0.500).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

# High-quality k = 2 panel: Se = Sp = 0.97, pi = 0.08, N = 200
Y_k2 <- simulate_panel(
  seed = 17L, k = 2L, N = 200L,
  Se = rep(0.97, 2), Sp = rep(0.97, 2),
  pi = 0.08
)
card_k2 <- grass_report(Y_k2)
card_k2
#> GRASS Report Card
#> 
#>   sample      = 2 raters, N = 200, pi_hat = 0.11
#>   PABAK  = 0.89  ->  50th percentile
#>   AC1    = 0.93  ->  55th percentile  (decisive)  <- primary
#>   delta       = 5.4 pp (aligned)
#>   thresholds  = (8.25, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 2, N = 200) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 3, N = 200).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.

7. Audit pass-rate detail

The paper’s App B (Reproducibility, grids, and DGPs) summarizes audit pass rates at Bonferroni-corrected \(|z| < 4.80\). This companion lists the per-cell audit results for reviewers wanting to verify any specific corner.

8. Confidence-interval methods detail

The paper App B carries the load-bearing CI numbers (Monte-Carlo error on percentile, nearest-neighbour cost, bootstrap coverage at one calibration point). This companion adds the implementation detail.

References

The paper’s bibliography is the authoritative source. Key methodological references this companion leans on:

• Byrt, Bishop, Carlin (1993) — PABAK derivation.
• Cohen (1960), Fleiss (1971) — \(\kappa\) family.
• Gwet (2008) — AC1.
• Krippendorff (2011, 2019) — \(\alpha\).
• Koo & Li (2016) — ICC variant selection.
• Hui & Walter (1980) — \(k = 2\) identifiability bounds.
• Dawid & Skene (1979) — latent-class fit.
• Neyman & Pearson (1933) — size-controlled testing.

(Full citations in references.bib next to the paper source.)