--- title: "GRASS methods companion: notation, calibration detail, and auxiliary diagnostics" author: - Austin Semmel - Rachel Gidaro date: "2026-07-04" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{GRASS methods companion: notation, calibration detail, and auxiliary diagnostics} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ## Scope of this companion This vignette accompanies the GRASS methods paper. It is a parallel reference: the paper is self-contained for review, while this companion carries the procedural detail and auxiliary diagnostics a practitioner running `grassr` may want at hand. Section numbering does not parallel the paper. Content moved here from earlier paper appendices during the 2026-05-04 appendix consolidation. **The paper does not cite this vignette**; this is a free-standing artifact for `grassr` users. --- ## 1. Notation glossary Symbols used throughout the package and the paper: - `Y` --- the $N \times k$ binary rating matrix; rows are subjects, columns are raters. - `pi_hat` --- observed marginal positive rate. - `tau2_hat` --- between-subject variance of the per-subject mean call rate (sample-side, computed from `Y`). - `Se_j`, `Sp_j` --- per-rater sensitivity and specificity, inferred from the rating matrix, not against external truth. - `q` --- diagonal-DGP rater quality; $\mathrm{Se}_j = \mathrm{Sp}_j = q$ for every rater under the diagonal case the surfaces are calibrated on. - `q_hat` --- internal scaffolding only. Used by `position_on_surface()` to select the sampling distribution from which the percentile is read; not exposed on the Report Card. - `delta_hat` --- cross-coefficient surface-percentile spread. --- ## 2. Recovering $\hat q$ from an observed coefficient For each agreement coefficient, `position_on_surface()` inverts the closed-form expectation curve to recover $\hat q$ at the observed coefficient value. - **PABAK**: algebraic inverse, $\hat q = \tfrac{1}{2}(1 + \sqrt{\mathrm{PABAK}})$. - **Fleiss $\kappa$, Krippendorff $\alpha$, AC1, ICC**: 501-point $q$-grid lookup with linear interpolation between the two nearest grid points where the curve crosses the observed value. - **Out-of-range observations**: clamped to the boundary $q$ value with a `clamped = TRUE` flag. --- ## 3. ICC and the subject-prevalence distribution The Report Card prints ICC alongside PABAK, AC1, and Fleiss $\kappa$, but with a `[distribution-sensitive]` marker, and ICC is excluded from the cross-coefficient stability signal $\hat\delta$ by construction. The agreement-family coefficients each have a closed-form expectation that depends only on the panel marginal $\hat\pi_+$ and the rater quality $q$. ICC depends additionally on the full distribution $F$ of subject-level positive probabilities $p_i$ across the rated subjects --- the shape of how the underlying positive rate varies from one subject to the next. Two studies can both have an overall positive rate of 30% and yet differ on whether that 30% comes from a clear class of obvious-positives (high between-subject variance) or from a mass of borderline-50/50 cases (low between-subject variance). The agreement-family coefficients can't tell those two studies apart at the panel level. ICC can, and so its reference surface must commit to a specific $F$. The bundled `grassr` reference covers a 48-point logit-normal grid in $(\mu, \tau^2)$ plus four discrete-mixture profiles (uniform, bimodal-low-high, skew-rare-positive, and similar shapes). A panel whose true $F$ sits cleanly inside that grid reads ICC's surface percentile faithfully. A panel whose $F$ sits between grid points or under a different family carries a distribution-mismatch cost that the agreement-family percentiles do not. This is why ICC carries a `[distribution-sensitive]` marker on the Report Card and does not enter $\hat\delta$. **Variant selection** (ICC(1,1), ICC(2,1), ICC(3,1)) follows the Koo and Li (2016) decision tree on both the inter-rater and intra-rater axes. The package exposes each variant through the `metric` argument of `position_on_surface()`. The four-question decision tree (same-rater-set / rater sampling / type / definition) is reproduced with worked examples per branch in the package help files. --- ## 4. Coefficient recovery: the full diagnostic The paper's App C carries the lead recovery table plus the asymmetry-cost figures. This companion expands the same dataset across the full grid. --- ## 5. $\hat\delta$ thresholds and operating characteristics The paper's Appendix~D carries the per-$(k, N)$ calibrated threshold table, the adopted-thresholds at the modal applied design, the TPR-by-asymmetry-magnitude tables for both caution and divergent flags, and the shared-bias vs split-bias detection comparison. This section populates the practitioner-facing detail that didn't make the paper. **How the thresholds are derived.** At each $(k, N)$ design point, the cuts are size-$\alpha$ calibrated against the diagonal-symmetric null distribution of $\hat\delta$ following the Neyman-Pearson construction: pick the smallest $t$ such that $P(\hat\delta > t \mid \text{symmetric panel at } (k, N)) \leq \alpha$ on the calibration grid, with $\alpha = 0.05$ for caution and $\alpha = 0.01$ for divergent. The thresholds are therefore a function of $(k, N)$, not a fixed pair. At the modal applied design ($k = 5$, $N = 200$) the cuts are $(9.25, 10.75)$ pp; at smaller designs they shift upward (the snapped pair at $k = 5$, $N = 50$ is $(10.25, 16.00)$ pp). **Small-design behavior.** At $k = 2$ and at $k = 3$ with $N \leq 30$, the diagonal-null distribution of $\hat\delta$ has heavy tails that no size-$\alpha$ cut controls. The lookup snaps such panels to the nearest fully-calibrated cell on the $(k, \log_{10} N)$ grid and surfaces the snap on the Report Card; this is an optimistic lower bound rather than a calibrated cut at the user's design. At $k = 2$ the snap collapses to $k = 3$ at the same $N$, because the panel-level coefficient battery at $k = 2$ (PABAK, AC1, Cohen's $\kappa$) measures effectively the same agreement structure under both the symmetric null and the asymmetric alternatives. This is why the framework's $k = 2$ prescription routes to direct pairwise reporting and identifiable bounds on per-rater $(\Se_j, \Sp_j)$ rather than relying on $\hat\delta$ to fire. **What $\hat\delta$ targets.** $\hat\delta$ targets cross-coefficient surface-percentile inversion --- the asymmetry pattern in which different chance-correction baselines respond differently to per-rater heterogeneity. Independently-drawn-sign asymmetry produces this inversion; uniform shared bias, conditional dependence, and other panel pathologies that move all coefficients together do not. Paper Appendix~D documents the gap: at $A = 0.20$ the divergent flag's TPR plateaus near $0.40$--$0.46$ under split-bias asymmetry, while shared-bias panels move all three coefficients together and leave $\hat\delta$ nearly unmoved; the percentile, not $\hat\delta$, carries the shared-bias signal. --- ## 5b. Pairwise PABAK and pooled-reference recovery When the divergent flag fires, the Report Card abandons the panel-aggregate summary and routes to a pairwise PABAK matrix plus per-rater pooled-reference $(\widetilde\Se_j, \widetilde\Sp_j)$. **Why PABAK at the pair level.** PABAK is the framework's pairwise coefficient because under symmetric raters its closed-form expectation reduces to $E[\mathrm{PABAK}] = (2q-1)^2$, a function of rater quality $q$ alone --- independent of prevalence and of the subject-prevalence distribution $F$. Pairwise reading needs that property because every rater pair has its own observed marginal $\hat\pi_{+, ij}$, and those per-pair marginals can move substantially with rater asymmetry. Cohen's $\kappa$ at the pair level is prevalence-attenuated (the $\kappa$ paradox of Byrt, Bishop, and Carlin (1993), the original motivation for PABAK); AC1's chance term carries a milder but non-zero prevalence dependence; ICC at $k = 2$ is degenerate because one pair of raters per subject leaves the variance decomposition under-identified. **Pooled-reference identifiability.** The pooled-reference estimates are label-flip clean conditional on the panel-majority direction matching the truth direction: the reference is the observable panel majority rather than a latent class, so the estimates do not depend on which mode an EM solver lands in. They read each rater's profile relative to whichever direction dominates the panel. Under a 2:3 panel where the minority direction happens to be the truth direction, the pooled-reference inverts and reads each rater's profile backwards. **Empirical recovery (54-cell Monte Carlo).** A sweep over $k \in \{3, 5, 7\}$, $\pi \in \{0.10, 0.30, 0.50\}$, asymmetry magnitude $A \in \{0.10, 0.20, 0.30\}$, and both shared-bias and split-bias configurations at $N = 200$ with $500$ replications per cell: - Pooled-reference recovers each rater's true bias direction in $94\%$ of split-bias in-majority-direction rater-replicates (rate improving in $k$ from $90\%$ at $k = 3$ to $97\%$ at $k = 7$). - Correctly inverts the direction in $98\%$ of minority-direction rater-replicates --- exactly as the conditional predicts. - Mean bias on $\widetilde\Se_j$ is $0.06$ and on $\widetilde\Sp_j$ is $0.03$ at $A \geq 0.20$. The pooled-reference summary is meaningful only at $k \geq 3$ (no rest-of-panel to pool against at $k = 2$) and is noisier on subjects where the panel is split. The package reports the per-rater pool size $\widetilde n_j$ alongside each pooled estimate so the reader can see how much information backs each one. --- ## 6. Worked code examples per Report Card branch A small helper generates an `N x k` rating matrix under the paper's diagonal DGP at a stated `seed`, `k`, `N`, per-rater `Se`/`Sp`, target prevalence `pi`, and logit-normal spread parameter `F_sigma2`: ```r library(grassr) simulate_panel <- function(seed, k, N, Se, Sp, pi, F_sigma2 = 0.25) { stopifnot(length(Se) == k, length(Sp) == k) set.seed(seed) mu_logit <- qlogis(pi) p_i <- plogis(rnorm(N, mu_logit, sqrt(F_sigma2))) C <- rbinom(N, 1L, p_i) Y <- matrix(0L, N, k) for (j in seq_len(k)) { Y[, j] <- rbinom(N, 1L, ifelse(C == 1L, Se[j], 1 - Sp[j])) } Y } ``` ### Aligned case (paper §4.1) ```r # Se = Sp = 0.85, k = 5, N = 1000, seed = 5 Y_aligned <- simulate_panel( seed = 5L, k = 5L, N = 1000L, Se = rep(0.85, 5), Sp = rep(0.85, 5), pi = 0.08 ) card_aligned <- grass_report(Y_aligned) card_aligned #> GRASS Report Card #> #> sample = 5 raters, N = 1000, pi_hat = 0.21 #> PABAK = 0.49 -> 46th percentile (decisive) <- primary #> AC1 = 0.62 -> 46th percentile #> Fleiss kappa = 0.23 -> 47th percentile #> ICC = 0.33 -> 50th percentile [distribution-sensitive] #> delta = 1 pp (aligned) #> thresholds = (8.50, 9.75) [calibrated at nearest grid cell (snapped)] #> #> Notes: #> - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500). #> - Fitted-ICC F_key picked via glmer: mu_hat=-1.716, tau2_hat=1.633 -> F_key tau2=1.0000, mu=-1.386. #> - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=-1.386_tau2=1.0000, k=5, N=1000 (family=logit_normal, M1=0.239). #> #> See `summary(...)` for full panel and CI details. #> See `plot(...)` for a surface-position visualization. ``` The three agreement coefficients land within ~1 pp of each other on their reference surfaces; the panel is *aligned* and the primary coefficient (PABAK) is safe to cite. ICC is reported alongside with `[distribution-sensitive]` since its reference surface depends on the full subject-prevalence distribution rather than on `(q, pi_hat)` alone. ### Divergent case (paper §4.2) ```r # op_strong panel: R1,R3,R5 at Se=0.95/Sp=0.75; R2,R4 inverted # k = 5, N = 1000, seed = 6, balanced prevalence Y_divergent <- simulate_panel( seed = 6L, k = 5L, N = 1000L, Se = c(0.95, 0.75, 0.95, 0.75, 0.95), Sp = c(0.75, 0.95, 0.75, 0.95, 0.75), pi = 0.50 ) card_divergent <- grass_report(Y_divergent) card_divergent #> GRASS Report Card #> #> sample = 5 raters, N = 1000, pi_hat = 0.50 #> PABAK = 0.49 -> 40th percentile <- primary #> AC1 = 0.49 -> 58th percentile #> Fleiss kappa = 0.49 -> 41st percentile #> ICC = 0.62 -> 50th percentile [distribution-sensitive] #> panel-agg. = suppressed (divergent) #> delta = 18 pp (divergent) #> thresholds = (8.50, 9.75) [calibrated at nearest grid cell (snapped)] #> #> pairwise PABAK / surface percentile (lower / upper): #> R1 R2 R3 R4 R5 #> R1 -- 25% 79% 43% 98% #> R2 0.47 -- 97% 90% 17% #> R3 0.51 0.41 -- 98% 96% #> R4 0.48 0.52 0.42 -- 62% #> R5 0.55 0.46 0.54 0.50 -- #> #> per-rater vs panel-majority of OTHER raters (pooled-reference): #> R1 Se_tilde = 0.94 Sp_tilde = 0.77 (n_pos = 434, n_neg = 471, excl = 95) #> R2 Se_tilde = 0.71 Sp_tilde = 0.95 (n_pos = 466, n_neg = 439, excl = 95) #> R3 Se_tilde = 0.95 Sp_tilde = 0.71 (n_pos = 433, n_neg = 485, excl = 82) #> R4 Se_tilde = 0.74 Sp_tilde = 0.95 (n_pos = 468, n_neg = 441, excl = 91) #> R5 Se_tilde = 0.96 Sp_tilde = 0.78 (n_pos = 428, n_neg = 476, excl = 96) #> #> per-rater (latent-class fit; alongside pairwise): #> R1 Se = 0.95 (0.21, 0.97) Sp = 0.78 (0.05, 0.82) #> R2 Se = 0.73 (0.05, 0.77) Sp = 0.95 (0.27, 0.97) #> R3 Se = 0.96 (0.27, 0.98) Sp = 0.72 (0.04, 0.76) #> R4 Se = 0.75 (0.05, 0.79) Sp = 0.95 (0.25, 0.97) #> R5 Se = 0.96 (0.22, 0.98) Sp = 0.78 (0.04, 0.82) #> #> Notes: #> - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500). #> - Fitted-ICC F_key picked via glmer: mu_hat=0.005, tau2_hat=5.278 -> F_key tau2=4.0000, mu=0.000. #> - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=+0.000_tau2=4.0000, k=5, N=1000 (family=logit_normal, M1=0.500). #> #> See `summary(...)` for full panel and CI details. #> See `plot(...)` for a surface-position visualization. ``` All three agreement coefficients return values around 0.49 (uniformly *moderate* by Landis-Koch) but the surface percentiles spread by ~18 pp: AC1 lands materially higher than the κ-family. The *divergent* flag fires; the panel-aggregate summary is suppressed; the Report Card emits the pairwise PABAK matrix and per-rater pooled-reference `(Se_tilde, Sp_tilde)` against the panel-majority of the other `k - 1` raters. ### $k = 2$ counterexample (paper §4.3) ```r # High-quality k = 2 panel: Se = Sp = 0.97, pi = 0.08, N = 200 Y_k2 <- simulate_panel( seed = 17L, k = 2L, N = 200L, Se = rep(0.97, 2), Sp = rep(0.97, 2), pi = 0.08 ) card_k2 <- grass_report(Y_k2) card_k2 #> GRASS Report Card #> #> sample = 2 raters, N = 200, pi_hat = 0.11 #> PABAK = 0.89 -> 50th percentile #> AC1 = 0.93 -> 55th percentile (decisive) <- primary #> delta = 5.4 pp (aligned) #> thresholds = (8.25, 9.75) [calibrated at nearest grid cell (snapped)] #> #> Notes: #> - (k = 2, N = 200) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 3, N = 200). #> #> See `summary(...)` for full panel and CI details. #> See `plot(...)` for a surface-position visualization. ``` Single-pair high-quality panel; the bundled k = 2 surface (densified to 12 q-points in v0.5.2) places the panel near the median of its `(k, N, pi_hat)` reference cohort. --- ## 7. Audit pass-rate detail The paper's App B (Reproducibility, grids, and DGPs) summarizes audit pass rates at Bonferroni-corrected $|z| < 4.80$. This companion lists the per-cell audit results for reviewers wanting to verify any specific corner. --- ## 8. Confidence-interval methods detail The paper App B carries the load-bearing CI numbers (Monte-Carlo error on percentile, nearest-neighbour cost, bootstrap coverage at one calibration point). This companion adds the implementation detail. --- ## References The paper's bibliography is the authoritative source. Key methodological references this companion leans on: - Byrt, Bishop, Carlin (1993) — PABAK derivation. - Cohen (1960), Fleiss (1971) — $\kappa$ family. - Gwet (2008) — AC1. - Krippendorff (2011, 2019) — $\alpha$. - Koo & Li (2016) — ICC variant selection. - Hui & Walter (1980) — $k = 2$ identifiability bounds. - Dawid & Skene (1979) — latent-class fit. - Neyman & Pearson (1933) — size-controlled testing. (Full citations in `references.bib` next to the paper source.)