---
title: "GRASS methods companion: notation, calibration detail, and auxiliary diagnostics"
author:
  - Austin Semmel
  - Rachel Gidaro
date: "2026-07-04"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{GRASS methods companion: notation, calibration detail, and auxiliary diagnostics}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---



## Scope of this companion

This vignette accompanies the GRASS methods paper. It is a parallel
reference: the paper is self-contained for review, while this companion
carries the procedural detail and auxiliary diagnostics a practitioner
running `grassr` may want at hand. Section numbering does not parallel
the paper.

Content moved here from earlier paper appendices during the 2026-05-04
appendix consolidation. **The paper does not cite this vignette**; this
is a free-standing artifact for `grassr` users.

---

## 1. Notation glossary

Symbols used throughout the package and the paper:

- `Y` --- the $N \times k$ binary rating matrix; rows are subjects,
  columns are raters.
- `pi_hat` --- observed marginal positive rate.
- `tau2_hat` --- between-subject variance of the per-subject mean call
  rate (sample-side, computed from `Y`).
- `Se_j`, `Sp_j` --- per-rater sensitivity and specificity, inferred
  from the rating matrix, not against external truth.
- `q` --- diagonal-DGP rater quality; $\mathrm{Se}_j = \mathrm{Sp}_j = q$
  for every rater under the diagonal case the surfaces are calibrated on.
- `q_hat` --- internal scaffolding only. Used by `position_on_surface()`
  to select the sampling distribution from which the percentile is
  read; not exposed on the Report Card.
- `delta_hat` --- cross-coefficient surface-percentile spread.

---

## 2. Recovering $\hat q$ from an observed coefficient

For each agreement coefficient, `position_on_surface()` inverts the
closed-form expectation curve to recover $\hat q$ at the observed
coefficient value.

- **PABAK**: algebraic inverse, $\hat q = \tfrac{1}{2}(1 + \sqrt{\mathrm{PABAK}})$.
- **Fleiss $\kappa$, Krippendorff $\alpha$, AC1, ICC**: 501-point
  $q$-grid lookup with linear interpolation between the two nearest
  grid points where the curve crosses the observed value.
- **Out-of-range observations**: clamped to the boundary $q$ value
  with a `clamped = TRUE` flag.

---

## 3. ICC and the subject-prevalence distribution

The Report Card prints ICC alongside PABAK, AC1, and Fleiss
$\kappa$, but with a `[distribution-sensitive]` marker,
and ICC is excluded from the cross-coefficient stability signal
$\hat\delta$ by construction.

The agreement-family coefficients each have a closed-form
expectation that depends only on the panel marginal $\hat\pi_+$ and
the rater quality $q$. ICC depends additionally on the full
distribution $F$ of subject-level positive probabilities $p_i$ across
the rated subjects --- the shape of how the underlying positive rate
varies from one subject to the next. Two studies can both have an
overall positive rate of 30% and yet differ on whether that 30% comes
from a clear class of obvious-positives (high between-subject variance)
or from a mass of borderline-50/50 cases (low between-subject
variance). The agreement-family coefficients can't tell those two
studies apart at the panel level. ICC can, and so its reference
surface must commit to a specific $F$.

The bundled `grassr` reference covers a 48-point logit-normal grid in
$(\mu, \tau^2)$ plus four discrete-mixture profiles (uniform,
bimodal-low-high, skew-rare-positive, and similar shapes). A panel
whose true $F$ sits cleanly inside that grid reads ICC's surface
percentile faithfully. A panel whose $F$ sits between grid points or
under a different family carries a distribution-mismatch cost that the
agreement-family percentiles do not. This is why ICC carries a `[distribution-sensitive]`
marker on the Report Card and does not enter $\hat\delta$.

**Variant selection** (ICC(1,1), ICC(2,1), ICC(3,1)) follows the
Koo and Li (2016) decision tree on both the inter-rater and
intra-rater axes. The package exposes each variant through the
`metric` argument of `position_on_surface()`. The four-question
decision tree (same-rater-set / rater sampling / type / definition)
is reproduced with worked examples per branch in the package help
files.

---

## 4. Coefficient recovery: the full diagnostic

The paper's App C carries the lead recovery table
plus the asymmetry-cost figures. This
companion expands the same dataset across the full grid.

---

## 5. $\hat\delta$ thresholds and operating characteristics

The paper's Appendix~D carries the per-$(k, N)$ calibrated threshold
table, the adopted-thresholds at the modal applied design, the
TPR-by-asymmetry-magnitude tables for both caution and divergent
flags, and the shared-bias vs split-bias detection comparison. This
section populates the practitioner-facing detail that didn't make
the paper.

**How the thresholds are derived.** At each $(k, N)$ design point,
the cuts are size-$\alpha$ calibrated against the diagonal-symmetric
null distribution of $\hat\delta$ following the Neyman-Pearson
construction: pick the smallest $t$ such that
$P(\hat\delta > t \mid \text{symmetric panel at } (k, N)) \leq \alpha$
on the calibration grid, with $\alpha = 0.05$ for caution and
$\alpha = 0.01$ for divergent. The thresholds are therefore a function
of $(k, N)$, not a fixed pair. At the modal applied design ($k = 5$,
$N = 200$) the cuts are $(9.25, 10.75)$ pp; at smaller designs they
shift upward (the snapped pair at $k = 5$, $N = 50$ is
$(10.25, 16.00)$ pp).

**Small-design behavior.** At $k = 2$ and at $k = 3$ with
$N \leq 30$, the diagonal-null distribution of $\hat\delta$ has heavy
tails that no size-$\alpha$ cut controls. The lookup snaps such
panels to the nearest fully-calibrated cell on the
$(k, \log_{10} N)$ grid and surfaces the snap on the Report Card; this
is an optimistic lower bound rather than a calibrated cut at the
user's design. At $k = 2$ the snap collapses to $k = 3$ at the same
$N$, because the panel-level coefficient battery at $k = 2$ (PABAK, AC1,
Cohen's $\kappa$) measures effectively the same
agreement structure under both the symmetric null and the asymmetric
alternatives. This is why the framework's $k = 2$ prescription routes
to direct pairwise reporting and identifiable bounds on per-rater
$(\Se_j, \Sp_j)$ rather than relying on $\hat\delta$ to fire.

**What $\hat\delta$ targets.** $\hat\delta$ targets cross-coefficient
surface-percentile inversion --- the asymmetry pattern in which
different chance-correction baselines respond differently to per-rater
heterogeneity. Independently-drawn-sign asymmetry produces this
inversion; uniform shared bias, conditional dependence, and other
panel pathologies that move all coefficients together do not. Paper Appendix~D documents the gap: at $A = 0.20$ the divergent
flag's TPR plateaus near $0.40$--$0.46$ under split-bias asymmetry,
while shared-bias panels move all three coefficients together and
leave $\hat\delta$ nearly unmoved; the percentile, not $\hat\delta$,
carries the shared-bias signal.

---

## 5b. Pairwise PABAK and pooled-reference recovery

When the divergent flag fires, the Report Card abandons the
panel-aggregate summary and routes to a pairwise PABAK matrix plus
per-rater pooled-reference $(\widetilde\Se_j, \widetilde\Sp_j)$.

**Why PABAK at the pair level.** PABAK is the framework's pairwise
coefficient because under symmetric raters its closed-form expectation
reduces to $E[\mathrm{PABAK}] = (2q-1)^2$, a function of rater quality $q$
alone --- independent of prevalence and of the subject-prevalence
distribution $F$. Pairwise
reading needs that property because every rater pair has its own
observed marginal $\hat\pi_{+, ij}$, and those per-pair marginals can
move substantially with rater asymmetry. Cohen's $\kappa$ at the pair
level is prevalence-attenuated (the $\kappa$ paradox of
Byrt, Bishop, and Carlin (1993), the original motivation for PABAK); AC1's chance
term carries a milder but non-zero prevalence dependence; ICC at
$k = 2$ is degenerate because one pair of raters per subject leaves
the variance decomposition under-identified.

**Pooled-reference identifiability.** The pooled-reference estimates
are label-flip clean conditional on the panel-majority direction
matching the truth direction: the reference is the observable panel
majority rather than a latent class, so the estimates do not depend on
which mode an EM solver lands in. They read each rater's profile
relative to whichever direction dominates the panel. Under a 2:3
panel where the minority direction happens to be the truth direction,
the pooled-reference inverts and reads each rater's profile backwards.

**Empirical recovery (54-cell Monte Carlo).** A sweep over
$k \in \{3, 5, 7\}$, $\pi \in \{0.10, 0.30, 0.50\}$, asymmetry
magnitude $A \in \{0.10, 0.20, 0.30\}$, and both shared-bias and
split-bias configurations at $N = 200$ with $500$ replications per
cell:

- Pooled-reference recovers each rater's true bias direction in $94\%$
  of split-bias in-majority-direction rater-replicates (rate improving
  in $k$ from $90\%$ at $k = 3$ to $97\%$ at $k = 7$).
- Correctly inverts the direction in $98\%$ of minority-direction
  rater-replicates --- exactly as the conditional predicts.
- Mean bias on $\widetilde\Se_j$ is $0.06$ and on $\widetilde\Sp_j$ is
  $0.03$ at $A \geq 0.20$.

The pooled-reference summary is meaningful only at $k \geq 3$ (no
rest-of-panel to pool against at $k = 2$) and is noisier on subjects
where the panel is split. The package reports the per-rater pool size
$\widetilde n_j$ alongside each pooled estimate so the reader can see
how much information backs each one.

---

## 6. Worked code examples per Report Card branch

A small helper generates an `N x k` rating matrix under the
paper's diagonal DGP at a stated `seed`, `k`, `N`, per-rater
`Se`/`Sp`, target prevalence `pi`, and logit-normal spread
parameter `F_sigma2`:


```r
library(grassr)

simulate_panel <- function(seed, k, N, Se, Sp, pi, F_sigma2 = 0.25) {
  stopifnot(length(Se) == k, length(Sp) == k)
  set.seed(seed)
  mu_logit <- qlogis(pi)
  p_i <- plogis(rnorm(N, mu_logit, sqrt(F_sigma2)))
  C   <- rbinom(N, 1L, p_i)
  Y   <- matrix(0L, N, k)
  for (j in seq_len(k)) {
    Y[, j] <- rbinom(N, 1L, ifelse(C == 1L, Se[j], 1 - Sp[j]))
  }
  Y
}
```

### Aligned case (paper §4.1)


```r
# Se = Sp = 0.85, k = 5, N = 1000, seed = 5
Y_aligned <- simulate_panel(
  seed = 5L, k = 5L, N = 1000L,
  Se = rep(0.85, 5), Sp = rep(0.85, 5),
  pi = 0.08
)
card_aligned <- grass_report(Y_aligned)
card_aligned
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.21
#>   PABAK        = 0.49  ->  46th percentile  (decisive)  <- primary
#>   AC1          = 0.62  ->  46th percentile
#>   Fleiss kappa = 0.23  ->  47th percentile
#>   ICC          = 0.33  ->  50th percentile  [distribution-sensitive]
#>   delta       =   1 pp (aligned)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=-1.716, tau2_hat=1.633 -> F_key tau2=1.0000, mu=-1.386.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=-1.386_tau2=1.0000, k=5, N=1000 (family=logit_normal, M1=0.239).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.
```

The three agreement coefficients land within ~1 pp of each
other on their reference surfaces; the panel is *aligned* and
the primary coefficient (PABAK) is safe to cite. ICC is
reported alongside with `[distribution-sensitive]` since its
reference surface depends on the full subject-prevalence
distribution rather than on `(q, pi_hat)` alone.

### Divergent case (paper §4.2)


```r
# op_strong panel: R1,R3,R5 at Se=0.95/Sp=0.75; R2,R4 inverted
# k = 5, N = 1000, seed = 6, balanced prevalence
Y_divergent <- simulate_panel(
  seed = 6L, k = 5L, N = 1000L,
  Se = c(0.95, 0.75, 0.95, 0.75, 0.95),
  Sp = c(0.75, 0.95, 0.75, 0.95, 0.75),
  pi = 0.50
)
card_divergent <- grass_report(Y_divergent)
card_divergent
#> GRASS Report Card
#> 
#>   sample      = 5 raters, N = 1000, pi_hat = 0.50
#>   PABAK         = 0.49  ->  40th percentile  <- primary
#>   AC1           = 0.49  ->  58th percentile
#>   Fleiss kappa  = 0.49  ->  41st percentile
#>   ICC           = 0.62  ->  50th percentile  [distribution-sensitive]
#>   panel-agg.  = suppressed (divergent)
#>   delta       =  18 pp (divergent)
#>   thresholds  = (8.50, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   pairwise PABAK / surface percentile (lower / upper):
#>     R1     R2     R3     R4     R5    
#>   R1    --     25%    79%    43%    98% 
#>   R2    0.47   --     97%    90%    17% 
#>   R3    0.51   0.41   --     98%    96% 
#>   R4    0.48   0.52   0.42   --     62% 
#>   R5    0.55   0.46   0.54   0.50   --  
#> 
#>   per-rater vs panel-majority of OTHER raters (pooled-reference):
#>     R1   Se_tilde = 0.94  Sp_tilde = 0.77  (n_pos = 434, n_neg = 471, excl = 95)
#>     R2   Se_tilde = 0.71  Sp_tilde = 0.95  (n_pos = 466, n_neg = 439, excl = 95)
#>     R3   Se_tilde = 0.95  Sp_tilde = 0.71  (n_pos = 433, n_neg = 485, excl = 82)
#>     R4   Se_tilde = 0.74  Sp_tilde = 0.95  (n_pos = 468, n_neg = 441, excl = 91)
#>     R5   Se_tilde = 0.96  Sp_tilde = 0.78  (n_pos = 428, n_neg = 476, excl = 96)
#> 
#>   per-rater (latent-class fit; alongside pairwise):
#>     R1   Se = 0.95  (0.21, 0.97)   Sp = 0.78  (0.05, 0.82)
#>     R2   Se = 0.73  (0.05, 0.77)   Sp = 0.95  (0.27, 0.97)
#>     R3   Se = 0.96  (0.27, 0.98)   Sp = 0.72  (0.04, 0.76)
#>     R4   Se = 0.75  (0.05, 0.79)   Sp = 0.95  (0.25, 0.97)
#>     R5   Se = 0.96  (0.22, 0.98)   Sp = 0.78  (0.04, 0.82)
#> 
#>   Notes:
#>     - (k = 5, N = 1000) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 5, N = 500).
#>     - Fitted-ICC F_key picked via glmer: mu_hat=0.005, tau2_hat=5.278 -> F_key tau2=4.0000, mu=0.000.
#>     - Fitted-ICC reference (GLMM-gap corrected) at F_key=LN_mu=+0.000_tau2=4.0000, k=5, N=1000 (family=logit_normal, M1=0.500).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.
```

All three agreement coefficients return values around 0.49
(uniformly *moderate* by Landis-Koch) but the surface
percentiles spread by ~18 pp: AC1 lands materially higher
than the κ-family. The *divergent* flag fires; the
panel-aggregate summary is suppressed; the Report Card emits
the pairwise PABAK matrix and per-rater pooled-reference
`(Se_tilde, Sp_tilde)` against the panel-majority of the
other `k - 1` raters.

### $k = 2$ counterexample (paper §4.3)


```r
# High-quality k = 2 panel: Se = Sp = 0.97, pi = 0.08, N = 200
Y_k2 <- simulate_panel(
  seed = 17L, k = 2L, N = 200L,
  Se = rep(0.97, 2), Sp = rep(0.97, 2),
  pi = 0.08
)
card_k2 <- grass_report(Y_k2)
card_k2
#> GRASS Report Card
#> 
#>   sample      = 2 raters, N = 200, pi_hat = 0.11
#>   PABAK  = 0.89  ->  50th percentile
#>   AC1    = 0.93  ->  55th percentile  (decisive)  <- primary
#>   delta       = 5.4 pp (aligned)
#>   thresholds  = (8.25, 9.75) [calibrated at nearest grid cell (snapped)]
#> 
#>   Notes:
#>     - (k = 2, N = 200) has no fully-calibrated threshold pair on the grid (or is off-grid); using nearest fully-calibrated cell (k = 3, N = 200).
#> 
#>   See `summary(...)` for full panel and CI details.
#>   See `plot(...)` for a surface-position visualization.
```

Single-pair high-quality panel; the bundled k = 2 surface
(densified to 12 q-points in v0.5.2) places the panel near
the median of its `(k, N, pi_hat)` reference cohort.

---

## 7. Audit pass-rate detail

The paper's App B (Reproducibility, grids, and DGPs) summarizes audit
pass rates at Bonferroni-corrected $|z| < 4.80$. This companion lists
the per-cell audit results for reviewers wanting to verify any
specific corner.

---

## 8. Confidence-interval methods detail

The paper App B carries the load-bearing CI numbers (Monte-Carlo error
on percentile, nearest-neighbour cost, bootstrap coverage at one
calibration point). This companion adds the implementation detail.

---

## References

The paper's bibliography is the authoritative source. Key
methodological references this companion leans on:

- Byrt, Bishop, Carlin (1993) — PABAK derivation.
- Cohen (1960), Fleiss (1971) — $\kappa$ family.
- Gwet (2008) — AC1.
- Krippendorff (2011, 2019) — $\alpha$.
- Koo & Li (2016) — ICC variant selection.
- Hui & Walter (1980) — $k = 2$ identifiability bounds.
- Dawid & Skene (1979) — latent-class fit.
- Neyman & Pearson (1933) — size-controlled testing.

(Full citations in `references.bib` next to the paper source.)
