| Title: | Pre-Data Recovery Tests for Planned Study Designs |
|---|---|
| Description: | A prototype implementation of the pre-data recovery test: a standardized simulation protocol for evaluating whether a planned design-analysis pair can recover its target estimand before data collection. Researchers declare the estimand, data-generating assumptions, data strategy, missing-data process, and analysis strategy; the package simulates a crossed scenario grid (null and target effects, each under declared and pessimistically perturbed nuisance assumptions), applies the planned analysis to each simulated dataset, and reports target-null rejection, power, target bias with its exact decomposition into estimator bias and estimand drift, coverage, Type S and Type M errors, precision, and classified model failure, each with Monte Carlo uncertainty and explicit inclusion rules. A versioned threshold profile converts the diagnosands into a PASS/RISK/FAIL verdict - a decision convention, not a validity classification. Built on the 'DeclareDesign' framework. |
| Authors: | Heidi Helena Andersen [aut, cre] (ORCID: <https://orcid.org/0000-0001-5923-5865>) |
| Maintainer: | Heidi Helena Andersen <[email protected]> |
| License: | Apache License (>= 2) |
| Version: | 0.2.0 |
| Built: | 2026-07-05 17:04:59 UTC |
| Source: | https://github.com/heidihelena/recoverlite |
Dropout follows a declared response model rather than an unstated one.
Under mechanism = "differential", the probability of dropout is
depending on assignment and the observed baseline — not on the
unobserved post-test outcome. Because dropout depends only on observed
quantities (Z, B), missingness is MAR given the baseline; this keeps
the drift mechanism transparent and makes baseline-based repairs
(e.g. planned_analysis() with estimator = "mi_baseline_adjusted")
identifiable for the ITT estimand. The arm-specific intercepts alpha_Z
are calibrated numerically so that each arm's marginal attrition rate
equals its declared value: arm imbalance in retention is a declared
quantity, not a by-product.
attrition_model( rate, mechanism = c("differential", "mcar"), rate_control = NULL, rate_treated = NULL, baseline_slope_treated = -0.5, baseline_slope_control = 0, max_rate = 0.6, evidence = NULL )attrition_model( rate, mechanism = c("differential", "mcar"), rate_control = NULL, rate_treated = NULL, baseline_slope_treated = -0.5, baseline_slope_control = 0, max_rate = 0.6, evidence = NULL )
rate |
Numeric in [0, 1). Anticipated overall marginal attrition rate; used for both arms unless arm-specific rates are given. |
mechanism |
Character. |
rate_control, rate_treated
|
Numeric or |
baseline_slope_treated |
Numeric. gamma_1, the log-odds change in
treated-arm dropout per baseline SD (default -0.5: poorer baseline
prognosis, higher dropout). Ignored under |
baseline_slope_control |
Numeric. gamma_0 (default 0). Ignored
under |
max_rate |
Numeric. Field-relevant cap applied when the pessimistic scenario multiplies the declared rates by 1.5; must be stated and justified. |
evidence |
Character or |
With a negative treated-arm slope and higher scores indicating better functioning, intervention participants with poorer baseline prognosis are more likely to drop out. This mechanism persists when the treatment effect is zero, so under null scenarios the analyzable-data contrast is displaced from 0 and rejections are false claims about the target.
mechanism = "mcar" drops observations completely at random at the
declared rate(s).
An object of class recovery_attrition.
Cluster-level random assignment with individual outcomes and cluster sizes fixed by design. For pupil i in cluster j,
with total individual-level variance standardized to 1 and
ICC = var(u) / (var(u) + var(e)). The declared effect is expressed in
individual-level (total) standard-deviation units. The simulated data
contain a cluster id column for use in the analysis formula, e.g.
y_observed ~ treatment + (1 | cluster).
cluster_trial( n_clusters, n_per_cluster, icc, icc_pessimistic = NULL, allocation = 0.5, evidence = NULL )cluster_trial( n_clusters, n_per_cluster, icc, icc_pessimistic = NULL, allocation = 0.5, evidence = NULL )
n_clusters |
Integer. Total number of clusters randomized. |
n_per_cluster |
Integer. Individuals measured per cluster. |
icc |
Numeric in [0, 1). Declared intraclass correlation. |
icc_pessimistic |
Numeric or |
allocation |
Numeric in (0, 1). Share of clusters assigned to treatment. |
evidence |
Character or |
An object of class recovery_data_strategy.
Step 2 of the protocol: assemble the target estimand, the model and data strategy, and the answer strategy into a single executable declaration. Every major design feature is either declared here or recorded as omitted in the report.
declare_recovery( target, data_strategy, measurement = NULL, missingness = NULL, answer_strategy, effect = NULL )declare_recovery( target, data_strategy, measurement = NULL, missingness = NULL, answer_strategy, effect = NULL )
target |
A |
data_strategy |
A |
measurement |
A |
missingness |
An |
answer_strategy |
A |
effect |
Numeric or |
An object of class recovery_design.
design <- declare_recovery( target = target_estimand( estimand = "ITT mean difference at 12 weeks", scale = "latent-outcome standardized mean difference", sesoi = 0.40 ), data_strategy = two_arm_trial(n_per_arm = 100), measurement = measured_outcome(reliability = 0.70), missingness = attrition_model(rate = 0.15, mechanism = "differential"), answer_strategy = planned_analysis( estimator = "linear_model", formula = y_observed ~ treatment ) )design <- declare_recovery( target = target_estimand( estimand = "ITT mean difference at 12 weeks", scale = "latent-outcome standardized mean difference", sesoi = 0.40 ), data_strategy = two_arm_trial(n_per_arm = 100), measurement = measured_outcome(reliability = 0.70), missingness = attrition_model(rate = 0.15, mechanism = "differential"), answer_strategy = planned_analysis( estimator = "linear_model", formula = y_observed ~ treatment ) )
Shows how power, Type M exaggeration, and precision behave as the true effect shrinks below the declared target, with all nuisance assumptions held at their declared values. This is deliberately outside the PASS/RISK/FAIL verdict: shrinking the target effect changes the inferential target, so pessimistic scenario rows perturb nuisance assumptions only, and fragility to the effect size itself is reported as a separate continuous curve.
effect_fragility(design, effects = NULL, sims = 500, seed = NULL)effect_fragility(design, effects = NULL, sims = 500, seed = NULL)
design |
A |
effects |
Numeric vector of true effects to evaluate. Defaults to an even grid from a quarter of the declared effect up to the declared effect, with the SESOI included. |
sims |
Simulations per effect value (default 500; increase for smoother curves). |
seed |
Optional integer seed. |
A data frame of class recovery_fragility with one row per
effect: power, Type M, and precision, each with its MCSE.
Classical additive measurement error, per the manuscript (section 3.1):
the observed outcome is y_observed = y_true + e with
Var(e) = 1/reliability - 1, so that reliability =
Var(y_true)/Var(y_observed). Under this model the raw treatment
contrast is not attenuated in expectation: measurement error inflates
residual variance (costing precision, power, and — downstream of low
power — Type M exaggeration), and is charged to the variance account,
not the bias account. (Attenuation by sqrt(reliability) would apply
only if the estimator standardized by the observed-outcome SD.)
measured_outcome(reliability, evidence = NULL)measured_outcome(reliability, evidence = NULL)
reliability |
Numeric in (0, 1]. Declared reliability (e.g. test-retest) of the outcome measure. |
evidence |
Character or |
If measurement is omitted from declare_recovery(), the outcome is
treated as perfectly reliable and the report states this explicitly
(silence must not imply ideality).
An object of class recovery_measurement.
Fragility curves over a nuisance parameter the diagnosis identifies as binding (manuscript, section 2.2, Step 3): the verdict uses the pre-specified point scenarios; these curves show where the verdict would change. The chosen diagnosands are power, target-null-relevant quantities being outside scope here, target bias, coverage, and estimand drift across the grid, at the target effect (the SESOI).
nuisance_fragility( design, parameter = c("attrition_rate", "reliability", "icc"), values, sims = 500, seed = NULL )nuisance_fragility( design, parameter = c("attrition_rate", "reliability", "icc"), values, sims = 500, seed = NULL )
design |
A |
parameter |
One of |
values |
Numeric grid of parameter values to evaluate. |
sims |
Simulations per grid point (default 500). |
seed |
Optional integer seed. |
A data frame of class recovery_fragility.
The analysis that will actually be applied to the observed data, stated exactly — including the specific inference method, since finite-sample behavior can differ sharply across methods that share a model formula (manuscript, section 3.2).
planned_analysis( estimator = c("linear_model", "lmm_random_intercept", "cluster_mean_ttest", "mi_baseline_adjusted"), formula, alpha = 0.05, inference = c("satterthwaite", "kenward_roger", "wald_z"), m_imputations = 20, degenerate_counts = FALSE )planned_analysis( estimator = c("linear_model", "lmm_random_intercept", "cluster_mean_ttest", "mi_baseline_adjusted"), formula, alpha = 0.05, inference = c("satterthwaite", "kenward_roger", "wald_z"), m_imputations = 20, degenerate_counts = FALSE )
estimator |
Character; one of the estimators above. |
formula |
The planned model formula. The simulated data provide
|
alpha |
Numeric. Two-sided significance level of the planned
decision rule (default 0.05); also fixes the nominal |
inference |
Character. Inference method for
|
m_imputations |
Integer. Number of imputations for
|
degenerate_counts |
Logical. Whether degenerate fits (singular or
boundary variance estimates; failure class (c) of the manuscript's
taxonomy) count against the model-failure threshold. Design-specific
and must be pre-specified; the choice is echoed in the report. Fatal
errors and nonconvergence always count; diagnostic warnings are
reported but never count. Default |
Available estimators:
"linear_model" — stats::lm() on the retained sample
(complete-case), Wald t interval. Baseline adjustment is expressed
through the formula, e.g. y_observed ~ treatment + baseline.
"lmm_random_intercept" — linear mixed model via lmerTest::lmer()
on the retained sample; inference selects "satterthwaite"
(default), "kenward_roger" (requires pbkrtest), or "wald_z"
(large-sample z interval).
"cluster_mean_ttest" — two-sample t-test on equal-weighted cluster
means (with cluster sizes fixed by design, equal and pupil-count
weighting coincide), n_clusters - 2 degrees of freedom.
"mi_baseline_adjusted" — multiple imputation of missing outcomes
from a normal model drawing on (treatment, baseline), followed by the
baseline-adjusted linear model on each completed dataset and Rubin's
rules with Barnard-Rubin degrees of freedom. Identifiable for the ITT
estimand when dropout is MAR given the observed baseline; per the
manuscript (section 2.3), evaluated through bias against the target,
not through drift.
An object of class recovery_analysis.
Simulates the declared design over the crossed scenario grid (manuscript, section 2.2, Step 3): null and target effects, each under declared and pessimistically perturbed nuisance assumptions, plus an informational expected-effect row when the declared effect exceeds the SESOI. The planned analysis is applied to every simulated dataset and the diagnosands of section 2.3 are computed with Monte Carlo standard errors and explicit inclusion rules.
recovery_test( design, sims = 2000, scenarios = "confirmatory_grid", thresholds = recovery_thresholds(), seed = NULL )recovery_test( design, sims = 2000, scenarios = "confirmatory_grid", thresholds = recovery_thresholds(), seed = NULL )
design |
A |
sims |
Integer. Simulations per scenario row. The default 2000 is an initial working number, not a standard: the relevant stopping rule is whether the MCSE is small enough to support the verdict, and verdicts near a threshold may require many more. |
scenarios |
|
thresholds |
A |
seed |
Optional integer seed for reproducibility. |
An object of class recovery_result. Use verdict() and
report() on it.
Wraps recovery_test() with the pre-declared simulation stopping
rule of the protocol (manuscript, section 2.4). The number of
simulations S starts at start_sims and is doubled whenever a required
threshold margin is still within mcse_margin (default 2) Monte Carlo
standard errors of its threshold, up to a pre-declared ceiling
max_sims. The rule stops as soon as the verdict is determined: a stable
declared failure locks FAIL, or all required margins resolve (clean PASS
or determined RISK). Anything still within mcse_margin MCSEs of a
threshold at max_sims is reported as RISK: the run cannot resolve
whether the design clears the bright line, and that irresolution is
itself the finding.
recovery_test_stable( design, start_sims = 2000L, max_sims = 16000L, scenarios = "confirmatory_grid", thresholds = recovery_thresholds(), seed = NULL, verbose = FALSE )recovery_test_stable( design, start_sims = 2000L, max_sims = 16000L, scenarios = "confirmatory_grid", thresholds = recovery_thresholds(), seed = NULL, verbose = FALSE )
design |
A |
start_sims |
Initial simulations per scenario row (default 2000). |
max_sims |
Pre-declared maximum simulations per scenario row (default 16000, i.e. up to three doublings from 2000). |
scenarios |
|
thresholds |
A |
seed |
Optional integer seed for reproducibility. |
verbose |
Logical; if |
Doubling reuses the same seed; each (seed, S) run is fully
reproducible.
A recovery_result (as from recovery_test()) carrying an
additional "stopping" attribute: a list with start_sims,
final_sims, max_sims, resolved (whether the verdict was
determined before the ceiling), hit_ceiling, and unresolved (a data
frame of any required margins still within the MCSE band at the final
S).
The recovery-test protocol ships threshold profiles matched to design goals (manuscript, section 2.5). Thresholds are conventions: their function is not to be correct but to prevent post hoc negotiation with borderline results. A profile must be selected or justified before simulation; every deviation from a shipped profile is recorded and echoed in the report, and the report recomputes the verdict under the shipped strict and lenient profiles with signed margins to every threshold.
recovery_thresholds( profile = c("default", "strict", "lenient", "estimation"), null_rejection_mult = NULL, power = NULL, target_bias = NULL, coverage = NULL, type_s = NULL, type_m = NULL, model_failure = NULL, drift = NULL, overcoverage_flag = 0.975, mcse_margin = 2, min_conditional_n = 200, max_width = NULL )recovery_thresholds( profile = c("default", "strict", "lenient", "estimation"), null_rejection_mult = NULL, power = NULL, target_bias = NULL, coverage = NULL, type_s = NULL, type_m = NULL, model_failure = NULL, drift = NULL, overcoverage_flag = 0.975, mcse_margin = 2, min_conditional_n = 200, max_width = NULL )
profile |
One of |
null_rejection_mult |
Maximum target-null rejection rate as a multiple of the nominal alpha. |
power |
Minimum power at the SESOI. |
target_bias |
Maximum absolute target bias in units of the declared substantive scale Delta. |
coverage |
Minimum coverage of a nominal 95% interval (lower bound; one-sided). |
type_s |
Maximum Type S rate among significant estimates, when stably estimable. |
type_m |
Maximum Type M exaggeration among significant estimates, when stably estimable. |
model_failure |
Maximum counted model-failure rate (see
|
drift |
Maximum absolute estimand drift in Delta units; thresholded by the estimation profile only (reported, not thresholded, in confirmatory profiles, where its effect is captured by target bias and target-null rejection). |
overcoverage_flag |
Coverage above this value is flagged as inefficiency (never failure). |
mcse_margin |
All required threshold margins must exceed this many Monte Carlo standard errors for a PASS (default 2). |
min_conditional_n |
Minimum contributing simulations for a conditional diagnosand (Type S, Type M) to be treated as stable (default 200 — a stability convention chosen so that the MCSE of a conditional proportion cannot exceed ~0.035). |
max_width |
Maximum acceptable confidence-interval width for the
estimation profile's precision criterion; usually inherited from
|
The shipped confirmatory profiles are, exactly:
| Criterion | Lenient | Default | Strict |
| Target-null rejection | <= 1.50 a | <= 1.25 a | <= 1.10 a |
| Power | >= 0.70 | >= 0.80 | >= 0.90 |
| Target bias | <= 0.10 D | <= 0.05 D | <= 0.025 D |
| Coverage | >= 0.900 | >= 0.925 | >= 0.940 |
| Type S | <= 0.05 | <= 0.01 | <= 0.005 |
| Type M | <= 2.00 | <= 1.50 | <= 1.25 |
| Model failure | <= 0.05 | <= 0.01 | <= 0.005 |
The rejection-rate threshold is one-sided: excess false claims trigger
failure, while conservative behavior is flagged through power and
precision. The coverage threshold is a lower bound; overcoverage above
overcoverage_flag (default 0.975) is flagged as inefficiency and
evaluated through precision, not treated as failure.
The "estimation" profile replaces the rejection-rate and Type S/M
criteria with target bias, coverage, precision against a declared
maximum acceptable width (see target_estimand()), and estimand
drift, evaluated on the target rows only.
An object of class recovery_thresholds, carrying the
threshold-set version, the profile name, and a record of any values
changed from the shipped profile.
Renders the full recovery report (protocol Step 6): target estimand and SESOI with scales, the declaration (including explicit statements of what was not modeled), the scenario grid with the null-world specification and the evidence tier of every pessimistic value, all diagnosands under every scenario with MCSEs, contributing counts, failure-class counts, and the bias decomposition, the threshold profile with signed margins, the verdict under the selected and the shipped strict/lenient profiles, the binding failure mode, and the computation settings. The report is standalone: a collaborator, reviewer, supervisor, or funder should be able to read it without running the code. A FAIL report is as complete as a PASS report.
report(result, file = NULL)report(result, file = NULL)
result |
A |
file |
Optional path; if supplied, the report is also written there as plain text. |
The report, invisibly, as a character vector of lines.
Step 1 of the recovery-test protocol: state, in one sentence, the quantity the study is designed to estimate, the scale on which it is expressed, and the smallest effect size of interest (SESOI). The SESOI should be justified on substantive grounds, not reverse-engineered from the available sample size.
target_estimand( estimand, scale, sesoi, bias_scale_unit = NULL, max_width = NULL )target_estimand( estimand, scale, sesoi, bias_scale_unit = NULL, max_width = NULL )
estimand |
Character. One-sentence statement of the target estimand (quantity, population, scale). |
scale |
Character. The scale on which the estimand is expressed,
e.g. |
sesoi |
Numeric. The smallest effect size of interest, on the estimand scale. Must be strictly positive. Target scenario rows are simulated at the SESOI. |
bias_scale_unit |
Numeric or |
max_width |
Numeric or |
An object of class recovery_target.
target_estimand( estimand = "ITT mean difference at 12 weeks", scale = "latent-outcome standardized mean difference", sesoi = 0.40 )target_estimand( estimand = "ITT mean difference at 12 weeks", scale = "latent-outcome standardized mean difference", sesoi = 0.40 )
Individual random assignment to two arms, with an observed baseline measure. The data-generating model follows the manuscript (section 3.1): for participant i with assignment Z,
so the true (latent) outcome has unit SD given Z and the declared
effect is expressed in latent standard-deviation units. The baseline is
observed at randomization for every participant and is available to the
answer strategy as the column baseline.
two_arm_trial( n_per_arm, allocation = 0.5, baseline_outcome_cor = 0.5, noncompliance = 0 )two_arm_trial( n_per_arm, allocation = 0.5, baseline_outcome_cor = 0.5, noncompliance = 0 )
n_per_arm |
Integer. Participants recruited per arm. |
allocation |
Numeric in (0, 1). Share assigned to treatment. |
baseline_outcome_cor |
Numeric in [0, 1). Correlation rho between the observed baseline and the true outcome (default 0.5). |
noncompliance |
Numeric in [0, 1). Share of treated participants who do not receive treatment (one-sided never-takers). The target estimand remains the declared ITT effect. |
n_per_arm is the number of participants recruited per arm (under
equal allocation); attrition, if declared via attrition_model(), is
applied after recruitment.
An object of class recovery_data_strategy.
Applies the fixed verdict rule (manuscript, section 2.2, Step 5)
mechanically to a recovery_test() result. Once thresholds and
scenarios are fixed, the computation is mechanical — the judgment
lives in the declaration and the scenario grid, not here.
verdict(result)verdict(result)
result |
A |
PASS — all required thresholds met under all scenario rows the
selected profile requires, with every margin exceeding
mcse_margin (default 2) Monte Carlo standard errors.
RISK — thresholds met under the declared-nuisance rows but not
under a pessimistic row; or any margin within mcse_margin MCSEs of
a threshold; or any required conditional diagnosand too unstable to
evaluate; or required scenario rows missing from the run.
FAIL — one or more required thresholds fail under a declared-nuisance row — including an inflated target-null rejection rate under Null-declared. The planned study cannot recover the target even if the declared assumptions are correct.
The verdict is recomputed under the shipped strict and lenient
profiles (section 2.5): agreement across profiles carries more
decision weight than any one profile, and a verdict that flips is
itself a finding — the RISK category exists to hold it. The verdict is
a decision convention, not a validity classification, and the full
report() always travels with it.
An object of class recovery_verdict.