Output files¶
Every file a fresh run writes, with its columns or keys. The lists below were taken from
runs of match run + match report, cis run (five runs) + cis compare, and trans run
(five runs) + trans compare on the example data. Paths are relative to the run (or comparison) directory; file names come
from one place, anqchor.pipelines.layout.Layout. Conventions:
- TSV files have a header (except the plain ID lists and maps noted below), use
NAfor missing values and1/0for booleans;.gzfiles are gzip/bgzip compressed. - Every JSON file has
schema_version(currently1;3for the cissummary.jsonof a run or comparison, which a comparison requires of every run;2for the transsummary.jsonof a run or comparison, whichtrans compareandtrans reportrequire). - Paths inside summaries are relative to the run directory.
- In a flags-only run the dataset is called
datasetand the referencereference, so<ds>=datasetand<ref>=referencebelow.
Run-level files¶
| File | Content |
|---|---|
anqchor.resolved.yaml |
complete settings of the last command that ran steps in the directory (defaults filled in, paths absolute, flags included; a dry run or anqchor steps leaves it unchanged); a valid --config file |
run.json |
provenance; see Running |
Makefile |
generated; rewritten on every invocation |
.anqchor.lock |
run-directory lock held while a command runs (a second command on the same directory exits with code 2) |
steps/<step>.json |
step signature: name kind pipeline inputs outputs commands params, plus params.qpgen_sha256 for steps that run qpgen and anqchor_version for Python steps |
steps/<step>.done, .seconds |
stamp and wall time of the last successful run |
logs/<step>.log |
step stdout and stderr |
prepare¶
prepare/genotypes/¶
| File | Content |
|---|---|
pgen_list.tsv |
qpgen pgen list, 4 columns CHROM BEG END PREFIX, absolute prefixes, no header |
links/ |
symlinks giving non-standard file names a common prefix (only when needed) |
samples.txt |
genotype IIDs, one per line, psam order |
genotypes.json |
pgen_list or pfile, n_samples, chrom_style (chr/plain), chromosomes, entries (chrom beg end pgen psam pivar), n_entries |
prepare/references/<ref>/¶
| File | Content |
|---|---|
pairs.tsv.gz |
normalised pairs: trait variant beta se pval (pval may be NA); variant IDs in the genotype chromosome style; invalid rows and duplicates removed |
variants.tsv |
one row per unique variant: variant chrom pos ref alt is_indel present |
variants.present.txt |
present variants, one CPRA per line, sorted, no header |
reference.json |
reference kind n_pairs_input n_pairs n_invalid n_duplicates n_traits n_variants n_variants_list n_variants_list_invalid n_present n_absent n_absent_snv n_absent_indel absent_examples invalid_examples per_chrom chrom_renamed chrom_style presence_method |
prepare/datasets/<ds>/¶
| File | Content |
|---|---|
pheno.regenie.tsv.gz |
phenotypes, regenie layout FID IID trait…, the dataset's own sample IDs |
cov.regenie.tsv.gz |
covariates, regenie layout FID IID sex PC1 … (only with cov); categorical covariates keep their text levels (sex is XX/XY), constant columns are dropped and missing values are handled by cov_missing. There is no numeric copy: qpgen encodes the categorical columns (--cat-cov-names) |
cov.json |
raw_columns (the columns of cov.regenie.tsv.gz), categorical_names (the categorical covariates, passed to qpgen as --cat-cov-names), categorical (name → {level: code}, levels in sorted order), names, columns and groups (original name → columns of the equivalent numeric encoding: L − 1 per categorical covariate; used only to count covariates for degrees of freedom and to resolve --cov-include), dropped_constant, dropped_all_missing, n_samples, missing_policy, n_missing_handled, dropped_samples |
traits.tsv |
trait chrom start end for every phenotype trait (NA when unknown), genotype chromosome style |
samples.txt |
phenotype sample IDs |
sample_map.tsv |
geno_id pheno_id, no header, IDs present on both sides (identity pairs without a sample_map) |
dataset.json |
dataset omics pheno pheno_format pheno_fast_path n_samples_pheno n_traits n_missing_values n_samples_geno n_samples_geno_overlap n_samples_mapped n_samples_analysable (phenotyped, genotyped and with covariates: usable bycis/trans) warnings (dropped or covariate-less samples,sample_mapID mismatch) n_pheno_unmapped pheno_unmapped_examples sample_map [n_sample_map_rows n_sample_map_rows_geno_missing n_sample_map_rows_pheno_missing n_geno_with_several_pheno] has_cov n_cov_columns cov_columns n_cov_samples n_pheno_without_cov cov_categorical (name → {level: code}) cov_dropped_constant cov_missing_policy cov_n_missing_handled has_loci n_traits_with_loci trait_annot_source trait_annot_rule trait_id_rule seconds_pheno seconds_total |
prepare/datasets/<ds>/ref.<ref>/¶
| File | Content |
|---|---|
trait_map.tsv |
ref_trait pheno_trait, no header, mapped traits only |
pairs.tsv.gz |
trait variant beta se pval ref_trait present: reference pairs rewritten to dataset trait IDs (present 0/1) |
variants.txt |
present variants of the mapped pairs, one CPRA per line, sorted |
traits.txt |
dataset trait IDs of the mapped reference traits |
mapping.json |
rule trait_map n_ref_traits n_dataset_traits n_mapped_traits n_unmapped_traits unmapped_examples n_pairs n_pairs_mapped n_pairs_present n_variants_present n_traits_with_present_variant |
PRS¶
| File | Content |
|---|---|
prs/<ref>/prs.prs.tsv.gz |
pair-prs scores, regenie layout FID IID trait…, one row per genotyped sample |
prs/<ref>/prs.se.tsv.gz |
standard errors, same layout |
match¶
Under match/<ds>/.
| File | Content |
|---|---|
qpgen.weights.tsv.gz |
qpgen: Trait Weight N.Obs |
qpgen.match.assigned.tsv.gz |
qpgen: mapped samples, ID.Pheno N.Traits MatchStatus ID.self Z.self COR.self Rank.self; MatchStatus is BEST_MATCH, LENIENT_MATCH, NO_MATCH or NO_OBS_TRAITS |
qpgen.match.all.tsv.gz |
qpgen: all phenotyped samples, the same seven columns plus ID/Z/COR.{1st…5th}; here MatchStatus is the detailed status (SELF_BEST, SELF_LENIENT, SINGLE_NEW_BEST, MULTI_NEW_BEST, UNCLEAR, NO_SELF, NO_OBS_TRAITS) that anqchor uses |
qpgen.params.json |
argv qpgen dataset reference mahalanobis lambda_config lambda_mode (independent/auto/fallback/fixed) lambda_used rint use_cov z_threshold z_diff min_weight n_traits_used n_trait_map_rows n_dropped_* n_prs_samples n_samples_pheno n_samples_mapped n_samples_mapped_dataset n_sample_map_rows_dropped trait_tsv_used sample_tsv_used warnings log_excerpt seconds |
trait_map.used.tsv |
trait map given to qpgen, no header |
samples.tsv |
one row per phenotyped sample, see below |
swaps.tsv |
samples whose best match is another individual: kind (reciprocal/reciprocal-weak/one-way) pheno_id geno_id_assigned group best_id z_best z_self rank_self partner_pheno_id partner_geno_id_assigned partner_group partner_best_id |
candidates.tsv |
rescue candidates (unmapped samples with a clear best match that is not another sample's confirmed match): pheno_id candidate_geno_id z_best second_id z_second z_gap n_traits candidate_assigned_to |
duplicates.tsv |
samples that match two genotype IDs about equally (rule in match): pheno_id geno_id_assigned group geno_id_1 z_1 geno_id_2 z_2 z_3 assigned_in_pair |
related.tsv |
possible relatives (a block of two or more close candidates; samples in duplicates.tsv left out): pheno_id geno_id_assigned group n_close close_ids close_z assigned_in_close |
weights.tsv |
trait ref_trait weight n_obs weight_le0 |
summary.json |
see below |
samples.tsv: pheno_id geno_id_assigned qpgen_status group z_self cor_self rank_self n_traits
best_id z_best cor_best second_id z_second z_gap proposed_geno_id note id_1…id_5 z_1…z_5
(meanings in match).
Also report/match.html after match report or match all.
summary.json keys: pipeline dataset omics reference verdict verdict_status (good/warning/critical)
status_counts group_counts n_pheno n_mapped n_unmapped n_scored n_not_scored n_fail
n_fail_two_ids n_reciprocal_swaps n_rescue_candidates n_duplicates n_related n_status_disagreements failing (list of failing
samples) n_prs_samples n_traits_used n_traits_weight_le0 weights (min q05 q25 median mean q75
q95 max) mahalanobis lambda_mode lambda_used rint use_cov z_threshold z_diff
trait_tsv_used sample_tsv_used qpgen_argv qpgen_log_excerpt dataset_counts warnings (warnings includes
the number of samples not scored for lack of covariates).
cis run¶
Under cis/<ds>/. A flags-only run has one strategy, default (so <s> = default), and
its rows are labelled with the run label, the name of the run directory (run and
label columns). A configuration file listing several strategies labels them by strategy
name. A run tests one phenotype transform: as given, or RINT with --rint
(pipelines.cis.rint).
| File | Content |
|---|---|
strategies/<s>/cov.regenie.tsv.gz |
the run's covariates, regenie layout, categorical covariates as text levels (absent without covariates) |
strategies/<s>/pheno.regenie.tsv.gz |
strategy phenotypes (only with a pheno override in a config file) |
strategies/<s>/strategy.json |
pipeline dataset strategy cov_source cov_columns n_cov [cat_cov_names cov_include cov_empty cov_file cov_encoding] pheno; cov_columns are the covariate names in the file, n_cov counts the numeric columns after encoding (degrees of freedom), cat_cov_names the categorical covariates passed to qpgen as --cat-cov-names (absent without covariates) |
strategies/<s>/pair_assoc.tsv.gz |
pair-assoc output (pair engine): TRAIT CHROM GENPOS ID ALLELE0 ALLELE1 A1FREQ N N_RR N_RA N_AA INFO BETA SE TSTAT LOG10P, plus BETA_RINT SE_RINT TSTAT_RINT LOG10P_RINT with --rint |
strategies/<s>/rect_assoc.<t>.tsv.gz |
rect-assoc output (rect engine); <t> = raw, or rint with --rint |
pairs.long.tsv.gz |
one row per mapped pair, see below |
strategies.tsv |
one row per run, see below |
traits.tsv |
trait ref_trait n_pairs n_variant_absent, per run label: best_log10p:<run>, level:<run> (strongest level reached, in level_order), then delta_log10p:<run> (vs the first run; only when there is more than one), and best_run |
power_curve.tsv |
run threshold_log10p n_tested n_replicated frac_replicated (sign-concordant with p < 10^−threshold, as the levels; thresholds 0, 0.5, 1, …) |
summary.json |
see below |
Also report/cis.html after cis report or cis all.
pairs.long.tsv.gz, one row per (run, mapped pair):
| Column | Meaning |
|---|---|
run |
the run label |
trait, ref_trait, variant |
dataset trait ID, reference trait ID, CPRA |
beta_ref, se_ref |
reference effect (ALT allele) and standard error |
z_ref |
reference z-score: beta_ref / se_ref, or, without se_ref, the signed normal quantile of pval_ref |
pval_ref, log10p_ref |
the reference's pval column (or the column named by columns.pval) and its −log10, read from the text so that p-values below the double range (1e-400) keep their size; without a usable p-value, log10p_ref is the two-sided normal p-value of z_ref |
status |
tested, variant_absent, not_tested (no association row, or no finite statistic) |
a1freq, n, beta, se, tstat, log10p |
study statistics (ALT allele, the same allele as beta_ref); RINT statistics with --rint |
z |
signed study z-score beta / se (as z_ref); without a usable se, the normal quantile of the two-sided p-value with the sign of beta |
sign_concordant |
1/0, NA when not tested |
rep_<level> |
1/0 for each configured level (sign nominal fdr bonferroni strict) |
strategies.tsv, one row per run: run label rint (1 with RINT) n_cov n_pairs n_tested
n_variant_absent n_not_tested n_rep_<level>… frac_rep_<level>… median_log10p mean_z2
beta_corr frac_sign_discordant pi1 bonferroni_log10p; with the rect engine also
n_control_cells lambda_gc_control n_control_p_lt_1e5 n_control_cross_chrom
lambda_gc_control_cross_chrom n_control_cross_chrom_p_lt_1e5; last mean_z2_common (mean
z² over the pairs tested in every row's run) and rel_neff = (mean_z2_common − 1) / (first
run's mean_z2_common − 1), the effective sample size relative to the first run (1 for a
single run).
summary.json keys: schema_version dataset reference engine rint levels (rule per level)
level_order (levels from weakest to strongest by their −log10 p cut-off for this run)
transform_semantics (as_given, rint, engines, degrees_of_freedom) runs (label strategy rint
n_cov per row) baseline n_ref_pairs n_pairs_mapped n_pairs_unmapped n_traits_mapped
n_traits_unmapped n_variant_absent n_absent_variants n_absent_variants_indel
n_absent_variants_snv absent_variant_examples unresolved_fraction unresolved_by_run
max_unresolved_fraction recomputed_log10p per_run (the strategies.tsv numbers by label)
ranking (labels per level, best first) recommended (run label key_level reason) warnings
sentinels (reserved, null).
cis compare¶
anqchor cis compare RUN_DIR... --out DIR writes into DIR. Each compared run is identified
by its label (run = label column); a run whose configuration listed several strategies
appears as <label>:<strategy>. Runs may differ in the phenotype transform. A run written
by an anqchor version with another cis summary schema is refused with a request to re-run it.
| File | Content |
|---|---|
runs.tsv |
one row per run: label run_dir dataset genotypes n_geno_samples pheno n_pheno_samples n_samples_analysable cov cov_include (all, the names given, or NA) n_cov cov_columns rint reference reference_pairs engine n_pairs_mapped |
strategies.tsv |
the columns of a run's strategies.tsv (run = label = run label), plus n_common_tested and n_rep_<level>_common (replication counted only over the pairs tested in every compared run); mean_z2_common and rel_neff are over those pairs, relative to the first run |
pairs.long.tsv.gz |
the runs' pairs.long.tsv.gz stacked, run = run label; levels re-derived with the first run's level settings |
traits.tsv |
as in a run, one column group per run label; delta_log10p is relative to the first run |
power_curve.tsv |
as in a run, per run label |
summary.json |
the keys of a run's summary.json (without rint, n_traits_unmapped, recomputed_log10p, unresolved_by_run), plus kind (comparison), runs (the runs.tsv rows) and comparability (same_reference, reference_sha256 per label, rint per label); baseline is the first run's label; warnings include each run's warnings prefixed by its label |
compare.json |
manifest: kind (cis), runs (label run_dir dataset settings), labels, argv, cwd |
settings/<label>.anqchor.yaml |
a copy of each run's anqchor.resolved.yaml |
report/cis.html |
the comparison report (unless --no-report) |
trans¶
Under trans/<ds>/ (<ds> is dataset in a flags-only run). trans run writes no HTML;
trans report or trans all writes report/trans.html. One run per directory, labelled
run (the run directory name). Summary schema 2.
| File | Content |
|---|---|
variants.txt |
rectangle variants, CPRA per line |
traits.txt |
rectangle genes (dataset IDs) |
lists.json |
variants {source n_listed n_invalid n_renamed_chrom n_off_genome n_kept}, traits {source trait_chroms n_missing_from_dataset allowed_chroms n_candidates n_kept n_dropped_off_chrom n_dropped_no_locus examples_*} |
strategies/<s>/cov.regenie.tsv.gz, strategy.json |
the run's covariates (<s> is default in a flags-only run) |
strategies/<s>/rect_assoc.<t>.tsv.gz |
rect-assoc output (<t> = raw, or rint with --rint): #CHROM GENPOS ID ALLELE0 ALLELE1 A1FREQ N N_RR N_RA N_AA TEST then BETA.T SE.T TSTAT.T LOG10P.T per gene |
reference_pairs.tsv |
one row per reference pair: run trait variant ref_trait pair_class (cis/trans/no_locus) trans_kind (cross_chrom/same_chrom_distal) chrom pos trait_chrom trait_start distance beta_ref se_ref pval_ref status (tested/variant_absent/not_in_rectangle/not_tested) variant_tested a1freq n beta se z log10p sign_concordant rep_sign rep_nominal (p < 0.05) rep_strict (p < 1e-5) |
hotspots.tsv |
one row per bin with a tested variant, in genome order: chrom bin_start bin_end n_variants n_genes_<threshold>… top_trait top_variant top_log10p; <threshold> is p1e-4 p1e-5 p1e-6 p5e-8 p1e-8 p1e-10, the configured levels (e.g. p0.001) and bonferroni; the top cell is the strongest trans cell of the bin among genes not in cis with it (NA without one at the loosest threshold) |
hotspot_genes.tsv |
one row per bin and gene with a trans cell below the loosest grid threshold (genes in cis with the bin excluded): chrom bin_start bin_end trait trait_chrom trait_start variant beta se log10p (the strongest variant of the bin for that gene) |
qq.tsv |
the cross-chromosome cells used for inflation, at most ~2,000 points (the 250 strongest, then log-spaced ranks): n_cells rank expected_log10p observed_log10p band_lo_log10p band_hi_log10p (95% range of the order statistic, Beta(rank, n − rank + 1)) |
variants.tsv |
one row per tested variant: variant chrom pos bin_start a1freq n n_cis n_cross_chrom n_same_chrom_distal n_sig_trans (p < nominal) max_log10p_trans top_trans_trait max_log10p_cis top_cis_trait excluded_from_calibration |
pairs.sig.tsv.gz |
every cell with p < levels.nominal (any class), strongest first, at most max_sig_rows: variant chrom pos trait trait_chrom trait_start cell_class (cis/cross_chrom/same_chrom_distal/no_locus) reference_pair distance a1freq n beta se log10p |
summary.json |
see below |
summary.json keys: schema_version dataset reference run strategy rint n_cov cov_source
cov_columns n_variants n_traits (listed) n_variants_tested n_traits_tested n_samples n_cells
n_cells_trans cells (cis cross_chrom same_chrom_distal no_locus) variants traits trait_chroms
variant_selection trait_selection known_pairs n_reference_pairs; replication: levels
(nominal 0.05, strict 1e-5) and per class cis, trans, no_locus: n_pairs n_tested
n_rep_sign n_rep_nominal n_rep_strict median_log10p (trans also n_pairs_cross_chrom
n_pairs_same_chrom_distal); hotspots: bin_bp n_bins thresholds (key label alpha log10p
roles, loosest first) k counts (thresholds × k: bins with at least k genes) default (threshold
k n_hotspots) bonferroni_p; inflation: n_cells n_cells_excluded n_reference_trans_pairs
n_variants_excluded n_genes_excluded exclusion_window_bp median_log10p lambda_gc counts (alpha
label observed expected at p < 1e-3 and 1e-5); n_sig_cells sig_rows_written params
definitions warnings memory_bound seconds. Definitions: trans.
trans compare¶
anqchor trans compare RUN_DIR... --out DIR writes into DIR. Each compared run is
identified by its label (the run directory name or --label; <label>:<ds> for a run with
several datasets). A run written by an anqchor version with another trans summary schema is
refused with a request to re-run it. Summary schema 2.
| File | Content |
|---|---|
runs.tsv |
one row per run: label run_dir dataset genotypes n_geno_samples pheno n_samples cov cov_include (all, the names given, or NA) n_cov cov_columns rint reference reference_pairs; per class <c> = cis, trans: <c>_n_pairs <c>_n_tested <c>_n_rep_sign <c>_n_rep_nominal (p < 0.05) <c>_n_rep_strict (p < 1e-5), and over the reference pairs tested in every run <c>_n_common <c>_n_rep_sign_common <c>_n_rep_nominal_common <c>_n_rep_strict_common; then n_variants_tested n_traits_tested n_cells n_cells_cis n_cells_cross_chrom n_cells_same_chrom_distal n_cells_trans cis_window_bp lambda_gc n_inflation_cells n_inflation_cells_excluded observed_p1e-3 expected_p1e-3 observed_p1e-5 expected_p1e-5 hotspot_bin_bp n_bins bonferroni_p hotspot_threshold hotspot_min_genes n_hotspots n_hotspots_shared (n_hotspots: the hotspot count at the run's own opening threshold and minimum; n_hotspots_shared: at the first run's, the same setting for every run, empty when a run's grid lacks it) |
hotspot_grid.tsv |
one row per run, threshold and minimum genes: run threshold (key, e.g. p1e-5, bonferroni) label alpha log10p k n_hotspots |
hotspots.tsv |
the runs' hotspots.tsv stacked: run then the run's columns (n_genes_<threshold> missing in a run is NA) |
hotspot_genes.tsv |
the runs' hotspot_genes.tsv stacked, run first |
qq.tsv |
the runs' qq.tsv stacked, run first |
reference_pairs.tsv |
the runs' reference_pairs.tsv stacked (run = label), plus tested_in_all (1 when the pair, matched by ref_trait and variant, was tested in every run) |
summary.json |
schema_version kind (comparison) pipeline (trans) dataset (comparison) labels reference runs (the input columns of runs.tsv) replication_levels n_common_pairs (cis, trans) thresholds (key label alpha log10p of every run's grid, loosest first) comparability (same_genotype_samples same_reference same_rectangle same_cells same_samples (same number and same phenotype samples) same_cis_window same_bin_width reference_sha256 rectangle_sha256 n_samples rint, per label where not a flag) per_run (per label: run dataset rint n_cov n_samples n_variants n_traits n_variants_tested n_traits_tested n_cells n_cells_trans cells n_reference_pairs replication hotspots inflation params warnings, as in a run's summary.json) definitions warnings (comparability warnings, then each run's warnings prefixed by its label) |
compare.json |
manifest: kind (trans), runs (label run_dir dataset settings), labels, argv, cwd |
settings/<label>.anqchor.yaml |
a copy of each run's anqchor.resolved.yaml |
report/trans.html |
the comparison report (unless --no-report) |
report¶
| File | Content |
|---|---|
report/match.html |
written by match report or match all |
report/cis.html |
written by cis report or cis all in a run directory, and by cis compare (or cis report) in a comparison directory |
report/trans.html |
written by trans report or trans all in a run directory, and by trans compare (or trans report) in a comparison directory |
report/index.html |
only from the top-level anqchor run with more than one pipeline: one tile per pipeline and dataset, with links |
Reports are self-contained interactive pages (plotly; every chart has a table view and TSV
download); with the default --plotlyjs embed the example reports are 5-10 MB. With
--plotlyjs cdn and no network, each chart area shows a notice suggesting
--plotlyjs embed. Reports are rendered only from the summaries above, never from qpgen output.