match: sample identity¶
Question: does each omics sample come from the genotyped individual it is labelled with?
This page is the reference for the identity check: commands, steps, the qpgen calls, the status rules and every output file. For a guided walk-through on the example data, see the identity matching use case.
How it works¶
Known xQTL effects applied to the genotypes give every genotyped individual a predicted value
(a polygenic score, PRS) for each reference trait. qpgen's match-prs-pheno correlates each
omics sample's measured traits with every individual's PRS profile. It weights each trait by
how well its PRS predicts it, turns the correlations into Z-scores across all genotyped
individuals, and checks whether the labelled genotype is the best match.
flowchart TB
R[("Reference pairs<br/>trait variant beta se")] --> P
G["Genotypes"] --> P["pair-prs:<br/>PRS per individual and trait"]
P --> M["match-prs-pheno:<br/>Z of every individual<br/>for every omics sample"]
O["Omics matrix<br/>(+ covariates)"] --> M
M --> S["summarize:<br/>calls, swaps, relatives"]
S --> H["match report:<br/>report/match.html"]
Genotyped individuals without omics data are scored too. They make it possible to name the true source of a mislabelled sample and to rescue omics samples that have no genotype label.
Commands¶
| Command | What it does |
|---|---|
anqchor match run |
prepares the inputs, computes the PRS, runs the matching and writes the summary tables. It writes no HTML. |
anqchor match report --out RUN_DIR |
renders report/match.html from the summary tables of a finished run, in about a second. It never runs qpgen. |
anqchor match all |
match run followed by match report. |
Inputs are given as flags; a configuration file (--config) is optional:
anqchor match run \
--geno-list examples/mage/data/geno/mage.pgen_list.tsv \
--pheno examples/mage/data/pheno/mage_expr.chr21_22.regenie.tsv.gz \
--cov examples/mage/data/pheno/mage_covariates.regenie.tsv.gz \
--pairs examples/mage/data/reference/gtex_lcl_eqtl.chr21_22.tsv.gz \
--out results/match
anqchor match report --out results/match
| Flag | Meaning | Default |
|---|---|---|
--geno-list / --pfile |
genotypes: a qpgen pgen list, or one PLINK2 prefix with .pgen, .psam, .pvar.idx.gz |
required |
--pheno, --pheno-format |
omics matrix and its layout | required, regenie |
--cov, --cov-format |
covariates regressed out of the omics data before matching | none, regenie |
--cat-cov-names |
categorical covariates coded as numbers (e.g. a batch number); non-numeric columns such as sex are categorical automatically |
none |
--pairs |
reference pairs (trait variant beta se [pval]) |
required |
--sample-map |
geno_id pheno_id table when omics IDs differ from genotype IDs |
identical IDs |
--trait-id-rule |
how omics trait IDs relate to reference trait IDs | as_is |
--use-cov / --no-use-cov |
adjust for --cov before matching |
on |
--rint / --no-rint |
rank-based inverse normal transform after the adjustment | on |
--mahalanobis / --independent |
whiten correlated traits, or score them as independent | independent |
--lambda |
with --mahalanobis: auto or a value in [0, 1] |
auto |
--z-threshold |
Z above which a non-best own genotype is a lenient match | 1.96 |
--z-diff |
Z gap that makes a leading genotype a clear match | 2.0 |
--prs-samples |
file of genotype IDs to use as candidates | all genotyped individuals |
--dataset-name, --reference-name |
names used in output paths with flag-only inputs | dataset, reference |
--threads |
threads for match-prs-pheno (changing it re-runs nothing) |
1 |
The configuration keys of the same settings are under pipelines.match and
datasets.<name> (Configuration).
examples/mage/match.yaml is a complete example. Every run directory also holds
anqchor.resolved.yaml, the full settings of the run including those given as flags, so
repeats a run exactly. run.json records the command line, the qpgen binary and the time
of each step.
Steps¶
<ref> and <ds> are the reference and dataset names (reference and dataset with
flag-only inputs).
| Step | Kind | What it does | Run by |
|---|---|---|---|
prepare.genotypes |
Python | checks the genotype files; writes an absolute pgen list and the sample list | run, all |
prepare.reference.<ref> |
Python | normalises the reference pairs; checks which variants exist in the genotypes | run, all |
prepare.dataset.<ds> |
Python | converts the omics matrix and covariates to regenie layout (categorical covariates such as sex keep their text levels); reads the sample map |
run, all |
prepare.mapping.<ds>.<ref> |
Python | maps reference traits onto omics traits | run, all |
prs.<ref> |
qpgen | pair-prs over all genotyped individuals (or --prs-samples) |
run, all |
match.<ds>.qpgen |
Python + qpgen | restricts the trait and sample maps to what the PRS and the omics matrix contain, decides λ, runs match-prs-pheno |
run, all |
match.<ds>.summarize |
Python | per-sample calls, swaps, rescue candidates, relatives, weights, summary.json |
run, all |
report.match |
Python | report/match.html |
report, all |
Re-running with the same settings runs nothing. Changing only --z-threshold, --z-diff,
--rint, --use-cov, --mahalanobis or --lambda re-runs match.<ds>.qpgen and
match.<ds>.summarize; the prepared inputs and the PRS are reused. After such a re-run,
run anqchor match report again: an existing report/match.html is not updated by
match run, which then prints match report: ... is older than the results; update it with
'anqchor match report --out DIR'.
The qpgen calls¶
From the example run above (paths relative to the run directory):
qpgentools pair-prs --pgen-list prepare/genotypes/pgen_list.tsv \
--pairs prepare/references/reference/pairs.tsv.gz --out prs/reference/prs
qpgentools match-prs-pheno --prs prs/reference/prs.prs.tsv.gz \
--pheno prepare/datasets/dataset/pheno.regenie.tsv.gz --out match/dataset/qpgen \
--cov prepare/datasets/dataset/cov.regenie.tsv.gz --cat-cov-names sex \
--rint --z-threshold 1.96 --z-diff 2 --threads 1
--cat-cov-names lists the categorical covariates of the file: the non-numeric ones (here
sex, XX/XY) plus any named with anqchor's --cat-cov-names (config
datasets.<ds>.cov_categorical), for example a batch number. qpgen encodes each as L − 1
indicator columns (covariates). Without --cov (or with
--no-use-cov) both flags are left out.
Flags added when needed: --sample on pair-prs with --prs-samples; --sample-tsv with a
sample map; --trait-tsv when reference and omics trait IDs differ; --mahalanobis with
--auto-lambda or --lambda X; --min-weight; --missing-str. The --out prefix is a temporary directory inside match/dataset/
(shown as match/dataset/qpgen above). The exact command of a run
is in match/<ds>/qpgen.params.json (argv) and at the top of logs/match.<ds>.qpgen.log.
Scoring: independence (default) or Mahalanobis¶
By default traits are scored as independent, the fast path. With --mahalanobis
(mahalanobis: true) qpgen whitens correlated traits:
--lambda auto(the default) uses qpgen's--auto-lambdawhen at leastmin_self_for_auto(100) samples are mapped. With fewer, anqchor uses--lambda 0.5and says so in the log,summary.jsonwarningsand the report.- a number in [0, 1] is passed as
--lambda.λ = 1is equivalent to independence.
--lambda without --mahalanobis is accepted and has no effect. On the example data,
auto-λ tunes to 0.58 and confirms 11 more samples strictly than independence scoring
(details).
Status rules¶
anqchor ports qpgen's MatchStatus rules line by line, and the report ports them to
JavaScript; shared test vectors pin both. With Z(1) ≥ … ≥ Z(5) the top five Z-scores,
t = z_threshold and d = z_diff, checked in order:
NO_OBS_TRAITS: no observed trait with non-zero weight.SELF_BEST: mapped, and the labelled genotype ranks first.SINGLE_NEW_BEST:Z(1) > Z(2) + d.MULTI_NEW_BEST:Z(k) > Z(k+1) + dfor some k in 2..4.SELF_LENIENT: mapped andZ.self > t.UNCLEARwhen mapped,NO_SELFotherwise.
Both comparisons are strict: a gap of exactly z_diff is not clear.
| Group | Statuses | Meaning |
|---|---|---|
PASS |
SELF_BEST |
the labelled genotype is the best match |
WARN |
SELF_LENIENT, UNCLEAR |
the labelled genotype is not the best, but no other genotype is clearly ahead |
FAIL |
SINGLE_NEW_BEST, MULTI_NEW_BEST of a mapped sample |
one (or a few) other genotypes match clearly better |
NO_TEST |
NO_SELF, NO_OBS_TRAITS, every unmapped sample, NOT_SCORED |
nothing to confirm or reject |
NOT_SCORED is anqchor's status for an omics sample missing from qpgen's output, for
example a sample without covariates.
A NOT_SCORED sample keeps its assigned genotype ID (from the sample map) in
geno_id_assigned, so it is counted as mapped, and summary.json warnings says how many
samples were not scored.
Proposed relabel. A mapped FAIL sample gets proposed_geno_id = its best genotype when
Z(1) − Z(2) ≥ z_diff, unless that genotype is the confirmed (PASS) match of another
omics sample. Relabelling would then map two samples to one genotype, so
proposed_geno_id stays empty and the note says "no relabel proposed: best genotype is the
confirmed match of another sample". The sample is still a mismatch.
Rescue candidates. An unmapped sample with SINGLE_NEW_BEST stays NO_TEST (there is no
label to confirm) and is listed in candidates.tsv with its best genotype, unless that
genotype is the confirmed match of another sample (the same rule as above; the report shows
such a sample as "ID already used").
Two genotype IDs (duplicates). A sample matches two genotype IDs about equally when
its top two candidates stand clear of the third by more than z_diff (qpgen's
MULTI_NEW_BEST test at k = 2), the second scores above z_threshold, and the second
scores at least 0.8 × the first. The two genotypes then most likely come from one person: a
genotype sample listed twice, or identical twins. Relatives do not qualify: a first-degree
relative scores about half the self Z (0.51–0.74 of it on the example data, for samples
whose own score is strong), a duplicated genotype 1.00 (0.90 with 2% of its calls
changed). A duplicate whose second score does not stand clear of the background
cannot be told from noise and is not flagged. The flag does not change qpgen_status or
group; such samples are listed in duplicates.tsv and named in the verdict. When the
assigned ID is one of the two (or no ID is assigned), the sample is not a mismatch: it gets
no proposed_geno_id, is not a rescue candidate, and is left out of the verdict's
mismatches, n_fail, failing and swaps.tsv (n_fail_two_ids counts such FAILs).
Duplicates alone make the verdict a warning. With a small
z_diff (1 on the example data) a near-tie at background level can qualify too.
Swaps. A pair A, B where A's best genotype is B's label and B's best is A's is listed as
reciprocal when at least one side is a clear mismatch (SINGLE_NEW_BEST, a FAIL), and as
reciprocal-weak otherwise (for example two SELF_LENIENT samples whose own genotype is
second by a hair). Only reciprocal pairs count in n_reciprocal_swaps. Every other FAIL
sample is listed as one-way, with the sample (if any) that holds its best genotype as
partner.
Possible relatives. A sample is flagged when its top five start with a block of
k = 2..4 candidates, the best included, that stands out: the largest drop Z(k) − Z(k+1)
(k = 2..4) is at least 1.0 and Z(k) ≥ z_threshold. A first-degree relative's Z is roughly
half the self Z, and the best unrelated individuals reach Z ≈ 3 (on the example data the
median third-best Z is 2.7), so neither "within z_diff of the best" nor a Z threshold alone
describes a relative; the drop to the background does. The flag is informational and does
not change the group. A sample that matches two genotype IDs about equally is listed in
duplicates.tsv instead. The report does not show relatives.
Outputs¶
Everything is under the run directory; per-dataset files under match/<ds>/. Full schemas
are in Output files.
| File | Content |
|---|---|
match/<ds>/samples.tsv |
one row per omics sample: status, group, Z-scores, top five, proposal and note |
match/<ds>/swaps.tsv |
swaps: kind = reciprocal, reciprocal-weak (not counted) or one-way |
match/<ds>/candidates.tsv |
rescue candidates (unmapped samples with one clearly leading genotype) |
match/<ds>/duplicates.tsv |
samples that match two genotype IDs about equally |
match/<ds>/related.tsv |
samples whose top candidates start with a block of possible relatives |
match/<ds>/weights.tsv |
per-trait weight with the reference trait and a weight_le0 flag |
match/<ds>/summary.json |
counts per status and group, verdict, failing samples, weight quantiles, parameters |
match/<ds>/qpgen.weights.tsv.gz, qpgen.match.assigned.tsv.gz, qpgen.match.all.tsv.gz |
qpgen's own outputs, unchanged |
match/<ds>/qpgen.params.json |
resolved parameters: argv, λ mode and value, trait and sample counts, warnings, log excerpt |
match/<ds>/trait_map.used.tsv |
the trait pairs used (traits with both a PRS and an omics column) |
prs/<ref>/prs.prs.tsv.gz, prs.se.tsv.gz |
PRS of every genotyped individual (regenie layout) |
report/match.html |
the report, written by match report or match all |
anqchor.resolved.yaml, run.json, logs/<step>.log |
settings, provenance, one log per step |
samples.tsv¶
| Column | Meaning |
|---|---|
pheno_id |
omics sample ID |
geno_id_assigned |
labelled genotype (from the sample map, or the identical ID; empty when unmapped); kept for NOT_SCORED samples |
qpgen_status |
status as above (NOT_SCORED if qpgen did not score the sample) |
group |
PASS, WARN, FAIL, NO_TEST |
z_self, cor_self, rank_self |
Z-score, weighted correlation and rank of the labelled genotype |
n_traits |
traits observed for the sample |
best_id, z_best, cor_best |
best-matching genotype |
second_id, z_second, z_gap |
second-best genotype and z_best − z_second |
proposed_geno_id |
best genotype for a FAIL with a clear gap; empty otherwise, and empty when that genotype is another sample's confirmed match |
note |
who holds the best genotype, why no relabel is proposed, rescue candidate, two genotype IDs, several leading genotypes, possible relatives |
id_1…id_5, z_1…z_5 |
the top five candidates |
The other tables¶
| File | Columns |
|---|---|
swaps.tsv |
kind pheno_id geno_id_assigned group best_id z_best z_self rank_self partner_pheno_id partner_geno_id_assigned partner_group partner_best_id |
candidates.tsv |
pheno_id candidate_geno_id z_best second_id z_second z_gap n_traits candidate_assigned_to |
duplicates.tsv |
pheno_id geno_id_assigned group geno_id_1 z_1 geno_id_2 z_2 z_3 assigned_in_pair |
related.tsv |
pheno_id geno_id_assigned group n_close close_ids close_z assigned_in_close |
weights.tsv |
trait ref_trait weight n_obs weight_le0 (weight = correlation of PRS and trait over mapped samples) |
summary.json¶
The main keys: verdict (one sentence) and verdict_status (good, warning, or
critical when any sample fails); status_counts and group_counts; failing (each
FAIL with Z-scores, proposal and note, except samples that match two genotype IDs, one of
them their own); n_fail (the same samples) and n_fail_two_ids; n_reciprocal_swaps,
n_rescue_candidates, n_duplicates, n_related; n_pheno, n_scored, n_not_scored, n_mapped,
n_unmapped, n_prs_samples; n_traits_used,
n_traits_weight_le0 and weights (quantiles); the parameters (use_cov, rint,
mahalanobis, lambda_mode, lambda_used, z_threshold, z_diff); warnings;
qpgen_argv; and n_status_disagreements, which is 0 unless the qpgen build applies
different status rules from anqchor's port.
The report¶
report/match.html is a single file that works offline. Its title is "Sample identity
matching with anQChor". From top to bottom:
- Header: counts (omics samples, genotypes, traits) and the scoring used; with
independent scoring, a short tip to try
--mahalanobis. - Dashboard: tiles (number first) for samples with an assigned ID, strict match to own
ID (
SELF_BEST), lenient match to own ID (SELF_BEST+SELF_LENIENT), matches another sample (SINGLE_NEW_BEST), matches two genotype IDs (every sample in Duplicates), matches several (MULTI_NEW_BEST), own ID too weak (UNCLEAR) and not scored (NO_OBS_TRAITS,NOT_SCORED); the two-IDs, several and not-scored tiles appear only when non-zero. A "No assigned ID" tile appears when some samples have no ID; its line splits them into new ID found, ID already used, two genotype IDs, no clear match and not scored. Tiles, cards, chart legend, table filters and table use the same result names. Clicking a tile filters the sample table. - Sliders for the lenient cutoff (
z_threshold) and the clear lead (z_diff), each with a one-line hint (no info icons), and one line explaining the match score. Every count, card, chart point, duplicate and table row is reclassified in the browser with qpgen's rules. Copy as config gives the YAML lines that make the values permanent; the files in the run directory keep the values of the run. - Samples to check: one card per sample with an assigned ID that is not a strict match, per sample that matches two genotype IDs, and per sample without an assigned ID. When more than six samples have no ID, one card summarises them instead (counts per result, and how many with no clear match have a best genotype that no other sample uses). Two or more samples not scored share one card. An own-ID-too-weak card names the best genotype, its lead and the lead needed, and whether that genotype matches its own sample.
- How clearly each sample matches one genotype: every scored sample, assigned or not, by its best match score (across) and its runner-up score (up). Below the dashed line the best genotype leads by more than the clear lead (grey shade, "one genotype clearly best"; a mismatch can lie there too); on the dotted diagonal the two tie. Samples without an assigned ID are hollow markers. The report calls qpgen's Z-score the "match score".
- Duplicates: samples that match two genotype IDs about equally (see the rule above), with the two scores and the next one. A compact line says when there are none. Mismatches and relatives are not listed here.
- All samples: searchable, sortable table (the page scrolls; the header stays on screen), 500 rows per page by default (100 on phones), with a TSV download of the current view. Columns (short one-line headers, fixed widths): sample, result, own score, own rank, best other (ID), its score, lead (own score minus the best other score; negative when another genotype fits better, red only for mismatches; for samples without an assigned ID, the best genotype's lead over the next, in italics) and suggested ID; an "Assigned ID" column appears when some assigned IDs differ from the sample IDs. On phones the table shows sample (with the best other genotype under it), result, own score and lead.
- Run details (collapsed): settings, the trait-weight histogram, the report's result
names with their
qpgen_statuscodes, the qpgen command and a log excerpt.
Interpretation¶
FAILwith aproposed_geno_idand areciprocalpartner: a sample swap; relabel both. Areciprocal-weakpair is not evidence of a swap on its own.- One-way
FAILwhose best genotype belongs to aPASSsample: a mismatch; no relabel is proposed because that individual already has a matching omics sample. Exclude the sample or check the lab records. - A sample in
duplicates.tsv: its two leading genotypes are probably one person under two genotype IDs (a duplicated genotype sample or identical twins). When the own ID is one of the two, the omics label is fine: check the genotype set. qpgen calls itMULTI_NEW_BEST(orSELF_BESTwhen the own ID ranks first by a hair). - One-way
FAILwith aproposed_geno_id: the sample probably comes from that individual, who has no confirmed omics sample of their own; relabel after checking. - Other
MULTI_NEW_BEST: several leading genotypes, possibly a mixed sample or a block of close relatives. WARN(SELF_LENIENT): the own genotype is close to the best. With few informative traits this is noise; a larger reference separates better.- Many
UNCLEARsamples or low Z overall: too few or weak reference traits, a wrong trait map, or phenotypes in another orientation.
After match
cis and trans require identical sample IDs in genotypes and phenotypes and do not
read the match results. Rename or drop samples in your input files before running them.
Example data¶
On the shipped MAGE chromosome 21/22 example (729 samples × 367 genes, 729 genotyped individuals, known eQTLs from GTEx), with the defaults:
| Group | Count | Statuses |
|---|---|---|
| PASS | 713 | SELF_BEST |
| WARN | 16 | 14 SELF_LENIENT; 2 UNCLEAR: NA11919 (own Z 0.41, rank 253; best HG00237 at 3.60 leads the next by only 0.32) and NA12717 (own Z 1.94, rank 13; every candidate at background level) |
| FAIL | 0 | |
| NO_TEST | 0 |
All 367 traits are used, 14 with weight ≤ 0 (median 0.24, range −0.35 to 0.80), and 8 samples are flagged as possibly related. The use case explains these results and how they change with the thresholds and scoring options.