Rename the Variation record to Variant; add a precomputed SNV characteristics search - #212
Merged
Conversation
IsolatesHTS no longer describes how DNASeq datasets are processed on this branch,
and both of its methods fail on the new shape:
- injectTemplates() derived per-sample gbrowse/jbrowse databases from getSampleList(),
which keys samples on organismAbbrev + datasetClassCategory + experimentName. The
new dnaseqExperiment class carries an empty category while its samples span two
("Genetic variation" for SNPs, "Structural variation" for CNVs), so the key cannot
match and getSampleList() throws "No sample names found for dataset ...".
- addModelReferences() registered SnpQuestions / SnpRecordClasses references. That XML
is not in the compiled model and is to be superseded by the variation record.
Both bodies are commented out rather than deleted, and the class stays in place as a
no-op so the presenters naming it keep resolving. getPropertiesDeclaration() is left
alone: presenters still supply hasCNVData, and DatasetInjector.addPropValues validates
supplied props against the declaration.
Note the deeper issue for whoever writes the replacement: keying the experiment ->
sample lookup on a single datasetClassCategory cannot work for an experiment that
aggregates SNP and CNV samples, even once dnaseqExperiment declares a category.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit 643ddb4 commented out the CommunityCount intermediate table and the communityCount column of OrganismAttributes in apiTuningManager.xml (it read from a hardcoded usercomments_userdb_betan.mappedComment), but left every consumer in place, so model load failed with: Database error while attempting to parse sqlQuery OrganismAttributes.organismAttrs: ERROR: column oa.communitycount does not exist Consumers brought into line with the producer, following the snpcount/popsetcount precedent already in these files: - organismAttributeQueries.xml: the communitycount column declaration and its select expression are commented out. - organismRecords.xml: the communitycount columnAttribute is commented out. - organismQuestions.xml: communitycount removed from five summary attribute lists, which is mandatory since WDK resolves every attribute a summary names. community_flag is kept, as a literal 'none' rather than derived from the missing column. That is exactly what "CASE oa.communitycount WHEN 0 THEN 'none'" produced for a zero count, so the organism page renders as it would with no comments, rather than losing an attribute that organismRecords.xml:305 and its red-dot <img> still use. Deriving it from NULL would have been worse than removing it: CASE NULL WHEN 0 falls through to ELSE, so every organism would have advertised comments it does not have. The individuals.txt ontology entry is left alone, matching how snpcount and popsetcount were handled: the category tree tolerates entries for absent attributes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These two questions were written by hand in geneQuestions.xml while the attribute
their summaries name is generated by the presenter:
summary="...,exprGraphAttrpfal3D7_microarrayExpression_Derisi_TimeSeries_RSRC_expr_graph"
Expression.java:136 builds that name and injects the textAttribute that defines it. So
a hand-written question depended on generated output, coupled only by a name string
with nothing enforcing that both exist. On an instance where this dataset is not
loaded, the presenter is skipped and the question survives, referencing an attribute
nobody generated:
Summary attribute field [exprGraphAttrpfal3D7_..._expr_graph] defined in question
[GeneQuestions.GenesByMicroarraypfal3D7_..._RSRC] is invalid
The questions are curated rather than generic -- they use the PFTimeSeries vocabulary
queries, an extra samples_fc_comp_generic param, and fold_change_chosen_display in
place of fold_change_chosen -- which is why the presenter sets hasMultipleSamples,
hasPercentileData and hasPageData to false and why they were hardcoded in the first
place. So they move to their own microarrayDeRisiTimeSeries.dst rather than being
folded into the generic expression templates, and MicroarrayDeRisiTimeSeries injects
them. Question and attribute now appear and disappear together.
Only three values are parameterised -- ${datasetName} for the question and graph
attribute names, ${includeProjects} and ${includeProjectsExcludeEuPathDB} for scoping,
both already set by Expression.java to precisely the literals that were hardcoded.
Display strings stay literal because they describe this experiment. addModelReferences
now derives the question names from getDatasetName() rather than spelling out the
dataset, so the references cannot drift from what is injected.
Not yet verified: that the generated question is equivalent to the deleted static XML.
Nothing is generated on an instance where this dataset is absent, so this branch can
only show that its absence is handled. Before this reaches a full-data site, run a
build with the dataset declared and diff the generated question against the static
version in this commit's parent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CompoundQuestions.CompoundsByFoldChange named one dataset's graph column in its summary:
summary="...,metaboliteProfiles_LlinasMetabolites_RSRC_metabolite_graph"
That attribute is injected per metabolomics dataset by metabolomics.dst, so naming a
particular dataset's copy in a generic search has two consequences: model load fails
wherever that dataset is not loaded ("Summary attribute field [...] is invalid"), and
where it is loaded the search shows that one dataset's graph regardless of what the
user searched for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One row per variant locus (4,390,908 in unidb_shu_a), backing a new "variation" WDK record that replaces the deprecated snp record. Holds only columns that are derived, aggregated, or require a join. Intrinsic per-locus facts -- variant_type, is_coding, all 22 snp_*/indel_* allele columns, the strain counts -- are read directly from apidb.VariationFeature, which is already one row per locus with a unique source_id. That split keeps ~570MB of derived strings out of the tuning table and gives every column one known source. Both internalDependency elements are load-bearing: built before TranscriptAttributes the gene aggregate is silently empty rather than an error, and built before GenomicSeqAttributes the project_id/organism join drops every row. Notable derivations: - gene_ids is a string_agg, not a lookup: 25,545 loci overlap more than one gene, so the multi-gene case is correct by construction. - most_severe_impact and effect_summary are split per caller, because snpeff and product_call disagree on 19% of paired calls and that disagreement is content rather than noise. - collapsed_allele renders both classes for a MIXED locus (e.g. "A>C; A>AC"), which a single major/minor pair would have destroyed. The SELECT was verified against unidb_shu_a before being committed: 4,390,908 rows, 100% join coverage against GenomicSeqAttributes, and the expected values at three loci covering SNV, INDEL, and MIXED. Design: docs/superpowers/specs/2026-07-30-variation-record-design.md and plans/2026-07-30-variation-record.md in agentic-veupath-dev. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record class, alias query, and the tuning-backed identity/location attributes. Builds green and loads by ID; 13 attributes registered. sequence_source_id and location are exposed as first-class non-internal attributes rather than being parsed back out of source_id, because they are the coordinate a future VCF/tabix lookup and an EDA sample join both key on. Two things differ from the written plan, both discovered by building it: - The snp imports in apiCommonModel.xml sit inside an "UNCOMMENT WHEN SNPS are AVAILABLE" comment block, so inserting after them would have silently disabled the variation record. Imports are placed after that block instead. - variationTableQueries.xml is not created yet. WDK's RELAX NG schema requires a querySet to contain at least one query, so the intended empty-but-valid placeholder fails validation. The file arrives with its first real query. Temporarily reads jbrestel.VariationAttributes; see Task 14 of the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First attribute query reading apidb.VariationFeature directly, per the sourcing rule: intrinsic per-locus facts come from the base table, only derived/joined columns come from the tuning table.
Both allele classes are preserved rather than collapsed: 129,850 MIXED loci populate snp_* and indel_* simultaneously and describe the same strains two ways, so a single major/minor pair would silently lose half the data. HGVS attributes carry help text explaining that the major-allele HGVS is empty whenever the major allele equals the reference.
Grouped together deliberately: these aggregates are what the future per-strain VCF-backed table will detail, so it lands beside them.
…lumns Gene linkage is an aggregate rather than a lookup, so the 25,545 multi-gene loci are correct by construction instead of by a later fix. Effect rollups stay split by caller: snpeff and product_call disagree on 19% of paired calls, 62% of which are product_call's strain-aware downstream_frameshift. Collapsed allele and MAF columns exist only because WDK tables render on record pages, so results pages need pre-aggregated values.
Review of the Task 8 commit found two issues. most_severe_impact_snpeff and most_severe_impact_product_call hold text, so WDK sorted them alphabetically (HIGH, LOW, MODERATE, MODIFIER) -- contradicting the severity ranking their own help text advertises. Adds computed rank columns and points sortingColumn at them, entirely within the WDK query so no tuning table change or rebuild is needed. Also completes collapsed_minor_allele_frequency's help text, which named itself derived but omitted the pointer to the canonical SNP Alleles and Indel Alleles sections that the spec requires and collapsed_allele has.
Overview shows both allele sections so a MIXED locus reads honestly. Summary columns use the collapsed allele and MAF, since results pages cannot render tables.
One row per transcript and observed codon. This is where gene strand lives, since strand is only unambiguous per transcript once a locus can overlap two genes.
The help said "Number of strains observed with this codon". It is not per-codon: it is per amino-acid PRODUCT, repeated across every codon row sharing that product. At Variant_Pf3D7_01_v3_29514 four synonymous V codons (GTA/GTC/GTG/GTT) all read 162 while the single A codon reads 1. Summing per-codon gives 649 strains at a locus whose distinct_strain_count is 160. Verified at scale rather than inferred from one locus: across all 1,192,971 product groups having more than one codon row, strain_count is constant within the group -- zero exceptions. A scientist summing this column would have over-counted badly, so the displayName now says "(per product)" and the help states the repetition explicitly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table with a visible Source column, not two tables, so caller provenance travels with every row and disagreements stay legible. LEFT joins TranscriptAttributes because na_feature_id is null for intergenic calls; an inner join would drop them silently.
Adds SNP Alleles, Indel Alleles, and Strain Statistics category nodes so the two allele classes read as distinct sections on the record page. Requires wb ontology; wb model alone does not regenerate the OWL and would leave every attribute uncategorized with no error.
The overview used '<dt><b>SNP Alleles</b></dt><dd></dd>' as a section header, but the client DROPS any dt/dd pair whose dd is empty. Both headers therefore vanished, and a MIXED locus rendered two identical Reference/Major/Minor triplets with nothing indicating which was the SNP view and which the indel view -- precisely the ambiguity the two-section design exists to prevent. Verified on Variant_Pf3D7_01_v3_12, which reads: SNP Reference A | SNP Major A (0.9316) | SNP Minor C (0.0598) Indel Reference A | Indel Major A (0.9316) | Indel Minor AC (0.0085) The same empty-dd rule is why Gene(s) and Most Severe Impact (Product Call) are correctly absent on that locus: both are genuinely null there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A textAttribute template of "$$allele$$ ($$frequency$$)" renders a bare " ()" when both operands are null, so every pure-SNV record page showed "Indel Major ()" and "Indel Minor ()" -- about 4M of the 4.39M records, reading as a rendering bug rather than as "no indel at this locus". Moving the concatenation into SQL with a CASE yields NULL instead, and the client already drops any dt/dd pair whose dd is empty (the same rule that correctly hides Gene(s) on an intergenic locus). Verified on all three variant types via the service: SNV -> snp_* populated, indel_* null INDEL -> indel_* populated, snp_* null MIXED -> both populated and on the rendered SNV page: zero occurrences of "()" in the overview. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Final review found a critical layout defect plus four cheaper ones. CRITICAL -- the three new category nodes were attached to the ontology ROOT, not to DNA polymorphism. individuals.txt column 2 is the parent IRI and column 3 its label; the hand-typed rows left both empty and put the IRI in column 4 (recordClassName) with 'category' in column 5, so each row asserted a nonexistent record class and declared no parent. SNP Alleles, Indel Alleles and Strain Statistics rendered as top-level peers of Genomic Location, ahead of it. Now verified as "Genetic variation > DNA polymorphism > SNP Alleles" in the assembled tree, with H2/H3/H4 nesting on the page. record_overview was parented to topic_2885 with record+download scope, unlike all 15 other record classes; it rendered as an ordinary attribute row labeled "Overview" inside the DNA polymorphism subsection and became a selectable report column carrying ~4KB of HTML per record. Now topic_0219 / record-internal. The tuning table joins DatasetPresenter but never declared the dependency, the exact failure mode its own comment warns about -- an undeclared dependency yields a silently stale or absent join rather than an error. linkedGeneIds rendered text byte-identical to gene_ids, so the page and column chooser showed two columns both named "Gene ID(s)" with the same content. Dropped; gene_ids is used directly. The four *_and_freq columns also duplicated their base column's displayName and are now "... Allele + Frequency". effect_summary_* had no help, so a blank "Effects (Product Call)" next to a populated SnpEff column read as "this caller found no effect" rather than "did not call here" -- true for 2,700,000 of 4,390,908 loci. Also set displayOrder within the allele and statistics sections: they sorted alphabetically, which put "Reference Allele" LAST. Now reference -> major -> minor -> HGVS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ence)" The old label "Strain Count" sat beside "Called Strain Count" (159) and "No-Call Strain Count" (57) and read as though those two should sum to it. They do not, and are not meant to. called + no_call is the genotyped panel -- 216 at Pf3D7_01_v3:29514, and call_rate = 159/216 = 0.7361 confirms that denominator on all 4,390,908 loci. distinct_strain_count is called + 1, verified as exactly +1 on every row, the extra strain being the reference, which carries the reference allele by definition. It is the denominator behind the allele frequencies. Renamed in both places it appears -- the attribute displayName and the record overview -- so the same number is not shown under two names. 160 now sits directly beside 159, making the +1 self-evident without help text. Reference-strain explanation confirmed by John rather than inferred. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the temporary jbrestel.VariationAttributes stub with ApidbTuning.VariationAttributes now that the tuning job has built it, and removes the TEMPORARY STUB comment from both query files. 6 SQL references flipped: 4 in variationAttributeQueries.xml, 2 in variationTableQueries.xml. The real table was compared against the stub before flipping rather than assumed equivalent: identical column sets and ordering, and EXCEPT in both directions returned 0 rows over all 4,390,908 rows. Aggregates populated as expected (gene_ids on 2,879,337 loci -- zero would have meant the TranscriptAttributes dependency failed to resolve). Record pages verified unchanged across all five test loci: SNV, INDEL, MIXED (both allele classes), multi-gene, and a caller-disagreement locus. Assembled SQL confirmed to read ApidbTuning via wdkQuery -showQuery. grep -rn jbrestel over Model/lib/wdk and Model/lib/xml returns nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These were authored in the agentic-veupath-dev control-plane checkout because that is where the build commands run from, but they describe work that lands entirely here. Moved to sit beside the code they specify, matching the convention already used for the EDA data tables and the fungidb strain-segment record. Content is carried over verbatim; the plan's progress block is stale relative to the commits on this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The progress block had been stale since Task 4, claiming Task 5 was next while tasks 5-14 were all committed on this branch, and all 91 step checkboxes were unchecked. Replaced with a task-to-commit map covering 0-14, and checked the boxes. Also retires the stub warning. Both halves are settled: Task 14 (2723cda) flipped all six query references to ApidbTuning.VariationAttributes, and a tuning run built the real table on unidb_shu_a (variationattributes1121, 1688 MB, 4,390,908 rows, plus the view; all 17 va.* columns the model reads resolve). jbrestel.VariationAttributes no longer exists in the database. The plan's original assumption that the flip would cost dev-instance buildability no longer holds, and said so misleadingly. Task bodies keep their jbrestel snippets — that is what was executed — with a banner not to copy them forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the question-side scaffolding for the variation record -- a variationParams paramSet, a VariationsBy querySet, a VariationQuestions questionSet, and their apiCommonModel.xml imports -- plus the first search, VariationBySourceId, ported from the deprecated NgsSnpBySourceId. Settled decisions worth the reader's time: the nine-project includeProjects list is deliberate and must not be narrowed to the subset appDb's loaded three; ID matching is exact source_id only, with the alias query named as the home for any future tolerance; the question inherits the record's default summary columns rather than overriding them. The four remaining snp searches are out of scope and explained: they are HSSS processQuery searches over per-strain data, which is the seam the record spec deferred. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five tasks: create the three question-side XML files, import them and confirm wb model loads, add the individuals.txt row and run wb ontology, verify the search in the browser, close out the spec. Uses a cheapest-rung-first verification ladder (xmllint, wb, the WDK service, then the browser) in place of unit tests, since the change is declarative model XML and the sanity model is dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three new files -- a variationParams paramSet holding the ID datasetParam, a VariationsBy querySet, and a VariationQuestions questionSet -- giving the variation record its first search, ported from the deprecated snp record's NgsSnpBySourceId. Not yet imported into apiCommonModel.xml, so this commit changes nothing that builds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"See spec section 5.1" did not say which of the two specs in docs/superpowers/specs/ it meant. The claim is accurate against the searches spec, but the record spec's section 5 has no subsections, so a maintainer who opens the record spec -- the likelier bookmark when editing this record -- finds nothing and concludes the comment is stale. Names the spec, matching the disambiguation style already used in variationQueries.xml. Comment-only; no element, attribute, value or SQL change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the new variation search files into the model. wb model succeeds and the assembled PlasmoDB model now contains VariationQuestions.VariationBySourceId. The search is not yet categorized, so it will not appear in the searches menu until the individuals.txt row lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Places the search under edamontology topic_0199 with menu and webservice scopes, matching the Popset ID search and the parent already used by the variation record's gene-linkage and effect-rollup attributes. Verified by running wb ontology and grepping the freshly built gus_home categories_merged.owl: the class appears exactly once, with rdfs:subClassOf topic_0199, targetType search, and scopes menu and webservice. The /service/ontologies/Categories check could not be run -- the browser is blocked at the veupathdb.org pre-release autologin gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scaffolding and VariationBySourceId are built and verified on the plasmodb dev instance: wb model and wb ontology both pass, the search appears in the searches menu, a single ID goes straight to the record page, and a multi-ID run shows the record's default summary columns. Records in the plan what was substituted and what remains unrun -- the WDK service endpoints were unreachable behind the autologin gate so server-side equivalents were used, and the empty-project case still needs a site whose project lacks variation data. The four HSSS-backed searches remain out of scope pending the per-strain seam. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"SNV Gene ID(s)" and "SNV Genomic Location" named their input; all four HSSS searches compute differences within a sample group, so they now say so: "SNV Differences for Gene(s)" and "SNV Differences for Genomic Location". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…search The sibling of GenesByNgsSnps. That one runs HSSS over a sample set the user picks and takes minutes; this one reads apidbtuning.GeneVariationSummary, returns in milliseconds, and reports the same numbers the gene record page shows. Neither replaces the other: dropping the HSSS search would delete the sample-set comparison, and not having this one leaves the record page and the search disagreeing about the same gene with no explanation. GenesByNgsSnps is renamed to "SNV Characteristics Within a Group of Samples" to say so. Filtering is ONE filterParam over sixteen statistics, not sixteen range params. The range-param version was written first and abandoned: a numberRangeParam cannot express "untouched", so every range applies on every run and any gene NULL in that column is excluded - and GeneVariationSummary NULLs its frequency statistics below their allele floors on purpose, which for tbruTREU927 is 100% of rows for three columns. Working around that needed a full-span escape clause with bounds hardcoded in two files, which then broke when NumberRangeParamHandler rounded a value onto the declared ceiling. The filterParam has none of it: a gene with no value contributes no metadata row, distributions come from the data, and adding a statistic is one ontology row plus one VALUES tuple. Also adds organismVQ.withVariantSummary, scoping the organism tree to organisms that actually have a summary row rather than to a datasource type. Verified against unidb_shu_a on a dev instance: the impact distribution sums to all 5,579 pfal genes (HIGH 2,797, matching SQL), piN/piS reports 2,370 - exactly the genes with a non-null value - and filtered runs match direct SQL counts (piN/piS >= 1.5 -> 271 genes/272 transcripts; n_lof >= 5 -> 317/319). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ones TranscriptRecordClass already publishes this tuning table's sortable subset, and gene_pi_n_pi_s there is gvs.pi_n_pi_s_snpeff - the same column the search's filter reads. The dynamic columns were showing identical numbers under parallel names, which is a divergence waiting to happen. Drops the dynamicAttributes block and the gvs_* columns from the id query, leaving it returning only the four columns WDK needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Not just the wsColumn: both postCacheUpdateSql blocks enumerate the result columns explicitly in their INSERT and SELECT lists, so without adding it there the sibling transcripts they backfill would carry NULL span density while the matched transcripts carried a value. Required by the plugin change in ApiCommonWebService 7a61e99 - getColumns now advertises span_snp_density, and PluginExecutor.validateColumns rejects a plugin advertising a column the wsColumn list lacks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds SNVs per kb (gene span) alongside the now-actually-CDS density, renames the ratio to say it is site-normalized, and corrects three labels that never matched the classifier: nonsynonymous excludes stop-gained, and non-coding is really unclassified - class 0 means no product byte was available, which covers unclassifiable positions as well as genuinely non-coding ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Density is CDS density, the ratio is site-normalized, and the class enum terms follow the corrected column names. Internal enum values are unchanged - they are a contract with legalParams and with the filter's branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PlasmoDB description promised normalized Dn/Ds in a future release; it is here. Also states in both descriptions that these statistics are sample-set scoped and so not comparable to the precomputed ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cription The bullet keeps saying the ratio is site-normalized; it no longer quotes the 17.49% figure or the 1.43x factor at the user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These are computed over the samples the user selected, but a results table can carry them alongside the whole-cohort columns from the gene record (Total Variants, Variants per kb, piN/piS) with nothing to tell them apart. Each now carries a (sample group) qualifier. The two density labels swap their existing parentheses for commas so no header ends up with two bracketed clauses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CASE in SampleOntologyByStudy translates EDA data_type values into WDK filter-param types, and NULL is meaningful: it marks a BRANCH node. So an unmapped type does not degrade gracefully - it makes a LEAF look like a branch, and WDK throws "ontology items have no children ... but have a null item type", 500ing the whole search. 'date' had no arm, which broke the 13 dnaseq studies carrying a Collection date. The partial test appDb held none of them, so nothing exercised it. Add the date arm and record the invariant in the comment: every data_type EDA can emit needs a case here. The full appDb uses category/date/integer/number/ string, all now mapped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IsolatesHTS was a no-op, so no dnaseq:: properties were injected at all and
/service/jbrowse/dnaseq/<org> answered {"tracks":[]} for every organism.
Re-enable only the per-dataset half. The per-sample half is not repaired but
removed: it called getSampleList(), which keys samples on organismAbbrev +
datasetClassCategory + experimentName, and the merged dnaseqExperiment class
carries an empty category while its samples span two ("Genetic variation" for
SNPs, "Structural variation" for CNVs), so that key cannot match and the call
throws. Dropping the per-sample path removes its only caller.
The sample list, and which measures each sample has, now come from the
webservices tree at request time - the pipeline that writes those files is the
only thing that knows them, and LOH in particular exists only for diploid
organisms, which no per-dataset property could express.
Retiring the template also means retiring its anchor: presenterInjectTemplates
fails on an anchor naming a template that no longer exists, and no amount of
re-running fixes it.
hasCNVData is no longer emitted. 118 presenters say true and 77 say false, but
the pipeline now writes _normalisedCoverage.bw regardless, so the flag stopped
describing the data. It stays declared in getPropertiesDeclaration because
presenters still pass it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both classes derive label, id and track type from the dataset, which suits one track per dataset but not a caller emitting several per sample. label is the JBrowse track id and must be unique across the whole response, so uniqueness needs to come from construction rather than from sample names happening to be distinct. SingleCoverageTrackConfig: honour an explicit label, and an explicit track_type_display. The latter matters because the existing "Coverage (ploidy Normalized)" default is applied to every order-less, dbid-less caller - correct for the CNV track it was written for, wrong for a caller emitting SNP density or LOH. MultiBigWigTrackConfig: honour explicit label and id, applied last so they also beat the jbrowse2 branch. Add has_alignment => 0 for callers with no unique/non-unique read split at all; without it every track claims "unique and non-unique" and shows up under the RNA-Seq Alignment facet. track_type_display is deliberately NOT overridable here - the ::Density and ::XY subclasses set it after SUPER::new returns, so anything set here is clobbered, and a hook that silently does nothing is worse than no hook. All defaults are unchanged, so existing callers are unaffected: the only other caller of either class, JbrowseRnaAndChipSeqTracks, passes none of these. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shape by variant type and colour by predicted effect, both resolved per feature
so one track can mix them: substitutions get a diamond, indels a box (which also
renders at the variant's real width, where an SNV is always one base).
Colour follows the varianteffect plugin's convention - green synonymous, purple
non-synonymous, red truncating, blue intron/other - read from snpEff's
Annotation (ANN field 2). Impact (field 3) was tried first and rejected: it has
only four ordinal levels, and as a warm ramp its top two were indistinguishable
at glyph size.
The exact hexes are load-bearing. The convention's nominal green/red pair sits at
deltaE 3.3 under deuteranopia, a hard fail on the most consequential pair here -
a silent change against a truncation. Under deuteranopia red and green separate
mainly by lightness, so red is deliberately dark (#b71c1c), lifting the worst
all-pairs separation to deltaE 9.7 with all four clearing 3:1 on white.
Two traps this file sets for anything added to it, both silent:
- A blank line inside a function body terminates the multi-line value. The
fragment then fails to compile and the key is left a plain string, so the
track renders in a default colour with no error anywhere. Neither function
here contains one; verify typeof is 'function' after building.
- Do not deref track.browser.config in a function used for glyph, the way
snpColorFxn does. getConfForFeature resolves glyph with a null track, so the
deref throws. Both functions here are self-contained and feature-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
glyph and track_type_display become optional args defaulting to today's values, so a caller can supply a per-feature glyph function while the existing per-sample ebi_VCF tracks are unchanged. The old default also hardcoded "VCF from VectorBase" as the user-visible track type, which is simply untrue on any other site. Raise maxFeatureScreenDensity to 50. This - not chunkSizeLimit - is what produces "Too many features to show" when zoomed out: CanvasFeatures.fillBlock refuses a block whose feature density exceeds it, and the JBrowse default is 0.5 features/pixel. A merged call set runs about 6.8 variants/kb, so in a ~1000px window a 640kb chromosome measures ~2.4 features/pixel and the largest ~3.3Mb P. falciparum chromosome ~12 - both well past 0.5. 50 clears a whole chromosome with headroom, and is deliberately not the 9999999 RNASeqJunctionTrackConfig uses, which would have JBrowse attempt 100k+ glyphs at genome scale. Raise the store fetch ceilings to match the size of a merged multi-sample call set, and emit all of these as numbers rather than the quoted string chunkSizeLimit used to send. Note VCFStore also sets a chunk_size_limit that nothing reads, so it does not participate in any of this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rewrite around a single glob of the organism's dnaseq/bigwig directory rather
than a build-time injected sample list. The old arrangement had the track list
and the files it pointed at coming from separate sources of truth; they diverged
and the endpoint silently served nothing. Reading the tree makes "a track
exists" and "its file exists" the same fact - and is the only workable source
for LOH, which the pipeline emits per sample for diploid organisms only
(tbruTREU927 has it, the haploid pfal3D7 does not).
The presenter still owns what only it knows - display name, summary,
attribution, category - so the two sources of truth are split by what each can
actually know.
Per sample: coverage and ploidy-normalised coverage as individual tracks, the
two density measures overlaid in one autoscaling XY track, and LOH standalone.
Coverage is deliberately not paired with anything: raw coverage runs to
3367-50590 against normalised coverage's ~5, and a ~200x gap on a shared axis
renders the smaller trace as a flat line. Pooling every sample's densities into
one plot was tried and reverted - with only two colours available, one per
measure, a few hundred overlaid samples are mutually indistinguishable. Data
volume was never the constraint; legibility was.
Also emits the merged annotated VCF as a variant track. It is per organism, a
merge across all experiments, so it is built once outside the dataset loop and
carries its own metadata, there being no single presenter to inherit from. Its
summary doubles as the legend, since shape and colour are the only channels
distinguishing type and effect on screen.
Removed with the old design: the BAM alignment track (b71 ships no BAMs), the
${organismAbbrev}_HTS_SNP_(.+)_RSRC regex (cannot match _dnaseqExperiment_RSRC
names), and a hardcoded afumAf293/V157-47 exclusion.
A missing bigwig directory for an organism that has dnaseq datasets is now a
hard failure naming the path, because an unreadable mirror must not look
identical to "this organism has no DNASeq data" - that equivalence is what let
the original bug hide. A declared dataset with no files is still skipped
quietly, since presenters are deliberately a superset of any one build.
Note this module's stderr is merged into its own JSON response by
JBrowseService.responseFromCommand, so a Perl warning corrupts the output. That
is why cov_max_score_default is always passed explicitly (the setter does
"$value + 0", and undef there both warns and stores 0) and why the VCF track
passes an empty-string rather than undef attribution.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The include had been commented out long before this branch, so the genome browser never asked for dnaseq tracks even once the endpoint returned them - they would have existed and been unreachable. The phenotypeTracks set already included it. This is what makes the volume user-visible: pfal3D7 contributes 1612 tracks, taking the faceted selector from 1392 to about 3000. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The accessor pair and the setChunkSizeLimit(10000000) call in new() wrote a chunk_size_limit that nothing ever read: the value the track config actually carries is emitted by VcfTrackConfig. Raising it here therefore looked effective and did nothing, which is worse than it being absent, so leave a note saying where the live one is. AlignmentsTrackConfig keeps its identically-named pair - that one is on a different key and its own getJBrowseObject genuinely emits it. Verified behaviour-neutral: the dnaseq endpoint's output is canonically identical before and after (byte comparison is meaningless here, since Perl randomises hash key order between processes). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two reasons, either sufficient on its own.
It was misplaced. The merged annotated VCF is one file per organism, a merge
across every DNASeq experiment, so it never belonged in a per-dataset module - it
had to be emitted outside the dataset loop with hand-written metadata precisely
because no single presenter owns it.
And it was slow where it mattered. The gene page's SNP links use the
geneticVariationTracks set, which includes organismSpecific but not dnaseq. With
the VCF on the dnaseq endpoint those links had to load and parse ~1600 per-sample
coverage and density track configs to draw variants; via organismSpecific the same
set is ~100. This also lets the geneticVariationTracks/syntenyTracks branch of
jbrowseTracks stay as it was - an earlier attempt added dnaseq to it, which would
have dragged all 1600 into the synteny view too.
Replaces the "SNPs by coding potential" unified SNP track, removed here. It was
backed by the SNP:Population REST feature, part of the SNP infrastructure the
Variant record supersedes, and was already unreachable: no organism in build 71
sets the hasSnp property that gated it, and the gate could not have fired anyway,
since the fallback is an empty hashref and {} == 1 compares a ref address to 1.
UnifiedSnpTrackConfig is deleted with it, having no remaining references.
Verified: dnaseq 1612 -> 1611 tracks with zero merged-VCF tracks, organismSpecific
102 -> 103 with exactly one, all of its glyph, colour, fetch and density settings
carried over intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SnpsGbrowseUrl was commented out and targeted the gbrowse_img CGI and the HtsAlignmentSNPsByClass gbrowse track, neither of which exists. Restore it against the merged annotated VCF, and rename it "Short Variants in the Genome Browser" - it covers indels as well as substitutions, so "SNPs" undersold it. snpsJbrowseUrl and snpsJbrowseFullUrl were live but pointed at "SNPs by coding potential", which no organism in build 71 emitted, so those links were already dead rather than merely dated. Same for dynSpanRecord's span link. Track sets are chosen per link rather than uniformly: the gene page's SNPs view and snpsJbrowseUrl use geneticVariationTracks (~100 track configs), while snpsJbrowseFullUrl keeps the full default set, which is the point of a "full" view. On dynSpanRecord the organism-dependent track name is appended in the two consuming URLs rather than inside the nested jbrowseTracks attribute they interpolate: there is no precedent in this model for a macro inside a composed textAttribute, so recursive expansion is unproven, whereas $$organismAbbrev$$ demonstrably expands in the consumers. jbrowseTracks keeps only the static part. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GenesByVariantCharacteristics covers indels as well as substitutions, so naming it "SNV Characteristics" was too narrow. GenesByNgsSnps keeps its "SNV Characteristics Within a Group of Samples" name deliberately: the per-sample-group statistics really are SNV-only. Also updates the four help texts that referred to the renamed search by its display name, which would otherwise send readers looking for a search that no longer exists. The fifth such reference is left alone - it points at the per-sample-group search, whose name is unchanged. Note "SNV Characteristics" is a prefix of "SNV Characteristics Within a Group of Samples", so these are anchored rather than blind-replaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 10, 2026
…eanup (#215) * Design: per-strain variant tables read directly from the merged VCF Rebuilds the legacy SNP record's per-strain and per-country tables on the Variant record, reading the published merged.ann.vcf.gz at request time instead of loading anything. Three deliverables: a VariantStrains table (one row per VCF sample), a VariantCountrySummary table (one row per country, country-having samples only), and a port of the legacy record-page strain filterParam. Two findings drove the shape. Record-page tables backed by a processQuery are NOT cached - SingleRecordAnswerValue routes them to TableFieldProcessQueryResult, which calls getUncachedResults(). So there is no cache table, no per-page-view DDL, and nothing to evict. This is specific to record tables; a process query reached any other way caches unconditionally, which is why deliverable C stays SqlQuery-backed. And the frequency semantics have to be taken from processSequenceVariations.jl rather than reinvented: chromosome-weighted denominators (not strain counts), IUPAC ambiguity codes for hets, AO/(RO+AO) read frequencies, and coverage-filled reference calls that carry no AO/RO and must render as 100%. Getting any of these wrong yields numbers that silently disagree with the attributes shown in the overview panel on the same page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Resolve the country attribute for the variant country summary Confirmed against genomicsdb_071n. The grouping attribute is provider_label ["country"], keyed on the label rather than the VAR_<hash> stable_id, which is a digest of the label and site-specific. Legacy used geographic_location, and porting that verbatim would have been wrong: both attributes exist in the P. falciparum dnaseq study, but country covers 536 of 537 samples against geographic_location's 111. Also records that 14 of the 62 dnaseq studies carry no country attribute at all, and that no fallback is worth building - 13 of them have no geography attribute of any kind, and the one exception (P. yoelii 17X) has a single populated value of "Lowlands West and Central Africa" across 6 samples. Table B renders empty for those organisms rather than erroring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Plan: implement the variant strain tables Eleven tasks, TDD throughout, over the design in 2026-08-11-variant-strain-tables-design.md. Task 1 retires both htsjdk risks before anything depends on them: whether the version-less dependency resolves, and whether VCFCodec tolerates this file's under-declared header. Tasks 2-5 build the WDK-free parsing and composition layer against a fixture that reproduces those header quirks on purpose. Tasks 6-8 add the two tables and their model wiring; 9-10 port the record-page strain filter; 11 verifies the whole instance, including an organism whose study has no country attribute. Every command is concrete: the locus SQL was run against genomicsdb_071n, the individuals.txt rows carry the verified 14-field tab layout, and the build uses wb full (a Java plugin changed) followed by wb ontology (individuals.txt changed) - wb model alone would silently miss both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix a self-contradiction in the plan's read-frequency definition The stated formula gave the alt branch AO[altIndex] and the reference branch sum(AO) - deliberately asymmetric, since reference support is measured against all non-reference reads. But the Step 5 code implemented both branches as sum(AO), so on a multi-allelic record a sample's read frequency would be diluted by reads supporting an alt it does not carry. Caught in spec review of Task 3, where the implementer had transcribed the code faithfully and the fixture - biallelic, like every real record - could not expose the disagreement. Latent rather than live: measured 548,040 alt calls across 40,000 loci of the real merged VCF with no record carrying more than one ALT and no sample carrying more than one nonzero AO, because write_vcf_entry emits one record per unique alt. Corrected anyway, and the measurement recorded, because a formula that is only accidentally right is a trap for whoever reads it next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix intsOf slot compaction and the coverage-fill wording Two corrections found in code review of Task 3, both defects in these documents rather than in the implementation. intsOf dropped '.' slots instead of preserving them, which compacts the array and misaligns every later index: AO="1,.,4" became [1,4], so a sample called for alt2 would read alt3's support as its own. That silently defeats the indexed lookup in readFrequency, which exists precisely for the multi-allelic case. It now preserves slot positions with a NO_VALUE sentinel, while still reporting a wholly-'.' value as absent, which is what the coverage-filled discrimination keys on. The design doc claimed a coverage-filled row has "every other FORMAT field '.'". Only the original FORMAT set is blanked - write_vcf_entry appends CA and DFS after the fill, so a filled row carries both and neither can be used to detect one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Record Task 4's as-built divergences from the plan Three review-driven changes: the SQLException catch narrowed to 42P01, the EDA-suffix guard split so a null suffix yields an empty map rather than an exception, and vcfPath validating its inputs instead of building 'build-null'. The plan's code snippets are left in place but marked non-authoritative for this task - git is. Recording the divergence rather than silently letting the plan drift, since a plan that disagrees with the code is what produced two of the defects found so far. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add the Strains / Samples table to the Variant record Backed by a processQuery reading the merged VCF at request time. Record-page table queries run uncached, so this persists nothing and needs no data load. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Hide the raw GT column on the VariantStrains table A merged genotype is no longer one real VCF genotype once a locus spans several merged VCF records (see the ApiCommonWebService fix on this same branch: MergedVcfReader now merges every record at a locus, and a sample with contributing alts on more than one record gets a slash-joined GT that was never a literal call). Requested by the user, and doubly right now that the value can be a synthetic join rather than a raw "0/1". Follows the existing internal="true" inReportMaker="true" idiom used on TranscriptProducts.gene_source_id: hidden on the record page, still available in the report maker and file downloads. * Hide Variant Products by Transcript from the variant record page TranscriptProducts enumerates per-transcript products for every possible alt allele, so on a multi-record locus it shows rows no strain can reach. Drop the 'record' scope (keeping 'download') so it stops rendering on the record page while remaining available in the report maker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use record-internal scope for TranscriptProducts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add the Country Summary table to the Variant record Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Say what a blank allele cell means in the Country Summary help At a biallelic SNP - the overwhelming majority - Other Allele is blank in every row, and Minor Allele is blank at a monomorphic country. Nothing said that a blank means "this country has no such allele", so an entirely empty column read as missing data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add the record-page strain filter's model pieces A filterParam has to live in a question, so VariantAlignmentForm is a query returning no rows whose only job is to host the params - the same device the legacy SnpAlignmentForm used, rewritten here because the snp block is unimported dead code. Points at SamplesMetadataByStudy rather than the WithRef variant: this filter selects VCF sample columns, and the reference strain is not one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix double-quoting and unscoped organism default in VariantAlignmentForm organismSinglePick defaults to quote="true", so WDK quoted the value and the '$$organismSinglePick$$' inside the vocab query VariantVQ.EdaSampleTableSuffix quoted it again, producing ''pfal3D7'' and a 500 from the searches service - the record page's strain filter rendered as a silently empty section. The missing queryRef also left the default coming from the unscoped organism vocabulary, giving an abbrev where that query matches on a scientific name. Both attributes now match the four VariantsBy* searches in this file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Tidy the Variant record page organization Reparent SNP Alleles, Indel Alleles, and Strain Statistics from the DNA polymorphism subheading up to Genetic Variation, so the Variant record's sections sit together rather than nested a level down. The GeneVariation* categories keep their DNA polymorphism parent. Set nine attributes the record summary already renders at the top of the page to record-internal, so they appear once rather than twice: Variant Type, Coding, Reference Strain, Gene ID(s), both Effects columns, both Most Severe Impact columns, and Minor Allele Frequency. They keep their download scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * More Variant record page tidying Set Allele, Sequence, Position, and Variant Call Set to record-internal - all four are already shown in the record summary at the top of the page. Rename the strain filter section to 'Explore Sample Details and Multiple Sequence Alignment Tool'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Hide the Genomic Location section on the Variant record page Location, NCBI Taxon ID, and Organism are the last three record-scoped attributes in that category and all three are already in the record summary directly above, so the section now has nothing visible and disappears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add the genome browser section to the Variant record page Mirrors the Gene record's SnpsGbrowseUrl: the same geneticVariationTracks endpoint and the same <abbrev>_dnaseq_merged_short_variants track, windowed on the variant position +/- 1kb with the position itself highlighted. The coordinates live in a new VariantBrowser attribute query rather than in VariantTuning, which also backs result grids - folding the organism join into that query would charge every row of every search for two attributes only the record page reads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Order the Variant record sections explicitly Every direct child of Genetic Variation had an empty displayOrder, so section order was implicit. Make it explicit, with the genome browser first: browser, SNP Alleles, Indel Alleles, Strain Statistics, Predicted Effects, Strains / Samples, Country Summary, the strain filter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix the Variant genome browser: undefined iframe src, and section order The two JBrowse URL attributes were marked internal="true", so the service never sent them to the client; the Gbrowse wrapper read undefined and the iframe resolved 'undefined&tracklist=0...' as a relative app route, giving a 404 on record-types/variation/records. They stay off the page body because nothing in individuals.txt references them, not because they are internal. Section order: WDK renders sub-categories before leaf attributes, so displayOrder=1 only made the browser first among the leaves - it still landed below Strain Statistics. Give it its own Genome Browser category ordered ahead of SNP Alleles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Give the Variant JBrowse URL attributes record-internal ontology rows The record page only requests attributes present in the category tree, so with no rows at all the client received neither URL and the iframe src was undefined. The gene page does exactly this for snpsJbrowseUrl and snpsJbrowseFullUrl: record-internal means fetched but never rendered, which is what these need. Clearing internal="true" in the XML was necessary but not sufficient. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move the genome browser to Genomic Location and widen it to 4kb Genomic Location is a top-level section above Genetic Variation and had no visible attributes left after the summary-duplicate cleanup, so putting the browser there both places it above everything and removes the extra Genome Browser category, which was nesting a section inside a section. Window is now the variant position +/- 4kb. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add JBrowse functions for the Variant record popup variantRecordUrlFxn rebuilds a Short Variant record id from a VCF feature: the merged VCF's ID column is '.' on every row, so the id comes from the locus. VCFFeature sets start = POS-1, and VariationAttributes.location IS the VCF POS, so start+1 recovers it - indels included, verified against real records. variantRecordTitleFxn carries the link to the full record page in the dialog title, which dijit renders as HTML. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add VariantTrackConfig: the merged VCF's Variant record popup A subclass rather than a flag on VcfTrackConfig, which is shared with addVCF's per-sample ebi_VCF tracks - those have no Variant records, so an unconditional onClick would link them to a 404. Left-click opens the Short Variant record (Overview + Predicted Effects) in an iframe dialog, with a link to the full record page in the dialog title. menuTemplate is deliberately not set: Util.deepUpdate merges arrays index-wise, so any menuTemplate here would silently clobber the leading entries of JBrowse's default menu - whose 'View details' is the raw-VCF escape hatch we want to keep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Return VcfTrackConfig to per-sample-VCF defaults c8baece added the glyph/track_type_display arg hooks and raised the rendering ceilings, all for the merged short-variant track. VariantTrackConfig now owns those, so the base class goes back to serving the per-sample ebi_VCF tracks it was written for. The string->number fix for chunkSizeLimit stays: that was a real bug independent of the merged track. BEHAVIOUR CHANGE: VectorBase's per-sample ebi_VCF tracks lose maxFeatureScreenDensity 50 and revert to JBrowse's default 0.5. On a single-sample VCF that is the behaviour they had before 2026-08-10, but if anything has come to rely on those tracks drawing while zoomed out, this is the commit that changed it back. fetchSizeLimit disappears too, but unlike the density change that is a pure return to pre-c8baecec8 behaviour: per-sample VCFs never set it before. Note for bisecting: this commit alone leaves the merged short-variant track broken. addMergedShortVariants still constructs VcfTrackConfig directly and still passes glyph/color/track_type_display, which this commit stops honouring, so the merged call set would render mislabeled and with a 10MB chunk ceiling. The next commit points that caller at VariantTrackConfig, which is where those values now live. Do not build from this commit in isolation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Build the merged short-variant track from VariantTrackConfig The subclass owns the glyph, colour and track type now, so the caller stops passing them - one way to configure each thing, not two. This also completes the previous commit: VcfTrackConfig stopped honouring those args, and this points the caller at the class that does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Model-side of the SNV work. Must merge together with
ApiCommonWebServicePR #20 — that PR's plugin advertises aspan_snp_densitycolumn andvalidateColumnsrejects it unless this PR'swsColumndeclaration is present.Renames the Variation record to Variant
The record class was
Variationwhile everysource_idit holds readsVariant_<sequence>_<location>, and its own attributes already said "Variant Type" and "Variant Call Set". "Variation" is the phenomenon; a record is one concrete allele at a locus.Renames the WDK layer only — record class, question set and its five searches, query sets, param set, category ontology nodes, and six model files — and sets
displayNameto Short Variant / Short Variants, naming what the record actually holds (SNVs and small indels).Deliberately unchanged:
urlNamestaysvariationso existing record URLs resolve (the feature is not public yet, so this can move separately);variation_sample_meta{,_a,_b}andchromosomeOptionalForVariationsare contracts with the HSSS plugins and renaming them breaks the searches at run time, not build time.The four HSSS searches now lead with SNV — HSSS computes over single-nucleotide variants only and cannot return an indel however the record class is named.
VariantBySourceIdis not prefixed, because an ID lookup goes to the tuning table and does return both.Adds GenesByVariantCharacteristics
A precomputed sibling to
GenesByNgsSnps. That one runs HSSS over a user-selected sample set and takes minutes; this readsapidbtuning.GeneVariationSummary, returns in milliseconds, and reports the same numbers as the gene record page. Neither replaces the other — dropping the HSSS search would delete the sample-set comparison, and without this one the record page and the search disagree about the same gene with no explanation.Filtering is one
filterParamover sixteen statistics, not sixteen range params. The range-param version was built first and abandoned: anumberRangeParamcannot express "untouched", so every range applies on every run and any gene NULL in that column is excluded — andGeneVariationSummaryNULLs its frequency statistics below their allele floors by design, which fortbruTREU927is 100% of rows for three columns. Working around that needed a full-span escape clause with bounds hardcoded in two files, which then broke whenNumberRangeParamHandlerrounded a value onto the declared ceiling. ThefilterParamhas none of it, and distributions come from the data rather than from hand-declared bounds.Result columns reuse the transcript record's own attributes rather than new dynamic ones —
gene_pi_n_pi_sthere isgvs.pi_n_pi_s_snpeff, the same column the filter reads, so the filtered value and the displayed value agree by construction.Corrects GenesByNgsSnps labels
Its statistics changed in PR #20, so the strings describing them changed with them: density is now genuinely CDS with the old gene-span value beside it, the ratio says it is site-normalized, "Nonsynonymous SNPs" → Missense SNVs (it never included stop-gained), "Non-coding SNPs" → Unclassified SNVs (class 0 means no product byte was available). All eight result columns carry a (sample group) qualifier, since a results table can hold them next to the whole-cohort columns from the gene record.
Also adds
span_snp_densityto bothpostCacheUpdateSqlblocks, not just thewsColumnlist — those blocks enumerate result columns explicitly and backfill sibling transcripts, so omitting it would leave siblings NULL while matched transcripts carried a value.Verification
Built and exercised on a dev instance throughout. Category tree keeps all 59 nodes after the rename with no stale
VariationRecordClassreferences; all five renamed searches resolve withexpandParams; the new search's filter distributions match SQL (impact classes sum to all 5,579 pfal genes; piN/piS reports exactly the 2,370 genes that have a value).Note on scope
This branch carries ~90 prior commits of dnaseq work authored before this session — CNV search ports, the
GeneVariationSummarytuning table, gene-record variation wiring. They are on the branch but are not part of this change.🤖 Generated with Claude Code
Related PRs
The variants work spans four repos. These four merge together —
ApiCommonModel's record rename is a contract the other three depend on by name:ApiCommonModelRename the Variation record to Variant; add a precomputed SNV characteristics search #212 — record rename + precomputed SNV characteristics searchApiCommonWebServiceFix SNV density and dN/dS in the sample-set SNV search, and revive its test suite ApiCommonWebService#20 — density and dN/dS fixes, test-suite repairApiCommonWebsitePrefix HSSS result IDs with Variant_ instead of NGS_SNP. ApiCommonWebsite#310 — HSSS result-ID prefix (NGS_SNP.→Variant_)web-monorepoShow the Variant record id without the record-type prefix web-monorepo#1841 — record-heading override, filename tracking the renameCoupling worth knowing when sequencing the merges: #1841's customization is resolved by
recordClass.fullName, so without #212 it matches nothing and the override silently stops applying — no error, just the default heading back. #310's prefix and #20's separator together produce the ids #212's record class resolves; any two without the third leaves HSSS emitting ids the record class rejects.Related but independent, cherry-picked onto
master— merge before the four above:EbrcModelCommonOptional dataset gate for presenter template injection (partial-appDb builds) EbrcModelCommon#131 — optional dataset gate for presenter template injection. Not part of the variants work; needed to build a site against a partial appDb.The release note originally in VEuPathDB/ApiCommonWebsite#310 has been deferred (reverted in
965009fdd) because itsdateattribute was an unresolvable placeholder. Recover the drafted text withgit show a039aeee9in ApiCommonWebsite and re-PR it once the release date is known.