Extending¶
Varda's core is small on purpose. Cost centers, retention policy, data classification, ownership — these differ per organization, and none of them belong in Varda. The extension mechanism is not bolted on the side; it is the reason the core can stay small enough to be correct.
The smallest useful extension has no Python in it¶
Write a LinkML schema declaring your vocabulary:
id: https://acme.example/varda/acme
name: acme
prefixes:
linkml: https://w3id.org/linkml/
acme: https://acme.example/varda/acme/
default_prefix: acme
default_range: string
imports:
- linkml:types
enums:
Sensitivity:
permissible_values:
public:
description: May leave Acme.
internal:
description: Acme staff only.
restricted:
description: Named access list only.
classes:
AcmeTableAnnotations:
annotations:
acme:applies_to: table
attributes:
cost_center:
description: The cost center that pays for this table's storage.
required: true
retention_days:
range: integer
AcmeColumnAnnotations:
annotations:
acme:applies_to: column
attributes:
sensitivity:
range: Sensitivity
Then three lines of configuration:
From then on acme:cost_center is first-class. It is checked for typos, its
enum values are enforced, what the profile declares about each annotation is
held to — required, the range, multivalued — and varda ext lists it:
$ varda check mart.yaml
ERROR V001 DimStore
unknown table annotation 'acme:cost_centre'; declare it in acme.yaml or fix the typo
ERROR V002 DimSegment
acme:sensitivity is 'secret', which is not a value of Sensitivity;
expected one of public, internal, restricted
ERROR V006 DimStore
no acme:cost_center; acme.yaml requires it on every table
ERROR V007 FctSales
acme:retention_days is 'thirty', which is not a whole number
A structured annotation — a range that is one of the profile's classes — is
checked field by field, and written as a mapping. A class may name the one
field a bare value stands for, as Varda's Level does for column:
AcmeOwner:
annotations:
acme:shorthand: team
attributes:
team:
required: true
escalation:
range: string
That is Varda's own validator policing your vocabulary — which is what makes writing an extension worth doing rather than just agreeing on a convention in a wiki.
Adding rules¶
An extension with code behind it imports varda.ext and nothing else. Every
type it runs on is there, including the ones its rules are annotated with:
Extension, RuleSet, Finding, Generator, Context, the Model a
rule walks and everything under it, and analysis.
import pathlib
from collections.abc import Iterator
from varda.ext import Extension, Finding, Model, RuleSet
HERE = pathlib.Path(__file__).parent
RULES = RuleSet(tag="ACME")
@RULES.rule("ACME101", "error", "Every fact says how long its rows are kept")
def acme101(model: Model) -> Iterator[Finding]:
for table in model.facts:
if table.annotations.of("acme").get("retention_days") is None:
yield Finding(
table,
"no acme:retention_days; a fact grows until somebody says "
"when its rows go",
)
EXTENSION = Extension(
name="acme",
prefix="acme",
version="1.2.0",
profile=HERE / "acme.yaml",
rules=RULES,
package="acme_ext",
)
Write a rule for what a profile cannot declare. That every table names a
cost center is required: true in acme.yaml, and V006 enforces it; that
only facts name a retention period is a condition on the role, which no
LinkML declaration states.
A rule yields what is wrong and with what: the table, column, hierarchy or
unique key, and a message. The code and severity are the decorator's, and
the engine adds them — along with the file the subject was declared in, when
it arrived through imports:.
Ship it as a package advertising the entry point:
Every rule code must begin with the extension's tag, which defaults to the prefix upper-cased. That is checked at registration, where the error can name the offending rule — because a rule code is a public identifier that ends up in commit messages and exemption lists, and two meanings for one code is a suppression that silently changes what it suppresses.
Reading the model¶
A rule and a generator both walk a Model: frozen data, read once, holding
nothing from LinkML. model.tables are sorted by name, each table's
columns are in declaration order with inherited ones included, and the
selections a rule usually wants are already there — model.facts,
table.foreign_keys, table.measures, table.versioning.
Every element carries its annotations as written, across every prefix.
An extension reads its own with annotations.of, which strips the prefix
and sees nothing outside it:
A name the model wrote that should resolve to something — a foreign key's
target, a grain column, a hierarchy level's key — is a Ref: the name, and
the element it resolved to, or None. A model that fails varda check is
still read, so a rule can report what is wrong with it.
What a model says an element means is read from LinkML's own metaslots, not from annotations, and kept as the model wrote it:
table.uri # class_uri: "schema:Product", or None when none was declared
column.uri # slot_uri, likewise
table.mappings.exact # exact_mappings; also close, related, narrow, broad
column.unit_of_measure # every part of `unit`; column.unit is its label
model.enum("Channel") # an enum's values, and the `meaning` of each
model.expand("schema:Product") # "http://schema.org/Product"
An IRI the model did not declare is None rather than minted, and a CURIE
whose prefix the model never declared expands to None: both are a rule's
to report, as a Ref without a target is. What one of Varda's own values
means is model.vocabulary.meaning("ColumnRole", "VERSION_START"), already
expanded, because a model need not declare the profile's prefixes.
What Varda concludes from a model is analysis, not the model itself:
from varda.ext import analysis
analysis.unique_claims(table) # what the DDL makes UNIQUE
analysis.reference(column) # the table and key a foreign key lands on
analysis.creation_order(model) # every reference target before its referrer
analysis.identities(hierarchy) # what identifies a member of each level
The two carry different promises. The model's fields change only when the
vocabulary does. An answer from analysis may become stricter in a minor
release — a derived key that declines to guess in one more case — while
its signature holds. An extension that needs the model's literal statement
reads the model.
Adding claims¶
A rule asks whether a model is legal. A claim is what a legal model says
about the rows it will hold that no constraint can state — that one column
set determines another, that the versions of one entity never overlap.
Varda renders every claim as a query in sql/expectations.sql, returning
the rows that break it and nothing when it holds, and renders an
extension's beside its own.
An extension hands over a function from the model to its claims:
import itertools
from collections.abc import Iterator
from varda.ext import Claim, Determines, Extension, Model
def claims(model: Model) -> Iterator[Claim]:
for table in model.dimensions:
marked = table.annotations.of("acme").get("strict_hierarchies")
if str(marked).lower() != "true":
continue
for hierarchy in table.hierarchies:
levels = hierarchy.levels
for parent, child in itertools.pairwise(levels):
if parent.is_reference or child.is_reference:
continue
if parent.column is None or child.column is None:
continue
yield Determines(
hierarchy,
f"each {child.spec} is in one {parent.spec}",
(child.column,),
(parent.column,),
)
EXTENSION = Extension(name="acme", prefix="acme", claims=claims, ...)
That is a claim Varda does not make itself. It reads a plain level's column as naming a member only under its parent — several states have a Springfield — so it holds only a level reached through a foreign key to one parent. Acme's marked dimensions mean more, and say so here. The query carries who claimed it:
The shapes are closed, as the vocabulary is: Determines,
PeriodNonEmpty, PeriodsDisjoint, AtMostOneFlagged, FlaggedIsLatest,
Requires, Differs, Reflexive and Distance, each carrying its subject — the table, column or
hierarchy it is about — and a statement, the sentence a report shows. A
renderer dispatches on them exhaustively, so a shape only one renderer knew
would be a claim the others skip without a word. Every claim is about one
table and names only its columns, and one that does not, or names a table of
another model, stops the run with a ClaimError naming the extension: the
query would otherwise fail in the warehouse, long after the run that wrote
it. Varda's own are claims.implied(model), public on the same weaker
promise as analysis.
Adding generators¶
from varda.ext import Generator
def run(ctx):
return {"acme/inventory.csv": render(ctx.model)}
GENERATOR = Generator(
name="acme-inventory",
artifacts=("acme/inventory.csv",),
run=run,
)
run returns {path: content} and never touches the disk. That is what makes
generation fail closed. Paths are declared up front so that two extensions
claiming the same output file are caught at load rather than by one silently
overwriting the other.
An extension that declares a type — not an annotation, an actual LinkML
type a model will name as a range — puts it in a separate schema and points
Extension.types at it. That schema, and only that schema, goes into the
import map, so a model can write imports: - acme and get the type without
getting the vocabulary. A LinkML import is a union: anything an importable
schema declares turns up in every generator's output for every model that
imports it, which is why types refuses a class or an enum at load. An
extension adding annotations only — most of them — declares no types schema
and is not importable at all.
ctx is a closed record — the model, and the Options the invocation chose:
ctx.options.schema, ctx.options.dialect, and ctx.options.constraints,
one of enforced, asserted or none. All three are plain strings rather
than resolved objects, so a generator emitting something other than SQL can
ignore them and one emitting SQL for an engine the core has never heard of is
not limited to the core's table. Every field is defaulted, so a generator
written against an earlier version keeps working when a new one is added.
Output named by the model¶
Some output has a file per table, and its names exist only once the model is
read. A declaration ending in / claims a directory for that:
def run(ctx):
return {
f"acme/tables/{table.physical}.md": render(table)
for table in ctx.model.tables
}
GENERATOR = Generator(name="acme-pages", artifacts=("acme/tables/",), run=run)
The claim is the whole directory, and it is checked the way a file is, with three additions:
- Nothing may overlap it. Another generator declaring a file or a
directory inside
acme/tables/, or a directory containing it, is refused at load. Claims are compared ignoring case, because macOS and Windows do. - Every produced path is checked again when it is produced. The names
under a claim come from the model, so a path that leaves the output tree, is
spelled more than one way (
acme//x.md, a backslash), or differs from another only in case stops the run naming the generator — before anything is written. varda verifyjudges the whole directory. A file under it that the run did not produce — a table since removed from the model — is reported as no longer generated. So a hand-written file does not belong under a claimed directory; put it beside it.
Settings¶
What a generator needs to know and the model does not say — where the tables
physically live, the address a published page will have — goes in
varda.toml, in a table named for the generator:
The generator declares what it reads, and finds it on ctx.settings:
def run(ctx):
base = ctx.settings.get("base_url", "/")
...
GENERATOR = Generator(
name="acme-pages",
artifacts=("acme/tables/",),
settings={"base_url": str},
run=run,
)
A setting is a str, int, float or bool. The file is checked against
the declaration when the registry is built, and refused naming the file for a
table naming no registered generator, a key the generator does not declare, or
a value of the wrong type — a misspelled setting would otherwise do nothing
while reading exactly like one that works. A whole number is accepted where a
float is declared; a boolean is never accepted where a number is.
ctx.settings holds only this generator's table, read-only, and is empty
where the file gives none, so defaults are the generator's to choose. A file
rather than a flag, because it is committed beside the model: generate and
verify read the same one, CI reads what a developer does, and changing it
is drift verify reports like a change to the model.
Using Varda as a library¶
The command line is a thin layer over varda.api, so anything it does is
available without it — to a pre-commit hook, a dbt integration or a
notebook — with the same refusals:
from varda import api
model = api.load("mart.yaml") # extensions found from the model's directory
report = api.check(model) # report.errors, report.warnings
generation = api.generate(model) # raises api.NonconformingError on errors
api.write(generation, "out")
drift = api.compare(generation, "out") # what `varda verify` reports
assert drift.clean # drift.changed, drift.missing, drift.extra
api.load discovers the extension set the way varda check does: installed
entry points, and the nearest varda.toml above the model. To decide it
yourself, pass a Registry — Registry.of(EXTENSION) is Varda plus exactly
that extension and nothing installed, which is what a test of an extension
wants.
Severity¶
An extension may propose a different severity for any rule, including one it does not own:
Two extensions naming different severities for one code is refused, with both parties named:
group wants V705 at 'error' and team wants 'info'. Nothing here can
adjudicate that — set it in varda.toml, which overrides both
Resolving that by load order would make behavior depend on discovery order — a difference between two machines that nothing in either repository explains.
varda.toml overrides everything, because the repository can see itself and
an upstream cannot:
The three severities are error, warning and info. Varda's own rules use
only the first two — a rule that is arguable is a warning, and one that is
unambiguously illegal is an error. info is there for extensions, which
have reason to report things Varda does not: a house convention worth naming
in a build log without failing it, or a metric an extension wants to surface
alongside the findings. A rule registered at info never stops a run, not
even under --strict.
There is deliberately no severity floor an upstream can set and a team cannot lower. It would not work: the repository can exempt the rule outright, so a floor is a lock with its key taped beside it.
What an extension may not do¶
One party, one namespace; extensions add, they never redefine.
An extension may introduce annotations, enumerations and rules under its own prefix. It may not:
- claim a reserved prefix (
varda,linkml,skos,dcterms,owl, …) - claim a prefix another extension already has
- redeclare an enumeration or class another extension owns
- add a value to
TableRole,Additivity, or any other core enum
The last is the important one. Every generator dispatches exhaustively on those closed vocabularies and raises on a value it cannot map. Widening one from outside turns that discipline into a silent fallback path.
All of it is refused at load with an error naming both parties, rather than warned about. The cost of raising is a clear error at startup; the cost of warning is an estate that half-works for two years because nobody reads startup warnings.
A missing enum value is a request to file, not a reason to fork
Varda may widen its own enums in a minor release. That is backward compatible — no existing model breaks when a new value becomes legal.
Migrating an extension written for 0.3¶
The interface changed once, deliberately, before any extension outside this repository depended on it. What moved:
| 0.3 | Now |
|---|---|
from varda.rules import Finding, RuleSet |
from varda.ext import Finding, RuleSet |
from varda.model import DimensionalModel |
from varda.ext import Model |
Finding("ACME101", "error", str(table), msg) |
Finding(table, msg) |
Extension.reader("acme").get(table.cls, "cost_center") |
table.annotations.of("acme").get("cost_center") |
column.references — a string |
column.references — a Ref; .name is the string, .target the table |
table.unique_claims, level.identity |
analysis.unique_claims(table), analysis.identities(hierarchy) |
hierarchy.resolved, hierarchy.levels |
hierarchy.levels (the Levels), hierarchy.specs (as written) |
ctx.schema, ctx.dialect, ctx.constraints |
ctx.options.schema, ctx.options.dialect, ctx.options.constraints |
registry.using(EXTENSION) in a test |
api.load(path, Registry.of(EXTENSION)) |
Roles, additivity, slowly-changing types and fact types are StrEnums now.
A member compares equal to its string, so column.role == "MEASURE" keeps
working; formatting one with !r shows the enum rather than the string.
Seeing what is active¶
$ varda ext
varda 0.4.0 [varda:]
origin built in
rule tag V
rules 58
table varda:fact_type, varda:grain, varda:grain_statement, ...
column varda:additivity, varda:physical_name, varda:references, ...
generators sql, assertions, expectations, docs, sqlalchemy
claims rendered in sql/expectations.sql
acme 1.2.0 [acme:]
origin varda.toml (/srv/mart/varda.toml)
rule tag ACME
rules 2
table acme:cost_center, acme:retention_days, acme:strict_hierarchies
column acme:sensitivity
claims rendered in sql/expectations.sql
Varda itself appears in that list because it is an Extension. If the core
ever needed a privilege the mechanism does not offer, that would be a defect
in the mechanism — and this way it is a load error rather than something a
third party discovers the hard way.