Skip to content

Extending

Varda's core is small on purpose. Cost centers, retention policy, data classification, ownership — these differ per organization, and none of them belong in Varda. The extension mechanism is not bolted on the side; it is the reason the core can stay small enough to be correct.

The smallest useful extension has no Python in it

Write a LinkML schema declaring your vocabulary:

profiles/acme.yaml
id: https://acme.example/varda/acme
name: acme
prefixes:
  linkml: https://w3id.org/linkml/
  acme: https://acme.example/varda/acme/
default_prefix: acme
default_range: string
imports:
  - linkml:types

enums:
  Sensitivity:
    permissible_values:
      public:
        description: May leave Acme.
      internal:
        description: Acme staff only.
      restricted:
        description: Named access list only.

classes:
  AcmeTableAnnotations:
    annotations:
      acme:applies_to: table
    attributes:
      cost_center:
        description: The cost center that pays for this table's storage.
        required: true
      retention_days:
        range: integer

  AcmeColumnAnnotations:
    annotations:
      acme:applies_to: column
    attributes:
      sensitivity:
        range: Sensitivity

Then three lines of configuration:

varda.toml
[[extension]]
name = "acme"
prefix = "acme"
profile = "profiles/acme.yaml"

From then on acme:cost_center is first-class. It is checked for typos, its enum values are enforced, what the profile declares about each annotation is held to — required, the range, multivalued — and varda ext lists it:

$ varda check mart.yaml
ERROR V001  DimStore
        unknown table annotation 'acme:cost_centre'; declare it in acme.yaml or fix the typo
ERROR V002  DimSegment
        acme:sensitivity is 'secret', which is not a value of Sensitivity;
        expected one of public, internal, restricted
ERROR V006  DimStore
        no acme:cost_center; acme.yaml requires it on every table
ERROR V007  FctSales
        acme:retention_days is 'thirty', which is not a whole number

A structured annotation — a range that is one of the profile's classes — is checked field by field, and written as a mapping. A class may name the one field a bare value stands for, as Varda's Level does for column:

  AcmeOwner:
    annotations:
      acme:shorthand: team
    attributes:
      team:
        required: true
      escalation:
        range: string

That is Varda's own validator policing your vocabulary — which is what makes writing an extension worth doing rather than just agreeing on a convention in a wiki.

Adding rules

An extension with code behind it imports varda.ext and nothing else. Every type it runs on is there, including the ones its rules are annotated with: Extension, RuleSet, Finding, Generator, Context, the Model a rule walks and everything under it, and analysis.

acme_ext/__init__.py
import pathlib
from collections.abc import Iterator

from varda.ext import Extension, Finding, Model, RuleSet

HERE = pathlib.Path(__file__).parent
RULES = RuleSet(tag="ACME")


@RULES.rule("ACME101", "error", "Every fact says how long its rows are kept")
def acme101(model: Model) -> Iterator[Finding]:
    for table in model.facts:
        if table.annotations.of("acme").get("retention_days") is None:
            yield Finding(
                table,
                "no acme:retention_days; a fact grows until somebody says "
                "when its rows go",
            )


EXTENSION = Extension(
    name="acme",
    prefix="acme",
    version="1.2.0",
    profile=HERE / "acme.yaml",
    rules=RULES,
    package="acme_ext",
)

Write a rule for what a profile cannot declare. That every table names a cost center is required: true in acme.yaml, and V006 enforces it; that only facts name a retention period is a condition on the role, which no LinkML declaration states.

A rule yields what is wrong and with what: the table, column, hierarchy or unique key, and a message. The code and severity are the decorator's, and the engine adds them — along with the file the subject was declared in, when it arrived through imports:.

Ship it as a package advertising the entry point:

pyproject.toml
[project.entry-points."varda.extensions"]
acme = "acme_ext:EXTENSION"

Every rule code must begin with the extension's tag, which defaults to the prefix upper-cased. That is checked at registration, where the error can name the offending rule — because a rule code is a public identifier that ends up in commit messages and exemption lists, and two meanings for one code is a suppression that silently changes what it suppresses.

Reading the model

A rule and a generator both walk a Model: frozen data, read once, holding nothing from LinkML. model.tables are sorted by name, each table's columns are in declaration order with inherited ones included, and the selections a rule usually wants are already there — model.facts, table.foreign_keys, table.measures, table.versioning.

Every element carries its annotations as written, across every prefix. An extension reads its own with annotations.of, which strips the prefix and sees nothing outside it:

table.annotations.of("acme").get("cost_center")
column.annotations.of("acme").get("sensitivity")

A name the model wrote that should resolve to something — a foreign key's target, a grain column, a hierarchy level's key — is a Ref: the name, and the element it resolved to, or None. A model that fails varda check is still read, so a rule can report what is wrong with it.

What a model says an element means is read from LinkML's own metaslots, not from annotations, and kept as the model wrote it:

table.uri  # class_uri: "schema:Product", or None when none was declared
column.uri  # slot_uri, likewise
table.mappings.exact  # exact_mappings; also close, related, narrow, broad
column.unit_of_measure  # every part of `unit`; column.unit is its label
model.enum("Channel")  # an enum's values, and the `meaning` of each
model.expand("schema:Product")  # "http://schema.org/Product"

An IRI the model did not declare is None rather than minted, and a CURIE whose prefix the model never declared expands to None: both are a rule's to report, as a Ref without a target is. What one of Varda's own values means is model.vocabulary.meaning("ColumnRole", "VERSION_START"), already expanded, because a model need not declare the profile's prefixes.

What Varda concludes from a model is analysis, not the model itself:

from varda.ext import analysis

analysis.unique_claims(table)  # what the DDL makes UNIQUE
analysis.reference(column)  # the table and key a foreign key lands on
analysis.creation_order(model)  # every reference target before its referrer
analysis.identities(hierarchy)  # what identifies a member of each level

The two carry different promises. The model's fields change only when the vocabulary does. An answer from analysis may become stricter in a minor release — a derived key that declines to guess in one more case — while its signature holds. An extension that needs the model's literal statement reads the model.

Adding claims

A rule asks whether a model is legal. A claim is what a legal model says about the rows it will hold that no constraint can state — that one column set determines another, that the versions of one entity never overlap. Varda renders every claim as a query in sql/expectations.sql, returning the rows that break it and nothing when it holds, and renders an extension's beside its own.

An extension hands over a function from the model to its claims:

import itertools
from collections.abc import Iterator

from varda.ext import Claim, Determines, Extension, Model


def claims(model: Model) -> Iterator[Claim]:
    for table in model.dimensions:
        marked = table.annotations.of("acme").get("strict_hierarchies")
        if str(marked).lower() != "true":
            continue
        for hierarchy in table.hierarchies:
            levels = hierarchy.levels
            for parent, child in itertools.pairwise(levels):
                if parent.is_reference or child.is_reference:
                    continue
                if parent.column is None or child.column is None:
                    continue
                yield Determines(
                    hierarchy,
                    f"each {child.spec} is in one {parent.spec}",
                    (child.column,),
                    (parent.column,),
                )


EXTENSION = Extension(name="acme", prefix="acme", claims=claims, ...)

That is a claim Varda does not make itself. It reads a plain level's column as naming a member only under its parent — several states have a Springfield — so it holds only a level reached through a foreign key to one parent. Acme's marked dimensions mean more, and say so here. The query carries who claimed it:

-- dim_product.merchandise: each brand is in one category (claimed by acme)

The shapes are closed, as the vocabulary is: Determines, PeriodNonEmpty, PeriodsDisjoint, AtMostOneFlagged, FlaggedIsLatest, Requires, Differs, Reflexive and Distance, each carrying its subject — the table, column or hierarchy it is about — and a statement, the sentence a report shows. A renderer dispatches on them exhaustively, so a shape only one renderer knew would be a claim the others skip without a word. Every claim is about one table and names only its columns, and one that does not, or names a table of another model, stops the run with a ClaimError naming the extension: the query would otherwise fail in the warehouse, long after the run that wrote it. Varda's own are claims.implied(model), public on the same weaker promise as analysis.

Adding generators

from varda.ext import Generator


def run(ctx):
    return {"acme/inventory.csv": render(ctx.model)}


GENERATOR = Generator(
    name="acme-inventory",
    artifacts=("acme/inventory.csv",),
    run=run,
)

run returns {path: content} and never touches the disk. That is what makes generation fail closed. Paths are declared up front so that two extensions claiming the same output file are caught at load rather than by one silently overwriting the other.

An extension that declares a type — not an annotation, an actual LinkML type a model will name as a range — puts it in a separate schema and points Extension.types at it. That schema, and only that schema, goes into the import map, so a model can write imports: - acme and get the type without getting the vocabulary. A LinkML import is a union: anything an importable schema declares turns up in every generator's output for every model that imports it, which is why types refuses a class or an enum at load. An extension adding annotations only — most of them — declares no types schema and is not importable at all.

ctx is a closed record — the model, and the Options the invocation chose: ctx.options.schema, ctx.options.dialect, and ctx.options.constraints, one of enforced, asserted or none. All three are plain strings rather than resolved objects, so a generator emitting something other than SQL can ignore them and one emitting SQL for an engine the core has never heard of is not limited to the core's table. Every field is defaulted, so a generator written against an earlier version keeps working when a new one is added.

Output named by the model

Some output has a file per table, and its names exist only once the model is read. A declaration ending in / claims a directory for that:

def run(ctx):
    return {
        f"acme/tables/{table.physical}.md": render(table)
        for table in ctx.model.tables
    }


GENERATOR = Generator(name="acme-pages", artifacts=("acme/tables/",), run=run)

The claim is the whole directory, and it is checked the way a file is, with three additions:

  • Nothing may overlap it. Another generator declaring a file or a directory inside acme/tables/, or a directory containing it, is refused at load. Claims are compared ignoring case, because macOS and Windows do.
  • Every produced path is checked again when it is produced. The names under a claim come from the model, so a path that leaves the output tree, is spelled more than one way (acme//x.md, a backslash), or differs from another only in case stops the run naming the generator — before anything is written.
  • varda verify judges the whole directory. A file under it that the run did not produce — a table since removed from the model — is reported as no longer generated. So a hand-written file does not belong under a claimed directory; put it beside it.

Settings

What a generator needs to know and the model does not say — where the tables physically live, the address a published page will have — goes in varda.toml, in a table named for the generator:

[generator.acme-pages]
base_url = "https://data.acme.example/mart/"

The generator declares what it reads, and finds it on ctx.settings:

def run(ctx):
    base = ctx.settings.get("base_url", "/")
    ...


GENERATOR = Generator(
    name="acme-pages",
    artifacts=("acme/tables/",),
    settings={"base_url": str},
    run=run,
)

A setting is a str, int, float or bool. The file is checked against the declaration when the registry is built, and refused naming the file for a table naming no registered generator, a key the generator does not declare, or a value of the wrong type — a misspelled setting would otherwise do nothing while reading exactly like one that works. A whole number is accepted where a float is declared; a boolean is never accepted where a number is.

ctx.settings holds only this generator's table, read-only, and is empty where the file gives none, so defaults are the generator's to choose. A file rather than a flag, because it is committed beside the model: generate and verify read the same one, CI reads what a developer does, and changing it is drift verify reports like a change to the model.

Using Varda as a library

The command line is a thin layer over varda.api, so anything it does is available without it — to a pre-commit hook, a dbt integration or a notebook — with the same refusals:

from varda import api

model = api.load("mart.yaml")  # extensions found from the model's directory
report = api.check(model)  # report.errors, report.warnings
generation = api.generate(model)  # raises api.NonconformingError on errors
api.write(generation, "out")
drift = api.compare(generation, "out")  # what `varda verify` reports
assert drift.clean  # drift.changed, drift.missing, drift.extra

api.load discovers the extension set the way varda check does: installed entry points, and the nearest varda.toml above the model. To decide it yourself, pass a Registry — Registry.of(EXTENSION) is Varda plus exactly that extension and nothing installed, which is what a test of an extension wants.

Severity

An extension may propose a different severity for any rule, including one it does not own:

EXTENSION = Extension(..., severity_defaults={"V705": "error"})

Two extensions naming different severities for one code is refused, with both parties named:

group wants V705 at 'error' and team wants 'info'. Nothing here can
adjudicate that — set it in varda.toml, which overrides both

Resolving that by load order would make behavior depend on discovery order — a difference between two machines that nothing in either repository explains.

varda.toml overrides everything, because the repository can see itself and an upstream cannot:

varda.toml
exempt = ["V705"]

[severity]
V502 = "error"

The three severities are error, warning and info. Varda's own rules use only the first two — a rule that is arguable is a warning, and one that is unambiguously illegal is an error. info is there for extensions, which have reason to report things Varda does not: a house convention worth naming in a build log without failing it, or a metric an extension wants to surface alongside the findings. A rule registered at info never stops a run, not even under --strict.

There is deliberately no severity floor an upstream can set and a team cannot lower. It would not work: the repository can exempt the rule outright, so a floor is a lock with its key taped beside it.

What an extension may not do

One party, one namespace; extensions add, they never redefine.

An extension may introduce annotations, enumerations and rules under its own prefix. It may not:

  • claim a reserved prefix (varda, linkml, skos, dcterms, owl, …)
  • claim a prefix another extension already has
  • redeclare an enumeration or class another extension owns
  • add a value to TableRole, Additivity, or any other core enum

The last is the important one. Every generator dispatches exhaustively on those closed vocabularies and raises on a value it cannot map. Widening one from outside turns that discipline into a silent fallback path.

All of it is refused at load with an error naming both parties, rather than warned about. The cost of raising is a clear error at startup; the cost of warning is an estate that half-works for two years because nobody reads startup warnings.

A missing enum value is a request to file, not a reason to fork

Varda may widen its own enums in a minor release. That is backward compatible — no existing model breaks when a new value becomes legal.

Migrating an extension written for 0.3

The interface changed once, deliberately, before any extension outside this repository depended on it. What moved:

0.3 Now
from varda.rules import Finding, RuleSet from varda.ext import Finding, RuleSet
from varda.model import DimensionalModel from varda.ext import Model
Finding("ACME101", "error", str(table), msg) Finding(table, msg)
Extension.reader("acme").get(table.cls, "cost_center") table.annotations.of("acme").get("cost_center")
column.references — a string column.references — a Ref; .name is the string, .target the table
table.unique_claims, level.identity analysis.unique_claims(table), analysis.identities(hierarchy)
hierarchy.resolved, hierarchy.levels hierarchy.levels (the Levels), hierarchy.specs (as written)
ctx.schema, ctx.dialect, ctx.constraints ctx.options.schema, ctx.options.dialect, ctx.options.constraints
registry.using(EXTENSION) in a test api.load(path, Registry.of(EXTENSION))

Roles, additivity, slowly-changing types and fact types are StrEnums now. A member compares equal to its string, so column.role == "MEASURE" keeps working; formatting one with !r shows the enum rather than the string.

Seeing what is active

$ varda ext
varda 0.4.0  [varda:]
    origin      built in
    rule tag    V
    rules       58
    table       varda:fact_type, varda:grain, varda:grain_statement, ...
    column      varda:additivity, varda:physical_name, varda:references, ...
    generators  sql, assertions, expectations, docs, sqlalchemy
    claims      rendered in sql/expectations.sql

acme 1.2.0  [acme:]
    origin      varda.toml (/srv/mart/varda.toml)
    rule tag    ACME
    rules       2
    table       acme:cost_center, acme:retention_days, acme:strict_hierarchies
    column      acme:sensitivity
    claims      rendered in sql/expectations.sql

Varda itself appears in that list because it is an Extension. If the core ever needed a privilege the mechanism does not offer, that would be a defect in the mechanism — and this way it is a load error rather than something a third party discovers the hard way.