Skip to content

Concepts

Varda models the parts of dimensional design that a general modeling language has no words for. LinkML already knows about classes, slots, ranges, cardinality, enumerations, inheritance and imports. It does not know about facts, dimensions, grain, additivity or slowly-changing dimensions, because those are dimensional-modeling concepts rather than general modeling concepts.

Three of them carry most of the weight: what one row of a fact is, how a measure may be aggregated, and what happens when a dimension value changes. Everything below builds to those.

Tables have a role

Every annotated class declares varda:role: FACT, DIMENSION or BRIDGE. Nothing is inferred from the class name — a table named FctSale is a fact because it says so, which is what lets one organization write Fct, another Fact, and a third nothing at all.

Facts are measurement events. They have a declared grain, foreign keys to dimensions, and zero or more measures. Nothing points at them.

Dimensions are descriptive context. One surrogate key, at least one natural key, and attributes that are almost always textual or categorical.

Bridges resolve a many-to-many — a customer in three segments — and usually carry an allocation factor, because a customer counted once per segment is a customer counted three times.

Columns have a role too

varda:role on a slot turns a flat list of columns into a structure that generators and validators can reason about.

Role On What it is
SURROGATE_KEY dimension The meaningless key facts join to. Exactly one.
NATURAL_KEY dimension The business identity a loader matches on.
DURABLE_KEY dimension One entity across all its versions. At most one.
FOREIGN_KEY fact, bridge, dimension A reference to another table's surrogate key, or its durable key.
DEGENERATE_DIMENSION fact An identifier with no dimension table of its own.
MEASURE fact, bridge A numeric quantity that is aggregated.
ATTRIBUTE any Descriptive context, grouped and filtered on.
VERSION_START, VERSION_END, IS_CURRENT, VERSION_NUMBER type-2 dimension What marks one version of a row off from another.

The natural key is what a loader matches on — what makes two source rows the same business entity. Without one, every load either creates duplicates or has the matching rule written somewhere the model cannot see. The surrogate key is the meaningless integer the facts actually join to.

A degenerate dimension is an identifier living on the fact with no dimension table of its own — an order number, a ticket reference. It groups the lines of one transaction and has no attributes worth a table.

The versioning roles and the durable key belong to history, below.

Grain: what one row is

A fact declares its grain twice, in two forms that do different jobs.

varda:grain names the columns at which rows are unique — the set that together identifies exactly one row. This is the grain proper: in every formal treatment of dimensional modeling the grain is that set. Stating it makes the claim testable. V203 checks the columns exist, V204 checks each is a foreign key or a degenerate dimension — a grain is what a row is, not what it records — and the SQL generator turns the set into a UNIQUE constraint. Whether the database polices that constraint on every write, or the claim is checked once per load by the generated assertions instead, is what --constraints chooses; either way the claim is stated somewhere a machine reads.

varda:grain_statement says the same thing in words, conventionally "one row per …". It carries what a column list cannot: why those columns, and what a row means to somebody reading the model rather than querying it. A grain that cannot be stated in one sentence is a sign the table is doing two jobs, and writing it is most of the value. V202 checks it is a sentence and nothing further.

Nothing compares the sentence to the columns, deliberately — see Design notes.

Additivity is the expensive one

Every measure declares varda:additivity.

ADDITIVE Sums across every dimension. Sales amount, quantity, cost.
SEMI_ADDITIVE Sums across some dimensions but not others — almost always not across time. Account balance, inventory level, headcount.
NON_ADDITIVE Never sum. Ratios, percentages, unit prices, temperatures. Aggregate the components and recompute.

A semi-additive measure must also declare varda:semi_additive_over, naming the foreign keys it may not be summed across — one name, almost always the date, or a list — and V703 checks each is actually a foreign key of the table. A constraint naming a dimension the fact does not have is a constraint that silently never applies, and it looks exactly like one that does.

It should also say what it takes across them instead of the sum, because "not the sum" is half an answer and every report would otherwise supply the other half for itself:

quantity_on_hand:
  annotations:
    varda:role: MEASURE
    varda:additivity: SEMI_ADDITIVE
    varda:semi_additive_over: date_key
    varda:semi_additive_aggregate: LAST
varda:semi_additive_aggregate Across the exception, take For
LAST the value at the latest member a closing balance, month-end stock
FIRST the value at the earliest member an opening balance
MIN the lowest value the low point of a balance
MAX the highest value peak headcount
AVERAGE the mean over the members present an average daily balance

Across every other dimension the measure still sums: month-end stock for all stores is each store's month-end stock, added up. LAST and FIRST pick a member by its place in an order, so they take one dimension to order by (V709); a measure semi-additive over two dimensions takes MIN, MAX or AVERAGE. Leaving the aggregate out is a warning (V710) rather than an error: the model is not wrong, only unfinished.

Why this family is error and not warning

Summing something that must not be summed produces a number that looks fine. Nobody's query breaks, no test fails, and the wrong figure reaches someone who acts on it. This is the one class of modeling error that is both easy to make and invisible afterwards.

History: what happens when a value changes

varda:scd declares how a dimension responds when a source attribute changes: TYPE_0 retain the original, TYPE_1 overwrite it, TYPE_2 add a row.

It sits on the table rather than per column, deliberately. A mixed dimension — some attributes overwritten, others versioned — is a genuinely harder object that this core does not model. Split it into two dimensions, or handle it in an extension.

V502 is a warning rather than an error, because a great many dimensions are genuinely type 1 and saying so feels like ceremony. It stays a rule because "we never decided" and "we decided overwrite" look identical in the model and cost very differently two years later.

How a type-2 dimension versions

Type 2 keeps a row per change. It does not say how the current row is found, because the field does not agree on that, and three mechanisms are all in normal use:

Strategy Columns The current row is
window VERSION_START + VERSION_END the one whose end is null or far-future
flagged VERSION_START + IS_CURRENT the flagged one; the end derived from the next row
counter VERSION_NUMBER the highest per natural key

Varda accepts all three, and insists that something able to discriminate be marked — a start or a counter. A dimension versioning by an undeclared mechanism cannot have its uniqueness generated. That constraint is the return on saying so: a type-2 dimension is unique on its natural key plus the discriminator, never the natural key alone, which would reject the second version of every row.

IS_CURRENT is not a discriminator. It is true of exactly one version, so a key carrying it lets every superseded row repeat — which is why all three strategies above pair the flag with a start rather than relying on it.

The period follows SQL:2011 — closed at the start, open at the end. A row is in force from VERSION_START up to but not including VERSION_END, which is what stops consecutive versions overlapping at their boundary.

Version time is usually not business time

VERSION_START is named for the version rather than for validity because in practice it records when the warehouse noticed a change, not when the change was true. dbt's dbt_valid_from and Data Vault's LOAD_DATE are both this, whatever they are called. Keeping both would be bitemporality, which this core does not model.

Reporting as it was, and as it is now

A fact's foreign key to a type-2 dimension names one version: the customer as they were when the sale happened. That is usually the right answer, and sometimes the wrong question — "sales by the region each customer is in today" needs every version of a customer to lead to the same row.

The DURABLE_KEY role is the column that makes this possible: one value per entity across all its versions, assigned by the warehouse and kept when the natural key itself changes. A fact carries it beside the surrogate key, as a second foreign key saying varda:lands_on: DURABLE_KEY — Kimball's type 7:

DimCustomer:
  annotations: {varda:role: DIMENSION, varda:scd: TYPE_2}
  attributes:
    customer_key:         {annotations: {varda:role: SURROGATE_KEY}}
    customer_id:          {annotations: {varda:role: NATURAL_KEY}}
    customer_durable_key: {annotations: {varda:role: DURABLE_KEY}}
    # ... the versioning columns

FctSale:
  attributes:
    customer_key:          # the customer as they were
      annotations: {varda:role: FOREIGN_KEY, varda:references: DimCustomer}
    customer_durable_key:  # the customer as they are now
      annotations:
        varda:role: FOREIGN_KEY
        varda:references: DimCustomer
        varda:lands_on: DURABLE_KEY

Join on the first for history, and on the second plus the current flag for today's attributes.

A durable key repeats once per version, exactly as the natural key does, so Varda derives its uniqueness the same way — UNIQUE (customer_durable_key, valid_from) — and the history expectations are checked per durable key as well as per natural key. For the same reason no FOREIGN KEY can land on it: there is no single row to point at. The DDL says so in a comment, the SQLAlchemy module carries no ForeignKey for it, and the orphan check in assertions.sql still runs, because every value must name some version. V308 allows one durable key per dimension, V309 asks for it to be required, and V409 refuses a reference landing on a durable key its target does not have.

One step of history, in a column

Type 3 keeps a change without adding a row: beside category sits prior_category, holding what category was before its last change, so a report can group sales by the new grouping or by the one it replaced. varda:previous_of names the attribute a column is the earlier value of:

DimProduct:
  annotations: {varda:role: DIMENSION, varda:scd: TYPE_1}
  attributes:
    category:
      annotations: {varda:role: ATTRIBUTE, varda:max_length: 60}
    prior_category:
      annotations:
        varda:role: ATTRIBUTE
        varda:previous_of: category
        varda:max_length: 60

Both are attributes of one dimension, and the previous value stores the same type at least as wide, since every change copies one into the other (V507, V508). Not on a fact or a bridge, whose rows are not revised, nor on a TYPE_0 dimension, which keeps every original value (V509). Previous values may chain — prior_prior_category naming prior_category — but not in a cycle.

History stays a property of the table. A dimension whose columns change in different ways — some overwritten, some versioned — is still split in two or handled in an extension; the durable key is what lets the two halves, or the fact, find the same entity, and a previous value names what one column means rather than declaring a second slowly-changing type.

When one thing has two identities

varda:role says what part a column plays. LinkML's own unique_keys says which combinations are unique. They are different claims — a surrogate key is unique and is not a business key, and valid_from belongs in a type-2 dimension's unique key while being a version marker rather than an identity — so Varda reads both.

Most dimensions need only the roles. Varda derives the constraint from them: a type-0 or type-1 dimension is unique on its natural key, because neither keeps a second row for one business entity, and a type-2 dimension is unique on that key plus whatever marks its versions apart.

A dimension that declares no varda:scd gets no derived constraint, because the answer depends on which type it is and guessing wrong costs rows — V502 reports the missing declaration.

That derivation needs one natural key, and stops at more than one. A dimension loaded from several sources often has more than one, because each source identifies the thing its own way — and two of them are ambiguous in a way a role cannot resolve. Either they are one compound identity, a store known by its chain code and its store number together, or they are two alternatives, and the two want opposite constraints. So the model says which:

DimProduct:
  annotations:
    varda:role: DIMENSION
    varda:scd: TYPE_2
  unique_keys:
    by_barcode:
      description: Sources that supply a barcode.
      unique_key_slots: [gtin, valid_from]
    by_supplier_part:
      description: Sources identifying an item by supplier and part number.
      unique_key_slots: [supplier_code, supplier_part_number, valid_from]

Two keys are two constraints, and that is the whole point. Merging them into one is weaker than either alone, and inert as well: a row whose gtin is null passes a merged key without being checked at all. Kept apart, each row is caught by whichever key its own source populated.

Declaring them replaces the derived constraint rather than adding to it, so a table states its uniqueness in one place or the other. V306 requires it wherever there is more than one natural key, as an error: the merged constraint was derived silently for a long time and enforced neither identity, and deriving nothing instead would leave a dimension whose identity the database does not hold. V307 warns when a dimension's only natural key is nullable, because SQL counts nulls as distinct and the uniqueness then does not hold for exactly the rows a repeated load produces — where there are several keys, being absent is the point and it says nothing. V303 checks the columns exist, because LinkML accepts a key over a misspelled slot without complaint. V304 checks a business key on a type-2 dimension carries a version marker — without one it claims the business key does not repeat, which on a type-2 table is false. V305 warns about a natural key none of them cover.

Two sources, one thing, two rows

A unique key stops one source loading the same product twice. Nothing structural can tell you that a row from one source and a row from another are the same product — that is a matching decision the ETL makes, and Varda has no column for where it landed.

Hierarchies: how a dimension is drilled

varda:hierarchies declares the named paths a reader drills down. Levels run from least to most granular, which is the direction every system that models hierarchies uses:

DimStore:
  annotations:
    varda:role: DIMENSION
    varda:hierarchies:
      - name: geography
        levels: [country, region, store_code]

When the coarser levels are their own tables — a snowflake — the only geography columns on DimCity are the keys, and a path of keys reads as integers. Reach through the key instead:

DimCity:
  annotations:
    varda:role: DIMENSION
    varda:hierarchies:
      - name: geography
        levels: [country_key.country_name, state_key.state_name, city_name]

Both halves are checked: the near name must be a foreign key of this table, and the far one a column of the dimension it points at. A level that names a foreign key without reaching through it is an error, because 4718 is not something anyone drills into.

A hierarchy makes claims of two kinds. The path — offer country, then region, then store — is a declaration, and Varda checks that the levels are real columns, distinct, at least two of them, on a dimension, and the kind of column that can be a level. The rest is about data: that every member has a parent, and that it has only one. Those are checked against the data, in sql/expectations.sql, as far as the hierarchy says they hold:

Declared Checked
complete: true, the default A row holding a level holds the level above it.
complete: false Nothing: rows may skip a level — an online store sells to its country, with no region.
strict: true Each member of a level is in one member of the level above.
strict: false, the default Only for a level reached through a foreign key, which names one row on its own.

strict is off by default for the reason the next section gives: a plain level's column usually names a member only under its parent. Say it where each level's values stand alone — a month written 2024-03 is one month, in one year — and every level is then identified by its own key. Neither roll-up is checked on a type-2 dimension, whose rows are versions; completeness is.

Three things go wrong often enough to be worth naming.

Two members can share a name. A level answers two questions: what a member is called, and which member it is. Usually one column does both. When it does not — city_name holds "Springfield" for cities in three states — nothing extra is needed, because the hierarchy has already said what tells them apart:

levels: [country_name, state_name, city_name]

A level's identity is its own key preceded by the key of every coarser level, so city_name is identified by country, state and city together. Varda derives that from the order; it is never written out.

Declare a key only when what names a level is not what tells its members apart — a level showing product_name where two products share a name:

levels:
  - brand
  - {column: product_name, key: sku}

The key defaults to the foreign key for a level reached through one and to the naming column otherwise. A surrogate key or a foreign key may be a key and never a column, which is one rule from both sides: they identify well and read badly.

V607 checks a declared key exists and is the kind of column that can identify something. Whether members are actually distinct is a claim about data, and nothing checks it.

Weeks do not roll up into months. [year, quarter, month, week, date] looks like the obvious calendar hierarchy and is wrong: a week straddles month boundaries and, under ISO 8601, year boundaries. Real calendars carry two paths that share their finest level and diverge in the middle, which is why a column may appear in more than one hierarchy:

    varda:hierarchies:
      - name: calendar
        description: How the business reports externally.
        levels: [calendar_year, calendar_month, calendar_date]
      - name: iso_week
        description: How operations plans its weeks.
        levels: [iso_year, iso_week, calendar_date]

A description is optional and earns its place exactly here: when a dimension carries several paths, the names alone rarely say which one a reader wants.

A hierarchy in a type-1 dimension rewrites history. Districts get redrawn and products get recategorized, and when the dimension overwrites, last year's numbers move into this year's regions. That is often what people want; it is never what they expect. A dimension whose hierarchy has to stay put under historical reporting is type 2, and the reasoning is the same one in History — the choice is cheap to make now and expensive to discover later.

Not every path is a list of levels. A reporting line is as deep as the organization is, and its branches end at different depths, so it is no fixed set of columns. Two relational forms hold it, and a hierarchy says which one with kind:

DimEmployee:
  annotations:
    varda:hierarchies:
      - name: reporting_line
        kind: PARENT_CHILD      # each row names its manager
        parent: manager_key     # a FOREIGN_KEY into DimEmployee
      - name: organization
        kind: BRIDGED           # the same tree, flattened
        bridge: BridgeReporting
        ancestor: manager_key   # the bridge's two keys into DimEmployee
        descendant: employee_key
        depth: depth            # 0 for a salesperson and themselves

The parent-child form is what a source system holds; the bridged one is what a query wants, because "every sale in this manager's organization" is then one join rather than a recursion. Each kind uses its own fields and no other's (V608): a parent is a foreign key into the dimension's own table (V609), and the bridge is a BRIDGE whose ancestor and descendant both reference the dimension (V610). The expectations check that no member is its own parent, that the bridge lists every member as its own ancestor, and that the depth is zero there and nowhere else. A longer cycle — A reports to B reports to A — needs a recursive query, which every engine spells differently, and is not checked.

A child with more than one parent is not a tree at all. That many-to-many is what a plain BRIDGE resolves.

What Varda does not model

Data quality — whether the rows are right — is a different layer with a different audience. Varda's rules answer "is this a legal dimensional model?", never "is this data correct" and never "is this a good model". The last is a review, and a rule that produces an argument gets switched off.