Structural Data Quality

Structural Data Quality verifies that a connector implemented each enabled Input Layer Model correctly. It evaluates the Warehouse Tables or Views produced by Tuva Core's Input Layer Wrappers before users rely on downstream models.

How to Use Structural Data Quality

Structural Data Quality evaluates the Warehouse Tables or Views produced by Tuva Core's Input Layer Wrappers. Build the current Input Layer Models and Wrappers successfully before running it. Otherwise, an older Warehouse Table or View or an older Structural Data Quality table may still exist and appear queryable even though it does not represent the current input data.

Set data_quality_enabled: true and the applicable domain variables in the connector project's dbt_project.yml. For example, claims_enabled: true enables the eligibility, medical_claim, and pharmacy_claim Input Layer Models and their corresponding Wrappers.

Build the Input Layer and Structural Data Quality in Three Steps

The commands below assume the connector tags its staging models and final Input Layer Models with input_layer. Replace <your_connector_project_name> with the name declared in the connector's dbt_project.yml. If the connector uses a different selector, substitute its explicit staging and final model selector in Step 1. Run all three commands against the same dbt target and with the same domain-variable configuration.

Build the connector-owned Input Layer Models that map source-system data to Tuva's contract:
```
dbt build --select "package:<your_connector_project_name>,tag:input_layer"
```
After Step 1 succeeds, build the package-owned Tuva Core Input Layer Wrappers that reference those Models and create the Warehouse Tables or Views that Structural Data Quality inspects:
```
dbt run --select "package:the_tuva_project,tag:input_layer"
```
Only after every enabled Wrapper succeeds, build Structural Data Quality:
```
dbt build --select tag:dq_structural
```

The dq_structural tag builds the readiness matrix, the normalized structural results, the three failure-only detail tables, and the internal helper tables they require. If the connector does not define an enabled Input Layer Model, dbt fails while resolving the corresponding Wrapper. If an Input Layer Model or Wrapper fails to build, stop and correct that error before running or trusting Structural Data Quality. Run these commands serially for each refresh; Tuva Core publishes current-state results, while Tuva Data Quality Intelligence maintains refresh history and remediation workflow.

Tuva runs four structural checks on every enabled Input Layer Model:

Columns exist. Verifies that the Warehouse Table or View contains every column required by the Tuva Input Layer contract.
Data types are correct. Verifies that each required column uses a data type compatible with the type declared by Tuva.
Table is populated. Verifies that the Warehouse Table or View contains at least one record for the data_source being evaluated.
Primary key is correct. Verifies that the columns declared as the Input Layer Model's primary key are non-null and unique among the records for each data_source.

The data_quality.structural results table contains one row for each enabled Input Layer Model and each in-scope data_source. Tuva discovers the in-scope sources separately for the claims and clinical domains so that a claims source is not incorrectly expected to populate clinical Models, or vice versa. When provider attribution is enabled, it participates in the claims domain. If no non-null source can be discovered anywhere in an enabled domain, Tuva produces one fallback row with data_source = null for each Model in that domain. Each result is pass, fail, or not evaluated. Every fail must be fixed. not evaluated means a failed prerequisite prevented that check from running; it is not readiness. Fix the prerequisite, rerun, and continue only when all four results pass.

The following query returns the four structural results:

select
    data_source,
    input_table_name,
    columns_exist,
    data_types_correct,
    table_populated,
    primary_key_correct,
    row_count
from data_quality.structural
order by
    data_source,
    input_table_name;

An example result might look like this:

Example structural Data Quality results for two data sources
data_source	input_table_name	columns_exist	data_types_correct	table_populated	primary_key_correct
Medicare MSSP	eligibility	pass	pass	pass	pass
Medicare MSSP	medical_claim	pass	pass	pass	pass
Medicare MSSP	pharmacy_claim	pass	pass	fail	not evaluated
UHC Medicaid	eligibility	pass	pass	pass	pass
UHC Medicaid	medical_claim	pass	pass	pass	fail
UHC Medicaid	pharmacy_claim	pass	pass	pass	pass

Let's unpack what these example results mean. The Warehouse Table or View for the pharmacy_claim Input Layer Model contains no records with data_source = 'Medicare MSSP', so the population check fails and the primary-key check is not evaluated. The records with data_source = 'UHC Medicaid' in the Warehouse Table or View for the medical_claim Input Layer Model contain a null or duplicate primary key. The remaining Input Layer Model and data_source combinations satisfy all four structural requirements.

Public Structural Data Quality Relations

Structural Data Quality exposes exactly five stable public relations:

Public relation	How to use it
`data_quality.structural`	Start here. It is the readiness matrix with one row per enabled Input Layer Model and in-scope `data_source`.
`data_quality.structural_test_results`	Use the normalized one-row-per-check representation for integrations and other normalized Data Quality consumers.
`data_quality.structural_missing_columns`	Identify columns required by the Input Layer contract that are absent from a Warehouse Table or View.
`data_quality.structural_data_type_mismatches`	Identify present columns whose warehouse data types are incompatible with the Input Layer contract.
`data_quality.structural_primary_key_failure_counts`	Determine whether a source-scoped primary-key failure was caused by null values, duplicate values, or both.

The readiness matrix query above should be the first query a user runs. To inspect the same results in the normalized public contract, run:

select
    data_source,
    input_table_name,
    test_name,
    display_name,
    description,
    grain,
    test_type,
    check_category,
    severity,
    total_row_count,
    tested_count,
    failed_count,
    passed_count,
    not_evaluated_count
from data_quality.structural_test_results
order by
    data_source,
    input_table_name,
    test_name;

When columns_exist = 'fail', query the missing-column detail:

select
    input_table_name,
    column_name,
    expected_data_type
from data_quality.structural_missing_columns
order by
    input_table_name,
    column_name;

When data_types_correct = 'fail', or when a known type mismatch accompanies a missing column, query the data-type detail:

select
    input_table_name,
    column_name,
    expected_data_type,
    actual_data_type
from data_quality.structural_data_type_mismatches
order by
    input_table_name,
    column_name;

When primary_key_correct = 'fail', query the source-scoped primary-key failure counts:

select
    data_source,
    input_table_name,
    failure_type,
    primary_key_columns,
    failed_record_count
from data_quality.structural_primary_key_failure_counts
order by
    data_source,
    input_table_name,
    failure_type,
    primary_key_columns;

These examples use the default data_quality schema. If tuva_schema_prefix is configured, replace data_quality with <tuva_schema_prefix>_data_quality and include the database or catalog qualifier required by your warehouse.

The five relations above are the stable Structural Data Quality integration contract. Other materialized data_quality.structural_* relations are internal pipeline helpers. Users and downstream integrations should not depend on their names, schemas, or row grains because Tuva may change them between releases.

Use the readiness matrix and failure details to correct the connector, rebuild the Input Layer Wrappers, and rerun tag:dq_structural. Continue to Logical Data Quality and the rest of Tuva Core only after all four readiness results are pass for every row.

How Structural Data Quality Works

data_quality.structural is produced by a dbt pipeline that compares Tuva's Input Layer contract with the Warehouse Tables or Views produced by the Input Layer Wrappers. The pipeline combines project configuration, Tuva Core's Input Layer YAML, warehouse metadata, and targeted scans of records in those objects.

Before explaining the pipeline, we define three terms that distinguish the contract, code, and warehouse output involved in the comparison:

An Input Layer Model is a dbt model in the connector project, such as medical_claim. Its SQL maps source-system data to Tuva's Input Layer contract.
An Input Layer Wrapper is the corresponding package-owned Tuva Core model, such as input_layer__medical_claim. It references the Input Layer Model and carries Tuva's contract metadata.
A Warehouse Table or View is the physical object produced when dbt builds an Input Layer Wrapper. Structural Data Quality inspects this object.

We use expected to describe structural requirements derived from Tuva's Input Layer contract. We use actual to describe metadata or records read from the Warehouse Tables or Views. The same distinction appears in internal names such as structural_expected_columns and structural_actual_columns.

The Tuva-specific resources connect as follows:

dq_enabled_input_layer_model_domains() groups the enabled Input Layer Wrappers into claims and clinical domains, and dq_enabled_input_layer_model_names() flattens those groups into the compile-time Wrapper list. Neither macro creates a warehouse table.
The expected-metadata branch uses data_quality__structural_expected_columns to create data_quality.structural_expected_columns, with one row per expected column in each enabled Input Layer Wrapper.
The actual-metadata branch uses data_quality__structural_actual_columns to create data_quality.structural_actual_columns, with one source-neutral row per actual column, including the adapter-reported physical column name, warehouse data type, and portable type family.
The population branch uses data_quality__structural_source_populations to create data_quality.structural_source_populations. It scans each enabled Warehouse Table or View once and counts records by its actual data_source value.
data_quality__structural_data_sources creates data_quality.structural_data_sources, with one row for each distinct, non-null data_source found anywhere among the enabled Models in a domain. Claims and clinical sources are listed separately. If no Model in a domain contains a non-null data_source, the table contains one placeholder row with data_source = null for that domain.
data_quality__structural_evaluation_scope creates data_quality.structural_evaluation_scope, with every Input Layer Model and data_source combination Tuva must evaluate and its row count.
The primary-key branch uses data_quality__structural_primary_key_tests to create data_quality.structural_primary_key_tests, with one row per null or duplicate-key subcheck and in-scope data_source. It derives all key metrics from one grouped record scan per enabled Warehouse Table or View.
data_quality__structural_column_details compares every expected column with its actual column in the source-neutral data_quality.structural_column_details helper table.
data_quality__structural combines the column comparison, evaluation scope, and primary-key results in data_quality.structural, with one row per enabled Input Layer Model and data_source.
data_quality__structural_test_results publishes the same four results at one row per check so they can be consumed alongside normalized Data Quality results.
Three failure-only detail models publish the missing columns, incompatible data types, and primary-key failure counts needed to correct a failed structural result.

The tuva_schema_prefix configuration applies to every materialized Data Quality table in this list.

1. How Tuva Determines Which Input Layer Models to Evaluate

Structural Data Quality begins by determining which Input Layer Models it needs to evaluate.

The connector project's dbt configuration includes variables that tell Tuva which types of data the connector maps. These include claims_enabled, clinical_enabled, and provider_attribution_enabled. For example, claims_enabled: true tells Tuva that the connector should provide the eligibility, medical_claim, and pharmacy_claim Input Layer Models.

These dbt variables do more than identify the Input Layer Models a connector provides. They also enable the Tuva Core transformations that consume the mapped data. For example, claims_enabled: true enables claims-specific models in the Normalized Layer, Claims Preprocessing, and Core Data Model. When data_quality_enabled: true is also set, the two variables together enable claims-specific Data Quality checks.

The dq_enabled_input_layer_model_domains() macro translates these settings into groups of Input Layer Wrappers. It places the eleven clinical Wrappers in the clinical domain and the three claims Wrappers in the claims domain. When provider_attribution_enabled: true, it also places input_layer__provider_attribution in the claims domain. The dq_enabled_input_layer_model_names() macro then flattens the enabled domain groups into the exact Wrapper list Structural Data Quality evaluates.

These groups and the flattened list are defined in Tuva Core code; they are not stored as standalone tables in the warehouse. While dbt compiles the Data Quality models, the macros read the dbt variables and return temporary Jinja objects in memory. The structural models use those objects to generate their SQL and dbt dependencies. Derived helper tables such as structural_expected_columns later represent the expected structural metadata in the warehouse.

For the claims example, the macro returns input_layer__eligibility, input_layer__medical_claim, and input_layer__pharmacy_claim. Each Input Layer Wrapper references the corresponding Input Layer Model. For example, input_layer__medical_claim selects from ref('medical_claim').

Tuva creates this enabled wrapper list before it inspects the warehouse. The references from the structural helper models to those wrappers enforce the dbt build order. If an Input Layer Model is missing, dbt cannot resolve the wrapper. If an Input Layer Wrapper fails to build, dbt skips its downstream structural models. Both conditions are dbt build failures that must be fixed before a structural matrix can describe the current Input Layer.

The expected-column, actual-column, source-population, and primary-key models register dependencies on every Input Layer Wrapper in the enabled list. These dependencies establish the dbt build order when the Wrappers and Structural Data Quality are selected together. If a user selects only Structural Data Quality, the Warehouse Tables or Views produced by the Wrappers must already exist from an earlier run. If a required object is unavailable, Tuva raises an error instead of producing incomplete structural results.

The connector must provide Input Layer Models

The connector must define every enabled Input Layer Model. For example, input_layer__medical_claim calls ref('medical_claim'), so the connector must provide an enabled Input Layer Model named medical_claim. A preexisting warehouse object is not sufficient by itself, although the Input Layer Model can select from that object and map it to Tuva's contract.

If the medical_claim Input Layer Model is missing or disabled, dbt cannot resolve the reference and stops before Structural Data Quality can run. This is a dbt build error, so it does not appear as a result in data_quality.structural.

The enabled Input Layer Wrapper list defines the evaluation scope. In Section 2, we describe how Tuva builds the expected structural metadata for that scope.

2. How Tuva Builds Expected Structural Metadata from the Input Layer Contract

After Tuva determines which Input Layer Wrappers are in scope, it reads the required columns, portable data types, and primary-key columns for each corresponding Input Layer Model. Tuva Core defines this contract in the Input Layer YAML files under models/input_layer. Connector authors map their source-system data to the contract; they do not redefine its structural requirements.

Every column declared in these files is required. config.meta.data_type defines the portable data type Tuva expects, and config.meta.is_primary_key: true identifies a column as part of the Input Layer Model's composite primary key. Tuva evaluates that key independently within each data_source.

For example, the contract for the medical_claim Input Layer Model is declared on the input_layer__medical_claim Input Layer Wrapper in models/input_layer/input_layer__medical_claim.yml. The following excerpt contains three primary-key columns and one ordinary required column:

models:
  - name: input_layer__medical_claim
    columns:
      - name: claim_id
        config:
          meta:
            data_type: varchar
            is_primary_key: true
      - name: claim_line_number
        config:
          meta:
            data_type: integer
            is_primary_key: true
      - name: bill_type_code
        config:
          meta:
            data_type: varchar
      - name: data_source
        config:
          meta:
            data_type: varchar
            is_primary_key: true

This excerpt says that all four columns must exist with compatible data types. It also says that claim_id, claim_line_number, and data_source together identify one medical-claim line within one data_source. The primary-key declaration is a structural contract requirement; it does not create a primary-key constraint in the warehouse.

During compilation, the dq_expected_input_layer_models() macro resolves each name returned by the dq_enabled_input_layer_model_names() macro to the corresponding Input Layer Wrapper in dbt's parsed project metadata. The dq_expected_columns() macro then walks the column metadata for each Wrapper, lowercases each column name, and extracts meta.data_type, meta.is_primary_key, and declaration order.

The data_quality__structural_expected_columns dbt model converts that metadata to SQL and creates the data_quality.structural_expected_columns helper table. It contains one row per expected column, including:

the table_name, model_name, and column_name fields, which identify the Input Layer Model (medical_claim), Input Layer Wrapper (input_layer__medical_claim), and column (claim_id);
the expected data type and its warehouse-independent type family;
whether the column belongs to the primary key; and
the column's declaration order, which Tuva retains as descriptive metadata but does not evaluate as a structural requirement.

For the claims example, data_quality__structural_expected_columns produces expected-column rows for every declared column in eligibility, medical_claim, and pharmacy_claim. These rows do not yet contain individual data_source values because the same model, column, type, and key contract applies to every data_source.

Before generating the expected-column SQL, Tuva validates the enabled contract metadata. Compilation fails if an Input Layer Wrapper has duplicate column names after case normalization, a column is missing meta.data_type, an expected type cannot be mapped to a supported portable family, no column is marked as part of the primary key, or the contract does not declare exactly one data_source column as part of that key. These errors indicate an invalid Tuva Core contract, not a connector data-quality result. They stop the run because Tuva cannot evaluate the mapped data safely without a valid expected contract.

3. How Tuva Captures Actual Structural Results

Tuva has now defined the expected structural metadata. It next collects the actual structural evidence from the Warehouse Table or View produced by each enabled Input Layer Wrapper. At this stage, "actual structural results" means the metadata and record-level measurements Tuva collects before it assigns the final pass, fail, or not evaluated results.

Tuva collects four types of actual evidence:

Actual columns and data types: the columns and warehouse data types that the Wrapper built, along with the portable type family Tuva assigns to each type.
Actual source populations: the distinct data_source values in each Warehouse Table or View and the number of records for each Input Layer Model and actual data_source combination.
The structural evaluation scope: every Input Layer Model and data_source combination Tuva must evaluate, including combinations with zero records.
Primary-key subcheck results: the number of records containing a null in each declared primary-key column and the number of records belonging to a non-unique composite-key group, calculated separately for each Input Layer Model and data_source combination.

Five internal dbt models create this evidence. The actual-column model reads warehouse metadata. The source-population model scans mapped records and provides the counts used to build the domain roster and evaluation scope. The primary-key model performs a second, grouped scan of each Model whose required key inputs are usable. Keeping these branches separate avoids copying one schema description for every source and avoids repeatedly scanning the same claims-scale table for individual counts.

Actual Columns and Data Types

For each enabled Input Layer Wrapper, the data_quality__structural_actual_columns model first uses dq_required_actual_relation() to locate the Warehouse Table or View at the Wrapper's configured database, schema, and alias. If that object does not exist, Tuva raises an error because no current object is available to inspect.

The dq_actual_columns() macro then reads the column metadata reported by the warehouse adapter. For each physical column, Tuva stores both a lowercase name for case-insensitive contract matching and the exact adapter-reported identifier for SQL that reads the column later. This distinction matters on warehouses that preserve case or require quoted identifiers. If two physical identifiers normalize to the same lowercase name, such as claim_id and CLAIM_ID, Tuva raises an ambiguity error rather than choosing one.

Warehouse data types use different names for the same underlying concept. The dq_type_family() macro dispatches to warehouse-specific logic and normalizes those names into portable families such as string, integer, numeric, boolean, date, and timestamp. Tuva applies this macro to both the expected data types from the Input Layer contract and the actual data types read from the Warehouse Tables or Views. For example, an expected varchar and an actual STRING both map to the string family. The default implementation covers common aliases, while BigQuery and Fabric provide additional overrides. A numeric or decimal type maps to integer only when its reported type explicitly has a scale of zero, such as NUMBER(38,0); a bare NUMBER remains numeric.

The dq_type_families_match_sql() macro defines how Tuva compares the expected and actual families in the next stage of the pipeline. It accepts equal families and also accepts an actual integer for an expected boolean or numeric value. Other cross-family pairs fail; for example, an expected integer does not match an actual date. Types outside Tuva's supported portable families remain unsupported; for example, BigQuery BYTES is treated as binary rather than as a string.

This is the complete Structural check for Input Layer date types. A column declared as date must resolve to the warehouse's native date family; a string column containing values such as 2026-08-24 fails Structural validation even though the text uses the recommended serialized format. Value range is handled separately by Logical Data Quality after the type contract passes.

The resulting data_quality.structural_actual_columns helper has one source-neutral row for each actual column in each enabled Input Layer Model. It contains input_layer_domain, table_name, model_name, the normalized column_name, actual_column_name, actual_data_type, and actual_type_family. It does not contain data_source or row_count, and it does not scan mapped records. Every source in one Warehouse Table or View shares the same physical schema.

Source Populations and the Evaluation Scope

The data_quality__structural_source_populations model reads the mapped records separately from the column-metadata branch. For each enabled Input Layer Model, it generates one grouped query against the Warehouse Table or View. That query uses the exact adapter-reported data_source identifier, counts the records for each actual data_source value, and stores each count as a 64-bit integer. The 64-bit count prevents ordinary claims-scale populations from overflowing a 32-bit integer.

The resulting data_quality.structural_source_populations helper contains one row per enabled Input Layer Model and actual data_source, with input_layer_domain, table_name, model_name, data_source, an internal null-safe data_source_key, and row_count. An empty Warehouse Table or View still produces one fallback row with data_source = null and row_count = 0.

If the actual data_source column is missing or its warehouse type does not map to the string family, Tuva cannot safely derive named source populations. The helper represents all records as one null-source population, and values from a non-string data_source do not enter the shared source roster. The final matrix reports population and primary-key results as not evaluated until the connector supplies a usable data_source column with the expected type. When the column exists with a non-string type, data_types_correct fails.

Tuva must know which Input Layer Model and data_source combinations to evaluate before it can identify a missing source population. Grouping the records in one Warehouse Table or View is not sufficient because SQL cannot return a zero-count group for a data_source value that is completely absent from that object.

Tuva solves this problem by building a shared source roster separately for each enabled Input Layer domain:

The claims domain contains eligibility, medical_claim, and pharmacy_claim. When provider_attribution_enabled: true, it also contains provider_attribution.
The clinical domain contains appointment, condition, encounter, immunization, lab_result, location, medication, observation, patient, practitioner, and procedure.

The data_quality__structural_data_sources model reads the actual source populations from structural_source_populations. For each enabled domain, it unions the distinct, non-null data_source values found across all of the domain's Warehouse Tables or Views. Claims sources therefore form only claims expectations, and clinical sources form only clinical expectations. If one source legitimately supplies both kinds of data, it appears independently in both rosters.

The data_quality__structural_evaluation_scope model then combines each domain roster with every enabled Input Layer Model in that domain. It left joins the actual row counts and uses 0 when a particular Model contains no records for a roster source. This produces one explicit row for every Model and source combination Tuva must evaluate.

Consider a claims connector with these mapped record counts:

Input Layer Model	Medicare MSSP records	UHC Medicaid records
`eligibility`	100,000	80,000
`medical_claim`	1,200,000	900,000
`pharmacy_claim`	0	75,000

Medicare MSSP does not appear in pharmacy_claim, but it appears elsewhere in the claims domain. The shared claims roster therefore contains Medicare MSSP and UHC Medicaid. Combining those two sources with the three claims Models creates all six required combinations, including Medicare MSSP and pharmacy_claim.

For that missing combination, the evaluation scope records row_count = 0. The final structural row reports table_populated = 'fail' and primary_key_correct = 'not evaluated' because there are no records whose keys can be checked. Column presence and data types can still be evaluated because they describe the shared pharmacy_claim Warehouse Table or View rather than one source's records. The generated row does not invent pharmacy records; it makes their absence visible.

The shared domain roster contains only non-null source names. The evaluation scope separately preserves a Model-specific null-source row when that Model actually contains one or more records with data_source = null. This keeps the invalid records visible for primary-key evaluation without creating null-source expectations for every other Model in the domain. A zero-count null fallback from an otherwise empty Model is not added as an extra row when the domain has a named source roster.

If no non-null data_source appears anywhere in an enabled domain, structural_data_sources emits one null-source fallback for that domain and structural_evaluation_scope creates one null-source row for each enabled Model in it. These rows preserve any actual null-source population and show which Models have zero records, but Tuva cannot infer the name of a source that is absent from every Model in the domain. Detecting that refresh-level condition requires an external expectation about which source refreshes should have arrived. It belongs to Data Quality Intelligence refresh monitoring rather than Tuva Core's current-snapshot discovery.

Primary-Key Subcheck Results

The data_quality__structural_primary_key_tests model uses the primary-key declarations from Tuva's Input Layer YAML to generate two types of subchecks for each enabled Input Layer Model and data_source:

Null-value subchecks. The model creates one subcheck for every declared primary-key column and counts the records in which that column is null.
Duplicate-value subcheck. The model creates one subcheck for the complete composite key and counts every record that belongs to a key group containing more than one record.

The dq_expected_pk_columns() macro reads the declared primary-key columns from the Input Layer Wrapper metadata. The model independently locates the Warehouse Table or View and its exact physical column identifiers, then confirms that every declared key column exists and maps to a supported portable type family. For data_source specifically, usable means the actual type maps to the string family. Another supported family is still incompatible with the source- partition contract and blocks the primary-key scan. If any key column is missing or unsupported, Tuva skips the record scan for that Input Layer Model and emits blocked subchecks with test_result = null. The final matrix reports primary_key_correct = 'not evaluated' until the key input is corrected.

When all required key inputs are usable, Tuva scans the Warehouse Table or View once. It groups the records by a collision-safe internal source key and the complete non-source portion of the composite primary key. From that grouped result, it derives every key metric for every source: one null-record count for each declared key column and one duplicate-record count for the complete key. This single grouped scan replaces a separate full-table scan for every key column and source.

For medical_claim, the declared composite key is claim_id, claim_line_number, and data_source. Because Tuva already partitions the subchecks by data_source, the duplicate calculation compares claim_id and claim_line_number within each data_source. The same claim and line can therefore appear once in Medicare MSSP and once in UHC Medicaid without being treated as a duplicate. If the same claim-line combination appears twice within UHC Medicaid, the duplicate subcheck reports two failed records. If one claim-line key appears twice and another appears three times, the subcheck reports five failed records.

data_quality.structural_primary_key_tests contains one row per Input Layer Model, in-scope data_source, and primary-key subcheck. Each row identifies the key column or composite key, the subcheck type, and test_result. When the subcheck can run, test_result = 0 means no violation was found and a positive value is the number of records that failed that subcheck. The counts are stored as 64-bit integers. An in-scope source with row_count = 0 can have zero-valued internal subchecks after the evaluation scope is joined to the grouped metrics, but the final matrix reports primary_key_correct = 'not evaluated' because there are no records whose keys can be assessed.

With valid physical key inputs, Structural Data Quality therefore performs two record scans per enabled Warehouse Table or View: one grouped source-population scan and one grouped primary-key scan. The number of sources and primary-key columns does not add more full-table scans. The remaining structural models read the much smaller metadata, scope, and metric helper tables.

Special Cases in the Mapped Data

The actual-evidence models handle four edge conditions explicitly:

If a required Warehouse Table or View is unavailable, Tuva raises an error directing the user to build the enabled Input Layer Wrappers before rerunning Structural Data Quality. Tuva does not produce partial structural results.
If one Warehouse Table or View is empty but another Model in the same domain supplies non-null source values, the evaluation scope creates one zero-count row for every source in the domain roster. Population fails and the primary-key check is not evaluated for each row. If the entire domain has no non-null source, Tuva creates one null-source fallback row per enabled Model.
If the Warehouse Table or View does not contain a usable data_source column, Tuva represents its records in a Model-specific null-source population. When other Models provide the domain roster, Tuva also creates zero-count rows for those non-null sources. A missing column makes columns_exist fail and leaves data_types_correct not evaluated; a present but incompatible type makes data_types_correct fail. In either case, table_populated and primary_key_correct are not evaluated because Tuva cannot safely evaluate records by source.
If the data_source column exists but some records contain a null value, Tuva preserves a Model-specific data_source = null row for those records in addition to the shared non-null source rows. The population check can still pass for that group, but the primary-key check fails because data_source is part of every declared primary key.

Tuva uses separate internal join-key encodings for null and non-null data_source values. This prevents a legitimate source name from being confused with SQL null. The public data_source value remains unchanged, and null values remain null in the results.

4. How Tuva Compares Expected and Actual Columns

Tuva now has the expected structural metadata described in Section 2 and the actual structural evidence collected in Section 3. It next aligns the expected and actual column metadata to identify where a Warehouse Table or View differs from the Input Layer contract. Source populations, row counts, and primary-key subchecks remain in their separate helper tables until the final aggregation.

The data_quality__structural_column_details dbt model performs this comparison and creates the source-neutral data_quality.structural_column_details helper table. Starting with every expected column, it left joins the actual column with the same input_layer_domain, table_name, model_name, and normalized column_name. Capitalization alone does not cause a failure, but the actual adapter-reported identifier remains available as actual_column_name.

For every expected column, the helper records:

whether the expected column was found;
the expected and actual data types;
whether those types are compatible;
whether the Input Layer contract declares the column as part of the primary key, which allows the final aggregation to identify a missing key column; and
the column's declaration order as descriptive metadata.

The helper does not contain data_source or row_count. A Warehouse Table or View has one schema shared by every source population, so repeating the same column comparison for every source would add rows without adding evidence.

The comparison begins with the expected-column list. An extra column in the Warehouse Table or View that is not part of Tuva's contract is ignored; it does not cause a structural failure. Ambiguous actual identifiers have already stopped the run in Section 3, so one actual column can match at most one normalized expected name.

If bill_type_code is declared in the YAML but absent from the Warehouse Table or View, its comparison row records a missing column and has no actual type. If claim_line_number exists as a date rather than an integer, the row records both the expected and actual types and marks them incompatible.

Structural Data Quality does not compare column descriptions or column order. It also does not separately enforce string length, numeric precision, or nullability for non-key columns. Null values are evaluated only where they are part of the primary-key check.

These column-level rows provide the evidence for columns_exist and data_types_correct. Together with the row counts and primary-key subcheck results captured in Section 3, these rows provide all of the evidence Tuva needs to create the final structural results.

5. How Tuva Creates `data_quality.structural`

At this point, Tuva has three types of evidence for the final aggregation:

data_quality.structural_column_details identifies missing required columns and incompatible data types for each Input Layer Model. This evidence is source-neutral because a Model has one physical schema.
data_quality.structural_evaluation_scope contains every in-scope Input Layer Model and data_source combination with its 64-bit row_count.
data_quality.structural_primary_key_tests contains the null and duplicate-key subcheck results for each Input Layer Model and data_source.

The data_quality__structural dbt model aggregates these inputs and creates the data_quality.structural results table. This table is the structural readiness matrix: one row answers whether the Warehouse Table or View for one enabled Input Layer Model is structurally ready for one data_source. This model does not scan the Warehouse Tables or Views again; it calculates the final results from the helper tables created in the preceding stages.

First, the model groups the source-neutral column-comparison rows by Input Layer Model and counts missing required columns, incompatible types, missing key columns, and data_source contract problems. It separately groups the primary-key subcheck rows by Input Layer Model and data_source. It then starts with the complete evaluation scope and joins both summaries to each Model and source row. The evaluation scope supplies row_count; no column helper carries or duplicates that measurement.

The aggregation applies dependency rules so that a missing prerequisite does not become an additional failure. Tuva reports the dependent check as not evaluated instead.

The public results table is data_quality.structural, or <tuva_schema_prefix>_data_quality.structural when a schema prefix is configured. It contains one row for each enabled Input Layer Model and each in-scope data_source, and it exposes four structural result columns. The input_table_name field identifies the Input Layer Model, such as medical_claim; row_count is supporting evidence and is not a fifth check.

Result column	How it is calculated
`columns_exist`	No columns required by the Input Layer Model contract are missing from the built Warehouse Table or View
`data_types_correct`	When all required columns are present, none uses an incompatible type
`table_populated`	The Warehouse Table or View contains at least one record for this `data_source`
`primary_key_correct`	For a populated Warehouse Table or View with every key column present, records for this `data_source` have no null or duplicate key values

Each result is pass, fail, or not evaluated. pass means the check ran and the requirement was satisfied. fail means the check ran and the requirement was not satisfied. not evaluated means a prerequisite prevented the check from running. Every fail must be fixed. A not evaluated result is also not ready: fix its failed prerequisite, rerun Structural Data Quality, and continue only when all four results pass.

For example, suppose the Warehouse Table or View for the medical_claim Input Layer Model contains all required columns with compatible types and has records for UHC Medicaid, but those records contain one duplicate claim-line key. Its matrix row contains three pass results and primary_key_correct = 'fail'. If the Warehouse Table or View for the pharmacy_claim Input Layer Model contains no Medicare MSSP records while that source appears in another claims Model, its Medicare MSSP row can have passing column and type results while table_populated = 'fail' and primary_key_correct = 'not evaluated'. Neither row is structurally ready.

These dependency rules apply after every enabled Input Layer Wrapper has built successfully. An unavailable Warehouse Table or View produced by an Input Layer Wrapper is a dbt error, not a structural result.

Check	Prerequisite	When the result is `not evaluated`
Columns exist	The enabled Input Layer Wrapper built successfully	Never
Data types correct	All expected columns exist	Any expected column is missing
Table populated	The required `data_source` column exists with a compatible type and its population can be measured	`data_source` is missing or incompatible, or a population count cannot be produced
Primary key correct	The source population is nonempty and every declared key column exists with a supported type	The population is empty or unavailable, `data_source` is missing or incompatible, another key column is missing or unsupported, or a key subcheck is blocked

These rules prevent one root problem from appearing as several misleading failures. For example, a missing required primary-key column makes columns_exist fail while data-type and primary-key evaluation wait until the column is added.

The matrix does not add a generic status column or one row per assertion. It also does not need a separate severity column. The normalized data_quality.structural_test_results table provides one row for each of the four checks and assigns S1 to a failed result (stored as the integer 1). There is no informational structural failure: a failure in columns_exist, data_types_correct, table_populated, or primary_key_correct means the row is not structurally ready and must be fixed.

6. How Tuva Publishes Evidence for Correcting Failures

The data_quality.structural readiness matrix identifies which structural requirement failed. Tuva publishes three purpose-built, failure-only tables so the user can identify the specific missing columns, incompatible data types, or kind of primary-key problem that caused that result:

Public table	Row grain	Fields
`data_quality.structural_missing_columns`	One Input Layer Model and missing expected column	`input_table_name`, `column_name`, `expected_data_type`
`data_quality.structural_data_type_mismatches`	One Input Layer Model and column with an incompatible type	`input_table_name`, `column_name`, `expected_data_type`, `actual_data_type`
`data_quality.structural_primary_key_failure_counts`	One source, Input Layer Model, and primary-key failure subcheck	`data_source`, `input_table_name`, `failure_type`, `primary_key_columns`, `failed_record_count`

These tables contain only conclusive failure evidence. A passing check does not produce a detail row. Missing-column and primary-key checks that are not evaluated do not produce detail because Tuva lacks the evidence needed to calculate it. A known type mismatch on a present column remains visible even when data_types_correct is not evaluated because a different required column is missing. This lets the user correct both problems before rerunning. As with the readiness matrix, tuva_schema_prefix changes the schema name to <tuva_schema_prefix>_data_quality.

Missing Columns

data_quality.structural_missing_columns contains one row for every column that the Input Layer contract requires but the Warehouse Table or View does not contain. For example, if medical_claim does not contain bill_type_code, the table contains this row:

input_table_name	column_name	expected_data_type
`medical_claim`	`bill_type_code`	`varchar`

Missing-column evidence does not include data_source. Every source in an Input Layer Model uses the same Warehouse Table or View and therefore the same schema. A missing column causes columns_exist = 'fail' for every in-scope source in data_quality.structural, but Tuva stores the underlying schema failure only once.

The public table is a failure-only projection of the expected-versus-actual column comparison described in Section 4. It does not repeat the internal source-neutral rows from structural_column_details.

Data-Type Mismatches

data_quality.structural_data_type_mismatches contains one row for every required column that exists but has a warehouse type incompatible with the expected type. For example:

input_table_name	column_name	expected_data_type	actual_data_type
`medical_claim`	`claim_line_number`	`integer`	`date`

This table also omits data_source because a Warehouse Table or View has one schema shared by all of its source populations. If a column is missing rather than incorrectly typed, it appears only in data_quality.structural_missing_columns; Tuva does not also report it as a type mismatch. However, a mismatch on another present column remains in this table even when the missing column prevents Tuva from assigning a final data_types_correct result.

Primary-Key Failure Counts

data_quality.structural_primary_key_failure_counts explains whether primary_key_correct = 'fail' was caused by null key values, non-unique keys, or both. Primary-key evidence is source-specific because Tuva evaluates records and uniqueness independently within each data_source.

The table uses two failure_type values with the following grains and count semantics:

`failure_type`	Row grain	`primary_key_columns`	`failed_record_count`
`null_value`	One `data_source`, Input Layer Model, and primary-key column containing nulls	The single key column evaluated	Number of records in which that column is null
`duplicate_value`	One `data_source` and Input Layer Model	The complete composite primary key in contract order	Number of records belonging to groups in which the complete key occurs more than once

primary_key_columns is a text field. A null_value row contains one column name. A duplicate_value row contains the complete key in Input Layer contract order, separated by a comma and a space; for example, claim_id, claim_line_number, data_source.

For example, if one medical_claim key occurs twice and another occurs three times within UHC Medicaid, its duplicate_value row contains failed_record_count = 5. Tuva counts all five non-unique records, not only the three records beyond the first occurrence of each key.

Null counts are calculated independently for each key column. A record with a null claim_id and a null claim_line_number contributes once to each column's null_value row. The failure-count rows therefore must not be added together and interpreted as a count of distinct failing records. A record can also contribute to both a null-value and duplicate-value result.

If an expected primary-key column is missing, the missing column appears in data_quality.structural_missing_columns and the primary-key result is not evaluated; Tuva does not publish a primary-key failure count. Similarly, an Input Layer Model and source combination with no records has row_count = 0 and table_populated = 'fail' in data_quality.structural, while its primary-key result is not evaluated and has no failure-count row.

Investigating Primary-Key Records

Tuva publishes counts rather than storing individual key values or complete healthcare records in the Data Quality schema. After identifying the failure type, query the corresponding Warehouse Table or View to inspect the records. The examples below use the default Warehouse Table or View produced by the input_layer__medical_claim Wrapper. If tuva_schema_prefix is configured, replace input_layer with <tuva_schema_prefix>_input_layer.

For example, this query returns medical-claim records with a null claim_line_number for one source:

select *
from input_layer.input_layer__medical_claim
where data_source = 'UHC Medicaid'
  and claim_line_number is null;

This query returns every medical-claim record belonging to a duplicate composite-key group:

with duplicate_keys as (
    select
        claim_id,
        claim_line_number,
        data_source
    from input_layer.input_layer__medical_claim
    where data_source = 'UHC Medicaid'
    group by
        claim_id,
        claim_line_number,
        data_source
    having count(*) > 1
)

select source_records.*
from input_layer.input_layer__medical_claim as source_records
inner join duplicate_keys
    on (
        source_records.claim_id = duplicate_keys.claim_id
        or (source_records.claim_id is null and duplicate_keys.claim_id is null)
    )
   and (
        source_records.claim_line_number = duplicate_keys.claim_line_number
        or (
            source_records.claim_line_number is null
            and duplicate_keys.claim_line_number is null
        )
    )
   and (
        source_records.data_source = duplicate_keys.data_source
        or (
            source_records.data_source is null
            and duplicate_keys.data_source is null
        )
    )
order by
    source_records.claim_id,
    source_records.claim_line_number;

Adapt the source filter and the key columns to the Input Layer Model reported in structural_primary_key_failure_counts. After finding the cause, correct the connector, rebuild the Input Layer Wrappers, and rerun Structural Data Quality.

How to Use Structural Data Quality​

Build the Input Layer and Structural Data Quality in Three Steps​

Public Structural Data Quality Relations​

How Structural Data Quality Works​

1. How Tuva Determines Which Input Layer Models to Evaluate​

2. How Tuva Builds Expected Structural Metadata from the Input Layer Contract​

3. How Tuva Captures Actual Structural Results​

Actual Columns and Data Types​

Source Populations and the Evaluation Scope​

Primary-Key Subcheck Results​

Special Cases in the Mapped Data​

4. How Tuva Compares Expected and Actual Columns​

5. How Tuva Creates data_quality.structural​

6. How Tuva Publishes Evidence for Correcting Failures​

Missing Columns​

Data-Type Mismatches​

Primary-Key Failure Counts​

Investigating Primary-Key Records​