Data Quality Tutorial

This tutorial uses Data Quality while mapping one claims source into the Tuva Input Layer. By the end, you will have:

• confirmed that each enabled Input Layer table is structurally ready;
• investigated logical failures at their native grain;
• corrected connector logic and rerun the checks; and
• made a source-specific decision about whether to continue building Tuva.

The examples call the source example_claims and assume Tuva writes Data Quality models to a schema named data_quality. If you set tuva_schema_prefix, replace that schema with <tuva_schema_prefix>_data_quality.

Prerequisites

Before starting:

• load the raw source data into your warehouse;
• create a connector project and confirm that dbt debug and dbt deps succeed;
• map the connector's final models to the Tuva Input Layer contract; and
• give every row from this source the same non-null data_source value, example_claims.

Claims enable three Input Layer tables: medical_claim, pharmacy_claim, and eligibility. The connector must define every enabled Input Layer Model so dbt can resolve and build its Tuva Core Input Layer Wrapper. Every resulting Warehouse Table or View must also contain records to pass Structural Data Quality.

Recommended Workflow

The workflow is intentionally iterative:

Structural checks come first because a logical result is not trustworthy when an enabled table is empty or its required fields, types, or native grain are wrong.

1. Configure Data Quality

Set the domain and Data Quality variables in the connector's dbt_project.yml:

vars:
  claims_enabled: true
  clinical_enabled: false
  provider_attribution_enabled: false
  data_quality_enabled: true
  enable_data_quality_failure_keys: false

Use native YAML booleans, without quotes.

Set enable_data_quality_failure_keys: true only when you need the optional key-only Logical Data Quality failure table. Leave it disabled for routine runs because the table can be large.

2. Map the source

A connector should do more than rename raw fields. It establishes the exact Input Layer grain and semantics that Data Quality evaluates. For a medical claim mapping, confirm that it:

• casts claim_line_number to an integer and dates and amounts to their declared types;
• creates a stable claim_id and positive line number;
• derives the accepted claim_type values;
• maps institutional fields such as bill_type_code and revenue_center_code only where appropriate;
• maps professional place_of_service_code values;
• keeps claim-header fields consistent across every line of a claim; and
• emits the composite primary key claim_id, claim_line_number, and data_source.

A simplified final select might end like this:

select
    cast(raw_claim_number as varchar) as claim_id,
    cast(raw_line_number as integer) as claim_line_number,
    /* map the remaining Input Layer columns */
    'example_claims' as data_source
from {{ ref('stg_example_claims__medical_claim') }}

Use the Input Layer page as the authoritative field contract. Do not remove required fields because the source cannot populate them; select a correctly typed null instead.

3. Build only the Input Layer

On a fresh installation, first load Core assets with dbt seed --select package:the_tuva_project. Reuse existing assets when their version, seed schema, and loader contract are unchanged.

Tag the connector's staging and final Input Layer models with input_layer, then build them without running the rest of Tuva:

dbt build --select "package:<your_connector_project_name>,tag:input_layer"
dbt run --select "package:the_tuva_project,tag:input_layer"

Replace <your_connector_project_name> with the root dbt project name. If the connector uses a different selector, select its staging and final models explicitly. The second command materializes the Core wrappers before the Structural Data Quality checks.

Confirm that one source value is used consistently:

select 'medical_claim' as input_table_name, data_source, count(*) as row_count
from input_layer.input_layer__medical_claim
group by data_source

union all

select 'pharmacy_claim', data_source, count(*)
from input_layer.input_layer__pharmacy_claim
group by data_source

union all

select 'eligibility', data_source, count(*)
from input_layer.input_layer__eligibility
group by data_source
order by input_table_name, data_source;

Unexpected spellings or null source values should be corrected in the connector before continuing.

4. Run Structural Data Quality

Run Structural Data Quality only after the enabled Input Layer Models and Tuva Core Input Layer Wrappers have built successfully. A missing or disabled Input Layer Model is a dbt build error before Data Quality runs, and a failed Wrapper causes dbt to skip its downstream structural models. Because Step 3 built the Wrappers first, the following selector can inspect their existing Warehouse Tables or Views. It raises a clear error if a required object is unavailable:

dbt build --select tag:dq_structural

The dq_structural tag includes the readiness matrix, its three failure-only detail tables, the normalized structural_test_results table, and the internal helpers they require.

Review the one-row-per-table-and-source readiness matrix:

select
    data_source,
    input_table_name,
    columns_exist,
    data_types_correct,
    table_populated,
    primary_key_correct,
    row_count
from data_quality.structural
where data_source = 'example_claims'
   or data_source is null
order by data_source, input_table_name;

Interpret the four results as follows:

Result	Action
columns_exist = fail	Add every required field, using a correctly typed null when the source has no value.
data_types_correct = fail	Cast the field to the Input Layer type in the connector.
table_populated = fail	Correct the mapping or domain configuration so the enabled Warehouse Table or View contains records, then rerun before continuing.
primary_key_correct = fail	Repair null or duplicate composite keys before continuing.

Every fail must be fixed. not evaluated means a failed prerequisite prevented that check from running; it is not readiness. Fix the prerequisite, rerun Structural Data Quality, and continue only when all four results pass.

Use the three failure-only structural detail tables to identify the exact kind of mapping problem:

select
    input_table_name,
    column_name,
    expected_data_type
from data_quality.structural_missing_columns
order by input_table_name, column_name;

select
    input_table_name,
    column_name,
    expected_data_type,
    actual_data_type
from data_quality.structural_data_type_mismatches
order by input_table_name, column_name;

select
    data_source,
    input_table_name,
    failure_type,
    primary_key_columns,
    failed_record_count
from data_quality.structural_primary_key_failure_counts
where data_source = 'example_claims'
order by input_table_name, failure_type, primary_key_columns;

For example, if claim_line_number is reported as a string instead of an integer, cast it in the staging mapping, rebuild the Input Layer, and rerun tag:dq_structural.

The primary-key detail table reports counts, not individual key values. The examples below query the Warehouse Table or View produced by the input_layer__medical_claim Wrapper. They assume the default input_layer schema. If tuva_schema_prefix is configured, replace input_layer with <tuva_schema_prefix>_input_layer.

If a null_value row identifies claim_line_number, inspect those records directly:

select *
from input_layer.input_layer__medical_claim
where data_source = 'example_claims'
  and claim_line_number is null;

If the detail reports a duplicate_value failure, use the composite key listed in primary_key_columns to locate every record in a duplicate group:

with duplicate_keys as (
    select
        claim_id,
        claim_line_number,
        data_source
    from input_layer.input_layer__medical_claim
    where data_source = 'example_claims'
    group by
        claim_id,
        claim_line_number,
        data_source
    having count(*) > 1
)

select source_records.*
from input_layer.input_layer__medical_claim as source_records
inner join duplicate_keys
    on (
        source_records.claim_id = duplicate_keys.claim_id
        or (source_records.claim_id is null and duplicate_keys.claim_id is null)
    )
   and (
        source_records.claim_line_number = duplicate_keys.claim_line_number
        or (
            source_records.claim_line_number is null
            and duplicate_keys.claim_line_number is null
        )
    )
   and (
        source_records.data_source = duplicate_keys.data_source
        or (
            source_records.data_source is null
            and duplicate_keys.data_source is null
        )
    )
order by
    source_records.claim_id,
    source_records.claim_line_number;

Adapt these queries to the Input Layer Model, source, and key columns reported by the failure-count table. Null counts for different key columns can overlap, and duplicate counts include every record in a non-unique group, so do not sum the rows and interpret the result as a distinct-record count.

Continue until all four structural results pass. The result tables do not automatically stop unrelated dbt models, so this readiness decision remains part of the user's workflow.

Trust the tables only after this Data Quality run completes successfully. A failed run can leave tables from an earlier run in place, and setting data_quality_enabled: false does not remove them. Do not run overlapping Data Quality builds into the same target schema. Tuva Core records neither the run that produced a table nor refresh history; Tuva DQI adds provenance, history, and remediation workflow.

5. Run Logical Data Quality

After all four Structural Data Quality requirements pass for the current Input Layer Wrapper build, build the Logical Data Quality flag tables and result tables:

dbt build --select tag:dq_logical

tag:dq_logical is the complete selector for the Logical Data Quality pipeline, including its public metadata and result relations and its validation tests.

Start with S1, then review S2 and S3:

select
    data_source,
    input_table_name,
    test_name,
    display_name,
    grain,
    test_type,
    severity,
    total_row_count,
    tested_count,
    failed_count,
    passed_count,
    not_applicable_count,
    case
        when tested_count = 0 then null
        else 1.0 * failed_count / tested_count
    end as failure_rate
from data_quality.logical_test_results
where data_source = 'example_claims'
  and failed_count > 0
order by severity, failure_rate desc, input_table_name, test_name;

failure_rate is calculated in this query; it is not stored in logical_test_results. Its denominator is tested_count, the records to which that individual test applies. A medical-claim-line result and a claim-level result have different denominators and must not be added or averaged.

Understand applicability

For the institutional bill type test:

• an institutional line with a missing bill type has flag 1;
• an institutional line with a bill type has flag 0; and
• a professional line has flag null because the test does not apply.

Tuva reports flags 0 and 1 in tested_count and null flags in not_applicable_count. For each test and data_source, total_row_count equals tested_count + not_applicable_count, and tested_count equals passed_count + failed_count.

Investigate Logical Results

Use the Data Quality Test Catalog for the complete list of tests. Query logical_test_catalog for the test's description, severity, native grain, key columns, flag table, and flag column:

select
    test_name,
    display_name,
    description,
    input_table_name,
    grain,
    key_columns,
    test_type,
    severity,
    flag_table_name,
    flag_column_name
from data_quality.logical_test_catalog
where test_name =
  'medical_claim__bill_type_code_null_for_institutional_claim';

Use that metadata to join the failed native-grain keys back to the Input Layer Wrapper. For example:

select source_rows.*
from input_layer.input_layer__medical_claim as source_rows
inner join data_quality.medical_claim_line_flags as flags
    on source_rows.claim_id = flags.claim_id
   and source_rows.claim_line_number = flags.claim_line_number
   and source_rows.data_source = flags.data_source
where flags.data_source = 'example_claims'
  and flags.bill_type_code_null_for_institutional_claim = 1;

Determine whether the failure is caused by the connector or by the source:

• If the raw bill type exists but was not mapped or normalized correctly, fix the connector.
• If the raw source truly lacks it, document the source limitation and evaluate whether the intended analytics can be trusted.
• If the rule's applicability is wrong, fix the test definition rather than forcing the source into an incorrect mapping.

Rebuild the connector-owned Input Layer Models and the Tuva Core Input Layer Wrappers, rerun Structural Data Quality, and then rerun Logical Data Quality after every correction. After a successful run, Tuva Core publishes the current state only, so save a result separately if you need a before-and-after comparison. Tuva DQI maintains refresh history in operational deployments.

6. Apply the readiness gate

Use this source-specific gate before building and trusting downstream Tuva models:

Checkpoint	Ready to continue when...
Structural	columns_exist, data_types_correct, table_populated, and primary_key_correct all equal pass.
Logical S1	No unresolved failure can prevent Tuva from running correctly.
Logical S2	Material issues are fixed where possible; every remaining issue and analytical limitation is understood.
Logical S3	Minor issues have been reviewed and prioritized for the use case.

Data Quality provides evidence for this decision; it does not automatically approve or reject the source refresh inside Tuva Core.

When the source is ready, build the rest of the project:

dbt build

Working with multiple data sources

Repeat the same workflow for every data_source. Never aggregate sources before the readiness decision: a large clean source can otherwise hide a serious issue in a smaller one.

After a successful Tuva Core run, all sources in the target schema reflect that run's current-state build artifacts. Tuva DQI adds independent refresh promotion: a healthy source can advance while a problematic source is held back for investigation.

Privacy and Failure Keys

Flag and failure-key tables contain identifiers that can be joined to healthcare records. They deliberately avoid copying full records, but should receive the same access controls, retention rules, and auditing as the Input Layer.

Enable the key-only failure relation only when needed. Rebuild the Logical Data Quality flag tables in the same command so the failure keys describe the current Wrapper build:

dbt build --select tag:dq_logical \
  --vars '{data_quality_enabled: true, enable_data_quality_failure_keys: true}'

Query a narrow test and source when retrieving examples for remediation:

select
    data_source,
    input_table_name,
    test_name,
    grain,
    key_columns,
    key_values_format,
    key_values
from data_quality.logical_failure_keys
where data_source = 'example_claims'
  and test_name =
    'medical_claim__bill_type_code_null_for_institutional_claim';

For key_values_format = 'percent_escaped_v1', key_columns lists the key fields in order and key_values stores their values in the same order. Components are separated by |; N means null, while V begins a non-null value, including an empty string. Within a non-null value, % is encoded as %25 and | as %7C. See How to Use Logical Data Quality for the complete decoding rules.

Do not export unrestricted failure-key tables to tickets or chat tools.

Troubleshooting

No Data Quality models were built

Confirm that data_quality_enabled: true is set and that the claims or clinical dbt variable is enabled. Run dbt ls --select tag:dq_structural to inspect the selected nodes.

An enabled Input Layer Model or Wrapper is unavailable

The connector must define every enabled Input Layer Model so dbt can resolve the Wrapper's reference. A missing or disabled Input Layer Model is a dbt build error before Structural Data Quality runs. If an Input Layer Wrapper fails in a combined build, dbt skips its downstream structural models. Fix the upstream error and rebuild the Input Layer before rerunning the four structural checks. Running tag:dq_structural after a separate Input Layer build requires the Wrapper Warehouse objects from that build to remain available. Tuva raises a clear error directing you to rebuild them when one is unavailable.

A source is missing from the structural matrix

Confirm that the same non-null data_source value appears in at least one enabled Input Layer Model in the same domain. Tuva derives the claims and clinical source rosters independently, so a source found only in claims does not create clinical rows, and vice versa. When provider attribution is enabled, it participates in the claims roster.

A named source that is absent from every Model in a domain cannot be discovered from the current Input Layer. Detect that refresh-level condition through Data Quality Intelligence monitoring.

A logical test has no applicable records

Confirm that the source contains records to which the test should apply and inspect the flag's applicability branch. A zero applicable count is different from passing; it means the rule did not evaluate any records.

Results look stale after a connector change

First confirm that the most recent Data Quality run completed successfully with data_quality_enabled: true. A failed or disabled run can leave earlier result tables in the schema. Then rebuild the connector's Input Layer models before rerunning Data Quality. Use a full refresh if your warehouse still contains an obsolete incremental or table shape from an earlier mapping, and avoid overlapping runs that target the same schema.