Provider Quality

Provider quality is a companion query catalog, not another data pool. Provider tables hold business records at providers.*; providers_quality holds measurements about those tables. Use the latest snapshot across the Workspace or the history for one Provider table:

Surface	Address	Use it for
Latest results	`providers_quality.summary.{table}`	Workspace-wide health, failing checks, SPC breaches, and shape.
Per-table history	`providers_quality.{connector}.{table}__{suffix}`	How a measurement changed across runs.

Only main Provider tables receive quality sidecars. Identity tables, artifacts, and Parable results do not.

Understand the measurements

Quality checks

Quality checks compare a Provider table with thresholds defined by its stream’s quality configuration and schema roles. Each row represents one check on one table.

Status	Meaning
`PASS`	The measurement satisfied its threshold.
`FAIL`	The measurement breached its threshold.
`SKIPPED`	The check lacked a required schema role or evaluation window, or at promote time its target column was absent from the promoted table. `timestamp_coverage` and `timeline_gaps` also skip when the event-time column is not a `Date32` or `Timestamp` column. The evidence names the missing role, window, or column, or the unsupported column type.

SKIPPED is not a pass. It means the rule could not evaluate. Promote-time checks scan the committed Delta table with its stored column types. A reseal replays the day’s promote transaction, so a day committed under an earlier plan keeps the types it was written with; a check whose target column is stored as a string skips on that stored type instead of failing the unit. Retyping a stored column takes a table restatement. no_future_timestamp needs to know the unit of the column it reads. A timestamp column carries its own. An integer column is compared only when its schema field declares x-temporal-format (unix, unix_millis, and so on); an integer metadata_timestamp with no declared format is SKIPPED with integer metadata timestamp declares no x-temporal-format, because seconds, milliseconds and microseconds are all plausible readings (PARABLE-4232).

Check ID	What it measures
`null_rate`	Required-column null rate against `maxRate`.
`dupe_rate`	Duplicate rate on logical business keys against `maxRate`.
`timestamp_coverage`	Event-time values inside the expected window against `minRate`.
`temporal_order`	`effective_from` after `effective_to` against `maxViolations`.
`no_future_timestamp`	Metadata timestamps in the future against `maxViolations`.
`timeline_gaps`	Event-time gaps against `maxGapDays`.
`string_pattern_rate`	Values matching a pattern against `maxRate`.
`referential_integrity`	Orphan rate to a referenced column against `maxOrphanRate`.
`row_count_not_suspicious`	Row count is not one of the configured suspicious values.

Checks run when a table is promoted and on its scheduled cadence. Read the lifecycle column with the result: the two evaluations cover different rows.

Lifecycle	Rows checked	What a clean duplicate check establishes
`promote`	Rows carrying the promote unit’s `dt` in the committed table.	No repeated key in that load when `observed = 0`.
`cadence`	The whole committed Provider table across all `dt` values.	No repeated key across the table when `observed = 0`.

On a table that holds current state, the rows carrying a load’s dt are the load’s own rows and the rows it newly marked missing, not the whole table (see Read row state). A promote PASS can coexist with duplicates across loads. Use the latest cadence result to assess the whole table, and check captured_at against its latest load. A missing or older cadence result does not establish current whole-table uniqueness. Production Workspaces may have no cadence rows at all: on 2026-09-14 every published Career Certified result was a promote result (PARABLE-3989). Until that is resolved, a whole-table duplicate measurement needs the SQL check over the table itself. dupe_rate counts rows beyond the distinct combinations of the schema’s key columns, divided by the total rows checked. A PASS means the rate satisfied maxRate; only an observed rate of zero establishes uniqueness. Before treating a repeated entity ID as a defect, confirm the table’s record grain: an event or history table can legitimately hold several records for one entity and needs a key that identifies each record. For a query restricted to a date window, a nonzero whole-table duplicate rate does not locate the duplicates inside that window. Inspect the records in the same window and with the same key before drawing that conclusion.

Statistical process control

Statistical process control (SPC) uses a Shewhart control chart for platform-owned series such as row count and column null percentage. It complements explicit quality checks; it does not replace them.

Series status	Meaning
`IN_BOUNDS`	The latest point is inside the warning limits.
`WARNING`	The latest point is outside two sigma but inside three sigma.
`OUT_OF_BOUNDS`	The latest point is outside the three-sigma control limits.
`INSUFFICIENT_HISTORY`	The series has fewer points than the configured minimum.

Tracked series kinds include TABLE_ROW_COUNT, COLUMN_NULL_PCT, COLUMN_VALIDATION_FAILURE_PCT, COLUMN_DISTINCT_COUNT, and COLUMN_AVG.

Shape and health

Shape statistics record measurements such as row count, column count, and bytes. Distinct counts skip engine-owned _parable_*, _raw_*, and _custom_fields columns — those payloads are not user columns, and a distinct count of the raw row is the row count. Every other distinct count is a HyperLogLog approximation (approx_distinct), so its memory does not grow with the table (PARABLE-4794). distinct_counts_exact is false whenever any column was counted, and true only when none was. A table whose Delta log counts more rows than the stats job’s scan limit (PROVIDER_TRANSFORMATIONS_STATS_SCAN_MAX_ROWS, 50 million by default) is read through a deterministic sample of its files that holds about that many rows. The snapshot records the rows it read as sampled_row_count, which is null for a whole-table snapshot. In a sampled snapshot:

Field	Source
`row_count`, `total_byte_size`	The Delta log, for the whole table.
A column’s null count and min/max	The Delta log where its file statistics record them, which by default is the first 32 columns. Otherwise the sample.
`null_pct`, `validation_failure_count`, `validation_failure_pct`	Scaled from the sample to the table’s row count when the Delta log has no null count.
`avg`, `std`, `q25`, `q50`, `q75`	The sample.
`distinct_count`	Null, because a sample’s distinct count says nothing about the table’s.

The same files are sampled on every run against the same table version, so a series keeps its meaning from run to run. The Query Layer passes a sampled snapshot’s column values to the query planner as estimates, never as exact values. Health is the catalog annotation for a table, including its state, reason, and watermark. Health and the quality/SPC rollups are related views, but they are computed through different paths.

Query the latest Workspace snapshot

These summary tables contain the current result per Provider table:

Table	Contents
`providers_quality.summary.quality_checks`	Latest quality check per table and lifecycle.
`providers_quality.summary.spc_results`	Latest SPC series per table.
`providers_quality.summary.table_stats`	Latest row, column, and byte counts.
`providers_quality.summary.table_health`	Health state, reason, and watermark.
`providers_quality.summary.table_commits`	Each table’s newest version (`version`) and when it was committed (`committed_at`, to the second): how old the data a live read sees is.

Each summary row describes one Provider table, so it follows that table’s Provider Policy: someone the Policy narrows or hides the table for gets none of its rows here. In all four tables, connector and table_name are the catalog’s schemaName and logicalName for the Provider table, the schema and table of its SQL address: the row for providers.google.users has connector = 'google' and table_name = 'users'. They follow the rules in Provider table names, so a connector slug written with - appears here with _. table_commits reads each table’s Delta log when asked, at most once a minute per table. Filter it on connector so only those Providers’ logs are read:

SELECT connector, max(committed_at) AS data_as_of
FROM providers_quality.summary.table_commits
WHERE connector IN ('google', 'zoom')
GROUP BY connector

Failing checks for Google Workspace data:

SELECT
  connector,
  table_name,
  lifecycle,
  check_id,
  check_status,
  observed,
  threshold,
  comparator,
  captured_at
FROM providers_quality.summary.quality_checks
WHERE connector = 'google'
  AND check_status = 'FAIL'
ORDER BY captured_at DESC

Every check that did not pass:

SELECT
  connector,
  table_name,
  lifecycle,
  check_id,
  check_status,
  observed,
  threshold
FROM providers_quality.summary.quality_checks
WHERE check_status IN ('FAIL', 'SKIPPED')
ORDER BY connector, table_name, lifecycle, check_id

Whole-table duplicate measurements for the Workspace:

SELECT
  connector,
  table_name,
  check_status,
  observed,
  threshold,
  captured_at
FROM providers_quality.summary.quality_checks
WHERE lifecycle = 'cadence'
  AND check_id = 'dupe_rate'
ORDER BY connector, table_name

This query returns only tables with a cadence result. Check the per-table history and load status when an expected table has no row. SPC series with a warning, control-limit breach, or recorded violation:

SELECT
  connector,
  table_name,
  series_id,
  series_kind,
  latest_value,
  mean,
  stddev,
  series_status,
  overall_status,
  captured_at
FROM providers_quality.summary.spc_results
WHERE has_violations = true
   OR series_status IN ('WARNING', 'OUT_OF_BOUNDS')
ORDER BY captured_at DESC

Largest Provider tables by row count:

SELECT
  connector,
  table_name,
  row_count,
  column_count,
  total_byte_size,
  captured_at
FROM providers_quality.summary.table_stats
ORDER BY row_count DESC
LIMIT 20

Provider tables with stale watermarks:

SELECT
  connector,
  table_name,
  health_state,
  health_reason,
  watermark,
  watermark_stale,
  computed_at
FROM providers_quality.summary.table_health
WHERE watermark_stale = true

Query one table’s history

History suffixes map to result kinds:

Suffix	History of
`__checks`	Quality checks, one row per check per run.
`__spc`	SPC series, one row per series per run.
`__stats`	Shape statistics.
`__runs`	Maintenance runs such as compaction and vacuum, not quality scores.

Recent checks for Google Workspace directory users:

SELECT
  captured_at,
  check_id,
  check_status,
  observed,
  threshold,
  lifecycle,
  overall_status
FROM providers_quality.google.users__checks
ORDER BY captured_at DESC
LIMIT 20

Row-count SPC over time for the same table:

SELECT
  captured_at,
  series_id,
  series_kind,
  latest_value,
  mean,
  upper_control3_sigma,
  lower_control3_sigma,
  series_status
FROM providers_quality.google.users__spc
WHERE series_kind = 'TABLE_ROW_COUNT'
ORDER BY captured_at DESC
LIMIT 20

History tables retain each run. They can also expose result_id, run_id, evidence_json for checks, and violations_json for SPC when an application needs the payload behind a status.

Older sidecar records can use statuses such as Fail or OutOfBounds; newer records use FAIL and OUT_OF_BOUNDS. Summary tables use typed decoding and return uppercase values. If a history query unexpectedly returns no rows, inspect the available values before choosing a casing-sensitive filter.

Admin

​Understand the measurements

​Quality checks

​Statistical process control

​Shape and health

​Query the latest Workspace snapshot

​Query one table’s history

Understand the measurements

Quality checks

Statistical process control

Shape and health

Query the latest Workspace snapshot

Query one table’s history