Labs · Product

Data Control Tower

An operational-intelligence layer for enterprise data platforms. It sits across the integration framework and connects telemetry across systems — sources, jobs, orchestrations, targets — so a failure reads as one explanation chain instead of five dashboards.
Talk about a pilot In development
← Labs

Enterprise data platforms are better instrumented than ever and still take hours to diagnose, because every tool sees one layer and the failure spans several. The Control Tower reads the telemetry those tools already produce and connects it: which source feeds which job, which job feeds which target, which target feeds which dashboard. A failure then reads as one explanation chain instead of five windows.

What it looks like

Three lenses on the same graph — the whole estate, the cause of one failure, the reach of one failure — and the two pages an operator works from. Every screen opens full size on click.

The Operational lens: one tenant as one lineage graph. Sources and systems on the left, assets and processes in the middle, targets on the right; the minimap shows the whole estate at once. The status bar counts 62 nodes — 6 failed, 19 healthy, 7 at risk. Type by icon, health by colour.
The Root Cause lens, zoomed to the failed Check DB process. Everything else is dimmed and the upstream path to the S3 source is lit. The drawer shows the failed job, the two actions — Show Impact and Trace Root Cause — and the root cause the engine found.
The Impact lens, zoomed to the failed SQL Server system with its blast radius applied. The four-node path from the replication jobs through rideshare zipcodes and test_data to SQL Server is lit; the drawer counts two assets at risk, lists each with its hop count and risk score, and names the root cause: the connection test failed.

The operational surfaces: the run queue with its dead-letter tab, and the connections with their failures and inline retry.

The Jobs page, by category. The Movement / ELT group is expanded: S3 imports and SQL Server replications with type, status, enabled toggle, last run and per-row actions. The Dead-Letter Queue tab counts what needs a human; the chips filter running, failed and disabled.
The Connections table. Nine connections with their status, type and provider; the three that failed — GitHub, MS Teams, QuickSight — carry an inline Retry, the fix where the failure is shown. Hosts and descriptions are masked in this capture.

What it does

Lineage graph across sources, jobs and targets

Every connection, job, orchestration gate and lineage record the platform already knows about is synced into one graph on every scheduler tick. Nobody draws it; it is derived, and it stays current.

Failure detection with downstream impact

When a node fails, the engine walks forward and lists what is now affected — the processes, the assets, the dashboards — so “who needs to know” is answered with the failure, not after it.

Root cause as a graph traversal

Walk upstream from the symptom, up to five hops, and score every candidate on severity, recency and proximity. A three-term weighted score you can read in ninety seconds. Deterministic, explainable, debuggable. Not a model.

Inline remediation on failed connections

A red row carries its own fix: retry, reconnect or edit credentials where the failure is shown, not three screens away.

Run queue and orchestration views

The queue, gates and sequences as an operator reads them, with schedules in plain words and the raw cron one hover away. A result is a result; only genuine failures are red.

Health and status without colour overload

Node type by shape and icon, health by colour, split into two legends. Empty states that say “no data yet” rather than looking broken.

Built on

The Control Tower runs on diF, dataMaven’s data integration framework — the same platform that runs the jobs it watches, so the graph, the signals and the fixes share one source of truth.

  • A control-plane API that owns configuration, authentication and a highly available scheduler with leader election, so exactly one trigger fires per job however many pods are running.
  • Specialized worker fleets — replication, push, files, analytics — that scale on queue depth and to zero when idle, draining in-flight jobs before they go.
  • Change data capture by timestamp, id or file hash, schema evolution on replication, and de-identification of sensitive fields before they reach a target.
  • Structured, context-aware logging with job and batch ids on every line, which is the telemetry the Control Tower reads.
  • Connectivity across cloud warehouses, relational databases, SaaS APIs and object storage, with credentials encrypted at rest.

The internals — how the graph is modelled, how lineage and signals are stored, how governance runs — are shown inside a pilot, not on a web page.

Read the thinking

Part 1Dashboards Don’t Run Data PlatformsRead it Part 2Why Did It Break? Root Cause Is a Graph Problem, Not an AI ProblemRead it
Operable data is part of being AI-ready.AI Readiness

Running a platform where a failed dashboard still takes five tools to explain?

The Control Tower is in development and piloting with a small number of platform teams. Forty-five minutes is enough to tell whether your estate is a fit.

Talk about a pilot