Handing the boring parts to agents

Most of data engineering is not clever. The clever parts — architecture, modeling, the occasional real algorithm — are maybe a tenth of the calendar. The rest is toil with a keyboard: the third nearly-identical pipeline this quarter, the schema that drifted over the weekend, the backfill, the lineage document that was accurate for one glorious week in March.

Last year I started wiring AI agents into that boundary — first as internal tooling at work, then as a habit I could not put down. These are my field notes so far.

What hands over cleanly

The tasks that agents absorb well share a shape: expensive to type, cheap to verify.

Pipeline boilerplate is the obvious one. The forty-seventh Spark job in a codebase is not designed, it is transcribed — the patterns already exist in the other forty-six, and an agent that has read them will produce a draft that a reviewer can check in minutes. Same for DAG scaffolding, test fixtures, backfill scripts, and the first pass of any migration that is really a thousand small renames.

Failure triage is the one I did not expect. A pipeline that died at 3 a.m. leaves logs, metrics, lineage, and a diff of yesterday’s inputs. Collecting those into one coherent story used to be the first forty minutes of every incident. An agent does the collection in seconds — not the judgment, the collection — and the human starts at minute forty-one.

Documentation, especially lineage and column descriptions, is the sleeper hit. Nobody writes it because the cost lands on the writer and the benefit lands on a stranger six months later. Agents invert that economics: drafts appear by default, and the human cost drops to editing.

What refuses

The failures are just as consistent. Anything without an oracle stays human.

An agent can tell you the column is a nullable string named customer_type_2. It cannot tell you whether the business means the same thing by it in both source systems, because the answer is not in any system — it is in a meeting. Data quality is a social problem wearing a technical costume, and agents do not attend meetings.

Ownership is the same. Half of platform work is deciding who is responsible for a dataset, and that is a negotiation, not a lookup.

And schemas — the contracts between teams — I keep firmly on the human side. Not because agents write bad schemas, but because a contract you did not read is a contract you did not sign.

The review is the job

The uncomfortable arithmetic: if drafting drops to nearly free, everything rides on the review. The quality gate stops being a checkbox at the end of the work — it becomes the work.

That changes what a good data engineer looks like. Less typing, more reading with intent. The skills that matter compound: knowing what wrong looks like, smelling a plausible-but-false join, insisting on the boring version even when the generated clever version compiles.

My rule of thumb so far fits in a comment:

# Hand to an agent when:
#   the pattern already exists elsewhere in the repo,
#   a reviewer can falsify the output quickly,
#   and being wrong is cheap to catch before it is expensive.
# Keep when any of those fail.

The itch, once you start, does not stay at work. It follows you home, which is roughly how this site got built.