The Pipeline That Replaced First-Pass Contract Review

Last year, a mid-size commercial real estate firm in Chicago let go of two of its three junior contract analysts. Not because the work disappeared — their lease portfolio actually grew — but because a single deployment of a retrieval-augmented LLM was handling the first eighty percent of what those analysts used to do. The remaining analyst now reviews flagged clauses and edge cases. She describes her job as "arguing with a very confident robot about indemnification language."

This is not a story about AI being smarter than lawyers. It is a story about a specific technical pipeline that turns a legal document into a structured risk report, and understanding that pipeline is the only way to understand what is actually being displaced and what is not.

The Input Problem

Contracts arrive in a remarkable variety of formats: scanned PDFs with inconsistent OCR quality, Word documents with tracked changes layered on top of tracked changes, emails with inline redlines, and occasionally, a fax. The first stage of any serious contract-review pipeline is document normalization. Optical character recognition tools — AWS Textract, Azure Document Intelligence, and open-source alternatives like Tesseract — convert scanned images into text. Tracked-changes documents are programmatically accepted or extracted into a diff representation so the model sees the proposed state of the contract, not a soup of strikethroughs.

The output of this stage is clean plain text, usually with some structural metadata preserved: section headings, clause numbers, page boundaries. This matters because the next stage depends on knowing where in the document a piece of text lives.

Chunking and Retrieval

A commercial lease can run to 150 pages. A merger agreement can exceed 400. Even the largest context-window models — as of early 2026, Gemini 1.5 Pro supports up to one million tokens, and several competitors are in similar territory — introduce latency and cost when you stuff an entire document in at once. More importantly, whole-document prompting degrades precision: the model tends to miss things buried in middle sections, a phenomenon researchers at Stanford have called "lost in the middle" attention degradation.

Production pipelines therefore chunk the contract into overlapping segments, typically by clause or by fixed token counts with a sliding overlap, and embed each chunk using a sentence-transformer or a provider's embedding API. These embeddings are stored in a vector database — Pinecone, Weaviate, and pgvector are common choices. When a reviewer submits a query ("find all indemnification obligations that are not capped") or when the pipeline runs a predefined playbook of risk checks, the system retrieves the top-k most semantically relevant chunks before passing them to the LLM.

This retrieval-augmented generation (RAG) architecture solves the context-length problem but introduces a new one: if a clause is relevant to a query but happens to be embedded in a way that scores poorly against it, it won't be retrieved. Indemnification language buried inside a "miscellaneous" section with unusual phrasing can slip through. Every team deploying this architecture knows about recall gaps and works around them differently — usually by running multiple query reformulations and merging results.

The Model Layer

The LLM itself receives a prompt that includes the retrieved chunks, the playbook question, and a structured output instruction. The playbook is the key artifact here. Legal teams encode their institutional knowledge into a playbook: a list of clause types to check, acceptable versus unacceptable formulations, jurisdiction-specific rules, and client-specific preferences. A typical commercial real estate playbook might contain forty to eighty checks — assignment restrictions, holdover rent multipliers, exclusivity clauses, HVAC operating hours, and so on.

The model is instructed to return a structured JSON object for each check: the clause it found, a verbatim quote, a risk rating (commonly a three-tier system: acceptable, needs negotiation, escalate), and a plain-language explanation. GPT-4-class models and their successors handle this reliably for well-formed contracts in standard English. The JSON parsing failure rate in production tends to be low — under two percent in most reported deployments — and is handled by a retry loop with a stricter prompt.

What the model does not do is reason across the entire contract simultaneously. It does not natively notice that clause 4.2 creates an obligation that clause 12.7 quietly eliminates. Cross-clause reasoning requires either a separate pipeline pass with both clauses retrieved together, or a human reviewer catching the interaction. This is one of the hardest unsolved problems in automated contract review.

Post-Processing and Confidence Scoring

Raw LLM output cannot go directly to a lawyer or a business stakeholder without some scaffolding. Most production systems apply a post-processing layer that deduplicates overlapping clause detections (because chunking overlap means the same clause might be analyzed twice), normalizes risk ratings against historical calibration data, and produces a summary report sorted by risk tier.

Some teams attach a confidence score derived from the model's logprob output or from consistency across multiple temperature-varied runs of the same query. Low-confidence items are automatically routed to human review regardless of risk tier. This is not a nicety — it is load management. The goal is to maximize the proportion of the document that a human never needs to read, while ensuring the highest-stakes and most ambiguous material always gets human eyes.

Where Supervision Remains Non-Negotiable

The pipeline does not make decisions. Every legal team that has deployed this in a production setting — and the landscape of vendors including Ironclad, Kira Systems (now part of Litera), Luminance, and SpotDraft are all explicit about this — maintains that a licensed attorney must review the escalated items and sign off before a contract proceeds. This is partly liability management and partly genuine necessity: the model cannot assess whether a clause is acceptable given the specific leverage and relationship dynamics of a deal.

The model also struggles with heavily negotiated contracts that deviate significantly from training distribution. A contract written in legalese native to a specific jurisdiction — say, Scots law lease terminology, or Brazilian civil code references in a Portuguese-English bilingual document — will produce higher error rates. Multilingual contracts remain an active area of work rather than a solved problem.

What Actually Got Displaced

The work that disappeared is the mechanical first pass: reading a 90-page lease to find every place the landlord's form deviates from the tenant's standard positions. That task required attention, pattern recognition, and familiarity with precedent — but not judgment about business consequences. The pipeline handles it faster, more consistently, and at a fraction of the cost.

The work that remained is everything requiring judgment about tradeoffs, client relationships, and strategic context. The Chicago analyst still catches things the model misses, still negotiates language, and still explains risk to clients. Her job changed more than it shrank — but for her two former colleagues, the arithmetic was less forgiving.

Aria Chen, who runs clankers.today and was herself displaced from a software engineering career in 2024, has noted that the hardest part of AI displacement is not that the technology is infallible — it clearly is not — but that it is good enough to eliminate the entry-level roles that used to serve as the training ground for the senior ones.

PROTECT YOUR INCOME NOW

If AI automation threatens your job, preparation is the only honest answer: know your risk, build a runway, and reskill before you have to. The Clankers 90-day programme structures exactly that — free, which is exactly why today is the right time to start.

Start the 90-Day Programme — Free →

GET THE CLANKERS BRIEF — FREE

One email every Monday: the week's AI displacement moves that matter, and what to do about them. No spam. Unsubscribe anytime.

60-second check Is your job at risk? Take the free AI displacement risk quiz → CV Review — from $20 Written from the hiring side Twenty years of hiring, applied to your CV →