There is a large gap between extracting a knowledge graph once and running extraction as a process. The first is a demo. The second is a workflow that a colleague can pick up, understand, and run without you in the room. Most teams build the demo, ship it, and then discover that nobody else can maintain it because the knowledge lives in someone's head and a few undocumented prompt tweaks.
A real workflow has discrete stages, defined inputs and outputs at each boundary, and artifacts that survive the person who created them. When extraction breaks, you should be able to point at the stage that failed rather than staring at a black box. When you onboard someone, they should be able to read the documentation and run the pipeline the same day.
This article describes how to turn knowledge graph extraction from an ad hoc task into a documented, repeatable, hand-off-able workflow. The stages build on each other, and each produces an artifact that the next stage consumes.
Stage One: Define the Target
The schema document
The workflow starts with a written schema, not a prompt. The schema document lists every entity type and relationship type you intend to extract, each with a one-line definition and a real example. It records the rules for ambiguous cases: when co-occurring entities imply a relationship and when they do not. This document is the contract every later stage references.
Why it comes first
If the target is undefined, every downstream stage produces output you cannot evaluate or aggregate. The schema is the cheapest thing to change and the most expensive thing to get wrong, so it leads. The reasoning behind tight schemas is laid out in Straight Answers on Turning Text Into Knowledge Graphs.
Stage Two: Build the Extraction Prompt
What goes into the prompt
The prompt embeds the schema, includes three to five diverse examples spanning different relationship types, requests structured output, and requires a source span for every triple. The prompt is a versioned artifact stored alongside the code, not a string pasted into a notebook. Every change to it is tracked.
The output contract
Define exactly what the prompt returns: a list of triples, each with subject, relationship, object, source span, and a confidence value. Downstream stages depend on this shape, so it becomes part of the documented interface. Change the contract and you change every consumer, which is why it is written down.
Stage Three: Resolve Entities
Maintaining the canon
Before relationships matter, entities must be canonical. This stage maintains a list of resolved entities and matches new mentions against it, adding genuinely new entities with a canonical form. Passing this list back into extraction keeps the model from minting duplicate nodes for spelling variants.
The artifact
The output is a resolved entity list per document and an updated global canon. Both are stored, because the next stage and every future re-extraction depend on them. Resolution that happens here, close to extraction, prevents the fragmented graphs that are painful to repair downstream.
Stage Four: Extract Relationships
The second pass
With entities resolved, a second extraction pass asks only about relationships among the known entities. For long documents this is where decomposition pays off, recovering relationships that span paragraphs a single pass would miss. The output is a set of candidate triples, each tied to a source span and a confidence value.
Keeping it auditable
Because relationships reference resolved entities and cite source spans, every triple is traceable back to the document that produced it. Auditability is not a feature you add later; it is a property of designing the stage to emit provenance from the start.
Stage Five: Validate
Two layers of checking
The validation stage applies structural checks (does the output parse and match the schema) and content checks (does the cited span exist, do the entities appear in it, is the relationship direction sensible). Triples below the confidence threshold route to a human review queue rather than the graph.
The handoff boundary
Validation is the boundary between machine output and trusted graph. Everything before it is a candidate; everything after it is committed. Making this boundary explicit is what lets a reviewer focus only on the uncertain cases instead of reading everything. The same surface-versus-substance distinction appears in A Step-by-Step Approach to Controlling Formality and Register in Output.
Stage Six: Load and Reconcile
Committing to the graph
Validated triples load into the graph store with full provenance: which document, which version, which span. When a source document changes, this stage re-extracts and reconciles, retiring triples that no longer have source support and adding new ones. Triple-level provenance is what makes incremental updates possible.
The documentation that travels
Each stage ships with a short runbook: its input, its output, how to run it, and how to tell when it has failed. Together these runbooks are the hand-off package. A new team member reads them and runs the workflow without decoding anyone's prompts. This is the difference between a process and a person.
Stage Seven: Monitor and Improve
Watching the workflow in production
A workflow that ships is not a workflow that is finished. The final stage watches the running pipeline for signals that quality is slipping: a rising rate of low-confidence triples, more documents landing in the review queue, or gold-set scores trending down after a model or prompt change. Each signal points at a stage to investigate rather than a vague sense that something is wrong.
Closing the loop
When monitoring surfaces a problem, the fix flows back through the staged structure. A spike in duplicate nodes points at the resolution stage; a drop in recall points at the relationship pass; a rise in fabricated triples caught at validation points at the prompt or schema. Because each stage has a defined input, output, and runbook, you can isolate and repair the failing stage without disturbing the rest, which is exactly the property that ad hoc pipelines lack. The same continuous-improvement discipline appears in Tone Discipline That Survives Real Production Volume.
Frequently Asked Questions
How is a workflow different from just running a good prompt?
A prompt produces output once. A workflow has staged boundaries, versioned artifacts, and documentation that lets someone else run it. The prompt is one component inside the workflow, not the whole thing. The workflow is what survives staff turnover.
Which stage should I document most carefully?
The schema and the output contract. These are the interfaces every other stage depends on, so ambiguity here propagates everywhere. If a new person can read the schema document and the contract and understand what the pipeline targets, the rest is easier to follow.
Can I skip entity resolution for a first version?
You can, but you will pay for it. Without resolution you get duplicate nodes and fragmented edges that are costly to merge later. If you skip it initially, isolate it as a clearly marked gap so it is easy to add rather than buried in extraction logic.
How do I keep prompt changes from silently breaking the workflow?
Version the prompt as code, and run every change against a gold set that reports precision and recall. A change that improves one document and quietly degrades the corpus shows up immediately. Treat the prompt with the same change discipline as any other code artifact.
What does a good handoff package contain?
The schema document, the versioned prompts, the output contract, a runbook per stage, and the gold set used for evaluation. With these, a competent newcomer can run, test, and modify the workflow without the original author. Anything less and the knowledge stays trapped in one head.
Key Takeaways
- A workflow has staged boundaries, versioned artifacts, and runbooks; a good prompt is just one component inside it.
- Start from a written schema document, not a prompt, because it is the contract every later stage references.
- Resolve entities close to extraction and pass the canon back into the model to prevent duplicate, fragmented nodes.
- Make validation the explicit boundary between candidate output and trusted graph, routing low-confidence triples to human review.
- Ship a runbook per stage and a gold set so a new team member can run and modify the workflow without reverse-engineering it.