Document extraction pipeline with Dapr Workflows: a demo implementation
A demo project shows how to build a document extraction pipeline using Dapr Workflows, with FastAPI services and OpenAI structured outputs. The pipeline handles large documents via fan-out and enforces durable workflow constraints, including deterministic functions and retry policies. Deployment considerations include lack of Workflow API support on Azure Container Apps, history size limits, and the need for versioning.

A local demo project in the dapr-workflow repository runs three FastAPI services alongside Dapr sidecars to build a document-extraction pipeline. Uploading a document and a JSON schema starts a workflow that converts the file to Markdown, extracts fields via OpenAI structured outputs, and writes the results somewhere. It is a local demo: Azurite development keys, permissive CORS, no authentication.
Dapr is an open-source CNCF runtime that runs beside an application as a sidecar. Workflow is one of its building blocks, stable since Dapr 1.15, with SDKs for Python, .NET, Java, Go, and JavaScript. The demo uses Python. The programming model is the same idea behind Azure Durable Functions and Temporal: an orchestrator function calls activities, and the engine keeps an append-only history in a state store. After each activity completes, the workflow function is replayed from the top, with recorded results substituted for completed steps. A process restart therefore costs only the step that was running. Each yield in the workflow function is a point where it can stop and resume.
The workflow uses common patterns. Most documents take a short path: retrieve, convert, plan chunks, extract. Large ones fan out, extracting chunks in parallel before a merge step. In the demo, the full text of the Odyssey, about 305,000 tokens, splits into two chunks and takes roughly two and a half minutes. Activities carry a retry policy: on failure, the engine waits two seconds, then four, up to three attempts, on durable timers that survive restarts.
Durable workflow constraints apply. Workflow functions must be deterministic: no datetime.now(), no random values, no new UUIDs, no direct I/O; anything that touches the outside world goes into an activity. Activity inputs and outputs end up in workflow history, which has a default maximum body size of 4 MiB. Passing a 1.7 MB PDF as base64 hit that limit; activities now pass blob names instead of content. Activities also run at least once, so repeating them must be harmless—the demo’s steps only write blobs, not emails or payments.
There are deployment caveats. Azure Container Apps has Dapr built in but does not support the Workflow API; on Azure, Kubernetes with AKS is the path. The state store must support actors, and completed history remains in the store until purged. The SDK does not enforce determinism, and changing a workflow while instances are running can break replays; plan for versioning before deploying changes.
The code is on GitHub at gbaeke/dapr-workflow. The same repository includes a second pipeline that wraps a LangChain agent as a workflow, where the agent chooses the steps and each model turn and tool call runs as an activity. That is the subject of a planned follow-up, and a test of the durability model without a fixed step order.
Read the original at baeke.info →