extraction-worker — Business Features

1. What it is

An admin uploads a policy, a manual or a safety procedure and expects a course. This worker is the part that actually reads the document: it finds the facts worth teaching, checks that they are genuinely supported by the source text, measures how much of the document it covered, and then arranges the survivors into sections, lessons and concepts. It never talks to a person and never writes lesson prose — authoring-service-v2 owns the human review gate and the draft, and content-worker writes the words.

Who uses it	No human calls it. authoring-service-v2 sends both phases over NATS on behalf of an admin who uploaded a document.
Runtime	FastAPI probe shell + 2 JetStream consumers · Deployment `extraction-worker`, `replicas: 1` (`extraction-worker/k8s/deployment.yaml:9`). Heaviest workload in the namespace: requests 200m/768Mi, limits 3000m/4Gi (`k8s/deployment.yaml:83-87`).
Database	None. Stateless — see §6.
Redis	None.
NATS streams	`AUTHORING_EXTRACT` (`authoring.extract.>`, 7-day retention; `app/config.py:15`); usage rows go to `AUTHORING_USAGE`
External APIs	OpenRouter (a dozen model roles — sweep, classify, merge, cluster, checker, vision — `k8s/configmap.yaml:48-58`) and S3 for artifacts and the LLM call cache (`ARCHITECTURE.md` §3.6)
Entry points	3 HTTP routes (health only) · 2 NATS consumers · 5 published subjects · 2 in-process tasks

Why its liveness probe is `exec true`

Extraction stages are CPU-bound and can starve the event loop for minutes, so any HTTP liveness probe — even a dependency-free one — got the container SIGKILLed mid-run (exit 137, twice, 2026-07-12). The liveness probe is now /bin/sh -c true: the kubelet only confirms the process exists. Real hangs are covered by JetStream redelivery plus the S3 call cache, which lets a redelivered run resume instead of re-spending on the model (k8s/deployment.yaml:59-70).

2. Feature map

flowchart LR
  ADMIN["Admin uploads a document"] --> AV2[authoring-service-v2]
  AV2 --> P1["Phase 1: find what is teachable"]
  P1 --> S1["authoring.extract.run.requested"]
  S1 --> G["gates.ready → human review in authoring-v2"]
  G --> P2["Phase 2: shape it into a course"]
  P2 --> S2["authoring.extract.structure.requested"]
  S2 --> T["tree.ready → draft + lesson fan-out"]
  P1 --> ART["artifacts + call cache in S3"]
  P2 --> ART

3. Features

Admin

No admin-facing entry point. The admin's experience of this worker is the review queue and the progress bar that authoring-service-v2 renders from the events below.

Employee (mobile)

None.

Internal (other services)

Phase 1 — find what is worth teaching, and prove it live

authoring-service-v2

The document is ingested, its language checked, and then swept for candidate learning points. Each candidate is cross-examined by a different model than the one that proposed it: quantities must match the source, polarity must not be flipped, and every claim must be entailed by the text it cites. The worker reports coverage and fidelity as gate results, which is what lets an admin see "we covered 78% of your manual, and these five points need a human look" before any money is spent on writing lessons.

Entry points: authoring.extract.run.requested (durable extraction-worker-run) → authoring.extract.gates.ready or authoring.extract.run.failed, with authoring.extract.progress ticks
Touches: S3 artifacts + call cache, OpenRouter (sweep/classify/merge/checker/vision roles), RAGFlow chunk ids supplied in the request
Related: EXTRACTION_FIGURE_POLICY is required with no default; an unset value fails every run loudly (app/main.py:57-64)
Evidence: consumer app/main.py:35 · publisher authoring-service-v2/app/services/extraction_orchestration.py:190 · gates consumed at authoring-service-v2/app/consumers/register.py:28

Phase 2 — turn approved points into a course tree live

authoring-service-v2

Once the gates pass (or a human releases them), the worker groups the surviving points into concepts, budgets them into lessons of a sensible length, orders the sections so prerequisites come first, and attaches the supporting evidence. The result is the outline the admin confirms and the unit of work every downstream lesson request is cut from.

Entry points: authoring.extract.structure.requested (durable extraction-worker-structure) → authoring.extract.tree.ready or authoring.extract.run.failed
Touches: S3 artifacts + call cache, OpenRouter (cluster/structure/adjudicate/score roles)
Related: Published either automatically when gates pass or on human release; the gate barrier itself lives in authoring-v2
Evidence: consumer app/main.py:42 · publisher authoring-service-v2/app/services/extraction_orchestration.py:635 · tree consumed at authoring-service-v2/app/consumers/register.py:29

Live progress and honest failures live

authoring-service-v2

Both phases take minutes to hours, so the worker ticks its stage as it goes and authoring-v2 turns those ticks into the admin's progress bar. On a permanent failure it always publishes one failure event before acknowledging the message, so a run ends visibly instead of hanging until a timeout.

Entry points: authoring.extract.progress, authoring.extract.run.failed (publish only)
Touches: AUTHORING_EXTRACT stream
Related: Transient trouble is retried instead: rate limits nak with a 60 s delay, other errors nak twice before a terminal failure (app/consumers/run_consumer.py:120-138)
Evidence: progress app/consumers/run_consumer.py:37, app/consumers/structure_consumer.py:61 · failures run_consumer.py:50, structure_consumer.py:46,97 · consumed at authoring-service-v2/app/consumers/register.py:30-31

Report what the reading cost live

authoring-service-v2

Extraction is the most model-hungry step in the platform — dozens of calls per document. Each phase reports its usage rows so authoring-v2 can charge the tenant's credits and so the cost of reading a document is visible next to the cost of writing the lessons.

Entry points: authoring.usage.event (publish only)
Touches: AUTHORING_USAGE stream, owned by rag-context-worker's stream definition
Related: Five of the six authoring pipeline workers publish this subject; authoring-v2's usage consumer is its only reader.
Evidence: app/consumers/run_consumer.py:149, app/consumers/structure_consumer.py:117 · consumed at authoring-service-v2/app/consumers/register.py:68

Background

NATS bootstrap that never gives up live

background

If NATS or its credentials are missing at boot, the worker stays HTTP-healthy and retries the subscription every five seconds forever, rather than crash-looping. A pod that cannot consume still fails readiness, so it receives no traffic and is visible in the cluster.

Entry points: lifespan task _connect_nats_with_retry
Touches: Both durables; the stream is created if absent
Related: House pattern shared with research-worker
Evidence: app/main.py:67 (task created in the lifespan), readiness gate app/health.py:32-38

Concurrent table sweep live

background

Tables in a manual are a rich source of teachable facts but slow to mine. That work runs alongside the ordinary text sweep instead of before it, so a document full of tables does not take twice as long; both results are joined before anything is merged, so the outcome is unchanged.

Entry points: in-process task llm_sweep.extract_tables (not scheduled)
Touches: OpenRouter sweep model, S3 call cache
Related: Phase 1, stages 2–4
Evidence: app/pipeline/phase_points.py:310

4. API reference

No business API. The real surface of this service is §5.

Method	Path	Auth	Feature	Callers	Verdict
GET	`/ready`	none	Readiness — NATS connected	kubelet `readinessProbe` (`k8s/deployment.yaml:51-58`)	live
GET	`/live`	none	Dependency-free liveness	No probe uses it — liveness is `exec ["/bin/sh","-c","true"]` after the exit-137 incident (`k8s/deployment.yaml:65-70`)	suspect
GET	`/health`	none	Diagnostics: S3 reachable, NATS connected, version	No probe and no in-repo client; useful by hand during an incident	suspect

5. Async contracts

Consumes

Subject	Stream	Durable	Published by	Feature	Verdict
`authoring.extract.run.requested`	`AUTHORING_EXTRACT`	`extraction-worker-run` (ack_wait 90 s + keepalive, max_deliver 3)	authoring-service-v2 — `app/services/extraction_orchestration.py:190`	Phase 1	live
`authoring.extract.structure.requested`	`AUTHORING_EXTRACT`	`extraction-worker-structure`	authoring-service-v2 — `app/services/extraction_orchestration.py:635`	Phase 2	live

Publishes

Subject	Consumed by	Feature	Verdict
`authoring.extract.gates.ready` run_consumer.py:145	authoring-service-v2 `extract_consumer.on_gates_ready` (`register.py:28`)	Phase 1 result → review gate	live
`authoring.extract.tree.ready` structure_consumer.py:115	authoring-service-v2 (`register.py:29`)	Phase 2 result → draft + lesson fan-out	live
`authoring.extract.run.failed` run_consumer.py:50; structure_consumer.py:46,97	authoring-service-v2 (`register.py:30`)	Terminal failure	live
`authoring.extract.progress` run_consumer.py:37; structure_consumer.py:61	authoring-service-v2 `extract_consumer.on_progress` (`register.py:31`) — progress bar only	Stage ticks	live
`authoring.usage.event` run_consumer.py:149; structure_consumer.py:117	authoring-service-v2 `usage_consumer` (`register.py:68`) on `AUTHORING_USAGE`	Credit metering	live

Background jobs

Job	Schedule	What it does	Verdict
`_connect_nats_with_retry` (`app/main.py:67`)	Once per pod start, retrying every 5 s until it succeeds	Connects, ensures the stream, binds both durables; keeps HTTP healthy while it retries	live
`llm_sweep.extract_tables` (`app/pipeline/phase_points.py:310`)	Not scheduled — one task per run that contains tables	Mines table segments concurrently with the text sweep	live

No cron and no CronJob: every unit of work arrives as a NATS message.

6. Data it owns

Stateless — this service owns no database and writes no other service's tables. ARCHITECTURE.md §3.4 lists it under "none". Everything it produces lands in two places: the events in §5, and S3 objects under a per-run prefix (app/services/s3_store.py:21-27) holding the phase artifacts plus the idempotency-keyed LLM call cache. That cache is what makes a redelivered run resume rather than re-spend, and it is the only durable state the worker relies on between deliveries — if the bucket is emptied, an in-flight run restarts from scratch at full model cost.

Table	What it holds	Written by
—	No tables. No alembic directory exists in this service.	—

7. Dependencies

flowchart LR
  AV2[authoring-service-v2] -. "run.requested · structure.requested" .-> EW[extraction-worker]
  EW -. "gates.ready · tree.ready · run.failed · progress · usage" .-> AV2
  EW -- "model calls" --> OR["OpenRouter (12 model roles)"]
  EW -- "artifacts + call cache" --> S3[("S3")]

Zero outbound HTTP: ARCHITECTURE.md §3.2 lists extraction-worker as zero-outbound, and its only non-NATS dependencies are OpenRouter and S3.

8. Dead-code verdicts

Every entry point with no in-repo caller. Deleting is a separate decision — see the hub roll-up.

Entry point	Kind	Verdict	Evidence
`GET /live`	HTTP route	suspect	Searched this service's k8s manifests and every other service's client code for the path: nothing calls it. It was written for an HTTP liveness probe that was deliberately replaced by `exec true` (`k8s/deployment.yaml:59-70`). Not dead — the handler is the correct one to re-adopt if the probe ever moves back to HTTP, and the comment explaining why it must not is on the manifest, not the route.
`GET /health`	HTTP route	suspect	Same search. No probe (`k8s/deployment.yaml` uses `/ready` + exec), no in-repo HTTP client — the service is NATS-only and nothing in the repo issues HTTP to it. Keeps value as a hand-run diagnostic: it is the only endpoint that reports S3 reachability (`app/health.py:19-29`).

Both consumers have a named authoring-v2 publisher and all five published subjects have a named authoring-v2 consumer, so no NATS entry point is dead.

9. Sources

extraction-worker/README.md — the phase/stage map, the required EXTRACTION_FIGURE_POLICY, resumability via the S3 call cache
app/main.py, app/consumers/run_consumer.py, app/consumers/structure_consumer.py, app/health.py, app/config.py, app/services/s3_store.py
k8s/deployment.yaml — the exec-liveness rationale and the 3000m/4Gi limits; k8s/configmap.yaml — model roles and stage flags
ARCHITECTURE.md §3.1 (spoke topology), §3.2 (zero-outbound), §3.3 (AUTHORING_EXTRACT and the doc→module flow), §3.4 (stateless), §3.6 (S3 + OpenRouter egress)
authoring-service-v2/docs/ — SDD-OPER-course-extraction rev. 8 and Addenda A/B, the specification this pipeline implements
No bug-hunt-reports/extraction-worker.md exists; bug-hunt-reports/_cross-service-consistency.md is the nearest review of its edges.
Scan output tools/feature-docs/out/extraction-worker.json

1. What it is

Why its liveness probe is exec true

2. Feature map

3. Features

Admin

Employee (mobile)

Internal (other services)

Phase 1 — find what is worth teaching, and prove it live

Phase 2 — turn approved points into a course tree live

Live progress and honest failures live

Report what the reading cost live

Background

NATS bootstrap that never gives up live

Concurrent table sweep live

4. API reference

5. Async contracts

Consumes

Publishes

Background jobs

6. Data it owns

7. Dependencies

8. Dead-code verdicts

9. Sources

Why its liveness probe is `exec true`