research-worker — Business Features

1. What it is

Sometimes an admin has no manual to upload — just a sentence like "forklift safety for warehouse staff". This worker does the reading for them: it searches the web, proposes the topics a course on that subject should cover, and after the admin picks the ones they want, researches each topic in depth and synthesises a full course outline grounded in cited sources. It is the prompt-to-module twin of extraction-worker, and it hands authoring-service-v2 the same shape of outline so everything downstream is identical.

This page is the only documentation of this service

research-worker has no README.md, no docs/ and no bug-hunt report. Every statement below is derived from its code and manifests, with the file and line named. The design it implements is referred to in code comments as SDD-research-worker / SDD-research-module; those specs are not present in this repository.

Who uses it	No human calls it. authoring-service-v2 sends both depths over NATS on behalf of an admin in the creation dialog.
Runtime	FastAPI probe shell + 2 JetStream consumers · Deployment `research-worker`, `replicas: 1` (`research-worker/k8s/deployment.yaml:9`); requests 100m/512Mi, limits 1000m/1Gi (`:100-104`)
Database	None. Stateless — see §6.
Redis	None.
NATS streams	`AUTHORING_RESEARCH` (`authoring.research.>`, 7 days; `app/config.py:15`) — the hottest authoring stream per `ARCHITECTURE.md` §3.3; usage rows go to `AUTHORING_USAGE`
External APIs	OpenRouter, using Perplexity Sonar search models for discovery and per-topic research and Claude/Sonnet for outline synthesis (`k8s/configmap.yaml:18-23`); S3 for artifacts and the call cache
Entry points	3 HTTP routes (health only) · 2 NATS consumers · 5 published subjects · 2 in-process tasks

2. Feature map

flowchart LR
  ADMIN["Admin types a course prompt"] --> AV2[authoring-service-v2]
  AV2 --> F1["Suggest the topics to cover"]
  F1 --> S1["authoring.research.run.requested"]
  S1 --> R1["topics.ready → admin picks topics (gate 1)"]
  R1 --> F2["Research the chosen topics deeply"]
  F2 --> S2["authoring.research.deep.requested"]
  S2 --> R2["tree.ready → outline confirm (gate 2)"]
  F2 --> PROG["progress ticks while it works"]
  R2 --> LESSONS["lesson generation fan-out"]

3. Features

Admin

No admin-facing entry point. The admin sees this worker's output as the topic picker, the progress bar and the outline confirmation screen, all rendered by authoring-service-v2.

Employee (mobile)

None.

Internal (other services)

Suggest what a course on this subject should cover live

authoring-service-v2

The fast first pass. One search-model call expands the admin's prompt into ranked candidate topics, each with a short rationale, a confidence and its sources — deliberately aimed at under a minute, because the admin is waiting at the create dialog. Those candidates become the first review gate: the admin ticks the topics worth teaching before any expensive research starts.

Entry points: authoring.research.run.requested (durable research-worker-run) → authoring.research.topics.ready or authoring.research.run.failed
Touches: OpenRouter perplexity/sonar-pro (discovery), S3 (topics.json artifact + per-call cache)
Related: Published by the request-module-from-prompt tool path in authoring-v2; capped at RESEARCH_MAX_TOPICS=12 (k8s/configmap.yaml:25)
Evidence: consumer app/main.py:35 · pipeline app/pipeline/discovery.py:1-6 · publisher authoring-service-v2/app/tools/handlers/orchestrator_handlers.py:224 · result consumed at authoring-service-v2/app/consumers/register.py:34

Research the chosen topics and write the outline live

authoring-service-v2

The expensive second pass, and the reason a prompt can become a real course. Each topic the admin confirmed is researched on its own — for high-richness runs with a deep-research model that plans and runs its own multi-step search, tens of minutes per topic — producing a dossier of passages judged for relevance and source quality. The dossiers are then synthesised into one outline: sections, lessons, concepts, learning points and the citations behind them. Individual topics are allowed to fail; the run only fails if too few dossiers survive, so one dead source does not throw away an hour of work.

Entry points: authoring.research.deep.requested (durable research-worker-deep) → authoring.research.tree.ready or authoring.research.run.failed
Touches: OpenRouter (sonar / sonar-pro / sonar-deep-research by richness, Claude for synthesis), S3 (dossiers + tree artifacts + call cache)
Related: Failure tolerance RESEARCH_DOSSIER_SUCCESS_FLOOR=0.5 (k8s/configmap.yaml:36); deep-call timeout 1800 s (:38). The worker mints every id in the tree and never trusts model-supplied ids.
Evidence: consumer app/main.py:42 · dossier floor app/pipeline/deep_research.py:430-435 · outline rules app/pipeline/outline.py:1-8 · publisher authoring-service-v2/app/services/research_orchestration.py:76 · result consumed at authoring-service-v2/app/consumers/register.py:35

Live progress and honest failures live

authoring-service-v2

A deep run can take half an hour, so the worker reports each topic as it completes — which topic, how many of how many. Those ticks drive the admin's progress bar and double as the heartbeat authoring-v2's stall sweeper looks for. Every terminal outcome publishes exactly one result or failure event before the message is acknowledged, so a run never silently disappears.

Entry points: authoring.research.progress, authoring.research.run.failed (publish only)
Touches: AUTHORING_RESEARCH stream
Related: Provider pushback (429/402) is retried in-client against Retry-After before it ever becomes a redelivery, because dropping a call throws away real research (app/config.py:32-38)
Evidence: progress app/consumers/deep_consumer.py:30 · failure app/consumers/run_consumer.py:43 · consumed at authoring-service-v2/app/consumers/register.py:36-37

Report what the research cost live

authoring-service-v2

Search models bill per call and deep research bills a lot per call, so every phase reports its usage rows. authoring-v2 turns them into the tenant's credit ledger, which is what makes a prompt-to-module run quotable before it starts and chargeable after it finishes.

Entry points: authoring.usage.event (publish only, best-effort)
Touches: AUTHORING_USAGE stream, whose definition lives with rag-context-worker
Related: Five of the six authoring pipeline workers publish this subject; authoring-v2's usage consumer is its only reader.
Evidence: app/consumers/run_consumer.py:129, app/consumers/deep_consumer.py:109,132,176 · consumed at authoring-service-v2/app/consumers/register.py:68

Background

NATS bootstrap that never gives up live

background

If NATS or its credentials are missing at boot, the worker stays HTTP-healthy and keeps retrying the subscription every five seconds instead of crash-looping. Readiness still fails while nothing is bound, so a pod that consumes nothing is visible rather than silently idle.

Entry points: lifespan task _connect_nats_with_retry
Touches: Both durables; creates the stream if absent
Related: Same house pattern as extraction-worker; preStop: sleep 15 lets an in-flight call land in the S3 cache before shutdown (k8s/deployment.yaml:83-90)
Evidence: app/main.py:59 · readiness gate app/health.py:35-39

Per-topic research fan-out live

background

Topics are researched concurrently rather than one after another, which is the difference between a deep run taking twenty minutes and taking two hours. Each topic's failure is contained to that topic, and a rate-limit error is deliberately re-raised so the whole message is retried instead of half-finished.

Entry points: in-process task _one per selected topic, inside a task group (not scheduled)
Touches: OpenRouter, S3 call cache
Related: Progress ticks are emitted as each task completes, so they arrive out of order by design and carry the topic id
Evidence: app/pipeline/deep_research.py:423, rate-limit unwrapping at :424-428

4. API reference

No business API. The real surface of this service is §5.

Method	Path	Auth	Feature	Callers	Verdict
GET	`/ready`	none	Readiness — NATS connected	kubelet `readinessProbe` (`k8s/deployment.yaml:63-70`)	live
GET	`/live`	none	Dependency-free liveness	No probe uses it — liveness is `exec ["/bin/sh","-c","true"]` because deep-research phases hold the loop busy (`k8s/deployment.yaml:71-80`)	suspect
GET	`/health`	none	Diagnostics: S3 reachable, NATS connected, version	No probe and no in-repo client	suspect

5. Async contracts

Consumes

Subject	Stream	Durable	Published by	Feature	Verdict
`authoring.research.run.requested`	`AUTHORING_RESEARCH`	`research-worker-run` (ack_wait 90 s + keepalive, max_deliver 3)	authoring-service-v2 — `app/tools/handlers/orchestrator_handlers.py:224` (prompt→module entry, `app/services/research_orchestration.py:8`)	Suggest topics	live
`authoring.research.deep.requested`	`AUTHORING_RESEARCH`	`research-worker-deep` (ack_wait 120 s + keepalive)	authoring-service-v2 — `app/services/research_orchestration.py:76`, after the admin confirms topics	Deep research + outline	live

Publishes

Subject	Consumed by	Feature	Verdict
`authoring.research.topics.ready` run_consumer.py:127	authoring-service-v2 `research_consumer.on_topics_ready` (`register.py:34`)	Topic picker / gate 1	live
`authoring.research.tree.ready` deep_consumer.py:174	authoring-service-v2 (`register.py:35`)	Outline confirm / gate 2	live
`authoring.research.run.failed` run_consumer.py:43 (shared by both consumers)	authoring-service-v2 (`register.py:36`)	Terminal failure	live
`authoring.research.progress` deep_consumer.py:30	authoring-service-v2 `research_consumer.on_progress` (`register.py:37`) — progress bar and stall heartbeat	Per-topic ticks	live
`authoring.usage.event` run_consumer.py:129; deep_consumer.py:109,132,176	authoring-service-v2 `usage_consumer` (`register.py:68`) on `AUTHORING_USAGE`	Credit metering	live

Background jobs

Job	Schedule	What it does	Verdict
`_connect_nats_with_retry` (`app/main.py:59`)	Once per pod start, retrying every 5 s until it succeeds	Connects, ensures the stream, binds both durables	live
`_one` (`app/pipeline/deep_research.py:423`)	Not scheduled — one task per selected topic	Researches one topic into a dossier; tolerated failure, re-raised rate limits	live

No cron and no CronJob: every unit of work arrives as a NATS message.

6. Data it owns

Stateless — no database, no alembic directory, no tables anywhere. ARCHITECTURE.md §3.4 lists research-worker under "none". Its durable output is the events in §5 plus S3 objects under a per-run prefix: topics.json, one dossier per topic, the outline tree, and the idempotency-keyed LLM call cache (app/services/s3_store.py, app/services/call_cache.py). The cache is load bearing: it is why a JetStream redelivery of a half-finished deep run resumes instead of re-spending on an hour of search calls.

Table	What it holds	Written by
—	No tables.	—

7. Dependencies

flowchart LR
  AV2[authoring-service-v2] -. "run.requested · deep.requested" .-> RW[research-worker]
  RW -. "topics.ready · tree.ready · run.failed · progress · usage" .-> AV2
  RW -- "search + synthesis calls" --> OR["OpenRouter (Sonar, Claude)"]
  RW -- "topics · dossiers · tree · call cache" --> S3[("S3")]

Zero outbound HTTP to other Oper services — ARCHITECTURE.md §3.2 lists research-worker as zero-outbound, and it talks only to authoring-service-v2 over NATS (§3.1).

8. Dead-code verdicts

Every entry point with no in-repo caller. Deleting is a separate decision — see the hub roll-up.

Entry point	Kind	Verdict	Evidence
`GET /live`	HTTP route	suspect	Searched this service's manifests for the path and every other service's client code for an HTTP call to research-worker: none. The liveness probe is deliberately `exec true` (`k8s/deployment.yaml:71-80`) after HTTP liveness SIGKILLed long runs in the sibling extraction worker. Not dead: it is the endpoint a future HTTP probe would use.
`GET /health`	HTTP route	suspect	Same search. No probe points at it and no service issues HTTP to research-worker (no `RESEARCH_WORKER_*` URL exists in `global-configs/k8s/global-configmap.yaml`). It remains the only endpoint that reports S3 reachability (`app/health.py:19-30`), so it is useful by hand during an incident.

Both consumers have a named authoring-v2 publisher and all five published subjects have a named authoring-v2 consumer, so no NATS entry point is dead.

9. Sources

No README, no docs/, no bug-hunt-reports/research-worker.md. Everything above is derived from code: app/main.py, app/consumers/run_consumer.py, app/consumers/deep_consumer.py, app/pipeline/discovery.py, app/pipeline/deep_research.py, app/pipeline/outline.py, app/health.py, app/config.py, app/schemas/events.py, app/services/.
k8s/deployment.yaml (probes, preStop, resources) and k8s/configmap.yaml (model map, topic caps, dossier floor, deep timeout)
ARCHITECTURE.md §3.1 (spoke topology), §3.2 (zero-outbound), §3.3 (AUTHORING_RESEARCH, the research→module flow), §3.4 (stateless), §3.6 (OpenRouter/Sonar + S3 egress)
authoring-v2 side of the contract: authoring-service-v2/app/services/research_orchestration.py, app/consumers/research_consumer.py, app/consumers/register.py:33-37
Scan output tools/feature-docs/out/research-worker.json