NLP

This document aims to provide help in configuring natural language processing (NLP) workflows as part of a Cumulus Library study.

Use cases

The coded metadata in FHIR inevitably only tells part of the story. There will always be some interesting extra bits in the clinical notes (which aren’t easily exposed in the core tables for study use). In those cases, you’ll want to define some NLP workflows that can run a custom prompt and clinical notes through an LLM.

The results of that prompt will be parsed into a structured format and written back into an Athena table for the rest of your study to reference.

As an example, you might be working on a study related to infectious diseases. You could write an NLP workflows with a prompt like “Analyze the following clinical note and indicate if the patient has a flu.” (That’s an overly simple prompt, but prompt engineering is outside the scope of this guide.)

Then you could capture the results of that prompt as "has_flu": True/False, which will get stored in an Athena table of your choosing (along with some metadata about the results).

And then the rest of your study SQL can reference those results to further categorize / process patients.

Configuring an NLP Workflow

The config you reference in your study manifest is expected to contain a number of field definitions. We strongly recommend starting from the below template, which contains details on the expectations of each value.

# This is a config file for defining one or more NLP tasks that will create tables in your study.

# config_type should be "nlp" - we use this to distinguish from other configurable builders
config_type="nlp"

# You define NLP tasks/tables with the 'tables' dictionary.
# The keys in the table dictionary define your table names, but the final table name also
# includes the study prefix from your manifest, an `nlp` marker, and the model you ran with.
# So an entry like [tables.my_table], run against the gpt-oss-120b model, results in a table
# like `my_study__nlp_my_table_gpt_oss_120b`.
# See "Result Table Names" below for more details.
[tables.table_1]

# `version` will be a number that you will increment every time you change the table definition
# in a clinically-relevant way. i.e. if you change the prompt or the response schema, you should
# bump the version. This version is used as part of the NLP caching layer, so if you don't bump it,
# you will get incorrect results from the cache.
version = 0

# `system_prompt` defines the system prompt to the model. System prompts are where you put your
# main instructions to the model. The system prompt will be the same for all the notes.
# If you put `%JSON-SCHEMA%` somewhere in the prompt, it will be replaced by a JSON schema of your
# expected response (see below). This is recommended, to help some models return the right format.
system_prompt = """
You are a clinical chart reviewer. Your task is to blah blah blah...

Pydantic Schema:
%JSON-SCHEMA%
"""

# `user_prompt` defines the user prompt to the model. This is where you might put the data to be
# processed.
# If you put `%CLINICAL-NOTE%` somewhere in the prompt, it will be replaced by the current clinical
# note. If you do not define this field or leave it empty, just the text of the clinical note will
# be used (i.e. the default is `"%CLINICAL-NOTE%"`).
user_prompt = ""

# `response_schema` points at a file relative to this workflow file. It contains a JSON schema
# for the expected response from the LLM. If the LLM's response cannot be parsed into this format,
# it will be ignored. This schema also helps defines the resulting table schema.
# Often, studies will use a bit of Python code to generate this file from some Pydantic models.
# See the example_nlp study or other studies for comparison.
response_schema = "schema.json"

# `select_by_word` allows you to define words that will cause notes to be included for NLP
# processing. If no selection rules are defined, all notes are selected by default. But once a
# selection rule is defined, a note that matches any of them will be included.
# Words are matched at word boundaries (e.g. "fever" will not match "fevers").
# Words can include whitespace.
select_by_word = ["fever", "severe cold"]

# `select_by_regex` allows you to define regular expressions that will cause notes to be included
# for NLP processing, just like `select_by_word`. Again, regexes will match at word boundaries.
# The regex rules are standard Python rules: https://docs.python.org/3/library/re.html
select_by_regex: ["fevers?"]

# `select_by_table` allows you to define a table name that will be searched for note identifiers,
# to choose which notes are selected for NLP processing. Column names that will be found include
# `note_ref`, `documentreference_id`, `documentreference_ref`, `diagnosticreport_id`, and
# `diagnosticreport_ref`. This is useful if your study wants to calculate which notes are most
# interesting to examine for NLP, then creates a table that holds such references.
select_by_table: "my_study__interesting_notes"

# `reject_by_word` allows you to define words that cause notes to **not** be included for NLP
# processing, even if selected by one of the above selection options.
reject_by_word: ["kidney", "fake"]

# `reject_by_regex` allows you to define regular expressions that will cause notes to **not** be
# included for NLP processing, even if selected by one of the above selection options.
reject_by_regex: ["kidneys?"]

# The `shared` dictionary allows you to share configuration between multiple tables.
# For example, you may use the same selection/rejection criteria for multiple related NLP tables.
# Or the same prompts, just with different schemas. Using the `shared` dictionary greatly reduces
# your configuration burden.
[shared]

# The following fields can be defined here, and will be used if a table does not define its own
# value for it. (i.e. these are fallback/default values)
system_prompt: "Default system prompt"
user_prompt: "Default user prompt"
select_by_word: ["default", "words"]
select_by_regex: ["default", "regex"]
select_by_table: "default_table"
reject_by_word: ["default", "words"]
reject_by_regex: ["default", regex"]

Result Table Names

NLP results are named the same way that Cumulus ETL names the NLP tables it creates, so that all NLP results in your database look alike:

{study_prefix}__nlp_{table_name}_{model}

For example, a [tables.treatment] entry in the ibd study, run with --nlp-model=gpt-oss-120b, creates the table ibd__nlp_treatment_gpt_oss_120b. (Hyphens in a model name become underscores, since the model name is part of a SQL table name.)

A few consequences worth knowing about:

• The model is part of the table name, so results from two different models don’t overwrite each other, and any study SQL that reads NLP results needs to name the model it expects.
• The task version is not part of the table name (it is available as the task_version column). Bumping a task version writes results to a new folder and re-points the table at it.
• nlp_ is a reserved table prefix, which means these tables are not dropped when a study is cleaned or rebuilt. That’s deliberate - NLP results are expensive to regenerate.

The parquet files behind these tables get uploaded to your Athena results bucket, in a folder named like the table plus the task version: cumulus_user_uploads/{database}/{study_prefix}/nlp_{table_name}_{model}_v{version}/

Result Table Format

Tables created by the NLP workflow will have the following fields:

• note_ref: a string like DocumentReference/abc
• encounter_ref: a string like Encounter/abc
• subject_ref: a string like Patient/abc
• generated_on: a string with the time-of-result-generation in UTC
• task_version: a number with the version field from the NLP table config
• model: a string with the name of the model used
• system_fingerprint: some LLMs provide a fingerprint, which can help track server-side changes
• result: a struct, with its shape defined by the response_schema field

Running an NLP Workflow

NLP workflows require extra configuration that normal study workflows do not. Namely, how to connect to the LLM of choice. You can pass these to Cumulus Library when building a study and any NLP workflows will use them.

• --note-dir=PATH: point this at the root folder of your FHIR NDJSON note documents
• --etl-phi-dir=PATH: point this at the PHI folder that you use for Cumulus ETL (the third argument when running the ETL process) - NLP caches are kept here as well as the information needed to compare anonymized IDs with the original note IDs
• --nlp-model=MODEL: choose a model to use for this run; pass help to get a list of options
• --nlp-provider=PROVIDER: choose a provider to use for this run; can be azure or bedrock but defaults to local (a locally run NLP)
  • If using azure, you also need to set the AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT environment variables.
  • If using bedrock, you need to make sure that your AWS configuration can be found (probably by setting the AWS_PROFILE environment variable).
  • If using local, see below for instructions on using Docker to run local LLMs
• --azure-deployment=NAME: when using the Azure provider, you may need to provide a deployment name (defaults to model name). Pass it more than once to spread requests across several deployments, which multiplies the quota available to a run (see below)
• --nlp-concurrency=N: how many NLP requests to keep in flight at once (defaults to one per deployment, so a single endpoint runs serially unless you ask for more). Workers are spread across your deployments as evenly as possible.
• --batch-nlp: if set, NLP will be done in batch mode, which can take up to a day to finish, but will be much cheaper
• --clean-nlp: if set, previous NLP results for the workflow will be deleted first
• --no-nlp-stats: if set, note and token stats will not be printed to the console

Some further arguments are only useful while you are writing a study, and are hidden behind dev mode. See Study Development Mode below.

Going Faster

By default, one note is processed at a time. Since the run spends almost all of its time waiting on the model to answer, things go slow. Here’s how we can move a bit faster!

--nlp-concurrency sets how many requests are in flight at once. It is a hard ceiling, not a target: the run never has more requests outstanding than you allow, so raising it cannot turn into a stampede against your server.

Which number is safe depends on where you’re sending the requests:

• Local (--nlp-provider=local): concurrency is close to a pure win. One deployment, but try the concurrency knob.
• Azure: quotas are per-deployment, so concurrency is bounded by deployment. If you have several deployments, pass --azure-deployment once per deployment and the run will spread its workers across all of them - which raises total throughput without raising the pressure on any one endpoint.
• Bedrock: throttling is account- and region-wide. Start small and watch for warnings.

A reasonable way to tune: start at 2, and if a run reports no rate limiting, go higher. If you can, add another deployment.

For example, three Azure deployments with two workers each:

cumulus-library build --target my_study \
  --nlp-model gpt-oss-120b --nlp-provider azure \
  --azure-deployment deploy-east \
  --azure-deployment deploy-west \
  --azure-deployment deploy-central \
  --nlp-concurrency 6

When You Do Get Rate Limited

Requests that come back rate limited are retried automatically, and the endpoint that throttled you is put in a brief cooldown that every worker respects.

If a note is still being rate limited after several attempts, it’s dropped and the run keeps going. Re-running picks them up: every note that did succeed is cached in your PHI dir, so a second pass only pays for what’s missing.

Note that concurrency does not apply to --batch-nlp, which is already a bulk API. Batch mode also only supports a single --azure-deployment.

Study Development Mode

Some NLP arguments only make sense while you are iterating on a study - choosing a prompt, tuning a schema, comparing two models. They are not part of the normal “build this study” workflow, so they are kept behind a --dev flag.

Pass --dev to unlock them. Without it, they aren’t merely hidden, they aren’t accepted at all, so a typo fails loudly instead of being quietly ignored:

cumulus-library build --target my_study --dev --nlp-subtask=age ...

To see everything dev mode adds, ask for help while in it:

cumulus-library build --dev --help

The arguments dev mode unlocks:

• --nlp-subtask=TABLE: only build this table from the workflow, instead of all of them. Pass it more than once to build a subset of tables (e.g. --nlp-subtask=age --nlp-subtask=race). This lets you build individual NLP tables in isolation without editing the workflow file. If a name isn’t found in the workflow, the build stops and lists the available tables.
• --select-by-table=TABLE: override the workflow’s select_by_table value, for testing. Can only be used together with --nlp-subtask.
• The MLflow arguments described below.

Tracking Experiments With MLflow

When you are iterating on a prompt, the question is usually “was that change an improvement?” Cumulus Library can record each NLP run to MLflow so you can answer that by comparing runs.

Setting Up a Tracking Server

MLflow is an optional dependency, since it is only useful for study development:

pip install 'cumulus-library[mlflow]'

That gives you the mlflow command, which includes a tracking server with a web UI. Start one:

mlflow server

It listens on http://127.0.0.1:5000 by default. Open that in a browser and you’ll get the MLflow UI, which is empty until your first tracked build. (mlflow ui is an alias for the same command.) Leave it running in its own terminal - Cumulus Library checks the connection before it processes any notes, so the server needs to be up before you start a build.

By default the server keeps its data in a SQLite file named mlflow.db, created in whatever directory you launched it from. That’s convenient but easy to litter with, so either run it from a scratch directory or point it somewhere deliberate:

mkdir -p /path/to/cumulus-mlflow
mlflow server --backend-store-uri sqlite:////path/to/cumulus-mlflow/mlflow.db

If a ./mlruns directory already exists where you start the server, MLflow uses that instead of SQLite.

Then tell Cumulus Library where the server is, either with --mlflow-uri on each build or once in your environment:

export MLFLOW_TRACKING_URI=http://localhost:5000

A server started this way runs on your machine and writes to your local disk, so tracking a run keeps everything local. If you point --mlflow-uri at a shared server, everything described below leaves your machine. See What Data Gets Sent Where.

Tracking a Build

With a server running, point a build at it:

cumulus-library build --target my_study --dev \
  --mlflow --mlflow-uri http://localhost:5000 \
  --mlflow-experiment my-prompt-tuning \
  --mlflow-tag phase=pilot \
  --nlp-model gpt-oss-120b --nlp-provider azure \
  --note-dir /path/to/notes --etl-phi-dir /path/to/phi

• --mlflow: turn on tracking. Nothing MLflow-related happens without it.
• --mlflow-uri=URI: your tracking server. Defaults to the MLFLOW_TRACKING_URI environment variable. The connection is checked before any notes are processed, so a bad URI fails immediately rather than after you have paid for a full pass.
• --mlflow-experiment=NAME: which experiment to log to. Defaults to the study name, so runs for a study land together without you having to pass anything.
• --mlflow-run-name=NAME: a base name for the runs. The table name is appended so each run stays distinguishable. Defaults to the table name, version, and model.
• --mlflow-tag=KEY=VALUE: tag every run. Pass it more than once for several tags.

One run per table. Each table in your workflow gets its own MLflow run, because that is the unit you actually compare - a prompt change to age shouldn’t move race’s numbers.

Each run records:

• Params: the study, table, task version, model, provider, concurrency, batch mode, and the note selection rules. Prompts and the response schema are stored as SHA-256 digests, so you can group runs by “same prompt” without diffing the text.
• Metrics: notes.* counts and yield rate, tokens.* usage and cache hit rate, cost.estimated_usd, and runtime.*.
• Artifacts: the full system prompt, user prompt, and response schema.

The metric names deliberately match the ones Cumulus ETL logs, so runs from both projects can sit in one experiment and be compared directly.

A few numbers are workflow-wide rather than per-table, and are logged with the same value on every run. Cumulus Library makes a single pass over your notes and serves every table from it, so notes.seen and notes.with_text are the same for each table by construction, runtime.* covers the whole interleaved pass rather than one table’s share of it, and workflow.throttle_dropped is counted before a note is attributed to a table. Token counts and cost are genuinely per-table.

Because results are cached in your PHI dir, re-running a workflow you have already run will show a run with results but zero tokens and zero cost. That is accurate: cached notes are free.

What Data Gets Sent Where

Naturally, NLP workflows deal with a lot of PHI since they work directly with clinical notes. Let’s look at how the data flows through the system.

1. The workflow sends the prompts and clinical note text to the model you specify.
2. Each note’s result is cached in the PHI folder (specified with --etl-phi-dir).
3. Any text span fragments that the model gives back are turned into text offsets (numbers) instead of actual clinical note fragments.
4. Results are packaged together and uploaded to the S3 bucket associated with the Athena workgroup (this is the same S3 bucket that Athena query results get stored and the same bucket that file_upload workflows use).
5. An Athena table is created that points at those parquet files in S3.

If you are using MLflow experiment tracking, there is one more destination to account for. By default, tracking sends only counts, costs, timings, your prompts, and your response schema - no note text. This should not include PHI, but configurations of MLflow could reasonably send PHI over the wire to that server.

Examining Results Before Sending to the Cloud

You may have requirements around inspecting data before uploading files to S3/Athena. You can first do an NLP run into a local duckdb database to inspect the resulting parquet files yourself. And then once satisfied, upload to Athena.

1. Start by passing arguments like --db-type duckdb --database ./testing.db instead of the usual AWS/Athena arguments.
   • NLP parquet files will be written to a user cache folder. On Linux, this will be somewhere like ~/.cache/cumulus-library/nlp/{my_study}/nlp_{table}_{model}_v0/
   • The tables themselves (that point to those parquet files) will be in the database path you gave Cumulus Library (i.e. ./testing.db).
2. You can inspect the parquet files with a tool like parquet-tools.
   • Call parquet-tools show ~/path/to/parquet/files/* to see a dump of their contents.
3. You can inspect the resulting database with a tool like duckdb-cli.
   • Call duckdb -ui ./testing.db and browse the tables in your browser.
4. If everything looks good, you can now rerun the NLP workflow but instead of --db-type duckdb, you can pass all the normal AWS Athena arguments. The existing cache of NLP results sitting in the ETL PHI dir will prevent this second run from actually consuming LLM tokens.

Batching

Some providers support NLP batching, which lets you bundle up a lot of requests at once and send them in one go to the LLM server at a discount. Then you wait up to day for the results. So you trade predictable timing for 50% cheaper tokens.

As of this writing, Cumulus Library only supports batching with the Azure provider. Make sure you’ve set up a deployment that uses batching, and then pass in the following arguments: --azure-deployment=xxx --batch-nlp.

Then be prepared to wait a little bit. Note that the 50% discount is mitigated somewhat by the fact that batching does not use token caching at all. So some of that discount is lost. But overall, you’ll probably save money.

Using a Local LLM

For cost, reproducibility, or PHI-control reasons, you may prefer a locally-run LLM instead of a cloud LLM.

We ship a convenient Docker compose file that makes it easy to launch a local LLM yourself.

You’ll find the docs/compose-nlp.yaml file (referenced in the below commands) in cumulus-library’s git files, so make sure you have a local checkout of that.

gpt-oss-120b

Run the following command on a machine with at least 80GB of GPU memory.

docker compose -f docs/compose-nlp.yaml up gpt-oss-120b --wait

llama4-scout

Llama4 local Docker support is offered as an experimental work-in-progress. The commands below may not work.

Llama4 is a gated model, so it can’t simply be downloaded at will. You’ll need a Hugging Face account and have approval to access the llama4 model.

Then, go to the “Access Tokens” section of your Hugging Face account settings and make a read-only access token, to use below so that Docker can download the model.

Then, you can run the following command on a machine with at least 80GB of GPU memory.

export HUGGING_FACE_HUB_TOKEN=xxx
docker compose -f docs/compose-nlp.yaml up llama4-scout --wait