Clinical NLP NER

The problem

The clinically useful information in a patient chart — diseases, medications, procedures — is buried in free-text notes, and pulling it out by hand takes 15–20 minutes per patient and is error-prone. At the scale of a research cohort or a coding department, that manual extraction is the bottleneck that blocks cohort discovery, billing, and quality reporting from ever operating on structured data.

Constraints

• Vocabulary. Clinical text is full of terms — "meningioma," "metformin," "cholecystectomy" — that general NLP models have never seen and shred into meaningless subwords. The model has to understand medical language, not just English.
• Data. Annotated clinical corpora are small and access-controlled. I built on n2c2_2018_track2 (clinical notes with medical-entity annotations) with bc5cdr as a biomedical fallback — real annotations, but limited in volume and domain coverage.
• Document length. Clinical notes routinely exceed 1,000 tokens, well past a transformer's 512-token window, so entities near boundaries get lost unless you handle length explicitly.
• Stakeholder / risk. For anything downstream of this (decision support, billing), a wrong extraction is worse than a missing one. The pipeline needed a way to defer uncertain calls to a human rather than silently emit them.

Approach

First attempt — an off-the-shelf general NER baseline. I started with SpaCy's general-purpose NER to establish a floor. It landed at ~0.60 F1 and failed exactly where expected: it didn't recognize drug names or procedure phrases, because its training data contains almost no clinical language. Useful only as a baseline.

Second attempt — a domain-pretrained transformer. I moved to BioBERT (dmis-lab/biobert-base-cased-v1.2), pretrained on 18B tokens of PubMed and PMC text. Because it already understands biomedical vocabulary, fine-tuning can spend its budget learning the NER task rather than the words. I framed it as token-level BIO classification (a linear head over the token embeddings) and fine-tuned on n2c2_2018_track2:

BioBERT-base (110M params) → Token Classification Head
  Learning rate: 2e-5 (linear warmup over 10% of steps)
  Batch size: 16 · Epochs: 5 · Max seq length: 512
  Optimizer: AdamW (weight decay 0.01)

Handling length and uncertainty. For notes over 512 tokens I used a sliding window with overlap so entities near a boundary appear whole in at least one window. The inference pipeline emits a per-entity confidence score, and a 0.7 threshold routes low-confidence spans to human review instead of auto-extracting them.

I also tested SciBERT (allenai/scibert_scivocab_cased) as an alternative domain model.

Evaluation

The evaluation is where this project earns its keep, so I looked at it three ways: against baselines, per entity type, and against what the number hides.

Against baselines.

The BioBERT result is a +0.22 F1 jump over the general baseline, and the SciBERT comparison is the more interesting one: BioBERT beat SciBERT despite both being domain-pretrained, because BioBERT's pretraining corpus (clinical/biomedical prose) is simply closer to the target text than SciBERT's general-scientific vocabulary. The lesson is that which domain you pretrain on matters, not just that you did.

Per entity — the errors aren't uniform.

Procedures are the weak class, and for a structural reason: they're long, variable phrases ("underwent a left anterior descending coronary artery stent placement") that span many tokens, so a single boundary error tanks the span. Medications, which are short and distinctive, are the strongest.

What the metric doesn't capture. Macro-F1 on n2c2 is measured on a curated benchmark with consistent annotation — real EHR notes are messier: typos, abbreviations, copy-forward boilerplate, institution-specific shorthand. The score also treats every entity as equally important, when clinically a missed medication is far more dangerous than a missed procedure. And token-level F1 rewards getting most of a span right; it doesn't tell you whether a partially-extracted drug dose ("500" without "mg") is safe to pass downstream — which, for anything touching prescribing, is the question that actually matters.

What I'd do differently

• Weight evaluation by clinical risk. A flat macro-F1 hides the asymmetry that matters. I'd report entity-type F1 with an explicit cost model — missing a medication or dose should count for more than missing a procedure — so the metric reflects patient risk, not just token accuracy.
• Span-level and normalization metrics, not just token BIO. Token F1 overstates real usefulness. I'd add strict span-match scoring and, more importantly, entity normalization (mapping "metformin 500mg" to a structured drug + dose), since the downstream systems need normalized values, not raw text spans.
• Stress-test on out-of-distribution notes. Everything here is n2c2 in-distribution. I'd assemble a small set of notes from a different institution or specialty and measure the drop — that gap, not the benchmark number, is what tells you whether this survives contact with a real hospital.