Starting in the ward, not with the model
The problem was not simply speech recognition. Clinicians were balancing patient care with paper records, noisy rooms, code-switched language, and infrastructure that could not be assumed to stay perfect.
SHIPPED HEALTH AI · GATES FOUNDATION
Awaaz-e-Sehat
Paper charting was consuming time an understaffed maternity hospital did not have. I shipped the React Native app end to end, frontend and backend, working with doctors to design a voice-first workflow for noisy, code-switched Urdu and Punjabi speech, then carried it through a seven-month deployment.
7-month live hospital deployment
Speech error rate · 46% → 13%
Structured records · 96% field accuracy
Risk detection · 7% → 40%
Teaching the system to hear the real environment
I fine-tuned Whisper with LoRA on Urdu and Punjabi clinical audio, then paired transcription with structured extraction so speech could become a usable medical record rather than another block of text.
Turning documentation into an earlier warning
The same record could do more than archive a visit. I built a retrieval service that matched symptoms against indexed clinical guidance and reranked the evidence before presenting a possible risk to a clinician.
The part after the demo
I built and shipped the backend myself: Bun/Hono and FastAPI microservices on Cloud Run, a PostgreSQL schema and data models designed around the clinical record rather than bolted on after, pooled connections for the hospital's flaky network, and the cloud stack defined as code with Pulumi so the team could change the system without turning every release into an event.