Julio Bermudez

The problem

Physicians spend a large portion of their day on documentation rather than patient care. A general surgery resident I know described the daily grind of hand-writing the same structured notes after every consultation. I wanted to build something that let him speak naturally and get back a properly formatted clinical document that followed the conventions his hospital actually uses.

How it was built

Coming from clinical laboratory science, I could make correct domain decisions instead of guessing at them. The data model reflects how patients, consultations, and clinical documents actually relate, and the generated documents follow real hospital conventions. The medical history, for instance, encodes a fixed set of physical-exam systems with canonical normal descriptions that fill in only when a system isn't mentioned, and abnormal findings that override the default. Those conventions came directly from the surgeon.

The most important design decision was drawing a hard line in the AI prompts between safe documentation inference (filling in standard normal exam findings) and unsafe clinical fabrication.

On the engineering side: browser audio capture with real-time waveform visualization (Web Audio API on canvas), a two-step upload-then-process flow for resilience on slow connections, and a single multimodal Gemini call that returns transcript and structured document together to halve API usage against a rate-limited tier. Full-stack TypeScript throughout — React, Node/Express, Prisma/PostgreSQL, JWT auth with strict per-doctor data scoping, Cloudinary for audio, deployed on Render and Supabase.

The name

It's named after Milo, my dog.

The problem

Reading a blood smear means sitting at a microscope and counting cells by eye, field after field. MiloDetects lets you photograph a microscope field, then detects and localizes the cells in it (white cells, red cells, platelets), draws labeled boxes over your own image, and reports per-field counts. It is deliberately assistive: it tells you where the cells are and what type they are. It does not perform a differential (for now).

How it was built

The backend has one function for inference: predict(image_bytes) returns a list of detections. It uses either a mock engine that makes up plausible results or the real one that runs YOLO. An environment variable picks which loads. I wrote the mock first and built the whole app on top of it: capture, batching, the box overlay, history. The model didn't exist yet. When the weights were ready I flipped the variable and nothing else changed.

Detections use normalized center-xywh so the model, the database, the API schema, and the canvas overlay all speak one language, avoiding bugs where boxes render fifteen pixels off. The model itself is a YOLOv8n fine-tuned on BCCD, a public dataset of 255 training images, transfer-learned from COCO weights: 0.919 mAP50 overall, 0.983 on white cells. It runs in-process behind FastAPI, offloaded to a thread pool so it doesn't block the event loop, with torch imported lazily to keep cold starts cheap.

What happened in the field

By every number I had, it worked. Then a bioanalyst friend held his phone to a microscope eyepiece in an actual lab and ran a real slide through it. It returned zero detections, on a field with cells plainly visible. Two reasons: 1) BCCD is built from 100× oil immersion images and he was working at 40×, so the model had never seen a cell at that scale. 2) BCCD images are clean and evenly lit, while a phone against an eyepiece produces glare, vignetting, a color cast from the lamp, and a hard dark circle where the field ends.

The differential is routinely read at 40×. Oil immersion is for particular cell morphology and inclusions. So a well-constructed, widely-used public dataset is built at a magnification most real users aren't working at, which resulted in the model being trained for a different problem.

What's next

v1 is deployed and does what it claims on images resembling what it was trained on, knowing the current limitations. v2 is being built around what the field test taught: a 40× dataset shot in real conditions with a written capture protocol and a clinical collaborator, since no public dataset for this exists and I'm qualified to annotate it correctly. After that, WBC differentiation, a real differential needs five subtypes, so the detector finds where the cells are and a second-stage classifier decides which kind. A differential is an actual diagnostic output.

The name

Also named after Milo, my dog.