Meet our CEO →Smear reader: computer vision that finds malaria-infected blood cells in the browser
Type
Industry
Work involved
Data
Built
About the project
Smear reader is Orangesoft’s own computer vision research project. It takes a photo of a stained blood smear, finds every red blood cell, flags the ones infected with the malaria parasite Plasmodium falciparum, and computes parasitemia, the share of infected cells.
We built it on open data to work through the full path of a medical imaging project, from labelling rules to an honest test on hospitals the model never saw. Everything runs in the browser tab: the photo is never uploaded.
The challenge
A parasite is 10 to 15 pixels across on a cell about 110 pixels wide, inside a 5312×2988 photo. A model that reads such photos has to stay right when the lab, the camera and the stain change.
The training data came from one hospital in Chittagong, Bangladesh: 965 photos of thin smears from 193 patients, taken with a phone through the microscope eyepiece, with 196,934 cells labelled by an expert. 7,952 of them, 4%, are infected (Kassim et al., 2020; images courtesy of the U.S. National Library of Medicine).
Three goals shaped the project:
- Find every red cell and flag the infected ones on a full-resolution photo.
- Measure the model only on patients and hospitals it never saw in training, and show where it fails.
- Run in an ordinary browser, with nothing sent to a server.
Delivered solution
The first network finds the centre of every cell on a copy of the photo four times smaller. The second looks at each cell at full resolution and decides whether it is infected. The app draws the result on the smear, counts red and white cells, computes parasitemia and shows the expert’s own count for the same image next to it.
The models run with ONNX Runtime Web on WebGPU and fall back to WebAssembly when the device has no suitable GPU. In Chrome on a laptop with an Apple M5 Pro, a 5312×2988 photo takes 84 to 120 ms with WebGPU and 1.2 to 1.6 s on a single WebAssembly thread. On six test photos the browser gave the same counts as the Python reference, with one photo off by a single cell.
Training took 12 minutes for the detector and 3.5 minutes for the classifier, on a laptop.
Key features
One screen: the smear with every cell marked, the counts next to the expert’s, and a threshold the lab controls.
Every red cell, found and counted
The detector rings each red blood cell. On 28 held-out patients it found 97.1% of the cells the expert labelled, and 98.4% of its marks were real cells.
Infected cells flagged, with a loupe to check
Infected cells get a violet ring, and a loupe magnifies any cell with the parasite in view. On the same patients the model found 81.7% of infected cells, and 86.9% of its flags were right.
Parasitemia next to the expert’s count
For each photo the app shows red cells, infected cells and parasitemia, with the expert’s own count for the same photo. Per patient, the model’s parasitemia was 0.89 percentage points away from the expert’s on average, with a correlation of 0.98.
The lab sets the threshold
A slider sets the confidence above which a cell is flagged: the balance between a missed infection and a false alarm is the lab’s call, not the model’s. The default, 80%, was chosen on separate patients: training batches held 25% infected cells, while the smears in this dataset hold 4%.
A clean smear stays clean
On a smear from a patient without infection the model rings every cell and flags none. On the four healthy test patients, its parasitemia never went above 0.36%.
How the model works
Two small networks instead of one. Shrink the photo enough to find cells quickly and a parasite shrinks to 3 pixels, too small to see. So one network finds cells on a small copy, and the other checks each cell at full resolution.
The photo
A thin smear stained with Giemsa, photographed with a phone through the microscope eyepiece at 5312×2988 pixels.
Find the cells
The detector reads a copy four times smaller and lights up the centre of each red cell. It also measures the image scale itself, from the cell edges, so a new lab needs no settings.
Check each cell
The classifier looks at every cell at full resolution and gives its confidence that the cell is infected.
Decide and count
Cells above the threshold are flagged in violet, the rest in teal, and the app computes parasitemia for the photo.
Results on patients the model never saw
Trained in one hospital, tested in six others
The model learned from 136 patients at one hospital. Then, without any retraining, we ran it on 285 photos from six French hospital labs: other cameras, another stain protocol, cells from 0.7 to 2.3 times the size in pixels. Images from these labs are not shown, only the numbers.
| Lab | Images | Cells found | Infected cells found | Flags that were right |
|---|---|---|---|---|
| Chittagong, held-out patientsSame hospital as the training data | 140 | 97% | 82% | 87% |
| Lille | 40 | 99% | 96% | 47% |
| Montpellier | 40 | 96% | 91% | 38% |
| Nantes | 25 | 99% | 80% | 98% |
| Rouen | 40 | 96% | 90% | 76% |
| Saint-Louis, Paris | 40 | 97% | 50% | 5% |
| Toulouse | 40 | 99% | 96% | 96% |
Cells found: the share of the expert’s red cells the model found. Infected cells found: the share of the expert’s infected cells the model flagged. Flags that were right: the share of the model’s flags that the expert also marked infected.
Sample of the French test set: up to 40 P. falciparum photos per lab, spread across smears (Nantes has 25 in total), plus 60 photos of Montpellier smears without infection. Counted are cells fully inside the frame. One run.
French data: Guemas et al., Zenodo, CC BY 4.0.
What carried over, and what did not
Finding cells carries over between labs. Spotting infection carries over much less reliably than finding cells.
Cells: 96 to 99% found in every lab
Across all six labs the model found 96 to 99% of the red cells fully inside the frame, and 92 to 98% of its marks were real cells. The scale is measured from each photo, with no settings per lab.
Infection: uneven, with one lab failing
In five of the six labs, 38 to 98% of the cells it flagged were really infected. In the sixth, Saint-Louis, the sample held only 6 infected cells and the model raised 53 false flags, so 3 of its 56 flags were right.
Labelling rules differ between labs
At first the model seemed to miss 14 to 35% of cells in the French labs. Of those misses, 87 to 99% were cells cut by the frame edge: the French labels include them, the training labels do not. Without that check, a good detector is easy to call a bad one.
Healthy smears: 1.7% false flags
On 60 photos of Montpellier smears without infection, the model flagged 1.7% of red cells as infected.
Where it fails
The worst calls on held-out patients the model never trained on. Each crop is one cell, with the model’s confidence that it is infected.
Missed infections
Labelled infected by the expert, scored low by the model. Most come from smears stained pinker than the training set, or the parasite sits right on the cell edge.
<1% infected
<1% infected
<1% infected
<1% infected
1% infected
1% infected
1% infected
1% infected
2% infected
2% infected
False alarms
Labelled uninfected by the expert, scored high by the model. Several show a dot that looks like a parasite: one reader labelled the reference, and a single reader can miss them too. In practice such disagreements go to a second reader.
>99% infected
>99% infected
>99% infected
>99% infected
>99% infected
>99% infected
>99% infected
>99% infected
99% infected
99% infected
What we learned
What this project taught us applies to most medical imaging work.
A research model is not yet a product that can inform a diagnosis: see what medical device software development takes. For the wider field, read our overview of AI in medical imaging.
Split by patient, not by photo
Each patient has five photos of the same smear. Split photos at random and the model learns to recognise the smear, and the numbers come out too good. All our tests hold out whole patients.
The reference makes mistakes too
Among the most confident false alarms are cells with a dot that looks like a parasite. The reference was labelled by one expert, so measured precision is capped by the reference itself.
Stain colour is the main source of misses
Colour augmentation in training (channel gain, contrast, blur) covered most of the difference between labs, but not the pinkest smears. In the French labs, false flags gather where platelets lie on a red cell and where the stain shifts the colour.
Check the data before the model
Three NIH annotation files carry a typo in the frame height (29888 instead of 2988). Folder names carry patient codes and photos carry EXIF data, so every image in the app is re-encoded and renamed.
Data and privacy
Open data still comes with conditions. We checked each licence at the source and published only what those terms allow.
Nothing leaves the browser
The models run in the tab. The photo is never uploaded to a server.
No patient codes, no metadata
Images are re-encoded without EXIF data and renamed, so no patient code reaches the page.
French images stay unpublished
The French dataset’s paper has no statement on patient consent, so we publish numbers from those labs and never their images.
Licences checked at the source
Both datasets are used under their published terms and credited the way each one asks.
Technology stack
Training in PyTorch on a laptop, inference in the browser through ONNX Runtime Web, with WebGPU first and WebAssembly as the fallback.
Model training
Python
PyTorch
Apple Metal (MPS)
NumPy
SciPy
Pillow
Export and inference
ONNX
ONNX Runtime
ONNX Runtime Web
WebGPU
WebAssembly
Web app
JavaScript
HTML5
CSS
Let’s talk about your product
Tell us where your product is today and what comes next. Our team will reply and suggest the next step.
Talk to our teamWe're trusted by:
- 1. Intro callDuring a 30-minute meeting, our domain expert dives into your business and describes the steps for future collaboration.
- 2. Free discovery workshopTogether with you, our technical team defines the user flow, feature list, and project risks.
- 3. Project planningWe provide the implementation plan, timelines and estimations for your project.
