Skip to content
Back to data work

Case study 03 / 06

BioStreamer

My own peer-reviewed anaerobic digestion research, turned into a hybrid data platform.

BioStreamer models daily bioreactor telemetry (pH, volatile fatty acids, alkalinity and biogas yield) for 100 simulated reactor lines, grounded in the co-digestion study I co-authored. Structured data and research literature flow through separate pipelines and meet in one API.

View on GitHubRead the paper
BioStreamer's fleet overview: simulated yields for 36 reference reactors next to the paper's published Table 2
Type
Hybrid data platform
Grounded in
My 2024 co-authored paper
Scale
100 reactors × 37 days
Code
Public on GitHub
Maximum deviation from the paper's published yields
0.042 ml/day
Pipeline throughput on ~3,700 rows
~27,000 rows/s
Simulated bioreactor lines
100

01

The problem

A 36-digester anaerobic digestion study, observed daily for 37 days, needs more than 1,300 manual water-displacement readings. Each one is read off a calibrated cylinder and typed into a spreadsheet by hand.

That logging burden, not the number of digesters, is what limits how large a study a small lab can run. The study BioStreamer is built on is one I co-authored on co-digesting bean and plantain peels, published in GSC Biological and Pharmaceutical Sciences in 2024.

BioStreamer asks what that research looks like when collecting and interpreting the data is automated for a fleet an order of magnitude larger, while staying accountable to the published results.

02

What I built

BioStreamer simulates, stores and reasons over daily telemetry for 100 bioreactor lines: pH, volatile fatty acids, alkalinity and biogas yield. A retrieval layer over the research literature sits beside it, so numbers and explanations both come from verified sources, never from a language model's memory.

  • 01

    Mechanistic simulator

    A coupled acidogenesis and methanogenesis model generates 37 days of telemetry for 100 reactors, across a continuous range of bean to plantain ratios at 5 to 25% total solids.

  • 02

    Two ingestion tracks

    Telemetry lands as Parquet in a MinIO lake and loads into PostgreSQL. Research text is chunked, embedded and indexed in Qdrant.

  • 03

    Validation gates

    Airflow refuses to load a dataset unless the 36 reference reactors reproduce the paper's published yields, and won't mark indexing complete unless retrieval probes return the right source type.

  • 04

    Hybrid answers

    A FastAPI /chat endpoint combines exact SQL aggregates with ranked literature passages, then asks the language model to compose an answer from both.

  • 05

    Traceable claims

    Every answer ships with its retrieval trace. Each passage is tagged as a published finding from the paper or as general domain context, so a claim can't be mistaken for a result the study didn't report.

  • 06

    Streamlit dashboard

    A fleet overview, a reactor explorer and a research chat that shows its retrieval trace.

03

How it works

Architecture

Hover a component to trace its connections.

BioStreamer architecture. Apache Airflow orchestrates Telemetry simulator. Apache Airflow orchestrates Literature corpus. Telemetry simulator feeds MinIO lake (Parquet). MinIO lake feeds PostgreSQL (load). Literature corpus feeds Chunk and embed (split). Chunk and embed feeds Qdrant (upsert). PostgreSQL feeds FastAPI hybrid RAG (SQL). Qdrant feeds FastAPI hybrid RAG (vectors). FastAPI hybrid RAG feeds Streamlit UI (/chat).
  1. 1

    Simulate

    A seeded simulator produces 100 reactors × 37 days of telemetry. A mechanistic pass sets the shape of each reactor's daily output, then a second pass calibrates the 36 reference reactors to the paper's Table 2 means.

  2. 2

    Validate, then load

    Airflow compares the reference cohort with all twelve cells of Table 2. Only if it passes does the data land in MinIO as Parquet and load into PostgreSQL.

  3. 3

    Index the literature

    A second DAG splits the paper and domain notes into chunks of 512 with an overlap of 64, embeds them as 384-dimension vectors and indexes them in Qdrant.

  4. 4

    Answer

    Every /chat question runs both retrievals: SQL for exact numbers, vector search for mechanism. The model composes the answer from that labelled context instead of doing the arithmetic itself.

  5. 5

    Explore

    Streamlit shows the fleet, any single reactor and the research chat. With no language model key set, /chat still returns the retrieved passages and aggregates.

04

Tech stack and why

  • PostgreSQL 15

    Reactor config and daily telemetry, with CHECK constraints that encode the scientific envelope.

  • Qdrant

    The literature's vector store: cosine distance over 384-dimension embeddings.

  • MinIO

    An S3-compatible landing zone for raw telemetry Parquet, the bronze layer.

  • Apache Airflow 2.8

    Two DAGs, each with a validation task in its critical path.

  • all-MiniLM-L6-v2

    An embedding model that runs on a CPU, at roughly 16 ms per query once loaded.

  • FastAPI

    Structured endpoints plus the hybrid /chat route.

  • Anthropic Claude

    Synthesis only. It never sees a question without grounding, and the platform works without it.

  • Streamlit and Plotly

    The fleet overview, reactor explorer and research chat.

  • NumPy and pandas

    The kinetic model, with a modified Gompertz envelope.

05

Challenges and what I learned

  • Hybrid, not vector-only

    A vector store can find a passage that discusses yields, but it can't compute a mean over 3,700 telemetry rows. SQL returns exact numbers with no explanation. Running both and labelling the context lets the model explain without doing the arithmetic.

  • Failing closed

    A pipeline that quietly drifts from the ground truth is worse than one that stops. Both DAGs fail closed: no load unless the reference cohort matches the paper, and no finished index unless retrieval returns the right provenance.

  • Being precise about what matches

    The 36 reference reactors match the paper's yields largely by construction, because they are calibrated to it. The leftover gap, at most 0.042 ml/day across twelve cells, measures calibration precision, not predictive accuracy. What the model decides on its own is which reactors sour and when: 361 soured reactor-days above roughly 20% total solids, and none inside the published range.

  • Degrading gracefully

    In testing, an exhausted API key made the language model call fail. /chat reported the billing error and still returned the full retrieval context, instead of failing the request.

What I learned

  • Ground truth belongs inside the pipeline. A validation gate in the DAG catches drift that a one-off check would miss.
  • Be precise about what a number proves. Calibration precision and predictive accuracy are different claims, and the page should say which one it's making.

Have something like this in mind?

Have a project, role or research opportunity in mind? Let's talk.

Book a callGet in touch