LLM systems for domains where being wrong has consequences
I'm Colin Peters, an AI engineer who has spent the last four years building LLM and agentic systems in healthcare. My work covers clinical decision support and autonomous patient intake, with agent orchestration, retrieval grounding, evals and guardrails, and HIPAA-compliant deployment on AWS and Azure.
- years in healthcare software & ML
- 7+
- years in healthcare software & ML
- hospitals & clinics deployed
- 50+
- hospitals & clinics deployed
- physicians using my CDS work
- 1,000+
- physicians using my CDS work
- clinical encounters handled
- 10,000+
- clinical encounters handled
Case studies
Production LLM and ML systems in healthcare: the problem, what I built, and what changed.
AdviNOW Medical
- LLM intake pipeline (LangChain, OpenAI + Anthropic) with NLP-driven symptom extraction
- Stateful LangGraph workflows with branching clinical paths, conditional tool calls and multi-turn context
- Schema-validated output (ranked differentials, recommended labs, medication suggestions) consumed directly by the frontend
- RAG over proprietary medical knowledge (Azure AI Search, pgvector) with Langfuse evals and HIPAA audit logging
30–50% shorter average intake time across 10,000+ clinical encounters.
AdviNOW Medical
- LiveKit telemedicine platform with AI voice agents (STT → LLM → TTS) for hands-free pre-consultation intake
- IoT medical-device streaming into active consultations
- Remote Medical Volunteer System pairing video with LangGraph triage agents
Deployed across 50+ hospitals and clinics.
AdviNOW Medical
- Benchmarked foundation models across AWS Bedrock, Azure OpenAI, Azure AI Foundry and Vertex AI on latency, clinical accuracy and cost
- Routed routine extraction to smaller models and reserved frontier models for diagnostic reasoning
- SageMaker MLOps to fine-tune, version, A/B test and monitor clinical NLP models
Lower inference spend with frontier-level quality where it matters.
Glass Health
- Core NLP and prompt framework for differential diagnosis, SOAP notes, ICD-10 suggestion and treatment planning
- Multi-step reasoning chains: case decomposition → retrieval over clinical references → structured assessment
- Eval tooling that benchmarked every prompt change against clinical-accuracy metrics, validated with licensed physicians
- FastAPI inference microservice streaming token-by-token to the physician UI
Higher diagnostic completeness on the internal case library, in production for 1,000+ physicians.
Independent consulting
- Retrieval-grounding checks and structured-output validation
- Regression suites over curated case sets, run on every prompt or model change
- Langfuse observability across prompt versions, token cost and latency
Prompt and model changes are measured before release instead of taken on trust.
Belun Technology
- Real-time ingestion of SpO2, HRV, accelerometry and sleep staging
- Anomaly detection flagging oxygen-desaturation and arrhythmia events
- Fault-tolerant AWS pipelines with buffering and deduplication; React clinician dashboards
10,000+ patients monitored across countries; contributed to FDA and international clearance.
Contract engagements
Where I've worked
7+ years shipping production software and ML in healthcare. The last four have been focused on LLM and agentic systems.
Independent Consultant
Senior AI / Full-Stack Engineer
Nov 2025 – Present · Remote
- Advise healthcare and enterprise clients on LLM architecture, from retrieval design and agent orchestration through HIPAA-compliant deployment on AWS and Azure.
- Ship production agentic workflows on LangGraph and LlamaIndex with function calling and MCP tool use, routing requests across OpenAI, Anthropic and Google models to balance latency, cost and accuracy.
- Build the eval harnesses and guardrails clients are missing (grounding checks, structured-output validation, regression suites), so a model change can be shown to improve results.
AdviNOW Medical
Senior AI / Full-Stack Engineer
Dec 2023 – Oct 2025 · Scottsdale, AZ
- Built the patient-intake LLM pipeline behind 10,000+ clinical encounters, cutting average intake time 30–50%.
- Engineered stateful LangGraph agents emitting ranked differentials, labs and medication suggestions as schema-validated output.
- Architected a LiveKit telemedicine platform with AI voice intake, deployed across 50+ hospitals and clinics.
- Benchmarked models across Bedrock, Azure OpenAI, AI Foundry and Vertex AI, and reduced inference spend by routing routine extraction to smaller models.
Glass Health
Full-Stack Engineer
Sep 2021 – Nov 2023 · San Francisco, CA
- Built the NLP and prompt-engineering framework behind differential diagnosis, SOAP notes, ICD-10 suggestion and treatment planning for 1,000+ physicians.
- Developed eval tooling benchmarking every prompt change against clinical-accuracy metrics, validated with licensed physicians.
- Designed a FastAPI inference microservice streaming token-by-token to the physician UI.
Belun Technology
Full-Stack Engineer
Dec 2018 – Aug 2021 · Hong Kong
- Built real-time BLE wearable ingestion pipelines for 10,000+ monitored patients across multiple countries.
- Developed biometric anomaly detection that contributed to the device's FDA and international regulatory clearance.
B.S. Computer Science
City University of Hong Kong
2014 – 2018
Technical depth
Work with me
I take on a small number of engagements with healthcare and enterprise teams building LLM systems that need to be right.
- 1
Assess
Review the current system, data and failure modes, and agree on what “better” means in measurable terms.
- 2
Build
Ship in small increments with evals in place from the start, so progress is measured against a baseline.
- 3
Hand off
Documentation, dashboards and a team that can run and extend the system without me.
Let's talk
I'm open to Senior AI Engineer roles and a small number of consulting engagements, especially LLM, agentic and voice systems in healthcare or other domains where accuracy matters.
Colin.Peters0410@outlook.com
View profile
Phone
(773) 466-4989
Location
Louisville, KY · open to remote