AI / ML Engineer · Toronto, ON, Canada

Sohen Patel

I build and evaluate LLM, agentic and applied ML systems, and I measure whether they actually work.

I am an MEng student at the University of Toronto (GPA 3.81/4.0, graduating Dec 2026) who builds LLM and agentic systems and then measures whether they actually work. I shipped an enterprise agent to 5,000+ employees at Ontario Health, led a student team building a six-agent consulting system for a Deloitte-sponsored practicum, and I am building a multi-agent legal reasoning system for my research project. Before AI, I spent a year turning plant data into decisions at ArcelorMittal Nippon Steel. I care about honest baselines, gold-standard datasets and systems people can trust.

Download resume View projects GitHub LinkedIn

5,000+: employees using the Ontario Health News Agent I built
68%: F1 gain from Chain-of-Thought prompting (0.25 → 0.42)
129: automated tests behind the 6-agent Deloitte practicum system
46%: NDCG@10 lift from a two-stage recommender

Experience

Where I have worked

Sep 2026 – Present

University of Toronto

Toronto, ON

Teaching Assistant

MIE354 Business Process Engineering: supporting course delivery, lab sessions, student assignments, exam grading and assessments.
MIE1628 Cloud-Based Data Analytics: supporting student assignments, exam grading and lab work.

Jul 2026 – Sep 2026

Ontario Health

Toronto, ON

AI Research Intern

Built an enterprise Ontario Health News Agent in Microsoft Copilot Studio end-to-end, deployed to 5,000+ employees org-wide.
Created a 259-item human-verified gold-standard dataset for 3 healthcare tasks; built an end-to-end LLM evaluation framework for 5 models (GPT-5, Claude, Gemini, MS Copilot, Perplexity) across 13+ metrics using RAGAS, DeepEval and LLM-as-a-Judge.
Ran prompt-engineering experiments across 4 strategies: Chain-of-Thought on a 5-year dataset increased the agent's F1 from 0.25 to 0.42 (+68%) and reduced hallucination rate by 51% (78% → 27%); runs were logged to Langfuse.

The agent

Three tasks, each with its own window: PHIPA privacy-decision news (last 6 months), hospital leadership and CEO changes (last 5 years), and interim supervisors appointed by the Province (last 5 years).
Guardrails written into the agent's instructions: defined news criteria (clinical services and leadership in, recruitment ads and fundraising out), an approved hospitals-by-region document as the single source of truth instead of geographic guesses, trusted sources first (mainstream media, official hospital sites, the privacy commissioner), and mandatory citations with date, publisher, title and URL. Results come back as a chronological table by region and hospital, exportable to Word.
Tested the same agent on Claude and GPT model variants inside Copilot Studio before choosing what to roll out.

Building the ground truth

Built the gold dataset by hand from primary sources such as hospital announcements and government orders, tiered by confidence (dated announcement, undated, inferred) so a recall number could be split by how certain the answer was.
Tagged every gold row by origin (agent-derived, other-tool-derived, independent research) and reported recall on the independent subset too. Several facts entered the gold set because a system under test surfaced them; without that split the agent would get credit for answers it helped write.

Evaluation framework

Combined Copilot Studio's built-in LLM-judge metrics (relevance, groundedness, completeness, abstention) with RAGAS and DeepEval answer relevancy and factual correctness, entity-level precision/recall/F1, and custom scoring for citation validity.
Custom metrics beyond F1: recall by confidence tier, specificity of dates, field-level accuracy (role, date, direction), omission rate, hallucination rate, completeness across hospitals, and run-to-run consistency measured as pairwise Jaccard similarity over repeated runs.
Every returned item with no gold match was adjudicated into one of five buckets (gold gap, hallucination, wrong attribution, out of window, out of scope), so precision and hallucination were not inflated by real-but-unlisted answers. Errors were then classified into a fixed six-way taxonomy. Confidence intervals came from bootstrap resampling.
Kept tasks separate instead of pooling: the PHIPA window is 6 months and the leadership window 5 years, so their numbers are not comparable.

Debugging the evaluator itself

Found and fixed three scorer bugs: table header rows counted as answers (inflating hallucination), ambiguous first names matching the wrong person, and a completeness score stuck at 1.0.
Later found that a person with several career events at one hospital was being matched to whichever gold row came first. Re-scoring by date and direction repointed 161 matches across 26 files and raised recall and date accuracy for every system.
Checked the automated judge against a stratified 70-item hand re-adjudication: 4 disagreements (about 6%), all traced to that same matching bug. I did not report an inter-rater kappa because the second reviewer was not independent.

What the experiments showed

Chain-of-thought helped at scale but hurt on the small set. On the 5-year dataset it raised the Copilot agent's F1 from 0.25 to 0.42 (precision from 0.19 to 0.69) and the strongest general chatbot's from 0.25 to 0.57, cutting hallucination by roughly 50 points in both cases (Copilot 81% to 31%, ChatGPT 78% to 27%). On the short 6-month set the same technique made models over-extract, and hallucination rose above 80%.
Structured, field-by-field prompts raised recall but pushed ChatGPT's hallucination rate from 0% to 40%: a straight recall-precision trade-off.
Consistency across repeat runs varied a lot: under the baseline prompt some general-purpose chatbots scored as low as 0.14, while the Copilot agent with chain-of-thought reached a perfect 1.0.
Even the best system found fewer than half of the leadership changes, so the honest conclusion was a useful first pass that needs human verification, not automation.
Most common real errors: no usable citation link (214 of 297 matched items), wrong role title (189) and wrong date (119). One person was completely fabricated by one run of one system, confirmed against the hospital's own records.
Root cause of many failures was the prompt, not the model. Windows like 'last 5 years' and terms like 'leadership' were ambiguous, so I proposed explicit date ranges and written in-scope and out-of-scope definitions.

Communication

Produced briefing decks on Copilot agents, evaluation methodology and results for supervisors and team leads, and drafted a research write-up of the evaluation.

May 2026 – Jun 2026

Deloitte (Sponsored Practicum)

Toronto, ON

AI Engineering Project Team Lead

Built a 6-agent system that turns a client brief into consulting deliverables (WBS, RACI, RAID, weekly status report) with a human-in-the-loop revision loop and Deloitte template sync for DOCX/XLSX exports, backed by 129 automated tests.
Added an LLM routing layer with fallback across 4 backends (Claude, GPT, Gemini, DeepSeek) and prompt-cache warm-up to cut repeated token cost.
Built a 13-rule Quality Gates engine (graph-based WBS cycle detection, RAID/RACI cross-artifact validation).

How I built it

Grew the system in stages: command-line prototype, then specialist agents with Pydantic schemas, a provider layer for multiple LLMs, rendering and templates, a FastAPI backend with a web UI, file upload and export, confirmation questions before generation, quality gates, and artifact versioning with streaming.
Grounded output in the firm's own templates. A sync script extracts column names, examples and structure from the real WBS, RAID, stakeholder and status-report templates into machine-readable specs that are injected into the prompts, so generated documents match the format reviewers expect.
Compared three generation strategies: free-form markdown, structured JSON rendered afterwards, and template-guided structured generation. Structured and template-guided output was the easiest to validate and the closest to the target format.
Ran model ablations by holding agents, schemas, prompts and templates fixed and swapping only the model, scoring completeness, template alignment, schema validity, speed and cost. Smaller models were cheaper and faster, larger ones reasoned better, and strict schemas plus template grounding narrowed the gap.

Human in the loop

The system asks the reviewer to confirm open questions from the brief before it generates anything, instead of guessing.
Reviewer feedback is scoped to one artifact at a time; each revision creates a new version with a diff, and linked artifacts (work packages, owners, risks) are reconciled afterwards.
A quality-gates checklist lists what a human must still verify, and a failure-mode table records each known way the agents go wrong with its mitigation.

Engineering

Prompt caching: stable context (parsed brief, WBS, RACI) is cached and warmed before the expensive generation calls to cut repeated token cost.
Runs offline with a mock LLM, so demos and CI never need API keys.
129 tests written alongside the code, covering agents, API, caching, exports and artifact links.

Aug 2024 – Aug 2025

ArcelorMittal Nippon Steel India (AM/NS)

Surat, India

Graduate Engineer Trainee

Used SAP datasets and project logs for real-time monitoring of compliance metrics and workflows, supporting executive decisions across 12 cross-functional departments.
Designed Power BI dashboards tracking KPIs, resources and progress for a plant expansion from 7 to 14 MMTPA.
Identified workflow bottlenecks and execution gaps, contributing to a 16.7% reduction in project timelines.

Scope

Worked with 12 department heads, under the plant's Executive Director, on the upstream expansion project. The goal was to shorten the timeline by two months (the 16.7% improvement) and double steel capacity from 7 to 14 MMTPA by December 2025.
Assessed optimum production levels and utilisation of machinery, raw materials and consumables at Hot Strip Mill #1 and #2.
Analysed production and indirect costs for plant facilities under the Operation Integration Group.
Ran HIRAC (hazard identification, risk assessment and control) and PSM (process safety management) reviews to strengthen health, safety and hazard prevention.

The analytics angle

The work was mostly turning messy operational data (SAP records, project logs, utilisation and cost figures) into numbers a decision-maker could act on: which mill or facility was under-utilised, where cost sat, and where the schedule was slipping.
Building KPI dashboards across 12 departments meant agreeing what each metric meant with the team that owned it before visualising it. I now do the same when defining evaluation metrics for LLM systems.

Selected work

Projects

MEng research project · University of Toronto

Multi-Agent Legal Reasoning System

0.72

judge AUC on 398 appeals from the lower decision alone (95% CI 0.66 to 0.77)

Agents that read a Federal Court of Appeal case and predict whether the moving party succeeds. The interesting part is the evaluation: I found that the inherited benchmark could not be trusted, rebuilt it from open court data, and showed that part of the earlier accuracy came from the test cases restating their own answer.

PythonClaude APILangChainQdrantBM25 + RRF

LangGraph · RAG · forecasting · LLMOps

Market Research Agent: Agentic RAG over SEC Filings

0.95

RAGAS faithfulness on a 99-question golden set, at about $0.0013 per question

A research assistant that answers questions about five large-cap companies by combining SEC filing text, trained volatility forecasts and database lookups. Every answer is cited, a judge model verifies it before it is shown, and every quality claim is measured on a 99-question golden set.

PythonLangGraphLangChainPostgreSQLpgvector

Deep learning · evaluation · MLOps · live demo

ASL Fingerspelling Recognition

0.307

test CER on 14 held-out signers (0.334 for the first deployed checkpoint)

Video of ASL fingerspelling in, text out. A Conformer-Transformer reads hand landmarks, scored on signers it never saw, with an error analysis that found where it fails, a confidence rule that flags those cases, a from-scratch retraining that honestly did not win, and the full MLOps loop around it: versioned data, tracked experiments, a gated registry, monitored serving, CI and a free public demo.

PyTorchMediaPipeConformerCTCOpenCV

Ontario Health internship · LLM evaluation · Langfuse

Hospital News Agent Evaluation

0.25 → 0.42

agent F1 on the 5-year leadership task with chain-of-thought prompting (ChatGPT 0.25 → 0.57)

A measured comparison of the Microsoft 365 Copilot news agent I built at Ontario Health against Perplexity, Claude, ChatGPT and Gemini on three hospital monitoring tasks, scored against a hand-verified gold dataset, with a prompt-engineering study, an error taxonomy, and the limits of the evidence stated plainly.

PythonpandasMicrosoft Copilot StudioLLM-as-a-JudgeLangfuse

LLM app · guardrails · cost control · live on this site

Ask About Sohen: a public, abuse-resistant chatbot

0 of 197

attack and off-topic questions that got through, out of 335 test questions

The assistant in the corner of this site. Strangers can type questions and it answers from my own projects and experience, but only about me: it refuses everything else before spending a token, caps what any visitor can spend, and was tested against 335 questions that include attacks.

TypeScriptNext.jsOpenAI API (Flex)Upstash RedisServer-sent streaming

Deloitte sponsored practicum · multi-agent

Consulting Workplan Agent

129

automated tests across agents, API and exports

A client brief goes in; a WBS, RACI matrix, RAID log and weekly status report come out, with a human-in-the-loop revision cycle, cross-artifact validation and Word/Excel exports in the firm's template.

PythonPydanticFastAPIClaudeGPT

Recommender systems

Two-Stage Recommender: Retrieval + LambdaRank

+46%

NDCG@10 vs MF (0.161 → 0.235); Hit Rate@10 0.53 → 0.63

A Two-Tower retrieval model with hard negatives feeds a candidate-aware ranker trained with LambdaRank, built the way production recommenders are staged.

PyTorchInfoNCELambdaRankMovieLens 100KPandas

Probabilistic modelling

Fraud Detection with Gaussian Mixtures

0.697

PR-AUC (primary) · ROC-AUC 0.969 · F1 0.727 on held-out test

A generative fraud detector: one Gaussian mixture for normal transactions, one for fraud, scored by log-likelihood ratio, with the decision threshold tuned on validation data.

scikit-learnNumPyPandasMatplotlib

Time series · model comparison

Forecasting MSFT Returns

0.000875

test MSE (ARIMA) · 62.2% directional accuracy

Twelve-plus models predict Microsoft's 5-day log return from a multi-asset feature set. The question: does model complexity buy anything on a target this noisy?

Statsmodelsscikit-learnPyTorchyfinanceta

More work

Toolbox

Skills

Languages & Data

PythonSQL (MySQL, PostgreSQL)PandasNumPyStatsmodelsMatplotlibSeabornPlotlyPower BI

ML & Deep Learning

scikit-learnPyTorchComputer Vision (MediaPipe, OpenCV)Model Evaluation (F1, NDCG, RAGAS, DeepEval)

LLMs

LangChainLangGraphLangfuseRAG (hybrid search, Qdrant)DSPyClaude/OpenAI APIsPydanticStreamlitHugging Face

MLOps

GitGitHub ActionsDockerFastAPIMLflowDVCpytestPrometheus/GrafanaONNXAWS (S3, EC2, SageMaker, Bedrock)

Background

Education

Sep 2025 – Expected Dec 2026

University of Toronto

Master of Engineering, Data Analytics and Machine Learning

GPA 3.81/4.0

Scholarship: Aga Khan Foundation International Scholarship (Jul 2025), awarded to one of the few students from India and covering the full program.
Coursework: Introduction to Machine LearningIntroduction to Deep LearningData Science Methods and Statistical LearningFoundations of Data Analytics and MLAI in Finance: Neural Networks to Deep LearningManagement Consulting for EngineersProject ManagementMEng Research Project · in progress

Jul 2021 – Jul 2024

L.D. College of Engineering, Gujarat Technological University

B.E. Mechanical Engineering

CGPA 8.40/10

Exam: All-India Rank 3465 in GATE XE 2023, a national-level engineering examination.
Provisional patents: Provisional patent applications, filed from my ISRO research internship.
Coarse Actuation Mechanism for Primary Optics Deployment in CubeSat-Based Space Telescopes for Sub-Meter Resolution Imaging ApplicationsIN:202421088072 · Filed Nov 14, 2024
Fine Actuation Device for Precision Alignment of Primary Optics in Deployable Space TelescopesIN:202421087180 · Filed Nov 12, 2024

Beyond the day job

Extracurricular and recognition

Recognition

Delegate, Harvard Project for Asian and International Relations (HPAIR) Asia Conference
Hong Kong, August 2023 · attended on a scholarship
HPAIR is a student-run Harvard College organisation, founded in 1991, that hosts an annual conference for students and young professionals on the economic, political and social issues facing the Asia-Pacific. The 2023 conference in Hong Kong brought together keynotes, fireside chats and team impact challenges. It deepened my understanding of global issues and collaboration, and I am grateful for the scholarship that made it possible.
Reach for the Stars (RFS) Mentorship Program, Cycle 6
National-level programme
Selected among the top 60 students for a mentorship programme organised at the national level.

Volunteering

Jun 2024 – Jul 2024

Speaker · Aga Khan Foundation

Gave a national webinar, 'Dream of Becoming an Engineer?', for the CareerCraft program of the Aga Khan Education Board for India, guiding students across India on choosing engineering career paths. Received a Letter of Appreciation.

Mar 2019 – Mar 2024

MIS Incharge · Aga Khan Foundation

Maintained and regulated attendance records in the portal for religious classes run under the Ismaili Tariqah and Religious Education Board for India (ITREB).

Mar 2020 – Mar 2022

Volunteer · Aga Khan Foundation

Managed and oversaw virtual religious classes during the COVID-19 pandemic under ITREB.

Mar 2019 – Mar 2021

Member · Aga Khan Foundation

Part of a volunteer team supporting young people, with a focus on sports, under the Aga Khan Youth and Sports Board for India (AKYSB).

Contact

Let us talk

The fastest way to reach me is email.

sohenpatel.work@gmail.com sohen.patel@mail.utoronto.ca +1 416-666-8954 LinkedIn GitHub