Market Research Agent: Agentic RAG over SEC Filings

The problem and the scope

A financial analyst asking 'what supply chain risks does Apple report, and what is the model's volatility outlook?' needs two kinds of evidence at once: text from regulatory filings and a number from a model. A plain chatbot invents both. The goal was an assistant whose every claim can be traced back to a filing excerpt, a database row or a model output, and which says so when it cannot.

• Coverage: five large-cap companies (AAPL, MSFT, NVDA, JPM, XOM), their most recent 10-K and 10-Q filings, about ten years of daily prices and quarterly fundamentals.
• Question types it handles: what a company says in its filings, price and fundamentals lookups, near-term volatility and direction forecasts, comparisons across companies, and questions that mix these.
• Question types it refuses: trades, personalised financial advice, and attempts to override its instructions. Obvious jailbreak phrases are refused by a pattern check before any model call, and the rest are refused at the first routing step, before any tool runs.
• Positioning: research and education only. The interface and every forecast say it is not investment advice.

How one question flows through the agent

The agent is a fixed LangGraph state machine rather than a free-running tool-calling loop, so cost and latency are bounded and the behaviour is testable with fake models.

• Route: an LLM returns a validated decision (intent, tickers, which tools to use, a standalone search query). Out-of-scope requests go straight to a refusal. If routing itself fails, the agent falls back to a plain filings search instead of crashing.
• Gather: runs the chosen tools. Filing search runs on every pass; forecasts and database lookups run once per question, not once per retry. If a question names several companies, each company is searched separately and the results are interleaved, so a comparison cannot be answered from one company's text alone. Tool failures are recorded and shown, never hidden.
• Generate: drafts a structured answer using only the retrieved excerpts, forecasts and rows, citing sources by number.
• Grade: a separate judge model checks the draft against the same evidence and returns whether it is grounded, whether it is relevant, and a 0 to 1 score. An answer passes only if it is both grounded and relevant and scores at least 0.7. If the grader itself fails, the answer is marked unverified rather than passed.
• Rewrite and retry: on failure, the grader's feedback is used to write a better search query and the agent gathers again, at most twice, with LangGraph's recursion limit as a second guard.
• Finalize: citations are verified against what was actually retrieved (hallucinated source numbers are dropped), excerpts are sanitised, and the answer is returned with its forecasts, data, retry count and a quality flag.
• Every step returns a validated Pydantic object through structured output; if a model replies in plain text instead of calling the schema function, the call is retried with a changed prompt.

The three tools

Data pipeline

• Sources: yfinance for prices and fundamentals; SEC EDGAR for filings, with the descriptive User-Agent the SEC requires and throttled requests to stay under its fair-access limit.
• Chunking: filings are split by their 'Item' headings (table-of-contents lines are merged away) so a chunk knows which section it came from, then cut into 1,000-character pieces with 150 characters of overlap. The result is about 5,400 chunks.
• Embeddings: computed locally with a Hugging Face model (BAAI/bge-small-en-v1.5, 384 dimensions), so no external embedding API is needed.
• Storage: one PostgreSQL database with pgvector holds prices, fundamentals (stored long and tidy because the metrics vary by company) and the document chunks, with an HNSW index for vectors and a full-text index for keywords.
• Reliability: every insert is idempotent, every raw response is cached so re-runs do not hit the sources, and raw data is versioned with DVC. A weekly workflow re-runs ingestion against the production database (Neon).
• A lesson from the first production seed: serverless databases drop idle connections, so ingestion keeps transactions short, disables prepared statements behind the connection pooler and normalises connection URLs.

Retrieval, measured instead of assumed

Dense and keyword search results are fused with reciprocal rank fusion, then a cross-encoder rereads the top candidates. I tested whether each part earns its place on the 50 single-company filing questions, with no LLM calls involved.

• The finding that surprised me: plain hybrid slightly trailed dense-only on this data, because the keyword leg alone is weak on filing prose. The reranker is what pays off (+0.12 NDCG@6), so it is on by default. The ordering was identical on the original 23 questions.
• Down-weighting the keyword leg helped un-reranked hybrid but not the reranked pipeline, so I kept standard equal-weight fusion.
• Caveat I state in the repo: 50 questions make the gaps clearer but I did not significance-test them.

Forecasting models, including where they lose

The agent calls trained models for near-term outlook questions. They are small and evaluated strictly: one chronological 70/15/15 split shared by all tickers, with the last 5 (volatility) or 21 (direction) days of train and validation purged so overlapping forward-looking targets cannot leak across a boundary.

• Target: log annualised volatility over the next five trading days, estimated from daily highs and lows with the Parkinson estimator. Features are daily, weekly and monthly log volatility plus returns, standardised per ticker using training rows only.
• The LSTM is deliberately small (22-day window, 32 hidden units, one layer), trained with Adam and early stopping on validation loss. Because forecast errors overlap, significance uses a Diebold-Mariano test with Newey-West errors: the LSTM beats the HAR-RV baseline with p = 0.001.
• It does not win everywhere. Skill against HAR by ticker: AAPL +7.7%, JPM +8.6%, MSFT +6.2%, NVDA +2.2%, and XOM -5.1%, where the simple baseline is better. The model card says so.
• The direction classifier (momentum, volatility, drawdown, volume and RSI features) is weak: test ROC-AUC 0.62 and accuracy 0.55, below the 0.65 always-up base rate. I kept it, labelled it as weak in the interface, and do not present it as a signal.
• Operations: models are tracked in MLflow and saved as versioned bundles. A monthly workflow runs a walk-forward check: it holds out the latest 60 trading days, trains a challenger on the data before them, scores both on those days, and opens a pull request only if the challenger improves by more than 2%. It also compares the current model's holdout error with its original test error as a drift signal, and costs no LLM calls.
• Forecasts and an out-of-sample volatility backtest are published back to Postgres so a Power BI dashboard can chart them.

Evaluation designed so it can fail

LLM judges are useful but noisy and cost money, while some behaviours can be checked exactly. So evaluation has layers, all driven by one thresholds file.

• Layer 1, free and exact: refusals happen exactly when expected, the forecast and database tools were actually called, numeric facts from the database appear in the answer (tolerating '$90.0 billion' versus '90,007,000,000'), and citations are present.
• Layer 2, run deliberately: RAGAS faithfulness, answer relevancy, context precision and context recall on the filing questions, with cost and latency per run.
• Layer 3, in CI: a DeepEval faithfulness and relevancy gate on six items against a small committed corpus, so it works on an empty database and skips itself when no API key is present.
• The first 38 questions were frozen when the set grew to 99, so earlier results stay comparable.

Results, and what the expanded set exposed

• Cross-company comparisons were the weak spot: only 4 of 6 cited both companies, because all six retrieved excerpts came from one company and the agent said it could not compare. I fixed it by searching each company separately and interleaving the results (rate 0.67 to 1.0 on a re-run).
• Other failures I found and kept in the record: one answer with no citation (retrieval returned only XBRL tables for an XOM question) and one forecast question that ended in 'I could not produce a valid answer' after two retries because the provider returned no usable structured output.
• The abstention check is a regex heuristic, so the unanswerable answers were also read by hand.
• DeepEval's relevancy score is coarse: the same correct answer scored between 0.45 and 1.0 across runs. I set the CI threshold to 0.55 and wrote the reason next to it, and use RAGAS answer relevancy as the finer measure.

Choosing the model by cost per correct answer

• The cheapest model was not the best outcome: GPT-4o mini wrongly refused four legitimate filing questions, which also cost it citations, and picked a stale quarter for one lookup. Cost per correct answer, not per call, is the number that matters.
• Paying more did not buy faithfulness here: Sonnet 5.5 cost about 14 times DeepSeek for the same faithfulness (0.916 versus 0.908), and Haiku 4.5 cost about 5.6 times for +0.026, within noise. DeepSeek stayed the default.
• The judge is a separate model (and configurable), so the model under test is not grading itself. The whole comparison cost about $1.40.
• Caveats stated in the repo: one run per model, RAGAS on only 20 filing questions, so gaps of a few hundredths are noise; the judge is DeepSeek, which may favour its own phrasing and was not tested for that; prices are list prices from 2 October 2026.

Observability and cost control

• Every question is one Langfuse trace: route, tool calls and every LLM call with tokens, cost and latency, tagged with provider, model and git commit, plus scores for quality passed, grade, retries, refusal and tool errors. With no keys configured it is a no-op, so tests, CI and forks need no Langfuse.
• Prompts can be synced to Langfuse's prompt registry and loaded from there, falling back to the local text if the registry is unreachable or an edit changes the template variables.
• Cost levers: one inexpensive model for both agent and judge by default, an on-disk cache of identical LLM calls so re-running evaluations is free, retries capped at two, and non-query-dependent tools run once. A typical question is about 5k tokens and a refusal about 1k.
• Tracing is thread-bound, so streamed requests run entirely on one worker thread.

Serving and integrations

• Request bodies are validated (question length, ticker pattern, horizon). Blocking work runs in a thread pool so the event loop is never blocked. The agent runtime is built lazily, so /health and /forecast work with no LLM key, and a missing key is a clean 503 rather than a stack trace.
• Two limits protect the paid model: a per-client sliding window (10 requests a minute) and a daily cap of 50 chat questions across all clients, with an output token cap. Obvious jailbreak phrases are refused by a pattern check before any model call, and only the DeepSeek key is sent to the Space. Error responses never include internal details. Both limiters are in memory, which is enough for a single-process demo.
• A Gradio interface sits on the same code path: a live progress checklist with a sources table, a Plotly forecast chart, and market-data charts.
• An MCP server exposes filing search, forecasts, market-data lookups and the full agent to any MCP client such as Claude Desktop.
• A Power BI recipe covers the connection, data model, DAX measures and three dashboard pages over the published forecasts, backtest and SQL views.

Engineering and delivery

• The app is live on a Hugging Face Gradio Space, reading from a Neon Postgres database with pgvector. Free hosting was the hard part: it needed a pinned torch release that ZeroGPU accepts, Gradio's server-side rendering turned off so uvicorn could bind its port, a small GPU probe app to pass the platform's startup check, and explicit CPU loading for the embedding model and reranker. The fixes are in the deploy module and written up in an architecture decision record.
• The Docker image is multi-stage: dependencies from a lockfile, CPU-only PyTorch, no compilers or training libraries in the runtime stage, both models baked in so it starts offline, a non-root user (UID 1000), a health check, and no secrets in any layer.
• Secrets live only in the CI and runtime environment, and values pasted from .env files into CI secrets are cleaned of stray quotes (a bug I hit and fixed with a test).
• Six short architecture decision records document the trade-offs below, written when each decision was made.

Design decisions I would defend

Limits I state openly

• Five tickers and one market regime: no crisis or regime-shift testing beyond what ten years of data contain. Unseen tickers are rejected, not extrapolated.
• The direction model is weak and the volatility model loses on XOM. Outputs are statistical estimates, not trading signals, with no transaction costs or portfolio construction.
• The golden set was drafted by an LLM and curated by one person (me). Ninety-nine items catches regressions but is not enough to claim fine-grained accuracy differences, and a second labeller is on the to-do list.
• Only DeepSeek has been fully evaluated with RAGAS on the whole set; the other providers were compared on a 69-question subset with one run each.
• The live demo has no authentication, so its limits are global: 10 requests a minute per client and 50 chat questions a day across everyone, held in memory (they reset when the process restarts). Per-user limits need accounts, which is on the to-do list.

What I learned, and what I would do next

• A grader that can say 'this answer isn't supported' is worth more than a clever generator: surfacing the quality flag to the user changed how the product behaves, not just how it scores.
• Measure before believing. Hybrid retrieval trailed dense-only until the reranker, and the LSTM beats the baseline overall but not for every ticker.
• Noisy LLM-judge metrics need calibrated thresholds with the reason written down, and cheap deterministic checks catch what judges miss.
• Next: enlarge the golden set with a second labeller and multi-turn questions, repeat the provider comparison with several runs per model to put error bars on the small gaps, add distribution-level drift monitoring and more tickers, and add authentication so the daily cap can be per user.