Senior Data Scientist
IBM · Oct 2024 – Present
- Built an agentic AI system to accurately answer queries over large volumes of educational PDFs, addressing poor answer quality in standard RAG and high latency caused by repeated correction loops in an earlier Corrective RAG approach.
- Designed a LangGraph-based agentic workflow with retrieval, correction, and routing nodes connected through conditional edges to orchestrate the end-to-end RAG process.
- Applied query expansion to reformulate ambiguous or under-specified questions before retrieval, improving recall over the educational PDF corpus.
- Used a map-reduce style chunk summarisation strategy to summarise retrieved chunks in parallel and combine them into a coherent context, keeping latency low as document volume grew.
- Implemented Corrective RAG to re-evaluate and refine retrieved chunks against the query before generation, and Adaptive RAG routing to dynamically decide whether retrieval was needed, avoiding redundant loops.
- Added guardrails to validate retrieved content and reduce hallucinations, cutting query latency and improving answer accuracy while achieving scalable, controlled, and safe performance across large educational PDF datasets.
- Built a secure, lightweight RAG application to query unstructured airline PDFs (operational manuals, safety guidelines, policy documents) without relying on a vector database, reducing infrastructure overhead, embedding costs, and compliance complexity in a regulated aviation environment.
- Designed a vector-less retrieval pipeline using direct document chunking and BM25-based keyword retrieval instead of embedding-based similarity search, reducing infrastructure dependencies.
- Integrated an LLM for context-aware responses with full compliance to aviation data standards, delivering a simpler, auditable, and cost-effective GenAI application that meets aviation regulatory requirements without any vector database dependency.
