Senior Data Engineer
Capgemini · Apr 2023 – Present
- Architect and maintain scalable ETL/ELT pipelines on Azure Databricks (ADF, ADLS, PySpark, SQL), processing high-volume finance and sales data end-to-end into Power BI-ready curated datasets.
- Own a layered Landing → Loading → Certified architecture, applying full and incremental processing patterns tailored to specific business requirements within a Medallion-based data platform.
- Built an ADF-based gating mechanism that withholds downstream file delivery until upstream jobs succeed, closing a data-quality gap and lifting sales-reporting accuracy by 30%.
- Designed a configurable, PySpark-based rules engine that applies business logic dynamically from external rule files writing clean, maintainable code that cut downstream data volume by 70% and pipeline runtime by 40% without re-coding for each change.
- Diagnosed and resolved a major SQL performance bottleneck on a multi-million-record pipeline, debugging and optimizing execution time from 2 hours to 45 minutes (~60% faster).
- Troubleshooted production data issues affecting millions of records running root-cause analysis across pipeline layers and shipping fixes that restore reporting accuracy.
- Apply Delta Lake capabilities (OPTIMIZE, Z-ORDER, VACUUM, partitioning, time travel) and manage governed views in Unity Catalog so downstream teams consume curated data safely.
- Partner with business stakeholders and cross-functional teams across requirements, development, UAT, deployment, and production support within an Agile delivery model.
- Onboard new team members by walking them through project architecture, delivering knowledge-transfer (KT) sessions, and reviewing their initial pipeline work.