
Data Engineer
IBM · Mar 2024 – Present
- Designed and implemented an Azure Lakehouse platform processing over 2 TB of SAP data daily across 80+ production pipelines, enabling standardized ingestion, historical data management, and enterprise analytics using the Medallion Architecture.
- Architected reusable metadata-driven ETL/ELT frameworks supporting 80+ production pipelines and 50+ SAP tables, reducing development effort and accelerating onboarding of new data sources.
- Built reusable ingestion frameworks for 50+ SAP tables, reducing manual onboarding effort by 70% while improving maintainability and operational reliability.
- Optimized Spark workloads processing multi-terabyte production datasets through partitioning, Delta Lake optimization, predicate pushdown, caching, and file compaction, reducing query execution time by 3x and storage costs by 20%.
- Designed dimensional data models using fact and dimension tables with SCD Type 1 and Type 2 implementations, enabling historical reporting and business analytics.
- Developed automated data quality frameworks incorporating schema validation, source-target reconciliation, duplicate detection, completeness checks, and exception handling, significantly improving trust in downstream analytical datasets.
- Published curated Gold-layer datasets through Azure Synapse Serverless SQL, enabling Power BI reporting and self-service analytics for business users.
- Owned production support for Azure data pipelines by leveraging Azure Monitor for proactive monitoring, incident resolution, root cause analysis (RCA), Spark performance tuning, and SLA compliance for business-critical workloads.
- Integrated Azure Key Vault to securely manage secrets and service credentials across data pipelines.
- Improved reporting performance by 25% through SQL query optimization, efficient joins, CTE restructuring, window functions, and data model refinements.
- Implemented CI/CD pipelines using Azure DevOps and Databricks Workflows, reducing deployment effort and improving release consistency across environments.
- Collaborated with architects, product owners, analysts, and business stakeholders to translate business requirements into scalable Azure data solutions while mentoring junior engineers and driving engineering best practices.



