
Data Engineer
Tata Consultancy Services · Jun 2021 – Present
- Retail Data & Insights Platform
- • Led the development of a scalable Retail Data Lakehouse platform for a global FMCG enterprise using Azure
- Data Factory, Databricks, ADLS Gen2, and Delta Lake, seamlessly processing 3M+ sales records daily.
- • Built reusable PySpark transformation and data-quality pipelines for schema validation, de-duplication, hash-
- based CDC, and reconciliation, ensuring end-to-end traceability and more than 99.3% valid-record acceptance.
- • Optimized Spark and Delta Lake workloads by implementing partition pruning, Adaptive Query Execution
- (AQE), broadcast joins, and Z-Ordering, reducing query run-times and cutting Azure compute costs.
- • Implemented enterprise governance through Unity Catalog & Azure Key Vault, enabling centralized metadata
- management, data lineage, secure credential handling and fine-grained access control across Lakehouse assets.
- • Delivered curated sales, store-performance, and transaction-level KPI datasets modeled for Power BI, enabling
- cross-regional business teams to monitor performance and drive data-backed operational strategies.
- Manufacturing Data Lakehouse & Analytics Platform
- • Engineered and deployed an Azure Databricks Lakehouse platform for a large-scale manufacturing
- organization, ingesting 500K+ incremental ERP records daily from Azure SQL Database utilizing watermarking
- and JDBC predicate pushdown.
- • Built robust Medallion architecture using Databricks and PySpark, delivering business-ready analytical data
- models, aggregate tables, and Databricks SQL endpoints, improving visibility into regional manufacturing trends.
- • Standardized CI/CD practices using Azure DevOps to automate pipeline deployments, maintaining a 99.5%
- pipeline execution success rate and significantly reducing manual deployment effort.
- BFSI Data Migration & ETL Development
- • Orchestrated the complex migration of enterprise applications and transactional data across heterogeneous
- database environments for a leading international bank.
- • Designed strict source-to-target validation and automated reconciliation controls to guarantee zero data loss and
- accurate cutover during critical deployment windows.
- • Optimized legacy Informatica PowerCenter ETL workflows, improving batch processing efficiency, production
- reliability, and maintainability across legacy-to-target data movement.

