
Data Engineer
Tata Consultancy Services Ltd (TCS) · Jun 2023 – Present
- Designed and migrated enterprise-scale ETL/Data Engineering pipelines by converting legacy Oracle stored procedures and ETL logic into Hive (Tez) and Python-based pipelines on Hadoop/HDFS, reducing processing time from 48 hours to 6-7 hours (86% improvement) and enabling faster data availability for downstream analytics.
- Re-engineered legacy ETL workflows into optimized PySpark pipelines using partition pruning and broadcast joins, reducing batch processing time from 24 hours to 8 hours (89% improvement) on distributed Hadoop/HDFS environments.
- Developed automated data reconciliation, validation, and quality frameworks generating 20+ daily audit reports, eliminating 90% of manual reporting effort and improving operational efficiency.
- Built automated KPI and reporting pipelines using Python, Pandas, SQL, and ETL workflows, replacing manual data extraction processes with scheduled and validated reporting outputs.
- Developed, scheduled, and monitored production Apache Airflow DAGs integrated with Hive, HDFS, Oracle, and Sqoop workflows, ensuring SLA compliance for critical data pipelines.
- Managed code using Git for version control.
- Optimized Oracle ETL processes and OBIEE dashboards, improving report performance by 30-35% for business and leadership reporting.
- Performed PySpark and Python unit testing, production troubleshooting, data validation, and collaborated with cross-functional teams to analyze requirements and resolve data pipeline issues.

