Senior Data Engineer - Tech Lead
NextPathway, India · May 2022 – Present
- Led adoption of AI-assisted engineering workflows using ChatGPT, Claude Code, Windsurf, and Codex APIs to accelerate software delivery, automate SQL/PySpark generation, improve troubleshooting, and enhance engineering productivity.
- Built AI-powered modernization accelerators for DataStage-to-Databricks migration by leveraging LLMs for code translation, transformation logic generation, metadata interpretation, documentation generation, and engineering assistance.
- Implemented multi-model validation workflows where outputs generated by one model were reviewed and refined by alternate models, significantly improving solution quality while reducing hallucinations.
- Developed AI-assisted validation utilities for schema comparison, reconciliation reporting, test-case generation, defect analysis, automated root-cause identification, and engineering productivity improvements.
- Integrated AI into engineering workflows for code reviews, Spark optimization recommendations, documentation generation, and pre-production quality assurance.
- Led a 10-member engineering POD responsible for delivering enterprise-scale DataStage to Databricks modernization across more than 800+ data pipelines.
- Architected metadata-driven migration frameworks and reusable PySpark libraries to replicate complex DataStage stages, establishing enterprise cluster configurations and Git standards.
- Engineered automated translation and metadata-extraction utilities to convert DataStage jobs into standardized PySpark pipelines, preserving business logic while accelerating delivery.
- Built automated validation and reconciliation frameworks-featuring schema checks, collision detection, and incremental re-sync-improving validation throughput by 30-35%.
- Optimized Spark workloads using partitioning strategies, caching, broadcast joins, adaptive execution, and cluster tuning, delivering 20-30% performance improvements.
- Collaborated with client architects and business stakeholders to define migration strategy, solution architecture, technical standards, implementation roadmaps, and delivery planning.
- Modernized enterprise Informatica ETL workloads into modular DBT models executing on Snowflake to support scalable cloud-native analytics.
- Developed automated translation utilities to convert Informatica metadata and transformation logic into standardized DBT model templates and Snowflake-compatible SQL, significantly reducing manual effort.
- Built reusable DBT macros, transformation libraries, incremental models, snapshot strategies, and standardized project structures.
- Designed metadata-driven Airflow DAG generation utilities for automated orchestration, dependency management, and deployment.
- Implemented automated data quality validation, reconciliation reporting, deployment verification, and operational monitoring frameworks.
- Contributed to establishing reusable engineering standards, CI/CD practices, release management, and development best practices for enterprise cloud migration.
- Modernized enterprise Informatica ETL workflows into scalable PySpark applications executing on Databricks while integrating enterprise data with Google BigQuery.
- Developed metadata-driven, reusable ingestion frameworks and generic integration components for seamless data movement between multiple source systems and Google BigQuery.
- Optimized Spark workloads using partition pruning, broadcast joins, caching, and efficient file layouts, improving pipeline scalability through parallel processing and workload balancing.
- Built reusable audit, logging, monitoring, exception handling, restart, and recovery frameworks for production data pipelines.
- Designed metadata-driven incremental loading strategies supporting scalable processing of high-volume enterprise datasets.
- Developed automated validation utilities for schema comparison, record reconciliation, execution reporting, and production verification.
- Collaborated with client teams during migration planning, production deployments, hyper-care support, and performance optimization activities.
