
Senior Software Engineer
HCL Tech · Sep 2025 – Present
- Led the end-to-end migration of NBC Universal's on-premise Film Data Warehouse from Teradata + Informatica PowerCenter to a cloud-native lakehouse on Azure Databricks, modernizing 200+ legacy ETL workflows into PySpark notebooks and dbt models on Delta Lake.
- Architected the medallion (Bronze / Silver / Gold) data architecture on ADLS Gen2 + Delta Lake, processing 5+ TB of film, marketing, and financial data daily across box-office, distribution, and royalty domains.
- Built 60+ Airflow DAGs to orchestrate ingestion, transformation, and publishing pipelines, replacing Autosys schedules; implemented dynamic task generation, SLA monitoring, retries, and Slack-based alerting to reduce on-call incidents by ~40%.
- Developed reusable PySpark frameworks for incremental ingestion, schema-evolution handling, and SCD Type 2 maintenance on Delta tables adopted across 4+ downstream domains.
- Engineered a dbt-based transformation layer (60+ models) with tests, snapshots, and documentation, enabling self-service analytics for marketing and finance teams and cutting analyst data-prep time by ~50%.
- Implemented IICS (Informatica Intelligent Cloud Services) hybrid pipelines using Secure Agents to bridge on-premise Oracle and Teradata sources with Azure cloud targets during the multi-year migration.
- Tuned long-running Spark jobs by re-partitioning skewed datasets, adopting Delta Lake Z-Order indexing, and adjusting AQE settings
- reducing critical pipeline runtimes from ~3 hours to under 45 minutes.
- Set up Unity Catalog for fine-grained access control and lineage across the lakehouse; partnered with the governance team to define table ownership, tags, and column-level masking for PII fields.
- Mentored 3 junior engineers on PySpark best practices, code reviews, and Airflow DAG design; introduced a peer-review checklist that reduced production hotfixes by ~30%.

