Senior Data Engineer
Startappss System India PVT LTD · Jul 2021 – Present
- Architected and implemented a scalable Medallion Architecture (Bronze-Silver-Gold) on AWS for WTW's insurance data platform, processing 45+ GB of daily batch and streaming data across policy, claims, and risk datasets.
- Designed hybrid batch + streaming ingestion pipelines from on-prem SQL Server, RDS, and Kafka (MSK) into S3 data lake, using AWS Lambda for format conversion and enabling near real-time data availability.
- Built large-scale transformation jobs using AWS Glue (PySpark) in the Silver layer for data cleansing, SCD Type 2 historical tracking, and incremental processing, significantly improving data reliability and performance.
- Developed reusable Terraform modules for multi-region infrastructure, S3 replication, and IAM governance, and delivered analytics-ready Gold layer aggregates with Athena views for downstream reporting and API consumption.
- Architected and deployed a fully automated credential rotation solution for Snowflake using AWS Lambda, Secrets Manager, and Terraform, eliminating 100% manual secret management and strengthening security compliance across production environments.
- Implemented a secure multi-stage rotation workflow (create, set, test, finish) with RSA/JWT authentication and cross-account ECR policies using awsSourceArn conditions for controlled CI/CD access.
- Configured multi-region ECR replication and Terraform-driven infrastructure for Lambda, IAM roles, and scheduled rotations, ensuring high availability and seamless deployments.
- Designed and orchestrated modular Airflow DAGs to ingest change data streams from multiple source databases into Snowflake, enabling near real-time data availability for analytics teams.
- Built robust end-to-end ETL pipelines (raw
- staging
- reporting) with data validation, bad-record handling, and MERGE-based upsert logic, improving data accuracy and reducing manual reconciliation efforts.
- Implemented data quality checks, snapshot tracking, and historical retention mechanisms, while exposing key pipeline metrics for observability and faster troubleshooting.

