Harshal Mane
As a Senior Data Engineer, I am driven by a deep ambition to achieve mastery in crafting reliable and impactful data solutions. I take full accountability for my work, ensuring high-quality outcomes that reflect my idealistic vision for technology's positive influence, while valuing the autonomy to innovate and deliver with precision and empathy.
Experience
Senior Data Engineer
Startappss System India PVT LTD · Jul 2021 – Present
- Architected and implemented a scalable Medallion Architecture (Bronze-Silver-Gold) on AWS for WTW's insurance data platform, processing 45+ GB of daily batch and streaming data across policy, claims, and risk datasets.
- Designed hybrid batch + streaming ingestion pipelines from on-prem SQL Server, RDS, and Kafka (MSK) into S3 data lake, using AWS Lambda for format conversion and enabling near real-time data availability.
- Built large-scale transformation jobs using AWS Glue (PySpark) in the Silver layer for data cleansing, SCD Type 2 historical tracking, and incremental processing, significantly improving data reliability and performance.
- Developed reusable Terraform modules for multi-region infrastructure, S3 replication, and IAM governance, and delivered analytics-ready Gold layer aggregates with Athena views for downstream reporting and API consumption.
- Architected and deployed a fully automated credential rotation solution for Snowflake using AWS Lambda, Secrets Manager, and Terraform, eliminating 100% manual secret management and strengthening security compliance across production environments.
- Implemented a secure multi-stage rotation workflow (create, set, test, finish) with RSA/JWT authentication and cross-account ECR policies using awsSourceArn conditions for controlled CI/CD access.
- Configured multi-region ECR replication and Terraform-driven infrastructure for Lambda, IAM roles, and scheduled rotations, ensuring high availability and seamless deployments.
- Designed and orchestrated modular Airflow DAGs to ingest change data streams from multiple source databases into Snowflake, enabling near real-time data availability for analytics teams.
- Built robust end-to-end ETL pipelines (raw
- staging
- reporting) with data validation, bad-record handling, and MERGE-based upsert logic, improving data accuracy and reducing manual reconciliation efforts.
- Implemented data quality checks, snapshot tracking, and historical retention mechanisms, while exposing key pipeline metrics for observability and faster troubleshooting.
Skills 0 proven through work
Also works with
Proof of Work
Proof of Work
Harshal shares this with people who ask. You'll hear back either way.
Education
B.Tech, Information Technology
Chameli Devi Group of Institutions, Indore · 2019 — 2023
Contact details
Contact details
Harshal shares this with people who ask. You'll hear back either way.
What drives their work
The Architect
You design and construct robust, scalable systems from the ground up.
Harshal's superpower
You excel at architecting and implementing secure, reliable, and efficient data platforms that deliver measurable business impact.
How Harshal works
You are best suited for roles that require designing, building, and optimizing large-scale data platforms, with opportunities for technical leadership and mentorship.
Ask about this candidate
AI responses may contain errors. Proof status and source information are provided by Proof of Skill.
A Proof CV — one profile, kept current, with the proof attached. Back to the portal