HM

Harshal Mane

As a Senior Data Engineer, I am driven by a deep ambition to achieve mastery in crafting reliable and impactful data solutions. I take full accountability for my work, ensuring high-quality outcomes that reflect my idealistic vision for technology's positive influence, while valuing the autonomy to innovate and deliver with precision and empathy.

Work style

Remote

Availability

Available in 15 days

Contact details — on requestProof of Work — on request

Experience

S

Senior Data Engineer

Startappss System India PVT LTD · Jul 2021 – Present

Not yet confirmed
  • Architected and implemented a scalable Medallion Architecture (Bronze-Silver-Gold) on AWS for WTW's insurance data platform, processing 45+ GB of daily batch and streaming data across policy, claims, and risk datasets.
  • Designed hybrid batch + streaming ingestion pipelines from on-prem SQL Server, RDS, and Kafka (MSK) into S3 data lake, using AWS Lambda for format conversion and enabling near real-time data availability.
  • Built large-scale transformation jobs using AWS Glue (PySpark) in the Silver layer for data cleansing, SCD Type 2 historical tracking, and incremental processing, significantly improving data reliability and performance.
  • Developed reusable Terraform modules for multi-region infrastructure, S3 replication, and IAM governance, and delivered analytics-ready Gold layer aggregates with Athena views for downstream reporting and API consumption.
  • Architected and deployed a fully automated credential rotation solution for Snowflake using AWS Lambda, Secrets Manager, and Terraform, eliminating 100% manual secret management and strengthening security compliance across production environments.
  • Implemented a secure multi-stage rotation workflow (create, set, test, finish) with RSA/JWT authentication and cross-account ECR policies using awsSourceArn conditions for controlled CI/CD access.
  • Configured multi-region ECR replication and Terraform-driven infrastructure for Lambda, IAM roles, and scheduled rotations, ensuring high availability and seamless deployments.
  • Designed and orchestrated modular Airflow DAGs to ingest change data streams from multiple source databases into Snowflake, enabling near real-time data availability for analytics teams.
  • Built robust end-to-end ETL pipelines (raw
  • staging
  • reporting) with data validation, bad-record handling, and MERGE-based upsert logic, improving data accuracy and reducing manual reconciliation efforts.
  • Implemented data quality checks, snapshot tracking, and historical retention mechanisms, while exposing key pipeline metrics for observability and faster troubleshooting.

Skills 0 proven through work

Also works with

Security AutomationWorkflow Design (Process Mapping)API Authentication ImplementationRoot Cause AnalysisCloud Architecture AdvisoryCode ReviewDatabase Query OptimizationApplication Performance OptimizationTechnical Solution DesignTechnical TroubleshootingRisk and Controls AssessmentDependency AnalysisRequirements AnalysisAdaptabilityWork Planning and PrioritizationProblem SolvingMentoring and Peer CoachingMaintainability DesignCloud Cost Estimation & ComparisonProcessing Pipeline OptimizationUptime ManagementSenior Engineering LeadershipCompliance ManagementProcess ImprovementCI/CD Pipeline ImplementationInfrastructure ManagementDependency ManagementAWS ECR UsageDockerJWT Authentication Filter ImplementationBusiness Logic ImplementationDatabase Password Rotation ImplementationAudit Log ImplementationError HandlingTransaction ReconciliationBusiness Rules VerificationForeign Key ImplementationData Repetition PreventionData ValidationCloud Security ManagementEnvironment Consistency ManagementMulti-Region DeploymentAWS IAM & Access ManagementAPI IntegrationAWS Athena UsageIncremental Data ProcessingTabular Data Cleaning & ValidationAWS GlueParquet File Format UsageData TransformationWorking with CSV FilesAWS RDS ConfigurationData Architecture DesignData Integrity ManagementScalable System DesignTask AutomationDebezium UsageSlowly Changing Dimensions ImplementationMedallion Architecture DesignBatch Processing ImplementationStreaming Data ProcessingExploratory Data Analysis (EDA)ETL Pipeline DevelopmentIaC Principles & PracticeTerraformAWS Secrets Manager UsageAmazon Redshift UsageAWS LambdaAmazon S3Apache KafkaPySparkApache Airflow OrchestrationSQL ProgrammingPython DevelopmentData Engineering

Proof of Work

Proof of Work

Harshal shares this with people who ask. You'll hear back either way.

Education

B.Tech, Information Technology

Chameli Devi Group of Institutions, Indore · 2019 — 2023

Contact details

Contact details

Harshal shares this with people who ask. You'll hear back either way.

What drives their work

The Architect — Builds what holds

The Architect

You design and construct robust, scalable systems from the ground up.

Harshal's superpower

You excel at architecting and implementing secure, reliable, and efficient data platforms that deliver measurable business impact.

How Harshal works

You are best suited for roles that require designing, building, and optimizing large-scale data platforms, with opportunities for technical leadership and mentorship.

Ask about this candidate

AI responses may contain errors. Proof status and source information are provided by Proof of Skill.

A Proof CV — one profile, kept current, with the proof attached. Back to the portal