Associate Data Engineer
Growsoft India · Dec 2022 – Present
- Designed, built, and maintained scalable ETL pipelines across Raw, Bronze, and Silver layers using AWS Databricks, PySpark, and Delta Lake.
- Implemented incremental data ingestion from Amazon S3 using Databricks Auto Loader with schema evolution, rescue mode, and checkpointing.
- Configured secure connectivity between Databricks and Amazon S3 using AWS IAM Roles, Unity Catalog Storage Credentials, and External Locations.
- Developed Structured Streaming pipelines (readStream) to incrementally process data from Bronze Delta tables.
- Performed data cleaning and validation using PySpark, including null handling, data type casting, duplicate removal, and data quality checks before loading data into the Silver layer.
- Loaded curated data into Silver Delta tables using foreachBatch and Delta MERGE for incremental upserts.
- Contributed to business transformations for Gold datasets using SQL and PySpark, including joins, filtering, aggregations, and window functions.
- Supported loading Gold Delta datasets into Snowflake using the COPY INTO command for analytics and business reporting.
- Worked in a project environment where Apache Airflow was used for workflow orchestration.
- Optimized Spark and Delta Lake workloads using partitioning, caching, and Databricks cluster optimization techniques.
- Worked with Unity Catalog by managing catalogs, schemas, governed Delta tables, Storage Credentials, External Locations, and implementing row, column, and table-level access controls.
- Worked with dimensional data models, including Fact Tables, Dimension Tables, Star Schema, and table relationships to support ETL development and analytical reporting.

