Azure Data Engineer
Microgate Technologies Pvt Ltd. · Jan 2025 – Apr 2026
- Designed and developed end-to-end ETL/ELT data pipelines using Azure Databricks and PySpark.
- Implemented the Medallion Architecture (Bronze, Silver, Gold) for scalable and reliable data processing.
- Ingested raw CSV data into Azure Data Lake Storage Gen2 using Auto Loader (Streaming) and Batch processing.
- Developed Bronze layer pipelines to store raw data while maintaining the source of truth.
- Built Silver layer transformations to remove duplicates, handle null values, standardize data types, and apply business rules.
- Created Gold layer datasets for business reporting, including Ticket Volume Trends, SLA Compliance, Resolution Status, and Department-wise Analytics.
- Implemented Delta Live Tables (DLT) to automate pipeline orchestration, dependency management, and data quality validation.
- Configured DLT Expectations for enforcing data quality and maintaining data lineage.
- Managed Delta Lake tables using Unity Catalog for centralized governance, metadata management, and access control.
- Optimized Spark jobs using partitioning, caching, and Delta Lake optimization techniques to improve pipeline performance.
- Developed PySpark notebooks for data ingestion, transformation, and aggregation.
- Implemented incremental file processing using Auto Loader to process only newly arrived files.
- Created reusable notebooks and parameterized pipelines for different environments.
- Developed SQL queries for data validation and business metric generation.
- Built interactive dashboards using Streamlit to monitor DLT pipelines and visualize silver and gold layer data.

