Data Engineer
Sai Technologies Pvt Ltd · Apr 2023 – Present
- Designed and deployed DLT pipelines to automate ETL workflows, enabling real-time data ingestion, transformation, and improved data quality.
- Automated DLT deployments through CI/CD pipelines using Azure DevOps and Databricks Repos to enforce version control and promote changes across environments.
- Utilized Databricks Autoloader to efficiently ingest large-scale cloud data into Delta tables, supporting incremental loads and reducing latency.
- Implemented CDC frameworks using Delta Lake to track source-system changes, ensuring timely updates and consistent data across analytics environments.
- Optimized Delta tables through strategic partitioning, Z-Ordering, and vacuuming, boosting query performance and lowering storage costs.
- Built scalable ETL pipelines using ADF, Azure Databricks, and Python to cleanse, transform, and load data into Azure Data Lake and Azure Synapse Analytics.
- Developed real-time ingestion pipelines with Azure Stream Analytics and Event Hubs to process IoT, sensor, and web-log data for near-real-time insights.
- Automated data integration workflows via ADF scheduling and Azure Logic Apps, reducing manual intervention and operational overhead.
- Integrated data from Azure Blob Storage, SQL Server into Azure Data Lake Storage to support unified processing and analytics.
- Managed large-scale data storage solutions in Azure Data Lake and Azure SQL Database, optimizing, partitioning, indexing and compression for faster queries.
- Designed star schema models to support BI workloads, ensuring optimal performance for complex analytical queries.
- Led large-scale data processing using Apache Spark in Databricks, leveraging PySpark for transformations and aggregations, improving processing speed by 30%.
- Implemented Delta Lake features including ACID transactions and schema evolution to enhance data reliability and governance.
- Monitored and tuned pipeline performance using ADF Monitoring and Databricks metrics to ensure high-availability and efficient execution.
- Collaborated with data scientists, analysts, and architects to define data requirements and deliver insights that enabled data-driven decision-making.
- Partnered with DevOps teams to implement CI/CD pipelines in Azure DevOps, automating deployment and integration of data workflows.
- Utilized Azure Monitor and Log Analytics to track pipeline health, detect anomalies, and proactively address performance issues.
- Enforced data governance policies with Unity Catalog in streaming environments, incorporating data quality checks, schema validation, and error-handling controls.
- Defined data retention and archiving policies in Azure Data Lake Analytics to ensure compliant lifecycle management of real-time data streams.
- Built production-grade data transformations and stored results in managed Unity Catalog tables.

