Data Engineer
Covalense Global Pvt Ltd · Oct 2025 – Apr 2026
- Worked on an existing Enterprise Data Lake Analytics and Storage Optimization project using Azure Databricks, PySpark, SQL, and ADLS Gen2 to support enterprise reporting and storage analysis.
- Developed and supported PySpark-based ETL workflows to process weekly Azure Blob Inventory data stored in ADLS Gen2 and prepare curated datasets for business reporting.
- Applied data transformations, filtering, joins, aggregations, validation, and reconciliation to generate clean and reliable datasets for analytics.
- Calculated storage metrics, analyzed storage utilization trends, identified inactive data paths, and prepared reporting datasets to support storage optimization initiatives.
- Used SQL for data validation, reconciliation, and analysis to ensure data accuracy before publishing datasets for reporting.
- Prepared curated datasets consumed through Azure Synapse Analytics and Power BI dashboards for storage analytics and business reporting.
- Supported production activities by monitoring Databricks jobs, troubleshooting failures, rerunning failed jobs when required, validating processed data, and resolving production issues.
- Monitored workflow executions using Azure Monitor, analyzed execution logs, and supported issue resolution to maintain reliable data processing.
- Supported CI/CD deployment activities and version control using Azure DevOps and Git for Databricks notebooks and deployment changes.
- Worked with Unity Catalog and RBAC concepts to support secure, governed, and controlled access to enterprise datasets.
- Collaborated with business users, reporting teams, and project stakeholders in an Agile environment to deliver enhancements and support reporting requirements.

