MD Danish

As a Lead Engineer, I am driven by a strong ambition to tackle complex challenges, applying a meticulous and logical approach to every project. My diligent work ethic and resourcefulness ensure I consistently deliver high-quality, effective solutions, always striving for precision and optimal outcomes.

Experience

C

LEAD ENGINEER-I

CitiusTech Healthcare Private Limited · Apr 2025 – Present

Not yet confirmed

• Designed and developed metadata-driven ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, PySpark, and Azure Synapse Analytics for scalable healthcare data integration.
• Engineered complex transformation logic using SQL Stored Procedures, aligning business rules to deliver high-quality data to target tables and file-based outputs.
• Developed SQL Stored Procedures, Functions, and transformation logic to process data across Stage, Ingress, and Egress layers.
• Built and optimized PySpark and Spark SQL notebooks for data ingestion, transformation, validation, and automated generation of client-specific Egress files.
• Automated vendor-specific file generation processes using SQL Stored Procedures, supporting multiple client facilities and improving operational efficiency and consistency.
• Engineered Ingress frameworks to ingest data from SFTP/ADLS, perform historical backfill loads, automate file validation, and ensure reliable end-to-end data processing.
• Orchestrated end-to-end data workflows using ADF, integrating Databricks notebooks, SQL procedures, and multi-source data pipelines into Azure Synapse Analytics.
• Used Azure DevOps to manage Agile/Kanban boards, track PBIs, maintain project documentation, and coordinate production support activities.
• Managed production ETL operations by monitoring pipeline health, resolving job failures, performing root cause analysis, implementing corrective actions, and ensuring reliable end-to-end data processing.
• Supported Microsoft Fabric migration by assessing ETL workflows, validating migration readiness, documenting data assets, and assisting in migration planning and testing.
• Partnered with business and cross-functional teams to deliver client enhancements, implement change requests, prepare technical documentation, and support seamless production deployments.

LEAD ENGINEER-I

CitiusTech · Apr 2025 – Present

Not yet confirmed

• • Designed and developed end-to-end Azure data pipelines using Azure Data Factory (ADF), Azure Databricks, SQL Server, and Azure Data Lake Storage (ADLS) to automate ingestion, transformation, and delivery of structured and semi-structured data.
• • Built metadata-driven ETL frameworks to process Excel, CSV, and TXT files, leveraging SQL Server stored procedures and Databricks notebooks for scalable data transformation and file generation.
• • Implemented incremental and full-load data ingestion strategies, enabling efficient processing of source files from ADLS into SQL Server target tables while maintaining data consistency and performance.
• • Developed reusable and dynamic SQL Server stored procedures for automated table creation, schema-driven data processing, and file generation, reducing manual effort from 7–8 hours to under 30 minutes.
• • Engineered Databricks-based data processing solutions for bulk file ingestion, transformation, validation, and export, improving throughput and operational efficiency across multiple data feeds.
• • Created automated data quality monitoring and notification frameworks using Databricks notebooks, proactively alerting stakeholders about unmapped fields.
• • Orchestrated complex ETL workflows through Azure Data Factory, incorporating dependency management, error handling, parameterization, scheduling, and end-to-end process automation.
• • Performed ad-hoc data analysis, data validation, and root-cause investigations on large datasets, collaborating with business stakeholders and clients to resolve data discrepancies and support decision-making.

V

Data Engineer

Veritas Comunications Private Limited · Aug 2021 – Feb 2025

Not yet confirmed

• Designed and developed ELT pipelines using Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics to process structured data.
• Built scalable ETL pipelines using Azure SQL Database, ADF, and PySpark for data ingestion, transformation, and loading into analytical data stores.
• Implemented Delta Lake and Delta Tables to provide ACID-compliant storage, schema enforcement, and reliable data processing.
• Developed and optimized PySpark and Spark SQL transformations for large-scale data processing and improved query performance.
• Designed end-to-end data solutions using Azure Data Lake Storage (ADLS Gen2) with Azure Databricks to support batch and streaming workloads.
• Developed data models and transformation logic using SQL and Python to support reporting, analytics, and business intelligence requirements.

Data Engineer

VERITAS COMUNICATIONS PRIVATE LIMITED · Aug 2021 – Feb 2025

Not yet confirmed