
Azure Data Engineer
Signovate Technologies · Dec 2024 – Present
- Proficient Azure Data Engineer with 5+ years of experience in designing, implementing, and optimizing data solutions on Microsoft Azure.
- Expertise in developing and managing data pipelines, integrating data from multiple sources, and ensuring high-performance, secure, and scalable cloud data architectures
- Hands-on experience with Azure services such as Azure Data Factory, Azure Storage, Logic Apps, and Azure Databricks, implementing scalable data solutions.
- Proficient in developing and implementing data lake solutions using Azure Data Factory, enabling efficient data integration and transformation workflows.
- Extensive experience working with Azure Data Lake Gen2, ensuring secure and optimized data storage for large-scale data operations.
- Hands-on experience in creating and configuring Azure Self-hosted Integration Runtime to connect on-premises data sources with Azure cloud services.
- Solid hands-on experience writing Spark SQL queries for data processing, aggregation, and analysis in a distributed environment.
- Expertise in designing and developing data flows in Delta Lake, enhancing the performance and reliability of data operations.
- Proficient in data preparation and processing using SQL and Spark SQL, streamlining data workflows for data engineers and analysts.
- Experienced in creating clusters in Databricks to support large-scale data processing and analytics tasks.
- Skilled in developing and scheduling jobs in Databricks for automating data pipelines and ensuring seamless data flow.
- Strong expertise in creating Technical Design Documents, debugging code, and performing unit testing to ensure high-quality deliverables.
- Extensive experience in developing stored procedures, functions, exception handling, views, dynamic SQL, and complex queries using SQL Server and T-SQL.
- Good understanding of customer needs and industry trends to deliver tailored solutions.
- The objective of this project is to reduce the costs and risks involved in preparing reports from data coming from legacy systems.
- The solution implements a data warehouse to improve reporting timeliness and reduce the gap between business and customers.
- Understanding the client's business, requirement gathering, and creating a design document.
- Coordinate with data mapping, data cleansing and DV teams to fulfill requirements.
- Process data from source to Azure Synapse Analytics as tables.
- Develop Azure Data Factory pipelines using Copy Activity.
- Achieve business transformation using Azure Data Factory/PySpark Databricks notebooks using Delta tables.
- Automate data validation and quality checks using Python and Pandas.
- Develop PySpark scripts to move data from Data Lake to Azure SQL DB.
- Create Databricks notebooks for data transformation using PySpark and schedule those jobs.
- Use Microsoft Azure services including Azure Data Factory, Databricks, Functions, Key Vault, Data Lake and Azure Logic Apps.
- Use Generative AI (GenAI) to assist with SQL/PySpark development, data analysis, troubleshooting pipeline/code issues, and preparation of technical documentation.
- Conduct unit testing on pipeline functionality and code base and participate in internal UTA testing.
- Prepare documents for production move and attend daily client calls for query clarification
- Lincoln Financial Distributors data warehouse providing wholesalers and sales management access to accurate sales-compensation data and faster turnaround for timely reports through Territory Visualizer reporting.
- Understanding the client's business, requirement gathering, and creating a design specification document.
- Create mapping sheets based on requirements provided by the client.
- Ingest data from Blob Storage (Excel files) to SAP Foundation layer Delta tables using ADF and trigger pipelines.
- Ingest data from SAP Foundation Delta layer to trusted Delta layer using incremental loading in ADF and perform transformation logic in the trusted layer.
- Ingest data from trusted Delta layer to unified Delta layer (dimension and fact tables) using SCD Type 1 logic in ADB and perform unit testing/data validation.
- Closely work with Power BI developers on data-related issues.
- Use cloud connectors including Azure Synapse (SQL DW), Snowflake, AWS, Azure Data Lake Store, Azure Blob Storage, Oracle and SQL Server.
- Navistar telematics monitoring solution supporting batch processing, near real-time IoT data analysis and Power BI dashboards for device health and maintenance monitoring.
- Azure Data Factory processed batch data from six different cloud and on-premises sources every two hours.
- Data wrangling on near real-time IoT data for Azure Synapse for data analysis.
- Power BI dashboards with 20+ reports covering device health and maintenance monitoring.
- Involved as Technical Consultant and translated functional design into technical design.
- Understanding the client's business, requirement gathering, and creating design documents.
- Develop Azure Data Factory pipelines to move data from on-premises databases to Azure Data Lake Storage and Azure SQL Database.
- Develop Azure Data Factory pipelines using Copy Activity and Stored Procedure Activity.
- Created solutions using Databricks Spark/HDInsight batch to extract and load data on Azure Data Lake Store and used on-premises data gateway to load data into Azure Analysis Service.
- Achieve business transformation using Azure Data Factory/PySpark Databricks notebooks using Delta tables.
- Develop PySpark scripts to move data from Data Lake to Azure SQL DB.
- Create Databricks notebooks for transformation using PySpark and schedule those jobs.
- Use Microsoft Azure Cloud with Azure Data Factory, Databricks, Functions, Key Vault, Data Lake, Azure Logic Apps and CI/CD.
- Conduct unit testing on pipeline functionality and code base.
- Automate deployment using Azure DevOps CI/CD methodology.

