SP
sai poorna
As an Azure Data Engineer, I approach my work with determination and meticulousness, driven by a logical and pragmatic mindset. I am dedicated to achieving mastery in my field, diligently crafting robust and efficient data solutions. My structured approach ensures high-quality outcomes, reflecting my commitment to precision and excellence.
Contact details — on requestProof of Work — on request
Experience
S
Azure Data Engineer
Signovate Technologies · Dec 2024 – Present
- Proficient Azure Data Engineer with 5+ years of experience in designing, implementing, and optimizing data solutions on Microsoft Azure.
- Expertise in developing and managing data pipelines, integrating data from multiple sources, and ensuring high-performance, secure, and scalable cloud data architectures
- Hands-on experience with Azure services such as Azure Data Factory, Azure Storage, Logic Apps, and Azure Databricks, implementing scalable data solutions.
- Proficient in developing and implementing data lake solutions using Azure Data Factory, enabling efficient data integration and transformation workflows.
- Extensive experience working with Azure Data Lake Gen2, ensuring secure and optimized data storage for large-scale data operations.
- Hands-on experience in creating and configuring Azure Self-hosted Integration Runtime to connect on-premises data sources with Azure cloud services.
- Solid hands-on experience writing Spark SQL queries for data processing, aggregation, and analysis in a distributed environment.
- Expertise in designing and developing data flows in Delta Lake, enhancing the performance and reliability of data operations.
- Proficient in data preparation and processing using SQL and Spark SQL, streamlining data workflows for data engineers and analysts.
- Experienced in creating clusters in Databricks to support large-scale data processing and analytics tasks.
- Skilled in developing and scheduling jobs in Databricks for automating data pipelines and ensuring seamless data flow.
- Strong expertise in creating Technical Design Documents, debugging code, and performing unit testing to ensure high-quality deliverables.
- Extensive experience in developing stored procedures, functions, exception handling, views, dynamic SQL, and complex queries using SQL Server and T-SQL.
- Good understanding of customer needs and industry trends to deliver tailored solutions.
- The objective of this project is to reduce the costs and risks involved in preparing reports from data coming from legacy systems.
- The solution implements a data warehouse to improve reporting timeliness and reduce the gap between business and customers.
- Understanding the client's business, requirement gathering, and creating a design document.
- Coordinate with data mapping, data cleansing and DV teams to fulfill requirements.
- Process data from source to Azure Synapse Analytics as tables.
- Develop Azure Data Factory pipelines using Copy Activity.
- Achieve business transformation using Azure Data Factory/PySpark Databricks notebooks using Delta tables.
- Automate data validation and quality checks using Python and Pandas.
- Develop PySpark scripts to move data from Data Lake to Azure SQL DB.
- Create Databricks notebooks for data transformation using PySpark and schedule those jobs.
- Use Microsoft Azure services including Azure Data Factory, Databricks, Functions, Key Vault, Data Lake and Azure Logic Apps.
- Use Generative AI (GenAI) to assist with SQL/PySpark development, data analysis, troubleshooting pipeline/code issues, and preparation of technical documentation.
- Conduct unit testing on pipeline functionality and code base and participate in internal UTA testing.
- Prepare documents for production move and attend daily client calls for query clarification
- Lincoln Financial Distributors data warehouse providing wholesalers and sales management access to accurate sales-compensation data and faster turnaround for timely reports through Territory Visualizer reporting.
- Understanding the client's business, requirement gathering, and creating a design specification document.
- Create mapping sheets based on requirements provided by the client.
- Ingest data from Blob Storage (Excel files) to SAP Foundation layer Delta tables using ADF and trigger pipelines.
- Ingest data from SAP Foundation Delta layer to trusted Delta layer using incremental loading in ADF and perform transformation logic in the trusted layer.
- Ingest data from trusted Delta layer to unified Delta layer (dimension and fact tables) using SCD Type 1 logic in ADB and perform unit testing/data validation.
- Closely work with Power BI developers on data-related issues.
- Use cloud connectors including Azure Synapse (SQL DW), Snowflake, AWS, Azure Data Lake Store, Azure Blob Storage, Oracle and SQL Server.
- Navistar telematics monitoring solution supporting batch processing, near real-time IoT data analysis and Power BI dashboards for device health and maintenance monitoring.
- Azure Data Factory processed batch data from six different cloud and on-premises sources every two hours.
- Data wrangling on near real-time IoT data for Azure Synapse for data analysis.
- Power BI dashboards with 20+ reports covering device health and maintenance monitoring.
- Involved as Technical Consultant and translated functional design into technical design.
- Understanding the client's business, requirement gathering, and creating design documents.
- Develop Azure Data Factory pipelines to move data from on-premises databases to Azure Data Lake Storage and Azure SQL Database.
- Develop Azure Data Factory pipelines using Copy Activity and Stored Procedure Activity.
- Created solutions using Databricks Spark/HDInsight batch to extract and load data on Azure Data Lake Store and used on-premises data gateway to load data into Azure Analysis Service.
- Achieve business transformation using Azure Data Factory/PySpark Databricks notebooks using Delta tables.
- Develop PySpark scripts to move data from Data Lake to Azure SQL DB.
- Create Databricks notebooks for transformation using PySpark and schedule those jobs.
- Use Microsoft Azure Cloud with Azure Data Factory, Databricks, Functions, Key Vault, Data Lake, Azure Logic Apps and CI/CD.
- Conduct unit testing on pipeline functionality and code base.
- Automate deployment using Azure DevOps CI/CD methodology.
B
Azure Data Engineer
Bosch · Nov 2023 – Nov 2024
B
Data Engineer
BSW SOFT · Apr 2021 – Feb 2023
Skills 0 proven through work
Also works with
Accurate Data DeliveryEnd-to-End Solution OwnershipTechnical DiscussionTechnical Knowledge SharingTeamworkIndependent WorkIncident ManagementTeam SupportTechnical TroubleshootingProcess ImprovementProduction Deployment VerificationCron Job SchedulingApproval ManagementDependency ManagementData Pipeline ParameterizationEnvironment-Specific Configuration ManagementApplication DeploymentCode ReviewBranching StrategyVersion Control with GitHubAzure DevOpsTelematics System EngineeringUnit TestingCI/CD Pipeline ImplementationAzure Logic Apps UsageAzure Key Vault UsageAWS LambdaAzure DeploymentAzure SQL UsagePySparkData MigrationTechnical Solution DesignData EngineeringBusiness Logic ImplementationData PreprocessingData MonitoringSQL Validation Template CreationSlowly Changing Dimensions ImplementationTabular Data Cleaning & ValidationError HandlingSQL ProgrammingPyCharm UsageService Layer Architecture DesignAPI IntegrationBusiness AcumenTesting under Varied ConditionsSolution ImplementationLog AnalysisRoot Cause AnalysisProblem SolvingData Partitioning and Segregation DesignIncremental Data ProcessingData Architecture DesignLarge Dataset HandlingAzure Synapse Analytics UsageData LoadingPython DevelopmentDatabricks UsageAzure Data Lake UsageData Copy Activity ImplementationETL Pipeline DevelopmentAzure Data Factory Pipeline DevelopmentWorkflow CoordinationData ValidationTechnical DocumentationData TransformationSystem AnalysisRequirements Analysis
Proof of Work
Proof of Work
sai shares this with people who ask. You'll hear back either way.
Education
B.Tech, ECE
JNTU Anantapur University
Contact details
Contact details
sai shares this with people who ask. You'll hear back either way.
What drives their work
The Builder
Turns data requirements into robust, operational pipelines.
sai's superpower
You excel at constructing and maintaining high-performance data pipelines that deliver reliable and accurate information.
How sai works
You are well-suited for teams focused on building and maintaining robust data infrastructure and pipelines in cloud environments.
Ask about this candidate
AI responses may contain errors. Proof status and source information are provided by Proof of Skill.
A Proof CV — one profile, kept current, with the proof attached. Back to the portal