Azure data engineer
DP World · May 2024 – Present
As an Azure data engineer, I am driven by a strong sense of accountability and ambition to achieve mastery in crafting robust data solutions. My approach is deeply rooted in logic and pragmatism, allowing me to apply insightful analysis to complex challenges and deliver effective, high-quality outcomes.
- Open to Bangalore · Hyderabad · San Francisco
- Remote, Hybrid, On-site
- In 15 days
Skills
Claimed
Artifact ManagementSpicebox UsageSchema Drift ManagementPerformance TestingRoot Cause AnalysisProcessing Pipeline OptimizationDecision MakingAdaptabilityDependency AnalysisSystem AnalysisRequirements AnalysisProcess Improvement
Proof of Experience
Azure Data Engineer
Cenit Technologies Pvt Ltd (Client: DP World (Multimodal Logistics)) · May 2024 – Present
- Built end-to-end ingestion workflows for multimodal logistics data (shipment, container, vessel, and terminal operations feeds), processing TRS files from Azure DevOps, and developed a Master Config framework in Databricks to automate table creation and schema mapping, cutting new-source onboarding effort from days to hours.
- Orchestrated Azure Data Factory (ADF) pipelines executing Databricks notebooks across Landing, Bronze, and Silver layers (Medallion Architecture) on ADLS Gen2, processing 20+ daily pipelines with 99%+ scheduled-run success rate.
- Implemented data validation during RAW-to-Landing movement and Delta Lake processing across layers, catching schema drift and bad records before downstream loads and reducing data-quality incidents in reporting by ~40%.
- Created database tables and the Azure SQL Database serving layer from Entity-Relationship Diagrams (ERDs), ensuring accurate mapping of entities, attributes, and relationships for shipment and container analytics.
- Collaborated with the BigID Data Governance team to identify PII fields and implement column-level encryption, ensuring compliance with security and organizational data-governance standards.
- Built a RAG workflow on prepared logistics datasets using Azure OpenAI, LangChain, embeddings, and vector search for document Q&A, and applied prompt engineering for summarization and data-catalog assistance.
- Managed Azure DevOps CI/CD processes, including pull requests, OPS PR configurations, production deployments, and release validation, cutting per-release deployment effort by ~30%.
- Monitored ADF and Databricks pipelines across dev and production, troubleshooting failures and tuning Spark configurations to cut job runtimes by ~25% on high-volume Delta MERGE workloads.
- Contributed across analysis, design, development, testing, deployment, and production support phases while documenting stored procedures, pipelines, and test scenarios for stakeholders.
Azure Data Engineer
Capgemini Technology Services Pvt. Ltd. (Client: HSBC) · Sep 2021 – May 2024
- Developed ingestion workflows for banking and financial data (transactions, customer, KYC, and regulatory reporting feeds), processing TRS files from DevOps and building a Master Config framework in Databricks to automate table creation and schema mapping across 50+ tables.
- Automated table creation, schema mapping, and ETL orchestration using Azure Data Factory pipelines to run Databricks notebooks across Landing, Bronze, and Silver layers, processing 10+ GB of banking data daily.
- Implemented Medallion architecture on ADLS Gen2 and maintained data integrity using Delta Lake ACID transactions and time travel for auditability.
- Prepared structured and semi-structured datasets using PySpark for downstream AI/ML consumption, supporting teams experimenting with Azure OpenAI and embedding-based semantic search.
- Partnered with the BigID Data Governance team to identify PII fields across customer and transaction datasets and implemented column-level encryption, supporting compliance with banking security and regulatory standards.
- Managed CI/CD lifecycle in Azure DevOps, including pull requests, production deployments, and OPS PR configurations, with zero rollback releases across the final year.
- Conducted data validation during RAW-to-Landing transitions, reconciling record counts and sums between source and target systems, and documented stored procedures, ADF pipelines, and test values for stakeholder sign-off.
Proof-of-Work
No completed sessions yet. Anything Sarath finishes shows up here with the evidence behind it.
Proof of Education
B.Tech / B.E., Electrical and Electronics Engineering (EEE)
Jawaharlal Nehru Technological University, Kakinada · 2016 — 2020
A Proof CV — one profile, kept current, with the proof attached. Back to the portal