Amrityanshu

+•• ••••• •••••••••••@•••••.com

Cloud Data Engineer · MANIERE SOFTWARE & TECHNOLOGIES PVT. LTD. · 5.1 yrs

As a Cloud Data Engineer with over five years of experience, I am driven by a strong ambition to achieve mastery in my field, always seeking logical and pragmatic solutions. I thrive on collaborating with teams, leveraging my resourcefulness to build efficient and robust data architectures.

Ask about Amrityanshu

Skills

Claimed

Problem SolvingTeamworkProcessing Pipeline OptimizationPrompt EngineeringModel Training ExecutionAI Solution ImplementationKnowledge Base Design & ManagementGemini AI IntegrationClaude UsageRetrieval-Augmented GenerationLarge Language Model ConceptsGenerative AI Application

Proof of Experience

Cloud Data Engineer

MANIERE SOFTWARE & TECHNOLOGIES PVT. LTD. · Oct 2023 – Present

  • Deployed and managed large-scale batch data processing pipelines using PySpark on AWS EMR to process high-volume datasets efficiently.
  • Implemented data partitioning strategies in PySpark jobs to optimize storage layout and speed up data retrieval in AWS S3 and Azure Cloud Storage.
  • Generated vector embeddings for RAG retrieval mechanisms and prepared structured clean datasets tailored for LLM fine-tuning tasks.
  • Designed and maintained DAGs in Apache Airflow and automated end-to-end multi-step Spark workflows on AWS EMR using AWS Step Functions.
  • Implemented optimized join strategies in PySpark to process relational data stored in S3 and eliminated shuffle bottlenecks through custom memory tuning.
  • Deployed PySpark jobs on AWS EMR clusters provisioned with custom EC2 instance families and auto-scaling rules for optimal performance and cost reduction.
  • Authored optimized AWS Hive queries for ad-hoc and scheduled analytical reporting over Spark-processed datasets residing in AWS S3.
  • Monitored EMR cluster health, resource allocation, and job execution logs to proactively resolve memory leaks and driver/worker failures.
  • Built automated data validation checks and quality control mechanisms within Spark ETL pipelines to maintain high data integrity.

Data Engineer

BT GLOBAL BUSINESS SERVICES PVT. LTD. · Aug 2021 – Sep 2023

  • Designed PySpark batch applications utilizing Spark SQL and DataFrames to perform complex business logic and aggregations over enterprise datasets.
  • Utilized Spark DataFrame persistence (`persist()`/`cache()`) and broadcast mechanisms to reduce processing overhead and improve query response time.
  • Optimized Airflow worker pool configurations and task execution logic to scale orchestration across high-concurrency environments.
  • Configured and scheduled automated Apache Sqoop jobs for full and incremental data transfers between MySQL/SQL Server databases and Hadoop/S3.
  • Processed semi-structured and serialized datasets using specialized Hive formats including Avro, Parquet, and ORC.
  • Orchestrated daily automated Spark pipelines using AWS Step Functions and configured EMR steps for reliable sequential processing.

Proof-of-Work

No completed sessions yet. Anything Amrityanshu finishes shows up here with the evidence behind it.

Proof of Education

Bachelor of Technology, Bachelor of Technology (B.Tech)

Bharati Vidyapeeth College of Engineering · 2017 — 2021

Secondary Education, Class 12th

Vivekanand School · 2016 — 2016

Secondary Education, Class 10th (Secondary)

Vivekananda Public School · 2014 — 2014

A Proof CV — one profile, kept current, with the proof attached. Back to the portal