Data Engineer(OPS)
TCS Ltd. · Jan 2021 – Present
- Worked on multiple tools like Qlik R, Databricks, Talend, Airflow and Incorta in order to support and monitor the end-to-end data pipeline and ensure the data completeness at the target system
- To timely resolve the errors occurring in the data pipeline by understanding the logs and taking actions accordingly
- To ensure smooth running of all the jobs in the pipeline daily and report any break in the flow of the jobs so that timely action can be taken by the concerned developers
- To provide inputs to the client on the possible improvements in the operational support process
- To continuously keep up with the current developments in the pipeline so as to ensure effective monitoring of the same and find out the gaps if any.
- Enhancing the current monitoring process by automating it using SQL queries, Excel reports
- Effectively used SQL in Data bricks notebooks and in Incorta to compare the source and target data
- Provide analysis and recommendations to the DE team based on the Light up alerts which is used as a data quality tool across the complete data pipeline.
- Alerts are related to anomalies in the data, like data delay and data volume.
- To monitor the dashboards in the target system (Incorta) and report in case of any data abnormalities.

