Badges
Certifications
chanddeepak has not earned any certificates yet.
Work Experience
Software Engineer
Nagarro• March 2019 - Present
Working in Big Data Project which involves replacing the existing client tool Xcalar with a Spark for processing huge amount of data and to decrease the time of processing the data. It also involves the tracking of a transformation of data at various stages in a pipeline. • Processing of large Structured data in Spark • Implemented data validation pipeline in PySpark and Python. • Implemented logic for validating different types of columns in data. • Implemented logic for validating Date and Numeric fields in Scala. • Optimization of Spark code to process huge amount of data in minimal time. • Worked on airflow dags which are responsible for orchestrating spark jobs. • Developed Airflow Plugin to prioritize airflow dags. • Created Airflow Dag for handling Dag Prioritization request for approval or rejection. • Created Custom Airflow Operator for checking response of Dag Prioritization Request. • Developed Flask Api’s for calculation of resources(memory, cores) required for spark jobs. • Developed Flask Api’s for saving resources and time taken by Spark jobs to Postgres DB. • Deployed Flask Api’s on Kubernetes.
Education
NIEC, Delhi (Northern India Engineering College)