Badges
avdeshmchandara has not unlocked any badges yet.
Certifications
avdeshmchandara has not earned any certificates yet.
Work Experience
Software Engineer
Wipro• February 2020 - Present
As a data expert I created and executed pyspark based ETL pipeline, created data warehouse on AWS • Worked on providing data for models by running multiple data pipelines which extract data from multiple data sources and export the required data and combine them. • Multiple legacy databricks notebooks to populate cloud data lake were refactored and optimized. • Code reusability was ensured by creating utils module which was ready to consume library. • Validation framework was created for input and output data validation. • Validation alerts were configured for data sanity failure. • Summary email with overall trend on key parameters was designed. o Databricks job configuraton with dbutils widgets written in Java/Scala o Refactoring of ETL databricks notebooks written in Python and Scala o Databricks dbutils usages and mounting to AWS S3, dynamoDB as logger o Standalone spark submit job deployment o JVM memory tuning for spark jobs via g1 gc o Reusable python library for job validations and alerts o Data pipeline configuration using databricks cluster • Fully sanitized and automated data pipeline which were storing data in different AWS cloud dataware house. • Worked on providing data for models by running multiple data pipelines which extract data from multiple data sources and export the required data. Used AWS services S3 for storage, datapipeline, (EC2, EMR) and serverless Lambda etc • Developed various needs for automation, application and monitoring tools in ElasticSearch and Kibana
Education
Chaudhary Charan Singh University
Computer Science, MS• August 1996 - August 1998