Badges
Certifications
kraer has not earned any certificates yet.
Work Experience
Data Engineer
Tiger analytics• May 2022 - Present
Responsible for deploying pipelines to cleanup data in Bronze layer and ingesting it in Silver layers using Delta Lake in Azure Databricks. Stack: • Python • Spark • Airflow • Azure (Databricks, ADF, Delta Lake) • NoSQL (CosmoDB) • SQL
Data Engineer
Santander• August 2021 - May 2022
Responsible for deploying, maintaining, integrating and developing all of the big data platform services for Santander Brazil. Stack: • Apache NiFi • SQL (MySQL, Impala) • Cloudera Hadoop • Shell Scripting • Python and Spark • Apache Kafka • Azure Cloud • Databricks • Apache Airflow • NoSQL (ScyllaDB & HBase)
Data Engineer
Engineering Ingegneria• April 2020 - August 2021
Project done in Sabesp, which is a Brazilian water and waste management company in the state of Sao Paulo, it basically provides water and sewage services to residential, commercial and industrial users in São Paulo. Basically, my goal there was to remodel all the data from Sabesp, which was in CSV, to parquet and ingest them in our Data Lake, then transform to ingest into our DW. To reach that goal, I worked with Sabesp’s Engineers and Product Owners to know what was important and what wasn't, so we could build the data structure. Then, myself with another group of Engineers from Engineering, deployed a OnPremise Hadoop Cluster to load data into HDFS as Parquets using ETLs and ELTs. We only used batch data on this project. Stack: • Hadoop • SQL (PostgreSQL) • Python and Spark • Shell Script • Data Modeling • Responsible for Ingestion and Data Processing in DW, Data Lake • Apache Airflow.
Education
Anhembi Morumbi
Computer Science, BS• August 2019 - Present
Links
Skills
kraer has not updated skills details yet.