Adriano Ferraro

Responsible for helping data teams to build data pipelines that are scalable and fault-tolerant.

• Development of a data pipeline for integrating CSV files with the dynamic layout in batch, whichresulted in the delivery of data without reoccupation with the new columns with information placed inthe files, providing an evolution of the table structure without our process breaking. Using Azure DataFactory, Databricks, Delta, Azure Data Lake, Python and SQL.
• Development of a data pipeline to transform a table that contains a column with several fieldsand values within it, this column should be transformed into several other columns. The delivery ofthese transformed data resulted in a speed gain for the business area that previously took 3 hours tomake this separation. With automation, we do it in less than 15 minutes. I used technologies such asBigQuery, Composer, Python, and SQL.
• Development of a data platform that automates the entire data integration process in the Data Lake,which resulted in the delivery of data in production by the analytical engineer more quickly. It used totake 120 hours to deliver a data ingestion in production now that time has dropped to 48 hours with thenew data platform. Using Google Cloud Storage, Composer, BigQuery, Cloud Functions, Python andSQL.

• Developed data pipelines for integrating Salesforce API data into Data Lake, which resulted in thedelivery of automated information to the business area, which previously needed to extract the data andmanually put it into a spreadsheet. This process took about 4 hours every day. With the automation, thedata was loaded and transformed in 20 minutes. Using Python, SQL, Composer, DataProc and Spark.
• Developed a system to monitor the quality of data that landed in the Data Lake, which resulted in thedelivery of information with quality and consistency by the Analytics Engineer Using Python, Pandas,SQL, Composer, BigQuery, Cloud Functions, Cloud Storage, and Power BI.

Responsible for helping data teams to build a new data platform on GCP.

• Developed a new data platform to support all business areas of the company, which resulted insavings of 2 million per year with infrastructure costs and a reduction from 4 days to two hours inprocessing customer bases. Using Talend, Airflow, BigQuery, Cloud Storage
• Developed an automation to handle files that arrived with the wrong header from healthcare providersbefore the process was executed in 4 hours because data platform operators needed to normalize thefile header manually, this automation resulted in a time-saving process now running in 4 minutes. UsingPython and Pandas.