Eduardo Najibe

• Replaced a 2-hour Alteryx CSV process with a 15-minute Airflow DAG using Python, SQL, and dbt to extract data from the Brazil Central Bank API and perform incremental transformations, loading it into BigQuery.
• Led the integration of Google Datastream for real-time data replication, ensuring seamless synchronization between production databases and BigQuery, reducing data latency and improving the timeliness of business insights across various teams.
• Built an Airflow DAG to automate the transfer of files from an payment SFTP server to both Google Cloud Storage and AWS S3, eliminating 100% of daily manual effort, enabling secure cross-platform delivery, historical traceability, and disaster recovery for business-critical reconciliation data.
• Developed a Databricks workflow using Spark and Python to automate Delta table cleanup by combining vacuum operations with custom logic to detect and delete orphaned files, reducing GCS storage from 300TB to 123TB (a 59% reduction) and significantly lowering cloud storage costs.
• Implemented a CI/CD workflow with GitHub Actions to automate the deployment of Airflow DAGs from the local environment to production, reducing manual steps and improving deployment reliability.

Technologies: Databricks, BigQuery, GCS, Airflow, Docker, CI/CD, GitHub Actions, SQL, Python, PySpark, dbt, Datastream, Datafusion, Spark, Terraform, Cloud Functions, Amazon S3, Git

• LIMS Project: Led a large-scale redesign of the laboratory analysis system (LIMS) for the Quality team by consolidating 26 fragmented pipelines into a single optimized pipeline—reducing execution time by over 60%, cutting external query costs, and preventing database locks previously caused by concurrent pipeline execution, improving system stability and data reliability.
• Implemented a custom logging and alert system for Azure Data Factory pipelines, enhancing monitoring and enabling rapid troubleshooting of data transformation errors, reducing downtime by 30%.
• Migration Project: Led the migration and optimization of all SQL Server procedures from Engineering and Risk to Databricks notebooks, supporting additional departments (Controllership, Agriculture, Quality); reduced query and table maintenance time by over 27% while improving data governance and cross-team scalability.

Technologies: SQL, Python, PySpark, Databricks, Azure Data Factory, Databricks, SQL Server, Power BI, Azure DevOps, CI/CD, Azure DevOps, Git, Azure Functions

• Ostensiva’s Project: Led the “Ostensiva” project, recovering 83% of $100,000 in fraud-related disputes in credit card.
• Launched an interactive dashboard in Power BI to monitor fraud and chargeback trends in real time, helping the risk management team identify irregularities faster and reduce fraud-related losses by 15%
• Created new metrics to monitor the company’s overall approval and rejection rates, segmented by bank, gateway, anti-fraud, and acquirers. This allowed us to anticipate potential low approval scenarios and collaborate with the sales/product team to create solutions.

Technologies: Redshift, SQL, Python, Amazon S3, Power BI, Looker

• Automated Table Clustering with Python and BigQuery: Developed two Python notebooks to automate table clustering in BigQuery—one for clustering existing tables based on the most used columns in queries, and another for clustering new tables using statistical metrics like skewness, optimizing query performance and storage.
• Optimized data storage costs by implementing partitioning strategies in BigQuery, reducing query costs by 25% while maintaining high query performance for the business intelligence team.
• Built and maintained data pipelines using Google Dataform, making business-relevant tables readily available for the business teams to leverage for decision-making and analysis.

Technologies: SQL, Python, Airflow, BigQuery, GCS, Airbyte, Datastream, Git, CI/CD, Dataform, Datastream, Dataplex