Build Stuff 2018

ETL pipelines ingest data from a variety of sources and must handle incorrect, incomplete or inconsistent records and produce curated, consistent data that delivers invaluable insight into the customers behaviour
 
In this talk Sam Elamin will relate his real life experience in building robust data processing pipelines powered by Spark that balances the considerations of extreme performance, speed of development, and cost of maintenance.

Sam will walk through building a Datalake using best practice patterns and running hundreds of jobs in parallel using open source tools including Apache Spark, Apache Airflow and Presto that underline systems which are dealing with £100,000 worth of transactions every hour, and more importantly will also highlight the pitfalls to avoid while providing scalable and reliable big data solutions

If you are curious about becoming a data engineer or fancy a move to big data then this is the talk for you!