The Elixir Of Web Scraping

Elixir & Phoenix enthusiast; FullStack Engineer with 10+ years of experience; CTO at https://www.craftingsoftware.com/ happy husband and father.

This talk is about how we implemented a simple crawler for an MVP for one of our clients. I will talk about how we used a Stream to implement the crawler, the Eager vs Lazy topic, applying concurrency to improve performance, understanding the difference between concurrency and parallelism and how it helped us optimise resources. I will also allocate some time to cover ethical web crawling.