PyCon X

Florence

2nd - 5th May 2019

Create a serverless infrastructure for data collection with Python and AWS

The talk illustrates a real-world example of how to collect data from your web, mobile, server and cloud apps and then send them to third party services and tools or load them into your data warehouse.

The data collection pipeline is integrated with multiple AWS services, such as Kinesis Firehose, Lambda functions and StepFunctions; Python is used to write each module. The data workflow is fully described pointing out how to store backup correctly, manage the conditional routing (in order to allow or discard data for specific services), implement a retry strategy on failure and finally compare performance and costs for each module.

Resources: clda.co/pycon8-data-collection-blogpost

Prerequisites: Basic knowledge of AWS services


Do you have some questions on this talk?

New comment