AutRmatic Reporting: billions of internet measurements, hundreds of reports and one repository to rule them all

Botan Agin - Data Analyst at SamKnows
Michael Stevens - Data Analyst at SamKnows
In this presentation, we will talk about how we structure our regularly‐scheduled reports as standardized entities within a monorepo. We will explain how this approach reduces the latency in setting up a report, makes it easier for new team members to contribute, and lets us uphold standards while retaining the flexibility to deliver work in diverse formats with a range of complexity levels and opportunities for man‐ ual intervention. We will go into detail on specific workflows that take the terabytes of data collected by SamKnows from cloud and on‐premises data sources, process them into an R Markdown document, formatted spreadsheet, and raw CSV output, and distribute them through cloud file storage, FTP servers, email, Slack and more