SETA - Search for Extraneous Test Automation

Every day our test automation finds failures! Do we need all 400+ desktop unittest jobs for every push?

Right now we skip many jobs due to coalescing, SETA aims to do this in a smarter way.

With some analysis of historical data we can find the minimal set of jobs required to run in order for us to catch every single failure!

Raw Data
Source Code

While it is true that past performance is not an indicator of future performance, we can use historical data to make smarter decisions.

We will eventually require new jobs to catch some new failures. Below is a calendar that shows the dates (bolded) where we would miss a failure and require us to change the jobs.