stockwell (intermittent failures) policy change - recommend disabling tests when 150 failures are seen in 21 days

Happy new year from the stockwell team!  We have been busy triaging bugs an=
d getting the test-verify job to be fine tuned for all platforms and test s=
uites.

If you want to read about what we plan to do in the near future, you can fo=
llow along in this tracking bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=3D1428828

One change I wanted to make everyone aware of is the threshold we use to se=
t the whiteboard tag: [stockwell disable-recommended].  For the last 4 mont=
hs that has been 200 failures in 30 days.  We look at this whiteboard tag t=
wice a week and focus on those bugs- basically if there isn't signs of an u=
pcoming patch we disable the test to reduce the pain on the trees.

What I would like to change on January 15th, is the threshold so that it is=
 150 failures over 21 days.  In analyzing data over the last 4 months there=
 would have been 2 bugs which we would have disabled which ended up getting=
 fixed a week later- the rest of the bugs had active development taking pla=
ce and would have been fixed/resolved as they were normally.

The advantage of doing this is we would disable tests 9 days faster, reduci=
ng failures on the tree and more importantly fewer failures on your try pus=
hes with almost the exact same end result.

Please reply if you have concerns or other ideas we should consider.
0
jmaher
1/9/2018 8:29:37 PM
mozilla.dev.platform 6264 articles. 0 followers. Post Follow

0 Replies
23 Views

Similar Articles

[PageSpeed] 28

Reply: