Inheriting annotations into included reftest.list files

This will probably come as a surprise to many (as it does to me each
time I rediscover it), but if, in a reftest.list file, you do
something like this (real example from [1]):

skip-if(browserIsRemote) include ogg-video/reftest.list

this may not do what you expect. My expectation, at least, is that the
entire ogg-video/reftest.list file is skipped if the "browserIsRemote"
condition is true.

However, what *actually* happens is that the skip-if expected status
(which is EXPECTED_DEATH [2]) gets "inherited" down into the included
reftest.list, and if there's a higher-valued [3] annotation on a
reftest inside that included list, then that will "win". So in
practice, this means that any reftest inside ogg-video/reftest.list
that has a fuzzy() annotation, or a fuzzy-if(x) annotation where x is
true, will still run.

This seems like a very unexpected result, and looking through some of
the cases where this happens it's not at all clear to me if this was
intentional, or if these tests are just running accidentally because
nobody realized this would happen.

I'm happy to make changes to the reftest manifest parser to remove
this footgun (most likely by disallowing annotations on include
statements) but we would need to go through each instance of this in
the reftest.list files and fix things up so that the tests that are
running are in line with the expectations of whoever added/owns the

I wanted to open this up for discussion in case people have any
thoughts on it before I move forward and try to clean this up.

1/10/2018 3:49:22 PM 6481 articles. 0 followers. Post Follow

0 Replies

Similar Articles

[PageSpeed] 11