It is now possible to apply arbitrary tags to tests/manifests and run all tests with a given tag

As of bug 987360, you can now run all tests with a given tag for 
mochitest (and variants), xpcshell and marionette based harnesses. Tags 
can be applied to either individual tests, or the DEFAULT section in 
manifests. Tests can have multiple tags, in which case they should be 
comma delimited. To run all tests with a given tag, pass in --tag <tag 
name> to the mach command.

For example, let's say we want to group all mochitest-plain tests 
related to canvas together. First we'd add a 'canvas' tag to the DEFAULT 
section in 
https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini

[DEFAULT]
tags = canvas

We notice there is also a canvas related test under dom/media, namely:
https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541

Let's pretend it is already tagged with the 'media' tag, but that's ok, 
we can add a second tag no problem:

[test_video_to_canvas.html]
tags = media,canvas

Repeat above for any other tests or manifests scattered in the tree that 
are related to canvas. Now we can run all mochitest-plain tests with:

../mach mochitest-plain --tag canvas

You can also run the union of two tags by specifying --tag more than 
once (though the intersection of two tags is not supported):

../mach mochitest-plain --tag canvas --tag media

So far the xpcshell (./mach xpcshell-test --tag name) and marionette 
(./mach marionette-test --tag name) commands are also supported. Reftest 
is not supported as it has its own special manifest format.

Applying tags to tests will not affect automation or other people's 
tags. So each organization or team should feel free to use tags in 
whatever creative ways they see fit. Eventually, we'll start using tags 
as a foundation for some more advanced features and analysis. For 
example, we may implement a way to run all tests with a given tag across 
multiple different suites.

If you have any questions or things aren't working, please let me know!

Cheers,
Andrew
0
Andrew
3/31/2015 4:30:05 PM
mozilla.dev.platform 6651 articles. 0 followers. Post Follow

18 Replies
486 Views

Similar Articles

[PageSpeed] 16

On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
ahalberstadt@mozilla.com> wrote:

>
> =E2=80=8BFor example, we may implement a way to run all tests with a give=
n tag
> across multiple different suites.
>
>
=E2=80=8BDoes this mean that I can't do `./mach=E2=80=8B test --tag foobar`=
 to run both
xpcshell and mochitests tagged with foobar yet? Do I have to do `./mach
xpcshell-test --tag foobar` and `./mach mochitest-whatever --tag foobar`
separately?

Anyways, this looks like a very cool capability. Looking forward to using
it!
0
Nick
3/31/2015 4:52:23 PM
This sounds awesome! Is there an estimate of when we'll be able to use it
for try pushes?

On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
ahalberstadt@mozilla.com> wrote:

> As of bug 987360, you can now run all tests with a given tag for mochitest
> (and variants), xpcshell and marionette based harnesses. Tags can be
> applied to either individual tests, or the DEFAULT section in manifests.
> Tests can have multiple tags, in which case they should be comma delimited.
> To run all tests with a given tag, pass in --tag <tag name> to the mach
> command.
>
> For example, let's say we want to group all mochitest-plain tests related
> to canvas together. First we'd add a 'canvas' tag to the DEFAULT section in
> https://dxr.mozilla.org/mozilla-central/source/dom/
> canvas/test/mochitest.ini
>
> [DEFAULT]
> tags = canvas
>
> We notice there is also a canvas related test under dom/media, namely:
> https://dxr.mozilla.org/mozilla-central/source/dom/
> media/test/mochitest.ini#541
>
> Let's pretend it is already tagged with the 'media' tag, but that's ok, we
> can add a second tag no problem:
>
> [test_video_to_canvas.html]
> tags = media,canvas
>
> Repeat above for any other tests or manifests scattered in the tree that
> are related to canvas. Now we can run all mochitest-plain tests with:
>
> ./mach mochitest-plain --tag canvas
>
> You can also run the union of two tags by specifying --tag more than once
> (though the intersection of two tags is not supported):
>
> ./mach mochitest-plain --tag canvas --tag media
>
> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
> (./mach marionette-test --tag name) commands are also supported. Reftest is
> not supported as it has its own special manifest format.
>
> Applying tags to tests will not affect automation or other people's tags.
> So each organization or team should feel free to use tags in whatever
> creative ways they see fit. Eventually, we'll start using tags as a
> foundation for some more advanced features and analysis. For example, we
> may implement a way to run all tests with a given tag across multiple
> different suites.
>
> If you have any questions or things aren't working, please let me know!
>
> Cheers,
> Andrew
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Bobby
3/31/2015 5:06:47 PM
It's technically already possible by modifying the in-tree mozharness 
configs here:
https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness

However it's not easy to figure out what needs to be modified to get the 
desired results. Some sort of |mach try| like command is going to be 
worked on in Q2 to make pushing various test configurations to try 
easier, I'm not 100% sure how it will be implemented yet though.

In the meantime, if you get stuck trying to modify the mozharness 
configs, let me know and I can come up with an example patch.

-Andrew


On 31/03/15 01:06 PM, Bobby Holley wrote:
> This sounds awesome! Is there an estimate of when we'll be able to use 
> it for try pushes?
>
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt 
> <ahalberstadt@mozilla.com <mailto:ahalberstadt@mozilla.com>> wrote:
>
>     As of bug 987360, you can now run all tests with a given tag for
>     mochitest (and variants), xpcshell and marionette based harnesses.
>     Tags can be applied to either individual tests, or the DEFAULT
>     section in manifests. Tests can have multiple tags, in which case
>     they should be comma delimited. To run all tests with a given tag,
>     pass in --tag <tag name> to the mach command.
>
>     For example, let's say we want to group all mochitest-plain tests
>     related to canvas together. First we'd add a 'canvas' tag to the
>     DEFAULT section in
>     https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
>
>     [DEFAULT]
>     tags = canvas
>
>     We notice there is also a canvas related test under dom/media, namely:
>     https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
>
>     Let's pretend it is already tagged with the 'media' tag, but
>     that's ok, we can add a second tag no problem:
>
>     [test_video_to_canvas.html]
>     tags = media,canvas
>
>     Repeat above for any other tests or manifests scattered in the
>     tree that are related to canvas. Now we can run all
>     mochitest-plain tests with:
>
>     ./mach mochitest-plain --tag canvas
>
>     You can also run the union of two tags by specifying --tag more
>     than once (though the intersection of two tags is not supported):
>
>     ./mach mochitest-plain --tag canvas --tag media
>
>     So far the xpcshell (./mach xpcshell-test --tag name) and
>     marionette (./mach marionette-test --tag name) commands are also
>     supported. Reftest is not supported as it has its own special
>     manifest format.
>
>     Applying tags to tests will not affect automation or other
>     people's tags. So each organization or team should feel free to
>     use tags in whatever creative ways they see fit. Eventually, we'll
>     start using tags as a foundation for some more advanced features
>     and analysis. For example, we may implement a way to run all tests
>     with a given tag across multiple different suites.
>
>     If you have any questions or things aren't working, please let me
>     know!
>
>     Cheers,
>     Andrew
>     _______________________________________________
>     dev-platform mailing list
>     dev-platform@lists.mozilla.org <mailto:dev-platform@lists.mozilla.org>
>     https://lists.mozilla.org/listinfo/dev-platform
>
>

0
Andrew
3/31/2015 5:39:19 PM
Not yet, you still need to run the two suites separately. But we are 
working towards that. Eventually ./mach test will be able to not only 
run tags across suites, but run directories across suites and 
auto-determine which suite a test path belongs to. In other words, 
../mach test will become the norm, and running a specific test harness 
directly will become the edge case. There is some work around this 
specifically slotted in our Q2 goals.

-Andrew

On 31/03/15 12:52 PM, Nick Fitzgerald wrote:
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>>
>> ​For example, we may implement a way to run all tests with a given tag
>> across multiple different suites.
>>
>>
> ​Does this mean that I can't do `./mach​ test --tag foobar` to run both
> xpcshell and mochitests tagged with foobar yet? Do I have to do `./mach
> xpcshell-test --tag foobar` and `./mach mochitest-whatever --tag foobar`
> separately?
>
> Anyways, this looks like a very cool capability. Looking forward to using
> it!
>

0
Andrew
3/31/2015 5:44:40 PM
Not yet, you still need to run the two suites separately. But we are 
working towards that. Eventually ./mach test will be able to not only 
run tags across suites, but run directories across suites and 
auto-determine which suite a test path belongs to. In other words, 
../mach test will become the norm, and running a specific test harness 
directly will become the edge case. There is some work around this 
specifically slotted in our Q2 goals.

-Andrew

On 31/03/15 12:52 PM, Nick Fitzgerald wrote:
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>>
>> ​For example, we may implement a way to run all tests with a given tag
>> across multiple different suites.
>>
>>
> ​Does this mean that I can't do `./mach​ test --tag foobar` to run both
> xpcshell and mochitests tagged with foobar yet? Do I have to do `./mach
> xpcshell-test --tag foobar` and `./mach mochitest-whatever --tag foobar`
> separately?
>
> Anyways, this looks like a very cool capability. Looking forward to using
> it!
>

0
Andrew
3/31/2015 5:44:40 PM
I filed bug 1149670 for the mach try feature.

On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
ahalberstadt@mozilla.com> wrote:

> It's technically already possible by modifying the in-tree mozharness
> configs here:
> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>
> However it's not easy to figure out what needs to be modified to get the
> desired results. Some sort of |mach try| like command is going to be worked
> on in Q2 to make pushing various test configurations to try easier, I'm not
> 100% sure how it will be implemented yet though.
>
> In the meantime, if you get stuck trying to modify the mozharness configs,
> let me know and I can come up with an example patch.
>
> -Andrew
>
>
> On 31/03/15 01:06 PM, Bobby Holley wrote:
>
>> This sounds awesome! Is there an estimate of when we'll be able to use it
>> for try pushes?
>>
>> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
>> ahalberstadt@mozilla.com <mailto:ahalberstadt@mozilla.com>> wrote:
>>
>>     As of bug 987360, you can now run all tests with a given tag for
>>     mochitest (and variants), xpcshell and marionette based harnesses.
>>     Tags can be applied to either individual tests, or the DEFAULT
>>     section in manifests. Tests can have multiple tags, in which case
>>     they should be comma delimited. To run all tests with a given tag,
>>     pass in --tag <tag name> to the mach command.
>>
>>     For example, let's say we want to group all mochitest-plain tests
>>     related to canvas together. First we'd add a 'canvas' tag to the
>>     DEFAULT section in
>>     https://dxr.mozilla.org/mozilla-central/source/dom/
>> canvas/test/mochitest.ini
>>
>>     [DEFAULT]
>>     tags = canvas
>>
>>     We notice there is also a canvas related test under dom/media, namely:
>>     https://dxr.mozilla.org/mozilla-central/source/dom/
>> media/test/mochitest.ini#541
>>
>>     Let's pretend it is already tagged with the 'media' tag, but
>>     that's ok, we can add a second tag no problem:
>>
>>     [test_video_to_canvas.html]
>>     tags = media,canvas
>>
>>     Repeat above for any other tests or manifests scattered in the
>>     tree that are related to canvas. Now we can run all
>>     mochitest-plain tests with:
>>
>>     ./mach mochitest-plain --tag canvas
>>
>>     You can also run the union of two tags by specifying --tag more
>>     than once (though the intersection of two tags is not supported):
>>
>>     ./mach mochitest-plain --tag canvas --tag media
>>
>>     So far the xpcshell (./mach xpcshell-test --tag name) and
>>     marionette (./mach marionette-test --tag name) commands are also
>>     supported. Reftest is not supported as it has its own special
>>     manifest format.
>>
>>     Applying tags to tests will not affect automation or other
>>     people's tags. So each organization or team should feel free to
>>     use tags in whatever creative ways they see fit. Eventually, we'll
>>     start using tags as a foundation for some more advanced features
>>     and analysis. For example, we may implement a way to run all tests
>>     with a given tag across multiple different suites.
>>
>>     If you have any questions or things aren't working, please let me
>>     know!
>>
>>     Cheers,
>>     Andrew
>>     _______________________________________________
>>     dev-platform mailing list
>>     dev-platform@lists.mozilla.org <mailto:dev-platform@lists.mozilla.org
>> >
>>     https://lists.mozilla.org/listinfo/dev-platform
>>
>>
>>
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Christopher
3/31/2015 5:57:16 PM
Is there any reason we couldn't put it in the try syntax instead?

On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
ahalberstadt@mozilla.com> wrote:

>  It's technically already possible by modifying the in-tree mozharness
> configs here:
> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>
> However it's not easy to figure out what needs to be modified to get the
> desired results. Some sort of |mach try| like command is going to be worked
> on in Q2 to make pushing various test configurations to try easier, I'm not
> 100% sure how it will be implemented yet though.
>
> In the meantime, if you get stuck trying to modify the mozharness configs,
> let me know and I can come up with an example patch.
>
> -Andrew
>
>
>
> On 31/03/15 01:06 PM, Bobby Holley wrote:
>
> This sounds awesome! Is there an estimate of when we'll be able to use it
> for try pushes?
>
> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>> As of bug 987360, you can now run all tests with a given tag for
>> mochitest (and variants), xpcshell and marionette based harnesses. Tags can
>> be applied to either individual tests, or the DEFAULT section in manifests.
>> Tests can have multiple tags, in which case they should be comma delimited.
>> To run all tests with a given tag, pass in --tag <tag name> to the mach
>> command.
>>
>> For example, let's say we want to group all mochitest-plain tests related
>> to canvas together. First we'd add a 'canvas' tag to the DEFAULT section in
>> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
>>
>> [DEFAULT]
>> tags = canvas
>>
>> We notice there is also a canvas related test under dom/media, namely:
>>
>> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
>>
>> Let's pretend it is already tagged with the 'media' tag, but that's ok,
>> we can add a second tag no problem:
>>
>> [test_video_to_canvas.html]
>> tags = media,canvas
>>
>> Repeat above for any other tests or manifests scattered in the tree that
>> are related to canvas. Now we can run all mochitest-plain tests with:
>>
>> ./mach mochitest-plain --tag canvas
>>
>> You can also run the union of two tags by specifying --tag more than once
>> (though the intersection of two tags is not supported):
>>
>> ./mach mochitest-plain --tag canvas --tag media
>>
>> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
>> (./mach marionette-test --tag name) commands are also supported. Reftest is
>> not supported as it has its own special manifest format.
>>
>> Applying tags to tests will not affect automation or other people's tags.
>> So each organization or team should feel free to use tags in whatever
>> creative ways they see fit. Eventually, we'll start using tags as a
>> foundation for some more advanced features and analysis. For example, we
>> may implement a way to run all tests with a given tag across multiple
>> different suites.
>>
>> If you have any questions or things aren't working, please let me know!
>>
>> Cheers,
>> Andrew
>> _______________________________________________
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
>
0
Bobby
3/31/2015 6:12:29 PM
The main problem is that the current syntax is very inflexible. And tags 
are just one of many things we want to make easier when pushing to try 
(think of things like running directories or individual tests and all 
the debug options like --run-until-failure). A try syntax that knows how 
to handle all those cases will be just as convoluted as editing the 
mozharness configs.

Also, how things are currently set up, we'd need to get the try syntax 
to somehow modify how buildbot runs tests. This would be difficult if 
not impossible. Taskcluster may make this easier, but I wouldn't count 
on it..


On 31/03/15 02:12 PM, Bobby Holley wrote:
> Is there any reason we couldn't put it in the try syntax instead?
>
> On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>>   It's technically already possible by modifying the in-tree mozharness
>> configs here:
>> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>>
>> However it's not easy to figure out what needs to be modified to get the
>> desired results. Some sort of |mach try| like command is going to be worked
>> on in Q2 to make pushing various test configurations to try easier, I'm not
>> 100% sure how it will be implemented yet though.
>>
>> In the meantime, if you get stuck trying to modify the mozharness configs,
>> let me know and I can come up with an example patch.
>>
>> -Andrew
>>
>>
>>
>> On 31/03/15 01:06 PM, Bobby Holley wrote:
>>
>> This sounds awesome! Is there an estimate of when we'll be able to use it
>> for try pushes?
>>
>> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
>> ahalberstadt@mozilla.com> wrote:
>>
>>> As of bug 987360, you can now run all tests with a given tag for
>>> mochitest (and variants), xpcshell and marionette based harnesses. Tags can
>>> be applied to either individual tests, or the DEFAULT section in manifests.
>>> Tests can have multiple tags, in which case they should be comma delimited.
>>> To run all tests with a given tag, pass in --tag <tag name> to the mach
>>> command.
>>>
>>> For example, let's say we want to group all mochitest-plain tests related
>>> to canvas together. First we'd add a 'canvas' tag to the DEFAULT section in
>>> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
>>>
>>> [DEFAULT]
>>> tags = canvas
>>>
>>> We notice there is also a canvas related test under dom/media, namely:
>>>
>>> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
>>>
>>> Let's pretend it is already tagged with the 'media' tag, but that's ok,
>>> we can add a second tag no problem:
>>>
>>> [test_video_to_canvas.html]
>>> tags = media,canvas
>>>
>>> Repeat above for any other tests or manifests scattered in the tree that
>>> are related to canvas. Now we can run all mochitest-plain tests with:
>>>
>>> ./mach mochitest-plain --tag canvas
>>>
>>> You can also run the union of two tags by specifying --tag more than once
>>> (though the intersection of two tags is not supported):
>>>
>>> ./mach mochitest-plain --tag canvas --tag media
>>>
>>> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
>>> (./mach marionette-test --tag name) commands are also supported. Reftest is
>>> not supported as it has its own special manifest format.
>>>
>>> Applying tags to tests will not affect automation or other people's tags.
>>> So each organization or team should feel free to use tags in whatever
>>> creative ways they see fit. Eventually, we'll start using tags as a
>>> foundation for some more advanced features and analysis. For example, we
>>> may implement a way to run all tests with a given tag across multiple
>>> different suites.
>>>
>>> If you have any questions or things aren't working, please let me know!
>>>
>>> Cheers,
>>> Andrew
>>> _______________________________________________
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>>
>>

0
Andrew
3/31/2015 6:26:57 PM
On Tue, Mar 31, 2015 at 10:44 AM, Andrew Halberstadt <
ahalberstadt@mozilla.com> wrote:

> Not yet, you still need to run the two suites separately. But we are
> working towards that. Eventually ./mach test will be able to not only run
> tags across suites, but run directories across suites and auto-determine
> which suite a test path belongs to. In other words, ./mach test will become
> the norm, and running a specific test harness directly will become the edge
> case. There is some work around this specifically slotted in our Q2 goals.


The ability to `mach test <dir>` and have it run multiple suites is already
supported.
https://hg.mozilla.org/mozilla-central/file/0aa30282f496/testing/mach_commands.py#l166
0
Gregory
3/31/2015 6:31:43 PM
On 31/03/15 02:31 PM, Gregory Szorc wrote:
> On Tue, Mar 31, 2015 at 10:44 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>> Not yet, you still need to run the two suites separately. But we are
>> working towards that. Eventually ./mach test will be able to not only run
>> tags across suites, but run directories across suites and auto-determine
>> which suite a test path belongs to. In other words, ./mach test will become
>> the norm, and running a specific test harness directly will become the edge
>> case. There is some work around this specifically slotted in our Q2 goals.
>
>
> The ability to `mach test <dir>` and have it run multiple suites is already
> supported.
> https://hg.mozilla.org/mozilla-central/file/0aa30282f496/testing/mach_commands.py#l166
>

Cool, in that case getting it to run tags across suites that support 
--tag shouldn't be terribly difficult. I filed bug 1149695.
0
Andrew
3/31/2015 6:39:44 PM
Minor update. It was pointed out that other list-like manifestparser 
attributes (like head and support-files) are whitespace delimited 
instead of comma delimited. To be consistent I switched tags to 
whitespace delimitation as well.

E.g both these forms are ok:

[test_foo.html]
tags = foo bar baz

[test_bar.html]
tags =
     foo
     bar
     baz

-Andrew

On 31/03/15 12:30 PM, Andrew Halberstadt wrote:
> As of bug 987360, you can now run all tests with a given tag for
> mochitest (and variants), xpcshell and marionette based harnesses. Tags
> can be applied to either individual tests, or the DEFAULT section in
> manifests. Tests can have multiple tags, in which case they should be
> comma delimited. To run all tests with a given tag, pass in --tag <tag
> name> to the mach command.
>
> For example, let's say we want to group all mochitest-plain tests
> related to canvas together. First we'd add a 'canvas' tag to the DEFAULT
> section in
> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
>
>
> [DEFAULT]
> tags = canvas
>
> We notice there is also a canvas related test under dom/media, namely:
> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
>
>
> Let's pretend it is already tagged with the 'media' tag, but that's ok,
> we can add a second tag no problem:
>
> [test_video_to_canvas.html]
> tags = media,canvas
>
> Repeat above for any other tests or manifests scattered in the tree that
> are related to canvas. Now we can run all mochitest-plain tests with:
>
> ./mach mochitest-plain --tag canvas
>
> You can also run the union of two tags by specifying --tag more than
> once (though the intersection of two tags is not supported):
>
> ./mach mochitest-plain --tag canvas --tag media
>
> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
> (./mach marionette-test --tag name) commands are also supported. Reftest
> is not supported as it has its own special manifest format.
>
> Applying tags to tests will not affect automation or other people's
> tags. So each organization or team should feel free to use tags in
> whatever creative ways they see fit. Eventually, we'll start using tags
> as a foundation for some more advanced features and analysis. For
> example, we may implement a way to run all tests with a given tag across
> multiple different suites.
>
> If you have any questions or things aren't working, please let me know!
>
> Cheers,
> Andrew

0
Andrew
4/2/2015 9:22:35 PM
You can now add "--tag" arguments to try syntax and they will get passed to
test harnesses in your try push. Details of the implementation are in bug
978846, but if you're interested in passing other arguments from try syntax
to a test harness, this can be done by adding those arguments to
testing/config/mozharness/try_arguments.py. Note this is still rather
coarse in the sense that arguments are forwarded without regard for whether
a harness supports a particular argument, but I can imagine it being useful
in a number of cases (for instance, when testing the feature with xpcshell
and "--tag devtools", I was able to get feedback in about ten minutes
whether things were working rather than waiting for every xpcshell test to
run).

Chris

On Thu, Apr 2, 2015 at 2:22 PM, Andrew Halberstadt <ahalberstadt@mozilla.com
> wrote:

> Minor update. It was pointed out that other list-like manifestparser
> attributes (like head and support-files) are whitespace delimited instead
> of comma delimited. To be consistent I switched tags to whitespace
> delimitation as well.
>
> E.g both these forms are ok:
>
> [test_foo.html]
> tags = foo bar baz
>
> [test_bar.html]
> tags =
>     foo
>     bar
>     baz
>
> -Andrew
>
>
> On 31/03/15 12:30 PM, Andrew Halberstadt wrote:
>
>> As of bug 987360, you can now run all tests with a given tag for
>> mochitest (and variants), xpcshell and marionette based harnesses. Tags
>> can be applied to either individual tests, or the DEFAULT section in
>> manifests. Tests can have multiple tags, in which case they should be
>> comma delimited. To run all tests with a given tag, pass in --tag <tag
>> name> to the mach command.
>>
>> For example, let's say we want to group all mochitest-plain tests
>> related to canvas together. First we'd add a 'canvas' tag to the DEFAULT
>> section in
>>
>> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
>>
>>
>> [DEFAULT]
>> tags = canvas
>>
>> We notice there is also a canvas related test under dom/media, namely:
>>
>> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
>>
>>
>> Let's pretend it is already tagged with the 'media' tag, but that's ok,
>> we can add a second tag no problem:
>>
>> [test_video_to_canvas.html]
>> tags = media,canvas
>>
>> Repeat above for any other tests or manifests scattered in the tree that
>> are related to canvas. Now we can run all mochitest-plain tests with:
>>
>> ./mach mochitest-plain --tag canvas
>>
>> You can also run the union of two tags by specifying --tag more than
>> once (though the intersection of two tags is not supported):
>>
>> ./mach mochitest-plain --tag canvas --tag media
>>
>> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
>> (./mach marionette-test --tag name) commands are also supported. Reftest
>> is not supported as it has its own special manifest format.
>>
>> Applying tags to tests will not affect automation or other people's
>> tags. So each organization or team should feel free to use tags in
>> whatever creative ways they see fit. Eventually, we'll start using tags
>> as a foundation for some more advanced features and analysis. For
>> example, we may implement a way to run all tests with a given tag across
>> multiple different suites.
>>
>> If you have any questions or things aren't working, please let me know!
>>
>> Cheers,
>> Andrew
>>
>
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Christopher
4/30/2015 11:21:58 PM
Wait - you're telling me that it is now possible to limit try pushes but
not just jobs but tests within jobs?! Stop the presses: this is huge! If
used by the masses, this could drastically reduce try turnaround times and
decrease automation load and costs.

Could we encourage use of --tag by having the automation scheduler
up-weight jobs that opt in to reduced load?

On Thu, Apr 30, 2015 at 4:21 PM, Christopher Manchester <
chmanchester@gmail.com> wrote:

> You can now add "--tag" arguments to try syntax and they will get passed to
> test harnesses in your try push. Details of the implementation are in bug
> 978846, but if you're interested in passing other arguments from try syntax
> to a test harness, this can be done by adding those arguments to
> testing/config/mozharness/try_arguments.py. Note this is still rather
> coarse in the sense that arguments are forwarded without regard for whether
> a harness supports a particular argument, but I can imagine it being useful
> in a number of cases (for instance, when testing the feature with xpcshell
> and "--tag devtools", I was able to get feedback in about ten minutes
> whether things were working rather than waiting for every xpcshell test to
> run).
>
> Chris
>
> On Thu, Apr 2, 2015 at 2:22 PM, Andrew Halberstadt <
> ahalberstadt@mozilla.com
> > wrote:
>
> > Minor update. It was pointed out that other list-like manifestparser
> > attributes (like head and support-files) are whitespace delimited instead
> > of comma delimited. To be consistent I switched tags to whitespace
> > delimitation as well.
> >
> > E.g both these forms are ok:
> >
> > [test_foo.html]
> > tags = foo bar baz
> >
> > [test_bar.html]
> > tags =
> >     foo
> >     bar
> >     baz
> >
> > -Andrew
> >
> >
> > On 31/03/15 12:30 PM, Andrew Halberstadt wrote:
> >
> >> As of bug 987360, you can now run all tests with a given tag for
> >> mochitest (and variants), xpcshell and marionette based harnesses. Tags
> >> can be applied to either individual tests, or the DEFAULT section in
> >> manifests. Tests can have multiple tags, in which case they should be
> >> comma delimited. To run all tests with a given tag, pass in --tag <tag
> >> name> to the mach command.
> >>
> >> For example, let's say we want to group all mochitest-plain tests
> >> related to canvas together. First we'd add a 'canvas' tag to the DEFAULT
> >> section in
> >>
> >>
> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
> >>
> >>
> >> [DEFAULT]
> >> tags = canvas
> >>
> >> We notice there is also a canvas related test under dom/media, namely:
> >>
> >>
> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
> >>
> >>
> >> Let's pretend it is already tagged with the 'media' tag, but that's ok,
> >> we can add a second tag no problem:
> >>
> >> [test_video_to_canvas.html]
> >> tags = media,canvas
> >>
> >> Repeat above for any other tests or manifests scattered in the tree that
> >> are related to canvas. Now we can run all mochitest-plain tests with:
> >>
> >> ./mach mochitest-plain --tag canvas
> >>
> >> You can also run the union of two tags by specifying --tag more than
> >> once (though the intersection of two tags is not supported):
> >>
> >> ./mach mochitest-plain --tag canvas --tag media
> >>
> >> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
> >> (./mach marionette-test --tag name) commands are also supported. Reftest
> >> is not supported as it has its own special manifest format.
> >>
> >> Applying tags to tests will not affect automation or other people's
> >> tags. So each organization or team should feel free to use tags in
> >> whatever creative ways they see fit. Eventually, we'll start using tags
> >> as a foundation for some more advanced features and analysis. For
> >> example, we may implement a way to run all tests with a given tag across
> >> multiple different suites.
> >>
> >> If you have any questions or things aren't working, please let me know!
> >>
> >> Cheers,
> >> Andrew
> >>
> >
> > _______________________________________________
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Gregory
5/4/2015 9:12:11 PM
Sounds great! I've filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1161282 for this.

According to
https://secure.pub.build.mozilla.org/builddata/reports/reportor/daily/highscores/highscores.html,
we still have a ton of people using '-p all -u all' on try

On Mon, May 4, 2015 at 5:12 PM, Gregory Szorc <gps@mozilla.com> wrote:

> Wait - you're telling me that it is now possible to limit try pushes but
> not just jobs but tests within jobs?! Stop the presses: this is huge! If
> used by the masses, this could drastically reduce try turnaround times and
> decrease automation load and costs.
>
> Could we encourage use of --tag by having the automation scheduler
> up-weight jobs that opt in to reduced load?
>
> On Thu, Apr 30, 2015 at 4:21 PM, Christopher Manchester <
> chmanchester@gmail.com> wrote:
>
> > You can now add "--tag" arguments to try syntax and they will get passed
> to
> > test harnesses in your try push. Details of the implementation are in bug
> > 978846, but if you're interested in passing other arguments from try
> syntax
> > to a test harness, this can be done by adding those arguments to
> > testing/config/mozharness/try_arguments.py. Note this is still rather
> > coarse in the sense that arguments are forwarded without regard for
> whether
> > a harness supports a particular argument, but I can imagine it being
> useful
> > in a number of cases (for instance, when testing the feature with
> xpcshell
> > and "--tag devtools", I was able to get feedback in about ten minutes
> > whether things were working rather than waiting for every xpcshell test
> to
> > run).
> >
> > Chris
> >
> > On Thu, Apr 2, 2015 at 2:22 PM, Andrew Halberstadt <
> > ahalberstadt@mozilla.com
> > > wrote:
> >
> > > Minor update. It was pointed out that other list-like manifestparser
> > > attributes (like head and support-files) are whitespace delimited
> instead
> > > of comma delimited. To be consistent I switched tags to whitespace
> > > delimitation as well.
> > >
> > > E.g both these forms are ok:
> > >
> > > [test_foo.html]
> > > tags = foo bar baz
> > >
> > > [test_bar.html]
> > > tags =
> > >     foo
> > >     bar
> > >     baz
> > >
> > > -Andrew
> > >
> > >
> > > On 31/03/15 12:30 PM, Andrew Halberstadt wrote:
> > >
> > >> As of bug 987360, you can now run all tests with a given tag for
> > >> mochitest (and variants), xpcshell and marionette based harnesses.
> Tags
> > >> can be applied to either individual tests, or the DEFAULT section in
> > >> manifests. Tests can have multiple tags, in which case they should be
> > >> comma delimited. To run all tests with a given tag, pass in --tag <tag
> > >> name> to the mach command.
> > >>
> > >> For example, let's say we want to group all mochitest-plain tests
> > >> related to canvas together. First we'd add a 'canvas' tag to the
> DEFAULT
> > >> section in
> > >>
> > >>
> >
> https://dxr.mozilla.org/mozilla-central/source/dom/canvas/test/mochitest.ini
> > >>
> > >>
> > >> [DEFAULT]
> > >> tags = canvas
> > >>
> > >> We notice there is also a canvas related test under dom/media, namely:
> > >>
> > >>
> >
> https://dxr.mozilla.org/mozilla-central/source/dom/media/test/mochitest.ini#541
> > >>
> > >>
> > >> Let's pretend it is already tagged with the 'media' tag, but that's
> ok,
> > >> we can add a second tag no problem:
> > >>
> > >> [test_video_to_canvas.html]
> > >> tags = media,canvas
> > >>
> > >> Repeat above for any other tests or manifests scattered in the tree
> that
> > >> are related to canvas. Now we can run all mochitest-plain tests with:
> > >>
> > >> ./mach mochitest-plain --tag canvas
> > >>
> > >> You can also run the union of two tags by specifying --tag more than
> > >> once (though the intersection of two tags is not supported):
> > >>
> > >> ./mach mochitest-plain --tag canvas --tag media
> > >>
> > >> So far the xpcshell (./mach xpcshell-test --tag name) and marionette
> > >> (./mach marionette-test --tag name) commands are also supported.
> Reftest
> > >> is not supported as it has its own special manifest format.
> > >>
> > >> Applying tags to tests will not affect automation or other people's
> > >> tags. So each organization or team should feel free to use tags in
> > >> whatever creative ways they see fit. Eventually, we'll start using
> tags
> > >> as a foundation for some more advanced features and analysis. For
> > >> example, we may implement a way to run all tests with a given tag
> across
> > >> multiple different suites.
> > >>
> > >> If you have any questions or things aren't working, please let me
> know!
> > >>
> > >> Cheers,
> > >> Andrew
> > >>
> > >
> > > _______________________________________________
> > > dev-platform mailing list
> > > dev-platform@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-platform
> > >
> > _______________________________________________
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Chris
5/4/2015 10:21:34 PM
On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester wro=
te:
> You can now add "--tag" arguments to try syntax and they will get passed =
to
> test harnesses in your try push. Details of the implementation are in bug
> 978846, but if you're interested in passing other arguments from try synt=
ax
> to a test harness, this can be done by adding those arguments to
> testing/config/mozharness/try_arguments.py. Note this is still rather
> coarse in the sense that arguments are forwarded without regard for wheth=
er
> a harness supports a particular argument, but I can imagine it being usef=
ul
> in a number of cases (for instance, when testing the feature with xpcshel=
l
> and "--tag devtools", I was able to get feedback in about ten minutes
> whether things were working rather than waiting for every xpcshell test t=
o
> run).

Can you provide a sample complete try syntax that uses this? I just did a t=
ry run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag =
apz| (after adding the apz tag to a mochitest.ini file) and it still ran al=
l the mochitests instead of just the ones in the mochitest.ini file. Does i=
t only work in some scenarios?

Cheers,
kats
0
kgupta
5/4/2015 11:23:26 PM
That looks like a valid use of the feature assuming the manifest was
annotated correctly. I'm not sure what went wrong, I'll reply here when I
figure out what went wrong or file a bug and investigate further.

Chris

On Mon, May 4, 2015 at 4:23 PM, <kgupta@mozilla.com> wrote:

> On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester
> wrote:
> > You can now add "--tag" arguments to try syntax and they will get passed
> to
> > test harnesses in your try push. Details of the implementation are in bug
> > 978846, but if you're interested in passing other arguments from try
> syntax
> > to a test harness, this can be done by adding those arguments to
> > testing/config/mozharness/try_arguments.py. Note this is still rather
> > coarse in the sense that arguments are forwarded without regard for
> whether
> > a harness supports a particular argument, but I can imagine it being
> useful
> > in a number of cases (for instance, when testing the feature with
> xpcshell
> > and "--tag devtools", I was able to get feedback in about ten minutes
> > whether things were working rather than waiting for every xpcshell test
> to
> > run).
>
> Can you provide a sample complete try syntax that uses this? I just did a
> try run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag
> apz| (after adding the apz tag to a mochitest.ini file) and it still ran
> all the mochitests instead of just the ones in the mochitest.ini file. Does
> it only work in some scenarios?
>
> Cheers,
> kats
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Christopher
5/4/2015 11:43:39 PM
It looks like I inadvertently landed this change on an the wrong branch, so
we aren't pointing to it in production (although we were for a short time
when I tested last week). I'll straighten this out in the morning. Sorry
for the inconvenience.

Chris

On Mon, May 4, 2015 at 4:43 PM, Christopher Manchester <
chmanchester@gmail.com> wrote:

> That looks like a valid use of the feature assuming the manifest was
> annotated correctly. I'm not sure what went wrong, I'll reply here when I
> figure out what went wrong or file a bug and investigate further.
>
> Chris
>
> On Mon, May 4, 2015 at 4:23 PM, <kgupta@mozilla.com> wrote:
>
>> On Thursday, April 30, 2015 at 7:22:26 PM UTC-4, Christopher Manchester
>> wrote:
>> > You can now add "--tag" arguments to try syntax and they will get
>> passed to
>> > test harnesses in your try push. Details of the implementation are in
>> bug
>> > 978846, but if you're interested in passing other arguments from try
>> syntax
>> > to a test harness, this can be done by adding those arguments to
>> > testing/config/mozharness/try_arguments.py. Note this is still rather
>> > coarse in the sense that arguments are forwarded without regard for
>> whether
>> > a harness supports a particular argument, but I can imagine it being
>> useful
>> > in a number of cases (for instance, when testing the feature with
>> xpcshell
>> > and "--tag devtools", I was able to get feedback in about ten minutes
>> > whether things were working rather than waiting for every xpcshell test
>> to
>> > run).
>>
>> Can you provide a sample complete try syntax that uses this? I just did a
>> try run with |try: -b do -p linux,macosx64,win32,win64 -u all -t none --tag
>> apz| (after adding the apz tag to a mochitest.ini file) and it still ran
>> all the mochitests instead of just the ones in the mochitest.ini file. Does
>> it only work in some scenarios?
>>
>> Cheers,
>> kats
>> _______________________________________________
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
0
Christopher
5/5/2015 12:47:53 AM
This landed as the mach try command. For more details run |./mach help try|
(excerpted below) and see the discussion in bug 1149670. The highlights
here are partly automated try syntax generation for testing changes
relevant to tests in certain directories, and pushing try syntax to the try
server without the benefit of mq.

Chris


"""
Pushes the specified tests to try. The simplest way to specify tests is
by using the -u argument, which will behave as usual for try syntax.
This command also provides a mechanism to select test jobs and tests
within a job by path based on tests present in the tree under that
path. Mochitests, xpcshell tests, and reftests are eligible for
selection by this mechanism. Selected tests will be run in a single
chunk of the relevant suite, at this time in chunk 1.

Specifying platforms is still required with the -p argument (a default
is taken from the AUTOTRY_PLATFORM_HINT environment variable if set).

Tests may be further filtered by passing one or more --tag to the
command.

To run suites in addition to those determined from the tree, they
can be passed to the --extra arguent.

The command requires either its own mercurial extension ("push-to-try",
installable from mach mercurial-setup) or a git repo using git-cinnabar
(available at https://github.com/glandium/git-cinnabar).

Global Arguments:
  -v, --verbose         Print verbose output.
  -l FILENAME, --log-file FILENAME
                        Filename to write log data to.
  --log-interval        Prefix log line with interval from last message
rather
                        than relative time. Note that this is NOT execution
                        time if there are parallel operations.
  --log-no-times        Do not prefix log lines with times. By default, mach
                        will prefix each output line with the time since
                        command start.
  -h, --help            Show this help message.
  --debug-command       Start a Python debugger when command is dispatched.

Command Arguments:
  paths                Paths to search for tests to run on try.
  -n                   Print detailed information about the resulting test
                       selection and commands performed.
  -p PLATFORMS         Platforms to run. (required if not found in the
                       environment)
  -u TESTS             Test jobs to run. These will be used in place of
suites
                       determined by test paths, if any.
  --extra EXTRA_TESTS  Additional tests to run. These will be added to
suites
                       determined by test paths, if any.
  -b BUILDS            Build types to run (d for debug, o for optimized)
  --tag TAGS           Restrict tests to the given tag (may be specified
                       multiple times)
  --no-push            Do not push to try as a result of running this
command
                       (if specified this command will only print calculated
                       try syntax and selection info).
"""

On Tue, Mar 31, 2015 at 10:57 AM, Christopher Manchester <
chmanchester@gmail.com> wrote:

> I filed bug 1149670 for the mach try feature.
>
> On Tue, Mar 31, 2015 at 10:39 AM, Andrew Halberstadt <
> ahalberstadt@mozilla.com> wrote:
>
>> It's technically already possible by modifying the in-tree mozharness
>> configs here:
>> https://dxr.mozilla.org/mozilla-central/source/testing/config/mozharness
>>
>> However it's not easy to figure out what needs to be modified to get the
>> desired results. Some sort of |mach try| like command is going to be worked
>> on in Q2 to make pushing various test configurations to try easier, I'm not
>> 100% sure how it will be implemented yet though.
>>
>> In the meantime, if you get stuck trying to modify the mozharness
>> configs, let me know and I can come up with an example patch.
>>
>> -Andrew
>>
>>
>> On 31/03/15 01:06 PM, Bobby Holley wrote:
>>
>>> This sounds awesome! Is there an estimate of when we'll be able to use
>>> it for try pushes?
>>>
>>> On Tue, Mar 31, 2015 at 9:30 AM, Andrew Halberstadt <
>>> ahalberstadt@mozilla.com <mailto:ahalberstadt@mozilla.com>> wrote:
>>>
>>>     As of bug 987360, you can now run all tests with a given tag for
>>>     mochitest (and variants), xpcshell and marionette based harnesses.
>>>     Tags can be applied to either individual tests, or the DEFAULT
>>>     section in manifests. Tests can have multiple tags, in which case
>>>     they should be comma delimited. To run all tests with a given tag,
>>>     pass in --tag <tag name> to the mach command.
>>>
>>>     For example, let's say we want to group all mochitest-plain tests
>>>     related to canvas together. First we'd add a 'canvas' tag to the
>>>     DEFAULT section in
>>>     https://dxr.mozilla.org/mozilla-central/source/dom/
>>> canvas/test/mochitest.ini
>>>
>>>     [DEFAULT]
>>>     tags = canvas
>>>
>>>     We notice there is also a canvas related test under dom/media,
>>> namely:
>>>     https://dxr.mozilla.org/mozilla-central/source/dom/
>>> media/test/mochitest.ini#541
>>>
>>>     Let's pretend it is already tagged with the 'media' tag, but
>>>     that's ok, we can add a second tag no problem:
>>>
>>>     [test_video_to_canvas.html]
>>>     tags = media,canvas
>>>
>>>     Repeat above for any other tests or manifests scattered in the
>>>     tree that are related to canvas. Now we can run all
>>>     mochitest-plain tests with:
>>>
>>>     ./mach mochitest-plain --tag canvas
>>>
>>>     You can also run the union of two tags by specifying --tag more
>>>     than once (though the intersection of two tags is not supported):
>>>
>>>     ./mach mochitest-plain --tag canvas --tag media
>>>
>>>     So far the xpcshell (./mach xpcshell-test --tag name) and
>>>     marionette (./mach marionette-test --tag name) commands are also
>>>     supported. Reftest is not supported as it has its own special
>>>     manifest format.
>>>
>>>     Applying tags to tests will not affect automation or other
>>>     people's tags. So each organization or team should feel free to
>>>     use tags in whatever creative ways they see fit. Eventually, we'll
>>>     start using tags as a foundation for some more advanced features
>>>     and analysis. For example, we may implement a way to run all tests
>>>     with a given tag across multiple different suites.
>>>
>>>     If you have any questions or things aren't working, please let me
>>>     know!
>>>
>>>     Cheers,
>>>     Andrew
>>>     _______________________________________________
>>>     dev-platform mailing list
>>>     dev-platform@lists.mozilla.org <mailto:dev-platform@lists.
>>> mozilla.org>
>>>     https://lists.mozilla.org/listinfo/dev-platform
>>>
>>>
>>>
>> _______________________________________________
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
0
Christopher
6/19/2015 2:22:38 AM
Reply: