Improved wpt-sync now running experimentally

The new sync for web-platform-tests is now running experimentally. This 
provides two way sync between the w3c/web-platform-tests repository on 
GitHub and mozilla-central, so allowing gecko developers to contribute 
to web-platform-tests using their normal gecko workflow, and ensuring 
that we get all the upstream changes submitted by the community 
including engineers at Google, Apple, and Microsoft.

The new code is intended to provide the following improvements over the 
old periodic batch sync approach:

* Faster sync. The code to actually land changes to mozilla-central is 
still undergoing testing, but the intent is that we can get at least one 
wpt update per day once the system is fully operational.

* One bug per PR we downstream, filed in a component determined by the 
files changed in the PR.

* One PR per bug we upstream. Currently this will be created when a 
patch lands on inbound or autoland and should be merged when the patch 
reaches central. In some hypothetical future world in which there's a 
single entry point for submitting code to land in gecko (e.g. 
phabricator) this will change so that the PR is created when the code is 
submitted for review, so that upstream test results are available before 
landing (see next point).

* Upstream CI jobs run on PRs originating from gecko repositories. 
Previously we skipped upstream travis jobs on pushes we landed, 
occasionally causing breakage as a result. Now these jobs are run on all 
our pushes and the original bug should get a notification if the jobs fail.

* Notifications of notable changes introduced by upstream PRs. In 
particular we will add a comment when tests that used to pass start to 
not pass, when there are crashes or disabled tests, and for new tests 
that fail. This notification happens in the bug for the sync, but there 
is already an issue open to move things that obviously require attention 
(e.g. crashes) into their own bug.

If you notice problems with the sync, please file an issue [1] or 
complain in #wpt-sync on irc.  The project team consists of:

* jgraham and maja_zf (development, primary contacts)
* AutomatedTester (project management)

Issues are not unanticipated at this time, so thanks in advance for your 
patience as we work out the kinks in the system.

[1] https://github.com/mozilla/wpt-sync/issues

0
James
2/9/2018 6:26:40 PM
mozilla.dev.platform 6467 articles. 0 followers. Post Follow

5 Replies
77 Views

Similar Articles

[PageSpeed] 27

On 2/9/18 1:26 PM, James Graham wrote:
> The new code is intended to provide the following improvements over the 
> old periodic batch sync approach:

Thank you.  This is awesome.

-Boris
0
Boris
2/9/2018 6:48:52 PM
On 2/9/18 1:26 PM, James Graham wrote:
> * One bug per PR we downstream, filed in a component determined by the 
> files changed in the PR.

What does this mean exactly? What is the desired outcome of these bugs?

Cheers,
Josh
0
Josh
2/9/2018 7:59:54 PM
On 09/02/2018 19:59, Josh Bowman-Matthews wrote:
> On 2/9/18 1:26 PM, James Graham wrote:
>> * One bug per PR we downstream, filed in a component determined by the 
>> files changed in the PR.
> 
> What does this mean exactly? What is the desired outcome of these bugs?

They're tracking the process and will be closed when the PR lands in 
central. They are used for notifying gecko developers about the incoming 
change, and in particular contain the information about tests that went 
from passing to failing, and other problems during the import.

They are not essential to the sync so if they end up not working well at 
keeping people informed we can revisit the approach.
0
James
2/9/2018 8:39:02 PM
On 02/09/2018 10:39 PM, James Graham wrote:
> On 09/02/2018 19:59, Josh Bowman-Matthews wrote:
>> On 2/9/18 1:26 PM, James Graham wrote:
>>> * One bug per PR we downstream, filed in a component determined by the files changed in the PR.
>>
>> What does this mean exactly? What is the desired outcome of these bugs?
> 
> They're tracking the process and will be closed when the PR lands in central. They are used for notifying gecko developers about the incoming change, 
> and in particular contain the information about tests that went from passing to failing, and other problems during the import.

I guess I don't understand the bugmail. Most of the time I don't see any information about something failing. Am I supposed to look at the commit?
Or are new failures in bugmail like
"
Ran 2 tests and 44 subtests
OK     : 2
PASS   : 34
FAIL   : 10
"

Are those 10 failures new failures, or failures from the test total?


-Olli

> 
> They are not essential to the sync so if they end up not working well at keeping people informed we can revisit the approach.

0
smaug
2/12/2018 8:08:39 PM
On 12/02/2018 20:08, smaug wrote:
> On 02/09/2018 10:39 PM, James Graham wrote:
>> On 09/02/2018 19:59, Josh Bowman-Matthews wrote:
>>> On 2/9/18 1:26 PM, James Graham wrote:
>>>> * One bug per PR we downstream, filed in a component determined by 
>>>> the files changed in the PR.
>>>
>>> What does this mean exactly? What is the desired outcome of these bugs?
>>
>> They're tracking the process and will be closed when the PR lands in 
>> central. They are used for notifying gecko developers about the 
>> incoming change, and in particular contain the information about tests 
>> that went from passing to failing, and other problems during the import.
> 
> I guess I don't understand the bugmail. Most of the time I don't see any 
> information about something failing. Am I supposed to look at the commit?
> Or are new failures in bugmail like
> "
> Ran 2 tests and 44 subtests
> OK     : 2
> PASS   : 34
> FAIL   : 10
> "
> 
> Are those 10 failures new failures, or failures from the test total?

That's the total failures. If that's all you see then nothing fell into 
one of the predefined categories of badness that get extra details added 
to the comment. If there is some information that you think should be 
present but is actually missing, please file an issue.
0
James
2/14/2018 10:34:51 AM
Reply: