Running on-device tests in CI

Hi all,

There have been a couple of threads related to test automation in B2G, 
asking why we haven't caught some especially egregious regressions; the 
kind that basically "break the phone".

To answer that, I'd like to describe how our on-device automation 
currently works, and what we're doing to expand it so we can more 
effectively address these concerns.

We currently have a smallish number of real devices, managed by WebQA, 
hooked up to on-device automation.  They run a bank of tests against a 
number of branches several times a day.  The devices are time-consuming 
to manage, since they occasionally get wedged during flashing, 
rebooting, or other operations, and require manual intervention to fix.  
For this reason, and because it's been very difficult to obtain 
significant numbers of devices for automation, we haven't been able to 
run any tests frequently enough to provide per-commit coverage.

When tests fail, WebQA engages in a fairly time-consuming process of 
investigation and bisection.  In the case of the homescreen breakage 
(caused by https://bugzilla.mozilla.org/show_bug.cgi?id=957086), our 
on-device tests did break, and the team was in the process of 
investigating these failures, which has to be done in order to be able 
to create specific, actionable bugs.

Clearly, what we really want is to be able to run at least a small set 
of tests per-commit, so that when things break, we don't need to spend 
lots of time investigating...we will already know which commit caused 
the problem, and can back it out or address it otherwise promptly.

That's exactly what we are planning for Q3, thanks to the Flame device.  
Jonathan Hylands has developed a power harness for this that allows us 
to remotely restart the phone, which addresses some of the device 
management concerns.   The A*Team, WebQA, and jhylands are working 
together to get 30 Flames in automation, and to reduce their management 
costs.  This is enough to allow us to run a small set of functional and 
performance tests per-commit, which should be enough to catch most 
"break the phone" problems.

Another issue we've had with device testing is test result visibility; 
currently, test results are available on Jenkins, for which you need VPN 
access.  This is awkward for people not closely involved in maintaining 
and running the tests.

Next quarter, we will be improving this as well.  Jonathan Eads on the 
A*Team is currently in the process of deploying Treeherder, a successor 
to TBPL.  Unlike TBPL, Treeherder is not tightly coupled with buildbot, 
and is capable of displaying test results from arbitrary data sources.  
As our bank of 30 Flames becomes available, we will start publishing 
on-device test results to Treeherder, in the same UI that will be used 
to display the per-commit tests being run in buildbot.  This will give 
people a "one-stop shop" for seeing test results for B2G, regardless of 
whether they're run on devices or in VM's managed by buildbot.

Both of these pieces together will give us the ability to manage some 
on-device tests in a manner similar to the way we currently handle 
desktop and emulator tests in TBPL; especially bad commits should break 
tests, the breakage should be visible in Treeherder, and the sheriffs 
will back out the offending commits.

We won't have enough device capacity to run all device tests per-commit, 
at least at first.  We'll have to carefully select a small set of tests 
that guard against the worst kinds of breakage. Whether we can scale 
beyond 30 devices will depend on how stable the devices are and what 
their management costs are, which is something we'll be looking at over 
the next few months.

Regards,

Jonathan

0
Jonathan
6/3/2014 6:23:39 PM
mozilla.dev.b2g 4036 articles. 0 followers. Post Follow

0 Replies
387 Views

Similar Articles

[PageSpeed] 31

Reply:

Similar Artilces:

Can we improve emulator testing? (Was: Running on-device tests in CI)
On Jun 3, 2014 2:25 PM, "Jonathan Griffin" <jgriffin@mozilla.com> wrote: > There have been a couple of threads related to test automation in B2G, asking why we haven't caught some especially egregious regressions; the kind that basically "break the phone". It's great to hear that we are making progress on on-device testing. Testing on device is always going to be needed and desired as ultimately, having FirefoxOS working on device is what matters to users. However I'm curious why emulator testing didn't catch these recent regressions? Did...

Can we improve emulator testing? (Was: Running on-device tests in CI)
On Jun 3, 2014 2:25 PM, "Jonathan Griffin" <jgriffin@mozilla.com> wrote: > There have been a couple of threads related to test automation in B2G, asking why we haven't caught some especially egregious regressions; the kind that basically "break the phone". It's great to hear that we are making progress on on-device testing. Testing on device is always going to be needed and desired as ultimately, having FirefoxOS working on device is what matters to users. However I'm curious why emulator testing didn't catch these recent regressions? Did...

media-queries, device-width/device-height, testing resolutions, and b2g-desktop. What should we do?
Short story, as briefly referenced in Miller's "Travis + Marionette JS gotchas" message, device-height and device-width may not do what you expect on b2g-desktop. If you built your profile with DEBUG=1, then they refer to the page size and exclude the Gaia status bar at the top of the screen that tells you what time it is and how amazing your internet connection is. Otherwise, you're dealing with the screen size of the X server you're running. We don't explicitly specify a screen configuration, but the travis default seems to be "1024x728x24&qu...

test test test test test test
test test test test test test test test test test test ...

TEST TEST TEST TEST TEST
from forums 11:55 AM PST 02/12/2008 ...

Tests running Tests
Hello all. I've got a test I want to write, but I don't know to write it (easily). I've got a test script, call it foo.t which uses Test::More and runs under Test::Harness. Now I want to make a new test script tweek-then-foo.t which tweeks the system and then ensures that foo.t still passes. How do I write tweek-then-foo.t? -sam On Tue, Apr 12, 2005 at 01:20:18PM -0400, Sam Tregar wrote: > Hello all. I've got a test I want to write, but I don't know to write > it (easily). I've got a test script, call it foo.t which uses > Test::More and run...

Running marionette tests on a device
Hi all, I'm trying to run a marionette test on my device to test ADN/FDN functionality but I'm encountering some difficulties. First of all I've setup everything as per the MDN page [1] using an engineering build, forwarding port 2828 and all. Then I tried running the test with the following command: ../test.sh marionette --address=localhost:2828 ../gecko/dom/icc/tests/marionette/test_icc_contact.js The first problem I hit was the minimal web server refusing to bind to the address it was given. Turns out this line was returning a weird address which I have no...

test test test test
what is the deal with these bizarre "test updates"? i did a couple, but they just keep coming. On Feb 20, 7:08=A0pm, inspector.arc...@gmail.com wrote: > what is the deal with these bizarre "test updates"? i did a couple, > but they just keep coming. what? my email is accessible?! what the hell are you thinking? please delete these posts and remove my email address from public access. unbelievable! ...

test test test test
OE is a pain in my ass, test test test -- http://www.spywareinfo.com PGP Public key at http://www.spywareinfo.com/Mike_Healan.txt ...

Only one test run at a time can run tests in ASP.NET
I am getting this error when I run my unit test:  The ASP.NET Web application at 'C:\Users\Jason\Desktop\Movies\Movies' is already configured for testing by another test run. Only one test run at a time can run tests in ASP.NET. If there are no other test runs using this Web application, ensure that the Web.config file does not contain an httpModule named HostAdapter. How do I run only one test? Kenshinofkin: I am getting this error when I run my unit test:  The ASP.NET Web application at 'C:\Users\Jason\Desktop\Movies\Movies' is already configured for testi...

Extra Tests for Test::Run (and evidently Test::Harness)
Hi all! These are some extra test files that I added to Test::Run, and possibly could be integrated into Test::Harness after some modification: t/output.t: <<<<<<<<<<<<<<<< #!/usr/bin/perl -w BEGIN { if( $ENV{PERL_CORE} ) { chdir 't'; @INC = ('../lib', 'lib'); } else { unshift @INC, 't/lib'; } } use strict; use Test::More tests => 2; use Test::Run::Obj; { open ALTOUT, ">", "altout.txt"; open SAVEOUT, "...

Merging dev-gaia and dev-b2g into dev-fxos
--001a113ce93ebce35d051e4c0c73 Content-Type: text/plain; charset=UTF-8 Hello people of Firefox OS, After a discussion we have decided that the distinction between dev-gaia and dev-b2g mailing lists is not enough to warrant maintaining two lists. So we are deprecating both in favor of dev-fxos. So if you are subscribed to one of the aforementioned lists, you will be subscribed to the new dev-fxos list and we will shortly be decommissioning dev-gaia and dev-b2g. Thanks! Michael --001a113ce93ebce35d051e4c0c73 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: qu...

Merging dev-gaia and dev-b2g into dev-fxos
--001a113ce93ebce35d051e4c0c73 Content-Type: text/plain; charset=UTF-8 Hello people of Firefox OS, After a discussion we have decided that the distinction between dev-gaia and dev-b2g mailing lists is not enough to warrant maintaining two lists. So we are deprecating both in favor of dev-fxos. So if you are subscribed to one of the aforementioned lists, you will be subscribed to the new dev-fxos list and we will shortly be decommissioning dev-gaia and dev-b2g. Thanks! Michael --001a113ce93ebce35d051e4c0c73 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: qu...

Running XPCShell tests on B2G emulator
--20cf303ea6a6a736c4050482e46a Content-Type: text/plain; charset=UTF-8 Hi, what's the best way to run XPCShell tests on the B2G emulator? In the past I managed to workaround the need of running these tests locally. But now I really have no other choice and I am having a really hard time trying to run them locally. I am following the guide at [1], but I found several issues, like [2] not existing anymore for instance. After struggling to install all the dependencies I think I managed to trigger the XPCShell tests machinery with the following command: runtestsb2g.py --adb...

Web resources about - Running on-device tests in CI - mozilla.dev.b2g

On Device Research - LinkedIn
Welcome to the company profile of On Device Research on LinkedIn. On Device Research uses the mobile internet to gain access to consumer opinions ...

Oracle Mobile Field Service for Mobile Devices on the App Store on iTunes
Get Oracle Mobile Field Service for Mobile Devices on the App Store. See screenshots and ratings, and read customer reviews.

'bpay' contactless devices on sale
A new range of tap and pay devices including wristbands, fobs and stickers will be unveiled in the UK.

Harper Government To Test Electronic Monitoring Device On High-Risk Inmates
... are among the inmates slated to take part in a federal prison service pilot project to test the effectiveness of electronic monitoring devices.Research ...

Following evasi0n jailbreak release, A4 devices on iOS 6.1.5 also untethered via “p0xisxpwn” Cydia package ...
... touch on iOS 6.1.3 through 6.1.5—versions previously impervious to the iOS 6 jailbreak—can also get an untethered jailbreak for their devices ...

BlackBerry's President of Devices on the Priv, Android, security and keyboards
In this exclusive interview with Ron Louks, BlackBerry's President of BlackBerry Devices, we talk about the upcoming Android-powered Priv and ...

BlackBerry Bought AndroidSecured.com and .net Domains, Is This More Evidence Of An Android Device On ...
On July 7th, the domains AndroidSecured.com and AndroidSecured.net were registered to BlackBerry Limited, the firm formerly known as Research ...

The Invisible Gamepad Launches For Mobile Devices On Kickstarter
... is the only product that lets you play with precision anywhere you want without a cumbersome accessory. You just pull out your mobile device ...

UBS: iPhone to be dominant device on Verizon at launch
An analyst with UBS AG says Apple's Verizon iPhone will immediately become the carrier's top model, eating up sales of other devices, including ...

RIMM: The BerryPad? New Devices On The Way From Waterloo
Research In Motion (RIMM) is testing a variety of new devices to compete with Apple (AAPL) and other rivals in the smart phone and tablet sectors, ...

Resources last updated: 12/14/2015 5:51:22 AM