PSA: Setting preferences and extensions in test harnesses

tl;dr - You can now set prefs and install extensions across multiple
harnesses by modifying the relevant profile under testing/profiles.

If you previously would have set a  pref in:
testing/profiles/prefs_general.js,

Use this instead:
testing/profiles/unittest/user.js


# Overview

I'm currently in the process of consolidating prefs and extensions
across all our test harnesses into a single shared location
(testing/profiles
<https://searchfox.org/mozilla-central/source/testing/profiles>). If you
navigate there you'll find several different
"profile-like" directories.

Each profile directory has a user.js file for setting prefs, and an
extensions dir for dropping in extensions. I call these directories
"profile-like" because they only currently support prefs and
extensions. Don't expect to be able to add other profile-related
files (though if there's a need, support for other kinds of profile
data can be implemented).

You'll also find a 'profiles.json' file, and a 'profile' utility script. The
JSON file is used to map test suites to a list of profile directories to
apply. For example, mochitest uses the 'common' and 'unittest'
profiles in that order. Prefs from the latter profiles will overwrite
prefs from the earlier ones. This means you can either set prefs for
a single harness (e.g by adding them to the latter profile), or set
prefs across all harnesses at the same time (e.g by adding them
to the 'common' profile). Same goes for extensions.

Because prefs are split across multiple directories, and those
directories can overwrite one another, the 'profile' utility script can
help you view and compare the contents of a profile. For example:

$ cd testing/profiles
$ ./profile show mochitest  # dumps all prefs that mochitest will use

You can also diff profiles or sort preference files alphabetically:

$ ./profile diff mochitest talos
$ ./profile sort mochitest

In the future, I may add a command to automatically set preferences
in a given list of suites.


# Motivations

One great use case is testing the impact prefs and extensions have
on our performance (and unittests). Simply drop an extension into
testing/profiles/common/extensions and push to try. It will be
installed in all of our harnesses (that use this system), including Talos.

Another reason for this change is that it provides more visibility into
which prefs are set in which harnesses. We can now diff the prefs set
in various suites to try and reason about whether those differences are
intentional or omissions.

Lastly, this will make it easier to set prefs. No more hunting around
the tree looking for preference files. This will help prevent cases
where a pref was accidentally omitted from a test harness.


# Further Work

I'm in the middle of migrating harnesses over to this new system.
Notably, reftest and xpcshell have not yet been converted, though
hopefully these will be done soon. There is a long tail of smaller
harnesses that I might not have time to get around to however.

For current support, just look here:
https://searchfox.org/mozilla-central/source/testing/profiles/profiles.json

Here's the relevant bug tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=1451159&hide_resolved=0

Let me know if you have any questions or concerns,
Andrew
0
Andrew
5/16/2018 2:52:29 PM
mozilla.dev.platform 6452 articles. 0 followers. Post Follow

0 Replies
36 Views

Similar Articles

[PageSpeed] 45

Reply: