env vars for slow tests

Currently we seem to have a collection of ad-hoc environment variables for
skipping slow tests on slow machines and/or restricting the number or type
of test runs.

I'd like to rationalise these.

For example looking for strings of the form $ENV{PERL_....} under t/ and
in *,t test files, shows the following apparently timing-related vars
(along with their counts):

   2 PERL_DEBUG_FULL_TEST
        used by a few test files. Mentioned briefly in pod/perllocale.pod
        causes more tests to be run

   5 PERL_RUN_SLOW_TESTS
        used by Storable, lib/charnames.t and t/re/fold_grind.t
        but not documented outside those files

   1 PERL_SKIP_PSYCHO_TEST
        used by t/re/pat_psycho.t to skip the entire test file

   1 PERL_TEST_NUMCONVERTS
        used only by t/op/numconvert.t. Seems to affect how many tests it
        runs.

   2 PERL_TEST_TIME_OUT_FACTOR
        used by t/re/fold_grind.t and t/re/pat_psycho.t

Most of these vars are undocumented.  There's also

  14 PERL_TEST_MEMORY
    actually documented! Says how many Gb available for testing.
    If set, runs 7 test scripts under t/bigmem.
    Also used by Storable, 

I would like to eliminate most of these, and where possible:

1) auto-adapt - for example, in each problematic test script, time the
running of a first batch of tests, and based on that result, decide how
many more batches to run and/or skip.

2) If not possible to auto-adapt, have a *single* documented var which
indicates a 'slowness' factor for the host machine's CPU (c.f.
PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
all test scripts to use that one var.

Then owners of smoke machines can just set that one var (and perhaps
PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
timeouts.

I haven't (yet) made any firm suggestions, in case anyone already has some
ideas or WIP.

-- 
"Emacs isn't a bad OS once you get used to it.
It just lacks a decent editor."
0
davem
8/10/2018 8:28:44 AM
perl.perl5.porters 47302 articles. 0 followers. Follow

1 Replies
9 Views

Similar Articles

[PageSpeed] 0

--Sig_/5OsINmKWsXFI9RNAS7xo9S.
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Fri, 10 Aug 2018 09:28:44 +0100, Dave Mitchell <davem@iabyn.com>
wrote:

> Currently we seem to have a collection of ad-hoc environment variables for
> skipping slow tests on slow machines and/or restricting the number or type
> of test runs.
>=20
> I'd like to rationalise these.
>=20
> For example looking for strings of the form $ENV{PERL_....} under t/ and
> in *,t test files, shows the following apparently timing-related vars
> (along with their counts):
>=20
>    2 PERL_DEBUG_FULL_TEST
>         used by a few test files. Mentioned briefly in pod/perllocale.pod
>         causes more tests to be run
>=20
>    5 PERL_RUN_SLOW_TESTS
>         used by Storable, lib/charnames.t and t/re/fold_grind.t
>         but not documented outside those files
>=20
>    1 PERL_SKIP_PSYCHO_TEST
>         used by t/re/pat_psycho.t to skip the entire test file
>=20
>    1 PERL_TEST_NUMCONVERTS
>         used only by t/op/numconvert.t. Seems to affect how many tests it
>         runs.
>=20
>    2 PERL_TEST_TIME_OUT_FACTOR
>         used by t/re/fold_grind.t and t/re/pat_psycho.t
>=20
> Most of these vars are undocumented.  There's also
>=20
>   14 PERL_TEST_MEMORY
>     actually documented! Says how many Gb available for testing.
>     If set, runs 7 test scripts under t/bigmem.
>     Also used by Storable,=20
>=20
> I would like to eliminate most of these, and where possible:
>=20
> 1) auto-adapt - for example, in each problematic test script, time the
> running of a first batch of tests, and based on that result, decide how
> many more batches to run and/or skip.
>=20
> 2) If not possible to auto-adapt, have a *single* documented var which
> indicates a 'slowness' factor for the host machine's CPU (c.f.
> PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
> all test scripts to use that one var.
>=20
> Then owners of smoke machines can just set that one var (and perhaps
> PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
> timeouts.
>=20
> I haven't (yet) made any firm suggestions, in case anyone already has some
> ideas or WIP.

I have no suggestions other than GO for it!
I specifically cheer them being well-documented (for slow smoke
machines)

--=20
H.Merijn Brand  http://tux.nl   Perl Monger  http://amsterdam.pm.org/
using perl5.00307 .. 5.29   porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/        http://www.test-smoke.org/
http://qa.perl.org   http://www.goldmark.org/jeff/stupid-disclaimers/

--Sig_/5OsINmKWsXFI9RNAS7xo9S.
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBAgAGBQJbbVCXAAoJEAOhR6E+XcCYmD0H/RHT7D4tVWySxoVFE8/p7McM
TS4/baF+JYJh5faVYVfWIaDaJd8kWqSmOFnlZ/FHRjGKRkskxpsJpLeqvR5VmZSz
CPTRuFXp8ETaziRtov7mWFaf9ZfKRAvCJ4TjkN3Ef/k/Z1P+xXkVozJE83Xg23DO
nZHwVQc7JHMZpWkC0UBgrWQvAHI+0q2v8qL8TqaYv169Me9NwAQoxON+69vIVrxr
tXBxBE6uoFEEZhHJVXx94Ids4uXhXlmV1cMg3L7cXxJi8kiui1k9V5OSNseZNhrR
EFVzRi1QOgmchbfGr7FTJkvZBnrO+hNLxbhMwqQJfrwGccKl4K6HgO5aqjDGig0=
=8wr+
-----END PGP SIGNATURE-----

--Sig_/5OsINmKWsXFI9RNAS7xo9S.--
0
h
8/10/2018 8:45:01 AM
Reply: