env vars for slow tests

Currently we seem to have a collection of ad-hoc environment variables for
skipping slow tests on slow machines and/or restricting the number or type
of test runs.

I'd like to rationalise these.

For example looking for strings of the form $ENV{PERL_....} under t/ and
in *,t test files, shows the following apparently timing-related vars
(along with their counts):

   2 PERL_DEBUG_FULL_TEST
        used by a few test files. Mentioned briefly in pod/perllocale.pod
        causes more tests to be run

   5 PERL_RUN_SLOW_TESTS
        used by Storable, lib/charnames.t and t/re/fold_grind.t
        but not documented outside those files

   1 PERL_SKIP_PSYCHO_TEST
        used by t/re/pat_psycho.t to skip the entire test file

   1 PERL_TEST_NUMCONVERTS
        used only by t/op/numconvert.t. Seems to affect how many tests it
        runs.

   2 PERL_TEST_TIME_OUT_FACTOR
        used by t/re/fold_grind.t and t/re/pat_psycho.t

Most of these vars are undocumented.  There's also

  14 PERL_TEST_MEMORY
    actually documented! Says how many Gb available for testing.
    If set, runs 7 test scripts under t/bigmem.
    Also used by Storable, 

I would like to eliminate most of these, and where possible:

1) auto-adapt - for example, in each problematic test script, time the
running of a first batch of tests, and based on that result, decide how
many more batches to run and/or skip.

2) If not possible to auto-adapt, have a *single* documented var which
indicates a 'slowness' factor for the host machine's CPU (c.f.
PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
all test scripts to use that one var.

Then owners of smoke machines can just set that one var (and perhaps
PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
timeouts.

I haven't (yet) made any firm suggestions, in case anyone already has some
ideas or WIP.

-- 
"Emacs isn't a bad OS once you get used to it.
It just lacks a decent editor."
0
davem
8/10/2018 8:28:44 AM
perl.perl5.porters 47366 articles. 0 followers. Follow

5 Replies
30 Views

Similar Articles

[PageSpeed] 24

--Sig_/5OsINmKWsXFI9RNAS7xo9S.
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Fri, 10 Aug 2018 09:28:44 +0100, Dave Mitchell <davem@iabyn.com>
wrote:

> Currently we seem to have a collection of ad-hoc environment variables for
> skipping slow tests on slow machines and/or restricting the number or type
> of test runs.
>=20
> I'd like to rationalise these.
>=20
> For example looking for strings of the form $ENV{PERL_....} under t/ and
> in *,t test files, shows the following apparently timing-related vars
> (along with their counts):
>=20
>    2 PERL_DEBUG_FULL_TEST
>         used by a few test files. Mentioned briefly in pod/perllocale.pod
>         causes more tests to be run
>=20
>    5 PERL_RUN_SLOW_TESTS
>         used by Storable, lib/charnames.t and t/re/fold_grind.t
>         but not documented outside those files
>=20
>    1 PERL_SKIP_PSYCHO_TEST
>         used by t/re/pat_psycho.t to skip the entire test file
>=20
>    1 PERL_TEST_NUMCONVERTS
>         used only by t/op/numconvert.t. Seems to affect how many tests it
>         runs.
>=20
>    2 PERL_TEST_TIME_OUT_FACTOR
>         used by t/re/fold_grind.t and t/re/pat_psycho.t
>=20
> Most of these vars are undocumented.  There's also
>=20
>   14 PERL_TEST_MEMORY
>     actually documented! Says how many Gb available for testing.
>     If set, runs 7 test scripts under t/bigmem.
>     Also used by Storable,=20
>=20
> I would like to eliminate most of these, and where possible:
>=20
> 1) auto-adapt - for example, in each problematic test script, time the
> running of a first batch of tests, and based on that result, decide how
> many more batches to run and/or skip.
>=20
> 2) If not possible to auto-adapt, have a *single* documented var which
> indicates a 'slowness' factor for the host machine's CPU (c.f.
> PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
> all test scripts to use that one var.
>=20
> Then owners of smoke machines can just set that one var (and perhaps
> PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
> timeouts.
>=20
> I haven't (yet) made any firm suggestions, in case anyone already has some
> ideas or WIP.

I have no suggestions other than GO for it!
I specifically cheer them being well-documented (for slow smoke
machines)

--=20
H.Merijn Brand  http://tux.nl   Perl Monger  http://amsterdam.pm.org/
using perl5.00307 .. 5.29   porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/        http://www.test-smoke.org/
http://qa.perl.org   http://www.goldmark.org/jeff/stupid-disclaimers/

--Sig_/5OsINmKWsXFI9RNAS7xo9S.
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBAgAGBQJbbVCXAAoJEAOhR6E+XcCYmD0H/RHT7D4tVWySxoVFE8/p7McM
TS4/baF+JYJh5faVYVfWIaDaJd8kWqSmOFnlZ/FHRjGKRkskxpsJpLeqvR5VmZSz
CPTRuFXp8ETaziRtov7mWFaf9ZfKRAvCJ4TjkN3Ef/k/Z1P+xXkVozJE83Xg23DO
nZHwVQc7JHMZpWkC0UBgrWQvAHI+0q2v8qL8TqaYv169Me9NwAQoxON+69vIVrxr
tXBxBE6uoFEEZhHJVXx94Ids4uXhXlmV1cMg3L7cXxJi8kiui1k9V5OSNseZNhrR
EFVzRi1QOgmchbfGr7FTJkvZBnrO+hNLxbhMwqQJfrwGccKl4K6HgO5aqjDGig0=
=8wr+
-----END PGP SIGNATURE-----

--Sig_/5OsINmKWsXFI9RNAS7xo9S.--
0
h
8/10/2018 8:45:01 AM
On 08/10/2018 02:28 AM, Dave Mitchell wrote:
> Currently we seem to have a collection of ad-hoc environment variables for
> skipping slow tests on slow machines and/or restricting the number or type
> of test runs.
> 
> I'd like to rationalise these.
> 
> For example looking for strings of the form $ENV{PERL_....} under t/ and
> in *,t test files, shows the following apparently timing-related vars
> (along with their counts):
> 
>     2 PERL_DEBUG_FULL_TEST
>          used by a few test files. Mentioned briefly in pod/perllocale.pod
>          causes more tests to be run

Actually, in the cases I know of, this doesn't cause more tests to be 
run, but increases the verbosity of the output.  Perhaps it could be 
changed to a better name
> 
>     5 PERL_RUN_SLOW_TESTS
>          used by Storable, lib/charnames.t and t/re/fold_grind.t
>          but not documented outside those files

This increases the number of tests run in the cases I know about.
> 
>     1 PERL_SKIP_PSYCHO_TEST
>          used by t/re/pat_psycho.t to skip the entire test file
> 
>     1 PERL_TEST_NUMCONVERTS
>          used only by t/op/numconvert.t. Seems to affect how many tests it
>          runs.
> 
>     2 PERL_TEST_TIME_OUT_FACTOR
>          used by t/re/fold_grind.t and t/re/pat_psycho.t

This is used where a time-out has been instituted because they can take 
too long on slow systems.  But if you don't care about that and want 
them to finish if they are going to, you can use this to set a ratio to 
compensate for the slowness of the system.
> 
> Most of these vars are undocumented.  There's also
> 
>    14 PERL_TEST_MEMORY
>      actually documented! Says how many Gb available for testing.
>      If set, runs 7 test scripts under t/bigmem.
>      Also used by Storable,
> 
> I would like to eliminate most of these, and where possible:
> 
> 1) auto-adapt - for example, in each problematic test script, time the
> running of a first batch of tests, and based on that result, decide how
> many more batches to run and/or skip.
> 
> 2) If not possible to auto-adapt, have a *single* documented var which
> indicates a 'slowness' factor for the host machine's CPU (c.f.
> PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
> all test scripts to use that one var.
> 
> Then owners of smoke machines can just set that one var (and perhaps
> PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
> timeouts.
> 
> I haven't (yet) made any firm suggestions, in case anyone already has some
> ideas or WIP.
> 
0
public
8/22/2018 12:28:40 AM
On Fri, Aug 10, 2018 at 10:28 AM, Dave Mitchell <davem@iabyn.com> wrote:
> Currently we seem to have a collection of ad-hoc environment variables for
> skipping slow tests on slow machines and/or restricting the number or type
> of test runs.
>
> I'd like to rationalise these.
>
> For example looking for strings of the form $ENV{PERL_....} under t/ and
> in *,t test files, shows the following apparently timing-related vars
> (along with their counts):
>
>    2 PERL_DEBUG_FULL_TEST
>         used by a few test files. Mentioned briefly in pod/perllocale.pod
>         causes more tests to be run
>
>    5 PERL_RUN_SLOW_TESTS
>         used by Storable, lib/charnames.t and t/re/fold_grind.t
>         but not documented outside those files
>
>    1 PERL_SKIP_PSYCHO_TEST
>         used by t/re/pat_psycho.t to skip the entire test file
>
>    1 PERL_TEST_NUMCONVERTS
>         used only by t/op/numconvert.t. Seems to affect how many tests it
>         runs.
>
>    2 PERL_TEST_TIME_OUT_FACTOR
>         used by t/re/fold_grind.t and t/re/pat_psycho.t
>
> Most of these vars are undocumented.  There's also
>
>   14 PERL_TEST_MEMORY
>     actually documented! Says how many Gb available for testing.
>     If set, runs 7 test scripts under t/bigmem.
>     Also used by Storable,
>
> I would like to eliminate most of these, and where possible:
>
> 1) auto-adapt - for example, in each problematic test script, time the
> running of a first batch of tests, and based on that result, decide how
> many more batches to run and/or skip.
>
> 2) If not possible to auto-adapt, have a *single* documented var which
> indicates a 'slowness' factor for the host machine's CPU (c.f.
> PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
> all test scripts to use that one var.
>
> Then owners of smoke machines can just set that one var (and perhaps
> PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
> timeouts.
>
> I haven't (yet) made any firm suggestions, in case anyone already has some
> ideas or WIP.

At the Lancaster QA Hackathon[1] we defined the EXTENDED_TESTING
environmental variable for CPAN Testers to signal that more extensive
testing is welcome. I think it would be approprate to reuse that here
(in particular in dual-life modules like Storable).

Leon

1: https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md
0
fawaka
8/22/2018 12:14:37 PM
On Wed, 22 Aug 2018 at 02:30, Karl Williamson <public@khwilliamson.com> wrote:
>
> On 08/10/2018 02:28 AM, Dave Mitchell wrote:
> > Currently we seem to have a collection of ad-hoc environment variables for
> > skipping slow tests on slow machines and/or restricting the number or type
> > of test runs.
> >
> > I'd like to rationalise these.
> >
> > For example looking for strings of the form $ENV{PERL_....} under t/ and
> > in *,t test files, shows the following apparently timing-related vars
> > (along with their counts):
> >
> >     2 PERL_DEBUG_FULL_TEST
> >          used by a few test files. Mentioned briefly in pod/perllocale.pod
> >          causes more tests to be run
>
> Actually, in the cases I know of, this doesn't cause more tests to be
> run, but increases the verbosity of the output.  Perhaps it could be
> changed to a better name
> >
> >     5 PERL_RUN_SLOW_TESTS
> >          used by Storable, lib/charnames.t and t/re/fold_grind.t
> >          but not documented outside those files
>
> This increases the number of tests run in the cases I know about.
> >
> >     1 PERL_SKIP_PSYCHO_TEST
> >          used by t/re/pat_psycho.t to skip the entire test file

PSYCHO takes forever and makes certain type of regex enging hacking
very frustrating, so I hacked this in so I could easily skip the test.

> >     1 PERL_TEST_NUMCONVERTS
> >          used only by t/op/numconvert.t. Seems to affect how many tests it
> >          runs.
> >
> >     2 PERL_TEST_TIME_OUT_FACTOR
> >          used by t/re/fold_grind.t and t/re/pat_psycho.t
>
> This is used where a time-out has been instituted because they can take
> too long on slow systems.  But if you don't care about that and want
> them to finish if they are going to, you can use this to set a ratio to
> compensate for the slowness of the system.
> >
> > Most of these vars are undocumented.  There's also
> >
> >    14 PERL_TEST_MEMORY
> >      actually documented! Says how many Gb available for testing.
> >      If set, runs 7 test scripts under t/bigmem.
> >      Also used by Storable,
> >
> > I would like to eliminate most of these, and where possible:
> >
> > 1) auto-adapt - for example, in each problematic test script, time the
> > running of a first batch of tests, and based on that result, decide how
> > many more batches to run and/or skip.
> >
> > 2) If not possible to auto-adapt, have a *single* documented var which
> > indicates a 'slowness' factor for the host machine's CPU (c.f.
> > PERL_TEST_MEMORY to indicate how many Gb the host has for testing) and get
> > all test scripts to use that one var.
> >
> > Then owners of smoke machines can just set that one var (and perhaps
> > PERL_TEST_MEMORY) and avoid the current random failures due to watchdog
> > timeouts.
> >
> > I haven't (yet) made any firm suggestions, in case anyone already has some
> > ideas or WIP.
> >



-- 
perl -Mre=debug -e "/just|another|perl|hacker/"
0
demerphq
8/23/2018 7:49:25 AM

On 08/22/2018 03:14 PM, Leon Timmermans wrote:
> On Fri, Aug 10, 2018 at 10:28 AM, Dave Mitchell <davem@iabyn.com> wrote:
>> Currently we seem to have a collection of ad-hoc environment variables for
>> skipping slow tests on slow machines and/or restricting the number or type
>> of test runs.
>>
>> I'd like to rationalise these.
>>
>>
>> [...]
>>
> At the Lancaster QA Hackathon[1] we defined the EXTENDED_TESTING
> environmental variable for CPAN Testers to signal that more extensive
> testing is welcome. I think it would be approprate to reuse that here
> (in particular in dual-life modules like Storable).

+1
0
xsawyerx
8/24/2018 2:55:32 PM
Reply: