1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-10-30 05:47:53 +01:00

Merge branch 'en/test-selector'

Our test scripts can be told to run only individual pieces while
skipping others with the "--run=..." option; they were taught to
take a substring of test title, in addition to numbers, to name the
test pieces to run.

* en/test-selector:
  test-lib: reduce verbosity of skipped tests
  t6006, t6012: adjust tests to use 'setup' instead of synonyms
  test-lib: allow selecting tests by substring/glob with --run
This commit is contained in:
Junio C Hamano 2020-11-02 13:17:42 -08:00
commit 9879f3b3f6
5 changed files with 75 additions and 49 deletions

View file

@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that
only some tests should be run or that some tests should be
excluded from a run.
The argument for --run is a list of individual test numbers or
ranges with an optional negation prefix that define what tests in
a test suite to include in the run. A range is two numbers
separated with a dash and matches a range of tests with both ends
been included. You may omit the first or the second number to
mean "from the first test" or "up to the very last test"
respectively.
The argument for --run, <test-selector>, is a list of description
substrings or globs or individual test numbers or ranges with an
optional negation prefix (of '!') that define what tests in a test
suite to include (or exclude, if negated) in the run. A range is two
numbers separated with a dash and matches a range of tests with both
ends been included. You may omit the first or the second number to
mean "from the first test" or "up to the very last test" respectively.
Optional prefix of '!' means that the test or a range of tests
should be excluded from the run.
The argument to --run is split on commas into separate strings,
numbers, and ranges, and picks all tests that match any of the
individual selection criteria. If the substring of the description
text that you want to match includes a comma, use the glob character
'?' instead. For example --run='rebase,merge?cherry-pick' would match
on all tests that match either the glob *rebase* or the glob
*merge?cherry-pick*.
If --run starts with an unprefixed number or range the initial
set of tests to run is empty. If the first item starts with '!'
@ -275,9 +280,6 @@ all the tests are added to the initial set. After initial set is
determined every test number or range is added or excluded from
the set one by one, from left to right.
Individual numbers or ranges could be separated either by a space
or a comma.
For example, to run only tests up to a specific test (21), one
could do this:
@ -290,7 +292,7 @@ or this:
Common case is to run several setup tests (1, 2, 3) and then a
specific test (21) that relies on that setup:
$ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
$ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
or:
@ -298,17 +300,17 @@ or:
or:
$ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
$ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
As noted above, the test set is built by going through the items
from left to right, so this:
$ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
$ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
will run tests 1, 2, and 4. Items that come later have higher
precedence. It means that this:
$ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
$ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
would just run tests from 1 to 4, including 3.
@ -317,6 +319,18 @@ test in the test suite except from 7 up to 11:
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
Sometimes there may be multiple tests with e.g. "setup" in their name
that are needed and rather than figuring out the number for all of them
we can just use "setup" as a substring/glob to match against the test
description:
$ sh ./t0050-filesystem.sh --run=setup,9-11
or one could select both the setup tests and the rename ones (assuming all
relevant tests had those words in their descriptions):
$ sh ./t0050-filesystem.sh --run=setup,rename
Some tests in a test suite rely on the previous tests performing
certain actions, specifically some tests are designated as
"setup" test, so you cannot _arbitrarily_ disable one test and

View file

@ -430,7 +430,7 @@ test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
test_expect_success '--run basic' "
run_sub_test_lib_test run-basic \
'--run basic' --run='1 3 5' <<-\\EOF &&
'--run basic' --run='1,3,5' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -472,7 +472,7 @@ test_expect_success '--run with a range' "
test_expect_success '--run with two ranges' "
run_sub_test_lib_test run-two-ranges \
'--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
'--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -556,7 +556,7 @@ test_expect_success '--run with basic negation' "
test_expect_success '--run with two negations' "
run_sub_test_lib_test run-two-neg \
'--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
'--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -577,7 +577,7 @@ test_expect_success '--run with two negations' "
test_expect_success '--run a range and negation' "
run_sub_test_lib_test run-range-and-neg \
'--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
'--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -620,7 +620,7 @@ test_expect_success '--run range negation' "
test_expect_success '--run include, exclude and include' "
run_sub_test_lib_test run-inc-neg-inc \
'--run include, exclude and include' \
--run='"'1-5 !1-3 2'"' <<-\\EOF &&
--run='"'1-5,!1-3,2'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -664,7 +664,7 @@ test_expect_success '--run include, exclude and include, comma separated' "
test_expect_success '--run exclude and include' "
run_sub_test_lib_test run-neg-inc \
'--run exclude and include' \
--run='"'!3- 5'"' <<-\\EOF &&
--run='"'!3-,5'"' <<-\\EOF &&
for i in 1 2 3 4 5 6
do
test_expect_success \"passing test #\$i\" 'true'
@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' "
EOF
"
test_expect_success '--run invalid range start' "
test_expect_success '--run substring selector' "
run_sub_test_lib_test run-substring-selector \
'--run empty selectors' \
--run='relevant' <<-\\EOF &&
test_expect_success \"relevant test\" 'true'
for i in 1 2 3 4 5 6
do
test_expect_success \"other test #\$i\" 'true'
done
test_done
EOF
check_sub_test_lib_test run-substring-selector <<-\\EOF
> ok 1 - relevant test
> ok 2 # skip other test #1 (--run)
> ok 3 # skip other test #2 (--run)
> ok 4 # skip other test #3 (--run)
> ok 5 # skip other test #4 (--run)
> ok 6 # skip other test #5 (--run)
> ok 7 # skip other test #6 (--run)
> # passed all 7 test(s)
> 1..7
EOF
"
test_expect_success '--run keyword selection' "
run_sub_test_lib_test_err run-inv-range-start \
'--run invalid range start' \
--run='a-5' <<-\\EOF &&
@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' "
EOF_ERR
"
test_expect_success '--run invalid selector' "
run_sub_test_lib_test_err run-inv-selector \
'--run invalid selector' \
--run='1?' <<-\\EOF &&
test_expect_success \"passing test #1\" 'true'
test_done
EOF
check_sub_test_lib_test_err run-inv-selector \
<<-\\EOF_OUT 3<<-\\EOF_ERR
> FATAL: Unexpected exit with code 1
EOF_OUT
> error: --run: invalid non-numeric in test selector: '1?'
EOF_ERR
"
test_set_prereq HAVEIT
haveit=no

View file

@ -339,7 +339,7 @@ commit $head1
.. (hinzugef${added_utf8_part_iso88591}gt) foo
EOF
test_expect_success 'prepare expected messages (for test %b)' '
test_expect_success 'setup expected messages (for test %b)' '
cat <<-EOF >expected.utf-8 &&
commit $head3
This commit message is much longer than the others,

View file

@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '
#
# This example is explained in Documentation/rev-list-options.txt
test_expect_success 'rebuild repo' '
test_expect_success 'setup rebuild repo' '
rm -rf .git * &&
git init &&
git switch -c topic &&

View file

@ -769,15 +769,17 @@ match_pattern_list () {
}
match_test_selector_list () {
operation="$1"
shift
title="$1"
shift
arg="$1"
shift
test -z "$1" && return 0
# Both commas and whitespace are accepted as separators.
# Commas are accepted as separators.
OLDIFS=$IFS
IFS=' ,'
IFS=','
set -- $1
IFS=$OLDIFS
@ -805,13 +807,13 @@ match_test_selector_list () {
*-*)
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in range" \
echo "error: $operation: invalid non-numeric in range" \
"start: '$orig_selector'" >&2
exit 1
fi
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in range" \
echo "error: $operation: invalid non-numeric in range" \
"end: '$orig_selector'" >&2
exit 1
fi
@ -819,9 +821,11 @@ match_test_selector_list () {
*)
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
then
echo "error: $title: invalid non-numeric in test" \
"selector: '$orig_selector'" >&2
exit 1
case "$title" in *${selector}*)
include=$positive
;;
esac
continue
fi
esac
@ -1031,7 +1035,7 @@ test_skip () {
skipped_reason="GIT_SKIP_TESTS"
fi
if test -z "$to_skip" && test -n "$run_list" &&
! match_test_selector_list '--run' $test_count "$run_list"
! match_test_selector_list '--run' "$1" $test_count "$run_list"
then
to_skip=t
skipped_reason="--run"
@ -1058,7 +1062,6 @@ test_skip () {
" <skipped message=\"$message\" />"
fi
say_color skip >&3 "skipping test: $@"
say_color skip "ok $test_count # skip $1 ($skipped_reason)"
: true
;;