From 106834e34a2cc11b4a36a536c5b96c2c1c916cbd Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 9 Oct 2024 15:25:18 +0200 Subject: [PATCH 1/5] t7300: work around platform-specific behaviour with long paths on MinGW Windows by default has a restriction in place to only allow paths up to 260 characters. This restriction can nowadays be lifted by setting a registry key, but is still active by default. In t7300 we have one test that exercises the behaviour of git-clean(1) with such long paths. Interestingly enough, this test fails on my system that uses Windows 10 with mingw-w64 installed via MSYS2: instead of observing ENAMETOOLONG, we observe ENOENT. This behaviour is consistent across multiple different environments I have tried. I cannot say why exactly we observe a different error here, but I would not be surprised if this was either dependent on the Windows version, the version of MinGW, the current working directory of Git or any kind of combination of these. Work around the issue by handling both errors. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- t/t7300-clean.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 0aae0dee67..12ab25296b 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -747,7 +747,7 @@ test_expect_success MINGW 'handle clean & core.longpaths = false nicely' ' test_must_fail git clean -xdf 2>.git/err && # grepping for a strerror string is unportable but it is OK here with # MINGW prereq - test_grep "too long" .git/err + test_grep -e "too long" -e "No such file or directory" .git/err ' test_expect_success 'clean untracked paths by pathspec' ' From 91839a882779012a2bc18c1bfd1d8d11fe1099aa Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 9 Oct 2024 15:25:21 +0200 Subject: [PATCH 2/5] ci: create script to set up Git for Windows SDK In order to build and test Git, we have to first set up the Git for Windows SDK, which contains various required tools and libraries. The SDK is basically a clone of [1], but that repository is quite large due to all the binaries it contains. We thus use both shallow clones and sparse checkouts to speed up the setup. To handle this complexity we use a GitHub action that is hosted externally at [2]. Unfortunately, this makes it rather hard to reuse the logic for CI platforms other than GitHub Actions. After chatting with Johannes Schindelin we came to the conclusion that it would be nice if the Git for Windows SDK would regularly publish releases that one can easily download and extract, thus moving all of the complexity into that single step. Like this, all that a CI job needs to do is to fetch and extract the resulting archive. This published release comes in the form of a new "ci-artifacts" tag that gets updated regularly [3]. Implement a new script that knows how to fetch and extract that script and convert GitHub Actions to use it. [1]: https://github.com/git-for-windows/git-sdk-64/ [2]: https://github.com/git-for-windows/setup-git-for-windows-sdk/ [3]: https://github.com/git-for-windows/git-sdk-64/releases/tag/ci-artifacts/ Helped-by: Johannes Schindelin Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- .github/workflows/main.yml | 16 ++++++++++------ ci/install-sdk.ps1 | 12 ++++++++++++ 2 files changed, 22 insertions(+), 6 deletions(-) create mode 100755 ci/install-sdk.ps1 diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 916a64b673..9301a1edd6 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -113,13 +113,15 @@ jobs: cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }} steps: - uses: actions/checkout@v4 - - uses: git-for-windows/setup-git-for-windows-sdk@v1 + - name: setup SDK + shell: powershell + run: ci/install-sdk.ps1 - name: build - shell: bash + shell: powershell env: HOME: ${{runner.workspace}} NO_PERL: 1 - run: . /etc/profile && ci/make-test-artifacts.sh artifacts + run: git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts' - name: zip up tracked files run: git archive -o artifacts/tracked.tar.gz HEAD - name: upload tracked files and build artifacts @@ -147,10 +149,12 @@ jobs: - name: extract tracked files and build artifacts shell: bash run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz - - uses: git-for-windows/setup-git-for-windows-sdk@v1 + - name: setup SDK + shell: powershell + run: ci/install-sdk.ps1 - name: test - shell: bash - run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10 + shell: powershell + run: git-sdk/usr/bin/bash.exe -l -c 'ci/run-test-slice.sh ${{matrix.nr}} 10' - name: print test failures if: failure() && env.FAILED_TEST_ARTIFACTS != '' shell: bash diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1 new file mode 100755 index 0000000000..66f24838a4 --- /dev/null +++ b/ci/install-sdk.ps1 @@ -0,0 +1,12 @@ +param( + [string]$directory='git-sdk', + [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip' +) + +Invoke-WebRequest "$url" -OutFile git-sdk.zip +Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory" +Remove-Item -Path git-sdk.zip + +New-Item -Path .git/info -ItemType Directory -Force +New-Item -Path .git/info/exclude -ItemType File -Force +Add-Content -Path .git/info/exclude -Value "/$directory" From b7a08e947eb42de6f272d60b68a2e30790e7c0d5 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 9 Oct 2024 15:25:23 +0200 Subject: [PATCH 3/5] ci: handle Windows-based CI jobs in GitLab CI We try to abstract away any differences between different CI platforms in "ci/lib.sh", such that knowledge specific to e.g. GitHub Actions or GitLab CI is neatly encapsulated in a single place. Next to some generic variables, we also set up some variables that are specific to the actual platform that the CI operates on, e.g. Linux or macOS. We do not yet support Windows runners on GitLab CI. Unfortunately, those systems do not use the same "CI_JOB_IMAGE" environment variable as both Linux and macOS do. Instead, we can use the "OS" variable, which should have a value of "Windows_NT" on Windows platforms. Handle the combination of "$OS,$CI_JOB_IMAGE" and introduce support for Windows. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- ci/lib.sh | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/ci/lib.sh b/ci/lib.sh index 74b430be23..95f39a26ea 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -250,8 +250,13 @@ then CI_TYPE=gitlab-ci CI_BRANCH="$CI_COMMIT_REF_NAME" CI_COMMIT="$CI_COMMIT_SHA" - case "$CI_JOB_IMAGE" in - macos-*) + + case "$OS,$CI_JOB_IMAGE" in + Windows_NT,*) + CI_OS_NAME=windows + JOBS=$NUMBER_OF_PROCESSORS + ;; + *,macos-*) # GitLab CI has Python installed via multiple package managers, # most notably via asdf and Homebrew. Ensure that our builds # pick up the Homebrew one by prepending it to our PATH as the @@ -259,9 +264,12 @@ then export PATH="$(brew --prefix)/bin:$PATH" CI_OS_NAME=osx + JOBS=$(nproc) + ;; + *,alpine:*|*,fedora:*|*,ubuntu:*) + CI_OS_NAME=linux + JOBS=$(nproc) ;; - alpine:*|fedora:*|ubuntu:*) - CI_OS_NAME=linux;; *) echo "Could not identify OS image" >&2 env >&2 @@ -272,6 +280,7 @@ then CI_JOB_ID="$CI_JOB_ID" CC="${CC_PACKAGE:-${CC:-gcc}}" DONT_SKIP_TAGS=t + handle_failed_tests () { create_failed_test_artifacts return 1 @@ -280,7 +289,6 @@ then cache_dir="$HOME/none" distro=$(echo "$CI_JOB_IMAGE" | tr : -) - JOBS=$(nproc) else echo "Could not identify CI type" >&2 env >&2 From 05a928a93e1a6a46902fca76a37c7876ace7f87e Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 9 Oct 2024 15:25:26 +0200 Subject: [PATCH 4/5] gitlab-ci: introduce stages and dependencies We're about to add a couple of jobs for Windows. As the Windows runners are quite slow, we will split those up across two stages: one stage to build the artifacts, and one stage that runs test slices in parallel. Introduce stages and "needs" dependencies for the preexisting jobs as a preparatory step. The stages will lead to a more natural representation of jobs in the UI, whereas the "needs" dependency ensures that jobs do not have to wait for all jobs in the preceding stage to finish. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- .gitlab-ci.yml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4abfbc3e20..e13e8b3866 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,10 @@ default: timeout: 2h +stages: + - test + - analyze + workflow: rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" @@ -9,6 +13,8 @@ workflow: test:linux: image: $image + stage: test + needs: [ ] tags: - saas-linux-medium-amd64 variables: @@ -67,6 +73,8 @@ test:linux: test:osx: image: $image + stage: test + needs: [ ] tags: - saas-macos-medium-m1 variables: @@ -102,6 +110,8 @@ test:osx: test:fuzz-smoke-tests: image: ubuntu:latest + stage: test + needs: [ ] variables: CC: clang before_script: @@ -111,6 +121,8 @@ test:fuzz-smoke-tests: static-analysis: image: ubuntu:22.04 + stage: analyze + needs: [ ] variables: jobname: StaticAnalysis before_script: @@ -121,6 +133,8 @@ static-analysis: check-whitespace: image: ubuntu:latest + stage: analyze + needs: [ ] before_script: - ./ci/install-dependencies.sh # Since $CI_MERGE_REQUEST_TARGET_BRANCH_SHA is only defined for merged @@ -135,6 +149,8 @@ check-whitespace: check-style: image: ubuntu:latest + stage: analyze + needs: [ ] allow_failure: true variables: CC: clang @@ -153,6 +169,8 @@ check-style: documentation: image: ubuntu:latest + stage: analyze + needs: [ ] variables: jobname: Documentation before_script: From 631ddbbcbd912530e1b78e5d782e72879f7f1fb2 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Wed, 9 Oct 2024 15:25:29 +0200 Subject: [PATCH 5/5] gitlab-ci: exercise Git on Windows Add jobs that exercise Git on Windows. Unfortunately, building and especially testing Git on Windows is inherently slower compared to other Unix-like systems, mostly because spawning processes is way slower. We thus use the same layout as we use in GitHub Actions, where we have one build job, and then pass on the resulting build artifacts to ten test jobs that split up the work across each other. Unfortunately, the GitLab runners for Windows machines are embarassingly slow by themselves. So while this strategy leads to around 20 minutes of build time in GitHub Actions, the same pipeline takes around an hour in GitLab CI. Still, having late coverage is certainly better than having none at all. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- .gitlab-ci.yml | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e13e8b3866..526ecfe030 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -2,6 +2,7 @@ default: timeout: 2h stages: + - build - test - analyze @@ -108,6 +109,38 @@ test:osx: - t/failed-test-artifacts when: on_failure +build:mingw64: + stage: build + tags: + - saas-windows-medium-amd64 + variables: + NO_PERL: 1 + before_script: + - ./ci/install-sdk.ps1 -directory "git-sdk" + script: + - git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts' + artifacts: + paths: + - artifacts + - git-sdk + +test:mingw64: + stage: test + tags: + - saas-windows-medium-amd64 + needs: + - job: "build:mingw64" + artifacts: true + before_script: + - git-sdk/usr/bin/bash.exe -l -c 'tar xf artifacts/artifacts.tar.gz' + - New-Item -Path .git/info -ItemType Directory + - New-Item .git/info/exclude -ItemType File -Value "/git-sdk" + script: + - git-sdk/usr/bin/bash.exe -l -c "ci/run-test-slice.sh $CI_NODE_INDEX $CI_NODE_TOTAL" + after_script: + - git-sdk/usr/bin/bash.exe -l -c 'ci/print-test-failures.sh' + parallel: 10 + test:fuzz-smoke-tests: image: ubuntu:latest stage: test