mirrors/git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2024-10-30 05:47:53 +01:00
Code Activity
git/azure-pipelines.yml

325 lines
12 KiB
YAML
Raw Normal View History

Add a build definition for Azure DevOps This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. The main idea is to replicate the Travis configuration as faithfully as possible, to make it easy to compare the Azure Pipeline builds to the Travis ones (spoiler: some parts, especially the macOS jobs, are way faster in Azure Pileines). Meaning: the number and the order of the jobs added in this commit faithfully replicates what we have in .travis.yml. Note: Our .travis.yml configuration has a Windows part that is *not* replicated in the Azure Pipelines definition. The reason is easy to see: As Travis cannot support our Windws needs (even with the preliminary Windows support that was recently added to Travis after waiting for *years* for that feature, our test suite would simply hit Travis' timeout every single time). To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. (While this seemingly contradicts the intention to replicate the Travis configuration as faithfully as possible, it is just too nice to show off that capability here already.) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:29 +01:00
resources:
- repo: self
fetchDepth: 1
jobs:
ci: add a Windows job to the Azure Pipelines definition Previously, we did not have robust support for Windows in our CI definition, simply because Travis cannot accommodate our needs (even after Travis added experimental Windows support very recently, it takes longer than Travis' 50 minute timeout to build Git and run the test suite on Windows). Instead, we used a hack that started a dedicated Azure Pipeline from Travis and waited for the output, often timing out (which is quite fragile, as we found out). With this commit, we finally have first-class support for Windows in our CI definition (in the Azure Pipelines one, that is). Due to our reliance on Unix shell scripting in the test suite, combined with the challenges on executing such scripts on Windows, the Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). Now, Azure Pipelines's free tier for open source projects (such as Git) offers up to 10 concurrent jobs for free, meaning that the overall run time will be dominated by the slowest job(s). Therefore, it makes sense to start the Windows job first, to minimize the time the entire build takes from start to end (which is now pretty safely the run time of the Windows job). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:30 +01:00
- job: windows
displayName: Windows
condition: succeeded()
pool: Hosted
timeoutInMinutes: 240
steps:
- powershell: |
if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") {
net use s: \\gitfileshare.file.core.windows.net\test-cache "$GITFILESHAREPWD" /user:AZURE\gitfileshare /persistent:no
cmd /c mklink /d "$(Build.SourcesDirectory)\test-cache" S:\
}
displayName: 'Mount test-cache'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- powershell: |
ci: use git-sdk-64-minimal build artifact Instead of a shallow fetch followed by a sparse checkout, we are better off by using a separate, dedicated Pipeline that bundles the SDK as a build artifact, and then consuming that build artifact here. In fact, since this artifact will be used a lot, we spent substantial time on figuring out a minimal subset of the Git for Windows SDK, just enough to build and test Git. The result is a size reduction from around 1GB (compressed) to around 55MB (compressed). This also comes with the change where we now call `usr\bin\bash.exe` directly, as `git-cmd.exe` is not included in the minimal SDK. That reduces the time to initialize Git for Windows' SDK from anywhere between 2m30s-7m to a little over 1m. Note: in theory, we could also use the DownloadBuildArtifacts@0 task here. However, restricted permissions that are in effect when building from forks would let this fail for PR builds, defeating the whole purpose of the Azure Pipelines support for git.git. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:31 +01:00
$urlbase = "https://dev.azure.com/git-for-windows/git/_apis/build/builds"
$id = ((Invoke-WebRequest -UseBasicParsing "${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1").content | ConvertFrom-JSON).value[0].id
$downloadUrl = ((Invoke-WebRequest -UseBasicParsing "${urlbase}/$id/artifacts").content | ConvertFrom-JSON).value[1].resource.downloadUrl
(New-Object Net.WebClient).DownloadFile($downloadUrl,"git-sdk-64-minimal.zip")
Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force
Remove-Item git-sdk-64-minimal.zip
ci: add a Windows job to the Azure Pipelines definition Previously, we did not have robust support for Windows in our CI definition, simply because Travis cannot accommodate our needs (even after Travis added experimental Windows support very recently, it takes longer than Travis' 50 minute timeout to build Git and run the test suite on Windows). Instead, we used a hack that started a dedicated Azure Pipeline from Travis and waited for the output, often timing out (which is quite fragile, as we found out). With this commit, we finally have first-class support for Windows in our CI definition (in the Azure Pipelines one, that is). Due to our reliance on Unix shell scripting in the test suite, combined with the challenges on executing such scripts on Windows, the Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). Now, Azure Pipelines's free tier for open source projects (such as Git) offers up to 10 concurrent jobs for free, meaning that the overall run time will be dominated by the slowest job(s). Therefore, it makes sense to start the Windows job first, to minimize the time the entire build takes from start to end (which is now pretty safely the run time of the Windows job). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:30 +01:00
# Let Git ignore the SDK and the test-cache
ci: use git-sdk-64-minimal build artifact Instead of a shallow fetch followed by a sparse checkout, we are better off by using a separate, dedicated Pipeline that bundles the SDK as a build artifact, and then consuming that build artifact here. In fact, since this artifact will be used a lot, we spent substantial time on figuring out a minimal subset of the Git for Windows SDK, just enough to build and test Git. The result is a size reduction from around 1GB (compressed) to around 55MB (compressed). This also comes with the change where we now call `usr\bin\bash.exe` directly, as `git-cmd.exe` is not included in the minimal SDK. That reduces the time to initialize Git for Windows' SDK from anywhere between 2m30s-7m to a little over 1m. Note: in theory, we could also use the DownloadBuildArtifacts@0 task here. However, restricted permissions that are in effect when building from forks would let this fail for PR builds, defeating the whole purpose of the Azure Pipelines support for git.git. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:31 +01:00
"/git-sdk-64-minimal/`n/test-cache/`n" | Out-File -NoNewLine -Encoding ascii -Append "$(Build.SourcesDirectory)\.git\info\exclude"
displayName: 'Download git-sdk-64-minimal'
ci: add a Windows job to the Azure Pipelines definition Previously, we did not have robust support for Windows in our CI definition, simply because Travis cannot accommodate our needs (even after Travis added experimental Windows support very recently, it takes longer than Travis' 50 minute timeout to build Git and run the test suite on Windows). Instead, we used a hack that started a dedicated Azure Pipeline from Travis and waited for the output, often timing out (which is quite fragile, as we found out). With this commit, we finally have first-class support for Windows in our CI definition (in the Azure Pipelines one, that is). Due to our reliance on Unix shell scripting in the test suite, combined with the challenges on executing such scripts on Windows, the Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). Now, Azure Pipelines's free tier for open source projects (such as Git) offers up to 10 concurrent jobs for free, meaning that the overall run time will be dominated by the slowest job(s). Therefore, it makes sense to start the Windows job first, to minimize the time the entire build takes from start to end (which is now pretty safely the run time of the Windows job). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:30 +01:00
- powershell: |
ci: use git-sdk-64-minimal build artifact Instead of a shallow fetch followed by a sparse checkout, we are better off by using a separate, dedicated Pipeline that bundles the SDK as a build artifact, and then consuming that build artifact here. In fact, since this artifact will be used a lot, we spent substantial time on figuring out a minimal subset of the Git for Windows SDK, just enough to build and test Git. The result is a size reduction from around 1GB (compressed) to around 55MB (compressed). This also comes with the change where we now call `usr\bin\bash.exe` directly, as `git-cmd.exe` is not included in the minimal SDK. That reduces the time to initialize Git for Windows' SDK from anywhere between 2m30s-7m to a little over 1m. Note: in theory, we could also use the DownloadBuildArtifacts@0 task here. However, restricted permissions that are in effect when building from forks would let this fail for PR builds, defeating the whole purpose of the Azure Pipelines support for git.git. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:31 +01:00
& git-sdk-64-minimal\usr\bin\bash.exe -lc @"
ci: add a Windows job to the Azure Pipelines definition Previously, we did not have robust support for Windows in our CI definition, simply because Travis cannot accommodate our needs (even after Travis added experimental Windows support very recently, it takes longer than Travis' 50 minute timeout to build Git and run the test suite on Windows). Instead, we used a hack that started a dedicated Azure Pipeline from Travis and waited for the output, often timing out (which is quite fragile, as we found out). With this commit, we finally have first-class support for Windows in our CI definition (in the Azure Pipelines one, that is). Due to our reliance on Unix shell scripting in the test suite, combined with the challenges on executing such scripts on Windows, the Windows job currently takes a whopping ~1h20m to complete. Which is *far* longer than the next-longest job takes (linux-gcc, ~35m). Now, Azure Pipelines's free tier for open source projects (such as Git) offers up to 10 concurrent jobs for free, meaning that the overall run time will be dominated by the slowest job(s). Therefore, it makes sense to start the Windows job first, to minimize the time the entire build takes from start to end (which is now pretty safely the run time of the Windows job). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:30 +01:00
export DEVELOPER=1
export NO_PERL=1
export NO_SVN_TESTS=1
export GIT_TEST_SKIP_REBASE_P=1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
"@
if (!$?) { exit(1) }
displayName: 'Build & Test'
env:
HOME: $(Build.SourcesDirectory)
MSYSTEM: MINGW64
- powershell: |
if ("$GITFILESHAREPWD" -ne "" -and "$GITFILESHAREPWD" -ne "`$`(gitfileshare.pwd)") {
cmd /c rmdir "$(Build.SourcesDirectory)\test-cache"
}
displayName: 'Unmount test-cache'
condition: true
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'windows'
platform: Windows
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
Add a build definition for Azure DevOps This commit adds an azure-pipelines.yml file which is Azure DevOps' equivalent to Travis CI's .travis.yml. The main idea is to replicate the Travis configuration as faithfully as possible, to make it easy to compare the Azure Pipeline builds to the Travis ones (spoiler: some parts, especially the macOS jobs, are way faster in Azure Pileines). Meaning: the number and the order of the jobs added in this commit faithfully replicates what we have in .travis.yml. Note: Our .travis.yml configuration has a Windows part that is *not* replicated in the Azure Pipelines definition. The reason is easy to see: As Travis cannot support our Windws needs (even with the preliminary Windows support that was recently added to Travis after waiting for *years* for that feature, our test suite would simply hit Travis' timeout every single time). To make things a bit easier to understand, we refrain from using the `matrix` feature here because (while it is powerful) it can be a bit confusing to users who are not familiar with CI setups. Therefore, we use a separate phase even for similar configurations (such as GCC vs Clang on Linux, GCC vs Clang on macOS). Also, we make use of the shiny new feature we just introduced where the test suite can output JUnit-style .xml files. This information is made available in a nice UI that allows the viewer to filter by phase and/or test number, and to see trends such as: number of (failing) tests, time spent running the test suite, etc. (While this seemingly contradicts the intention to replicate the Travis configuration as faithfully as possible, it is just too nice to show off that capability here already.) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-01-29 15:19:29 +01:00
- job: linux_clang
displayName: linux-clang
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
sudo apt-get update &&
sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&
export CC=clang || exit 1
ci/install-dependencies.sh || exit 1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-build-and-tests.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'linux-clang'
platform: Linux
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: linux_gcc
displayName: linux-gcc
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&
sudo apt-get update &&
sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1
ci/install-dependencies.sh || exit 1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-build-and-tests.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'linux-gcc'
platform: Linux
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: osx_clang
displayName: osx-clang
condition: succeeded()
pool: Hosted macOS
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
export CC=clang
ci/install-dependencies.sh || exit 1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-build-and-tests.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'osx-clang'
platform: macOS
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: osx_gcc
displayName: osx-gcc
condition: succeeded()
pool: Hosted macOS
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
ci/install-dependencies.sh || exit 1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-build-and-tests.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'osx-gcc'
platform: macOS
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: gettext_poison
displayName: GETTEXT_POISON
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
sudo apt-get update &&
sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&
export jobname=GETTEXT_POISON || exit 1
ci/run-build-and-tests.sh || {
ci/print-test-failures.sh
exit 1
}
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-build-and-tests.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'gettext-poison'
platform: Linux
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: linux32
displayName: Linux32
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
res=0
sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
sudo chmod a+r t/out/TEST-*.xml
test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
exit $res
displayName: 'ci/run-linux32-docker.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TEST-*.xml'
inputs:
mergeTestResults: true
testRunTitle: 'linux32'
platform: Linux
publishRunAttachments: false
condition: succeededOrFailed()
- task: PublishBuildArtifacts@1
displayName: 'Publish trash directories of failed tests'
condition: failed()
inputs:
PathtoPublish: t/failed-test-artifacts
ArtifactName: failed-test-artifacts
- job: static_analysis
displayName: StaticAnalysis
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
sudo apt-get update &&
sudo apt-get install -y coccinelle &&
export jobname=StaticAnalysis &&
ci/run-static-analysis.sh || exit 1
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1
displayName: 'ci/run-static-analysis.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
- job: documentation
displayName: Documentation
condition: succeeded()
pool: Hosted Ubuntu 1604
steps:
- bash: |
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
sudo apt-get update &&
sudo apt-get install -y asciidoc xmlto asciidoctor &&
export ALREADY_HAVE_ASCIIDOCTOR=yes. &&
export jobname=Documentation &&
ci/test-documentation.sh || exit 1
test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || exit 1
displayName: 'ci/test-documentation.sh'
env:
GITFILESHAREPWD: $(gitfileshare.pwd)
Copy permalink