Checkout part of a branch in Azure DevOps Pipelines (GetSources)
Solution 1
In Azure DevOps you don't have option to get only part of the repository, but there is a workaround: Disable the "Get sources" step and get only the source you want by manually executing the according git commands in a script.
To disable the default "Get Sources" just specify none
in the checkout statement:
- checkout: none
In the pipeline add a CMD/PowerShell task to get the sources manually with one of the following 2 options:
1. Get only part of the repo with git sparse-checkout.
For example, get only the directories src_1
and src_2
within the test
folder (lines starting with REM ###
are just the usual batch comments):
- script: |
REM ### this will create a 'root' directory for your repo and cd into it
mkdir myRepo
cd myRepo
REM ### initialize Git in the current directory
git init
REM ### set Git sparsecheckout to TRUE
git config core.sparsecheckout true
REM ### write the directories that you want to pull to the .git/info/sparse-checkout file (without the root directory)
REM ### you can add multiple directories with multiple lines
echo test/src_1/ >> .git/info/sparse-checkout
echo test/src_2/ >> .git/info/sparse-checkout
REM ### fetch the remote repo using your access token
git remote add -f origin https://[email protected]/repo
REM ### pull the files from the source branch of this build, using the build-in Azure DevOps variable for the branch name
git pull origin $(Build.SourceBranch)
displayName: 'Get only test/src_1 & test/src_2 directories'
Now in the builds task make myRepo
the working directory.
Fetching the remote repo using an access token is necessary, since using checkout: none
will prevent your login credentials from being used.
In the end of the pipeline you may want to add step to clean the myRepo
directory.
2. Get parts of the repo with Azure DevOps Rest API (Git - Items - Get Items Batch).
Solution 2
Maybe it is helpful for you to check out only a specific branch. This works by:
resources:
repositories:
- repository: MyGitHubRepo
type: github
endpoint: MyGitHubServiceConnection
name: MyGitHubOrgOrUser/MyGitHubRepo
ref: features/tools
steps:
- checkout: MyGitHubRepo
Or by using the inline syntax like so
- checkout: git://MyProject/MyRepo@features/tools # checks out the features/tools branch
- checkout: git://MyProject/MyRepo@refs/heads/features/tools # also checks out the features/tools branch
- checkout: git://MyProject/MyRepo@refs/tags/MyTag # checks out the commit referenced by MyTag.
More information can be found here
Solution 3
The other answers work well but I found a different way using potentially newer features of git.
This will fetch to a depth of 1 and show all the files in the root folder plus folder1
, folder2
and folder3
- task: CmdLine@2
inputs:
script: |
git init
git sparse-checkout init --cone
git sparse-checkout set folder1 folder2 folder3
git remote add origin https://<github-username>:%GITHUB_TOKEN%@<your-git-repo>
git fetch --progress --verbose --depth=1 origin
git switch develop
env:
GITHUB_TOKEN: $(GITHUB_TOKEN)
Related videos on Youtube
MikeLimaSierra
I like code. I like when it works and when i understand why it doesn't. Many times i just code for fun or out of curiosity.
Updated on September 16, 2022Comments
-
MikeLimaSierra over 1 year
My repository in my organisation's devops project contains a lot of .net solutions and some unity projects as well. When I run my build pipeline, it fails due to several of these:
Error MSB3491: Could not write lines to file "obj\Release\path\to\file". There is not enough space on the disk.
I would like the pipeline to only checkout and fetch parts of the repository that are required for a successful build. This might also help with execution time of the pipeline since it currently also fetches the whole of my unity projects with gigabytes of resources which takes forever.
I would like to spread my projects across multiple repositories but the admin won't give me more than the one I already have. It got a lot better when I configured git fetch as shallow (
--depth=1
) but I still get the error every now and then.This is how I configured the checkout:
steps: - checkout: self clean: true # shallow fetch fetchDepth: 1 lfs: false submodules: false
The build is done using
VSBuild@1
task.I can't find a valid solution to my problem except for using multiple repositories, which is not an option right now.
Edit: Shayki Abramczyk's solution #1 works perfectly. Here is my full implementation.
GitSparseCheckout.yml
:parameters: access: '' repository: '' sourcePath: '' steps: - checkout: none - task: CmdLine@2 inputs: script: | ECHO ##[command] git init git init ECHO ##[command] git sparse-checkout: ${{ parameters.sourcePath }} git config core.sparsecheckout true echo ${{ parameters.sourcePath }} >> .git/info/sparse-checkout ECHO ##[command] git remote add origin https://${{ parameters.repository }} git remote add origin https://${{ parameters.access }}@${{ parameters.repository }} ECHO ##[command] git fetch --progress --verbose --depth=1 origin master git fetch --progress --verbose --depth=1 origin master ECHO ##[command] git pull --progress --verbose origin master git pull --progress --verbose origin master
Checkout is called like this (where template path has to be adjusted):
- template: ../steps/GitSparseCheckout.yml parameters: access: anything:<YOUR_PERSONAL_ACCESS_TOKEN> repository: dev.azure.com/organisation/project/_git/repository sourcePath: path/to/files/
-
MikeLimaSierra almost 5 years@DanielBMann9000 we are not committing build output to source control. unity projects can contain huge amounts of resources like shaders, textures or 3d models/meshes.
-
-
Tim Wilson over 3 yearsThis is works great! I added "git clean -ffdx" after git init. Our repository is huge, so this helped save a lot of time. Thanks!
-
Tim Wilson over 3 yearsWe ended up not needing to perform a git clean. We just cleaned the workspace at the start of the job (since we recently switched to self-hosted agents - not needed for Microsoft-hosted).
-
user1324887 over 3 yearsdoesn't seem to work. Still download the whole thing.
-
Peter Grainger over 3 years@user1324887 maybe it's your version of git, this assumes the latest version
-
user1324887 over 3 yearsThis is on Azure Dev ops with vsts on latest version. I changed it to
git clone --filter=blob:none --depth 1 --sparse REPO_URL
followed by sparse settings andgit sparse-checkout add FOLDER_OR_FILE
and that worked -
Kat Lim Ruiz about 3 yearsI don't think achieves what is asked, this resolves to checkout certain branch or tag. What is asked here is to get certain path even in master branch (to checkout only one project in a monorepo)
-
Matthias Güntert about 3 yearsIndeed, I must have misunderstood. I will still leave it.
-
Vertexwahn about 2 years@MatthiasGüntert No! Delete it! I upvoted it before I realized this does not solve the issue - was a mistake. Can not downvote it again, since votes are looked at for me now...