Executing git commands inside a build job in Visual Studio Team Services (was VSO)

Question

[Environment: Team Services, GIT, hosted build agent]

I'd like to create a Team Services build definition that is able to do the following:

1. Executing a script to generate some new files based on existing files in the repo

2. Commit/push those generated files back to repo

I can do #1 with no problem. But I'm not sure how I can do #2.

I discovered I was actually able to run git.exe from inside a build job. But I'm not sure how I can pass the credential to git. Based on the build logs, it's failing because it's trying to get the username from stdin.

I tried adding a step in the build definition with something like "git config --global user.name xxxx" but it still didn't help.

Is this a supported scenario at all? Any suggestions?

Thanks!

---

Edit

I tried the following approach in my build script:

git -c http.extraheader="AUTHORIZATION: bearer %SYSTEM_ACCESSTOKEN%" pull ...

It seemed to work for commands like pull, etc. But when I was trying to push the changes, I got the below error:

fatal: unable to access 'https://example.visualstudio.com/SampleTeam/_git/SampleRepo/': SSL read: error:00000000:lib(0):func(0):reason(0), errno 10054

Thoughts?

Solution 1

Sorry to answer my own question here...

I just got some hint from some TFS expert, who pointed me to this article: https://www.visualstudio.com/en-us/docs/build/scripts/git-commands, which perfectly solved my problem.

I think I should share it out to help whoever might run into the same situation as I did.

Here I am quoting the key steps (reformatted a bit):

Grant version control permissions to the build service

Go to the Version Control control panel tab

• Team Services: https://{your-account}.visualstudio.com/DefaultCollection/{your-team-project}/_admin/_versioncontrol

• On-premises: https://{your-server}:8080/tfs/DefaultCollection/{your-team-project}/_admin/_versioncontrol

On the Version Control tab, select the repository in which you want to run Git commands, and then select Project Collection Build Service (account_name). Grant permissions needed for the Git commands you want to run. Typically you'll want to grant:

• Branch creation: Allow
• Contribute: Allow
• Read: Inherited allow
• Tag creation: Inherited allow

When you're done granting the permissions, make sure to click Save changes.

Enable your build definition to run Git.exe

• On the variables tab set this variable: system.prefergit = true
• On the options tab select Allow scripts to access OAuth token.

With these settings, there is no need to install the Git Build Tools extension or tweak the Credential Manager. You don't need to explicitly set the extra header for OAuth token, either. I feel it's indeed a very neat solution. :)

But really appreciate the help from Eddie and VonC!

Solution 2

Visual Studio Team Services (VSTS) now has built in functionality to do this:

1. Grant the account Project Collection Build Service (account_name) access to the appropriate repository in VSTS.
2. In the Agent phase, check the box to Allow scripts to access OAuth token.
3. Now within the task you can reference the variable SYSTEM_ACCESSTOKEN to access the git repository: git clone https://randomusername:${SYSTEM_ACCESSTOKEN}@instance.visualstudio.com/proj1/_git/repo

Ref: https://github.com/Microsoft/vsts-tasks/issues/962

Solution 3

You can install Git Build Tools extension and then add "Allow Git remote access" task in your build definition. Make sure "Allow Scripts to Access OAuth Token" feature under "Options" tab is enabled.

Enable Git Remote Access

Certain operations require access to the remote repository from during a build. This task updates a remote of the Git repository on the agent to allow access to the upstream repository on Visual Studio Team Services.

Requirements

For this build task to work it is required that the Allow Scripts to Access OAuth Token option is set in the build definition options.

Parameters

Enable Git Remote Access

Remote name: Name of the remote which should be updated. Default is origin.

Related Tasks

Restore Git Remote should be called at the end of the build definition to restore the remote to its original value.

Known issues

Git-Lfs operations, like git lfs fetch still won't work with this. See this Git-Lfs issue

Add the steps for using the powershell script in the extension:

1. Create a power-shell script with the code in the "EnableGitRemoteAccess.ps1" script and add the script into source control.
2. Enable the "Allow Scripts to Access OAuth Token" option in the build definition.
3. Add a PowerShell task in build definition and set the script path the script to enable the git remote access.
4. Add another PowerShell task in build definition to commit and push the changes.

The code I use to commit and push changes:

git add .
git commit -m "changesinbuild"
git push origin master 2>&1 | Write-Host

Solution 4

Any file that you can generate from the source is generally considered as build artifact, and not added/committed/pushed to a git repo.

That being said, if you can, you should use an ssh url instead of an https one: ssh would require an ssh key, and if your private ssh key is passphrase-less, git won't have to query anything on stdin.

Another way is to use the Microsoft GCH (Git Credential Helper), which is included in Git for Windows (since Git 2.7.3, March 2016).
See this answer for an example. That would cache your login/password within the Windows Credential store.

View more solutions

Comments

[Yutao Huang]

I do need to check in those generated files back to the repo because I need to build them. I will try the GCH approach as you suggested. Thanks VonC!

[Yutao Huang]

Ok. Now I'm trying the GCH approach on a on-premises build machine. I installed the latest version of Git. I confirmed it is indeed using "manager" as "credential.helper". I pushed some changes locally from that build machine. But Git didn't ask my username/password at all! I tried removing the cached credential from the Control Panel "Windows Credentials". Git still didn't ask for username/password. A credential was automatically added to Windows Credentials. And no matter what I've tried, the VSO build is still failing... :( BTW, the build machine is domain-joined, if that matters...

[VonC]

@TonyBlues Just to be sure: git would only ask for credential when pushing to a repo, not cloning/fetching from one (unless it is a private repo)

[Yutao Huang]

It didn't ask me for credential when I was pushing changes to the repo (which succeeded). Strange...

[VonC]

@TonyBlues either your credentials were already cached, or your remote repo does not require authentication.

[Yutao Huang]

I'm sure the remote repo requires authentication. But I'm not sure whether my credentials were still cached... I already remove it from the Control Panel. I also tried "git credential-manager delete url" but it didn't work for me (complaining about unable to read from stdin).

[Yutao Huang]

Thanks for the detailed steps and screenshot! I followed every steps here (added the PowerShell scripts and checked the "Allow Scripts to Access OAuth Token" option). But I am still getting the same error when executing the push command as I mentioned in my previous comment. BTW, that was on a on-prem build agent. If I do all these on a hosted build agent, I will get a different failure message "fatal: Authentication failed for 'OAuth:*****@sample.visualstudio.com:443/SampleProject/_git/‌​SampleRepo/'".

[VonC]

Nice answer. Thank you for this follow-up. +1

[pableiros]

Links to external resources are encouraged, but please add context around the link so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline.

[xmashallax]

Option is now in Agent phase...I died while looking for this checkbox :-D

[xmashallax]

I followed the instructions but when my Powershell script arrives at the git commands nothing happens. I am using a local build agent on my development machine with VSTS, git is installed and working. Is there anything I have to do in addition?

[BBlackwo]

For me the OAuth check box was under: Task --> Phase (Run on Agent) --> Allow scripts to access OAuth token. Thanks stackoverflow.com/a/49489983/2443005.