Using msbuild I want to update a config file with values from teamcity

Question

I have some XML that looks something like this:

<?xml version="1.0" encoding="utf-8"?>
<XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>C:\DevPath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>C:\DevPath2</value>
    <encrypted>False</encrypted>
  </item>

   ....

</Provisioning.Lib.Processing.XmlConfig>

In TeamCity I have many system properties:

 system.HlrFtpPutDir     H:\ReleasePath1
 system.HlrFtpPutCopyDir H:\ReleasePath2

What sort of MsBuild magic can I use to push these values into my XML file? In all there are 20 or so items.

Solution 1

I just blogged about this (http://sedodream.com/2011/12/29/UpdatingXMLFilesWithMSBuild.aspx) but I'll paste the info here for you as well.

Today I just saw a question posted on StackOverflow asking how to update an XML file using MSBuild during a CI build executed from Team City.

There is not correct single answer, there are several different ways that you can update an XML file during a build. Most notably:

1. Use SlowCheetah to transform the files for you
2. Use the TransformXml task directly
3. Use the built in (MSBuild 4.0) XmlPoke task
4. Use a third party task library

1 Use SlowCheetah to transform the files for you

Before you start reading too far into this post let me go over option #3 first because I think it’s the easiest approach and the most easily maintained. You can download my SlowCheetah XML Transforms Visual Studio add in. Once you do this for your projects you will see a new menu command to transform a file on build (for web projects on package/publish). If you build from the command line or a CI server the transforms should run as well.

2 Use the TransformXml task directly

If you want a technique where you have a “main” XML file and you want to be able to contain transformations to that file inside of a separate XML file then you can use the TransformXml task directly. For more info see my previous blog post at http://sedodream.com/2010/11/18/XDTWebconfigTransformsInNonwebProjects.aspx

3 Use the built in XmlPoke task

Sometimes it doesn’t make sense to create an XML file with transformations for each XML file. For example if you have an XML file and you want to modify a single value but to create 10 different files the XML transformation approach doesn’t scale well. In this case it might be easier to use the XmlPoke task. Note this does require MSBuild 4.0.

Below are the contents of sample.xml (came from the SO question).

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>C:\DevPath1</value>
    <encrypted>False</encrypted>
  </item>
  <item
    <key>HlrFtpPutCopyDir</key>
    <value>C:\DevPath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So in this case we want to update the values of the value element. So the first thing that we need to do is to come up with the correct XPath for all the elements which we want to update. In this case we can use the following XPath expressions for each value element.

• /Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value
• /Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value I’m not going to go over what you need to do to figure out the correct XPath because that’s not the purpose of this post. There are a bunch of XPath related resources on the interwebs. In the resources section I have linked to the online XPath tester which I always use.

Now that we’ve got the required XPath expressions we need to construct our MSBuild elements to get everything updated. Here is the overall technique:

1. Place all info for all XML updates into an item
2. Use XmlPoke along with MSBuild batching to perform all the updates

For #2 if you are not that familiar with MSBuild batching then I would recommend buying my book or you can take a look at the resources I have online relating to batching (the link is below in resources section). Below you will find a simple MSBuild file that I created, UpdateXm01.proj.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <ItemGroup>
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates-SampleXml">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The parts to pay close attention to is the XmlConfigUpdates item and the contents of the UpdateXml task itself. Regarding the XmlConfigUpdates, that name is arbitrary you can use whatever name you want, you can see that the Include value (which typically points to a file) is simply left at ConfigUpdates-SampleXml. The value for the Include attribute is not used here. I would place a unique value for the Include attribute for each file that you are updating. This just makes it easier for people to understand what that group of values is for, and you can use it later to batch updates. The XmlConfigUpdates item has these two metadata values:

• XPath -- This contains the XPath required to select the element which is going to be updated
• NewValue -- This contains the new value for the element which is going to be updated Inside of the UpdateXml target you can see that we are using the XmlPoke task and passing the XPath as %(XmlConfigUpdate.XPath) and the value as %(XmlConfigUpdates.NewValue). Since we are using the %(…) syntax on an item this start MSBuild batching. Batching is where more than one operation is performed over a “batch” of values. In this case there are two unique batches (1 for each value in XmlConfigUpdates) so the XmlPoke task will be invoked two times. Batching can be confusing so make sure to read up on it if you are not familiar.

Now we can use msbuild.exe to start the process. The resulting XML file is:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>H:\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>H:\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

So now we can see how easy it was to use the XmlPoke task. Let’s now take a look at how we can extend this example to manage updates to the same file for an additional environment.

How to manage updates to the same file for multiple different results

Since we’ve created an item which will keep all the needed XPath as well as the new values we have a bit more flexibility in managing multiple environments. In this scenario we have the same file that we want to write out, but we need to write out different values based on the target environment. Doing this is pretty easy. Take a look at the contents of UpdateXml02.proj below.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="UpdateXml" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  
  <PropertyGroup>
    <SourceXmlFile>$(MSBuildProjectDirectory)\sample.xml</SourceXmlFile>
    <DestXmlFiles>$(MSBuildProjectDirectory)\result.xml</DestXmlFiles>
  </PropertyGroup>

  <PropertyGroup>
    <!-- We can set a default value for TargetEnvName -->
    <TargetEnvName>Env01</TargetEnvName>
  </PropertyGroup>
  
  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env01' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>H:\ReleasePath1</NewValue>
    </XmlConfigUpdates>
    
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>H:\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>

  <ItemGroup Condition=" '$(TargetEnvName)' == 'Env02' ">
    <!-- Create an item which we can use to bundle all the transformations which are needed -->
    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath1</NewValue>
    </XmlConfigUpdates>

    <XmlConfigUpdates Include="ConfigUpdates">
      <XPath>/Provisioning.Lib.Processing.XmlConfig/item[key='HlrFtpPutCopyDir']/value</XPath>
      <NewValue>G:\SomeOtherPlace\ReleasePath2</NewValue>
    </XmlConfigUpdates>
  </ItemGroup>
  
  <Target Name="UpdateXml">
    <Message Text="Updating XML file at $(DestXmlFiles)" />
    <Copy SourceFiles="$(SourceXmlFile)"
          DestinationFiles="$(DestXmlFiles)" />
    <!-- Now let's execute all the XML transformations -->
    <XmlPoke XmlInputPath="$(DestXmlFiles)"
             Query="%(XmlConfigUpdates.XPath)"
             Value="%(XmlConfigUpdates.NewValue)"/>
  </Target>
</Project>

The differences are pretty simple, I introduced a new property, TargetEnvName which lets us know what the target environment is. (note: I just made up that property name, use whatever name you like). Also you can see that there are two ItemGroup elements containing different XmlConfigUpdate items. Each ItemGroup has a condition based on the value of TargetEnvName so only one of the two ItemGroup values will be used. Now we have a single MSBuild file that has the values for both environments. When building just pass in the property TargetEnvName, for example msbuild .\UpdateXml02.proj /p:TargetEnvName=Env02. When I executed this the resulting file contains:

<Provisioning.Lib.Processing.XmlConfig instancetype="XmlConfig, Processing, Version=1.0.0.0, Culture=neutral">
  <item>
    <key>IsTestEnvironment</key>
    <value>True</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutDir</key>
    <value>G:\SomeOtherPlace\ReleasePath1</value>
    <encrypted>False</encrypted>
  </item>
  <item>
    <key>HlrFtpPutCopyDir</key>
    <value>G:\SomeOtherPlace\ReleasePath2</value>
    <encrypted>False</encrypted>
  </item>
</Provisioning.Lib.Processing.XmlConfig>

You can see that the file has been updated with different paths in the value element.

4 Use a third party task library

If you are not using MSBuild 4 then you will need to use a third party task library like the MSBuild Extension Pack (link in resources).

Hope that helps.

Resources

• MSBuild batching: http://sedotech.com/Resources#Batching
• SlowCheetah – XML Transforms extension: http://visualstudiogallery.msdn.microsoft.com/69023d00-a4f9-4a34-a6cd-7e854ba318b5
• MSBuild Extension Pack (has a task to update XML files): http://msbuildextensionpack.codeplex.com/
• Online XPath tester: http://www.whitebeam.org/library/guide/TechNotes/xpathtestbed.rhtm

Solution 2

We change config values for our different build environments (e.g. dev, staging, production) using config transformations. I assume that config transforms probably won't work for you, but if it's a possibility, check out this answer which shows how to apply .Net config transforms to any XML file.

An alternative would be to use the FileUpdate build task from the MSBuild Community Tasks project. This task allows you to use regular expressions to find and replace content in a file. Here's an example:

<FileUpdate Files="version.txt" Regex="(\d+)\.(\d+)\.(\d+)\.(\d+)" ReplacementText="$1.$2.$3.123" />

Because you'd be passing TeamCity system properties into FileUpdate if you decide to go with the second option, take a look at this question to see how system properties can be referenced in an MSBuild script.

Comments

[Loofer]

Epic answer! Thanks so much Sayed.

[GarethD]

Check out this answer if you want something more simple: stackoverflow.com/a/21733510/1979461

[Jason Slocomb]

@Sayed This is a great answer! Note: If you use XMLPoke and your values contain any msbuild special characters, you will need to encode them. See msdn.microsoft.com/en-us/library/ms228186.aspx