Real-world use of Mercurial with a Team Foundation Server?

Question

My shop uses TFS & is generally happy with it with the exception of the lack of local repository commits/reverts. I'm starting to use Mercurial locally myself to help manage smaller chunks of changes, then posting them to TFS. I see that Subversion has a 'bridge' component to automagically enable this if the central VCS is Subversion. I have not found one for Team System. This encourages me that other folks have gone down this path with integrating DVCS with CVCS systems.

(1) Does anyone know of one? I'm sort of doubting it (quick search didn't find anything).

(2) Is anyone using Mercurial/TFS in this manner? If so, can you share your experiences. I'm particularly looking for any insights into what issues might come up that aren't obvious regarding commits to TFS after significant activity via Mercurial.

It seems like a total win-win so far with just me using if for a few days--but I know enough then to think it's just that easy.

Solution 1

Not sure if this is anything you don't already know, but I've been using mercurial locally for a little while now, and so far I think the benefits are outweighing the added overhead of managing the two source control systems. Here is how I've been doing things:

1. I made my TFS checkout an HG repository which I consider my "master". I get updates from TFS and commit them to this repo, so this contains the most current state of the project from TFS. The important thing here is that there are no changes to this made independent of a TFS update or an Hg merge (which is part 2)

2. Anytime I need to make a change, I clone my "master" repo and do my work there. I've found that a clone per feature or story is actually pretty easy to manage, and feels pretty clean. Once I complete a feature, I do an Hg merge back to the "master" repo, which has had all of the TFS updates applied. This allows me to use Mercurials merge capabilities, which are so far superior to TFS as to call into question how TFS can claim to merge code at all. Once the merge is complete, I commit it in Hg, and then check those changes into TFS. The best part of this is that when I do the checkin to TFS, I do not have to merge anything. Very, very nice.

Now, here are the issues I've found with this approach:

1. The biggest is the fact that TFS is lousy at finding changes. There is a make writable plugin which you can use to make the modified files writable when they are updated/merged by Mercurial. There are two options I have found for this. You can either force TFS to go offline, at which point it will assume anything writable needs to be checked in, or you can use the compare tool in the source control tool and select the changed files and individually check them out. Both are crappy IMO

2. The source control bindings are still there at the project level, even if you exclude the TFS source control files from your hg repository (which you should do). This isn't entirly obvious until you add a file to the solution, at which point it tries adding it to source control. You can "Undo Pending Changes" and get rid of the source control add, but it's really annoying.

The good news is that I used this approach to work through a rather massive merge which I think would have caused me to turn to some form of hard drugs had I been forced to use the TFS tools to do it.

I have not yet applied this to updating branches within TFS, but my guess would be that it would be much much better than the options you are given for merging in TFS. On a related note, since you can check in chunks of working functionality at one time, using the TFS merge would be less problematic just because all changes needed for a feature would be together in one place.

One thing I have not tried to tackle is sharing this across the entire team. Part of the reason is that it really doesn't have to be a team-wide thing. I work remotely, so having a local repository is a big deal, and saves a lot of time. The other members of my dev team may or may not get the same benefit from this approach, but I find it pretty cool that I can without effecting the way they work.

Update I've been wanting to update this response for a while with additional info based on comments and some of my experiences working with large TFS repositories.

First as @Eric Hexter points out in the comments, you can utilize the rebase extension to better integrate commits from your work repositories into your main TFS repository. Though, depending on how you want your commits to appear to TFS you may want to use the collapse extension to squash your changes into a single commit (this can make rollbacks in TFS easier). There is also the "online" command from TFS PowerTools which can make the work of letting TFS know what has changed easier (thanks again to Eric for mentioning that in his blog post)

Now, when I originally wrote this I was working on a project that had only one TFS branch that developers were using, and was fairly small, so cloning repositories was no big deal. I later found myself working on a project that had a repo that was about 1.5GB after checkout, and much bigger after the build, and involved switching between branches in TFS quite often. Clearly this approach is not well suited to this environment (particularly since at one point it was impossible to build the solutions in an arbitrary directory.

The size problem is best handled by utilizing a technique similar to gits topic branches rather than cloning the repositories to new directories. There are a couple options for this. I think the best is actually to use the bookmark extension and create topic "bookmarks" rather than topic branches. You can use named branches as well, but they have the slight disadvantage of being permanent, and traveling with any clones you may do (if you want to share your nifty TFS-Hg hybrid with a colleague). Bookmarks are local to your repo, and effectively point to a commit and travel with the head. They are implemented so that they can be used in any place where Hg is expecting a revision (so merges, updates, etc). You can use these to create a TFS bookmark as your primary "branch" that only gets updates from TFS, and merges from the topic work, which would each have their own bookmarks, and you could delete once you have committed back to TFS. If you would rather use named branches, then you can apply exactly the same techniques, which is handy.

Now, the multiple branch problem is trickier, especially since TFS "branches" are actually copies of every file from the original branch, which means that each time you pull in branches from TFS, your repo is going to get that much bigger. One possible way to deal with this is use a combination of named Hg branches, and bookmarks, so that you have a branch for each TFS branch, and then create bookmarks for your work off of those branches. The real headache in these scenarios is actually dealing with TFS workspaces through all of this. You can remove the mappings in your workspaces and get pretty far, but once you map back to your working directory you've got to be careful of TFS stomping on files (this is actually where the TF PowerTools come in handy). Trying to leave the workspace attached while your switching branches around gets ugly quick. A couple tools that are nice to have in your toolbelt are the Hg purge extension and the TF PowerTools "scorch" command. Both effectivly remove files that are not in version control (technically "scorch" ensure TFS and your local working directory match, so it could update files as well).

For me, though, this process became quite burdensome and error prone. I've recently switched to useing git with git-tfs, since it manages TFS Workspaces for me, and removes a lot of the burden associated with that side. Sadly there does not appear to be an "hg-tfs" out there anywhere, or I would probably have chosen that.

Solution 2

If you're not stuck on mercurial, there is a sweet git/tfs integration project that I've been using called git-tfs. It's very similar to git-svn, but pushes/pulls from TFS instead. Check it out at http://github.com/spraints/git-tfs

Solution 3

@Eric, your post at lostechies was most helpful. With VS2010 I had to add options /diff and /deletes to the tftp online command in the push script to get changed and deleted files to be checked in to TFS. Initially I was getting an error from push when a file has been deleted (from -working) that hg update is
"unable to remove FileXyz : access is denied".
I installed the MakeWritable.py extension but that only works when files are opened not deleted. So I added a call to attrib to remove the READ-ONLY from all files in the project and then restore it afterwards (excluding the .hg folder) I also added the /diff option so that differences are detected by MD5 checksum instead of depending on the READ-ONLY attribute. Seems to be working fine now.

=====FILE: push.ps1=====
$projName = "TicTacToeCMMI"
$tftp = "C:\Program Files\Microsoft Team Foundation Server 2010 Power Tools\TFPT.exe"
$tf = "C:\Program Files\Microsoft Visual Studio 10.0\Common7\ide\tf.exe"

hg push
cd ..\$projName-tfs  
"Syncing -tfs workspace with TFS server"  
&$tftp scorch /noprompt /exclude:.hg',_Resharper*',*.user  
"Making all files in -tfs writable"
attrib -R /S /D *
"Updating -tfs with latest push from Mercurial"
hg update -C -y
attrib +R /S /D *
attrib -R /S /D .hg\*
"Resyncing Mercurial changes with TFS Server"  
&$tftp online /adds /deletes /diff /exclude:'.hgignore,.hg,bin,obj,*.ps1,_Resharper*,*.lnk,*.user,*.suo,*.vspscc'  
"Checkin"  
&$tf checkin  
cd ..\$projName-working  
cmd /c pause  

====FILE: pull.ps1=====
$projName = "TicTacToeCMMI"
$tf = "C:\Program Files\Microsoft Visual Studio 10.0\Common7\ide\tf.exe"
$username = cmd /c set USERNAME
$username = $username.SubString($username.IndexOf("=")+1)

function pull {
    cd ..\$projName-tfs
    &$tf get
    hg commit -A -m "from tfs" --user $username
    cd ..\$projName-working
    hg pull --rebase
}
pull  
cmd /c pause  

I had a bit of a learning curve with PowerShell scripts which I hadn't used before. For others like me the scripts are run with a shortcut like this:

TARGET: C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe C:\dev\TicTacToeCMMI-working\push.ps1
START IN: C:\dev\TicTacToeCMMI-working

I put push and pull shortcuts on my task bar so push/pull to/from TFS is a single click

Solution 4

I know that some people have used hgsubversion with the Subversion bridge. I don't know how well it worked, and I've never had to use TFS.

As far as I'm aware, there's no "more native" bridge than using TFS -> Subversion Bridge -> hgsubversion, but I've also heard that it works fairly well. My extremely limited understanding of TFS suggests that its internal model should be similar enough to Subversion for things like hgsubversion to work really well.

Solution 5

I'he just put together a small tool, HgTfs, which tries to accomplish the goal of syncing Mercurial and TFS repositories. It's really simple and and has only three commands: clone, pull and push. Here is my Bitbucket repo:

https://bitbucket.org/thepretender/hgtfs

There is also a blog post describing the workflow and usage scenarios (actually, wiki pages are just parts of this blog entry):

http://www.olegtarasov.me/Post/2013/07/Mercurial-to-TFS-bridge-(hgtfs)

The code is hacky, but it seems to get the job done. I would really appreciate any feedback or forks :)

View more solutions

Comments

[Tom Willis]

i did a similar approach except perforce was the blessed central repo. And the issues you listed exist there as well. I'm trying to adjust my workflow to feature focused and the patch queues functionality so far I've been pretty happy with it.

[MrHinsh - Martin Hinshelwood]

Were you not able to use the Codeplex bits? blogs.msdn.com/bharry/archive/2010/01/27/…

[Richard Banks]

As far as I'm aware the mercurial support is a completely separate server from TFS. It's not a bridging arrangement like it is with the Subversion support.

[StingyJack]

Merging sucks less for TFS when you use the built in tool. James Manning has a nice article on how to swap it out for something much better (I like SourceGear's DiffMerge) blogs.msdn.com/b/jmanning/archive/2006/02/20/…

[Sean Kearon]

You can use also Mercurial to front SVN, details here: abdullin.com/journal/2010/3/22/…

[Maggie]

Thanks Jon, I'll check it out.

[Lars Christian Jensen]

I blogged about my experience using this setup. lostechies.com/blogs/hex/archive/2010/06/22/… I added some powershell scripts to automate the use of hg and the TFS powertools. The other extension I used was hg rebase to keep the merging on the HG side really simple. I know have two simple powershell commands. push and pull, which do the heavy lifting of running the hg and tfs commands lines in the correct order.

[ckramer]

@Eric, thats some good stuff. When I wrote this I wasn't aware of the rebase extension, Its probably going to find its way into my toolbox in the near future though. The powershell commands are awesome, too.

[Jeramy Rutley]

I've been thinking about this for a while now, and ... I'm still stuck on how you would handle merging using Mercurial between existing TFS branches. If I make a change in a branch, I'm usually required to merge it to the root with TFS merge. Any ideas on how to leverage the Mercurial repository in that case? Would I need one hg clone-per-TFS-branch? Is there a way to automate the creation of hg clones when someone creates a TFS branch?

[ckramer]

The way I typically work is to do my change in the root, and then merge it into the branch. In that case the workflow is the same, there is just the added step of doing the merge in TFS afterward. Now, if you are unable to do that, then you are probably looking at creating a new Hg repo for your new TFS branch (which I've done in cases where branches are way different), but your still looking at duplicating some work there. What would be nice is to make a change in one Hg repo, merge it into TFS, then push the change to another Hg repo for your other TFS branch. Not sure if that works tho.

[Lee Harold]

This general approach seems to be working for me. It's not without friction, but is at least an improvement over dealing with TFS all the time. In addition to Eric Hexter's blog post, there are some good tips here as well: blog.dabide.no/idabide/2010/10/18/…

[ckramer]

I've been considering updating this response with some "lessons learned" since my original answer, but I don't know if it will be more or less useful that way. The tfpt online command (mentioned in the blog post @Lee linked above) is one I've recently found that helps quite a bit. I've also now found myself working on a project that is too large for the original approach of multiple clones to work well, which leads to Named Branches, and a whole new set of problems. Merging between TFS branches via Hg is a pain, but I've had some success using the Transplant Hg plugin to cherry-pick changes

[Eddie Groves]

@ckramer An edit to your original answer with further details/tips would be great.

[ckramer]

Took me a while, but I've finally gotten an update for those who may be curious about working with larger code bases and multiple TFS branches.

[Marek Dzikiewicz]

I use Mercurial with TFS on a large repository with many per-release branches. I store each TFS branch in separate subrepo (similar to the way the branches are stored in TFS). This way, when I work on the subrepo I have a relatively small hg repo (in contrast to storing all branches in a single hg repo) which makes Mercurial work faster. Additionally, when branches are created in TFS, I create a clone of the subrepo of the source branch and add it as a new subrepo. This way I have a complete history from the previous branch (which is missing in TFS).

[James Mills]

How well does this Powershell script work? Anyone have any experience using this? --James

[Sam Mackrill]

Nice, I will be using this when we move from ClearCase to TFS (/sigh) [Still looking for the "Shoot me with a railgun option" ... totally need that :p]