Why is msbuild and link.exe "hanging" during a build?

Question

We have a few C++ solutions and we run some build scripts using batch files that call msbuild.exe for each of the configurations in the solutions.

This had been working fine on 3 developer machines and one build machine, but then one of the projects started to hang when linking. This only happens on the newest machine which is a quad core, 2.8ghz I think. It runs on Windows Server 2003 and the others are on XP or Vista.

This happens consistently even if I change the order of builds in the bat file.

If I run the build from the IDE on that machine it does not hang.

Any ideas about what could possibly be causing this?

I am using Visual Studio 2008.

---

Edit:

I see now that when it is hung the following are running:

• link.exe (2 instances) One with large memory usage and one with a small amount of memory usage.
• vcbuild.exe
• msbuild.exe
• vcbuildhelper.exe
• mspdbsrv.exe

---

Edit:

The exe file exists and so does the pdb file.

The exe file is locked by some process, and I can't delete it or move it. I can delete the pdb file though.

I also have the problem if I just use VCBuild.exe.

I decided to try debugging the 2 link.exe processes and the mspdbsrv.exe processes.

When I attached the debugger/MSdev IDE to them I got a message box saying that the application was deadlocked and/or that "all threads have exited".

I guess I will have to check for a service pack for that msdev install on that machine.

---

Edit:

In the debug.htm output file I get all sorts of stuff output after the link.exe command is generated.

However, for the release buildlog.htm the linke.exe line is the last line.

This is clearly a hang in the linker. Definitely a Microsoft bug.

I am now trying to figure out what the .rsp (linker response) file is.

When I issue:

link.exe @c:\\Release\RSP00000535202392.rsp /NOLOGO /ERRORREPORT:QUEUE

That is the last line in the release build log. The debug one has lots more information after that.

Reinstalling a different version of Visual Studio did not solve the problem.

I will open an issue/ticket with Microsoft. I will post an answer if I can.

Solution 1

Whole-program optimization (/GL and /LTCG) and /MP don't mix -- the linker hangs. I raised this on Connect.

The upshot is that it's a confirmed bug in VS2008; contact PSS if you want a hotfix; and the fix is included in VS2010.

If you can't wait that long, turn off /MP (slower compiles) or /LTCG (slower code).

Solution 2

Are you using xcopy in your scripts? This suggests wrapping xcopy with cmd /c " .. " as a solution.

If that wasn't it, I'd recommend to narrow things down by only letting one cpu work (i.e. removing /maxcpucount) This would rule out any form of race condition between compilation processes.

Solution 3

I had a similar problem, but with Visual Studio 2010.

This is a project that had worked fine on another computer, but just not my new one. The symptoms described matched the original Visual Studio 2008 Issue.

I was able to resolve the issue by installing the Visual Studio 2010 Service Pack 1 (SP1)

http://www.microsoft.com/download/en/confirmation.aspx?id=23691 - or just go to microsoft and search for "Visual Studio 2010 Service Pack 1"

I had run my windows "check for updates" and had thought I had installed all service packs, but apparently, I had not installed any Visual Studio Service Packs.

After installing the VS2010 SP1, I no longer had this issue. I confirmed that I had installed VS2010 and the Service Pack 1 on other older computer with the working project a while back.

Solution 4

You could try this: Open the build dialog via Menu -> Tools -> Options -> Projects and Solutions -> Build and Run

Here you can set "MSBuild project build output verbosity" to "Diagnostic". Maybe this will deliver more information on what is going wrong.

In the same dialog you can set "Maximum number of parallel project builds" to 1. Maybe this works around the link.exe "hang".

Solution 5

mspdbsrv.exe is used to combine all debug info into one pdb file. The VS2005 version of mspdbsrv.exe is buggy, it might be that the VS2005 version has some of the same issues. Killing it before building is making a difference for some people. We're going to add it to our builds as well since we're regularly suffering from unknown PDB errors.

View more solutions

Comments

[Tim]

Hmm. I do have a post-commit step that calls a bat file which does a copy of *.exe and *.pdb to another directory... That sounds like a good culprit. i will take that out and see the results.

[Tim]

I think incremental linking is off because it cannot work with MP builds. I'll check. This script always does a clean then a build.

[Tim]

I have set the parallel to 1, but will also try diagnostic. thanks

[Tim]

That does nothing for me in a command line build from MSBuild... Unless I am missing something. I will try verbose switch for linker.

[Roger Lipscombe]

MSBuild verbosity doesn't affect VC++ projects; they use VCBuild.

[Tim]

Thanks. I will look into it. I appreciate the help.

[Roger Lipscombe]

It looks like a timing thing. Some do, some don't. On a single machine, sometimes it will, sometimes it won't. Compiling with CppUnit seemed to trigger it more reliably for us. Something about the build...

[willll]

I had exactly the same issue with MS visual studio 98, running in a Virtual Machine configured with 2 CPUs, removing one of the CPU solved the issue