How does the Visual Studio Attach to Process work?

Question

I've always wanted to know the inner-workings of Visual Studio's debugger and debuggers in general. How does it communicate and control your code, especially when it's running inside a host process or an external network server (attach to process)? Does the compiler or linker patch your code with callbacks so that the debugger is given control? If it indeed works this way, how do interpreted languages such as JavaScript containing no debug code work?

Solution 1

Generally speaking, Windows provides an API for writing debuggers that let you examine and modify memory in another process and to get notifications when exceptions happen in another process.

The debug process sits in a loop, waiting for notification from events from the process under inspection. To set a breakpoint, the debugger process modifies the code in the debugee to cause an exception (typically, an int 3 instruction for x86).

The compiler and linker work together to make the symbol information about a program available in a format that can be read by debuggers. On Windows, that's typically CodeView in a separate PDB file.

In the Unix-derived world, there's an API called ptrace that does essentially the same sorts of things as Windows debugging API.

For remote debugging, a small program is placed on the remote machine that communicates with and acts on behalf of the actual debugger running on the local machine.

For interpreted languages, like JavaScript, the debugger works with the interpreter to give the same sorts of functionality (inspecting memory, setting breakpoints, etc.).

Solution 2

Windows includes support for debuggers. A process has to enable debugger privilege, and once this is done that process can attach to any other process and debug it using windows debugger functions

http://msdn.microsoft.com/en-us/library/windows/desktop/ms679303(v=vs.85).aspx

For something like javascript, seems like you would need the equivalent of a javascript debugger.

In the case of a Visual Studio multi-process project, you typically have to switch which process the debugger is attached to in order to debug that process. I don't know if there's a way to have pending breakpoints set for multiple processes at the same time. There could be other debuggers that work better with multiple processes, but I haven't used such a tool.

Comments

[ATL_DEV]

So if an OS is very primitive and doesn't have specific debugging facilities available to a debugger, it won't work? Debugging has been around since DOS and I'm not certain it had any specific support for debuggers.

[ATL_DEV]

What about Linux and other more archaic OSes? Do they have similar facilities?

[rcgldr]

@ATL_DEV - The issue is that some OS's restrict the usage of debuggers. Windows doesn't do this, which allows trainer type debuggers to be used on any process. In the case of DOS in real mode, there aren't any privileged instructions needed, so DOS doesn't need to do anything special to allow debuggers. Posix systems may have an optional debugging kernel, mostly used to debug device drivers. Windows uses remote debugging, using two computers, to debug device drivers.

[Adrian McCarthy]

@ATL_DEV: On non-Windows OSes, debug info is often embedded in the binary using DWARF. I don't know about debugging APIs on other OSes. The question seemed to be specifically about Visual Studio and thus Windows. You can study the code for cross-platform open source debuggers, like LLDB, to learn more about the differences between OSes.

[Adrian McCarthy]

Some of the answers for this related question talks about other OSes: stackoverflow.com/questions/216819/…