Step into Kernel (VmWare+WinDbg)

May 31st, 2007 - Fernando Roberto

It is enough with this idle talk and let’s go to what really matters. Speaking about Kernel Debugging, I have written a post describing the steps needed to Kernel Debugging with WinDbg, using two machines and a serial cable. But having two machines dedicated to this practice is a luxury that not everyone has. Thus, in another post, I have commented about installing and using SoftIce to debug drivers on a “single machine dedicated to debug”. I tell you “dedicated machine to debug” because using your development machine to debug drivers may not be one of your best ideas. The most courageous and confident people in their own code still risk themselves in this practice. Well, I cannot say much, I myself have already done that in lean times. Anyway, we have not fled the need for two machines to have a minimal environment for development and test drivers. An alternative to these scenarios is the use of a single machine, but the one that can have enough memory and CPU to run a virtual machine, in order to debug the kernel. In this post, I will take the steps to use a virtual machine as a guinea pig to test and debug drivers.

Configuring a virtual machine

The natural communication way between real machines using WinDbg to do Kernel Debugging is a serial port. To follow the same steps, we need to make your virtual machine to gain a serial port to make this communication possible. With your virtual machine turned off, click on the Edit vitual machine settings option.

A window will appear with the list of devices already installed on your machine in Hardware tab. Click Add… in order to add a new device and select Serial Port in the device list that is displayed on the screen below.

After clicking on Next, select the Output named pipe. This will cause the serial port on the virtual machine to communicate with the real machine, via a named pipe. Clicking on OK, the specific settings of the named pipe will be presented. The pipe name will be used later in one of the WinDbg settings so, if you want to use a name other than the one suggested here, try to remember this same name later. Then, change the value of the second combo to indicate that the other communication ending will be an application, in this case WinDbg. After that, just click on Finish.

At the end of these settings, select the Yield CPU on poll option as shown below.

Configuring the TARGET machine

Made all that “hocus-pocus”, now we have to set the TARGET machine system in the way it will able to make the Kernel Debugging. Remember that from Windows that runs inside the virtual machine point of view, the named pipe does not exist, it is just a serial port. If you still do not know what TARGET machine is a or how it can be set, then take a look in this post and follow the described steps in the part where a TARGET machine is configured .

Configuring the HOST machine

Assuming that your TARGET machine has been configured, now we have to configure WinDbg in a way it can be connected to a machine using named pipe, instead of a serial port. For that, I usually create a batch file that contains the following command line:

start C:\Progra~1\Debugg~1\windbg.exe -b -k com:pipe,port=\\.\pipe\com_1,resets=0

Creating a batch file is not a required step, you might want to retype everything in the Run… window, every time the debugging starts; but it’s all up to you. Notice that the pipe name appears here. I hope you still remember which name you have chosen.


Now we have everything already configured, it is just plug and debug. In this story, the virtual machine is the one that creates the named pipe that is opened by WinDbg. Thus, if you start WinDbg with the shown parameters before starting the virtual machine, you will see the window below stating the pipe was not found.

So, the sequence is, firstly, to connect the virtual machine, select the debug option in the Boot as shown below, and only after that, you should start the WinDbg with the parameters described above.

When WinDbg is finally connected to the virtual machine, via the named pipe, we have the following messages in our Command Window within WinDbg.

Thereafter, you know… You pick up the bugs.

Once more I hope I have helped you.
See you next time. 🙂

2 Responses to “Step into Kernel (VmWare+WinDbg)”

  1. Kaveesh says:

    Good post. Nice job explaining. Take care…

  2. Florian Schaper says:

    thank you for this nice post!

Leave a Reply