Archive for the ‘Debug’ Category

Getting up-to-date

Thursday, June 5th, 2008

Once more I have not posted any for a long time. Now you might be thinking: “Here he comes with that chatter he has no time, everything is difficult and God does not like the kernel developers and so on.” Well, I am skipping this part of apologies and just saying what I have been doing during that time.

Windows Driver Training

Well, let me see where I have stopped. My last post was during the period when I was teaching a driver development course for a private class. The course was given in five Saturdays of eight hours each. Needless to say, it took me a while. Okay, okay, okay, no excuses.

Forth C/C++ Programmers’ Meeting

On next Saturday following the end of the course, I took a tour around this event. They left the door open and I could get through for a peek and talk a little about Windows drivers’ architecture and development using C language. I confess that the invitation to participate in this event was a surprise to me. I’m not a great C++ programmer but I use the C language with “classes”, as just my friend Strauss gets used to say. I can say it was very interesting to participate in this event with a very high technical level.  I had an opportunity to realize the way the language may be used in different scenarios. The slides from my talk are available at this link.

Going back to Boston

It seems that things only happen on Saturdays. Anyway, on the next Saturday following the meeting, I embarked to Boston on a trip taking about one month. The last time I had been there was due to the OSR training described in this post. This time, it was about an IBM integration program. I could personally meet the people I work with and previously had only met through Web Conferences. Again, it was very interesting and I tried to enjoy it mostly. We were two Brazilians and one Indian. I missed our Brazilian coffee a lot. In this photo, there are below, from left to right, David E. Jones (Manager), Scott D. Hankin (Documentation), William C. Oliveira (Linux), William H. Taber (Linux), Mridula Muppana (Testing), Paul Ganshirt (Windows), Niraj Kumar (AIX), Kaveesh Mishra (Windows), Fernando Roberto (Windows) and Richard Kissel (AIX and Linux). This is just one part of the whole MVFS team.

Don’t you know what MVFS is? It is a File System that is part of a product called ClearCase. Don’t you know ClearCase? Well, beyond Wikipedia, I had the pleasure of bumping into an observation about the ClearCase at Rajeev Nagar’s book. This book, as I have never gotten tired of saying, is still the only respectable reference on Windows File System development, even though it was first published in 1997. Where is ClearCase in this story? Well, if you’re sick like me and have this book, take a peek at the beginning of chapter nine. On the first page of this chapter, you’re going to find the following passages.

Yeah, that’s really cool! 🙂

Back to the Classes

For a whole month after I had come back to Brazil, I spent some time chasing the content I had lost at the university. Yeah, I’m still graduating in Computer Engineering. I have been working on my final project (Final Course Project). My project must obligatorily have a Windows driver; that is the least thing I could do. Finally, I will be able to show what I can do to my classmates. During the course, some friends had asked me what I’ve worked with, since I have often been reading big books from Microsoft. After trying to explain the simplest way possible, drawing or even using puppets, they still get with that interrogation face. Well, at the time that the blue screen shows up, they will eventually understand.

I took the tour in the United States to buy this development kit of Altera. This kit will be part of my project and in the future, I will be able to use it in my courses on driver development. Nowadays I have just counted on the OSR training boards, as I already talked about in this post. Unlike the OSR boards, this kit may have its hardware defined by a language called VHDL, and thus, it can cause the board to have a wide variety of behaviors and be able to illustrate the driver interface construction for every situation. This card costs around R$ 1,700.00 in Brazil, while I had only paid U$ 150.00 there in USA. Practically, a candy price. Do you know, when we used to go to the bakery to buy that ice cream cone that came with a stuck toy? So, it was almost the same thing. Another interesting thing in this kit, besides its price, is the possibility of using its USB and PCI interfaces. That will be very funny.

Portability and Performance Seminar

In this last weekend, I also attended that event organized by Brazil C/C++ group. It’s amazing to notice how these events are bringing more and more people to it. I took some pictures, but I’ll leave the comments on behalf of my friend Slug, who has a cool post about it.

Back to the bloggers’ world, I will try not to abandon you, guys for so long. Some excuses apart, it has not been that easy.

CYA.

Step into Kernel (Vista + USB2)

Thursday, March 6th, 2008

I have already mentioned into another post that it is possible to perform Kernel Debugging via a 2.0 USB port. In this post I’m going to demonstrate such a 2.0 USB connection working. Yeah, that’s a good thing. So let’s stop this idle talk and go to the business. If you know nothing about Kernel Debugging, you can read this post which brings the concepts and basic steps on the subject or you can ask yourself “How have I ever gotten into this site?” and return to ORKUT.

The USB bus does not allow us to have direct connections between two computers. In order to use a USB port for this purpose, we must rely on the additional hardware helping. Although it is named as Debug Cable, this is a small device that connects computers via USB ports. I do not know if there are already other manufacturers for the Debug Cable, but the one I’m using at this experiment is NET20DC.

Using the Debug Cable

Using the Debug Cable is a luxury that only Windows Vista and posterior systems can enjoy. It needs to be plugged into a 2.0 USB port without going through any hub at the TARGET side. But how am I supposed to know for sure which of my ports is actually 2.0?

An easy way to know that, especially for those ones who have installed WDK, is to build the USBView sample which is in the “C:\WinDDK\6000\src\usb\USBView” folder of WDK. This program lists the drivers and their respective connected USB devices. Thus, when connecting the Debug Cable to the TARGET machine, USBView should inform us what port this new device is connected to. Note that in this window we can see which controller the port we are using belongs to.

Windows will prompt a driver when detecting this new USB device presence. No driver must be installed to use the Debug Cable on the TARGET side. So, you can select “Do not show this message again for this device” in the window shown below. Not installing any driver to the TARGET side makes sense. Remember that this interface will be used by BIOS of the TARGET machine and its control wont be passed to the operating system core. Do not forget, yet, there is another requirement to use the Debug Cable. The TARGET machine’s BIOS must implement this debug interface control. The most unfortunate part of this story is about checking whether your BIOS offers such support. All that you need to do is just to try. That means you may have the Debug Cable, a free 2.0 USB port and the cables but even that, there is a risk of not connecting it because TARGET BIOS might not support the debug interface. I’m glad to know that my laptop offers this support and have not spent this money for anything importing the Debug Cable. Phew!

Modifying Boot.ini

Sorry? Boot.ini? I’m sorry to tell you that the Boot.ini has gone way. Windows Vista uses a new architecture called Boot Configuration Data (BCD). That has been the end for Boot.ini file. To edit the BCD settings, we use the BCDEdit.exe tool, a console application that needs to be run with administrative rights. Executing this tool without any parameters returns it to the following output.

To configure the system in order to have two boot options, the one with enabled debugging and the  other isn’t, follow the described steps below. The Boot Manager managers the Boot Loaders which are either possible to be the other operating systems, even prior to Windows Vista or configuration sets of the same system. Each of these items is identified by a GUID. Note that the first command makes a copy of the current configuration to the  one contains the “Windows Vista Guinea Pig Mode” string as a description. In response to this copy, the tool returns a resulting GUID of the copy. Also, observe that the second command enables the kernel debugging to the setting identified by the GUID just received. Right now, if we take a look at the settings, it will have a result like the same image being displayed below.

Nice!!! Now we have to configure the media to be used by the Kernel Debugger. I believe that most systems still use serial ports for this purpose. In this case, the most common configuration is to use the COM1 serial port with a 115200 baud rate. To configure these settings, use the following command line. To see the current settings, just run the second command line as it is shown below.

If you do not know how to use the serial port as a communication way for the kernel debug, this is the post that talks about it. However, in this post I am commenting about kernel debug using a 2.0 USB as a communication way. In this case, the settings are described at the command line that is shown below. The TARGETNAME is able to receive any name. I have used the “WINDBG” name for obvious reasons, but you can use anyone.

As you may have guessed, the settings on the debug mode of communication is global, that is not linked to one or another boot configuration identified by a GUID.

I believe that on the TARGET side everything is ready to make the link. Thus, when you reboot the victim machine, the following menu will be displayed by the Boot Manager:

Configuring the HOST

At the HOST side of this game, we must add the driver that controls the Debug Cable to it and allow Windbg to use this device. Although I’m using Windows Vista on the HOST side, nothing would prevent me from using Windows 2000 or higher on this side of the conversation. The fun starts when you plug the device and click “Locate and install driver (recommended)” into the “New Hardware Found” window that has already appeared into this post. At this time, Windows searches for the driver at Windows Update database.

The thing starts to get funny when Windows “asks” me to insert the disk that coming with the Debug Cable. Only two words had been on my mind at that moment, “What disk?”. Without some other options, I clicked on “I do not have that damn disk! Are you crazy? Show me anything, for the God sake.” Some of you might even have this sequence of windows decorated by just not finding drivers for all devices you have had.

Finally, when the last window has been shown to me, a light came to me, just like one of those brilliant ideas that I have had every leap year. That’s when I thought to myself “Now I know! I am going to visit the manufacturer’s website and look for a support session.”

And at the manufacturer’s web site…

And at the Microsoft’s web site…

In the end, I sent this e-mail to get any sign that I had not thrown my money into the trash can. Less than an hour after, I got the answer with a list of requirements and steps to be followed; but the most important thing had just been revealed to me at the end.

I knew I could count on Microsoft intelligence and agility to resolve this but, for those who are reading this post, my tip is: When Windows prompts for new device driver, point out the “C:\Program Files\Debugging Tools for Windows\usb” folder.

So, I’m glag to show you this next window into the figure below:

Configuring WinDbg

Now it’s children’s play. Select the “Kernel Debug …” item from the “File” menu. Click on the “USB 2.0” tab and type the same TARGETNAME you have chosen at TARGET configuration with BCDEdit.exe. In my case, the name is WINDBG as it is displayed at the figure.

Clicking OK, the WinDbg command window should display the text highlighted in red below and wait for the connect completion. The output below was obtained with a CTRL+Break after having the debugger connected.






Microsoft (R) Windows Debugger Version 6.8.0004.0 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
 
Using USB2 for debugging
Waiting to reconnect...
USB2: Write opened
Connected to Windows Vista 6000 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Vista Kernel Version 6000 MP (1 procs) Free x86 compatible
Built by: 6000.16584.x86fre.vista_gdr.071023-1545
Kernel base = 0x82000000 PsLoadedModuleList = 0x82111e10
System Uptime: not available
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run kd.exe) or,                                        *
*       CTRL+BREAK (if you run WinDBG),                                       *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
nt!RtlpBreakWithStatusInstruction:
820818d0 cc              int     3
0:kd>

Now the easy part has been finished. The next steps are referring to investigate the issue, however this will be described in a future post.

I hope having  helped.
CYA 🙂

Did you say I don’t like Blue Screens?

Monday, September 3rd, 2007

I’m not saying I love them. Blue screen is a sign that something went wrong but, it is better you see it than a customer calling you saying that he saw one. So we need to do our best to make it make it appear while your driver is under test. I’ve seen programmers escaping the blue screen just trying to hide themselves behind an exception handler. I do not need to say this is not that useful. This way, you just defer to find mistakes that you or someone else will find eventually. Computers that are using your drivers may have a sudden reboot here or there. But even blue screens can happen, it’s all a matter of luck; however, your customers can realize that using your drivers makes them too unlucky. In this post, I’ll give you some tips about how we can see more blue screens in our test drivers.

Going through with F8

Do not take this as “Test Yourself,” but take it as testing all the returning codes that you can. Remember that if something gets wrong, it is not just a MessageBox that will appear, but usually, everything ends up in “blue”, sooner or later. Let’s hope for “sooner”. I often use to say that every code deserves, at least, an F8 walkthrough, which is my Step Into key. Yeah, I know it’s different of the usual one, but when I started using Visual C/C++ 1.52, its keyboard layout was it. But back to the point, once I have made a function that should simply read a file, I went through the code with F8 and received a return code that was not STATUS_SUCCESS, but passed the NT_SUCCESS() macro. The haste induced me to ignore it. It was not an error, it was just a warning. Weeks later, the test team told me that, for any reason, the driver was returning garbage when reading the file. A bit of debugging showed me that same returning code. Only after having loked at its definition in the file ntstatus.h I could understand everything.

//
// MessageId: STATUS_PENDING
//
// MessageText:
//
//  The operation that was requested is pending completion.
//
#define STATUS_PENDING                   ((NTSTATUS)0x00000103L)    // winnt

During debug sessions, the system has enough time to do the I/O that was deferred previously; but, freely running the code, the story can change.

A code that has passed on its F8 test can run in various existing and imaginable environments. Therefore, this first phase is just to remove the grosser errors. After that, it is the test team should beat the victim. There are people who really have a gift to test software. I had worked at a company where the person who used to test the products should have had a personal problem with the software that we produced (or with us, go figure!). I could test the software for days, but when I delivered it for testing, it did not take half an hour for him to call me back saying that phrase that has become his trademark: “Too bad !!!”. There was no explanation. We used to say that his PC had been formatted over an Indian burial ground. No wonder that developer’s testing is so frowned upon.

Using ASSERT

I imagine most of you have already heard from ASSERT macro. This macro is intended to ensure that a certain condition is true. If the condition is false, a blue screen just pops up. Wow, what an incredible macro that helped us very much! Yeah, I know, it is not quite that. Actually, this is the behavior we would have if the debugger was not attached to the system. Then, that macro would be perfect only in Checked. Do I must remind you that  Checked means Debug and Free means Release? Well, already done. If we take a look at its definition, we will see that the obvious thing has already been thought.

#if DBG
 
#define ASSERT( exp ) \
    ((!(exp)) ? \
        (RtlAssert( #exp, __FILE__, __LINE__, NULL ),FALSE) : \
        TRUE)
...
 
#else
 
#define ASSERT( exp )         ((void) 0)
 
...
 
#endif // DBG

But what happens if we have the debugger attached to the system? Good question, this is a really interesting question; I’m glad you have asked it. Has anyone ever told you that you have some knack for programming? Anyway, I have changed one of our examples to enforce this condition.

extern "C"
NTSTATUS DriverEntry(IN PDRIVER_OBJECT pDriverObj,
                     IN PUNICODE_STRING pusRegistryPath)
{
    //-f--> I think I'm sure that this count is right.
    ASSERT(1 + 1 == 3);

If the condition fails and we have a kernel debugger attached to the system, it will display the condition that had failed at the output window and it will request us an answer among the  four alternatives, as it is shown below.

As you can see, the ASSERT is practical, easy and not fattening. It just requires a bit of brain, as well. I’m sying this because we’ve all had bad days, and after 11:00 pm, no programmers should answer for any produced code. Once, it took me some time to figure out why the code below was not working properly. Although everything was working perfectly fine when compiled on Checked, it seemed to me that a certain function simply was not being called in Free. If you take one more looking at the definition of this macro, you will see that the condition disappears when compiled in Free, and in this case, the call too.

    //-f--> Programming is just not enough, thinking is required.
    //      << Don't copy this >>
    ASSERT(DoSomeThing() == STATUS_SUCCESS);

A system in Checked

Wouldn’t it be nice to have an entire operating system full of ASSERTs and tests to detect the slightest sign of trouble and on finding one, the system would present us with a nice blue screen? Well, you’ve probably heard of the Checked Build versions. They are exactly what I have just written. An entire operating system built on Checked. This means that all of ASSERTs that were at the sources have been included in the final binaries and they are checking the system for you. It may seem silly to have to install your driver on one of these systems, but believe me, it is worth. I’ve had drivers that worked very well for months until my manager suggested that they should be tested on Checked versions. From the top of my arrogance, I thought to myself: “I see no reason for that”. Well, we are always learning as we are living. The test machine has not even has started with my drivers. Several blue screens were shown, one after another. Checked Build versions are still able to check for deadlocks. Could you guess what would happen if a spinlock was held for more than a certain time? I bet you do.

Since Checked Build versions are foolproof, I can use them as my default system? Actually you can, but everything is much slower, thousands of checks are being made all the time and the code has no optimization applied. In a fresh installation without installing any additional software, you can find ASSERTs at Internet Explorer or other programs. That’s right, not just the kernel is Checked, but the entire system is as well. You always have to let a kernel debugger attached to your system, because the slightest hint of smoke would be more than an enough reason for one more blue screen.

If my driver has passed by the Checked Build, then is the driver perfect? Sorry to disappoint you. Remember that the performance impact caused by so many tests can hide problems like race conditions. The ideal is to test your drivers in both versions. One setting allows us having only the system image and HAL in Checked, while the rest of the system stays in Free. This may provide you with additional tests in the kernel while the rest of the system runs the lighter version. This link explains how.

OK, is my driver beautiful now?

Actually, this still is only the minimum you should do. Another excellent tool for generating blue screens is the Driver Verifier. This application works in conjunction with the system to track actions from a list of drivers composed by you. From Windows 2000, DriverVerifier comes already installed. Try it now! It doesn’t require any skill or even practice. Type “Verifier” at the Windows “Run…” window and a simple Wizard starts. As I intend to finish this post still alive, I won’t describe how to use this tool step by step, but there are several links on the product page that can help you with this.

OK, now I know you really love Blue Screens!

To get an idea of how importnat a search for blue screens is for a software company, Microsoft promotes IFS Plug Fest in order to test the interoperability of different products that implement File Systems or filters for them. The event is for free, but each company must pay for travel expenses and lodging of its representatives. In these meetings, the professionals around the world gather together to do the tests with each other and they can get in touch and ask possible questions concerning this complex model of development. There are also sponsored seminars that discuss common problems faced by this community. I have not been at one of these, but maybe one day I will be able to.

Stay calm, there are even more

Beyond the mentioned tools, there are some others that I could not forget mentioning; however, I won’t take long on explaining how each of them works.

  • Prefast – A tool that analyses the source code to be compiled by the DDK, looking for usual programming mistakes for Kernel drivers.
  • Static Driver Verifier – That tool makes a binary analysis on the driver file already built. The test takes longer but, it detects more defects than using Prefast.
  • Hardware Compatibility Test – A set of software to apply tests to get drivers certified by Microsoft. It was discontinued a while ago.
  • Driver Test Manager – That’s the new test software used to get your drivers certified.

After so many blue screens, only drinking some coffee can help us!.
CYA!

Debug Option or Security Fault?

Tuesday, August 21st, 2007

Some time ago I have written a post that talks about replacing faulty drivers using WinDbg. This is nn excellent technique to substitute drivers which, because of some kind of problem, prevents the machine from starting. However, to be able to enjoy all of this facility, there must be a connection between the system and the Kernel Debugger. This brings us to my last post that ostensibly talks about how to enable this feature just being in front of the machine, and at worst, having a serial port. All of this is too easy, after all; you never know where an unknown problem can happen and much less, in which machine ir is going to happen. But all of this convenience has a price. The side effect is that anyone can debug the kernel of a machine without ever having any valid account on it. In fact, a laptop, a cable and a serial port are beyond than enough weapons to dismantle the system security, without a password, wireless network and even without brute-force, only doing Kernel Debugging.

Installing a Driver Through the Serial Port

Drivers are a kind of software that needs to be reliable and very well written. After all, they run at the lowest level of the system. With the drivers, it is actually very easy to cheat the system, since most of the system runs on the service offered by the kernel. The posts above mentioned have motivated me to write a small proof of concept that aims to inject a driver which installs itself on the system. This driver, in turn, extracts and installs a program into the Windows which will run every time one logs on the system. To do that, we only need a machine that can be debugged. My last post shows that this condition is not so difficult to be found. Want to try? What’s the percentage of computers around you that have serial ports?

Step by Step

This is one of those posts that come with a code available for downloading. The whole procedure shown here can be reproduced and modified by the readers of this blog. This is not a blog devoted to attack Windows; and my goal here is purely didactic. I won’t explain every line of the sample code; otherwise, this post would be very extensive. However, I am willing to answer any questions about this. The figure below outlines the steps to be taken to achieve our goal.

  1. The first step occurs when compiling the sample driver. The solution has three available projects. The first project is the application to be injected. In this case, a Win32 application that just displays a message and is ended up. We will refer to this application as MyFile.exe. The C/C++ Run Time was removed, in order the smallest possible application to be obtained. This will be worthy a lot when you have to transfer everything via a serial cable at 19200. The second project is just one tool. Using the archive MyFile.exe, which must already be compiled, this tool generates a header file that creates a variable of the type array of characters contained into its body, where the file passed as a command line. This is not the most advanced method of making a Self-Extractor, but it is enough for our purpose here. Let’s call this tool HeaderFile.exe. When the driver is compiled, this header file is included making the variable containing the application be injected. We will call our driver MapInject.sys. The driver in this example is compiled using ddkbuild.cmd with Visual Studio 2005. More details with this regard in this post.

    /****
    ***     WinMain
    **
    **      This application does almost nothing and doesn't use C/C++ Run Time
    **      so that we can avoid a bunch of codes.
    */
     
    int WINAPI WinMain(HINSTANCE   hInstance,
                       HINSTANCE   hPrevInstance,
                       LPSTR       lpCmdLine,
                       int         nShowCmd)
    {
        //-f--> That is it...
        MessageBox(NULL,
                   L"A serial port was found",
                   L"DriverEntry.com.br",
                   MB_OK | MB_ICONEXCLAMATION);
     
        return 0;
    }
  2. Using driver mapping techniques demonstrated in this post, we can make our driver be loaded in replacement of any driver in the pre-existing system. Obviously, our driver will not perform all the steps that the original driver should perform, thus, we must not choose a driver which does essential tasks for the system loading, such as disk drives, video or something like that. Particularly I have chosen using the Partport.sys parallel port driver, which is a driver that exists in any basic Windows installation and don’t not make us so missed while playing here. This driver is not also a Boot driver, because boot drivers need a special Loader on Windows XP to be mapped by the Debugger.

    map 
    \Windows\System32\drivers\parport.sys 
    Z:\MapInject\Driver\objfre_wnet_x86\i386\MapInject.sys
  3. After the driver has been loaded and mapped by the system, all that we need to do on the system is just some willingness. When our driver is first loaded, it is loaded in replacement of Parport.sys. In this condition, the driver must install itself on the system. Some things have never changed. To install a driver on Windows NT platform just create a few key values in the system registry, as described in this post. This will still work on Windows Vista. Our driver does this at this time.

  4. After making the registry modifications, now we have to rename our driver because it was mapped and, for that reason, it gets the replaced driver name. Thus, a function changes the name of the driver from Parport.sys to MapInject.sys. This driver could remain installed using the original name but, our driver would be replaced by the system restore when the driver mapping didn’t take place.

  5. As you know, our driver has a variable that contains the whole body of our application. Thus, it is very simple to extract the application and save it into the disk. The System32 directory comes in handy as a destination for our application. Since we’re running on system account at this time, we have no problem on doing it.

  6. Now that our application has been extracted, we have to make it run automagically. Some of you may know that there is no documented or reliable enough way to create a process from the system kernel however, there are several places on registry where we could write and ask an existing process in Windows to start our application. One of the most classic registry keys that allow this it is the Run of Windows Explorer; but, as shown in this application, there are many other keys to be chosen. When a user logs on the system, the Shell enumerates this key and executes each line in the user’s account that is logged on. This will make our program be executed.

Making it Work

Again, a virtual machine fits like a glove for this type of thing. If you do not know how to do Kernel Debugging using a virtual machine, this post can help you. Everything is very simple after the driver being already encoded. The only thing needed to complete the test set is a mapping that replaces the Parport.sys driver for a MapInject.sys file, as it is shown in the map file above. After that, just wait until the system loads up and everything is done. Any user who logs on to the system will see the message of our program. A point to note is that, even if the user is smart enough to find the MyFile.exe file, our change at the registry and delete them, they will be recreated when the MapInject driver starts on each system reboot. To stop this process, just remove the MapInject driver from the system.


Password? What For?

Windows protects its most important resource for most users, allowing only administrators to have access to the machine settings that can compromise the system stability. This phrase is kind of “cute” and makes several system administrators to feel comfortable. But life is a box of surprises, and in a nice sunny morning, a post is published on a RootKits website, providing tips on how to log into a system in Debug Mode without any password. Recently, another post has come out about it and I will not repeat everything here but, I can say that this same technique also works with Windows Vista. In fact, if you can log into the system without using any password, you will not need to map a driver to make it run on this machine. Just log in as an administrator, install it and go ahead but, I admit that it was at least funny to write this proof of concept, and moreover, the sources may still be an example for some tasks you want to do in Kernel Mode.

A possible solution? Maybe letting your serial port disabled in BIOS, which should be password protected, is something to be done. But there is not much you can do when someone has physical access to the machine.

Enough, I already wrote too much.
Hugs!

MapInject.zip

Emergency Debug

Wednesday, August 15th, 2007

Suddenly, a blue screen. That’s right, that driver which you had not seen its source files for so long, which was your model for stable driver, the one that was installed in the machine of the CEO where you have worked, the same one that suddenly decided to take revenge and simply got a blue screen at the system startup. That means the machine turns on and resets and resets and resets in an infinite loop that runs in parallel with your resignation letter printing which, by the way, has being printed on another PC. To debug this machine and see what is happening, and then save your project, your job, your dignity, your marriage, and why not, your life: you must configure the system to enter in Debug Mode so that our inseparable friend WinDbg can take action. But how about setting it to debug a machine that is not even starting? Today I will talk about the support that Windows offers to avoid such an embarrassing situation to connect a Debugger in a computer that has never its settings for Kernel Debugging enabled, either through BOOT.INI or BdcEdit.

This post assumes that you have already known how to do Kernel Debugging and uses some related terms with this regard. Since Windows 2000, if you press the F8 key while the machine is running the NT Loader, more precisely, 2 seconds before the system starts to loading up, the system will freeze up and displays the menu as it is shown below.

Select the “Debugging Mode” item. It will take us to the list of installed operating systems. Select the system and the rest is up to the WinDbg. Under these conditions, setting the serial connection would be applied a baud rate of 19200 on the higher serial port existing on your machine. That is, if your PC had two serial ports, then the connection would be made on COM2 at 19200. At least that’s what tells us the reference.

But in real life…

If you do some tests, will also conclude that we cannot believe everything we read. Both Windows 2000 and Windows XP always use COM2 port with 19200.

My computer is modern and it only has one serial port. Can I kill myself? That is it, even in theses cases, the system tries to use COM2, even if it does not exist. But don’t worry, life is beautiful and there is still hope. Some BIOS allow you to configure the base address and interrupt about the only existing serial port. This will help us greatly; after all, the NT Loader is not a Win32 application that requests a handle to the COM1 or COM2 device through the Object Manager. It simply follows the addresses adopted as standards.

  • COM1: Address 3F8h, IRQ 4
  • COM2: Address 2F8h, IRQ 3

In this way you can configure the only port (COM1) to answer for the addresses which are normally used for COM2. The BIOS usually has a list of features commonly used by serial ports. Thus, any autistic chimp could make this change.

But my computer is even more modern and has no serial port at all. Can I kill myself now? Hope it is the last to die. You can still install a PCI card Expander (PCMCIA in case of notebooks) that provides us with the needed serial ports. It is important to note that this board should create additional ports COM1 and COM2 at the above standard addresses.

You don’t understand, I am an unlucky bastard who bought a MacBook 13″ which, besides not having serial ports, also offers no PCMCIA or ExpressCard slot. Can I kill myself now? As I have mentioned in another post, serial ports created from USB ports are not an option for using in a TARGET machine during the Kernel Debugging. This connection is made by the NT Loader and serial ports that it knows and they are the only those ones the addresses mentioned above. OK, now you can kill yourself.

Anything that could get better?

Windows Vista ignores the documentation mercilessly. The default setting for Kernel Debugging when you press F8 during its Boot is for using  COM1 port with 115200. But if you used the Bcdedit application with the /DbgSettings to modify the connection settings for Kernel Debugging, then these same settings will be used  when F8 is pressed. That is, if you are an unlucky bastard that, like me, have configured the IEEE1397 port for debugging into your MacBook, then these are the settings used when you press F8.

Pode parecer brincadeira, mas é realmente frustrante ter que ir até à máquina do cliente onde seu driver está dando trabalho, armado dos seus vários aparatos tecnológicos, tais como notebook, porta FireWire, cabo serial e assim por diante, e não poder fazer nada simplesmente porque você não consegue conectar o depurador.

It may seem fun, but it’s really frustrating having to go to the customer machine where your driver is annoying, armed with all of your various technological devices such as laptop, FireWire port, serial cable and so on, and simply could not do anything because you cannot attach the debugger.

Once again, I hope I could help in anyway.
Cya!

Bug on my Boot driver! Now what?

Monday, July 16th, 2007

Writing drivers is a task that must be done with little care. After all, any unresolved situation between your code and the operating system will result in a beautiful blue screen. But everything in this life has a cure and fortunately God has created the debugger to deal with these situations. After fixing the problem, it is just to change the .sys file and that’s it. Just changing the file means that the machine is expected to be in a stable condition: then, replace the file on the system and reboot the machine. But life is a surprising box and you may not believe, but most drivers are automagically started. Well, in this case, the only think we can do is to pray that the error does not happen until the driver is replaced. Otherwise, we would have to use some resources, such as putting the hard drive of victim’s machine on another computer to replace the sick driver. When there is no one else’s computer, use the Recovery Console in XP to avoid the sick driver being loaded. With some luck, this driver is not a filter attached to something important like disk, video or File System. You know, if a filter is not able to be loaded, then the main driver, either. In summary, there is a lot of juggling you can invent to replace a buggy driver that is automatically loaded. Is there anything that does not depend much on luck or even on the creativity? Today, I will talk a bit about the replacing driver system with WinDbg.

Mapping driver files

You can map the driver files so the WinDbg can replace it at the time it’d be loaded. Is not that beautiful? For this, you must use the .kdfiles command. With this command, you make a link between the driver that runs on the Target machine and the one that has been fixed on the Host side. One way to do this is to initially create a file that lists these two drivers. This file should be a simple text file where the syntax is shown below. This file can have any name and extension.

map 
\??\C:\Windows\System32\drivers\MyDriver.sys 
C:\My_Driver_Project_Folder\MyDriver.sys

The word map marks the beginning of a mapping; this does not change. Next line is the driver file path to be replaced from. This line should have the same format used in the ImagePath value under the driver key that is on registry. The last line is another driver path. This path could point at a driver in the Host machine itself or on the network.

This mapping works only in Windows XP or higher, in the Target machine, obviously. If you’re not familiar with terms like Host and Target, check this post out.

Once created the file, you will use WinDbg to launch the following command:

kd>.kdfiles C:\Any_Folder\MyFileMap.txt

Thereafter, whenever the driver is loaded by the system, the kernel checks whether this file is mapped in the debugger, and if so, WinDbg sends the new driver through the debugger connection, such as serial, USB or firewire. For large files, I recommend using firewire. You must understand that the file on Target system’s disk is replaced by the new version. This means that in the forthcoming launches of our driver, the new version will still be loaded, even if the debugger is not connected to the system.

Does this really work?

To make things a little clearer, let’s see a practical example. To do this, we need a very simple driver source, or even useless, which can be found here. Let’s change its DriverEntry function, so that it will be as shown below:

NTSTATUS DriverEntry(IN PDRIVER_OBJECT  pDriverObject,
                     IN PUNICODE_STRING pusRegistryPath)
{
    //-f--> Let's use __DATE__ and __TIME__ to change this message
    //      each time this driver is built.
    DbgPrint("This driver was built on "__DATE__" "__TIME__"\n");
 
...

So, we build an initial version of the driver and we install it on the Target machine in the way you best think, but we’ll assume that after the driver has been installed, the registry will be as shown below.

If we start the driver, we will have something like the following string in the debugger output:

This driver was built on Jul 16 2007 00:04:03

Now, we’ll create the mapping file. Here I will save it as Z:\Sources\DriverEntry\Useless\map.txt. Following the same format the driver was registered in the registry, our mapping file should have the following content:

map
\??\C:\Windows\System32\drivers\Useless.sys
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

Notice that the folder where I saved my file map.txt is the same where the driver sources are. This is just a matter of organization. The mapping file could be in any folder. Then, use the command .kdfiles as shown below. Notice that we list the existing maps simply using the same command without any parameters.

kd> .kdfiles Z:\Sources\DriverEntry\Useless\map.txt
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
 
kd> .kdfiles
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
\??\C:\Windows\System32\drivers\Useless.sys ->
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

Afterwards, we’ll Rebuild the driver (that should change that time stamp) and we will restart it. If everything is right there by your side, you should have an output with a time stamp different from what we had before.

kd> g
But now? I did nothing so far...
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\??\C:\Windows\system32\drivers\Useless.sys)
  File size 2K.
MmLoadSystemImage: Pulled \??\C:\Windows\system32\drivers\Useless.sys from kd
This driver was built on Jul 16 2007 00:17:04

Well, if this works for a driver with manual start value, then the automatic one also should work. To see this, change your driver start value to System or Automatic, do a Rebuild again in the driver, and finally, we will restart the Target machine. When the driver is loaded, we have the automatic replacement of its image. In WinDbg, we will have the following:

Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp_sp2_gdr.070227-2254
Kernel base = 0x804d7000 PsLoadedModuleList = 0x805533a0
System Uptime: not available
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\??\C:\Windows\system32\drivers\Useless.sys)
  File size 2K.
MmLoadSystemImage: Pulled \??\C:\Windows\system32\drivers\Useless.sys from kd
This driver was built on Jul 16 2007 00:25:38

Yeah, it really works! But what if was my driver a Boot driver? This means that the driver image will be loaded even before the connection is established with WinDbg. Have you heard that what has no remedy, it is remedied? Fortunately, this is not applied here yet. There is a way to make this work, even with Boot drivers.

Mapping Boot drivers

To establish a connection with WinDbg, we need to replace the system loader by a special version. This version makes this connection with the Kernel Debugger, even before the Boot.ini file is read. For this reason, the connection parameters are fixed with COM1 and baud rate of 115200. This loader version is located in the directory C:\winddk\3790\debug of the DDK with its ntldr_dbg name. This file should replace the original loader system version that is at the boot drive root with the ntldr name. The debug version must stay with the same name as the original loader.

Before rebooting, we need to change the driver’s start value to Boot and remove the ImagePath value from the registry. As you may know, boot drivers do not have that luxury of using file paths that have drive letters. At the end of the changes, we should have the registry as shown below.

As I said earlier, the driver file path format should be the same as it is on registry, but knowing that now there’s no file path in the registry, so we adopt the same forms adopted by the system. Oh! okay! The same format. And what would it be? To see this format, simply restart the system with the debug loader, which should give us the following screen at boot-up.

This is the time to connect to Windbg using pre-determined parameters connection. This should result in the following output in the debugger.

Microsoft (R) Windows Debugger  Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.
 
Opened \\.\pipe\com_1
Waiting to reconnect...
BD: Boot Debugger Initialized
BD: osloader.exe base address 00400000
Connected to Windows Boot Debugger 3790 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Boot Debugger Kernel Version 3790 UP Checked x86 compatible
Primary image base = 0x00000000 Loaded module list = 0x00000000
System Uptime: not available
The call to LoadLibrary(bootext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

Now you see the Boot.ini selection menu. Select your option and continue loading the system. A list of boot drivers should be presented, which are the ones loaded before the first breakpoint that the debugger can stop.

BD: \WINDOWS\system32\ntoskrnl.exe base address 804EA000
BD: \WINDOWS\system32\hal.dll base address 806FF000
BD: \WINDOWS\system32\KDCOM.DLL base address 80720000
BD: \WINDOWS\system32\BOOTVID.dll base address 80010000
BD: \WINDOWS\system32\DRIVERS\ACPI.sys base address 80124000
BD: \WINDOWS\system32\DRIVERS\WMILIB.SYS base address 80001000
BD: \WINDOWS\system32\DRIVERS\pci.sys base address 80062000
BD: \WINDOWS\system32\DRIVERS\isapnp.sys base address 80003000
BD: \WINDOWS\system32\DRIVERS\compbatt.sys base address 8000C000
BD: \WINDOWS\system32\DRIVERS\BATTC.SYS base address 80013000
BD: \WINDOWS\system32\DRIVERS\intelide.sys base address 80017000
BD: \WINDOWS\system32\DRIVERS\PCIIDEX.SYS base address 80019000
BD: \WINDOWS\System32\Drivers\MountMgr.sys base address 80152000
BD: \WINDOWS\system32\DRIVERS\ftdisk.sys base address 8015D000
BD: \WINDOWS\System32\drivers\dmload.sys base address 80073000
BD: \WINDOWS\System32\drivers\dmio.sys base address 8017C000
BD: \WINDOWS\System32\Drivers\PartMgr.sys base address 801A2000
BD: \WINDOWS\System32\Drivers\VolSnap.sys base address 801A7000
BD: \WINDOWS\system32\DRIVERS\atapi.sys base address 801B4000
BD: \WINDOWS\system32\DRIVERS\vmscsi.sys base address 801CC000
BD: \WINDOWS\system32\DRIVERS\SCSIPORT.SYS base address 801CF000
BD: \WINDOWS\system32\DRIVERS\disk.sys base address 801E7000
BD: \WINDOWS\system32\DRIVERS\CLASSPNP.SYS base address 801F0000
BD: \WINDOWS\system32\DRIVERS\fltMgr.sys base address 801FD000
BD: \WINDOWS\system32\DRIVERS\sr.sys base address 8021D000
BD: \WINDOWS\System32\Drivers\KSecDD.sys base address 8022F000
BD: \WINDOWS\System32\Drivers\Ntfs.sys base address 80246000
BD: \WINDOWS\System32\Drivers\NDIS.sys base address 802D3000
BD: \WINDOWS\System32\Drivers\Useless.sys base address 8000F000
BD: \WINDOWS\System32\Drivers\Mup.sys base address 80300000
BD: \WINDOWS\system32\DRIVERS\agp440.sys base address 8031B000
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...

Here the connection to the loader is closed. A new connection would be established if you had selected the Debug entry from Boot.ini, but what we have to note here, is the file path format used to load drivers during system boot. Notice that our test driver is among the drivers from the above list. Let us adopt this same file path format on our mapping file.

map
\WINDOWS\System32\Drivers\Useless.sys
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

After modifying the mapping file and save its contents to disk, we update the WinDbg, so it takes this change. Then, we will restart the system.

kd> .kdfiles Z:\Sources\DriverEntry\Useless\map.txt
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
 
kd> .reboot
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...
BD: Boot Debugger Initialized
Connected to Windows Boot Debugger 3790 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.  (Initial Breakpoint requested)
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Module List address is NULL - debugger not initialized properly.
WARNING: .reload failed, module list may be incomplete
KdDebuggerData.KernBase < SystemRangeStart
Windows Boot Debugger Kernel Version 3790 UP Checked x86 compatible
Primary image base = 0x00000000 Loaded module list = 0x00000000
System Uptime: not available
The call to LoadLibrary(bootext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

The same message sequence happens, but this time, the mapping is done as it should and we have the following output when the boot drivers are loaded.

BD: osloader.exe base address 00400000
BD: \WINDOWS\system32\ntoskrnl.exe base address 804EA000
BD: \WINDOWS\system32\hal.dll base address 806FF000
BD: \WINDOWS\system32\KDCOM.DLL base address 80720000
BD: \WINDOWS\system32\BOOTVID.dll base address 80010000
BD: \WINDOWS\system32\DRIVERS\ACPI.sys base address 80124000
BD: \WINDOWS\system32\DRIVERS\WMILIB.SYS base address 80001000
BD: \WINDOWS\system32\DRIVERS\pci.sys base address 80062000
BD: \WINDOWS\system32\DRIVERS\isapnp.sys base address 80003000
BD: \WINDOWS\system32\DRIVERS\compbatt.sys base address 8000C000
BD: \WINDOWS\system32\DRIVERS\BATTC.SYS base address 80013000
BD: \WINDOWS\system32\DRIVERS\intelide.sys base address 80017000
BD: \WINDOWS\system32\DRIVERS\PCIIDEX.SYS base address 80019000
BD: \WINDOWS\System32\Drivers\MountMgr.sys base address 80152000
BD: \WINDOWS\system32\DRIVERS\ftdisk.sys base address 8015D000
BD: \WINDOWS\System32\drivers\dmload.sys base address 80073000
BD: \WINDOWS\System32\drivers\dmio.sys base address 8017C000
BD: \WINDOWS\System32\Drivers\PartMgr.sys base address 801A2000
BD: \WINDOWS\System32\Drivers\VolSnap.sys base address 801A7000
BD: \WINDOWS\system32\DRIVERS\atapi.sys base address 801B4000
BD: \WINDOWS\system32\DRIVERS\vmscsi.sys base address 801CC000
BD: \WINDOWS\system32\DRIVERS\SCSIPORT.SYS base address 801CF000
BD: \WINDOWS\system32\DRIVERS\disk.sys base address 801E7000
BD: \WINDOWS\system32\DRIVERS\CLASSPNP.SYS base address 801F0000
BD: \WINDOWS\system32\DRIVERS\fltMgr.sys base address 801FD000
BD: \WINDOWS\system32\DRIVERS\sr.sys base address 8021D000
BD: \WINDOWS\System32\Drivers\KSecDD.sys base address 8022F000
BD: \WINDOWS\System32\Drivers\Ntfs.sys base address 80246000
BD: \WINDOWS\System32\Drivers\NDIS.sys base address 802D3000
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\WINDOWS\System32\Drivers\Useless.sys)
  File size 2K.BD: Loaded remote file \WINDOWS\System32\Drivers\Useless.sys
 
BlLoadImageEx: Pulled \WINDOWS\System32\Drivers\Useless.sys from Kernel Debugger
BD: \WINDOWS\System32\Drivers\Useless.sys base address 8000F000
BD: \WINDOWS\System32\Drivers\Mup.sys base address 80300000
BD: \WINDOWS\system32\DRIVERS\agp440.sys base address 8031B000
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...

And later, our proof that the driver file has been successfully replaced.

Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.  (Initial Breakpoint requested)
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp_sp2_gdr.070227-2254
Kernel base = 0x804ea000 PsLoadedModuleList = 0x8056d620
System Uptime: not available
This driver was built on Jul 16 2007 01:18:53

I assure you that this could still save your life if you had a File System filter with a bug in its DriverEntry on a customer's machine. The expression of panic when the customer sees his machine restarting in an endless loop is interesting, but keeping your job is a little more interesting.

See you!

Step into Kernel (VmWare+WinDbg)

Thursday, May 31st, 2007

It is enough with this idle talk and let’s go to what really matters. Speaking about Kernel Debugging, I have written a post describing the steps needed to Kernel Debugging with WinDbg, using two machines and a serial cable. But having two machines dedicated to this practice is a luxury that not everyone has. Thus, in another post, I have commented about installing and using SoftIce to debug drivers on a “single machine dedicated to debug”. I tell you “dedicated machine to debug” because using your development machine to debug drivers may not be one of your best ideas. The most courageous and confident people in their own code still risk themselves in this practice. Well, I cannot say much, I myself have already done that in lean times. Anyway, we have not fled the need for two machines to have a minimal environment for development and test drivers. An alternative to these scenarios is the use of a single machine, but the one that can have enough memory and CPU to run a virtual machine, in order to debug the kernel. In this post, I will take the steps to use a virtual machine as a guinea pig to test and debug drivers.

Configuring a virtual machine

The natural communication way between real machines using WinDbg to do Kernel Debugging is a serial port. To follow the same steps, we need to make your virtual machine to gain a serial port to make this communication possible. With your virtual machine turned off, click on the Edit vitual machine settings option.

A window will appear with the list of devices already installed on your machine in Hardware tab. Click Add… in order to add a new device and select Serial Port in the device list that is displayed on the screen below.

After clicking on Next, select the Output named pipe. This will cause the serial port on the virtual machine to communicate with the real machine, via a named pipe. Clicking on OK, the specific settings of the named pipe will be presented. The pipe name will be used later in one of the WinDbg settings so, if you want to use a name other than the one suggested here, try to remember this same name later. Then, change the value of the second combo to indicate that the other communication ending will be an application, in this case WinDbg. After that, just click on Finish.

At the end of these settings, select the Yield CPU on poll option as shown below.

Configuring the TARGET machine

Made all that “hocus-pocus”, now we have to set the TARGET machine system in the way it will able to make the Kernel Debugging. Remember that from Windows that runs inside the virtual machine point of view, the named pipe does not exist, it is just a serial port. If you still do not know what TARGET machine is a or how it can be set, then take a look in this post and follow the described steps in the part where a TARGET machine is configured .

Configuring the HOST machine

Assuming that your TARGET machine has been configured, now we have to configure WinDbg in a way it can be connected to a machine using named pipe, instead of a serial port. For that, I usually create a batch file that contains the following command line:

start C:\Progra~1\Debugg~1\windbg.exe -b -k com:pipe,port=\\.\pipe\com_1,resets=0

Creating a batch file is not a required step, you might want to retype everything in the Run… window, every time the debugging starts; but it’s all up to you. Notice that the pipe name appears here. I hope you still remember which name you have chosen.

Connecting…

Now we have everything already configured, it is just plug and debug. In this story, the virtual machine is the one that creates the named pipe that is opened by WinDbg. Thus, if you start WinDbg with the shown parameters before starting the virtual machine, you will see the window below stating the pipe was not found.

So, the sequence is, firstly, to connect the virtual machine, select the debug option in the Boot as shown below, and only after that, you should start the WinDbg with the parameters described above.

When WinDbg is finally connected to the virtual machine, via the named pipe, we have the following messages in our Command Window within WinDbg.

Thereafter, you know… You pick up the bugs.

Once more I hope I have helped you.
See you next time. 🙂

Step into Kernel (SoftIce)

Saturday, December 23rd, 2006

Finally back from vacation. It has been nearly six years that I did not take vacations. Traveling, visiting new places and just relaxing were my main tasks during these last 20 days. Back to work, I’ve just arrived at the company and I was amazed by it. I could not believe they sent that old one away. We exchanged glances and obviously I got butterflies in my stomach. I got close enough to be able to smell it. I looked up and down. Simply perfect and it was an injustice to leave it alone. It didn’t take long and it was under my control. I had to prove everything it had to offer. After all, there were so many choices: Coffee, cappuccino, chocolate, tea with or without sugar. But it was all an illusion. “Oh my… that’s was a horrible coffee.” My friends, appearances can be deceiving. How can so beautiful machine make so bad coffee? I confess I miss the old machine. I didn’t mean that coffee was not so good, because the coffee was really horrible! Now I’m looking for alternative ways to keep my addiction. Perhaps there are caffeine patches to sell out there.

Don’t worry, this will no longer be an “Off Topic” post about changing coffee machines and the impact caused by them. In this post, which I had started last year and have just finished it now, I’ll take the steps needed to use SoftICE system debugger.

I still was new in SCUA and I was developing a GINA for our security product. For those who do not know, Gina is one small screen that appears during the Logon in NT platform. GINA is a DLL that is loaded by winlogon.exe process, which has the function of making the user’s interface to perform the login in the computer. The Msgina.dll GINA is the original one, but we can build our own GINA to provide alternative ways of authentication. A practical example of this is the product True Finger sold by SCUA. This product changes the system’s GINA, so that, with a biometric sensor help, it makes the system ask for the user’s fingerprint to logon the system. Debug GINA requires a few atypical steps, since to test it is necessary to do Logon, Logoff and things like that. You can also debug it using the Visual Studio Remote Debug, but at that time, debugging GINA was the need I had to learn how to use SoftICE.

There were two development teams, one for Kernel drivers and the other for applications. I still worked on application staff when in a meeting area, the team introduced me to SoftICE as an alternative to solve the problems about Kernel Debug I was facing. I was using techniques such as log files and MessageBox to debug GINA. You can imagine the MessageBox with the typical messages as “I called the X routine”, “Returning from Y routine.” After I learned how to use SoftICE, it solved many things for me. Specially since other Ginas had to be developed and as, the time passed by, I started to develop kernel drivers.

Most of the application team had a panicked about using SoftICE. A situation very similar to that described in my previous post. They said: “Oh no, using SoftICE is too complicated. I will continue with the MessageBox. If I cannot solve this, then I will use a Kernel Debugger.” I think it was just about getting used to it, SoftICE is not a rocket science and nor difficult to use. I found it amusing when one part of the team was caught trying to solve a particular problem, that kind of strange bug that nobody could find. I went around and asked: “Why don’t you just use SoftICE?”, “No, no, I guess it is not necessary.” they said. Later I also was called to help tracing the problem, and in most of the time I could, I’d picked up the problem that they’d chased for days in half an hour of Debug. I’m not saying I’m super Debug genius, or that they were technically incompetent. I’m saying that it is only a matter of using the right tool. You could dig a tunnel with a teaspoon, but with a shovel things would be much easier.

SoftICE is part of a development package called Driver Studio, which was sold by Compuware. For those who still do not know, Compuware has discontinued SoftICE and we cannot count on a version of this tool for Windows Vista. With SoftICE is possible to debug the kernel using only one machine, but the product still has the resources to do remote debug via serial ports or network adapters. Thus, they are also adopted the names Host for the machine that does the Debug and Target for the machine that will suffer the Debug (Similar to WinDbg). Here we will adopt the Host name for identifying the machine where we have the compiler and the installation type Host SoftICE. We will not do any remote debug on this post.

Very briefly, we can say that SoftICE works as it follows. At the Host machine, it is necessary to create a symbol file known as NMS. This file is copied along with the driver to be tested to the Target machine. In it, we load the symbols and start debugging the driver. We will describe these steps in more details next.

Installing the Host Machine

Installing SoftICE on the development machine is necessary for us to generate the NMS file. Don’t worry; you will not test your driver on your development machine. At least not while you’re sane. During the setup process, you will pass through the typical setup screens until you reach the screen shown above where you select installation type you want. Because we use only installation Host portion, select that option and skip to the next screen. Select only the option “SoftICE Host Components”, so that it appears on screen as below. The remainder of this process is next, next…

Installing the Target Machine

We will use the test machine in a way that it is not necessary to have two machines to do the debug, and so we will explore this feature that is usually used. To do this, select the “Full Instalation” in the first screen, then select only “SoftICE Host Components” items and “SoftICE Target Components” as it is shown below.

In this case, at the end of the register and file copy process, we will have to set some debugger properties. The first group of settings refers to initialization. We will select the Manual startup at this moment. All these settings can be changed after the product is installed. The setup screen should be as shown below.

The next group will be about general settings. In this item, we will change the field “Initialization” to stay with the configuration shown below. These are commands that are executed when the debugger is started. Each command is separated by a semicolon. These commands will make the following changes:

  • LINES 60; Change the total amount of lines to 60
  • WC 30; Change code window size to 30 lines
  • WL 10; Change local variable window size to 10 lines
  • X; And finally, the command that starts the system normal execution

The lack of “X” command will cause the system to become paralyzed at debugger startup time. This gives us the opportunity to put initial breakpoints and later we will be able to continue with the system execution by pressing F5, which is the hot-key for “X” command.

Now let’s set the video configuration. Right now you can ask me: “What do you mean by video configuration? Doesn’t Windows implement an abstraction layer called GDI?”

SoftICE is a system Kernel debugger. GDI is part of the system. How would you debug a video driver if the debugger itself utilizes this driver? The debugger should influence the debugged system as little as possible. It is for this reason that WinDbg need a serial cable and another whole machine to make the Kernel Debug possible. Similarly, SoftICE, once loaded, cannot use any system resource. This includes video drivers, disk, keyboard, mouse and so on. For the SoftICE being able to have a place in the video while debugging the system, it accesses the video memory directly and draws manually its entire interface. That’s why we have to configure the video.

This configuration can be chosen between two main options. The first is the “Universal Video Driver”, where the interface will appear in a window. This “window” is actually the result of writing in video memory that gives the impression of a window. This is not the type of window you’re used to seeing and drag with the mouse or even suffer the action of ALT + TAB. Remember, when SoftICE displays its user interface, the whole system is frozen. The other option is “Full screen”, where SoftICE switches the video to text mode to display its interface.

If you are installing SoftICE on a virtual machine, some steps are required:

  • Do not install tools such as video acceleration like VmTools.
  • Use the standard VGA driver in Windows and configure it so that, it is 640×480 with 16 colors. (Disgusting but necessary)
  • Select the “Full Screen” option in SoftICE.

If these steps are not followed, the machine will freeze when asked, but the debugger screen will not appear. Assuming that we are installing on a real machine, select the video option as shown below.

The remaining is next, next…

Generating the NMS File

On the Host machine, after compiling your driver (if you still don’t know how to do it, read this other post), you must now run the application “SoftICE Symbol Loader” that was already installed. This application does the symbol translation and generates the NMS file. Select Open… from the File menu and point out the driver you will debug. Once opened, the program must place the driver loaded name at the window title. I have numbered the three buttons you need for this operation.

  • Make sure that the second button is pressed. This will make any driver source code be appended to the resulting NMS.
  • Also make sure that the “3” button is released. This will prevent the program for asking for sources that it cannot find.
  • Click on “1” button that will do the symbol translation and the program will generate NMS file at the local directory.

Starting SoftICE

After installing your driver as usual in the Target machine, we will be ready to start the Debug of our driver. Remember that both our sample driver and SoftICE are configured to start manually. To start SoftICE, you can click the “Start SoftICE” icon that was installed in your Start menu or simply run the command “net start ntice” in the “Run…” window. To make sure that the debugger has loaded, press CTRL + D so that the system freezes and the screen is shown as the one below.

No, this is not a CRASH DUMP of your video card that has just exploded. Gentlemen, I am pleased to introduce you the SoftICE. Now can you understand the application staff panic? Don’t worry, it’s ugly but it doesn’t bite you. Pressing F5, the system returns to normal execution.

Now we have to load the NMS file into the debugger. This is easily done simply by double-clicking the NMA file. This file can be in any directory on the machine. The only time that this file is used is when you load it in the debugger. To verify whether the file was successfully loaded by SoftICE, use the “table” command that lists all loaded symbol tables. Several tables can be loaded simultaneously, but only one is active at a time. The “file *” command lists all files of a given table. To open a specific file, enter the file name in front of the command. See example below.

With the file already opened in the code window, press ALT + C to put the cursor into the source code. If this key sequence is not working, press ESC and then ALT + C. Note that you can only walk with the cursor up and down. Attempts to move the cursor sideways causes the command window to gain focus again. Place the cursor on the line you want to insert the breakpoint and press F9. At this time, the line was highlighted as shown in the picture below.

Now let’s press F5 to let the system return to normal operation. At the moment the driver starts, our breakpoint will interrupt this action and the SoftICE interface will be displayed. The highlighted line shows what the next instruction to be executed. Pressing ALT + L, the local variables window gains focus. You can navigate among the variables by moving the cursor up and down. Press ENTER key on the structures that begin with “+”, so that they are expanded. The image below shows register window and then, the local variable window.

I will leave a small table of commands here, but the full reference is in “SoftICE Command Reference” PDF file.

  • F5 : Run
  • F9 : Breakpoint
  • F8 : Step into
  • F10 : Step over
  • F12 : Step out
  • F7 : Run to cursor
  • WL : Local variable window
  • WW : Watch window
  • WD : Memory window
  • F3 : Swap among C source/Assembly/Mixed

Obviously there are numerous details and technical use, but let’s leave that to the documentations that coming with the product.

Wow, what a long post! You thought it would be easy, didn’t you? Until the next time …

Step into Kernel (Serial)

Friday, December 1st, 2006

I earned my first computer when I was 13 years-old, but seven years later, when I started my internship, I discovered that it was possible to debug programs that I used to write. Until then, my debug methods were always things like printing the variable value on screen. All right, in that time I didn’t use to develop complex programs enough to be stuck; over the time, I think I got used to working without the luxury as break-points and so on.

When I was introduced to a debugger, I even paid attention to it. I thought to myself: I can just put a printf and I’ll be OK. I don’t need all this stuff. At that time, we used the CodeView, a 16-bit debugger that runs on DOS which is displayed on the picture below.

It didn’t take long for me starting to depend on the debugger doing the simplest things, like brushing my teeth, for example. In fact, some situations, such as writing code for data collectors, writing firmware for hardware that has no display, situations in which we could not count on a serial port to see what was happening with the software, and the unforgettable situation where I had placed an oscilloscope on a particular processor pin to catch our software life signs, so, situations that we would give an arm to have a simple break-point made me realize that the debugger is really necessary.

The funny thing is that nowadays, in college, most students in my class don’t know how to debug software. I’ve tried to explain to my friends the tremendous advantage of using a debugger, but they end up having the same reaction I had years ago: “No way, Fernando, this is too complicated. Just by looking at the source code we end up finding the mistake”. So, that’s fine for me.

Well, we’re here to debug or to talk? Let us debug the driver generated by the traditional post Getting Started. This time we’ll do according to the tradition. We will need two machines, a Null Modem serial cable and last but not least, a copy of WinDbg. No, not worth reading this sentence again, you didn’t take it wrongly; you’ll really need two machines. I don’t know why some people have a certain resistance to believe in this. Well, I think we have overcome that phase and we can continue with the post.

If you do not have a copy of WinDbg, you can download it for free from the Windows Debugging Tools page at Microsoft site.

Let’s take this picture borrowed from the Windbg Help, which shows how we should set computers to do Kernel debug.

The machine named HOST is where WinDbg will run. This machine is, in most cases, the machine where the driver was developed. Let’s assume that this is the machine where the driver was developed and defer the details about how to set the symbol and source directory to another post. The machine named TARGET is where our driver is loaded by the system. The TARGET machine’s operating system don’t necessarily need to be Checked Build nor it is necessary to install any additional kit to the operating system. All you need is your driver and a serial port. All of the Windows NT platform (Windows NT, Windows 2000, Windows XP, Windows 2003 Server and Windows Vista) already bring their native debugger since its installation. Actually, this is a great feature in the use of WinDbg. Imagine that kind of problem that manifests itself on the client machine. You will not want to install anything aggressive or even that will change the scenario that would make the problem not manifest itself anymore. Believe me, this is not so rare as it seems to be. In these cases, simply enable the Debug mode on the TAGET computer and we’re done.

To enable the Debug mode in the TARGET machine, we need to edit some initialization parameters that are in the Boot.ini file located in the system root folder. This file has its attributes as system file, hidden file and also read-only. You may need to configure Windows Explorer to view these files. Using Windows Explorer, go to the root folder, remove the read-only property from the Boot.ini file, then open this file using a text editor like Notepad, for example. The contents of this file is similar to that shown below.

At Operating Systems section you will need to duplicate the referenced system line you will want to debug. Add to its end the /debugport=com1 /baudrate=115200 parameters that will configure the Debug mode. The first parameter selects the COM port being used, while the second selects communication speed. The serial protocol is not the fastest communication way between machines HOST and TARGET and in some cases is annoying having to wait for interactions between WinDbg and debugee system. There are other media that can be used if your computer does not have an available serial port or if you wish more speed and comfort during the Debug. See more details in the post Serial Killers.

The final result of our Boot.ini file should be something like shown below. I’ll skip the lines beginning that describe the operating system for better visualization putting “…” in their places.

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(2)\WINDOWS
[operating systems]
multi(0)... /NoExecute=OptIn
multi(0)... /NoExecute=OptIn /debugport=com1 /baudrate=115200

After making these changes, you can save the file and restore the original file attributes on it. In Windows XP and Windows 2003, the same described steps above could be done with the help of a tool named Bootcfg.exe. In Windows Vista, the process is quite different and I will describe it in a future post.

Back on the HOST machine, start WinDbg and select Kernel Debug… from the File menu. You should see a window that sets the communication way. Make it so in a compatible way with what was set in the Boot.ini on the TARGET machine. In this case, we are using COM1 serial port and baud rate of 115200. Clicking on OK will make WinDbg opening the serial port on the HOST machine and it will wait until the TARGET machine be ready. At this point, we get the following messages in WinDbg command window .

While the HOST machine waits, make sure your driver is installed on the TARGET machine and restart it. After that, when system is starting, the system loader is  displaying the options found in the Boot.ini edited by you. Select the option which appears [debugger enabled] as shown below.

After selecting this option, the machines HOST and TARGET should connect each other and the Command window would show a text similar to that shown below.

Microsoft (R) Windows Debugger  Version 6.6.0007.5
Copyright (c) Microsoft Corporation. All rights reserved.
 
Opened \\.\com1
Waiting to reconnect...
Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path.           *
* Use .symfix to have the debugger choose a symbol path.                   *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
*********************************************************************
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y  argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
*********************************************************************
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for
ntkrnlpa.exe -
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp_sp2_gdr.050301-1519
Kernel base = 0x804d7000 PsLoadedModuleList = 0x805531a0
 

Well, so far we know that the machines are connected. Pressing Ctrl + Break in Windbg, we will make the system freeze in the TARGET machine and the control is passed to the debugger. The debugger is ready to receive your commands when you get a command prompt KD indicated by the letters at Command window bottom left.

Just to make a simple test, type “lm” command and hit enter. This command lists the modules loaded by the system. Note that the Useless driver is not on this list yet. This happens because the driver instance is configured to have its execution started manually, so, the driver has not yet been loaded by the system. Then, select the item Open Source File… from File menu and open Useless.c file. Click on the DriverEntry function name and hit F9. Although there is no visible sign, we put a break-point on this function. The fact of having no visual change is due to the fact that our module has not been loaded yet. We can be sure that the breakpoint is installed listing the breakpoints with the “bl” command.

kd> bl
 0 e f9fb3430     0001 (0001) Useless!DriverEntry

Now we launch the “g” command that will cause the system to return to its normal execution.

Once the system is loaded, we will start the driver through the “net start Useless” command . When the driver starts, the DriverEntry entry point runs and consequently, we have its execution stopped at our breakpoint. The TARGET machine will freeze while the debugger has control over the system. Note that now our module is loaded, the breakpoints are easily identified by lines with red background color.

Well, the first steps have already been done, the commands and details on how to debug the kernel for Windows would be outside the scope of this post. But a good tip on how to obtain tons of details about this subject is the Windows 2000 Kernel Debugging book. The book is not one of my favorites; it gives a good introduction to the subject besides talking about how to use and build Debug Extensions, but does not bring the used techniques by real Kernel Debugging ninja masters. Another excellent information source on this subject is the discussion list Windows Debugger Users List dedicated to Windbg. That’s all without forgetting the Help.

This was an initial post about the subject, wait for future posts where I will give the steps about how to use Windbg with VMware and how to debug Windows Kernel with one machine using SoftIce (God rest his soul).

Have fun… 😉

Serial Killers

Tuesday, October 24th, 2006

There’s no doubt, they’re really giving up producing serial ports on notebooks and desktops. In this post, I will talk about some problems we encounter when debugging that driver that only reproduces the issue on that machine that has no serial ports. What alternatives do we have with this regard?

This month new computers have arrived here in the company. In these machines, we put some of our products in beta version. For any reason, only on new machines one of our drivers was not working as it should. As the quotes of the programmer’s code of ethics, the blame is on the trainee who did the tests until otherwise could be proved. After several unsuccessful attempts to discover what was happening, I had no doubts, let’s debug it. Avoiding moving the victim to my desk to connect the serial cable, I decided to install SoftIce. But life is a surprise box and we got a nice BSOD when SoftIce was started. Wow! This is rare, but it happens. In these cases, we don’t have much choice; we really have to use Windbg. The PCs we got were Dell Dimension 5150 model. Take a look at its cabinet back in the picture and answer quickly: where is the serial port? That’s it, there is none. By consulting the manufacturer’s Web site, we found that the computer had 8 USB ports but had neither a parallel nor a serial port.

At this moment, we made a circle and talk all together: “Oh God, what do we do now?” Some issues that happen in real machines are also reproduced in virtual ones, especially if the driver you are developing is a filter or a driver that does not deal directly with the hardware. Just for a test, we installed VMware on a new machine and fortunately the problem was reproduced. That’s one of those occasions that we get happy when the problem appears. Thereafter, I just used the old trick about having virtual serial port in a VM and redirect the data stream to a named pipe. This could become a post. 😉

Well, after connecting the debugger, having the break-point set, it did not take long to find the problem, generating a new version, testing, making it to work and live happily ever after. But not always the history is solved with these few steps. Some virtual machines do not support USB devices. Even VMware that offers this feature still does not support human interface devices (HID) like USB keyboards and mice. Not to mention the psychological and esoteric issues involving race conditions and/or a good joint action with Murphy. What can we do in these cases besides sitting and crying?

Fortunately, technology at the service of humanity predicted situations like these. In these cases, we traditionally have two alternatives:

1) Describe the bug you want to delete on a sheet of paper. It is recommended that this description has sections like “Go away strange bug …”. Sew a piece of paper inside a frog’s mouth and positive thinking in solving the problem; throw the frog into a river without seeing where it would fell. After seven days, do a “Rebuild All” in your project (A reboot before the build is recommended). If the problem persists, repeat the steps. Formatting the machine helps to remove bad fluids and scare evil and haunt spirits away that torment your code. Consult a software healer for best results.

2) Use a FireWire card and make a Windbg connection using an IEEE 1394 cable. This option is only available if the machine being debugged is a Windows XP or higher. These cards are installed on your PCI or PCMCIA bus. I did not have any chance of using this connection type to debug a machine, but it is certainly very fast and comfortable, since the data is transmitted at a rate of up to 400Mbps.

Can I use one of those famous serial ports to USB converters that are sold at Santa Ifigenia? Although it’s not required to install a driver on your system, these adapters use drivers to work. All USB stack is mounted to make these devices work. On a machine that is being debugged in Kernel, all the code that deals with serial communication or firewire is hard-coded in the system and it works with fixed standard addresses . These adapters can be used without problems by the debugger machine, but not by the debugee one.

What if I use a PCI multiport serial card? That depends on what I/O addresses and interrupts the card. Normally these cards use different addresses of the standards in order to not to create any compatibility issues with I/O addresses that probably pre-exist on the machine. If the card offers the option of using standard addresses, there is no problem.

Windows Vista will support Kernel Debug via USB 2.0 port, though some conditions are required. USB cable must be special for this purpose, the debugee computer USB interface must support Kernel Debug, and last but not least, the debugger computer USB controller should be Extended Host Controller Interface (EHCI) compliant.

If you are interested in acquiring IEEE 1394, remember that you must have, at least, two interfaces such that there is a link between the debugger and debugee machine. Although I have seen some lost comments on the Internet about FireWire x USB converter cables (for use in debugger machine), as far as I could see, they are electrically impossible to be built. That’s because the entire protocol is different. Whereas USB is a “Master x Slave” protocol, FireWire is “Peer to Peer” one. It would require some intelligent hardware along the way to do all translation.

The company I work for is acquiring these interfaces for PCMCIA FireWire in order to eliminate the problem of debugging customer’s notebook  that is crippled serial ports. I think soon we’ll give up the frog and all. I will make some comparisons with the good old serial cable and give some news on this subject.