Archive for July, 2007

ExAllocatePool with Tagging

Thursday, July 19th, 2007

My friend Slug was always a great reference on a different way of seeing things. He usually has a different viewpoint from mine, and I have always learned from these other viewpoints. This post will be different than you are accustomed to seeing here. Wanderley had introduced the subject “How to be a better programmer in the next six months” and brought us this new point of view (at least for me) to see blogs, known as Tagging. Here is my reply.

I cannot say that everything will be different in the next six months; I am really going to make every effort to turn a better programmer. I think my answer is similar to the response that Strauss has given us, but not equal.

As some of you know, beyond the work day at Open, I have still faced the university to complete my degree in Computer Engineering. This course has brought me new concepts about micro-processing, embedded systems, computer architecture and other subjects different from Windows programming in C/C++. I can say that it has opened up a little my range of opportunities to learn new things. Developing drivers is, in addition to programming, establishing communication between two near worlds, but virtually unknown to each other. Usually, who knows one of these sides very well, don’t do the same about the other one. It is natural. I have already known a bit of software, and every six months to go, I know some new things about hardware. Knowing better the hardware, I think I can better understand and program the operating system that interacts with it.

Speaking of new things, this month I have found myself buying a book about Device Drivers for Linux. I have been Microsoft ever since. Obviously, I am referring to the professional aspect. My days of MSX and CP200 exert the same influence that my kindergarten does about my resume. The book has not arrived yet, but certainly, it will feed my database about Software/Hardware interaction and it’ll allow me to draw a parallel between these two development platforms, Windows/Linux, and thus, I’ll be able to better understand what they have in common, both concerning Software/Hardware interaction about operating system concepts.

A friend of mine once told me that one of the best ways to learn something is teaching. I have been writing this blog for almost a year and have learned a lot from it. Writing about some things, which would be simple, made me take another view and see that there are still more and more details about it. Thus, I end up learning the details that were lost at first. Some lectures I have given the company and the device driver course work as a Boost for this learning way. I confess I’m enjoying it.

In conclusion and in summary, I will continue studying to get my degree, reading about Linux, continue blogging and being questioned about its issues. This subject is very fun for programmers, but we cannot forget that we are also husbands, sons, brothers, friends and so on. This post can offer you that courage to start studying everything and devote all your time available for the benefit of being the best developer but, cycling and going to the beach are also part of excellent programmer’s rising.

Enjoy in moderation. 😉
See you…

Bug on my Boot driver! Now what?

Monday, July 16th, 2007

Writing drivers is a task that must be done with little care. After all, any unresolved situation between your code and the operating system will result in a beautiful blue screen. But everything in this life has a cure and fortunately God has created the debugger to deal with these situations. After fixing the problem, it is just to change the .sys file and that’s it. Just changing the file means that the machine is expected to be in a stable condition: then, replace the file on the system and reboot the machine. But life is a surprising box and you may not believe, but most drivers are automagically started. Well, in this case, the only think we can do is to pray that the error does not happen until the driver is replaced. Otherwise, we would have to use some resources, such as putting the hard drive of victim’s machine on another computer to replace the sick driver. When there is no one else’s computer, use the Recovery Console in XP to avoid the sick driver being loaded. With some luck, this driver is not a filter attached to something important like disk, video or File System. You know, if a filter is not able to be loaded, then the main driver, either. In summary, there is a lot of juggling you can invent to replace a buggy driver that is automatically loaded. Is there anything that does not depend much on luck or even on the creativity? Today, I will talk a bit about the replacing driver system with WinDbg.

Mapping driver files

You can map the driver files so the WinDbg can replace it at the time it’d be loaded. Is not that beautiful? For this, you must use the .kdfiles command. With this command, you make a link between the driver that runs on the Target machine and the one that has been fixed on the Host side. One way to do this is to initially create a file that lists these two drivers. This file should be a simple text file where the syntax is shown below. This file can have any name and extension.

map 
\??\C:\Windows\System32\drivers\MyDriver.sys 
C:\My_Driver_Project_Folder\MyDriver.sys

The word map marks the beginning of a mapping; this does not change. Next line is the driver file path to be replaced from. This line should have the same format used in the ImagePath value under the driver key that is on registry. The last line is another driver path. This path could point at a driver in the Host machine itself or on the network.

This mapping works only in Windows XP or higher, in the Target machine, obviously. If you’re not familiar with terms like Host and Target, check this post out.

Once created the file, you will use WinDbg to launch the following command:

kd>.kdfiles C:\Any_Folder\MyFileMap.txt

Thereafter, whenever the driver is loaded by the system, the kernel checks whether this file is mapped in the debugger, and if so, WinDbg sends the new driver through the debugger connection, such as serial, USB or firewire. For large files, I recommend using firewire. You must understand that the file on Target system’s disk is replaced by the new version. This means that in the forthcoming launches of our driver, the new version will still be loaded, even if the debugger is not connected to the system.

Does this really work?

To make things a little clearer, let’s see a practical example. To do this, we need a very simple driver source, or even useless, which can be found here. Let’s change its DriverEntry function, so that it will be as shown below:

NTSTATUS DriverEntry(IN PDRIVER_OBJECT  pDriverObject,
                     IN PUNICODE_STRING pusRegistryPath)
{
    //-f--> Let's use __DATE__ and __TIME__ to change this message
    //      each time this driver is built.
    DbgPrint("This driver was built on "__DATE__" "__TIME__"\n");
 
...

So, we build an initial version of the driver and we install it on the Target machine in the way you best think, but we’ll assume that after the driver has been installed, the registry will be as shown below.

If we start the driver, we will have something like the following string in the debugger output:

This driver was built on Jul 16 2007 00:04:03

Now, we’ll create the mapping file. Here I will save it as Z:\Sources\DriverEntry\Useless\map.txt. Following the same format the driver was registered in the registry, our mapping file should have the following content:

map
\??\C:\Windows\System32\drivers\Useless.sys
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

Notice that the folder where I saved my file map.txt is the same where the driver sources are. This is just a matter of organization. The mapping file could be in any folder. Then, use the command .kdfiles as shown below. Notice that we list the existing maps simply using the same command without any parameters.

kd> .kdfiles Z:\Sources\DriverEntry\Useless\map.txt
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
 
kd> .kdfiles
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
\??\C:\Windows\System32\drivers\Useless.sys ->
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

Afterwards, we’ll Rebuild the driver (that should change that time stamp) and we will restart it. If everything is right there by your side, you should have an output with a time stamp different from what we had before.

kd> g
But now? I did nothing so far...
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\??\C:\Windows\system32\drivers\Useless.sys)
  File size 2K.
MmLoadSystemImage: Pulled \??\C:\Windows\system32\drivers\Useless.sys from kd
This driver was built on Jul 16 2007 00:17:04

Well, if this works for a driver with manual start value, then the automatic one also should work. To see this, change your driver start value to System or Automatic, do a Rebuild again in the driver, and finally, we will restart the Target machine. When the driver is loaded, we have the automatic replacement of its image. In WinDbg, we will have the following:

Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp_sp2_gdr.070227-2254
Kernel base = 0x804d7000 PsLoadedModuleList = 0x805533a0
System Uptime: not available
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\??\C:\Windows\system32\drivers\Useless.sys)
  File size 2K.
MmLoadSystemImage: Pulled \??\C:\Windows\system32\drivers\Useless.sys from kd
This driver was built on Jul 16 2007 00:25:38

Yeah, it really works! But what if was my driver a Boot driver? This means that the driver image will be loaded even before the connection is established with WinDbg. Have you heard that what has no remedy, it is remedied? Fortunately, this is not applied here yet. There is a way to make this work, even with Boot drivers.

Mapping Boot drivers

To establish a connection with WinDbg, we need to replace the system loader by a special version. This version makes this connection with the Kernel Debugger, even before the Boot.ini file is read. For this reason, the connection parameters are fixed with COM1 and baud rate of 115200. This loader version is located in the directory C:\winddk\3790\debug of the DDK with its ntldr_dbg name. This file should replace the original loader system version that is at the boot drive root with the ntldr name. The debug version must stay with the same name as the original loader.

Before rebooting, we need to change the driver’s start value to Boot and remove the ImagePath value from the registry. As you may know, boot drivers do not have that luxury of using file paths that have drive letters. At the end of the changes, we should have the registry as shown below.

As I said earlier, the driver file path format should be the same as it is on registry, but knowing that now there’s no file path in the registry, so we adopt the same forms adopted by the system. Oh! okay! The same format. And what would it be? To see this format, simply restart the system with the debug loader, which should give us the following screen at boot-up.

This is the time to connect to Windbg using pre-determined parameters connection. This should result in the following output in the debugger.

Microsoft (R) Windows Debugger  Version 6.7.0005.1
Copyright (c) Microsoft Corporation. All rights reserved.
 
Opened \\.\pipe\com_1
Waiting to reconnect...
BD: Boot Debugger Initialized
BD: osloader.exe base address 00400000
Connected to Windows Boot Debugger 3790 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Boot Debugger Kernel Version 3790 UP Checked x86 compatible
Primary image base = 0x00000000 Loaded module list = 0x00000000
System Uptime: not available
The call to LoadLibrary(bootext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

Now you see the Boot.ini selection menu. Select your option and continue loading the system. A list of boot drivers should be presented, which are the ones loaded before the first breakpoint that the debugger can stop.

BD: \WINDOWS\system32\ntoskrnl.exe base address 804EA000
BD: \WINDOWS\system32\hal.dll base address 806FF000
BD: \WINDOWS\system32\KDCOM.DLL base address 80720000
BD: \WINDOWS\system32\BOOTVID.dll base address 80010000
BD: \WINDOWS\system32\DRIVERS\ACPI.sys base address 80124000
BD: \WINDOWS\system32\DRIVERS\WMILIB.SYS base address 80001000
BD: \WINDOWS\system32\DRIVERS\pci.sys base address 80062000
BD: \WINDOWS\system32\DRIVERS\isapnp.sys base address 80003000
BD: \WINDOWS\system32\DRIVERS\compbatt.sys base address 8000C000
BD: \WINDOWS\system32\DRIVERS\BATTC.SYS base address 80013000
BD: \WINDOWS\system32\DRIVERS\intelide.sys base address 80017000
BD: \WINDOWS\system32\DRIVERS\PCIIDEX.SYS base address 80019000
BD: \WINDOWS\System32\Drivers\MountMgr.sys base address 80152000
BD: \WINDOWS\system32\DRIVERS\ftdisk.sys base address 8015D000
BD: \WINDOWS\System32\drivers\dmload.sys base address 80073000
BD: \WINDOWS\System32\drivers\dmio.sys base address 8017C000
BD: \WINDOWS\System32\Drivers\PartMgr.sys base address 801A2000
BD: \WINDOWS\System32\Drivers\VolSnap.sys base address 801A7000
BD: \WINDOWS\system32\DRIVERS\atapi.sys base address 801B4000
BD: \WINDOWS\system32\DRIVERS\vmscsi.sys base address 801CC000
BD: \WINDOWS\system32\DRIVERS\SCSIPORT.SYS base address 801CF000
BD: \WINDOWS\system32\DRIVERS\disk.sys base address 801E7000
BD: \WINDOWS\system32\DRIVERS\CLASSPNP.SYS base address 801F0000
BD: \WINDOWS\system32\DRIVERS\fltMgr.sys base address 801FD000
BD: \WINDOWS\system32\DRIVERS\sr.sys base address 8021D000
BD: \WINDOWS\System32\Drivers\KSecDD.sys base address 8022F000
BD: \WINDOWS\System32\Drivers\Ntfs.sys base address 80246000
BD: \WINDOWS\System32\Drivers\NDIS.sys base address 802D3000
BD: \WINDOWS\System32\Drivers\Useless.sys base address 8000F000
BD: \WINDOWS\System32\Drivers\Mup.sys base address 80300000
BD: \WINDOWS\system32\DRIVERS\agp440.sys base address 8031B000
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...

Here the connection to the loader is closed. A new connection would be established if you had selected the Debug entry from Boot.ini, but what we have to note here, is the file path format used to load drivers during system boot. Notice that our test driver is among the drivers from the above list. Let us adopt this same file path format on our mapping file.

map
\WINDOWS\System32\Drivers\Useless.sys
Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys

After modifying the mapping file and save its contents to disk, we update the WinDbg, so it takes this change. Then, we will restart the system.

kd> .kdfiles Z:\Sources\DriverEntry\Useless\map.txt
KD file assocations loaded from 'Z:\Sources\DriverEntry\Useless\map.txt'
 
kd> .reboot
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...
BD: Boot Debugger Initialized
Connected to Windows Boot Debugger 3790 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.  (Initial Breakpoint requested)
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Module List address is NULL - debugger not initialized properly.
WARNING: .reload failed, module list may be incomplete
KdDebuggerData.KernBase < SystemRangeStart
Windows Boot Debugger Kernel Version 3790 UP Checked x86 compatible
Primary image base = 0x00000000 Loaded module list = 0x00000000
System Uptime: not available
The call to LoadLibrary(bootext) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

The same message sequence happens, but this time, the mapping is done as it should and we have the following output when the boot drivers are loaded.

BD: osloader.exe base address 00400000
BD: \WINDOWS\system32\ntoskrnl.exe base address 804EA000
BD: \WINDOWS\system32\hal.dll base address 806FF000
BD: \WINDOWS\system32\KDCOM.DLL base address 80720000
BD: \WINDOWS\system32\BOOTVID.dll base address 80010000
BD: \WINDOWS\system32\DRIVERS\ACPI.sys base address 80124000
BD: \WINDOWS\system32\DRIVERS\WMILIB.SYS base address 80001000
BD: \WINDOWS\system32\DRIVERS\pci.sys base address 80062000
BD: \WINDOWS\system32\DRIVERS\isapnp.sys base address 80003000
BD: \WINDOWS\system32\DRIVERS\compbatt.sys base address 8000C000
BD: \WINDOWS\system32\DRIVERS\BATTC.SYS base address 80013000
BD: \WINDOWS\system32\DRIVERS\intelide.sys base address 80017000
BD: \WINDOWS\system32\DRIVERS\PCIIDEX.SYS base address 80019000
BD: \WINDOWS\System32\Drivers\MountMgr.sys base address 80152000
BD: \WINDOWS\system32\DRIVERS\ftdisk.sys base address 8015D000
BD: \WINDOWS\System32\drivers\dmload.sys base address 80073000
BD: \WINDOWS\System32\drivers\dmio.sys base address 8017C000
BD: \WINDOWS\System32\Drivers\PartMgr.sys base address 801A2000
BD: \WINDOWS\System32\Drivers\VolSnap.sys base address 801A7000
BD: \WINDOWS\system32\DRIVERS\atapi.sys base address 801B4000
BD: \WINDOWS\system32\DRIVERS\vmscsi.sys base address 801CC000
BD: \WINDOWS\system32\DRIVERS\SCSIPORT.SYS base address 801CF000
BD: \WINDOWS\system32\DRIVERS\disk.sys base address 801E7000
BD: \WINDOWS\system32\DRIVERS\CLASSPNP.SYS base address 801F0000
BD: \WINDOWS\system32\DRIVERS\fltMgr.sys base address 801FD000
BD: \WINDOWS\system32\DRIVERS\sr.sys base address 8021D000
BD: \WINDOWS\System32\Drivers\KSecDD.sys base address 8022F000
BD: \WINDOWS\System32\Drivers\Ntfs.sys base address 80246000
BD: \WINDOWS\System32\Drivers\NDIS.sys base address 802D3000
KD: Accessing 'Z:\Sources\DriverEntry\Useless\objchk_wxp_x86\i386\Useless.sys'
 (\WINDOWS\System32\Drivers\Useless.sys)
  File size 2K.BD: Loaded remote file \WINDOWS\System32\Drivers\Useless.sys
 
BlLoadImageEx: Pulled \WINDOWS\System32\Drivers\Useless.sys from Kernel Debugger
BD: \WINDOWS\System32\Drivers\Useless.sys base address 8000F000
BD: \WINDOWS\System32\Drivers\Mup.sys base address 80300000
BD: \WINDOWS\system32\DRIVERS\agp440.sys base address 8031B000
Shutdown occurred...unloading all symbol tables.
Waiting to reconnect...

And later, our proof that the driver file has been successfully replaced.

Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.  (Initial Breakpoint requested)
Symbol search path is: SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Kernel Version 2600 UP Free x86 compatible
Built by: 2600.xpsp_sp2_gdr.070227-2254
Kernel base = 0x804ea000 PsLoadedModuleList = 0x8056d620
System Uptime: not available
This driver was built on Jul 16 2007 01:18:53

I assure you that this could still save your life if you had a File System filter with a bug in its DriverEntry on a customer's machine. The expression of panic when the customer sees his machine restarting in an endless loop is interesting, but keeping your job is a little more interesting.

See you!

Installing and Using DSF

Tuesday, July 10th, 2007

You are probably already tired of reading that OSR has hardware kits for driver development training for USB and PCI. It’s really frustrating to want to learn how to develop drivers that control board without even having one around. But I am poor and I have no money to buy these imported toys. Actually, I also got some pain when I had to go to the post office to get two kits I had bought and paid a tax of U$ 183.59, not counting the kit’s cost. Well, this is a point.

Assuming you are already an experienced programmer and knows how to make drivers for USB devices with one tied eye behind your back, imagine that you have to write a driver for a USB device that is not ready yet. That is, you have the specification, you know its characteristics, but in fact, the device is not ready for you to do all the tests you need. In this case, the training kit would not help much. This is another point.

Joining these points, we can see that we are lost and the only way is just forget everything and go to sell coconut water at the beach.

A considerable alternative

Taking these same points, Microsoft has developed the Device Simulation Framework for USB Devices. This framework is composed of, among other components, a driver that is installed on your test machine that simulates an USB controller. This driver is Lower Filter Drivers for Windows USB controller drivers. It intercepts the interactions that Windows does with the actual hardware and simulate hardware interrupts. From driver’s viewpoint, there is no difference between DSF and real devices.

But how can the framework simulate a USB device that it doesn’t know? Actually, the framework redirects these interactions to components in User-Mode that can be written by you in any language that can use COM. This way, you can control device behavior being simulated. Observe the figure borrowed from the MSDN page, so you can have a clearer picture of how these components are organized.

Installing DSF

The DSF comes in the WDK ISO that you can download from Microsoft. It can only be installed on Windows XP SP2 or higher (including x64 platforms). When you install WDK on your machine, the DSF is not installed. You need to install it separately. The  dsfx86runtime.msi and dsfx64runtime.msi files, responsible for installation, are located in the \dsf of the ISO. The framework installation is extremely simple, but you can only see something when, after installed, you run “softehcicfg /install” command line at the “\Program Files\dsf\softehci” folder, as it is shown below.

This command creates a virtual USB controller in your device tree, which can be seen on Device Manager. Your system may ask the drivers for this new hardware that was added. If it does, enter the system directory (C:\Windows) for the search for drivers taking place. After the drivers have been installed, you should get two new devices. They are: Microsoft USB 2.0 EHCI Host Controller Simulator and USB Root Hub, as it is shown below.

Using VBScript to write devices

To simulate devices, DSF installs a group of components that implement COM objects, such as: Devices, Configurations, Interfaces, Endpoints, Descriptors, and so on. Who has already developed firmware for USB knows exactly what I mean. With these objects, you can write components that can simulate any USB device. Along the framework, three examples of devices that can be simulated are also installed. These devices’ sources are examples that come together in the WDK. They are located at \WinDDK6000\src\Test\DSF\USB. To use one of these examples, we just need a language script, or any language able to use COM interfaces.

Let’s take a look at the keyboard example and look for its IRPs. To make the keyboard work, go to the “\Program Files\dsf\usbhid” and run the following command line:  “cscript Create1.1Kbd.wsf” and follow the steps. If this is the first time you are doing this, Windows may ask the drivers for new devices that will be created. The script will create a Generic USB Hub and a USB Microsoft Natural Keyboard, as it is shown below.

Before removing these devices, let’s take a look using IRP Tracker, a software that I have mentioned in another post, and watch this simulated keyboard’s IRPs. Start the IRP Tracker and select all KbdClass driver. KbdClass is the driver responsible for centralizing and implementing interfaces for the keyboard device class. The IRPs from all keyboards are here.

While the first command prompt is stopped, start another one and from the same directory, run the following command line: “cscript Use1.1Kbd.wsf”. This script simulates pressing keys on the simulated keyboard writing the phrase “Hello World!”. For each key the system gets, records are generated from monitored IRPs. Thus, the IRP Tracker should look as it is shown below.

This is an excellent tool for using with virtual machines and it can anticipate a good amount of testing until the actual hardware is ready. In cases where the hardware and the driver are new, this could be a good way to discover which one is causing the issue. After all, hardware has bugs, too. Even if you’re a student and have no hardware, this could be useful to train your learning. At least, you will not burn anything.

See you!