Archive for May, 2007

Step into Kernel (VmWare+WinDbg)

Thursday, May 31st, 2007

It is enough with this idle talk and let’s go to what really matters. Speaking about Kernel Debugging, I have written a post describing the steps needed to Kernel Debugging with WinDbg, using two machines and a serial cable. But having two machines dedicated to this practice is a luxury that not everyone has. Thus, in another post, I have commented about installing and using SoftIce to debug drivers on a “single machine dedicated to debug”. I tell you “dedicated machine to debug” because using your development machine to debug drivers may not be one of your best ideas. The most courageous and confident people in their own code still risk themselves in this practice. Well, I cannot say much, I myself have already done that in lean times. Anyway, we have not fled the need for two machines to have a minimal environment for development and test drivers. An alternative to these scenarios is the use of a single machine, but the one that can have enough memory and CPU to run a virtual machine, in order to debug the kernel. In this post, I will take the steps to use a virtual machine as a guinea pig to test and debug drivers.

Configuring a virtual machine

The natural communication way between real machines using WinDbg to do Kernel Debugging is a serial port. To follow the same steps, we need to make your virtual machine to gain a serial port to make this communication possible. With your virtual machine turned off, click on the Edit vitual machine settings option.

A window will appear with the list of devices already installed on your machine in Hardware tab. Click Add… in order to add a new device and select Serial Port in the device list that is displayed on the screen below.

After clicking on Next, select the Output named pipe. This will cause the serial port on the virtual machine to communicate with the real machine, via a named pipe. Clicking on OK, the specific settings of the named pipe will be presented. The pipe name will be used later in one of the WinDbg settings so, if you want to use a name other than the one suggested here, try to remember this same name later. Then, change the value of the second combo to indicate that the other communication ending will be an application, in this case WinDbg. After that, just click on Finish.

At the end of these settings, select the Yield CPU on poll option as shown below.

Configuring the TARGET machine

Made all that “hocus-pocus”, now we have to set the TARGET machine system in the way it will able to make the Kernel Debugging. Remember that from Windows that runs inside the virtual machine point of view, the named pipe does not exist, it is just a serial port. If you still do not know what TARGET machine is a or how it can be set, then take a look in this post and follow the described steps in the part where a TARGET machine is configured .

Configuring the HOST machine

Assuming that your TARGET machine has been configured, now we have to configure WinDbg in a way it can be connected to a machine using named pipe, instead of a serial port. For that, I usually create a batch file that contains the following command line:

start C:\Progra~1\Debugg~1\windbg.exe -b -k com:pipe,port=\\.\pipe\com_1,resets=0

Creating a batch file is not a required step, you might want to retype everything in the Run… window, every time the debugging starts; but it’s all up to you. Notice that the pipe name appears here. I hope you still remember which name you have chosen.


Now we have everything already configured, it is just plug and debug. In this story, the virtual machine is the one that creates the named pipe that is opened by WinDbg. Thus, if you start WinDbg with the shown parameters before starting the virtual machine, you will see the window below stating the pipe was not found.

So, the sequence is, firstly, to connect the virtual machine, select the debug option in the Boot as shown below, and only after that, you should start the WinDbg with the parameters described above.

When WinDbg is finally connected to the virtual machine, via the named pipe, we have the following messages in our Command Window within WinDbg.

Thereafter, you know… You pick up the bugs.

Once more I hope I have helped you.
See you next time. 🙂

Now he only talks about that…

Monday, May 21st, 2007

Registration for Windows Drivers’ course has already started operating this weekend. Because of some delays, the course starting date was postponed and has also been confirmed for June 23.

To see the complete list of extension courses offered by the university, select the item “Information technology” item from “Extension” on the University web site.

For those who enjoy creating Penguins, my hard-coder friend William, will be giving a course about driver development for Linux at the same university. He will also use one of the OSR hardware training kits which appear in the photo above just to give you the opportunity to get your hands dirty. In the case of Linux course, the hardware is the USB one.

See you…

Windows Drivers Course

Thursday, May 10th, 2007

As some of you might read in another post, I was invited by Gama Filho University to give a course about Kernel Mode development for Windows. All the course details have already been determined and they would be producing material for publication; bu, as this post was being written, I was notified that there had been a delay in producing this material and it would be available for May 21. I already took a look in the Folder they were producing, and from what I got, they decided to summarize the course description that would be given. For this reason, I’ll put the full version of the description here. The following was the file I sent to the university.

This course is intended for developers or students who need to understand
the fundamental concepts on drivers' implementation for Windows. This course
would not cover specific drivers' implementation, such as printers, video,
SCSI, NDIS, USB, 1394 or UMDF. The aim of this course is to prepare students
who want to understand, test, complement or build drivers for Windows, using
general concepts involved in the process.
Knowledge in C Language 
Windows API Basics
Operating Systems Basics
Covered Topics:
System Architecture Overview
        Processes and Threads     
        Virtual Memory and Paging
        Kernel Mode x User Mode
        Subsystem and Native API
        Driver Stack and Plug-and-Play
        Object Manager
                Terminal Server
        Hardware Abstraction Layer (HAL)
Environment (Getting, installing and using)
        Windows Device Driver Kit
        Microsoft Visual Studio Express
        Microsoft Windows Debugging Tools
Writing a Driver
        Writing DriverEntry and DriverUnload
        Compiling a Driver
        Installing a Driver (Legacy)
                Load Order
        Debugging a Driver
                Checked Build Installations
                Driver Verifier
                Mapping an image for debugging
                Using Virtual Machines
        Creating DeviceObject
        Symbolic Links
        I/O Request Packets
        IOCTLs and DeviceIoControl
        Implementing Dispatch Routines
                Buffered I/O
                Direct I/O
                Neither I/O
        Objects, Handles and Pointers
        Arbitrary Context
        IRQLs, APCs, DPCs and WorkItems
                Spin Lock
        Events and Timers
        Custom Queues
Hardware Interaction
        Port I/O
        Interruptions and ISRs
Writing Filters
        Writing AddDevice routine
        Legacy Filter Drivers
        Forwarding IRPs
        Stack Locations
        Completion Routines
        Pending IRPs Handling
        IRPs Cancelation
        Creating IRPs for third Drivers
Drivers Types
        Legacy drivers
        WDM Drivers
        Creating an .INF file
        The use of SetupApi
        Web Sites
        Discussion Lists

By the time this post was written, the university attendance was not able to provide details about the course, such as registration dates, contents or due date. But I can anticipate what I know so far.

The course will be 40 hour-long and it is divided into 10 classes of four hours each. The classes will be on Saturdays from 1:00pm to 5:00pm.

The course is scheduled to have its first class starting May 26. It will necessary, at least, five students to form one class, but by taking the amount of people who have already contacted the university seeking for details (without disclosure) my concern is now the upper limit of 10 students. We decided to limit the class of 10 students in order there would be a better time usage for the content.

I agree that 40 hours is not enough to learn everything you need to develop drivers, but it will be an excellent starting point to have the first contact. If you are the type of person who needs to learn absolutely everything to start developing then I can antecipate that this course is not for you. The best kernel developers I know (from their Blogs, but I know) have been working with drivers for years and years, but they say they have never know everything. To get an idea, some developers focus on just certain issues within the kernel. One has been working only with disk drivers for about 10 years; another one,  for about seven years, only with network drivers. Tony Mason, for example, has been working for 18 years, only with File System Drivers. Now, ask them if they know everything! And I used to feel myself bad for trying to focus only on working with Kernel, refusing to work in User Mode. Sometimes, with friends, I have heard comments about how to do this or that using .Net, and I, having no idea what they were talking about. I didn’t know that inside the Kernel issue have still existed a myriad of details and specialization. Is it possible to learn everything?

I am preparing the content in a way that students may have some practical experience, such as writing and compiling a “Hello World” driver, install them in virtual machines and debug them. There is no way. It is necessary to get your hands dirty. Connect two real machines with a serial cable and do Kernel Debug. Generate and analyze Crash Dumps. I think we have to make the most possible to do things together in one classroom, and perform some experiments that the books attempt to describe only in words and pictures. This desire of putting things in practice was the responsible for contributing with me to get the training kit on sale at OSR Online. The kit that I’ll be taking to the course, and that appears in the photo above is basically a PCI Digital I/O. My goal is to make students’ drivers control this device.

Well, as soon as I’m getting news, I will let you know.
See you…

How was it in Boston?

Wednesday, May 2nd, 2007

As I commented in my last post, I attended the seminar for Windows File System driver development produced by OSR in Boston. In this post, I will talk a bit about how things went there.

To begin with, none other than Tony Mason to give this seminar that was very, very helpful. For those who are not familiar, Tony Mason is one of the four OSR partners. He has been working with drivers over 20 years, and particularly with File System drivers since, 1989. He began working with File Systems for Windows NT in 1993, but was already developing for Unix File Systems before that. Tony Mason is also the person responsible for reviewing and updating Windows NT File System Internals book contents by Rajeev Nagar, written and published in 1997. This book is now a reference in the subject and its revised edition, which still has no release date, will bring new issues that have been implemented in Windows 2000, XP and Vista. I have also commented a bit about this in this post.

You people can call me a “super fan”, but it is nice to see how it just takes implementation details of File Systems Drivers (FSD) and File Systems Filters (FSF). I obviously asked him to sign up my book copy. Now I can keep that autograph on my calendar next to my first absorbent packaging. I’d love to show here, but it is about his signature, I don’t think it would be a good idea to put it on the Internet.

We were fifteen students in the class, where twelve were Americans, one Romanian, one Canadian and one Brazilian. I had the opportunity to be the first Brazilian to take this course from OSR. There were people of various ages. The youngest was about 25 years-old, whereas the oldest had his hair completely white. I imagine that the class average age was about to 40-years old. I’m comfortable with the certainty that, in this development area, there is no risk of being replaced by kids who just got out of college, just by lower prices of labor. Tell me the truth! How many 40 years-old programmers do you know? I said programmers, not managers. Well, fortunately the driver development will still require a lot of experience.

It’s hard to believe, but a woman (about 43 years-old and really looking like a woman!) was also part of the group. It is not sexism. You know how it is unusual to see a woman (who does not look like a man) programming, but programming kernel it is even more unusual. Another specimen of this rare species is Molly Brown, a programmer responsible for the NT Cache Manager. Check here.

As it might be expected, the seminar rhythm was accelerated. After all, the content was quite dense. We had an overview of IoManager, virtual memory, multi-threaded, multi-processing and Object Manager just to begin with. These issues are still operating system part only and not specifically for FSD. We started to see File System contents itself at the middle of the second day. All content had been well distributed and for each point explained, we received comments focusing on the FSD, FSF and Redirectors. Interesting to see how each item was discussed, getting comments on the implementation differences among different Windows versions.

In our group, there was at least one person who would develop a Redirector, one that would develop a File System, but the public in general would improve their knowledge to develop anti-virus, anti-malware, access control to systems and data replication. All implemented as FSF.

Data replication? Would not it be better to develop data replication with a disk filter? Yeah, I think so either. The File System interface is much more complex than the disk interfaces, but try to explain that to that guy! My sincere wishes of good luck to him.

I can say that the student’s level was medium. Well, I’ve read the book from Rajeev a few times and I already had the opportunity to work with FSF. I was aware of I would not waste this opportunity to be there learning things that a book could teach me, but part of the group didn’t think the same. Maybe they were thinking it would be easy. It was funny when we had our coffeebreaks and they sed to commente that the matter was very thorough and very detailed. No doubt it was! Some of them never had contact with FSD before. I can say who already had contact with the subject and was able to enjoy the details of interaction between the FSD and other system components such as the Cache Manager, Memory Manager or even the LAN Manager Server, really enjoyned the course. There were some people who got stuck on comments about IoMarkIrpPending and Completion Routines. Seriously? I’m saying this issue is trivial, but you should not leave to learn this in a FSD seminar.

There were very good people there, too. Questions and discussions made Tony open FastFAT, CDFS or RDR sources to show how things would be implemented in the real world. And I thought that just the reference considered source samples as part of the documentation. Have you already searched for some information about IRP_MN_MDL in reference? Well, I was able to find a clue in the part that talk that IRP_MJ_READ was mentioned, where they listed this, among the other options and said:

“For more information about handling this IRP, study the CDFS and FASTFAT samples that are included in the Windows Driver Kit (WDK).”

According to Tony, developing an FSD requires much knowledge and experience, but unlikely what many people could think, developing filter is even worse. All begin with the false impression that it is necessary to change only a small part of the filter sample from WDK (SFilter), that everything would be okay. But a side effect will require just one more change here, another there, and when you realize we have a complexity greater than a FSD project. A strong argument he used was that, when you implement an FSD, you know everything what may happen, but when you develop a filter, you are at the interface of two black boxes, the operating system and FSD; and any change in behavior, generated by your filter, can generate the most bizarre, intriguing and unexpected situations to be debugged in both black boxes.

The OSR gave us a binder with all PPTs used in the seminar; also, it gave us an OSR portmanteau, a NT Insider edition and to some people’s surprise a crazy spring. A spring crazy? Well, in summary it is a spring, but a crazy one.

If you want to know more about File System Drivers, an excellent source of information is the OSR mailing lists. It’s worth takeing a look. Obviously, I will comment more on this subject here.

See you next time! 🙂