Archive for September, 2007

More Contacts

Friday, September 28th, 2007

It has become more frequent, at least for me, the demand for skilled developers in the Windows Kernel. What is frequent for me? Well, besides the companies that I have regularly worked for, which I imagine are already tired of asking me if I know more Kernel programmers, two companies asked me indications just this last month. One of them is the company that is taking one of my friends, Rodrigo Strauss, to Porto Alegre. If you think it is hard to find places for this type of development, it is even harder to find developers for this type of vacancy. Usually when someone asks me for any developer’s indication, I just end up saying that, after more than ten years as a programmer, I can count every kernel developer I have know using just one hand.

The fact is that both are hard to find, so I’m providing a forward invitations and opportunities of Windows Kernel area to this blog readers who are interested in it. Those interested ones should contact me by e-mail so I can expand our network of contacts. Please, try to avoid leaving your e-mail in the post comment area, because in that way, anyone else (including spammers) can see them. If you have not gotten my e-mail yet, just get it from my Blogger profile. Even those who are not interested in new challenges can send me a hello. So that I can know how many of you have already worked with Windows kernel and attempt to start using the fingers of the other hand on that score.

There is an initiative like this is on the OSR Online site where several specialized companies post jobs about Windows Kernel. Maybe in a few hundred years, when my blog is famous worldwide, companies want to publish their vacancies for Brazilian Kernel programmers at DriverEntry.com.br.

For now I just want to know how many and who are the Brazilian Kernel Coders, but I’ll be open to suggestions. Everything is depending on the volume and quantity of responses that I will get during my attempt.

CYA…

But I don’t know English

Thursday, September 27th, 2007

It was through this post that I found out that MSDN beta content (including WDK) is now available for the automatic translation into Portuguese and other languages.

For those who have trouble understanding the original text in English, they can now enjoy the comfort of having the same problems of understanding the same text in Portuguese. This is because the translation is not that good but, it helps for those who have little or no English knowledge.

The result is displayed on the already known MSDN layout, but the main text will come with a split that will separate the Portuguese text from its equivalent English. Maybe it gets better when it stops being beta, but anyway, it remains an alternative for it.

Take a look:
Introdução do kit de drivers do Windows

Automatic translations may be pretty funny. I’ve already seen nasty translations in manuals of imported goods. It’s a shame I have not found that PenDrive manual I have recently bought to share some “pearls” with you!

Device Driver Programming == Motorista de dispositivo programando

Be careful!

Try, Except, Finally and IoWriteErrorLogEntry (Part 1)

Tuesday, September 25th, 2007

Most of you have already known about exception handling in C. Knowing that the C Run Time Kernel does not support exception handling in C++, the exception handling in C fits us like a glove. Exceptions do not always mean that a critical error has occurred, but regardless, it is essential to treat them. An unhandled exception in kernel means blue screen. In the case of an unexpected exception, such as an Access Violation, using exception handlers could prevent the system from a blue finish. That’s great, congratulations, one more life saved, but we must not forget that there had been an unexpected exception. This post will not mention how to use exception handlers in details, but it will talk about how we can report such events to the system administrator.

A Typical Exception Handling

For those who are just waking up right now, I’ll just give you a brief description of exception handling in the source below. The details are in the reference. The following source was taken from one of the examples available on this blog. This feature can save us from a merciless blue screen whenever “a loser” who calls this function, passes an invalid string as a source of duplication.

/****
***     DupString
**
**      This routine receives a Unicode String and duplicates it.
**      The resulting string should be released
**      using the ExFreePool routine.
*/
 
NTSTATUS
DupString(IN PUNICODE_STRING   pusSource,
          OUT PUNICODE_STRING  pusTarget)
{
    NTSTATUS    nts = STATUS_SUCCESS;
 
    __try
    {
        __try
        {
            //-f--> Initializes the target
            pusTarget->Buffer = NULL;
 
            //-f--> Allocates the target buffer
            if (!(pusTarget->Buffer = (PWSTR)ExAllocatePool(PagedPool,
                                                            pusSource->Length)))
                ExRaiseStatus(STATUS_INSUFFICIENT_RESOURCES);
 
            //-f--> Copies the buffer
            RtlCopyMemory(pusTarget->Buffer,
                          pusSource->Buffer,
                          pusSource->Length);
 
            //-f--> Copies the buffer legths
            pusTarget->MaximumLength = pusTarget->Length = pusSource->Length;
        }
        __finally
        {
            //-f--> If something gets wrong, we must release the target buffer.
            if (AbnormalTermination())
                if (pusTarget->Buffer)
                    ExFreePool(pusTarget->Buffer);
        }
    }
    __except(EXCEPTION_EXECUTE_HANDLER)
    {
        nts = GetExceptionCode();
 
        DbgPrint("DupString >> An exception occourred at "__FUNCTION__
                 " with status 0x%08x.\n", nts);
 
        ASSERT(FALSE);
    }
    return nts;
}

Note that there are two blocks of execution. The outermost is the one, which will deal with unhandled exceptions, and the innermost the one, which will deal with resource releasing. The exception handler block will executes the __except block when any exception is thrown from the inside of __try block. This will give us the opportunity to find out what happened and, in some cases, to request that the instruction causing the exception is re-executed. The completion __finally block is executed when the thread goes out of the main block, either by reaching the block’s end, for a return, for a goto out of the block, or even an exception that was thrown. The __finally block is always executed. This gives us the opportunity to release any resource that has been pending during the execution of the block. Taken it together, we can imagine the following situation: suppose the function RtlCopyMemory() throw an exception. In this case, the __except block will run, but for this to occur, the execution flow will leave the inner block as well, thus causing the __finally block execution.

Within each handler, there is a special function call. Inside __except, we have GetExceptionCode(), which returns us the exception code that was thrown. This function can only be called within this context. Also note that, within __finally block, there is a call to the AbnormalTermination() function which indicates that the block was not executed until reaching its end. In our example, we use this function to determine that something wrong should have happened and it is necessary to release the buffer which would be returned to the calling function. Anyway, check out the reference for a complete description of these resources.

The main point here to note is the ASSERT at the end of the __except block. If this source is compiled in Checked, the ASSERT will cause a prompt in the debugger. If there is no debugger attached to the system, then a blue screen will appear. That’s because this macro will throw a break exception to the debugger but, as the debugger will not be there to handle this exception, figure it out. When compiled in Free, the ASSERT macro is translated into nothing: our function will handle the exception gracefully and it will return the exception code to the calling function. How sweet but, when will is the administrator going to know?

A Life Sign

The standard way to leave this signal is writing in the system event log. What is this? This link can give you detailed information about the system event logs. To write a record in this list from the kernel, we initially must allocate the buffer that will house all the information record using the IoAllocateErrorLogEntry() function.

PVOID 
  IoAllocateErrorLogEntry(
    IN PVOID  IoObject,
    IN UCHAR  EntrySize
    );

The pointer returned by this function, although being the type PVOID, points out to an IO_ERROR_LOG_PACKET structure, which has variable size depending on the information that composes it. This structure is used to package all the event data that will be logged.

typedef struct _IO_ERROR_LOG_PACKET
{
    UCHAR MajorFunctionCode;
    UCHAR RetryCount;
    USHORT DumpDataSize;
    USHORT NumberOfStrings;
    USHORT StringOffset;
    USHORT EventCategory;
    NTSTATUS ErrorCode;
    ULONG UniqueErrorValue;
    NTSTATUS FinalStatus;
    ULONG SequenceNumber;
    ULONG IoControlCode;
    LARGE_INTEGER DeviceOffset;
    ULONG DumpData[1];
 
} IO_ERROR_LOG_PACKET, *PIO_ERROR_LOG_PACKET;

Although this structure has three kilograms of members, you won’t need to fill most of them up. Then, the next step is to pass the structure to the IoWriteErrorLogEntry() function and that is it.

VOID 
  IoWriteErrorLogEntry(
    IN PVOID  ElEntry
    );

It is not that easy

So, is it just to allocate the structure, initialize it, call the function of writing and all is done? This question is simply funny. Not long ago, I was in Porto Alegre, and there someone told me that when one explain something starting with “It’s just…”, the actual work is five times bigger than the one explained. Let’s start writing that famous phrase “Hello World” to see the size of the mess!

Well… Where do I put error message? Oh yeah, that’s right, into the message file. Unlikely log files we usually do in applications, the messages are not passed directly to the writing function. Instead, the ErrorCode member of the IO_ERROR_LOG_PACKET structure receives a handle to the message which will be displayed. When the IoWriteErrorLogEntry() function is called, it puts the packet in a list in memory. Later, this package is written to the system events log file. The EventViewer, the software used to view such events, gets the message identifier contained in the package. With this handle, it finds the string that will be stored in a separated module. This module can be your own driver or a DLL. Thus, this string is read and displayed to the user’s module. Phew!

Huh? Who? When? Where? We will rely on the simplest driver example we have to add these features. Let’s start composing the module writing messages to a text file that takes the .mc extension. This text file will be compiled by the Message Compiler. This compilation will generate the resource file containing strings along with the header file that will refer to certain created symbols. See the template message file below:

MessageIdTypedef = NTSTATUS
 
SeverityNames =
(
    Success         = 0x0:STATUS_SEVERITY_SUCCESS
    Informational   = 0x1:STATUS_SEVERITY_INFORMATIONAL
    Warning         = 0x2:STATUS_SEVERITY_WARNING
    Error           = 0x3:STATUS_SEVERITY_ERROR
)
 
FacilityNames =
(
    System          = 0x0
    DriverEntryLogs = 0x2A:DRIVERENTRY_FACILITY_CODE
)
 
LanguageNames =
(
    Portuguese  = 0x0416:msg00001
    English     = 0x0409:msg00002
)
 
MessageId = 0x0001
Facility = DriverEntryLogs
Severity = Informational
SymbolicName = EVT_HELLO_MESSAGE
 
Language = Portuguese
"Ola mundo!"
.
Language = English
"Hello world!"
.

Notice that we have the same message in several languages. What a fancy thing, uh? Save this file with the name of LogMsgs.mc. You can name it of your choice, but remember that this name will be repeated at the generated files. To compile this file, just put it in the list of sources to be compiled, which is in the sources of your project file, as it is shown below. If you decide the messages should be contained within the driver itself, then you need to add the LogMsgs.rc file that will be generated by the Message Compiler.

TARGETNAME=Useless
TARGETPATH=obj
TARGETTYPE=DRIVER
 
SOURCES=LogMsgs.mc\
        Useless.c \
        LogMsgs.rc

Once compiled, the files LogMsgs.rc and LogMsgs.h will be generated. The header file defines the error codes defined at the message file and should be included in the sources files that will make calls for log creation. Watch an excerpt of this file that was generated automagically.

//
// MessageId: EVT_HELLO_MESSAGE
//
// MessageText:
//
//  "Ola mundo!"
//
#define EVT_HELLO_MESSAGE                ((NTSTATUS)0x402A0001L)

Finally we can write the code lines that will use all of this stuff we have created.

#include  
#include "LogMsgs.h"
 
/****
***     DriverEntry 
**
**      That's our driver entry point, everything starts from here,
**      and atfer gets worse and worse...
*/
 
NTSTATUS DriverEntry(IN PDRIVER_OBJECT  pDriverObject,
                     IN PUNICODE_STRING pusRegistryPath)
{
    PIO_ERROR_LOG_PACKET    pLogPacket;
 
    //-f--> Allocates an entry to the event.
    pLogPacket = IoAllocateErrorLogEntry(pDriverObject,
                                         sizeof(IO_ERROR_LOG_PACKET));
 
    //-f--> Inicializes all the structure
    RtlZeroMemory(pLogPacket, sizeof(IO_ERROR_LOG_PACKET));
 
    //-f--> Puts up the desired message
    pLogPacket->ErrorCode = EVT_HELLO_MESSAGE;
 
    //-f--> Sends the entry to the system event list
    IoWriteErrorLogEntry(pLogPacket);
 
    //-f--> Phew, we got to be here; that makes
    //      a success returning code to the system be worthy!
    return STATUS_SUCCESS;
}

Is everything done now?

From the driver viewpoint, it is all already done. The driver can now be compiled, loaded and thus able to create the entry in the log, carrying the message number to be displayed by EventViewer. After executing the driver, we can open the EventViewer; and it is able to find the entry generated by our driver, and then to see the message.

Oops! We still have to tell EventViewer what is the responsible module for storing the strings. In our case, the messages are in the driver itself. To achieve this, we must add a key with the name of our driver in the EventLog registry key. Within this key, we still have to create two values indicating the module path which contains the messages and what kinds of events our driver can generate. Look at carefully the registry path in the figure below.

Once we set the values described above in the registry, just reopen the same event so that the message is displayed as it should be. Remember that the driver does not need to send the event again for the message to be displayed correctly. The driver sends only one entry with the message identifier. This was already done regardless of the message module has been configured. In the figure below, we can see the messages in Portuguese and English, depending on the language of the operating system. What a cute!

As this post is getting bigger than I had expected (for change), I will divide it into parts. I still want to comment on how to use its input parameters to clarify the doubts that some of you may be asking now: “If the log entry is initially in a linked list, and only after a time that is actually going to the disk, can entries be lost if the machine falls in this interval? “.

Is the butler really culprit? Is the Matrix inside another Matrix? Does Tostines sell more because it’s fresh, or is it fresh because it sells more? Do not miss it!

CYA … 🙂

Did you say I don’t like Blue Screens?

Monday, September 3rd, 2007

I’m not saying I love them. Blue screen is a sign that something went wrong but, it is better you see it than a customer calling you saying that he saw one. So we need to do our best to make it make it appear while your driver is under test. I’ve seen programmers escaping the blue screen just trying to hide themselves behind an exception handler. I do not need to say this is not that useful. This way, you just defer to find mistakes that you or someone else will find eventually. Computers that are using your drivers may have a sudden reboot here or there. But even blue screens can happen, it’s all a matter of luck; however, your customers can realize that using your drivers makes them too unlucky. In this post, I’ll give you some tips about how we can see more blue screens in our test drivers.

Going through with F8

Do not take this as “Test Yourself,” but take it as testing all the returning codes that you can. Remember that if something gets wrong, it is not just a MessageBox that will appear, but usually, everything ends up in “blue”, sooner or later. Let’s hope for “sooner”. I often use to say that every code deserves, at least, an F8 walkthrough, which is my Step Into key. Yeah, I know it’s different of the usual one, but when I started using Visual C/C++ 1.52, its keyboard layout was it. But back to the point, once I have made a function that should simply read a file, I went through the code with F8 and received a return code that was not STATUS_SUCCESS, but passed the NT_SUCCESS() macro. The haste induced me to ignore it. It was not an error, it was just a warning. Weeks later, the test team told me that, for any reason, the driver was returning garbage when reading the file. A bit of debugging showed me that same returning code. Only after having loked at its definition in the file ntstatus.h I could understand everything.

//
// MessageId: STATUS_PENDING
//
// MessageText:
//
//  The operation that was requested is pending completion.
//
#define STATUS_PENDING                   ((NTSTATUS)0x00000103L)    // winnt

During debug sessions, the system has enough time to do the I/O that was deferred previously; but, freely running the code, the story can change.

A code that has passed on its F8 test can run in various existing and imaginable environments. Therefore, this first phase is just to remove the grosser errors. After that, it is the test team should beat the victim. There are people who really have a gift to test software. I had worked at a company where the person who used to test the products should have had a personal problem with the software that we produced (or with us, go figure!). I could test the software for days, but when I delivered it for testing, it did not take half an hour for him to call me back saying that phrase that has become his trademark: “Too bad !!!”. There was no explanation. We used to say that his PC had been formatted over an Indian burial ground. No wonder that developer’s testing is so frowned upon.

Using ASSERT

I imagine most of you have already heard from ASSERT macro. This macro is intended to ensure that a certain condition is true. If the condition is false, a blue screen just pops up. Wow, what an incredible macro that helped us very much! Yeah, I know, it is not quite that. Actually, this is the behavior we would have if the debugger was not attached to the system. Then, that macro would be perfect only in Checked. Do I must remind you that  Checked means Debug and Free means Release? Well, already done. If we take a look at its definition, we will see that the obvious thing has already been thought.

#if DBG
 
#define ASSERT( exp ) \
    ((!(exp)) ? \
        (RtlAssert( #exp, __FILE__, __LINE__, NULL ),FALSE) : \
        TRUE)
...
 
#else
 
#define ASSERT( exp )         ((void) 0)
 
...
 
#endif // DBG

But what happens if we have the debugger attached to the system? Good question, this is a really interesting question; I’m glad you have asked it. Has anyone ever told you that you have some knack for programming? Anyway, I have changed one of our examples to enforce this condition.

extern "C"
NTSTATUS DriverEntry(IN PDRIVER_OBJECT pDriverObj,
                     IN PUNICODE_STRING pusRegistryPath)
{
    //-f--> I think I'm sure that this count is right.
    ASSERT(1 + 1 == 3);

If the condition fails and we have a kernel debugger attached to the system, it will display the condition that had failed at the output window and it will request us an answer among the  four alternatives, as it is shown below.

As you can see, the ASSERT is practical, easy and not fattening. It just requires a bit of brain, as well. I’m sying this because we’ve all had bad days, and after 11:00 pm, no programmers should answer for any produced code. Once, it took me some time to figure out why the code below was not working properly. Although everything was working perfectly fine when compiled on Checked, it seemed to me that a certain function simply was not being called in Free. If you take one more looking at the definition of this macro, you will see that the condition disappears when compiled in Free, and in this case, the call too.

    //-f--> Programming is just not enough, thinking is required.
    //      << Don't copy this >>
    ASSERT(DoSomeThing() == STATUS_SUCCESS);

A system in Checked

Wouldn’t it be nice to have an entire operating system full of ASSERTs and tests to detect the slightest sign of trouble and on finding one, the system would present us with a nice blue screen? Well, you’ve probably heard of the Checked Build versions. They are exactly what I have just written. An entire operating system built on Checked. This means that all of ASSERTs that were at the sources have been included in the final binaries and they are checking the system for you. It may seem silly to have to install your driver on one of these systems, but believe me, it is worth. I’ve had drivers that worked very well for months until my manager suggested that they should be tested on Checked versions. From the top of my arrogance, I thought to myself: “I see no reason for that”. Well, we are always learning as we are living. The test machine has not even has started with my drivers. Several blue screens were shown, one after another. Checked Build versions are still able to check for deadlocks. Could you guess what would happen if a spinlock was held for more than a certain time? I bet you do.

Since Checked Build versions are foolproof, I can use them as my default system? Actually you can, but everything is much slower, thousands of checks are being made all the time and the code has no optimization applied. In a fresh installation without installing any additional software, you can find ASSERTs at Internet Explorer or other programs. That’s right, not just the kernel is Checked, but the entire system is as well. You always have to let a kernel debugger attached to your system, because the slightest hint of smoke would be more than an enough reason for one more blue screen.

If my driver has passed by the Checked Build, then is the driver perfect? Sorry to disappoint you. Remember that the performance impact caused by so many tests can hide problems like race conditions. The ideal is to test your drivers in both versions. One setting allows us having only the system image and HAL in Checked, while the rest of the system stays in Free. This may provide you with additional tests in the kernel while the rest of the system runs the lighter version. This link explains how.

OK, is my driver beautiful now?

Actually, this still is only the minimum you should do. Another excellent tool for generating blue screens is the Driver Verifier. This application works in conjunction with the system to track actions from a list of drivers composed by you. From Windows 2000, DriverVerifier comes already installed. Try it now! It doesn’t require any skill or even practice. Type “Verifier” at the Windows “Run…” window and a simple Wizard starts. As I intend to finish this post still alive, I won’t describe how to use this tool step by step, but there are several links on the product page that can help you with this.

OK, now I know you really love Blue Screens!

To get an idea of how importnat a search for blue screens is for a software company, Microsoft promotes IFS Plug Fest in order to test the interoperability of different products that implement File Systems or filters for them. The event is for free, but each company must pay for travel expenses and lodging of its representatives. In these meetings, the professionals around the world gather together to do the tests with each other and they can get in touch and ask possible questions concerning this complex model of development. There are also sponsored seminars that discuss common problems faced by this community. I have not been at one of these, but maybe one day I will be able to.

Stay calm, there are even more

Beyond the mentioned tools, there are some others that I could not forget mentioning; however, I won’t take long on explaining how each of them works.

  • Prefast – A tool that analyses the source code to be compiled by the DDK, looking for usual programming mistakes for Kernel drivers.
  • Static Driver Verifier – That tool makes a binary analysis on the driver file already built. The test takes longer but, it detects more defects than using Prefast.
  • Hardware Compatibility Test – A set of software to apply tests to get drivers certified by Microsoft. It was discontinued a while ago.
  • Driver Test Manager – That’s the new test software used to get your drivers certified.

After so many blue screens, only drinking some coffee can help us!.
CYA!