Advanced Search
Apple Developer Connection
Member Login Log In | Not a Member? Contact ADC


Hello Debugger: Debugging a Device Driver With GDB

This tutorial describes how to prepare to debug a device driver for Darwin/Mac OS X. You will learn how to set up a two-machine debugging environment and how to start using GDB, a command-line debugger, to perform remote debugging.

The tutorial assumes that you are working in a Mac OS X development environment.

Although this tutorial is written with a device driver as the example, the steps for debugging are similar for debugging any type of kernel extension (KEXT). If you wish, you can substitute your own code for the example. Note, however, that you may encounter a few inconsistencies. For example, examples of GDB commands may be dependent on the underlying source code language—I/O Kit extensions (drivers) use C++; the GDB commands for C may differ.

In this section:

Preparation
Roadmap
On Target Machine, Enable Kernel Debugging
On Development Machine, Set Up a Permanent Network Connection
On Target Machine, Create a Symbol File
On Development Machine, Import the Symbol File
On Development Machine, Start GDB
On Target Machine, Break into Kernel Debugging Mode
On Development Machine, Attach to the Target Machine
On Development Machine, Set Breakpoints in GDB
On Target Machine, Start Running the Device Driver
On Development Machine, Control the Driver With GDB
Stop the Debugger
On Target Machine, Unload the Driver
Where to Go Next


Preparation

Before you begin this tutorial, make sure you have met the following requirements.

  1. Be sure you understand the material presented in the KEXT and driver development tutorials.

    The tutorials "Hello Kernel: Creating a Kernel Extension With Xcode" and "Hello I/O Kit: Creating a Device Driver With Xcode" describe how to create a kernel extension's project, correct code errors, and how to create a simple I/O Kit device driver. This tutorial uses the example from “HelloIOKit”. Be sure that you have completed these tutorials (or are familiar with KEXT or driver development in Mac OS X) before beginning this one.

  2. For remote debugging, you need two machines. On the development machine, you create, build, and, for this tutorial, debug your driver. On the target machine, you load and run the driver.

    Before you begin this tutorial, you should prepare the two machines. Note that:

  3. Transfer a copy of your KEXT to the target machine.

    Before you can begin to debug your driver (and each time you rebuild it), you must transfer a copy of your KEXT to the target machine. You can do this in any of several ways. For example:

    You can also automate this process using ssh. This tutorial does not tell you how to do this, but you can find resources on the web that describe the process.

Roadmap

This tutorial has many steps. It is important to keep the two machines clear in your mind as you work through the steps. It may help if you take a piece of paper, tear it in half, and write “Development” on one piece and “Target” on the other. Then place the pieces of paper next to the two machines.

You will be moving back and forth between the two machines many times. Figure 1 illustrates the steps you will take for each machine.


Figure 1  Steps to set up a debugging environment

Figure 1 Steps to set up a debugging environment

After reading "Preparation", you should have already prepared the two machines and attached them to the network. The next two steps will enable kernel debugging and set up a permanent network connection between the two machines.

  1. "On Target Machine, Enable Kernel Debugging"

  2. "On Development Machine, Set Up a Permanent Network Connection"

Once these steps are completed, you will not need to repeat them, even if you need to start over part way through the tutorial. The next set of steps prepare the driver for debugging, set up GDB, and attach the two machines to begin debugging your driver. If you need to start over part way through the tutorial, you should repeat all of these steps from the beginning.

  1. "On Target Machine, Create a Symbol File"

  2. "On Development Machine, Import the Symbol File"

  3. "On Development Machine, Start GDB "

  4. "On Target Machine, Break into Kernel Debugging Mode"

  5. "On Development Machine, Attach to the Target Machine"

  6. "On Target Machine, Start Running the Device Driver"

  7. "On Development Machine, Control the Driver With GDB"

The last two steps stop the debugger and unload the driver. If you need to start over part way through, you may need to skip ahead to these steps to reset the environment completely, then start again at "On Target Machine, Create a Symbol File".

  1. "Stop the Debugger"

  2. "On Target Machine, Unload the Driver"

On Target Machine, Enable Kernel Debugging

In this step and the next, you will set up the two-machine environment. These two steps will enable kernel debugging and set up a permanent network connection between the two machines.

By default, Darwin/Mac OS X does not let you debug the kernel. Before you can debug your driver, you must first enable kernel debugging. On the target machine, do the following:

  1. Start the Terminal application. From a Finder window, locate and launch the Terminal application, found at /Applications/Utilities/Terminal.

  2. Set the kernel debug flags.

    To enable kernel debugging, you will be setting an NVRAM (non-volatile random access memory) variable. To do this, you must use the sudo command which gives permitted users the ability to execute a given command as root. When sudo prompts you for a password, enter your admin password. (Note that nothing is displayed as you type the password.)

    The sudo command will allow you to execute commands with root privileges without re-entering your password for a few minutes (usually 5) but after that time is up, it will ask you for your password again. This tutorial does not show the password prompt every time the sudo command is used.

    If you're using a beige Power Macintosh G3 and you’re running a version of Mac OS X prior to 10.1, enter this command:

    $ sudo nvram boot-command="0 bootr debug=0x14e"

    If your target machine is a PowerPC-based Macintosh model not listed above or an Intel-based Macintosh, enter this command:

    $ sudo nvram boot-args="debug=0x14e"
    Password:

    For more information on debugging flags, see Building and Debugging Kernels in Kernel Programming Guide.

  3. Restart the computer.

    The computer restarts and displays the login screen.

On Development Machine, Set Up a Permanent Network Connection

Your development and target machines must be continuously connected by a reliable network connection. In this section, you will create such a connection.

Note: If your target machine is running Mac OS X 10.3 or later and is configured with the 0x40 debug flag set (DB_ARP), you can skip this section.

After the target machine has restarted, do the following steps on the development machine:

  1. Start the Terminal application. From a Finder window, locate and launch the Terminal application, found at /Applications/Utilities/Terminal.

  2. Make sure your development machine can connect to your target machine.

    Use the ping command with your target machine’s hostname or IP address. For example:

    $ ping -c 1 target.apple.com

    or

    $ ping -c 1 192.102.207.119

    This command makes sure your development machine can reach your target machine and creates a temporary connection between them. The -c 1 option tells ping to stop after sending (and receiving) one ICMP (Internet Control Message Protocol) packet.

    You should see output similar to this:

    PING target.my-company.com (192.102.207.119): 56 data bytes 
    64 bytes from 192.102.207.119: icmp_seq=0 ttl=255 time=2.099 ms
     
    --- target.apple.com ping statistics ---
    1 packets transmitted, 1 packets received, 0% packet loss
    round-trip min/avg/max = 2.099/2.099/2.099 ms
  3. If you don't already know it, determine the hardware Ethernet address of the target machine.

    Enter this command:

    $ arp -a

    The arp command with the -a option lists the static hardware Ethernet addresses of all machines your development machine has recently accessed. Make a note of the entry for your target machine's address. Because this address is stored in the hardware, you should keep track of this number for future debugging sessions (or for debugging possible kernel panics).

    For example, you should see output similar to this:

    aniil33 (1192.102.207.102) at 0:0:f:0:86:c3
    target (192.102.207.119) at 0:5:2:b0:b3:20
    dev (192.102.207.124) at 0:5:2:59:2f:20

    In this example, the hardware Ethernet address to remember is 0:5:2:b0:b3:20.

  4. Remove the current, temporary, connection between your development and target machines. Use the arp command with your target machine’s hostname. Because you’ll be using the -d option to delete the entry for the target machine, this command must be executed as root, so use the sudo command.

    For example:

    $ sudo arp -d target.apple.com

    You should see output like this:

    target.apple.com (192.102.207.119) deleted
  5. Create a new, permanent, connection between your development and target machines. Use the arp command with the target machine’s hostname and Ethernet address. The arp command with the -s option must be executed as root, so use the sudo command.

    For example:

    $ sudo arp -s target.apple.com 0:5:2:b0:b3:20

    You can copy and paste the Ethernet address from the previous output in the Terminal window.

  6. Make sure the connection was made correctly.

    Enter this command:

    $ arp -a

    Make sure the entry for your target machine now contains the word “permanent.”

    You should see output similar to this:

     
    niil33 (192.102.207.102) at 0:0:f:0:86:c3
    target (192.102.207.119) at 0:5:2:b0:b3:20 permanent
    dev (192.102.207.124) at 0:5:2:59:2f:20

On Target Machine, Create a Symbol File

The previous steps prepared the two machines to communicate with each other. The next steps will load the driver, prepare and transfer a symbol file, and start the debugger.

!

Warning: If you make a mistake in any of the following steps, or need to retrace any steps, you should begin again at this step.

On the target machine, do the following:

  1. From the login screen, log in as console.

    Type >console as the user name, leave the password blank, and press Return. Be sure to include the > character at the beginning of the name. The screen turns black and looks like an old ASCII “glass terminal”. This is console mode.

  2. At the prompt, log in as your admin account.

    For example:

    Darwin/BSD (dev) (console)
     
    login: admin
    Password for admin: 
    admin$ 
  3. Copy your KEXT to /tmp.

    !

    Warning: If you use tab-completion to avoid typing all of HelloIOKit.kext, you’ll get a “/” after the KEXT name. Be sure to delete this slash before you press Return. If you don’t, the cp command will copy only the contents of the HelloIOKit.kext directory to /tmp, instead of copying the directory and its entire subtree.

    Because you will be loading your KEXT into the kernel, your KEXT must be owned by the user root and the group wheel and the folders and files of the KEXT bundle must have their permissions set so that they are not writable by any user other than the super user. If you haven't already done so, use the sudo command to copy the KEXT binary (HelloIOKit.kext) to the /tmp directory so you can load and unload the KEXT from there. Using sudo to copy the KEXT binary gives the KEXT the super user's ownership and permissions and leaves the original KEXT alone so you can revise and save it as you choose.

    Note: Every time you make changes to your KEXT and rebuild it, you need to repeat this step to copy the new version to the /tmp directory to load and debug it.

    For example:

    $ sudo cp -R HelloIOKit.kext /tmp
  4. For debugging purposes, you first use kextload with some options to load the driver and create a symbol file for it, but not start the driver. You'll register the driver and start it running (again using kextload) in a later step, "On Target Machine, Start Running the Device Driver". The kextload command must be run as root, so you must also use the sudo command.

    !

    Warning: In the KEXT development tutorials, "Hello Kernel: Creating a Kernel Extension With Xcode" and "Hello I/O Kit: Creating a Device Driver With Xcode", you used kextload to load your KEXT and start it running all in one step. Now you will break this step in two, using different options to kextload each time.

    Load the driver (without starting it) and create a symbol file for it.

    For example, enter the command:

    $ sudo kextload -ls /tmp HelloIOKit.kext

    The -l (this is a lower-case “L”) option tells kextload to load the KEXT's code into the kernel but not to start I/O Kit matching, which would start the KEXT running. The -s option tells kextload to create symbol files in /tmp for the KEXT and any of its dependencies. To ensure uniqueness the symbol files are named after each KEXT's CFBundleIdentifier, plus a .sym extension. You may want to rename symbol files to be shorter.

    The symbol file contains information that GDB needs to debug your driver. You must create the symbol file for a loaded driver, then copy the file to the development machine where it will be used. Note that you must recreate the symbol file again if you unload (and reload) the driver. This is because the KEXT may be loaded at a different address each time and the symbol files contain the actual virtual address of each symbol.

  5. Optionally, you can rename the symbol file.

    For example:

    $ mv /tmp/com.MyTutorial.driver.HelloIOKit.sym /tmp/HelloIOKit.sym

On Development Machine, Import the Symbol File

On the development machine, import the symbol file from the target machine, for use with GDB. Use the ftp (file transfer protocol) command. A sample ftp session is shown below. Enter the ftp commands as shown.

!

Warning: Be sure to use the actual name or IP address of the target machine; in this example it is target.apple.com. Be sure to log in with your actual account name; this example uses admin.

$ ftp target.apple.com
Connected to target.apple.com.
220 target.apple.com FTP server (Version 6.00) ready. 
Name (dev:admin): admin
331 Password required for admin.
Password:
 
230- Welcome to Darwin!
230 User admin logged in.
Remote system type is BSD.
ftp> bin
200 Type set to I.
ftp> get /tmp/com.MyTutorial.driver.HelloIOKit.sym
local: /tmp/com.MyTutorial.driver.HelloIOKit.sym remote: /tmp/com.MyTutorial.driver.HelloIOKit.sym
200 PORT command successful.
150 Opening BINARY mode data connection for '/tmp/com.MyTutorial.driver.HelloIOKit.sym' 
226 Transfer complete.
180356 bytes received in 0.15 seconds (1.15 MB/s)
ftp> bye

Note that you must recreate (and import) the symbol file again if you unload (and reload) the driver on the target machine. You should also note that any files in the /tmp directory are temporary and will be deleted whenever Darwin/Mac OS X is restarted.

On Development Machine, Start GDB

Now you can start GDB. On the development machine, do the following from the Terminal application:

  1. Start the debugger on the kernel.

    Enter this command:

    $ gdb -arch ppc /mach_kernel

    Substitute -arch i386 if your debug target is an Intel-based Macintosh. You should see output like this:

    GNU gdb 4.18-20000320 (Apple version gdb-172)
    Copyright 1998 Free Software Foundation, Inc.
    ...
    (gdb)
  2. Load the symbol file you created and transferred.

    At the GDB prompt, use the add-symbol-file command with the name of the symbol file you’ve created.

    For example:

    (gdb) add-symbol-file
    /tmp/com.MyTutorial.driver.HelloIOKit.sym
  3. Tell the debugger that you're remotely debugging a device driver.

    At the GDB prompt, enter this command:

    (gdb) target remote-kdp

On Target Machine, Break into Kernel Debugging Mode

Before you can connect GDB to the target machine, you must break into kernel debugging mode. There are four ways to break into kernel debugging mode.

Note: Unless you’re in Console mode, it may be difficult to tell when the kernel stops and the target machine is in kernel debugging mode. One way to tell is to enable the seconds display in the menu bar clock. When the seconds stop counting, the kernel is stopped. To enable the seconds display, open System Preferences and click Date & Time. Select the Clock tab and check the box next to Display the time with seconds.

On the target machine, do the following:

  1. Break into the debugger. Briefly press the power button on the machine.

  2. The kernel stops and the machine waits for a remote debugger connection.

    If you’re in Console mode, you’ll see the following message on the screen:

    ethernet MAC address: 0:5:2:b0:b3:20
    ip address: 192.102.207.119
  3. If the seconds don’t stop counting (or if you don't see the “Waiting...” message in Console mode), briefly press the power button again.

  4. Immediately move to the development machine and attach to the target machine from GDB.

On Development Machine, Attach to the Target Machine

Now you can tell GDB to attach to the target machine. On the development machine, do the following:

  1. Attach to the target machine. At the GDB prompt, use the attach command with the target machine’s name or IP address.

    For example:

    (gdb) attach target.apple.com
  2. The target machine prints:

    Connected to remote debugger.

    Note that the target machine is currently unable to accept keyboard input.

On Development Machine, Set Breakpoints in GDB

This is the best place to set breakpoints in your code, before you actually run the driver on the target machine. For this tutorial, set a breakpoint at the start method.

  1. In GDB, set a breakpoint in the start method.

    For example:

    (gdb) break 'com_MyTutorial_driver_HelloIOKit::start(IOService
    *)'
    Breakpoint 1 at 0x53d93d0: file HelloIOKit.cpp, line 54.

    Hint: You don't need to type the entire class name for the breakpoint; GDB can do name completion. Type a single quote, ', then the first part of the name, then press the tab key. If GDB does not complete the name, you may need to type a few more characters to allow it to choose an unambiguous match. For example, type

    (gdb) break 'com_<TAB>

    When you press the tab key, GDB will complete the name as far as it can, unambiguously. If it cannot complete the entire name of the method, you will need to give it more clues. For example, all of the methods in this tutorial begin with com_MyTutorial_driver_HelloIOKit. When GDB has completed that much, it will stop. Then you can type

    ::start<TAB>

    GDB will finish the entry. After GDB completes the method name (with a final closing single quote), be sure to press return.

  2. Allow the target machine to continue. On the development machine, enter the continue command at the GDB prompt. For example:

    (gdb) continue
  3. The target machine is again able to accept keyboard input. Now you can run the driver.

On Target Machine, Start Running the Device Driver

Now that you have GDB attached and breakpoints set, you can start the driver running. On the target machine, do the following.

  1. If you are not already there, move to the /tmp directory, using the cd command. For example

    $ cd /tmp
  2. Start the driver running, using the kextload command. Recall that you previously loaded the driver using kextload with the -l option, which does not start I/O Kit matching for the driver, meaning that the code doesn't start running. You must now use kextload with the -m option to finish the process and run the driver. For example:

    $ sudo kextload -m HelloIOKit.kext
  3. The driver executes until it hits a breakpoint.

On Development Machine, Control the Driver With GDB

Now that GDB is running on the development machine, the target machine is attached to the debugger, and the driver is running, you can actually start debugging!

The driver executes on the target machine until it hits a breakpoint. When this happens, GDB will print some status on the development machine. For example:

Breakpoint 1, com_MyTutorial_driver_HelloIOKit::start (this=0xcdcfc0, dict=0xbcfe40) at HelloIOKit.cpp:54
54 HelloIOKit.cpp: No such file or directory.
Current language: auto; currently c++

Now you can debug your driver (almost) as you would any other executable. However, because driver debugging happens at such a low level, you won't be able to take advantage of all of GDB’s features. For example:

For help with GDB, use its help command. Most GDB commands have shortcuts; for example, you can type c instead of continue.

Here are a few things you can do. From GDB, try the following:

Stop the Debugger

When you've finished debugging, you should stop the debugger and then unload the driver. Do the following:

  1. On the target machine, break into the kernel debugger. Briefly press the power button as described in "On Target Machine, Break into Kernel Debugging Mode".

    If you’re in Console mode, you’ll see the following message on the screen:

    Debugger(Programmer Key)
  2. On the development machine, enter this command at the GDB prompt:

    (gdb) quit
  3. Answer yes to the prompt that appears:

    The program is
    running. Exit anyway? (y or n)
  4. The target machine prints

    Remote debugger
    disconnected

The debugging session ends.

Note: If your target machine contains a PMU (for example, a PowerBook G4 or an early G5 desktop computer), you may find that it shuts down when you exit kernel debugging mode. One reason for this is that if a breakpoint causes kernel debugger entry in the middle of a PMU transaction, the PMU watchdog may trigger on a timeout and cause the machine to shut down. If you experience this, you may find that forcing the PMU driver to operate in polled mode fixes the problem. To do this, set the NVRAM variable pmuflags to 1 when you enable kernel debugging (see “On Target Machine, Enable Kernel Debugging”), as shown below:

$ sudo nvram boot-args="pmuflags=1"

You can set the pmuflags variable separately, as shown above, or you can set it at the same time you set the debug variable, as shown below:

$ sudo nvram boot-args="debug=0x14e pmuflags=1"

On Target Machine, Unload the Driver

On the target machine, unload the driver. Use the kextunload command. This command unloads your driver by running its termination (stop) function. For example:

$ sudo kextunload HelloIOKit.kext
IOCatalogueTerminate(Module com.MyTutorial.driver.HelloIOKit) [0]
done.

Where to Go Next

Congratulations! You've now learned how to set up a two-machine debugging environment to debug a driver (or another type of KEXT) using GDB.

If you're interested, you can use the man command to read the manual pages for kextload, kextstat, and kextunload. In particular, you may want to read about the many ways to use kextload to get debug symbols for all types of KEXTs, including I/O Kit drivers. For example, from a Terminal window, enter the command

$ man kextload

You may also want to familiarize yourself with GDB. Further information is available in Tools Documentation.





Last updated: 2006-10-03




Did this document help you?
Yes: Tell us what works for you.

It’s good, but: Report typos, inaccuracies, and so forth.

It wasn’t helpful: Tell us what would have helped.
Get information on Apple products.
Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Copyright © 2006 Apple Computer, Inc.
All rights reserved. | Terms of use | Privacy Notice