FreeBSD (v10.2) online debugging with two VMs -- the initial setup

Hello.

Here we talk about how to compile the FreeBSD (fbsd) kernel (v10.2) to enable online debugging. Since I cannot install fbsd natively, I resorted to using two VMs with VMware workstation.

To that end, I added a named pipe as serial port on both the VMs, with one end as the server and the other one as the client. Depending on your host OS, the location of the pipe will differ. If like me you're on windows, the most typical location would be something like \\.pipe\com_1 or if you are on Linux, you could probably select /tmp/socket1 or some such.

For the client, here's how it looks:

Make sure to not "yield the CPU" on the client. After having done that, we need to make sure that our VMs can actually "talk to each other".  Now fbsd, unlike Linux, has two serial interfaces for each physical serial port, a "dial-out" interface and a "dial-in" interface. Assuming that the kernel maps the very first serial line to our named pipe, the dial-out device is /dev/cuau0 and the dial-in device is the familiar /dev/ttyu0.

Now let's fire up our VMs and check if these VMs are indeed connected via the serial port. I used the cu utility to check this. 

On the client, use cu to connect to cuau0 like so: sudo cu -l /dev/cuau0 -s 9600 (baud rate of 9600). It will say "connected". Similarly on the server, connect to ttyu0 like so: sudo cu -l /dev/ttyu0 -s 9600 which will again say "connected". Then type characters on either of them. They will appear on the other end. Note that they will NOT be echoed so you won't see them on the originating end. Make sure you do this with 'sudo' or else it won't be able to create a 'lock' file under /var/spool/lock without which it won't connect.

Once having verified that they are indeed connected, we can prepare the kernel on the server (the debugged VM) to add debugging options to it. You may always refer to the handbook to understand the process of configuring and then compiling the kernel. Here are the various options related to debugging.

So here's what my config file looks like:

include GENERIC
ident 'some-string-to-identify-your-kernel'

# -- added some debug options
options     DDB
options     GDB
options     BREAK_TO_DEBUGGER

So here I include the factory default GENERIC kernel and only modify stuff that I need to modify thereby always referring other options back to the original one. ident allows us to give a 'name' to our kernel which we'll use later to identify the running kernel. This conf file lives in my home directory.

Now we cd to the conf directory of the underlying architecture which, in my case is /usr/src/sys/amd64/conf. We soft-link the above file here. And we go back to /usr/src/ and start the compilation (make sure you are root)

cd /usr/src/
make -j4 (for single CPU) buildkernel -DKERNFAST (if re-compiling) KERNCONF='name-of-the-conf-file'

and while it's working you may go and get some coffee/sleep ;)

It took around 20 mins on my VM to which I had given 256 MB ram, pretty decent IMHO. Now we need to install this kernel at the 'right' place for it to be picked up by the bootloader. The default kernel image picked up the bootloader is /boot/kernel/kernel. So just to be on the safe side, we need to make sure the"working" kernel need to be copied somewhere persistent.

cp /boot/kernel/kernel ~/kernel.original

then

cd /usr/src/
make installkernel KERNCONF='name-of-the-conf-file' | tee ~/installkernel.log


The object files generated by the build are stored inside /usr/obj/usr/src/sys. To know more about the build structure please read build(7). To learn more about the various make options, read make(1) and make.conf(5)

In order to have GDB debug macros work for KGDB remote debugging, one has to 'install' them in the same directory containing the kernel binaries (aka the object files).

Thusly:
cd /usr/obj/usr/src/sys/AIJAZ-DEBUG/
sudo make gdbinit

One can now copy the kernel (called kernel.debug) as the symbol file (called kernel.symbols) over to the client or the development VM, but if one is intending to debug KLD modules as well, preferable to copy the entire directory over to the dev VM. Also while building the kernel module, remember to pass the debug flag while compiling and installing:

Thusly,
make DEBUG_FLAGS=-g3
make install DEBUG_FLAGS=-g3

Or if you already have the exact same source on both the target VM as well as the dev VM, you could repeat the same steps on the dev VM as well.

Before you try to reboot make sure that the UART flags is "0x9" plus you can set the baud rate to 9600 (default). You do it by changing '/boot/device.hints' (read more about it at uart(4)). You want your UART to be able to participate in remote kernel debugging. I typically set it to 0x10 | 0x80 i.e. hint.uart.0.flags="0x90" . This is a VERY IMPORTANT STEP. You won't be able to connect via client KGDB instance to the server GDB stub without this.

After having done that, reboot the server. If you want boot messages to be shown on the console, you also need to do:

echo 'console="comconsole"' >> /boot/loader.conf

Remember, due to the serial nature of the console, boot messages might end up being garbled by the time the console shows it, so you can change the baud rate accordingly.

Now you are ready to reboot your server. After rebooting,you will see the menu like so:

Press 3 to see the loader prompt.

At the loader prompt, type 'boot -d'

If all is well, you will see this:

If GDB debug ports says NULL or not connected, it is mostly because one may have forgotten to change the UART flags, which as we showed above, should be 0x8 or above.

Assuming we are all set, go to the client machine and browse to the directory which has the debug version of the kernel (kernel.debug). Attach KGDB to this image (remember to be root)

sudo kgdb kernel.debug

Then set the baud rate

set remotebaud 9600

After that, go back to the debug prompt on the server and type gdb, The stub is now listening for the remote end to be connected. Now go back to the client and type

target remote /dev/cuau0

Now these VMs are finally connected via the named pipe. On the client end, you can see the portion where the server VM is blocked. In my case, this is what I see:

As we can see, execution stopped in the function 'devstat_remove_entry' in /usr/src/sys/kern/subr_devstat.c:205. The '205' at the end shows the line number. The GDB 'bt' command gives us the backtrace, which in my case is:

(kgdb) bt
#0  0xffffffff80993f0e in devstat_remove_entry (ds=0xffffffff8102e387) at /usr/src/sys/kern/subr_devstat.c:205
#1  0xffffffff80d52b26 in fpugetregs (td=<value optimized out>) at /usr/src/sys/amd64/amd64/fpu.c:721
#2  0xffffffff802e19e4 in btext () at /usr/src/sys/amd64/amd64/locore.S:79
#3  0x0000000000000000 in ?? ()

As we can see we are still in the very early stages of booting. 'Early' here is a relative word, as we are already way pass the earlier three-staged bootloader. 'locore.S' is actually the 'very first code' that gets executed from the kernel per-se. At line 79, it calls 'hammer_time', which is a machine dependent init code for the amd64 architecture. Many of this quirky stuff could be understood later as they lean towards the more complicated side of things and are best explored after one has gotten his/her hands dirty!!

So there we are folks! We've managed to get the setup ready!! You may now issue the GDB continue command (shorthand 'c') and the rest of the kernel can get initialized.

After the VM has completely initialized and you see the login prompt, let's check which modules are loaded

Enter KGDB on the target VM using
[aijazbaig1@aijazb-bsd-server ~/freebsd-device-drivers_code/ch01/echo]$ sudo sysctl debug.kdb.enter=1

This will lock the target VM and return control to the gdb connection on the dev VM

Now on the dev VM
(kgdb) kldstat
During symbol reading, Incomplete CFI data; unspecified registers at 0xffffffff8099497a.
Id Refs Address    Size     Name
 1    6 0x80200000 17e10c8  kernel
 2    1 0x819e2000 4cf0     vmxnet.ko
 3    1 0x81c11000 23dc     vmmemctl.ko

Now release control of the target VM and load a module there
(kgdb)c

On the target VM:

[aijazbaig1@aijazb-bsd-server ~/freebsd-device-drivers_code/ch01/echo]$ sudo kldload ./echo.ko
Echo driver loaded.

Enter KGDB on the target VM using
[aijazbaig1@aijazb-bsd-server ~/freebsd-device-drivers_code/ch01/echo]$ sudo sysctl debug.kdb.enter=1

If it enters the DDB prompt, we need to switch to GDB so that the KGDB backend can connect to it:
db>gdb

Now on the dev VM:
(kgdb) kldstat
During symbol reading, Incomplete CFI data; unspecified registers at 0xffffffff8099497a.
Id Refs Address    Size     Name
 1    8 0x80200000 17e10c8  kernel
 2    1 0x819e2000 4cf0     vmxnet.ko
 3    1 0x81c11000 23dc     vmmemctl.ko
 4    1 0x81c14000 47a      echo.ko

cool!!

Now :
(kgdb) add-kld /boot/kernel/echo.ko
add symbol table from file "/boot/kernel/echo.ko.symbols" at
        .text_addr = 0xffffffff81c14000
        set_modmetadata_set_addr = 0xffffffff81c14228
        set_sysinit_set_addr = 0xffffffff81c14238
        .rodata.str1.1_addr = 0xffffffff81c14240
        .data_addr = 0xffffffff81c142f8
        .bss_addr = 0xffffffff81c14420
(y or n) y
Reading symbols from /boot/kernel/echo.ko.symbols...
location expression too complex...done.

(kgdb) b echo_write
Breakpoint 1 at 0xffffffff81c14161: file echo.c, line 89.
(kgdb) c
Continuing.

On the target VM:
echo "don't panic" > /dev/echo
Opening echo device.

We hit the breakpoint on the dev VM:

[New Thread 100053]
[Switching to Thread 100053]

Breakpoint 1, echo_write (dev=0xfffff80002cb3e00, uio=0xfffffe004e725aa0, ioflag=0x0) at echo.c:89
89          char *buffer = echo_message->buffer;

You can now take it off from here!!!

My very first blog post

Hello People,
So after months of speculation, i've finally decided to give in to the excitement of having a personal blog and a place to share (hopefully) my personal thoughts and ideas with the people out there.

Nothing too special but still a step towards maybe something even better than this..who knows..

and yeah...ive been feeling drowsy now because of all the flu medications that i've taken. So people..i hope to c ya later and i'd surely start posting stuff soon...

regards