What to do if you have problems with Vinum

This document refers to “old” Vinum, the version that I wrote in 1997. Since I wrote it, FreeBSD has changed its I/O structure, breaking many things in Vinum. At the time of writing, a new version, provisionally called gvinum, is being written. The lead developer for this version is Lukas Ertl. Many of the recommendations in this document are also of relevance to gvinum, but I can't claim authority in this case.

I can't reproduce many of the problems that occur with Vinum; otherwise they would never have been seen in the field.

What information to supply

If you need to contact me because of problems with Vinum, please send me a mail message with the following information:

If you don't supply all this information, it will make it much more difficult to perform an analysis of the problem. If there is a good reason why you can't supply the information, contact me anyway, but I may not be able to do very much.

What not to supply

Please don't supply the following information unless I ask for it:

How to supply the information

Most of the information I ask for is relatively short, and the easiest way is to send it as part of a mail message. Please don't wrap the lines, even though they can be quite long. If your mailer mutilates text, I would strongly recommend that you get one which does not. See Using Internet mail for further details. If it's absolutely impossible to get the mailer to send unmutilated messages, please tar up the files and attach them to a mail message with an appropriate attachment type (such as application/octet-stream). If even this proves impossible, please put the tar archive in the directory ftp://ftp.lemis.com/incoming/vinum/ and refer to the name in your mail message.

How to ensure that I ignore your report

I look at Vinum problems out of interest in improving the product. A large number of error reports are due to incorrect usage of the product. That's OK: in many cases, it means that the documentation is inadequate, and I try to do something about it.

But I'm also supplying a service for free, and my time is limited. The least you can do is help make it easy for me. That's the reason for the paragraphs above. Of course, there's another side to the coin: if you want me to ignore your messages, the following will help:

The correct way to send the information is as part of a mail message in text/plain format. If your MUA doesn't handle this well, sorry, that's not my problem.

Even if you do all these things, I may still look at your report. But don't count on it.

Configuration problems

It is relatively easy (too easy) to run into problems with the vinum configuration. If you do, the first thing you should do is stop configuration updates:

# vinum setdaemon 4

This will stop updates and any further corruption of the on-disk configuration.

Next, look at the on-disk configuration, using a Bourne-style shell:

# rm -f log
# for i in /dev/da0s1h /dev/da1s1h /dev/da2s1h /dev/da3s1h; do
(dd if=$i skip=8 count=6|tr -d '\000-\011\200-\377'; echo) >> log

The names of the devices are the names of all vinum slices. The file log should then contain something like this:

IN VINOpanic.lemis.comdrive1}6E7~^K6T^Yfoovolume obj state up
volume src state up
volume raid state down
volume r state down
volume foo state up
plex name obj.p0 state corrupt org concat vol obj
plex name obj.p1 state corrupt org striped 128b vol obj
plex name src.p0 state corrupt org striped 128b vol src
plex name src.p1 state up org concat vol src
plex name raid.p0 state faulty org disorg vol raid
plex name r.p0 state faulty org disorg vol r
plex name foo.p0 state up org concat vol foo
plex name foo.p1 state faulty org concat vol foo
sd name obj.p0.s0 drive drive2 plex obj.p0 state reborn len 409600b driveoffset 265b plexoffset 0b
sd name obj.p0.s1 drive drive4 plex obj.p0 state up len 409600b driveoffset 265b plexoffset 409600b
sd name obj.p1.s0 drive drive1 plex obj.p1 state up len 204800b driveoffset 265b plexoffset 0b
sd name obj.p1.s1 drive drive2 plex obj.p1 state reborn len 204800b driveoffset 409865b plexoffset 128b
sd name obj.p1.s2 drive drive3 plex obj.p1 state up len 204800b driveoffset 265b plexoffset 256b
sd name obj.p1.s3 drive drive4 plex obj.p1 state up len 204800b driveoffset 409865b plexoffset 384b

The first line contains the vinum label and must start with the text IN VINO. It also contains the name of the system. The exact definition is contained in /usr/src/sys/dev/vinum/vinumvar.h. The saved configuration starts in the middle of the line with the text volume obj state up and starts in sector 9 of the disk. The rest of the output shows the remainder of the on-disk configuration. It may be necessary to increase the count argument of dd in order to see the complete configuration.

The configuration on all disks should be the same. If this is not the case, please report the problem with the exact contents of the file log. There is probably little that can be done to recover the on-disk configuration, but if you keep a copy of the files used to create the objects, you should be able to re-create them. The create command does not change the subdisk data, so this will not cause data corruption. You may need to use the resetconfig command if you have this kind of trouble.

Kernel Panics

In order to analyse a panic which you suspect comes from vinum you will need to build a debug kernel. You should also enable dumps. See your local copy of the online handbook (if installed) or the developer's handbook on FreeBSD.org for more details of how to do this.

Perform the following steps to analyse a vinum problem:

$ file /modules/vinum.ko
/modules/vinum.ko: ELF 32-bit LSB shared object, Intel 80386,
version 1 (FreeBSD), not stripped

If the output shows that /modules/vinum.ko is stripped, you will have to find a version which is not. Usually this will be either in /usr/obj/sys/modules/vinum/vinum.ko (if you have built vinum with a make world) or /usr/src/sys/modules/vinum/vinum.ko (if you have built vinum in this directory). Modify the file .gdbinit.vinum.paths accordingly.

# cd /var/crash
# gdb -k kernel.debug vmcore.5

This example assumes that you have installed the correct debug kernel at /var/crash/kernel.debug. If not, substitute the correct name of the debug kernel.

To perform remote serial debugging, link /var/crash/.gdbinit.serial to /var/crash/.gdbinit and enter

# cd /var/crash
# gdb -k kernel.debug

In this case, the .gdbinit file performs the functions necessary to establish connection. The remote machine must already be in debug mode: enter the kernel debugger and select gdb. The serial .gdbinit file expects the serial connection to run at 38400 bits per second; if you run at a different speed, edit the file accordingly (look for the remotebaud specification).

The following example shows a remote debugging session using the debug command of vinum(8):

GDB 4.16 (i386-unknown-freebsd), Copyright 1996 Free Software Foundation, Inc.
(kgdb) f 1          look at the calling frame
Debugger (msg=0xf1093174 "vinum debug") at ../../i386/i386/db_interface.c:318
318                 in_Debugger = 0;
#1  0xf108d9bc in vinumioctl (dev=0x40001900, cmd=0xc008464b, data=0xf6dedee0 "",
    flag=0x3, p=0xf68b7940) at
102             Debugger ("vinum debug");
(kgdb) bt
#0  Debugger (msg=0xf0f661ac "vinum debug") at ../../i386/i386/db_interface.c:318
#1  0xf0f60a7c in vinumioctl (dev=0x40001900, cmd=0xc008464b, data=0xf6923ed0 "",
    flag=0x3, p=0xf688e6c0) at
#2  0xf01833b7 in spec_ioctl (ap=0xf6923e0c) at ../../miscfs/specfs/spec_vnops.c:424
#3  0xf0182cc9 in spec_vnoperate (ap=0xf6923e0c) at ../../miscfs/specfs/spec_vnops.c:129
#4  0xf01eb3c1 in ufs_vnoperatespec (ap=0xf6923e0c) at ../../ufs/ufs/ufs_vnops.c:2312
#5  0xf017dbb1 in vn_ioctl (fp=0xf1007ec0, com=0xc008464b, data=0xf6923ed0 "",
    p=0xf688e6c0) at vnode_if.h:395
#6  0xf015dce0 in ioctl (p=0xf688e6c0, uap=0xf6923f84) at ../../kern/sys_generic.c:473
#7  0xf0214c0b in syscall (frame={tf_es = 0x27, tf_ds = 0x27, tf_edi = 0xefbfcff8,
    tf_esi = 0x1, tf_ebp = 0xefbfcf90, tf_isp = 0xf6923fd4, tf_ebx = 0x2,
    tf_edx = 0x804b614, tf_ecx = 0x8085d10, tf_eax = 0x36, tf_trapno = 0x7,
    tf_err = 0x2, tf_eip = 0x8060a34, tf_cs = 0x1f, tf_eflags = 0x286,
    tf_esp = 0xefbfcf78, tf_ss = 0x27}) at ../../i386/i386/trap.c:1100
#8  0xf020a1fc in Xint0x80_syscall ()
#9  0x804832d in ?? ()
#10 0x80482ad in ?? ()
#11 0x80480e9 in ?? ()

When entering from the debugger, it's important that the source of frame 1 (listed by the f 1 command at the top of the example) contains the text

Debugger ("vinum debug");

This is an indication that the address specifications are correct. If you get some other output, your symbols and the kernel module are out of sync, and the trace will be meaningless.

For an initial investigation, the most important information is the output of the bt (backtrace) command above.