/dev/mapper/<dm_name> symlink not being created

/dev/mapper/<dm_name> symlink not being created

[Nicholas Nevin]

[-- Attachment #1.1: Type: text/plain, Size: 1638 bytes --]

In nightly testing of our product the tests do a fair bit of
creating/destroying of device mapper devices and we have been seeing
infrequent cases where the device is created OK but the corresponding
/dev/mapper/<dm_name> symlink does not exist.

Here is an example of the dmsetup command used to create the devices.

# dmsetup create tap0 --table '0 47247164 linear /dev/xen/blktap-2/tapdev0
1'

I managed to reproduce this problem with udev logging enabled and 'udevadm
monitor' running and from the logs the cause appears to be that the udev
add action (from the device create) and the udev change action (from when
the device is resumed after loading the table) are running
concurrently. The processing of the rules for the change action creates the
/dev/mapper/tap0 symlink but then the processing of the add action deletes
the symlink because /dev/mapper/tap0 is not in the add action DEVLINKS.

The question I have is should dmsetup be ensuring that the add action
completes before the change action is handled or is this behaviour expected?

That /dev/mapper/tap0 is not in the DEVLINKS for the add action is AFAICT
because of a race between the creation of the sysfs entries for the device
and the processing of the add action in udevd. This can be fixed by
sprinkling in some WAIT_FOR conditions in the dm udev rules. Our system is
based on Ubuntu 12.04.

I have attached the udevadm monitor output and the relevant piece of syslog
for a test case where the /dev/mapper/tap0 symlink wasn't present when
dmsetup returned.

I hope this is the right place to ask this. If not please point me in the
right direction.

Thanks, Nick.

[-- Attachment #1.2: Type: text/html, Size: 1902 bytes --]

[-- Attachment #2: syslog --]
[-- Type: application/octet-stream, Size: 7993 bytes --]

Apr 27 18:53:46 HP6535B udevd[410]: seq 80951 queued, 'add' 'bdi' 
Apr 27 18:53:46 HP6535B udevd[410]: passed 153 bytes to netlink monitor 0x7ff22301f2d0 
Apr 27 18:53:46 HP6535B udevd[1486]: seq 80951 running 
Apr 27 18:53:46 HP6535B udevd[1486]: device 0x7ff22302dc10 has devpath '/devices/virtual/bdi/251:7' 
Apr 27 18:53:46 HP6535B udevd[410]: seq 80952 queued, 'add' 'block' 
Apr 27 18:53:46 HP6535B udevd[410]: passed 200 bytes to netlink monitor 0x7ff22301f2d0 
Apr 27 18:53:46 HP6535B udevd[1486]: no db file to read /run/udev/data/+bdi:251:7: No such file or directory 
Apr 27 18:53:46 HP6535B udevd[1556]: seq 80952 running 
Apr 27 18:53:46 HP6535B udevd[1556]: device 0x7ff22302d130 has devpath '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: passed -1 bytes to netlink monitor 0x7ff22302f030 
Apr 27 18:53:46 HP6535B udevd[1486]: seq 80951 processed with 0 
Apr 27 18:53:46 HP6535B udevd[1556]: device 0x7ff22302d130 filled with db file data 
Apr 27 18:53:46 HP6535B udevd[1556]: GROUP 6 /lib/udev/rules.d/50-udev-default.rules:67 
Apr 27 18:53:46 HP6535B udevd[1556]: IMPORT '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o name,uuid,suspended' /lib/udev/rules.d/55-dm.rules:69 
Apr 27 18:53:46 HP6535B udevd[410]: seq 80951 done with 0 
Apr 27 18:53:46 HP6535B udevd[8755]: starting '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o name,uuid,suspended' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o name,uuid,suspended'(err) 'Device does not exist.' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o name,uuid,suspended'(err) 'Command failed' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o name,uuid,suspended' [8755] exit with return code 1 
Apr 27 18:53:46 HP6535B udevd[1556]: IMPORT '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o suspended' /lib/udev/rules.d/55-dm.rules:70 
Apr 27 18:53:46 HP6535B udevd[8757]: starting '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o suspended' 
Apr 27 18:53:46 HP6535B udevd[410]: seq 80953 queued, 'change' 'block' 
Apr 27 18:53:46 HP6535B udevd[1486]: seq 80953 running 
Apr 27 18:53:46 HP6535B udevd[1486]: device 0x7ff22302eac0 has devpath '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: device 0x7ff22302eac0 filled with db file data 
Apr 27 18:53:46 HP6535B udevd[1486]: GROUP 6 /lib/udev/rules.d/50-udev-default.rules:67 
Apr 27 18:53:46 HP6535B udevd[1486]: IMPORT '/sbin/dmsetup udevflags 4223486' /lib/udev/rules.d/55-dm.rules:20 
Apr 27 18:53:46 HP6535B udevd[410]: passed 221 bytes to netlink monitor 0x7ff22301f2d0 
Apr 27 18:53:46 HP6535B udevd[8758]: starting '/sbin/dmsetup udevflags 4223486' 
Apr 27 18:53:46 HP6535B udevd[1486]: '/sbin/dmsetup udevflags 4223486'(out) 'DM_UDEV_PRIMARY_SOURCE_FLAG='1'' 
Apr 27 18:53:46 HP6535B udevd[1486]: '/sbin/dmsetup udevflags 4223486' [8758] exit with return code 0 
Apr 27 18:53:46 HP6535B udevd[1486]: RUN '/sbin/dmsetup udevcomplete $env{DM_COOKIE}' /lib/udev/rules.d/55-dm.rules:21 
Apr 27 18:53:46 HP6535B udevd[1486]: LINK 'mapper/tap1' /lib/udev/rules.d/55-dm.rules:80 
Apr 27 18:53:46 HP6535B udevd[1486]: LINK 'disk/by-id/dm-name-tap1' /etc/udev/rules.d/60-persistent-storage-dm.rules:9 
Apr 27 18:53:46 HP6535B udevd[1486]: IMPORT '/sbin/blkid -o udev -p /dev/dm-7' /etc/udev/rules.d/60-persistent-storage-dm.rules:18 
Apr 27 18:53:46 HP6535B udevd[8759]: starting '/sbin/blkid -o udev -p /dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o suspended'(out) 'DM_SUSPENDED='Active'' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/dmsetup info -j 251 -m 7 -c --nameprefixes --noheadings --rows -o suspended' [8757] exit with return code 0 
Apr 27 18:53:46 HP6535B udevd[1556]: LINK 'disk/by-id/dm-name-' /etc/udev/rules.d/60-persistent-storage-dm.rules:9 
Apr 27 18:53:46 HP6535B udevd[1556]: IMPORT '/sbin/blkid -o udev -p /dev/dm-7' /etc/udev/rules.d/60-persistent-storage-dm.rules:18 
Apr 27 18:53:46 HP6535B udevd[8760]: starting '/sbin/blkid -o udev -p /dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: '/sbin/blkid -o udev -p /dev/dm-7' [8759] exit with return code 2 
Apr 27 18:53:46 HP6535B udevd[1486]: no node name set, will use kernel supplied name 'dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: creating device node '/dev/dm-7', devnum=251:7, mode=0660, uid=0, gid=6 
Apr 27 18:53:46 HP6535B udevd[1486]: preserve file '/dev/dm-7', because it has correct dev_t 
Apr 27 18:53:46 HP6535B udevd[1486]: set permissions /dev/dm-7, 060660, uid=0, gid=6 
Apr 27 18:53:46 HP6535B udevd[1486]: creating symlink '/dev/block/251:7' to '../dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: creating link '/dev/disk/by-id/dm-name-tap1' to '/dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: creating symlink '/dev/disk/by-id/dm-name-tap1' to '../../dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: creating link '/dev/mapper/tap1' to '/dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: creating symlink '/dev/mapper/tap1' to '../dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: created db file '/run/udev/data/b251:7' for '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[8761]: starting '/sbin/dmsetup udevcomplete 4223486' 
Apr 27 18:53:46 HP6535B udevd[1556]: '/sbin/blkid -o udev -p /dev/dm-7' [8760] exit with return code 2 
Apr 27 18:53:46 HP6535B udevd[1556]: no node name set, will use kernel supplied name 'dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: update old name, '/dev/disk/by-id/dm-name-tap1' no longer belonging to '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: no reference left, remove '/dev/disk/by-id/dm-name-tap1' 
Apr 27 18:53:46 HP6535B udevd[1556]: update old name, '/dev/mapper/tap1' no longer belonging to '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: no reference left, remove '/dev/mapper/tap1' 
Apr 27 18:53:46 HP6535B udevd[1556]: creating device node '/dev/dm-7', devnum=251:7, mode=0660, uid=0, gid=6 
Apr 27 18:53:46 HP6535B udevd[1556]: preserve file '/dev/dm-7', because it has correct dev_t 
Apr 27 18:53:46 HP6535B udevd[1556]: preserve permissions /dev/dm-7, 060660, uid=0, gid=6 
Apr 27 18:53:46 HP6535B udevd[1556]: preserve already existing symlink '/dev/block/251:7' to '../dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: found 'b251:7' claiming '/run/udev/links/disk\x2fby-id\x2fdm-name-' 
Apr 27 18:53:46 HP6535B udevd[1556]: found 'b251:8' claiming '/run/udev/links/disk\x2fby-id\x2fdm-name-' 
Apr 27 18:53:46 HP6535B udevd[1556]: found 'b251:9' claiming '/run/udev/links/disk\x2fby-id\x2fdm-name-' 
Apr 27 18:53:46 HP6535B udevd[1556]: creating link '/dev/disk/by-id/dm-name-' to '/dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: preserve already existing symlink '/dev/disk/by-id/dm-name-' to '../../dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: created db file '/run/udev/data/b251:7' for '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: adding watch on '/dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: created db file '/run/udev/data/b251:7' for '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1556]: passed -1 bytes to netlink monitor 0x7ff22301ffc0 
Apr 27 18:53:46 HP6535B udevd[1556]: seq 80952 processed with 0 
Apr 27 18:53:46 HP6535B udevd[410]: seq 80952 done with 0 
Apr 27 18:53:46 HP6535B udevd[1486]: '/sbin/dmsetup udevcomplete 4223486' [8761] exit with return code 0 
Apr 27 18:53:46 HP6535B udevd[1486]: adding watch on '/dev/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: created db file '/run/udev/data/b251:7' for '/devices/virtual/block/dm-7' 
Apr 27 18:53:46 HP6535B udevd[1486]: passed -1 bytes to netlink monitor 0x7ff22302f030 
Apr 27 18:53:46 HP6535B udevd[1486]: seq 80953 processed with 0 
Apr 27 18:53:46 HP6535B udevd[410]: seq 80953 done with 0 

[-- Attachment #3: monitor.out --]
[-- Type: application/octet-stream, Size: 1488 bytes --]

KERNEL[79669.539249] add      /devices/virtual/bdi/251:7 (bdi)
ACTION=add
DEVPATH=/devices/virtual/bdi/251:7
SEQNUM=80951
SUBSYSTEM=bdi
UDEV_LOG=3

KERNEL[79669.539609] add      /devices/virtual/block/dm-7 (block)
ACTION=add
DEVNAME=dm-7
DEVPATH=/devices/virtual/block/dm-7
DEVTYPE=disk
MAJOR=251
MINOR=7
SEQNUM=80952
SUBSYSTEM=block
UDEV_LOG=3

UDEV  [79669.541404] add      /devices/virtual/bdi/251:7 (bdi)
ACTION=add
DEVPATH=/devices/virtual/bdi/251:7
SEQNUM=80951
SUBSYSTEM=bdi
UDEV_LOG=7
USEC_INITIALIZED=79669540071

KERNEL[79669.548290] change   /devices/virtual/block/dm-7 (block)
ACTION=change
DEVNAME=dm-7
DEVPATH=/devices/virtual/block/dm-7
DEVTYPE=disk
DM_COOKIE=4223486
MAJOR=251
MINOR=7
SEQNUM=80953
SUBSYSTEM=block
UDEV_LOG=3

UDEV  [79669.699756] add      /devices/virtual/block/dm-7 (block)
ACTION=add
DEVLINKS=/dev/disk/by-id/dm-name-
DEVNAME=/dev/dm-7
DEVPATH=/devices/virtual/block/dm-7
DEVTYPE=disk
DM_SUSPENDED=0
DM_UDEV_RULES=1
MAJOR=251
MINOR=7
SEQNUM=80952
SUBSYSTEM=block
UDEV_LOG=7
USEC_INITIALIZED=1092997836

UDEV  [79669.701727] change   /devices/virtual/block/dm-7 (block)
ACTION=change
DEVLINKS=/dev/disk/by-id/dm-name-tap1 /dev/mapper/tap1
DEVNAME=/dev/dm-7
DEVPATH=/devices/virtual/block/dm-7
DEVTYPE=disk
DM_COOKIE=4223486
DM_NAME=tap1
DM_SUSPENDED=0
DM_UDEV_PRIMARY_SOURCE_FLAG=1
DM_UDEV_RULES=1
MAJOR=251
MINOR=7
SEQNUM=80953
SUBSYSTEM=block
UDEV_LOG=7
USEC_INITIALIZED=1092997836

[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

Re: /dev/mapper/<dm_name> symlink not being created

[Nicholas Nevin]

[-- Attachment #1.1: Type: text/plain, Size: 1830 bytes --]

Anyone have any thoughts on this?

Thanks, Nick.

On Mon, Apr 30, 2012 at 8:08 AM, Nicholas Nevin <njnevin@gmail.com> wrote:

> In nightly testing of our product the tests do a fair bit of
> creating/destroying of device mapper devices and we have been seeing
> infrequent cases where the device is created OK but the corresponding
> /dev/mapper/<dm_name> symlink does not exist.
>
> Here is an example of the dmsetup command used to create the devices.
>
> # dmsetup create tap0 --table '0 47247164 linear /dev/xen/blktap-2/tapdev0
> 1'
>
> I managed to reproduce this problem with udev logging enabled and 'udevadm
> monitor' running and from the logs the cause appears to be that the udev
> add action (from the device create) and the udev change action (from when
> the device is resumed after loading the table) are running
> concurrently. The processing of the rules for the change action creates the
> /dev/mapper/tap0 symlink but then the processing of the add action deletes
> the symlink because /dev/mapper/tap0 is not in the add action DEVLINKS.
>
> The question I have is should dmsetup be ensuring that the add action
> completes before the change action is handled or is this behaviour expected?
>
> That /dev/mapper/tap0 is not in the DEVLINKS for the add action is AFAICT
> because of a race between the creation of the sysfs entries for the device
> and the processing of the add action in udevd. This can be fixed by
> sprinkling in some WAIT_FOR conditions in the dm udev rules. Our system is
> based on Ubuntu 12.04.
>
> I have attached the udevadm monitor output and the relevant piece of
> syslog for a test case where the /dev/mapper/tap0 symlink wasn't present
> when dmsetup returned.
>
> I hope this is the right place to ask this. If not please point me in the
> right direction.
>
> Thanks, Nick.
>
>

[-- Attachment #1.2: Type: text/html, Size: 2317 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: /dev/mapper/<dm_name> symlink not being created

[Zdenek Kabelac]

Dne 12.5.2012 16:59, Nicholas Nevin napsal(a):
> Anyone have any thoughts on this?
> 
> Thanks, Nick.
> 
> On Mon, Apr 30, 2012 at 8:08 AM, Nicholas Nevin <njnevin@gmail.com> wrote:
> 
>> In nightly testing of our product the tests do a fair bit of
>> creating/destroying of device mapper devices and we have been seeing
>> infrequent cases where the device is created OK but the corresponding
>> /dev/mapper/<dm_name> symlink does not exist.
>>
>> Here is an example of the dmsetup command used to create the devices.
>>
>> # dmsetup create tap0 --table '0 47247164 linear /dev/xen/blktap-2/tapdev0
>> 1'
>>
>> I managed to reproduce this problem with udev logging enabled and 'udevadm
>> monitor' running and from the logs the cause appears to be that the udev
>> add action (from the device create) and the udev change action (from when
>> the device is resumed after loading the table) are running
>> concurrently. The processing of the rules for the change action creates the
>> /dev/mapper/tap0 symlink but then the processing of the add action deletes
>> the symlink because /dev/mapper/tap0 is not in the add action DEVLINKS.
>>
>> The question I have is should dmsetup be ensuring that the add action
>> completes before the change action is handled or is this behaviour expected?
>>
>> That /dev/mapper/tap0 is not in the DEVLINKS for the add action is AFAICT
>> because of a race between the creation of the sysfs entries for the device
>> and the processing of the add action in udevd. This can be fixed by
>> sprinkling in some WAIT_FOR conditions in the dm udev rules. Our system is
>> based on Ubuntu 12.04.
>>
>> I have attached the udevadm monitor output and the relevant piece of
>> syslog for a test case where the /dev/mapper/tap0 symlink wasn't present
>> when dmsetup returned.
>>
>> I hope this is the right place to ask this. If not please point me in the
>> right direction.

You have not specified version of lvm2 tools in use.

But on udev system there are 2 options when nodes/links are created.

By default it's on RESUME - since this is the moment table is defined,
and proper messages are passed across udev system.

For backward compatibility (and for non-udev systems) there is
dmsetup create --addnodeoncreate  - which basically creates links on 'create'
phase - but with old 'badness' behind - since normally no one should create
nodes in /dev  - it's the  udev 'kingdom' which should not be externally
modified (udev might not know about such entries).

So the proper moment to work with created devices is after they have been
resumed - everything else will be ugly udev hack with unpredictable results.

Zdenek

Re: /dev/mapper/<dm_name> symlink not being created

[Nicholas Nevin]

[-- Attachment #1.1: Type: text/plain, Size: 3604 bytes --]

On Mon, May 14, 2012 at 9:43 AM, Zdenek Kabelac <zkabelac@redhat.com> wrote:

> Dne 12.5.2012 16:59, Nicholas Nevin napsal(a):
> > Anyone have any thoughts on this?
> >
> > Thanks, Nick.
> >
> > On Mon, Apr 30, 2012 at 8:08 AM, Nicholas Nevin <njnevin@gmail.com>
> wrote:
> >
> >> In nightly testing of our product the tests do a fair bit of
> >> creating/destroying of device mapper devices and we have been seeing
> >> infrequent cases where the device is created OK but the corresponding
> >> /dev/mapper/<dm_name> symlink does not exist.
> >>
> >> Here is an example of the dmsetup command used to create the devices.
> >>
> >> # dmsetup create tap0 --table '0 47247164 linear
> /dev/xen/blktap-2/tapdev0
> >> 1'
> >>
> >> I managed to reproduce this problem with udev logging enabled and
> 'udevadm
> >> monitor' running and from the logs the cause appears to be that the udev
> >> add action (from the device create) and the udev change action (from
> when
> >> the device is resumed after loading the table) are running
> >> concurrently. The processing of the rules for the change action creates
> the
> >> /dev/mapper/tap0 symlink but then the processing of the add action
> deletes
> >> the symlink because /dev/mapper/tap0 is not in the add action DEVLINKS.
> >>
> >> The question I have is should dmsetup be ensuring that the add action
> >> completes before the change action is handled or is this behaviour
> expected?
> >>
> >> That /dev/mapper/tap0 is not in the DEVLINKS for the add action is
> AFAICT
> >> because of a race between the creation of the sysfs entries for the
> device
> >> and the processing of the add action in udevd. This can be fixed by
> >> sprinkling in some WAIT_FOR conditions in the dm udev rules. Our system
> is
> >> based on Ubuntu 12.04.
> >>
> >> I have attached the udevadm monitor output and the relevant piece of
> >> syslog for a test case where the /dev/mapper/tap0 symlink wasn't present
> >> when dmsetup returned.
> >>
> >> I hope this is the right place to ask this. If not please point me in
> the
> >> right direction.
>
> You have not specified version of lvm2 tools in use.
>

Ah yes, here we go.

root@nick-lt:~# lvm version
  LVM version:     2.02.66(2) (2010-05-20)
  Library version: 1.02.48 (2010-05-20)
  Driver version:  4.22.0
root@nick-lt:~# dmsetup version
Library version:   1.02.48 (2010-05-20)
Driver version:    4.22.0
root@nick-lt:~#



> But on udev system there are 2 options when nodes/links are created.
>
> By default it's on RESUME - since this is the moment table is defined,
> and proper messages are passed across udev system.
>
> For backward compatibility (and for non-udev systems) there is
> dmsetup create --addnodeoncreate  - which basically creates links on
> 'create'
> phase - but with old 'badness' behind - since normally no one should create
> nodes in /dev  - it's the  udev 'kingdom' which should not be externally
> modified (udev might not know about such entries).
>
> So the proper moment to work with created devices is after they have been
> resumed - everything else will be ugly udev hack with unpredictable
> results.


> Zdenek
>

Makes sense thanks. The problem seems to be though that the handling of the
add (on create) and change (on resume) actions coming from the kernel to
udevd
are not handled sequentially by udevd which can cause problems such I
described.
It seems to me that dmsetup should be ensuring that the add action upon
create
has been fully handled before doing the resume which generates the change
action.
I'm no expert though so I could well have things wrong here.

-nick

[-- Attachment #1.2: Type: text/html, Size: 4917 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: /dev/mapper/<dm_name> symlink not being created

[Alasdair G Kergon]

dmsetup and udev have the functionality needed to do this in a race-free
manner, but it's necessarily complicated and takes a while to get your head
around it.  You'll need to use the dmsetup commands/options for udev cookies
described in the man page.

Alasdair

Re: /dev/mapper/<dm_name> symlink not being created

[Nicholas Nevin]

[-- Attachment #1.1: Type: text/plain, Size: 404 bytes --]

On Tue, May 15, 2012 at 10:07 AM, Alasdair G Kergon <agk@redhat.com> wrote:

> dmsetup and udev have the functionality needed to do this in a race-free
> manner, but it's necessarily complicated and takes a while to get your head
> around it.  You'll need to use the dmsetup commands/options for udev
> cookies
> described in the man page.
>
> Alasdair
>
>
Thanks. I'll take a look at doing this.

-nick

[-- Attachment #1.2: Type: text/html, Size: 740 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]