[linux-lvm] Dead device and cleaning up LVM

[Ray Van Dolson <rvandolson@esri.com>]

I have a volgroup that was created on top of an iSCSI mounted block
device (target was stgt on RHEL 5.3 and initiator is the default iSCSI
initiator in RHEL 4.7).

Through a goof the target went offline and I tried shutting down the
initiator software before disabling the volume group on the initiator
with vgchange -an.  Now I'm left with a bit of a mess including iSCSI
initiator modules that won't completely unload.  I'm trying to avoid a
reboot and we have been theorizing that we somehow need to tell LVM to
"let go" of the remaining ghosts of the iSCSI device before things can
go back to normal.

The /dev/sdXX device is gone, but how can I convice LVM to quit trying
to use /dev/dm-2?  Can I edit the /etc/lvm/cache/.cache file and remove
entries there?

I'm seeing the following stuff in dmesg:

  scsi7 (0:1): rejecting I/O to dead device

And all the various pvscan/vgscan/lvscan tools show something similar
to the following:

  /dev/dm-2: read failed after 0 of 4096 at 4398038056960: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 4398038114304: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 4096: Input/output error
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error

Anyone know how I can remove this device both from the system and from
LVM safely?  I can reboot, but saving that as a last resort if
possible.

Also, any best practices or suggestions for using iSCSI backed block
devices as volgroup members would be appreciated.  I'm worried about
LVM not recognizing the volume group of physical volumes correctly on
reboots...

Thanks,
Ray