nvmf question - synchronization between target/initiator regarding partitions

nvmf question - synchronization between target/initiator regarding partitions

[Guilherme G. Piccoli]

We observed that it's possible to perform partition operations in both
nvmf target and initiator block devices, like creating and deleting
partitions.

But there is no sync mechanism between target and initiator regarding
the partitions operations. After creating a partition on initiator, for
example, we don't see it on target side. We could format it like ext4 on
initiator, and still target cannot see it. It's possible to trigger a
BLKRRPART ioctl on target, which end up calling revalidate_disk() on
nvme driver and then partitions are perceived.

So, question: is this behavior expected/acceptable? Is it completely up
to userspace to deal with the synchronization between host/target?
I think answer might be yes since partitions are a higher level of
abstraction than nvmf (which is purely block device aware).

But if kernel could/should jump in, we possibly could try to issue a
revalidate_disk on target whenever this operation is performed on
initiator side (and vice-versa). I confess I couldn't find such sync
idea in NVMe over fabrics spec, though.
Also, reading NVMe spec 1.3, we do have the optional feature
"reservations", but seems it doesn't mention target (only multiple hosts).

Thanks in advance for any ideas on this subject.
Cheers,



Guilherme

nvmf question - synchronization between target/initiator regarding partitions

[Hannes Reinecke]

On 08/07/2017 03:42 PM, Guilherme G. Piccoli wrote:
> We observed that it's possible to perform partition operations in both
> nvmf target and initiator block devices, like creating and deleting
> partitions.
> 
> But there is no sync mechanism between target and initiator regarding
> the partitions operations. After creating a partition on initiator, for
> example, we don't see it on target side. We could format it like ext4 on
> initiator, and still target cannot see it. It's possible to trigger a
> BLKRRPART ioctl on target, which end up calling revalidate_disk() on
> nvme driver and then partitions are perceived.
> 
> So, question: is this behavior expected/acceptable? Is it completely up
> to userspace to deal with the synchronization between host/target?
> I think answer might be yes since partitions are a higher level of
> abstraction than nvmf (which is purely block device aware).
> 
Yes, this is to be expected.
After all, one could argue that no partition information should ever be
visible on the target seeing that the device is exported.
As the initiator has exclusive access, the target has no business
looking at the contents; in fact, this might lead to unexpected
corruptions if things like udev jump in on the target side and try to
'correct' things with journal replay etc.

> But if kernel could/should jump in, we possibly could try to issue a
> revalidate_disk on target whenever this operation is performed on
> initiator side (and vice-versa). I confess I couldn't find such sync
> idea in NVMe over fabrics spec, though.
> Also, reading NVMe spec 1.3, we do have the optional feature
> "reservations", but seems it doesn't mention target (only multiple hosts).
> 
See above.
Never try this :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)

nvmf question - synchronization between target/initiator regarding partitions

[Guilherme G. Piccoli]

Thanks for your feedback Hannes, agreed!

Cheers,


Guilherme

nvmf question - synchronization between target/initiator regarding partitions

[Christoph Hellwig]

On Mon, Aug 07, 2017@02:29:47PM -0300, Guilherme G. Piccoli wrote:
> Thanks for your feedback Hannes, agreed!

And btw, you'll see similar results with the SCSI target or nbd,
so it's not really nvme specific.

nvmf question - synchronization between target/initiator regarding partitions

[Guilherme G. Piccoli]

On 08/10/2017 06:16 AM, Christoph Hellwig wrote:
> On Mon, Aug 07, 2017@02:29:47PM -0300, Guilherme G. Piccoli wrote:
>> Thanks for your feedback Hannes, agreed!
> 
> And btw, you'll see similar results with the SCSI target or nbd,
> so it's not really nvme specific.

Thanks, I agree - noticed the same stuff. I've used nvmf as a "trigger"
to the subject, in order to discuss a possible generic solution. But
since everything is working as expected, no need to pursue a "fix" heheh