Re: Q: "- PDU header Digest" fetaure

[Boaz Harrosh <bharrosh-C4P08NqkoRlBDgjK7y7TUQ@public.gmane.org>]

Hi Mike, list.

Mike Christie has pointed out of a serious problem for us which we need
the list help of.

It started with a question by Ulrich Windl of why data-digests are
not supported/recommended by open-iscsi installations and distros.

[iscsi data-digests is when the complete payload of an iscsi transaction
 initiator-target is signed by an HMAC(SHA1) both read/write]

Mike Christie wrote:
> Ulrich Windl wrote:
>>
>
> Data digests were working but when upstream did the scatterlist changes 
> to the kernel it broke data digests. We have not found the cause yet.
> 
> For Red Hat, they do not support them for different reasons depending on 
> the version and arch. For example in RHEL5, the big endien crypto digest 
> code is busted. It needs a fix from upstream, and I think in general 
> there is still some other bugs in the digest code.
> 
>> I see the performance impact, but is there another reason against implementing it? 
>> Can I safely activate it on the target, or will it cause problems?
>>
> 
> Another reason a lot of distros do not support it is because a common 
> problem we always hit is that users will write out some data, then start 
> modifying it again. But the kernel will normally not do do a sync write 
> when you do a write. So once the write() returns, the kernel is still 
> sending it through the caches, block, scsi, and iscsi layers. If you are 
> writing to the data while the it is working its way through the iscsi 
> layers, the iscsi layer could have done the digest calculation, then you 
> could modify it and now when the target checks it the digest check will 
> fail. And so this happens over and over and you get digest errors all 
> over the place and the iscsi layers fire their error handling and retry 
> and retry, and in the end they just say forget it and do not support 
> data digests.
> 

Mike if what you said in the last paragraph is true, about FS modifying the data
while the request is in-flight, then it does not explain your statement above
about, things getting worse around the scatterlist changes.

The way I see it there can be two fundamental problems:
1. The FS is permitted to (or sinfully) modifies pages of memory while a request to
   write these pages is already in-flight. fsdevel guys might want to comment on that?
   Mike have you observed these problems with a particular file system?
   I can anticipate such a problem arising in a memory-mapped IO, while a page-cache
   write-back is in progress. Is that so? is Linux not safe in this regard? If so
   how does DM & MD do there raid parity calculations? do they copy the data?

2. iSCSI releases the request too soon, before the all data was actually used up by the
   network stack, and is allowing the FS to continue modifying these pages.
   This is a serious problem which means that there can be crashes and data corruption even
   if data-digest are not used.
   Actually we did move not long ago from copy of network data to been completely copy-less
   could that be the point in time things stopped working?

3. Plain coding bug, but I could not find any.

I know in the passed that data-digests are a grate tool for finding bugs that otherwise can
go undetected, it happened to me several times in the passed. All of these cases reviled a flaw
in the code, do to rebasing, things changing, plain programmer bugs.

Mike, I'm running here a plain iscsi initiator-target setup and the regression tests, and it
runs. What setup and tests did you run to trigger these digest retries, I would like to
reproduce this here, and investigate.

Thanks for any help
Boaz