Problems writing to intel P3700 NVMe drive

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

I am having problems with writing to an intel P3700 NVMe drive on a
ARM based platform and I would appreciate any help in debugging this
problem. Here are my experiments and observations so far:

   - The drive is usable for read operations and I can run fio
(randread, etc) but I cannot write to it.
   - Other manufacturer's NVMe drive can be inserted into this
platform and I can write to it
   - I can insert the intel P3700 into an x86 host and I can perform
write operations
   - I have tried using the nvme-cli
(https://github.com/linux-nvme/nvme-cli) tool and have read the error
codes from read. It just shows 'WRITE_FAULT' which is not very
helpful.
   - I have looked at the SmartLog reports and the 'Critical Warning'
is 0, i.e. the device is not in Read-Only mode
   - I have updated the firmware using the intel SSD-DataCenterTool
and updated the firmware on the drive the latest available version.
The updated drive works fine on x86 platform.

I think there is some combination of the platform and drive that is
causing the problem. Can anyone offer pointers on how to debug this?

Thx
Kishore

Problems writing to intel P3700 NVMe drive

[Greg Schulz]

I may have missed it somewhere, what OS (and version) are you using?
Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
partition or filesystem?
Gs


Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com   
Phone 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com 
Server StorageIO and UnlimitedIO LLC aka StorageIO  |  www.storageio.com 

Author of the books:
Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
Reading List) The Green and Virtual Data Center (CRC Press & Intel
Recommended Reading List) Resilient Storage Networks ? Designing Flexible
Scalable Data Infrastructures (Elsevier)

The information contained in this email and any attachments is confidential
and may be legally privileged and or confidential. If you are not the
intended recipient then you must not use, disseminate, distribute copy or in
any other way use, the information contained in this email or attachments.
If you receive this email in error, destroy?it immediately and contact
us.?Thank you.

-----Original Message-----
From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf
Of Pavilion Storage
Sent: Wednesday, May 27, 2015 4:30 PM
To: linux-nvme at lists.infradead.org
Subject: Problems writing to intel P3700 NVMe drive

I am having problems with writing to an intel P3700 NVMe drive on a ARM
based platform and I would appreciate any help in debugging this problem.
Here are my experiments and observations so far:

   - The drive is usable for read operations and I can run fio (randread,
etc) but I cannot write to it.
   - Other manufacturer's NVMe drive can be inserted into this platform and
I can write to it
   - I can insert the intel P3700 into an x86 host and I can perform write
operations
   - I have tried using the nvme-cli
(https://github.com/linux-nvme/nvme-cli) tool and have read the error codes
from read. It just shows 'WRITE_FAULT' which is not very helpful.
   - I have looked at the SmartLog reports and the 'Critical Warning'
is 0, i.e. the device is not in Read-Only mode
   - I have updated the firmware using the intel SSD-DataCenterTool and
updated the firmware on the drive the latest available version.
The updated drive works fine on x86 platform.

I think there is some combination of the platform and drive that is causing
the problem. Can anyone offer pointers on how to debug this?

Thx
Kishore

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

Greg,
I am running this on Linux (Ubuntu/3.12) and I am accessing the raw
device (/dev/nvme1).

I have tried accessing with dd, nvme-cli and fio
Thx
Kishore

On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
> I may have missed it somewhere, what OS (and version) are you using?
> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
> partition or filesystem?
> Gs
>
>
> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com
> Phone 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
> Server StorageIO and UnlimitedIO LLC aka StorageIO  |  www.storageio.com
>
> Author of the books:
> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
> Reading List) The Green and Virtual Data Center (CRC Press & Intel
> Recommended Reading List) Resilient Storage Networks ? Designing Flexible
> Scalable Data Infrastructures (Elsevier)
>
> The information contained in this email and any attachments is confidential
> and may be legally privileged and or confidential. If you are not the
> intended recipient then you must not use, disseminate, distribute copy or in
> any other way use, the information contained in this email or attachments.
> If you receive this email in error, destroy it immediately and contact
> us. Thank you.
>
> -----Original Message-----
> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf
> Of Pavilion Storage
> Sent: Wednesday, May 27, 2015 4:30 PM
> To: linux-nvme at lists.infradead.org
> Subject: Problems writing to intel P3700 NVMe drive
>
> I am having problems with writing to an intel P3700 NVMe drive on a ARM
> based platform and I would appreciate any help in debugging this problem.
> Here are my experiments and observations so far:
>
>    - The drive is usable for read operations and I can run fio (randread,
> etc) but I cannot write to it.
>    - Other manufacturer's NVMe drive can be inserted into this platform and
> I can write to it
>    - I can insert the intel P3700 into an x86 host and I can perform write
> operations
>    - I have tried using the nvme-cli
> (https://github.com/linux-nvme/nvme-cli) tool and have read the error codes
> from read. It just shows 'WRITE_FAULT' which is not very helpful.
>    - I have looked at the SmartLog reports and the 'Critical Warning'
> is 0, i.e. the device is not in Read-Only mode
>    - I have updated the firmware using the intel SSD-DataCenterTool and
> updated the firmware on the drive the latest available version.
> The updated drive works fine on x86 platform.
>
> I think there is some combination of the platform and drive that is causing
> the problem. Can anyone offer pointers on how to debug this?
>
> Thx
> Kishore
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>

Problems writing to intel P3700 NVMe drive

[Greg Schulz]

Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine.
I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else.
What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc
Cheers gs


-----Original Message-----
From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf Of Pavilion Storage
Sent: Wednesday, May 27, 2015 5:41 PM
To: greg at storageio.com
Cc: linux-nvme at lists.infradead.org
Subject: Re: Problems writing to intel P3700 NVMe drive

Greg,
I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1).

I have tried accessing with dd, nvme-cli and fio Thx Kishore

On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
> I may have missed it somewhere, what OS (and version) are you using?
> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via 
> partition or filesystem?
> Gs
>
>
> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone 
> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com 
> Server StorageIO and UnlimitedIO LLC aka StorageIO  |  
> www.storageio.com
>
> Author of the books:
> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended 
> Reading List) The Green and Virtual Data Center (CRC Press & Intel 
> Recommended Reading List) Resilient Storage Networks ? Designing 
> Flexible Scalable Data Infrastructures (Elsevier)
>
> The information contained in this email and any attachments is 
> confidential and may be legally privileged and or confidential. If you 
> are not the intended recipient then you must not use, disseminate, 
> distribute copy or in any other way use, the information contained in this email or attachments.
> If you receive this email in error, destroy it immediately and contact 
> us. Thank you.
>
> -----Original Message-----
> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On 
> Behalf Of Pavilion Storage
> Sent: Wednesday, May 27, 2015 4:30 PM
> To: linux-nvme at lists.infradead.org
> Subject: Problems writing to intel P3700 NVMe drive
>
> I am having problems with writing to an intel P3700 NVMe drive on a 
> ARM based platform and I would appreciate any help in debugging this problem.
> Here are my experiments and observations so far:
>
>    - The drive is usable for read operations and I can run fio 
> (randread,
> etc) but I cannot write to it.
>    - Other manufacturer's NVMe drive can be inserted into this 
> platform and I can write to it
>    - I can insert the intel P3700 into an x86 host and I can perform 
> write operations
>    - I have tried using the nvme-cli
> (https://github.com/linux-nvme/nvme-cli) tool and have read the error 
> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>    - I have looked at the SmartLog reports and the 'Critical Warning'
> is 0, i.e. the device is not in Read-Only mode
>    - I have updated the firmware using the intel SSD-DataCenterTool 
> and updated the firmware on the drive the latest available version.
> The updated drive works fine on x86 platform.
>
> I think there is some combination of the platform and drive that is 
> causing the problem. Can anyone offer pointers on how to debug this?
>
> Thx
> Kishore
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

Greg

Here is the output from the dd command on a good (non-intel;
/dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no
difference in displayed output of dd, but the write is successful in
the non-intel drive. The nvme-cli command tool correctly shows the
failure to write, but offers no further clues beyond the failed
status.

thx
Kishore

-------------------------------------------------------
[root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s
[root]#
-----------------------------------------------------
[root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s
[root]#
---------------------------------------------------------
[root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null
write: success
[root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null
write:WRITE_FAULT(2280)
---------------------------------------------------------

On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote:
> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine.
> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else.
> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc
> Cheers gs
>
>
> -----Original Message-----
> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage
> Sent: Wednesday, May 27, 2015 5:41 PM
> To: greg at storageio.com
> Cc: linux-nvme at lists.infradead.org
> Subject: Re: Problems writing to intel P3700 NVMe drive
>
> Greg,
> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1).
>
> I have tried accessing with dd, nvme-cli and fio Thx Kishore
>
> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
>> I may have missed it somewhere, what OS (and version) are you using?
>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
>> partition or filesystem?
>> Gs
>>
>>
>> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone
>> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
>> Server StorageIO and UnlimitedIO LLC aka StorageIO  |
>> www.storageio.com
>>
>> Author of the books:
>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
>> Reading List) The Green and Virtual Data Center (CRC Press & Intel
>> Recommended Reading List) Resilient Storage Networks ? Designing
>> Flexible Scalable Data Infrastructures (Elsevier)
>>
>> The information contained in this email and any attachments is
>> confidential and may be legally privileged and or confidential. If you
>> are not the intended recipient then you must not use, disseminate,
>> distribute copy or in any other way use, the information contained in this email or attachments.
>> If you receive this email in error, destroy it immediately and contact
>> us. Thank you.
>>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>> Behalf Of Pavilion Storage
>> Sent: Wednesday, May 27, 2015 4:30 PM
>> To: linux-nvme at lists.infradead.org
>> Subject: Problems writing to intel P3700 NVMe drive
>>
>> I am having problems with writing to an intel P3700 NVMe drive on a
>> ARM based platform and I would appreciate any help in debugging this problem.
>> Here are my experiments and observations so far:
>>
>>    - The drive is usable for read operations and I can run fio
>> (randread,
>> etc) but I cannot write to it.
>>    - Other manufacturer's NVMe drive can be inserted into this
>> platform and I can write to it
>>    - I can insert the intel P3700 into an x86 host and I can perform
>> write operations
>>    - I have tried using the nvme-cli
>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error
>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>>    - I have looked at the SmartLog reports and the 'Critical Warning'
>> is 0, i.e. the device is not in Read-Only mode
>>    - I have updated the firmware using the intel SSD-DataCenterTool
>> and updated the firmware on the drive the latest available version.
>> The updated drive works fine on x86 platform.
>>
>> I think there is some combination of the platform and drive that is
>> causing the problem. Can anyone offer pointers on how to debug this?
>>
>> Thx
>> Kishore
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>

Problems writing to intel P3700 NVMe drive

[j cagle]

Hi Kishore,

dd won't write anything if you use if=/dev/null.  You should use
if=/dev/zero instead. (This is probably what Greg intended.)

John

On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage
<storagepavilion@gmail.com> wrote:
> Greg
>
> Here is the output from the dd command on a good (non-intel;
> /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no
> difference in displayed output of dd, but the write is successful in
> the non-intel drive. The nvme-cli command tool correctly shows the
> failure to write, but offers no further clues beyond the failed
> status.
>
> thx
> Kishore
>
> -------------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s
> [root]#
> -----------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s
> [root]#
> ---------------------------------------------------------
> [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null
> write: success
> [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null
> write:WRITE_FAULT(2280)
> ---------------------------------------------------------
>
> On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote:
>> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine.
>> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else.
>> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc
>> Cheers gs
>>
>>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage
>> Sent: Wednesday, May 27, 2015 5:41 PM
>> To: greg at storageio.com
>> Cc: linux-nvme at lists.infradead.org
>> Subject: Re: Problems writing to intel P3700 NVMe drive
>>
>> Greg,
>> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1).
>>
>> I have tried accessing with dd, nvme-cli and fio Thx Kishore
>>
>> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
>>> I may have missed it somewhere, what OS (and version) are you using?
>>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
>>> partition or filesystem?
>>> Gs
>>>
>>>
>>> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone
>>> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
>>> Server StorageIO and UnlimitedIO LLC aka StorageIO  |
>>> www.storageio.com
>>>
>>> Author of the books:
>>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
>>> Reading List) The Green and Virtual Data Center (CRC Press & Intel
>>> Recommended Reading List) Resilient Storage Networks ? Designing
>>> Flexible Scalable Data Infrastructures (Elsevier)
>>>
>>> The information contained in this email and any attachments is
>>> confidential and may be legally privileged and or confidential. If you
>>> are not the intended recipient then you must not use, disseminate,
>>> distribute copy or in any other way use, the information contained in this email or attachments.
>>> If you receive this email in error, destroy it immediately and contact
>>> us. Thank you.
>>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>>> Behalf Of Pavilion Storage
>>> Sent: Wednesday, May 27, 2015 4:30 PM
>>> To: linux-nvme at lists.infradead.org
>>> Subject: Problems writing to intel P3700 NVMe drive
>>>
>>> I am having problems with writing to an intel P3700 NVMe drive on a
>>> ARM based platform and I would appreciate any help in debugging this problem.
>>> Here are my experiments and observations so far:
>>>
>>>    - The drive is usable for read operations and I can run fio
>>> (randread,
>>> etc) but I cannot write to it.
>>>    - Other manufacturer's NVMe drive can be inserted into this
>>> platform and I can write to it
>>>    - I can insert the intel P3700 into an x86 host and I can perform
>>> write operations
>>>    - I have tried using the nvme-cli
>>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error
>>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>>>    - I have looked at the SmartLog reports and the 'Critical Warning'
>>> is 0, i.e. the device is not in Read-Only mode
>>>    - I have updated the firmware using the intel SSD-DataCenterTool
>>> and updated the firmware on the drive the latest available version.
>>> The updated drive works fine on x86 platform.
>>>
>>> I think there is some combination of the platform and drive that is
>>> causing the problem. Can anyone offer pointers on how to debug this?
>>>
>>> Thx
>>> Kishore
>>>
>>> _______________________________________________
>>> Linux-nvme mailing list
>>> Linux-nvme at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

John
Here is the outputs with if=/dev/zero and oflag=direct and you can see
the error.
thx
Kishore
------------------------------------------------------
[root]# dd if=/dev/zero of=/dev/nvme0n1 oflag=direct bs=512
count=1000000 conv=notrunc
1000000+0 records in
1000000+0 records out
512000000 bytes (512 MB) copied, 77.6635 s, 6.6 MB/s
-----------------------------------------------------------
[root]# dd if=/dev/zero of=/dev/nvme1n1 oflag=direct bs=512
count=1000000 conv=notrunc
dd: writing '/dev/nvme1n1': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0212549 s, 0.0 kB/s
-------------------------------------------------------------
[root]# ./nvme write /dev/nvme0n1 -s 0 -z 4096 -d /dev/zero
write: success
-------------------------------------------------------------
[root]# ./nvme write /dev/nvme1n1 -s 0 -z 4096 -d /dev/zero
write:WRITE_FAULT(2280)
[root]#

On Wed, May 27, 2015@5:24 PM, j cagle <jcagle@gmail.com> wrote:
> Hi Kishore,
>
> dd won't write anything if you use if=/dev/null.  You should use
> if=/dev/zero instead. (This is probably what Greg intended.)
>
> John
>
> On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage
> <storagepavilion@gmail.com> wrote:
>> Greg
>>
>> Here is the output from the dd command on a good (non-intel;
>> /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no
>> difference in displayed output of dd, but the write is successful in
>> the non-intel drive. The nvme-cli command tool correctly shows the
>> failure to write, but offers no further clues beyond the failed
>> status.
>>
>> thx
>> Kishore
>>
>> -------------------------------------------------------
>> [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc
>> 0+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s
>> [root]#
>> -----------------------------------------------------
>> [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc
>> 0+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s
>> [root]#
>> ---------------------------------------------------------
>> [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null
>> write: success
>> [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null
>> write:WRITE_FAULT(2280)
>> ---------------------------------------------------------
>>
>> On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote:
>>> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine.
>>> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else.
>>> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc
>>> Cheers gs
>>>
>>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage
>>> Sent: Wednesday, May 27, 2015 5:41 PM
>>> To: greg at storageio.com
>>> Cc: linux-nvme at lists.infradead.org
>>> Subject: Re: Problems writing to intel P3700 NVMe drive
>>>
>>> Greg,
>>> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1).
>>>
>>> I have tried accessing with dd, nvme-cli and fio Thx Kishore
>>>
>>> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
>>>> I may have missed it somewhere, what OS (and version) are you using?
>>>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
>>>> partition or filesystem?
>>>> Gs
>>>>
>>>>
>>>> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone
>>>> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
>>>> Server StorageIO and UnlimitedIO LLC aka StorageIO  |
>>>> www.storageio.com
>>>>
>>>> Author of the books:
>>>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
>>>> Reading List) The Green and Virtual Data Center (CRC Press & Intel
>>>> Recommended Reading List) Resilient Storage Networks ? Designing
>>>> Flexible Scalable Data Infrastructures (Elsevier)
>>>>
>>>> The information contained in this email and any attachments is
>>>> confidential and may be legally privileged and or confidential. If you
>>>> are not the intended recipient then you must not use, disseminate,
>>>> distribute copy or in any other way use, the information contained in this email or attachments.
>>>> If you receive this email in error, destroy it immediately and contact
>>>> us. Thank you.
>>>>
>>>> -----Original Message-----
>>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>>>> Behalf Of Pavilion Storage
>>>> Sent: Wednesday, May 27, 2015 4:30 PM
>>>> To: linux-nvme at lists.infradead.org
>>>> Subject: Problems writing to intel P3700 NVMe drive
>>>>
>>>> I am having problems with writing to an intel P3700 NVMe drive on a
>>>> ARM based platform and I would appreciate any help in debugging this problem.
>>>> Here are my experiments and observations so far:
>>>>
>>>>    - The drive is usable for read operations and I can run fio
>>>> (randread,
>>>> etc) but I cannot write to it.
>>>>    - Other manufacturer's NVMe drive can be inserted into this
>>>> platform and I can write to it
>>>>    - I can insert the intel P3700 into an x86 host and I can perform
>>>> write operations
>>>>    - I have tried using the nvme-cli
>>>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error
>>>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>>>>    - I have looked at the SmartLog reports and the 'Critical Warning'
>>>> is 0, i.e. the device is not in Read-Only mode
>>>>    - I have updated the firmware using the intel SSD-DataCenterTool
>>>> and updated the firmware on the drive the latest available version.
>>>> The updated drive works fine on x86 platform.
>>>>
>>>> I think there is some combination of the platform and drive that is
>>>> causing the problem. Can anyone offer pointers on how to debug this?
>>>>
>>>> Thx
>>>> Kishore
>>>>
>>>> _______________________________________________
>>>> Linux-nvme mailing list
>>>> Linux-nvme at lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>>>
>>>
>>> _______________________________________________
>>> Linux-nvme mailing list
>>> Linux-nvme at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

Greg
I have verified that the same drive is working in an x86 system with
the same firmware. Also, this appears to be a problem with multiple
intel P3700 SSD drives, not just one. I have tried other manufacturers
drives in the same slots on our ARM platform and verified that the
platform (and the slot) is working.

Yes, these are PCIe x4 Gen3 slots.

>From the error log the (StatusField.SCT and StatusField.SC appear to
0x2 and 0x80) which for a write command is LBA out of range, but that
does not make sense because the error is the same regardless of the
LBA address used.

thx
Kishore

On Wed, May 27, 2015@5:09 PM, Greg Schulz <greg@storageio.com> wrote:
> Interesting, have you tried the bad card in a different x86 based server to see if it is the card?
> Also, if the card is good, yet wont write in the specific server, have you checked the servers BIOS, granted if you have one card working and one not, that?s a different ball game.
> Not sure if there is an ARM variation of the intel UEFI sysinfo tool that would give you more info.
> Whats not clear is if you are getting an actual write protect error or simply a write fault.
> Write protect are usually an indicator of something actually wrong with the underlying nand flash, or its FTL and with a good controller, it will try to work around before giving up.
> Write fault could be a write-protect, however could also be a driver, bios, firmware or even PCIe slot issue.
> Have not tried the 3700 in a PCIe G2 slot, assume you have them in a PCIe G3 slot or riser?
> Good luck
> Cheers
> Gs
>

On Wed, May 27, 2015 at 4:53 PM, Pavilion Storage
<storagepavilion@gmail.com> wrote:
> Greg
>
> Here is the output from the dd command on a good (non-intel;
> /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no
> difference in displayed output of dd, but the write is successful in
> the non-intel drive. The nvme-cli command tool correctly shows the
> failure to write, but offers no further clues beyond the failed
> status.
>
> thx
> Kishore
>
> -------------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s
> [root]#
> -----------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s
> [root]#
> ---------------------------------------------------------
> [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null
> write: success
> [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null
> write:WRITE_FAULT(2280)
> ---------------------------------------------------------
>
> On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote:
>> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine.
>> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else.
>> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc
>> Cheers gs
>>
>>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage
>> Sent: Wednesday, May 27, 2015 5:41 PM
>> To: greg at storageio.com
>> Cc: linux-nvme at lists.infradead.org
>> Subject: Re: Problems writing to intel P3700 NVMe drive
>>
>> Greg,
>> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1).
>>
>> I have tried accessing with dd, nvme-cli and fio Thx Kishore
>>
>> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
>>> I may have missed it somewhere, what OS (and version) are you using?
>>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
>>> partition or filesystem?
>>> Gs
>>>
>>>
>>> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone
>>> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
>>> Server StorageIO and UnlimitedIO LLC aka StorageIO  |
>>> www.storageio.com
>>>
>>> Author of the books:
>>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
>>> Reading List) The Green and Virtual Data Center (CRC Press & Intel
>>> Recommended Reading List) Resilient Storage Networks ? Designing
>>> Flexible Scalable Data Infrastructures (Elsevier)
>>>
>>> The information contained in this email and any attachments is
>>> confidential and may be legally privileged and or confidential. If you
>>> are not the intended recipient then you must not use, disseminate,
>>> distribute copy or in any other way use, the information contained in this email or attachments.
>>> If you receive this email in error, destroy it immediately and contact
>>> us. Thank you.
>>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>>> Behalf Of Pavilion Storage
>>> Sent: Wednesday, May 27, 2015 4:30 PM
>>> To: linux-nvme at lists.infradead.org
>>> Subject: Problems writing to intel P3700 NVMe drive
>>>
>>> I am having problems with writing to an intel P3700 NVMe drive on a
>>> ARM based platform and I would appreciate any help in debugging this problem.
>>> Here are my experiments and observations so far:
>>>
>>>    - The drive is usable for read operations and I can run fio
>>> (randread,
>>> etc) but I cannot write to it.
>>>    - Other manufacturer's NVMe drive can be inserted into this
>>> platform and I can write to it
>>>    - I can insert the intel P3700 into an x86 host and I can perform
>>> write operations
>>>    - I have tried using the nvme-cli
>>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error
>>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>>>    - I have looked at the SmartLog reports and the 'Critical Warning'
>>> is 0, i.e. the device is not in Read-Only mode
>>>    - I have updated the firmware using the intel SSD-DataCenterTool
>>> and updated the firmware on the drive the latest available version.
>>> The updated drive works fine on x86 platform.
>>>
>>> I think there is some combination of the platform and drive that is
>>> causing the problem. Can anyone offer pointers on how to debug this?
>>>
>>> Thx
>>> Kishore
>>>
>>> _______________________________________________
>>> Linux-nvme mailing list
>>> Linux-nvme at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

Thanks to Keith Busch for the providing the clues.
The problem was that the PCIe MPSS on the drive and the CPU root
complex did not match and that caused a problem. Setting it to a
common value fixed the problem.
Kishore

On Wed, May 27, 2015@7:33 PM, Greg Schulz <greg@storageio.com> wrote:
> No worries John, my bad, yes you are correct, s/b if=/dev/zero ;)
>
> Cheers
>
> Gs
>
>
>
>
>
> From: j cagle [mailto:jcagle at gmail.com]
> Sent: Wednesday, May 27, 2015 7:19 PM
> To: Pavilion Storage
> Cc: greg schulz; linux-nvme at lists.infradead.org
> Subject: Re: Problems writing to intel P3700 NVMe drive
>
>
>
> Hi Kishore,
>
>
>
> dd won't write anything if you use if=/dev/null.  You should use
> if=/dev/zero instead. (This is probably what Greg intended.)
>
>
>
> John
>
>
>
>
>
> On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage
> <storagepavilion@gmail.com> wrote:
>
> Greg
>
> Here is the output from the dd command on a good (non-intel;
> /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no
> difference in displayed output of dd, but the write is successful in
> the non-intel drive. The nvme-cli command tool correctly shows the
> failure to write, but offers no further clues beyond the failed
> status.
>
> thx
> Kishore
>
> -------------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s
> [root]#
> -----------------------------------------------------
> [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s
> [root]#
> ---------------------------------------------------------
> [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null
> write: success
> [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null
> write:WRITE_FAULT(2280)
> ---------------------------------------------------------
>
>
> On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote:
>> Thanks for the follow-up and extra info, I have 3700s running on x86/64
>> Ubuntu 14.04 and working fine.
>> I did update the drivers from the Intel site even though they were already
>> in the 14.04 kernel when I was troubel shooting something else.
>> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512
>> count=1000000 conv=notrunc
>> Cheers gs
>>
>>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf
>> Of Pavilion Storage
>> Sent: Wednesday, May 27, 2015 5:41 PM
>> To: greg at storageio.com
>> Cc: linux-nvme at lists.infradead.org
>> Subject: Re: Problems writing to intel P3700 NVMe drive
>>
>> Greg,
>> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device
>> (/dev/nvme1).
>>
>> I have tried accessing with dd, nvme-cli and fio Thx Kishore
>>
>> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote:
>>> I may have missed it somewhere, what OS (and version) are you using?
>>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via
>>> partition or filesystem?
>>> Gs
>>>
>>>
>>> Greg Schulz  |  Sr. Advisory Analyst  |  greg at storageio.com Phone
>>> 651-275-1563  |  twitter @storageio  |  blog www.storageioblog.com
>>> Server StorageIO and UnlimitedIO LLC aka StorageIO  |
>>> www.storageio.com
>>>
>>> Author of the books:
>>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended
>>> Reading List) The Green and Virtual Data Center (CRC Press & Intel
>>> Recommended Reading List) Resilient Storage Networks ? Designing
>>> Flexible Scalable Data Infrastructures (Elsevier)
>>>
>>> The information contained in this email and any attachments is
>>> confidential and may be legally privileged and or confidential. If you
>>> are not the intended recipient then you must not use, disseminate,
>>> distribute copy or in any other way use, the information contained in
>>> this email or attachments.
>>> If you receive this email in error, destroy it immediately and contact
>>> us. Thank you.
>>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>>> Behalf Of Pavilion Storage
>>> Sent: Wednesday, May 27, 2015 4:30 PM
>>> To: linux-nvme at lists.infradead.org
>>> Subject: Problems writing to intel P3700 NVMe drive
>>>
>>> I am having problems with writing to an intel P3700 NVMe drive on a
>>> ARM based platform and I would appreciate any help in debugging this
>>> problem.
>>> Here are my experiments and observations so far:
>>>
>>>    - The drive is usable for read operations and I can run fio
>>> (randread,
>>> etc) but I cannot write to it.
>>>    - Other manufacturer's NVMe drive can be inserted into this
>>> platform and I can write to it
>>>    - I can insert the intel P3700 into an x86 host and I can perform
>>> write operations
>>>    - I have tried using the nvme-cli
>>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error
>>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful.
>>>    - I have looked at the SmartLog reports and the 'Critical Warning'
>>> is 0, i.e. the device is not in Read-Only mode
>>>    - I have updated the firmware using the intel SSD-DataCenterTool
>>> and updated the firmware on the drive the latest available version.
>>> The updated drive works fine on x86 platform.
>>>
>>> I think there is some combination of the platform and drive that is
>>> causing the problem. Can anyone offer pointers on how to debug this?
>>>
>>> Thx
>>> Kishore
>>>
>>> _______________________________________________
>>> Linux-nvme mailing list
>>> Linux-nvme at lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>>
>>
>> _______________________________________________
>> Linux-nvme mailing list
>> Linux-nvme at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-nvme
>>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
>
>

Problems writing to intel P3700 NVMe drive

[Keith Busch]

Perhaps not a desirable/permanent solution, but for everyone's benefit,
we can work-around the problem with kernel parameter:

   pci=pcie_bus_safe

Using 'pcie_bus_perf' instead may also work, though your original
solution of just using x86 sounded perfectly reasonable to me. :)

On Thu, 28 May 2015, Pavilion Storage wrote:
> Thanks to Keith Busch for the providing the clues.
> The problem was that the PCIe MPSS on the drive and the CPU root
> complex did not match and that caused a problem. Setting it to a
> common value fixed the problem.
> Kishore

Problems writing to intel P3700 NVMe drive

[Pavilion Storage]

The root cause might be a bug in the kernel where pcie devices are
configured. Setting it to 128B will work, but may not perform to the
best capabilities allowed by the device and RC. If we have multiple
drives from different manufacturers in the system, we may penalize the
drive that has the hardware capability to handle higher MPSS.

thx
Kishore


On Thu, May 28, 2015@2:03 PM, Keith Busch <keith.busch@intel.com> wrote:
> Perhaps not a desirable/permanent solution, but for everyone's benefit,
> we can work-around the problem with kernel parameter:
>
>   pci=pcie_bus_safe
>
> Using 'pcie_bus_perf' instead may also work, though your original
> solution of just using x86 sounded perfectly reasonable to me. :)
>
> On Thu, 28 May 2015, Pavilion Storage wrote:
>>
>> Thanks to Keith Busch for the providing the clues.
>> The problem was that the PCIe MPSS on the drive and the CPU root
>> complex did not match and that caused a problem. Setting it to a
>> common value fixed the problem.
>> Kishore

Problems writing to intel P3700 NVMe drive

[Keith Busch]

Using 'pcie_bus_perf' will work just fine in most cases, selecting the
highest capable settings so as not to penalize other h/w. Though all
your controllers had the same max payload capabilities anyway, so they
should all be the same from the max TLP perspective.

On Thu, 28 May 2015, Pavilion Storage wrote:
> The root cause might be a bug in the kernel where pcie devices are
> configured. Setting it to 128B will work, but may not perform to the
> best capabilities allowed by the device and RC. If we have multiple
> drives from different manufacturers in the system, we may penalize the
> drive that has the hardware capability to handle higher MPSS.
>
> thx
> Kishore
>
>
> On Thu, May 28, 2015@2:03 PM, Keith Busch <keith.busch@intel.com> wrote:
>> Perhaps not a desirable/permanent solution, but for everyone's benefit,
>> we can work-around the problem with kernel parameter:
>>
>>   pci=pcie_bus_safe
>>
>> Using 'pcie_bus_perf' instead may also work, though your original
>> solution of just using x86 sounded perfectly reasonable to me. :)
>>
>> On Thu, 28 May 2015, Pavilion Storage wrote:
>>>
>>> Thanks to Keith Busch for the providing the clues.
>>> The problem was that the PCIe MPSS on the drive and the CPU root
>>> complex did not match and that caused a problem. Setting it to a
>>> common value fixed the problem.
>>> Kishore
>

Problems writing to intel P3700 NVMe drive

[Keith Busch]

+cc the pci list

Hi Bjorn,

A while back, there were a few proposals on changing the pci driver's
default MPS tuning from the existing do-nothing policy to something
safe so end-users don't need to remember kernel parameters as described
below. Is this still active, or can we kick that back to life if not?

Thanks,
Keith

On Thu, 28 May 2015, Keith Busch wrote:
> Perhaps not a desirable/permanent solution, but for everyone's benefit,
> we can work-around the problem with kernel parameter:
>
>  pci=pcie_bus_safe
>
> Using 'pcie_bus_perf' instead may also work, though your original
> solution of just using x86 sounded perfectly reasonable to me. :)
>
> On Thu, 28 May 2015, Pavilion Storage wrote:
>> Thanks to Keith Busch for the providing the clues.
>> The problem was that the PCIe MPSS on the drive and the CPU root
>> complex did not match and that caused a problem. Setting it to a
>> common value fixed the problem.
>> Kishore

Problems writing to intel P3700 NVMe drive

[Bjorn Helgaas]

On Wed, Jun 3, 2015@3:10 PM, Keith Busch <keith.busch@intel.com> wrote:
> +cc the pci list
>
> Hi Bjorn,
>
> A while back, there were a few proposals on changing the pci driver's
> default MPS tuning from the existing do-nothing policy to something
> safe so end-users don't need to remember kernel parameters as described
> below. Is this still active, or can we kick that back to life if not?

I'd love to see some activity there.  Somebody else asked me about
this a few weeks ago, and I sent him this list of some of the things I
think are wrong with the current situation:

- MPS configuration is done in pcie_bus_configure_settings().  This is
  not arch-specific, but it is done by arch code, and only arm,
  powerpc, tile, and x86 call it.

- MPS configuration should be done before a driver claims a device.
  This is currently wrong on arm.

- I would rather have MPS configuration done somewhere in the
  pci_scan_single_device() path.  That way it would be done on every
  arch and in every hotplug path.

- I know MPS depends on more than just the individual device and its
  upstream bridge, so it's not as simple as some other properties.
  But if we have a correctly-configured tree, we should be able to
  figure out what needs to change if we add one more device to it.

- I do not believe it is safe for drivers to manipulate MPS behind the
  back of the PCI core, so pcie_set_mps() should probably not be
  exported.

5f39e6705faa ("PCI: Disable MPS configuration by default") turned off
most the tuning we used to do.  I expect that if we turn it back on,
we'll trip over a few issues.  That in itself is not so much of a
problem, but before we turn things back on, I want to have better
infrastructure in place that makes it easier to diagnose and fix those
issues.  For example, I'd like to see just enough clues in dmesg to
enable us to see what the kernel's doing and verify that it's correct.
And I want it do be done cleanly on every arch for every device, with
hot-added ones being handled the same as those present at boot.

> On Thu, 28 May 2015, Keith Busch wrote:
>>
>> Perhaps not a desirable/permanent solution, but for everyone's benefit,
>> we can work-around the problem with kernel parameter:
>>
>>  pci=pcie_bus_safe
>>
>> Using 'pcie_bus_perf' instead may also work, though your original
>> solution of just using x86 sounded perfectly reasonable to me. :)
>>
>> On Thu, 28 May 2015, Pavilion Storage wrote:
>>>
>>> Thanks to Keith Busch for the providing the clues.
>>> The problem was that the PCIe MPSS on the drive and the CPU root
>>> complex did not match and that caused a problem. Setting it to a
>>> common value fixed the problem.
>>> Kishore

Problems writing to intel P3700 NVMe drive

[Keith Busch]

On Wed, 3 Jun 2015, Bjorn Helgaas wrote:
> On Wed, Jun 3, 2015@3:10 PM, Keith Busch <keith.busch@intel.com> wrote:
>> A while back, there were a few proposals on changing the pci driver's
>> default MPS tuning from the existing do-nothing policy to something
>> safe so end-users don't need to remember kernel parameters as described
>> below. Is this still active, or can we kick that back to life if not?
>
> I'd love to see some activity there.  Somebody else asked me about
> this a few weeks ago, and I sent him this list of some of the things I
> think are wrong with the current situation:

Thanks for the update. I got in contact with some comrades already working
on this, so we'll post an updated proposal when it's ready. This feature
is getting more attention with broader hot plug pci-e storage usage (among
other reasons), so I'm sure we'll give this its due consideration this time.