* Problems writing to intel P3700 NVMe drive @ 2015-05-27 21:30 Pavilion Storage 2015-05-27 22:34 ` Greg Schulz 0 siblings, 1 reply; 18+ messages in thread From: Pavilion Storage @ 2015-05-27 21:30 UTC (permalink / raw) I am having problems with writing to an intel P3700 NVMe drive on a ARM based platform and I would appreciate any help in debugging this problem. Here are my experiments and observations so far: - The drive is usable for read operations and I can run fio (randread, etc) but I cannot write to it. - Other manufacturer's NVMe drive can be inserted into this platform and I can write to it - I can insert the intel P3700 into an x86 host and I can perform write operations - I have tried using the nvme-cli (https://github.com/linux-nvme/nvme-cli) tool and have read the error codes from read. It just shows 'WRITE_FAULT' which is not very helpful. - I have looked at the SmartLog reports and the 'Critical Warning' is 0, i.e. the device is not in Read-Only mode - I have updated the firmware using the intel SSD-DataCenterTool and updated the firmware on the drive the latest available version. The updated drive works fine on x86 platform. I think there is some combination of the platform and drive that is causing the problem. Can anyone offer pointers on how to debug this? Thx Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 21:30 Problems writing to intel P3700 NVMe drive Pavilion Storage @ 2015-05-27 22:34 ` Greg Schulz 2015-05-27 22:40 ` Pavilion Storage 0 siblings, 1 reply; 18+ messages in thread From: Greg Schulz @ 2015-05-27 22:34 UTC (permalink / raw) I may have missed it somewhere, what OS (and version) are you using? Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via partition or filesystem? Gs Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone 651-275-1563 | twitter @storageio | blog www.storageioblog.com Server StorageIO and UnlimitedIO LLC aka StorageIO | www.storageio.com Author of the books: Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended Reading List) The Green and Virtual Data Center (CRC Press & Intel Recommended Reading List) Resilient Storage Networks ? Designing Flexible Scalable Data Infrastructures (Elsevier) The information contained in this email and any attachments is confidential and may be legally privileged and or confidential. If you are not the intended recipient then you must not use, disseminate, distribute copy or in any other way use, the information contained in this email or attachments. If you receive this email in error, destroy?it immediately and contact us.?Thank you. -----Original Message----- From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf Of Pavilion Storage Sent: Wednesday, May 27, 2015 4:30 PM To: linux-nvme at lists.infradead.org Subject: Problems writing to intel P3700 NVMe drive I am having problems with writing to an intel P3700 NVMe drive on a ARM based platform and I would appreciate any help in debugging this problem. Here are my experiments and observations so far: - The drive is usable for read operations and I can run fio (randread, etc) but I cannot write to it. - Other manufacturer's NVMe drive can be inserted into this platform and I can write to it - I can insert the intel P3700 into an x86 host and I can perform write operations - I have tried using the nvme-cli (https://github.com/linux-nvme/nvme-cli) tool and have read the error codes from read. It just shows 'WRITE_FAULT' which is not very helpful. - I have looked at the SmartLog reports and the 'Critical Warning' is 0, i.e. the device is not in Read-Only mode - I have updated the firmware using the intel SSD-DataCenterTool and updated the firmware on the drive the latest available version. The updated drive works fine on x86 platform. I think there is some combination of the platform and drive that is causing the problem. Can anyone offer pointers on how to debug this? Thx Kishore _______________________________________________ Linux-nvme mailing list Linux-nvme at lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 22:34 ` Greg Schulz @ 2015-05-27 22:40 ` Pavilion Storage 2015-05-27 23:05 ` Greg Schulz 0 siblings, 1 reply; 18+ messages in thread From: Pavilion Storage @ 2015-05-27 22:40 UTC (permalink / raw) Greg, I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). I have tried accessing with dd, nvme-cli and fio Thx Kishore On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: > I may have missed it somewhere, what OS (and version) are you using? > Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via > partition or filesystem? > Gs > > > Greg Schulz | Sr. Advisory Analyst | greg at storageio.com > Phone 651-275-1563 | twitter @storageio | blog www.storageioblog.com > Server StorageIO and UnlimitedIO LLC aka StorageIO | www.storageio.com > > Author of the books: > Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended > Reading List) The Green and Virtual Data Center (CRC Press & Intel > Recommended Reading List) Resilient Storage Networks ? Designing Flexible > Scalable Data Infrastructures (Elsevier) > > The information contained in this email and any attachments is confidential > and may be legally privileged and or confidential. If you are not the > intended recipient then you must not use, disseminate, distribute copy or in > any other way use, the information contained in this email or attachments. > If you receive this email in error, destroy it immediately and contact > us. Thank you. > > -----Original Message----- > From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf > Of Pavilion Storage > Sent: Wednesday, May 27, 2015 4:30 PM > To: linux-nvme at lists.infradead.org > Subject: Problems writing to intel P3700 NVMe drive > > I am having problems with writing to an intel P3700 NVMe drive on a ARM > based platform and I would appreciate any help in debugging this problem. > Here are my experiments and observations so far: > > - The drive is usable for read operations and I can run fio (randread, > etc) but I cannot write to it. > - Other manufacturer's NVMe drive can be inserted into this platform and > I can write to it > - I can insert the intel P3700 into an x86 host and I can perform write > operations > - I have tried using the nvme-cli > (https://github.com/linux-nvme/nvme-cli) tool and have read the error codes > from read. It just shows 'WRITE_FAULT' which is not very helpful. > - I have looked at the SmartLog reports and the 'Critical Warning' > is 0, i.e. the device is not in Read-Only mode > - I have updated the firmware using the intel SSD-DataCenterTool and > updated the firmware on the drive the latest available version. > The updated drive works fine on x86 platform. > > I think there is some combination of the platform and drive that is causing > the problem. Can anyone offer pointers on how to debug this? > > Thx > Kishore > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 22:40 ` Pavilion Storage @ 2015-05-27 23:05 ` Greg Schulz 2015-05-27 23:53 ` Pavilion Storage 0 siblings, 1 reply; 18+ messages in thread From: Greg Schulz @ 2015-05-27 23:05 UTC (permalink / raw) Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine. I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else. What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc Cheers gs -----Original Message----- From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf Of Pavilion Storage Sent: Wednesday, May 27, 2015 5:41 PM To: greg at storageio.com Cc: linux-nvme at lists.infradead.org Subject: Re: Problems writing to intel P3700 NVMe drive Greg, I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). I have tried accessing with dd, nvme-cli and fio Thx Kishore On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: > I may have missed it somewhere, what OS (and version) are you using? > Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via > partition or filesystem? > Gs > > > Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone > 651-275-1563 | twitter @storageio | blog www.storageioblog.com > Server StorageIO and UnlimitedIO LLC aka StorageIO | > www.storageio.com > > Author of the books: > Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended > Reading List) The Green and Virtual Data Center (CRC Press & Intel > Recommended Reading List) Resilient Storage Networks ? Designing > Flexible Scalable Data Infrastructures (Elsevier) > > The information contained in this email and any attachments is > confidential and may be legally privileged and or confidential. If you > are not the intended recipient then you must not use, disseminate, > distribute copy or in any other way use, the information contained in this email or attachments. > If you receive this email in error, destroy it immediately and contact > us. Thank you. > > -----Original Message----- > From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On > Behalf Of Pavilion Storage > Sent: Wednesday, May 27, 2015 4:30 PM > To: linux-nvme at lists.infradead.org > Subject: Problems writing to intel P3700 NVMe drive > > I am having problems with writing to an intel P3700 NVMe drive on a > ARM based platform and I would appreciate any help in debugging this problem. > Here are my experiments and observations so far: > > - The drive is usable for read operations and I can run fio > (randread, > etc) but I cannot write to it. > - Other manufacturer's NVMe drive can be inserted into this > platform and I can write to it > - I can insert the intel P3700 into an x86 host and I can perform > write operations > - I have tried using the nvme-cli > (https://github.com/linux-nvme/nvme-cli) tool and have read the error > codes from read. It just shows 'WRITE_FAULT' which is not very helpful. > - I have looked at the SmartLog reports and the 'Critical Warning' > is 0, i.e. the device is not in Read-Only mode > - I have updated the firmware using the intel SSD-DataCenterTool > and updated the firmware on the drive the latest available version. > The updated drive works fine on x86 platform. > > I think there is some combination of the platform and drive that is > causing the problem. Can anyone offer pointers on how to debug this? > > Thx > Kishore > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme > _______________________________________________ Linux-nvme mailing list Linux-nvme at lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 23:05 ` Greg Schulz @ 2015-05-27 23:53 ` Pavilion Storage 2015-05-28 0:24 ` j cagle ` (2 more replies) 0 siblings, 3 replies; 18+ messages in thread From: Pavilion Storage @ 2015-05-27 23:53 UTC (permalink / raw) Greg Here is the output from the dd command on a good (non-intel; /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no difference in displayed output of dd, but the write is successful in the non-intel drive. The nvme-cli command tool correctly shows the failure to write, but offers no further clues beyond the failed status. thx Kishore ------------------------------------------------------- [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc 0+0 records in 0+0 records out 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s [root]# ----------------------------------------------------- [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s [root]# --------------------------------------------------------- [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null write: success [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null write:WRITE_FAULT(2280) --------------------------------------------------------- On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote: > Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine. > I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else. > What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc > Cheers gs > > > -----Original Message----- > From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage > Sent: Wednesday, May 27, 2015 5:41 PM > To: greg at storageio.com > Cc: linux-nvme at lists.infradead.org > Subject: Re: Problems writing to intel P3700 NVMe drive > > Greg, > I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). > > I have tried accessing with dd, nvme-cli and fio Thx Kishore > > On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: >> I may have missed it somewhere, what OS (and version) are you using? >> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via >> partition or filesystem? >> Gs >> >> >> Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone >> 651-275-1563 | twitter @storageio | blog www.storageioblog.com >> Server StorageIO and UnlimitedIO LLC aka StorageIO | >> www.storageio.com >> >> Author of the books: >> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended >> Reading List) The Green and Virtual Data Center (CRC Press & Intel >> Recommended Reading List) Resilient Storage Networks ? Designing >> Flexible Scalable Data Infrastructures (Elsevier) >> >> The information contained in this email and any attachments is >> confidential and may be legally privileged and or confidential. If you >> are not the intended recipient then you must not use, disseminate, >> distribute copy or in any other way use, the information contained in this email or attachments. >> If you receive this email in error, destroy it immediately and contact >> us. Thank you. >> >> -----Original Message----- >> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On >> Behalf Of Pavilion Storage >> Sent: Wednesday, May 27, 2015 4:30 PM >> To: linux-nvme at lists.infradead.org >> Subject: Problems writing to intel P3700 NVMe drive >> >> I am having problems with writing to an intel P3700 NVMe drive on a >> ARM based platform and I would appreciate any help in debugging this problem. >> Here are my experiments and observations so far: >> >> - The drive is usable for read operations and I can run fio >> (randread, >> etc) but I cannot write to it. >> - Other manufacturer's NVMe drive can be inserted into this >> platform and I can write to it >> - I can insert the intel P3700 into an x86 host and I can perform >> write operations >> - I have tried using the nvme-cli >> (https://github.com/linux-nvme/nvme-cli) tool and have read the error >> codes from read. It just shows 'WRITE_FAULT' which is not very helpful. >> - I have looked at the SmartLog reports and the 'Critical Warning' >> is 0, i.e. the device is not in Read-Only mode >> - I have updated the firmware using the intel SSD-DataCenterTool >> and updated the firmware on the drive the latest available version. >> The updated drive works fine on x86 platform. >> >> I think there is some combination of the platform and drive that is >> causing the problem. Can anyone offer pointers on how to debug this? >> >> Thx >> Kishore >> >> _______________________________________________ >> Linux-nvme mailing list >> Linux-nvme at lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-nvme >> > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 23:53 ` Pavilion Storage @ 2015-05-28 0:24 ` j cagle 2015-05-28 0:37 ` Pavilion Storage 2015-05-28 0:46 ` Pavilion Storage [not found] ` <CA+At7pWf6Ge3pc7=aeiqz=iWxtYJSPVdM_W2XW9pL4d22q432Q@mail.gmail.com> 2 siblings, 1 reply; 18+ messages in thread From: j cagle @ 2015-05-28 0:24 UTC (permalink / raw) Hi Kishore, dd won't write anything if you use if=/dev/null. You should use if=/dev/zero instead. (This is probably what Greg intended.) John On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage <storagepavilion@gmail.com> wrote: > Greg > > Here is the output from the dd command on a good (non-intel; > /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no > difference in displayed output of dd, but the write is successful in > the non-intel drive. The nvme-cli command tool correctly shows the > failure to write, but offers no further clues beyond the failed > status. > > thx > Kishore > > ------------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s > [root]# > ----------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s > [root]# > --------------------------------------------------------- > [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null > write: success > [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null > write:WRITE_FAULT(2280) > --------------------------------------------------------- > > On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote: >> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine. >> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else. >> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc >> Cheers gs >> >> >> -----Original Message----- >> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage >> Sent: Wednesday, May 27, 2015 5:41 PM >> To: greg at storageio.com >> Cc: linux-nvme at lists.infradead.org >> Subject: Re: Problems writing to intel P3700 NVMe drive >> >> Greg, >> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). >> >> I have tried accessing with dd, nvme-cli and fio Thx Kishore >> >> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: >>> I may have missed it somewhere, what OS (and version) are you using? >>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via >>> partition or filesystem? >>> Gs >>> >>> >>> Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone >>> 651-275-1563 | twitter @storageio | blog www.storageioblog.com >>> Server StorageIO and UnlimitedIO LLC aka StorageIO | >>> www.storageio.com >>> >>> Author of the books: >>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended >>> Reading List) The Green and Virtual Data Center (CRC Press & Intel >>> Recommended Reading List) Resilient Storage Networks ? Designing >>> Flexible Scalable Data Infrastructures (Elsevier) >>> >>> The information contained in this email and any attachments is >>> confidential and may be legally privileged and or confidential. If you >>> are not the intended recipient then you must not use, disseminate, >>> distribute copy or in any other way use, the information contained in this email or attachments. >>> If you receive this email in error, destroy it immediately and contact >>> us. Thank you. >>> >>> -----Original Message----- >>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On >>> Behalf Of Pavilion Storage >>> Sent: Wednesday, May 27, 2015 4:30 PM >>> To: linux-nvme at lists.infradead.org >>> Subject: Problems writing to intel P3700 NVMe drive >>> >>> I am having problems with writing to an intel P3700 NVMe drive on a >>> ARM based platform and I would appreciate any help in debugging this problem. >>> Here are my experiments and observations so far: >>> >>> - The drive is usable for read operations and I can run fio >>> (randread, >>> etc) but I cannot write to it. >>> - Other manufacturer's NVMe drive can be inserted into this >>> platform and I can write to it >>> - I can insert the intel P3700 into an x86 host and I can perform >>> write operations >>> - I have tried using the nvme-cli >>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error >>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful. >>> - I have looked at the SmartLog reports and the 'Critical Warning' >>> is 0, i.e. the device is not in Read-Only mode >>> - I have updated the firmware using the intel SSD-DataCenterTool >>> and updated the firmware on the drive the latest available version. >>> The updated drive works fine on x86 platform. >>> >>> I think there is some combination of the platform and drive that is >>> causing the problem. Can anyone offer pointers on how to debug this? >>> >>> Thx >>> Kishore >>> >>> _______________________________________________ >>> Linux-nvme mailing list >>> Linux-nvme at lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-nvme >>> >> >> _______________________________________________ >> Linux-nvme mailing list >> Linux-nvme at lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-nvme >> > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-28 0:24 ` j cagle @ 2015-05-28 0:37 ` Pavilion Storage 0 siblings, 0 replies; 18+ messages in thread From: Pavilion Storage @ 2015-05-28 0:37 UTC (permalink / raw) John Here is the outputs with if=/dev/zero and oflag=direct and you can see the error. thx Kishore ------------------------------------------------------ [root]# dd if=/dev/zero of=/dev/nvme0n1 oflag=direct bs=512 count=1000000 conv=notrunc 1000000+0 records in 1000000+0 records out 512000000 bytes (512 MB) copied, 77.6635 s, 6.6 MB/s ----------------------------------------------------------- [root]# dd if=/dev/zero of=/dev/nvme1n1 oflag=direct bs=512 count=1000000 conv=notrunc dd: writing '/dev/nvme1n1': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0212549 s, 0.0 kB/s ------------------------------------------------------------- [root]# ./nvme write /dev/nvme0n1 -s 0 -z 4096 -d /dev/zero write: success ------------------------------------------------------------- [root]# ./nvme write /dev/nvme1n1 -s 0 -z 4096 -d /dev/zero write:WRITE_FAULT(2280) [root]# On Wed, May 27, 2015@5:24 PM, j cagle <jcagle@gmail.com> wrote: > Hi Kishore, > > dd won't write anything if you use if=/dev/null. You should use > if=/dev/zero instead. (This is probably what Greg intended.) > > John > > On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage > <storagepavilion@gmail.com> wrote: >> Greg >> >> Here is the output from the dd command on a good (non-intel; >> /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no >> difference in displayed output of dd, but the write is successful in >> the non-intel drive. The nvme-cli command tool correctly shows the >> failure to write, but offers no further clues beyond the failed >> status. >> >> thx >> Kishore >> >> ------------------------------------------------------- >> [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc >> 0+0 records in >> 0+0 records out >> 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s >> [root]# >> ----------------------------------------------------- >> [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc >> 0+0 records in >> 0+0 records out >> 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s >> [root]# >> --------------------------------------------------------- >> [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null >> write: success >> [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null >> write:WRITE_FAULT(2280) >> --------------------------------------------------------- >> >> On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote: >>> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine. >>> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else. >>> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc >>> Cheers gs >>> >>> >>> -----Original Message----- >>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage >>> Sent: Wednesday, May 27, 2015 5:41 PM >>> To: greg at storageio.com >>> Cc: linux-nvme at lists.infradead.org >>> Subject: Re: Problems writing to intel P3700 NVMe drive >>> >>> Greg, >>> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). >>> >>> I have tried accessing with dd, nvme-cli and fio Thx Kishore >>> >>> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: >>>> I may have missed it somewhere, what OS (and version) are you using? >>>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via >>>> partition or filesystem? >>>> Gs >>>> >>>> >>>> Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone >>>> 651-275-1563 | twitter @storageio | blog www.storageioblog.com >>>> Server StorageIO and UnlimitedIO LLC aka StorageIO | >>>> www.storageio.com >>>> >>>> Author of the books: >>>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended >>>> Reading List) The Green and Virtual Data Center (CRC Press & Intel >>>> Recommended Reading List) Resilient Storage Networks ? Designing >>>> Flexible Scalable Data Infrastructures (Elsevier) >>>> >>>> The information contained in this email and any attachments is >>>> confidential and may be legally privileged and or confidential. If you >>>> are not the intended recipient then you must not use, disseminate, >>>> distribute copy or in any other way use, the information contained in this email or attachments. >>>> If you receive this email in error, destroy it immediately and contact >>>> us. Thank you. >>>> >>>> -----Original Message----- >>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On >>>> Behalf Of Pavilion Storage >>>> Sent: Wednesday, May 27, 2015 4:30 PM >>>> To: linux-nvme at lists.infradead.org >>>> Subject: Problems writing to intel P3700 NVMe drive >>>> >>>> I am having problems with writing to an intel P3700 NVMe drive on a >>>> ARM based platform and I would appreciate any help in debugging this problem. >>>> Here are my experiments and observations so far: >>>> >>>> - The drive is usable for read operations and I can run fio >>>> (randread, >>>> etc) but I cannot write to it. >>>> - Other manufacturer's NVMe drive can be inserted into this >>>> platform and I can write to it >>>> - I can insert the intel P3700 into an x86 host and I can perform >>>> write operations >>>> - I have tried using the nvme-cli >>>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error >>>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful. >>>> - I have looked at the SmartLog reports and the 'Critical Warning' >>>> is 0, i.e. the device is not in Read-Only mode >>>> - I have updated the firmware using the intel SSD-DataCenterTool >>>> and updated the firmware on the drive the latest available version. >>>> The updated drive works fine on x86 platform. >>>> >>>> I think there is some combination of the platform and drive that is >>>> causing the problem. Can anyone offer pointers on how to debug this? >>>> >>>> Thx >>>> Kishore >>>> >>>> _______________________________________________ >>>> Linux-nvme mailing list >>>> Linux-nvme at lists.infradead.org >>>> http://lists.infradead.org/mailman/listinfo/linux-nvme >>>> >>> >>> _______________________________________________ >>> Linux-nvme mailing list >>> Linux-nvme at lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-nvme >>> >> >> _______________________________________________ >> Linux-nvme mailing list >> Linux-nvme at lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-27 23:53 ` Pavilion Storage 2015-05-28 0:24 ` j cagle @ 2015-05-28 0:46 ` Pavilion Storage [not found] ` <CA+At7pWf6Ge3pc7=aeiqz=iWxtYJSPVdM_W2XW9pL4d22q432Q@mail.gmail.com> 2 siblings, 0 replies; 18+ messages in thread From: Pavilion Storage @ 2015-05-28 0:46 UTC (permalink / raw) Greg I have verified that the same drive is working in an x86 system with the same firmware. Also, this appears to be a problem with multiple intel P3700 SSD drives, not just one. I have tried other manufacturers drives in the same slots on our ARM platform and verified that the platform (and the slot) is working. Yes, these are PCIe x4 Gen3 slots. >From the error log the (StatusField.SCT and StatusField.SC appear to 0x2 and 0x80) which for a write command is LBA out of range, but that does not make sense because the error is the same regardless of the LBA address used. thx Kishore On Wed, May 27, 2015@5:09 PM, Greg Schulz <greg@storageio.com> wrote: > Interesting, have you tried the bad card in a different x86 based server to see if it is the card? > Also, if the card is good, yet wont write in the specific server, have you checked the servers BIOS, granted if you have one card working and one not, that?s a different ball game. > Not sure if there is an ARM variation of the intel UEFI sysinfo tool that would give you more info. > Whats not clear is if you are getting an actual write protect error or simply a write fault. > Write protect are usually an indicator of something actually wrong with the underlying nand flash, or its FTL and with a good controller, it will try to work around before giving up. > Write fault could be a write-protect, however could also be a driver, bios, firmware or even PCIe slot issue. > Have not tried the 3700 in a PCIe G2 slot, assume you have them in a PCIe G3 slot or riser? > Good luck > Cheers > Gs > On Wed, May 27, 2015 at 4:53 PM, Pavilion Storage <storagepavilion@gmail.com> wrote: > Greg > > Here is the output from the dd command on a good (non-intel; > /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no > difference in displayed output of dd, but the write is successful in > the non-intel drive. The nvme-cli command tool correctly shows the > failure to write, but offers no further clues beyond the failed > status. > > thx > Kishore > > ------------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s > [root]# > ----------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s > [root]# > --------------------------------------------------------- > [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null > write: success > [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null > write:WRITE_FAULT(2280) > --------------------------------------------------------- > > On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote: >> Thanks for the follow-up and extra info, I have 3700s running on x86/64 Ubuntu 14.04 and working fine. >> I did update the drivers from the Intel site even though they were already in the 14.04 kernel when I was troubel shooting something else. >> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 count=1000000 conv=notrunc >> Cheers gs >> >> >> -----Original Message----- >> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf Of Pavilion Storage >> Sent: Wednesday, May 27, 2015 5:41 PM >> To: greg at storageio.com >> Cc: linux-nvme at lists.infradead.org >> Subject: Re: Problems writing to intel P3700 NVMe drive >> >> Greg, >> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device (/dev/nvme1). >> >> I have tried accessing with dd, nvme-cli and fio Thx Kishore >> >> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: >>> I may have missed it somewhere, what OS (and version) are you using? >>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via >>> partition or filesystem? >>> Gs >>> >>> >>> Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone >>> 651-275-1563 | twitter @storageio | blog www.storageioblog.com >>> Server StorageIO and UnlimitedIO LLC aka StorageIO | >>> www.storageio.com >>> >>> Author of the books: >>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended >>> Reading List) The Green and Virtual Data Center (CRC Press & Intel >>> Recommended Reading List) Resilient Storage Networks ? Designing >>> Flexible Scalable Data Infrastructures (Elsevier) >>> >>> The information contained in this email and any attachments is >>> confidential and may be legally privileged and or confidential. If you >>> are not the intended recipient then you must not use, disseminate, >>> distribute copy or in any other way use, the information contained in this email or attachments. >>> If you receive this email in error, destroy it immediately and contact >>> us. Thank you. >>> >>> -----Original Message----- >>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On >>> Behalf Of Pavilion Storage >>> Sent: Wednesday, May 27, 2015 4:30 PM >>> To: linux-nvme at lists.infradead.org >>> Subject: Problems writing to intel P3700 NVMe drive >>> >>> I am having problems with writing to an intel P3700 NVMe drive on a >>> ARM based platform and I would appreciate any help in debugging this problem. >>> Here are my experiments and observations so far: >>> >>> - The drive is usable for read operations and I can run fio >>> (randread, >>> etc) but I cannot write to it. >>> - Other manufacturer's NVMe drive can be inserted into this >>> platform and I can write to it >>> - I can insert the intel P3700 into an x86 host and I can perform >>> write operations >>> - I have tried using the nvme-cli >>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error >>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful. >>> - I have looked at the SmartLog reports and the 'Critical Warning' >>> is 0, i.e. the device is not in Read-Only mode >>> - I have updated the firmware using the intel SSD-DataCenterTool >>> and updated the firmware on the drive the latest available version. >>> The updated drive works fine on x86 platform. >>> >>> I think there is some combination of the platform and drive that is >>> causing the problem. Can anyone offer pointers on how to debug this? >>> >>> Thx >>> Kishore >>> >>> _______________________________________________ >>> Linux-nvme mailing list >>> Linux-nvme at lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-nvme >>> >> >> _______________________________________________ >> Linux-nvme mailing list >> Linux-nvme at lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-nvme >> ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CA+At7pWf6Ge3pc7=aeiqz=iWxtYJSPVdM_W2XW9pL4d22q432Q@mail.gmail.com>]
[parent not found: <000a01d098ee$b52ce0c0$1f86a240$@storageio.com>]
* Problems writing to intel P3700 NVMe drive [not found] ` <000a01d098ee$b52ce0c0$1f86a240$@storageio.com> @ 2015-05-28 20:53 ` Pavilion Storage 2015-05-28 21:03 ` Keith Busch 0 siblings, 1 reply; 18+ messages in thread From: Pavilion Storage @ 2015-05-28 20:53 UTC (permalink / raw) Thanks to Keith Busch for the providing the clues. The problem was that the PCIe MPSS on the drive and the CPU root complex did not match and that caused a problem. Setting it to a common value fixed the problem. Kishore On Wed, May 27, 2015@7:33 PM, Greg Schulz <greg@storageio.com> wrote: > No worries John, my bad, yes you are correct, s/b if=/dev/zero ;) > > Cheers > > Gs > > > > > > From: j cagle [mailto:jcagle at gmail.com] > Sent: Wednesday, May 27, 2015 7:19 PM > To: Pavilion Storage > Cc: greg schulz; linux-nvme at lists.infradead.org > Subject: Re: Problems writing to intel P3700 NVMe drive > > > > Hi Kishore, > > > > dd won't write anything if you use if=/dev/null. You should use > if=/dev/zero instead. (This is probably what Greg intended.) > > > > John > > > > > > On Wed, May 27, 2015 at 6:53 PM, Pavilion Storage > <storagepavilion@gmail.com> wrote: > > Greg > > Here is the output from the dd command on a good (non-intel; > /dev/nvme0n1) and bad (intel drive; /dev/nvme1n1). There is no > difference in displayed output of dd, but the write is successful in > the non-intel drive. The nvme-cli command tool correctly shows the > failure to write, but offers no further clues beyond the failed > status. > > thx > Kishore > > ------------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvmen0n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 1.62e-05 s, 0.0 kB/s > [root]# > ----------------------------------------------------- > [root]# dd if=/dev/null of=/dev/nvme1n1 bs=512 count=1000000 conv=notrunc > 0+0 records in > 0+0 records out > 0 bytes (0 B) copied, 0.00032024 s, 0.0 kB/s > [root]# > --------------------------------------------------------- > [root]# ./nvme-cli write /dev/nvme0n1 -s 0 -z 4096 -d /dev/null > write: success > [root]# ./nvme-cli write /dev/nvme1n1 -s 0 -z 4096 -d /dev/null > write:WRITE_FAULT(2280) > --------------------------------------------------------- > > > On Wed, May 27, 2015@4:05 PM, Greg Schulz <greg@storageio.com> wrote: >> Thanks for the follow-up and extra info, I have 3700s running on x86/64 >> Ubuntu 14.04 and working fine. >> I did update the drivers from the Intel site even though they were already >> in the 14.04 kernel when I was troubel shooting something else. >> What happens if you do a dd if=/dev/null of=/dev/nvme0n1 bs=512 >> count=1000000 conv=notrunc >> Cheers gs >> >> >> -----Original Message----- >> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On Behalf >> Of Pavilion Storage >> Sent: Wednesday, May 27, 2015 5:41 PM >> To: greg at storageio.com >> Cc: linux-nvme at lists.infradead.org >> Subject: Re: Problems writing to intel P3700 NVMe drive >> >> Greg, >> I am running this on Linux (Ubuntu/3.12) and I am accessing the raw device >> (/dev/nvme1). >> >> I have tried accessing with dd, nvme-cli and fio Thx Kishore >> >> On Wed, May 27, 2015@3:34 PM, Greg Schulz <greg@storageio.com> wrote: >>> I may have missed it somewhere, what OS (and version) are you using? >>> Also are you accessing the nvme device e.g. /dev/nvme0n1 raw or via >>> partition or filesystem? >>> Gs >>> >>> >>> Greg Schulz | Sr. Advisory Analyst | greg at storageio.com Phone >>> 651-275-1563 | twitter @storageio | blog www.storageioblog.com >>> Server StorageIO and UnlimitedIO LLC aka StorageIO | >>> www.storageio.com >>> >>> Author of the books: >>> Cloud and Virtual Data Storage Network (CRC Press & Intel Recommended >>> Reading List) The Green and Virtual Data Center (CRC Press & Intel >>> Recommended Reading List) Resilient Storage Networks ? Designing >>> Flexible Scalable Data Infrastructures (Elsevier) >>> >>> The information contained in this email and any attachments is >>> confidential and may be legally privileged and or confidential. If you >>> are not the intended recipient then you must not use, disseminate, >>> distribute copy or in any other way use, the information contained in >>> this email or attachments. >>> If you receive this email in error, destroy it immediately and contact >>> us. Thank you. >>> >>> -----Original Message----- >>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On >>> Behalf Of Pavilion Storage >>> Sent: Wednesday, May 27, 2015 4:30 PM >>> To: linux-nvme at lists.infradead.org >>> Subject: Problems writing to intel P3700 NVMe drive >>> >>> I am having problems with writing to an intel P3700 NVMe drive on a >>> ARM based platform and I would appreciate any help in debugging this >>> problem. >>> Here are my experiments and observations so far: >>> >>> - The drive is usable for read operations and I can run fio >>> (randread, >>> etc) but I cannot write to it. >>> - Other manufacturer's NVMe drive can be inserted into this >>> platform and I can write to it >>> - I can insert the intel P3700 into an x86 host and I can perform >>> write operations >>> - I have tried using the nvme-cli >>> (https://github.com/linux-nvme/nvme-cli) tool and have read the error >>> codes from read. It just shows 'WRITE_FAULT' which is not very helpful. >>> - I have looked at the SmartLog reports and the 'Critical Warning' >>> is 0, i.e. the device is not in Read-Only mode >>> - I have updated the firmware using the intel SSD-DataCenterTool >>> and updated the firmware on the drive the latest available version. >>> The updated drive works fine on x86 platform. >>> >>> I think there is some combination of the platform and drive that is >>> causing the problem. Can anyone offer pointers on how to debug this? >>> >>> Thx >>> Kishore >>> >>> _______________________________________________ >>> Linux-nvme mailing list >>> Linux-nvme at lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-nvme >>> >> >> _______________________________________________ >> Linux-nvme mailing list >> Linux-nvme at lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-nvme >> > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme > > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-28 20:53 ` Pavilion Storage @ 2015-05-28 21:03 ` Keith Busch 2015-05-28 21:19 ` Pavilion Storage 2015-06-03 20:10 ` Keith Busch 0 siblings, 2 replies; 18+ messages in thread From: Keith Busch @ 2015-05-28 21:03 UTC (permalink / raw) Perhaps not a desirable/permanent solution, but for everyone's benefit, we can work-around the problem with kernel parameter: pci=pcie_bus_safe Using 'pcie_bus_perf' instead may also work, though your original solution of just using x86 sounded perfectly reasonable to me. :) On Thu, 28 May 2015, Pavilion Storage wrote: > Thanks to Keith Busch for the providing the clues. > The problem was that the PCIe MPSS on the drive and the CPU root > complex did not match and that caused a problem. Setting it to a > common value fixed the problem. > Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-28 21:03 ` Keith Busch @ 2015-05-28 21:19 ` Pavilion Storage 2015-05-28 21:31 ` Keith Busch 2015-06-03 20:10 ` Keith Busch 1 sibling, 1 reply; 18+ messages in thread From: Pavilion Storage @ 2015-05-28 21:19 UTC (permalink / raw) The root cause might be a bug in the kernel where pcie devices are configured. Setting it to 128B will work, but may not perform to the best capabilities allowed by the device and RC. If we have multiple drives from different manufacturers in the system, we may penalize the drive that has the hardware capability to handle higher MPSS. thx Kishore On Thu, May 28, 2015@2:03 PM, Keith Busch <keith.busch@intel.com> wrote: > Perhaps not a desirable/permanent solution, but for everyone's benefit, > we can work-around the problem with kernel parameter: > > pci=pcie_bus_safe > > Using 'pcie_bus_perf' instead may also work, though your original > solution of just using x86 sounded perfectly reasonable to me. :) > > On Thu, 28 May 2015, Pavilion Storage wrote: >> >> Thanks to Keith Busch for the providing the clues. >> The problem was that the PCIe MPSS on the drive and the CPU root >> complex did not match and that caused a problem. Setting it to a >> common value fixed the problem. >> Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-28 21:19 ` Pavilion Storage @ 2015-05-28 21:31 ` Keith Busch 0 siblings, 0 replies; 18+ messages in thread From: Keith Busch @ 2015-05-28 21:31 UTC (permalink / raw) Using 'pcie_bus_perf' will work just fine in most cases, selecting the highest capable settings so as not to penalize other h/w. Though all your controllers had the same max payload capabilities anyway, so they should all be the same from the max TLP perspective. On Thu, 28 May 2015, Pavilion Storage wrote: > The root cause might be a bug in the kernel where pcie devices are > configured. Setting it to 128B will work, but may not perform to the > best capabilities allowed by the device and RC. If we have multiple > drives from different manufacturers in the system, we may penalize the > drive that has the hardware capability to handle higher MPSS. > > thx > Kishore > > > On Thu, May 28, 2015@2:03 PM, Keith Busch <keith.busch@intel.com> wrote: >> Perhaps not a desirable/permanent solution, but for everyone's benefit, >> we can work-around the problem with kernel parameter: >> >> pci=pcie_bus_safe >> >> Using 'pcie_bus_perf' instead may also work, though your original >> solution of just using x86 sounded perfectly reasonable to me. :) >> >> On Thu, 28 May 2015, Pavilion Storage wrote: >>> >>> Thanks to Keith Busch for the providing the clues. >>> The problem was that the PCIe MPSS on the drive and the CPU root >>> complex did not match and that caused a problem. Setting it to a >>> common value fixed the problem. >>> Kishore > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-05-28 21:03 ` Keith Busch @ 2015-06-03 20:10 ` Keith Busch 2015-06-03 20:10 ` Keith Busch 1 sibling, 0 replies; 18+ messages in thread From: Keith Busch @ 2015-06-03 20:10 UTC (permalink / raw) +cc the pci list Hi Bjorn, A while back, there were a few proposals on changing the pci driver's default MPS tuning from the existing do-nothing policy to something safe so end-users don't need to remember kernel parameters as described below. Is this still active, or can we kick that back to life if not? Thanks, Keith On Thu, 28 May 2015, Keith Busch wrote: > Perhaps not a desirable/permanent solution, but for everyone's benefit, > we can work-around the problem with kernel parameter: > > pci=pcie_bus_safe > > Using 'pcie_bus_perf' instead may also work, though your original > solution of just using x86 sounded perfectly reasonable to me. :) > > On Thu, 28 May 2015, Pavilion Storage wrote: >> Thanks to Keith Busch for the providing the clues. >> The problem was that the PCIe MPSS on the drive and the CPU root >> complex did not match and that caused a problem. Setting it to a >> common value fixed the problem. >> Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Problems writing to intel P3700 NVMe drive @ 2015-06-03 20:10 ` Keith Busch 0 siblings, 0 replies; 18+ messages in thread From: Keith Busch @ 2015-06-03 20:10 UTC (permalink / raw) To: Keith Busch, Bjorn Helgaas; +Cc: Pavilion Storage, linux-nvme, linux-pci +cc the pci list Hi Bjorn, A while back, there were a few proposals on changing the pci driver's default MPS tuning from the existing do-nothing policy to something safe so end-users don't need to remember kernel parameters as described below. Is this still active, or can we kick that back to life if not? Thanks, Keith On Thu, 28 May 2015, Keith Busch wrote: > Perhaps not a desirable/permanent solution, but for everyone's benefit, > we can work-around the problem with kernel parameter: > > pci=pcie_bus_safe > > Using 'pcie_bus_perf' instead may also work, though your original > solution of just using x86 sounded perfectly reasonable to me. :) > > On Thu, 28 May 2015, Pavilion Storage wrote: >> Thanks to Keith Busch for the providing the clues. >> The problem was that the PCIe MPSS on the drive and the CPU root >> complex did not match and that caused a problem. Setting it to a >> common value fixed the problem. >> Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-06-03 20:10 ` Keith Busch @ 2015-06-03 20:20 ` Bjorn Helgaas -1 siblings, 0 replies; 18+ messages in thread From: Bjorn Helgaas @ 2015-06-03 20:20 UTC (permalink / raw) On Wed, Jun 3, 2015@3:10 PM, Keith Busch <keith.busch@intel.com> wrote: > +cc the pci list > > Hi Bjorn, > > A while back, there were a few proposals on changing the pci driver's > default MPS tuning from the existing do-nothing policy to something > safe so end-users don't need to remember kernel parameters as described > below. Is this still active, or can we kick that back to life if not? I'd love to see some activity there. Somebody else asked me about this a few weeks ago, and I sent him this list of some of the things I think are wrong with the current situation: - MPS configuration is done in pcie_bus_configure_settings(). This is not arch-specific, but it is done by arch code, and only arm, powerpc, tile, and x86 call it. - MPS configuration should be done before a driver claims a device. This is currently wrong on arm. - I would rather have MPS configuration done somewhere in the pci_scan_single_device() path. That way it would be done on every arch and in every hotplug path. - I know MPS depends on more than just the individual device and its upstream bridge, so it's not as simple as some other properties. But if we have a correctly-configured tree, we should be able to figure out what needs to change if we add one more device to it. - I do not believe it is safe for drivers to manipulate MPS behind the back of the PCI core, so pcie_set_mps() should probably not be exported. 5f39e6705faa ("PCI: Disable MPS configuration by default") turned off most the tuning we used to do. I expect that if we turn it back on, we'll trip over a few issues. That in itself is not so much of a problem, but before we turn things back on, I want to have better infrastructure in place that makes it easier to diagnose and fix those issues. For example, I'd like to see just enough clues in dmesg to enable us to see what the kernel's doing and verify that it's correct. And I want it do be done cleanly on every arch for every device, with hot-added ones being handled the same as those present at boot. > On Thu, 28 May 2015, Keith Busch wrote: >> >> Perhaps not a desirable/permanent solution, but for everyone's benefit, >> we can work-around the problem with kernel parameter: >> >> pci=pcie_bus_safe >> >> Using 'pcie_bus_perf' instead may also work, though your original >> solution of just using x86 sounded perfectly reasonable to me. :) >> >> On Thu, 28 May 2015, Pavilion Storage wrote: >>> >>> Thanks to Keith Busch for the providing the clues. >>> The problem was that the PCIe MPSS on the drive and the CPU root >>> complex did not match and that caused a problem. Setting it to a >>> common value fixed the problem. >>> Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Problems writing to intel P3700 NVMe drive @ 2015-06-03 20:20 ` Bjorn Helgaas 0 siblings, 0 replies; 18+ messages in thread From: Bjorn Helgaas @ 2015-06-03 20:20 UTC (permalink / raw) To: Keith Busch; +Cc: Pavilion Storage, linux-nvme, linux-pci@vger.kernel.org On Wed, Jun 3, 2015 at 3:10 PM, Keith Busch <keith.busch@intel.com> wrote: > +cc the pci list > > Hi Bjorn, > > A while back, there were a few proposals on changing the pci driver's > default MPS tuning from the existing do-nothing policy to something > safe so end-users don't need to remember kernel parameters as described > below. Is this still active, or can we kick that back to life if not? I'd love to see some activity there. Somebody else asked me about this a few weeks ago, and I sent him this list of some of the things I think are wrong with the current situation: - MPS configuration is done in pcie_bus_configure_settings(). This is not arch-specific, but it is done by arch code, and only arm, powerpc, tile, and x86 call it. - MPS configuration should be done before a driver claims a device. This is currently wrong on arm. - I would rather have MPS configuration done somewhere in the pci_scan_single_device() path. That way it would be done on every arch and in every hotplug path. - I know MPS depends on more than just the individual device and its upstream bridge, so it's not as simple as some other properties. But if we have a correctly-configured tree, we should be able to figure out what needs to change if we add one more device to it. - I do not believe it is safe for drivers to manipulate MPS behind the back of the PCI core, so pcie_set_mps() should probably not be exported. 5f39e6705faa ("PCI: Disable MPS configuration by default") turned off most the tuning we used to do. I expect that if we turn it back on, we'll trip over a few issues. That in itself is not so much of a problem, but before we turn things back on, I want to have better infrastructure in place that makes it easier to diagnose and fix those issues. For example, I'd like to see just enough clues in dmesg to enable us to see what the kernel's doing and verify that it's correct. And I want it do be done cleanly on every arch for every device, with hot-added ones being handled the same as those present at boot. > On Thu, 28 May 2015, Keith Busch wrote: >> >> Perhaps not a desirable/permanent solution, but for everyone's benefit, >> we can work-around the problem with kernel parameter: >> >> pci=pcie_bus_safe >> >> Using 'pcie_bus_perf' instead may also work, though your original >> solution of just using x86 sounded perfectly reasonable to me. :) >> >> On Thu, 28 May 2015, Pavilion Storage wrote: >>> >>> Thanks to Keith Busch for the providing the clues. >>> The problem was that the PCIe MPSS on the drive and the CPU root >>> complex did not match and that caused a problem. Setting it to a >>> common value fixed the problem. >>> Kishore ^ permalink raw reply [flat|nested] 18+ messages in thread
* Problems writing to intel P3700 NVMe drive 2015-06-03 20:20 ` Bjorn Helgaas @ 2015-06-03 21:37 ` Keith Busch -1 siblings, 0 replies; 18+ messages in thread From: Keith Busch @ 2015-06-03 21:37 UTC (permalink / raw) On Wed, 3 Jun 2015, Bjorn Helgaas wrote: > On Wed, Jun 3, 2015@3:10 PM, Keith Busch <keith.busch@intel.com> wrote: >> A while back, there were a few proposals on changing the pci driver's >> default MPS tuning from the existing do-nothing policy to something >> safe so end-users don't need to remember kernel parameters as described >> below. Is this still active, or can we kick that back to life if not? > > I'd love to see some activity there. Somebody else asked me about > this a few weeks ago, and I sent him this list of some of the things I > think are wrong with the current situation: Thanks for the update. I got in contact with some comrades already working on this, so we'll post an updated proposal when it's ready. This feature is getting more attention with broader hot plug pci-e storage usage (among other reasons), so I'm sure we'll give this its due consideration this time. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Problems writing to intel P3700 NVMe drive @ 2015-06-03 21:37 ` Keith Busch 0 siblings, 0 replies; 18+ messages in thread From: Keith Busch @ 2015-06-03 21:37 UTC (permalink / raw) To: Bjorn Helgaas Cc: Keith Busch, Pavilion Storage, linux-nvme, linux-pci@vger.kernel.org On Wed, 3 Jun 2015, Bjorn Helgaas wrote: > On Wed, Jun 3, 2015 at 3:10 PM, Keith Busch <keith.busch@intel.com> wrote: >> A while back, there were a few proposals on changing the pci driver's >> default MPS tuning from the existing do-nothing policy to something >> safe so end-users don't need to remember kernel parameters as described >> below. Is this still active, or can we kick that back to life if not? > > I'd love to see some activity there. Somebody else asked me about > this a few weeks ago, and I sent him this list of some of the things I > think are wrong with the current situation: Thanks for the update. I got in contact with some comrades already working on this, so we'll post an updated proposal when it's ready. This feature is getting more attention with broader hot plug pci-e storage usage (among other reasons), so I'm sure we'll give this its due consideration this time. ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2015-06-03 21:38 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-05-27 21:30 Problems writing to intel P3700 NVMe drive Pavilion Storage
2015-05-27 22:34 ` Greg Schulz
2015-05-27 22:40 ` Pavilion Storage
2015-05-27 23:05 ` Greg Schulz
2015-05-27 23:53 ` Pavilion Storage
2015-05-28 0:24 ` j cagle
2015-05-28 0:37 ` Pavilion Storage
2015-05-28 0:46 ` Pavilion Storage
[not found] ` <CA+At7pWf6Ge3pc7=aeiqz=iWxtYJSPVdM_W2XW9pL4d22q432Q@mail.gmail.com>
[not found] ` <000a01d098ee$b52ce0c0$1f86a240$@storageio.com>
2015-05-28 20:53 ` Pavilion Storage
2015-05-28 21:03 ` Keith Busch
2015-05-28 21:19 ` Pavilion Storage
2015-05-28 21:31 ` Keith Busch
2015-06-03 20:10 ` Keith Busch
2015-06-03 20:10 ` Keith Busch
2015-06-03 20:20 ` Bjorn Helgaas
2015-06-03 20:20 ` Bjorn Helgaas
2015-06-03 21:37 ` Keith Busch
2015-06-03 21:37 ` Keith Busch
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.