fstrim error - AORUS NVMe Gen4 SSD

fstrim error - AORUS NVMe Gen4 SSD

[davidc502]

  Hello,

  I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.

  When attempting to fstrim the NVMe, I receive the following error.
  davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
  fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
  fstrim: /: FITRIM ioctl failed: Input/output error

  Anyhow, I have put some details below which might be helpful. Note 
that this NVMe is supposed to be TRIM and SMART compliant. The SMART 
outputs are available using the utility “nvme-cli”.
  I am willing to provide whatever command outputs that are needed to 
help solve this issue.

  OS= Ubuntu 18.4.2 LTS
  Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
  fstrim version =  fstrim from util-linux 2.31.1
  Firmware version for Aorus NVMe = EGFM11.0

  mount output
  /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
  /dev/nvme0n1p1 on /boot/efi type vfat 
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

  NVMe firmware version using utility ‘nvme’

  $ sudo nvme list /dev/nvme0n1

  Node             SN Model                                    Namespace 
Usage                      Format           FW Rev
  ---------------- -------------------- 
---------------------------------------- --------- 
-------------------------- ---------------- --------
  /dev/nvme0n1     SNX       GIGABYTE GP-ASM2NE6100TTTD               
1           1.00  TB /   1.00 TB    512   B +  0 B   EGFM11.0


  If there is any other additional information I can provide please let 
me know.

  Best Regards,

  David

Re: fstrim error - AORUS NVMe Gen4 SSD

[davidc502]

  Hello,

  I'm not sure my email from yesterday went through due to .txt 
requirements.  However, I've added some additional logs at the end of 
this email.

  If this email group does not deal with these type of issues, please 
let me know.

  Best Regards,

  David

  On 7/20/19 9:41 PM, davidc502 wrote:


   Hello,

   I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.

   When attempting to fstrim the NVMe, I receive the following error.
   davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
   fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
   fstrim: /: FITRIM ioctl failed: Input/output error

   Anyhow, I have put some details below which might be helpful. Note 
that this NVMe is supposed to be TRIM and SMART compliant. The SMART 
outputs are available using the utility “nvme-cli”.
   I am willing to provide whatever command outputs that are needed to 
help solve this issue.

   OS= Ubuntu 18.4.2 LTS
   Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
   fstrim version =  fstrim from util-linux 2.31.1
   Firmware version for Aorus NVMe = EGFM11.0

   mount output
   /dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)
   /dev/nvme0n1p1 on /boot/efi type vfat 
(rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

   NVMe firmware version using utility ‘nvme’

   $ sudo nvme list /dev/nvme0n1

   Node             SN Model Namespace Usage                      
Format           FW Rev
   ---------------- -------------------- 
---------------------------------------- --------- 
-------------------------- ---------------- --------
   /dev/nvme0n1     SNX       GIGABYTE GP-ASM2NE6100TTTD               
1           1.00  TB /   1.00 TB 512   B +  0 B   EGFM11.0

    More Log information

  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] print_req_error: I/O error, dev nvme0n1, 
sector 4160 flags 803
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] amd_iommu_report_page_fault: 28 callbacks 
suppressed
  [Fri Jul 19 17:05:08 2019] AMD-Vi: Event logged [IO_PAGE_FAULT 
device=01:00.0 domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] AMD-Vi: Event logged [IO_PAGE_FAULT 
device=01:00.0 domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:08 2019] print_req_error: I/O error, dev nvme0n1, 
sector 1141976 flags 803
  [Fri Jul 19 17:05:59 2019] amd_iommu_report_page_fault: 2 callbacks 
suppressed
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] print_req_error: I/O error, dev nvme0n1, 
sector 4160 flags 803
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] nvme 0000:01:00.0: AMD-Vi: Event logged 
[IO_PAGE_FAULT domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] AMD-Vi: Event logged [IO_PAGE_FAULT 
device=01:00.0 domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] AMD-Vi: Event logged [IO_PAGE_FAULT 
device=01:00.0 domain=0x0000 address=0x0 flags=0x0000]
  [Fri Jul 19 17:05:59 2019] print_req_error: I/O error, dev nvme0n1, 
sector 1141976 flags 803

  If there is any other additional information I can provide please let 
me know.

   Best Regards,

   David

Re: fstrim error - AORUS NVMe Gen4 SSD

[Ming Lei]

Hi,

On Sat, Jul 20, 2019 at 09:41:24PM -0500, davidc502 wrote:
>  Hello,
> 
>  I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.
> 
>  When attempting to fstrim the NVMe, I receive the following error.
>  davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
>  fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
>  fstrim: /: FITRIM ioctl failed: Input/output error
> 
>  Anyhow, I have put some details below which might be helpful. Note that
> this NVMe is supposed to be TRIM and SMART compliant. The SMART outputs are
> available using the utility “nvme-cli”.
>  I am willing to provide whatever command outputs that are needed to help
> solve this issue.
> 
>  OS= Ubuntu 18.4.2 LTS
>  Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
>  fstrim version =  fstrim from util-linux 2.31.1
>  Firmware version for Aorus NVMe = EGFM11.0
>

I saw discard timeout on HGST 1.6TB NVMe, not sure if yours is same with
that one.

Could you collect logs via the following steps?

Suppose your nvme disk name is /dev/nvme0n1:

1) queue limits log:

	#(cd /sys/block/nvme0n1/queue && find . -type f -exec grep -aH . {} \;)


2) NVMe IO trace

- enable nvme IO trace before running fstrim: 

	#echo 1  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
	#echo 1  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable

- run fstrim

- after the fstrim failure is triggered, disable the nvme io trace & post the log:

	#echo 0  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
	#echo 0  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable

	#cp	/sys/kernel/debug/tracing/trace /root/nvme_io_trace.log



thanks,
Ming

Re: fstrim error - AORUS NVMe Gen4 SSD

[Ming Lei]

Hi,

On Mon, Jul 22, 2019 at 09:39:07PM -0500, davidc502 wrote:
> See attached:  nvme_io_trace.log
> 
> 
> On 7/22/19 9:19 PM, Ming Lei wrote:
> > Hi,
> > 
> > On Sat, Jul 20, 2019 at 09:41:24PM -0500, davidc502 wrote:
> > >   Hello,
> > > 
> > >   I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.
> > > 
> > >   When attempting to fstrim the NVMe, I receive the following error.
> > >   davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
> > >   fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
> > >   fstrim: /: FITRIM ioctl failed: Input/output error
> > > 
> > >   Anyhow, I have put some details below which might be helpful. Note that
> > > this NVMe is supposed to be TRIM and SMART compliant. The SMART outputs are
> > > available using the utility “nvme-cli”.
> > >   I am willing to provide whatever command outputs that are needed to help
> > > solve this issue.
> > > 
> > >   OS= Ubuntu 18.4.2 LTS
> > >   Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
> > >   fstrim version =  fstrim from util-linux 2.31.1
> > >   Firmware version for Aorus NVMe = EGFM11.0
> > > 
> > I saw discard timeout on HGST 1.6TB NVMe, not sure if yours is same with
> > that one.
> > 
> > Could you collect logs via the following steps?
> > 
> > Suppose your nvme disk name is /dev/nvme0n1:
> > 
> > 1) queue limits log:
> > 
> > 	#(cd /sys/block/nvme0n1/queue && find . -type f -exec grep -aH . {} \;)
> > 
> > 
> > 2) NVMe IO trace
> > 
> > - enable nvme IO trace before running fstrim:
> > 
> > 	#echo 1  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
> > 	#echo 1  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
> > 
> > - run fstrim
> > 
> > - after the fstrim failure is triggered, disable the nvme io trace & post the log:
> > 
> > 	#echo 0  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
> > 	#echo 0  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
> > 
> > 	#cp	/sys/kernel/debug/tracing/trace /root/nvme_io_trace.log
> > 
> > 
> > 
> > thanks,
> > Ming
> 
> 
> Hello Ming
> 
> Thank you for the quick reply --  See attached

From the IO trace, discard command(nvme_cmd_dsm) is failed:

  kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_dsm nr=0, attributes=4)
          <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq: nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0, status=8198

And the returned error code is 0x8198, I am not sure how to parse the
'Command Specific Status Values' of 0x98, maybe Christoph, Keith or our other
NVMe guys can help to understand the failure.


Thanks,
Ming

Re: fstrim error - AORUS NVMe Gen4 SSD

[davidc502]

Hello,

On 7/22/19 11:38 PM, Ming Lei wrote:
> Hi,
>
> On Mon, Jul 22, 2019 at 09:39:07PM -0500, davidc502 wrote:
>> See attached:  nvme_io_trace.log
>>
>>
>> On 7/22/19 9:19 PM, Ming Lei wrote:
>>> Hi,
>>>
>>> On Sat, Jul 20, 2019 at 09:41:24PM -0500, davidc502 wrote:
>>>>    Hello,
>>>>
>>>>    I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.
>>>>
>>>>    When attempting to fstrim the NVMe, I receive the following error.
>>>>    davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
>>>>    fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
>>>>    fstrim: /: FITRIM ioctl failed: Input/output error
>>>>
>>>>    Anyhow, I have put some details below which might be helpful. Note that
>>>> this NVMe is supposed to be TRIM and SMART compliant. The SMART outputs are
>>>> available using the utility “nvme-cli”.
>>>>    I am willing to provide whatever command outputs that are needed to help
>>>> solve this issue.
>>>>
>>>>    OS= Ubuntu 18.4.2 LTS
>>>>    Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
>>>>    fstrim version =  fstrim from util-linux 2.31.1
>>>>    Firmware version for Aorus NVMe = EGFM11.0
>>>>
>>> I saw discard timeout on HGST 1.6TB NVMe, not sure if yours is same with
>>> that one.
>>>
>>> Could you collect logs via the following steps?
>>>
>>> Suppose your nvme disk name is /dev/nvme0n1:
>>>
>>> 1) queue limits log:
>>>
>>> 	#(cd /sys/block/nvme0n1/queue && find . -type f -exec grep -aH . {} \;)
>>>
>>>
>>> 2) NVMe IO trace
>>>
>>> - enable nvme IO trace before running fstrim:
>>>
>>> 	#echo 1  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
>>> 	#echo 1  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
>>>
>>> - run fstrim
>>>
>>> - after the fstrim failure is triggered, disable the nvme io trace & post the log:
>>>
>>> 	#echo 0  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
>>> 	#echo 0  > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
>>>
>>> 	#cp	/sys/kernel/debug/tracing/trace /root/nvme_io_trace.log
>>>
>>>
>>>
>>> thanks,
>>> Ming
>>
>> Hello Ming
>>
>> Thank you for the quick reply --  See attached
>  From the IO trace, discard command(nvme_cmd_dsm) is failed:
>
>    kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_dsm nr=0, attributes=4)
>            <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq: nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0, status=8198
>
> And the returned error code is 0x8198, I am not sure how to parse the
> 'Command Specific Status Values' of 0x98, maybe Christoph, Keith or our other
> NVMe guys can help to understand the failure.
>
>
> Thanks,
> Ming

Long story short we have 3 new PCI Gen 4 SSD - NVMe drives from 
Gigabyte. But actually, Gigabyte is just putting their name on it as I 
believe it is actually from "Phison".

Here is the website where you can see the new drives -- 
https://www.gigabyte.com/Solid-State-Drive/Gen-4

I have opened a ticket with Gigabyte, and have inquired about any 
available firmware updates. It will take 3-5 days to hear back from 
them, but will report back the finding.

Thank you for taking a look at the tracing file, and hopefully that 
gives enough insight as to what might be happening.

Best Regards,

David

Re: fstrim error - AORUS NVMe Gen4 SSD

[davidc502]

Ming,

On 7/23/19 4:38 PM, davidc502 wrote:
> Hello,
>
> On 7/22/19 11:38 PM, Ming Lei wrote:
>> Hi,
>>
>> On Mon, Jul 22, 2019 at 09:39:07PM -0500, davidc502 wrote:
>>> See attached:  nvme_io_trace.log
>>>
>>>
>>> On 7/22/19 9:19 PM, Ming Lei wrote:
>>>> Hi,
>>>>
>>>> On Sat, Jul 20, 2019 at 09:41:24PM -0500, davidc502 wrote:
>>>>>    Hello,
>>>>>
>>>>>    I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.
>>>>>
>>>>>    When attempting to fstrim the NVMe, I receive the following error.
>>>>>    davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
>>>>>    fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
>>>>>    fstrim: /: FITRIM ioctl failed: Input/output error
>>>>>
>>>>>    Anyhow, I have put some details below which might be helpful. 
>>>>> Note that
>>>>> this NVMe is supposed to be TRIM and SMART compliant. The SMART 
>>>>> outputs are
>>>>> available using the utility “nvme-cli”.
>>>>>    I am willing to provide whatever command outputs that are 
>>>>> needed to help
>>>>> solve this issue.
>>>>>
>>>>>    OS= Ubuntu 18.4.2 LTS
>>>>>    Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
>>>>>    fstrim version =  fstrim from util-linux 2.31.1
>>>>>    Firmware version for Aorus NVMe = EGFM11.0
>>>>>
>>>> I saw discard timeout on HGST 1.6TB NVMe, not sure if yours is same 
>>>> with
>>>> that one.
>>>>
>>>> Could you collect logs via the following steps?
>>>>
>>>> Suppose your nvme disk name is /dev/nvme0n1:
>>>>
>>>> 1) queue limits log:
>>>>
>>>>     #(cd /sys/block/nvme0n1/queue && find . -type f -exec grep -aH 
>>>> . {} \;)
>>>>
>>>>
>>>> 2) NVMe IO trace
>>>>
>>>> - enable nvme IO trace before running fstrim:
>>>>
>>>>     #echo 1  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
>>>>     #echo 1  > 
>>>> /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
>>>>
>>>> - run fstrim
>>>>
>>>> - after the fstrim failure is triggered, disable the nvme io trace 
>>>> & post the log:
>>>>
>>>>     #echo 0  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
>>>>     #echo 0  > 
>>>> /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
>>>>
>>>>     #cp    /sys/kernel/debug/tracing/trace /root/nvme_io_trace.log
>>>>
>>>>
>>>>
>>>> thanks,
>>>> Ming
>>>
>>> Hello Ming
>>>
>>> Thank you for the quick reply --  See attached
>>  From the IO trace, discard command(nvme_cmd_dsm) is failed:
>>
>>    kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd: 
>> nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0, 
>> cmd=(nvme_cmd_dsm nr=0, attributes=4)
>>            <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq: 
>> nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0, 
>> status=8198
>>
>> And the returned error code is 0x8198, I am not sure how to parse the
>> 'Command Specific Status Values' of 0x98, maybe Christoph, Keith or 
>> our other
>> NVMe guys can help to understand the failure.
>>
>>
>> Thanks,
>> Ming
>
> Long story short we have 3 new PCI Gen 4 SSD - NVMe drives from 
> Gigabyte. But actually, Gigabyte is just putting their name on it as I 
> believe it is actually from "Phison".
>
> Here is the website where you can see the new drives -- 
> https://www.gigabyte.com/Solid-State-Drive/Gen-4
>
> I have opened a ticket with Gigabyte, and have inquired about any 
> available firmware updates. It will take 3-5 days to hear back from 
> them, but will report back the finding.
>
> Thank you for taking a look at the tracing file, and hopefully that 
> gives enough insight as to what might be happening.
>
> Best Regards,
>
> David
>
>
I have attempted to CC ilnux-nvme@lists.infradead.org, but receive a 
immediate notification " 550 Unknown recipient ".  So it just gets 
bounced back to me.

Thanks,

David

Re: fstrim error - AORUS NVMe Gen4 SSD

[Ming Lei]

On Tue, Jul 23, 2019 at 04:56:10PM -0500, davidc502 wrote:
> Ming,
> 
> On 7/23/19 4:38 PM, davidc502 wrote:
> > Hello,
> > 
> > On 7/22/19 11:38 PM, Ming Lei wrote:
> > > Hi,
> > > 
> > > On Mon, Jul 22, 2019 at 09:39:07PM -0500, davidc502 wrote:
> > > > See attached:  nvme_io_trace.log
> > > > 
> > > > 
> > > > On 7/22/19 9:19 PM, Ming Lei wrote:
> > > > > Hi,
> > > > > 
> > > > > On Sat, Jul 20, 2019 at 09:41:24PM -0500, davidc502 wrote:
> > > > > >    Hello,
> > > > > > 
> > > > > >    I've assembled a X570 board with a 1TB AORUS NVMe Gen4 SSD.
> > > > > > 
> > > > > >    When attempting to fstrim the NVMe, I receive the following error.
> > > > > >    davidc502@Ryzen-3900x:~$ sudo fstrim -a -v
> > > > > >    fstrim: /boot/efi: FITRIM ioctl failed: Input/output error
> > > > > >    fstrim: /: FITRIM ioctl failed: Input/output error
> > > > > > 
> > > > > >    Anyhow, I have put some details below which might be
> > > > > > helpful. Note that
> > > > > > this NVMe is supposed to be TRIM and SMART compliant.
> > > > > > The SMART outputs are
> > > > > > available using the utility “nvme-cli”.
> > > > > >    I am willing to provide whatever command outputs that
> > > > > > are needed to help
> > > > > > solve this issue.
> > > > > > 
> > > > > >    OS= Ubuntu 18.4.2 LTS
> > > > > >    Different Kernels I’ve tried = 5.1.16, 5.2 rc7, and 4.18
> > > > > >    fstrim version =  fstrim from util-linux 2.31.1
> > > > > >    Firmware version for Aorus NVMe = EGFM11.0
> > > > > > 
> > > > > I saw discard timeout on HGST 1.6TB NVMe, not sure if yours
> > > > > is same with
> > > > > that one.
> > > > > 
> > > > > Could you collect logs via the following steps?
> > > > > 
> > > > > Suppose your nvme disk name is /dev/nvme0n1:
> > > > > 
> > > > > 1) queue limits log:
> > > > > 
> > > > >     #(cd /sys/block/nvme0n1/queue && find . -type f -exec
> > > > > grep -aH . {} \;)
> > > > > 
> > > > > 
> > > > > 2) NVMe IO trace
> > > > > 
> > > > > - enable nvme IO trace before running fstrim:
> > > > > 
> > > > >     #echo 1  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
> > > > >     #echo 1  >
> > > > > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
> > > > > 
> > > > > - run fstrim
> > > > > 
> > > > > - after the fstrim failure is triggered, disable the nvme io
> > > > > trace & post the log:
> > > > > 
> > > > >     #echo 0  > /sys/kernel/debug/tracing/events/nvme_setup_cmd/enable
> > > > >     #echo 0  >
> > > > > /sys/kernel/debug/tracing/events/nvme_complete_rq/enable
> > > > > 
> > > > >     #cp    /sys/kernel/debug/tracing/trace /root/nvme_io_trace.log
> > > > > 
> > > > > 
> > > > > 
> > > > > thanks,
> > > > > Ming
> > > > 
> > > > Hello Ming
> > > > 
> > > > Thank you for the quick reply --  See attached
> > >  From the IO trace, discard command(nvme_cmd_dsm) is failed:
> > > 
> > >    kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd:
> > > nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0,
> > > cmd=(nvme_cmd_dsm nr=0, attributes=4)
> > >            <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq:
> > > nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0,
> > > status=8198
> > > 
> > > And the returned error code is 0x8198, I am not sure how to parse the
> > > 'Command Specific Status Values' of 0x98, maybe Christoph, Keith or
> > > our other
> > > NVMe guys can help to understand the failure.
> > > 
> > > 
> > > Thanks,
> > > Ming
> > 
> > Long story short we have 3 new PCI Gen 4 SSD - NVMe drives from
> > Gigabyte. But actually, Gigabyte is just putting their name on it as I
> > believe it is actually from "Phison".
> > 
> > Here is the website where you can see the new drives --
> > https://www.gigabyte.com/Solid-State-Drive/Gen-4
> > 
> > I have opened a ticket with Gigabyte, and have inquired about any
> > available firmware updates. It will take 3-5 days to hear back from
> > them, but will report back the finding.
> > 
> > Thank you for taking a look at the tracing file, and hopefully that
> > gives enough insight as to what might be happening.
> > 
> > Best Regards,
> > 
> > David
> > 
> > 
> I have attempted to CC ilnux-nvme@lists.infradead.org, but receive a
> immediate notification " 550 Unknown recipient ".  So it just gets bounced
> back to me.

It should have been linux-nvme@lists.infradead.org.

Thanks,
Ming

Re: fstrim error - AORUS NVMe Gen4 SSD

[Keith Busch]

On Tue, Jul 23, 2019 at 12:38:04PM +0800, Ming Lei wrote:
> From the IO trace, discard command(nvme_cmd_dsm) is failed:
> 
>   kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_dsm nr=0, attributes=4)
>           <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq: nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0, status=8198
> 
> And the returned error code is 0x8198, I am not sure how to parse the
> 'Command Specific Status Values' of 0x98, maybe Christoph, Keith or our other
> NVMe guys can help to understand the failure.

The 198h status code is still marked reserved in the latest spec for an
NVM command set, so not sure what to make of it. I think we would have
to refer back to the vendor.

Re: fstrim error - AORUS NVMe Gen4 SSD

[davidc502]

Hello,

On 7/24/19 4:30 PM, Keith Busch wrote:
> On Tue, Jul 23, 2019 at 12:38:04PM +0800, Ming Lei wrote:
>>  From the IO trace, discard command(nvme_cmd_dsm) is failed:
>>
>>    kworker/15:1H-462   [015] .... 91814.342452: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=7, cmdid=552, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_dsm nr=0, attributes=4)
>>            <idle>-0     [013] d.h. 91814.342708: nvme_complete_rq: nvme0: disk=nvme0n1, qid=7, cmdid=552, res=0, retries=0, flags=0x0, status=8198
>>
>> And the returned error code is 0x8198, I am not sure how to parse the
>> 'Command Specific Status Values' of 0x98, maybe Christoph, Keith or our other
>> NVMe guys can help to understand the failure.
> The 198h status code is still marked reserved in the latest spec for an
> NVM command set, so not sure what to make of it. I think we would have
> to refer back to the vendor.

Thank you  for taking a look at this. I guess my question would be if 
you have a vehicle in place to be able to refer back to the vendor?

David