[LSF/FS TOPIC] Scaling file systems on high performance flash devices

[LSF/FS TOPIC] Scaling file systems on high performance flash devices

[Nauman Rafique]

Flash device vendors are coming up with faster and faster devices
every year.  Given the high performance supported by these devices,
there are thoughts about using them not only as high performance
storage but also as a replacement for huge quantities of DRAM. That
particular use case would put very stringent requirements on the
performance of file systems on these devices --- an issue that should
be discussed.

I will share our experience running some experiments on a high
performance flash device (FusionIO IODrive duo) with ext4 and XFS. We
have devised an extensive set of experiments focused on finding the
scaling and overhead problems in the kernel. Our experiments use
various IO sizes, and perform IO in both synchronous multi-threaded
mode and AIO mode. We configure our setup to bypass the block layer
(fusionIO driver supports that), and do IO in O_DIRECT mode to
minimize overhead in the kernel. In spite of such optimizations, we
still see performance issues especially while doing IO at the peak
throughput capacity available on these drives. The issues pertain to
CPU scheduling behavior, filesystem metadata manipulation, and
basically the whole kernel code path involved in doing IO to
such devices, that would not be involved if data was read from DRAM
directly.

--
Nauman

Re: [Lsf-pc] [LSF/FS TOPIC] Scaling file systems on high performance flash devices

[Ric Wheeler]

On 02/03/2011 07:39 PM, Nauman Rafique wrote:
> Flash device vendors are coming up with faster and faster devices
> every year.  Given the high performance supported by these devices,
> there are thoughts about using them not only as high performance
> storage but also as a replacement for huge quantities of DRAM. That
> particular use case would put very stringent requirements on the
> performance of file systems on these devices --- an issue that should
> be discussed.
>
> I will share our experience running some experiments on a high
> performance flash device (FusionIO IODrive duo) with ext4 and XFS. We
> have devised an extensive set of experiments focused on finding the
> scaling and overhead problems in the kernel. Our experiments use
> various IO sizes, and perform IO in both synchronous multi-threaded
> mode and AIO mode. We configure our setup to bypass the block layer
> (fusionIO driver supports that), and do IO in O_DIRECT mode to
> minimize overhead in the kernel. In spite of such optimizations, we
> still see performance issues especially while doing IO at the peak
> throughput capacity available on these drives. The issues pertain to
> CPU scheduling behavior, filesystem metadata manipulation, and
> basically the whole kernel code path involved in doing IO to
> such devices, that would not be involved if data was read from DRAM
> directly.
>
> --
> Nauman

Thanks, I think that this is a very good topic for us and should have broad 
interest...

Ric

Re: [LSF/FS TOPIC] Scaling file systems on high performance flash devices

[Dave Chinner]

On Thu, Feb 03, 2011 at 04:39:40PM -0800, Nauman Rafique wrote:
> Flash device vendors are coming up with faster and faster devices
> every year.  Given the high performance supported by these devices,
> there are thoughts about using them not only as high performance
> storage but also as a replacement for huge quantities of DRAM. That
> particular use case would put very stringent requirements on the
> performance of file systems on these devices --- an issue that should
> be discussed.
> 
> I will share our experience running some experiments on a high
> performance flash device (FusionIO IODrive duo) with ext4 and XFS. We
> have devised an extensive set of experiments focused on finding the
> scaling and overhead problems in the kernel. Our experiments use
> various IO sizes, and perform IO in both synchronous multi-threaded
> mode and AIO mode. We configure our setup to bypass the block layer
> (fusionIO driver supports that), and do IO in O_DIRECT mode to
> minimize overhead in the kernel. In spite of such optimizations, we
> still see performance issues especially while doing IO at the peak
> throughput capacity available on these drives. The issues pertain to
> CPU scheduling behavior, filesystem metadata manipulation, and
> basically the whole kernel code path involved in doing IO to
> such devices, that would not be involved if data was read from DRAM
> directly.

Seeing as I'm not going to be around for LSF, can you describe some
of your testing and the limitations you came across on XFS? I'm
especially interested in the metadata manipulation issues you saw as
we've done a fair bit of metadata and journal IO optimisation in the
past year....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/FS TOPIC] Scaling file systems on high performance flash devices

[Nauman Rafique]

Hi Dave,
I will share a detailed report on the mailing lists once I have it ready.

Thanks.

On Tue, Feb 8, 2011 at 2:38 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Feb 03, 2011 at 04:39:40PM -0800, Nauman Rafique wrote:
>> Flash device vendors are coming up with faster and faster devices
>> every year.  Given the high performance supported by these devices,
>> there are thoughts about using them not only as high performance
>> storage but also as a replacement for huge quantities of DRAM. That
>> particular use case would put very stringent requirements on the
>> performance of file systems on these devices --- an issue that should
>> be discussed.
>>
>> I will share our experience running some experiments on a high
>> performance flash device (FusionIO IODrive duo) with ext4 and XFS. We
>> have devised an extensive set of experiments focused on finding the
>> scaling and overhead problems in the kernel. Our experiments use
>> various IO sizes, and perform IO in both synchronous multi-threaded
>> mode and AIO mode. We configure our setup to bypass the block layer
>> (fusionIO driver supports that), and do IO in O_DIRECT mode to
>> minimize overhead in the kernel. In spite of such optimizations, we
>> still see performance issues especially while doing IO at the peak
>> throughput capacity available on these drives. The issues pertain to
>> CPU scheduling behavior, filesystem metadata manipulation, and
>> basically the whole kernel code path involved in doing IO to
>> such devices, that would not be involved if data was read from DRAM
>> directly.
>
> Seeing as I'm not going to be around for LSF, can you describe some
> of your testing and the limitations you came across on XFS? I'm
> especially interested in the metadata manipulation issues you saw as
> we've done a fair bit of metadata and journal IO optimisation in the
> past year....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html