[PATCH] Use direct I/O for loop devices

[PATCH] Use direct I/O for loop devices

[Demi Marie Obenour]

This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using
   synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and
   associated overhead.
---
 tools/hotplug/Linux/block | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/hotplug/Linux/block b/tools/hotplug/Linux/block
index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
--- a/tools/hotplug/Linux/block
+++ b/tools/hotplug/Linux/block
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
         xenstore_write "$XENBUS_PATH/node" "$loopdev"
         write_dev "$loopdev"
         release_lock "block"
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

Re: [PATCH] Use direct I/O for loop devices

[Jan Beulich]

On 17.08.2022 22:46, Demi Marie Obenour wrote:
> This is a huge performance improvement for two reasons:
> 
> 1. It uses the filesystem’s asynchronous I/O support, rather than using
>    synchronous I/O.
> 2. It bypasses the page cache, removing a redundant layer of caching and
>    associated overhead.
> ---
>  tools/hotplug/Linux/block | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/hotplug/Linux/block b/tools/hotplug/Linux/block
> index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
> --- a/tools/hotplug/Linux/block
> +++ b/tools/hotplug/Linux/block
> @@ -330,7 +330,7 @@ mount it read-write in a guest domain."
>          else
>            roflag=''
>          fi
> -        do_or_die losetup $roflag "$loopdev" "$file"
> +        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"

I guess you want to first check (maybe in tools/configure) that losetup
actually supports that option. The old-ish one I'm looking at doesn't,
according to its --help output at least.

Jan

Re: [PATCH] Use direct I/O for loop devices

[Demi Marie Obenour]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On Thu, Aug 18, 2022 at 09:02:11AM +0200, Jan Beulich wrote:
> On 17.08.2022 22:46, Demi Marie Obenour wrote:
> > This is a huge performance improvement for two reasons:
> > 
> > 1. It uses the filesystem’s asynchronous I/O support, rather than using
> >    synchronous I/O.
> > 2. It bypasses the page cache, removing a redundant layer of caching and
> >    associated overhead.
> > ---
> >  tools/hotplug/Linux/block | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/hotplug/Linux/block b/tools/hotplug/Linux/block
> > index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
> > --- a/tools/hotplug/Linux/block
> > +++ b/tools/hotplug/Linux/block
> > @@ -330,7 +330,7 @@ mount it read-write in a guest domain."
> >          else
> >            roflag=''
> >          fi
> > -        do_or_die losetup $roflag "$loopdev" "$file"
> > +        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
> 
> I guess you want to first check (maybe in tools/configure) that losetup
> actually supports that option. The old-ish one I'm looking at doesn't,
> according to its --help output at least.

What version are you referring to?

In Qubes OS the current plan is to use a block “script” written in C, to
improve performance and (when combined with kernel patches) eliminate
race conditions.  This code could be made a wrapper for the C version.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] Use direct I/O for loop devices

[Jan Beulich]

On 18.08.2022 16:04, Demi Marie Obenour wrote:
> On Thu, Aug 18, 2022 at 09:02:11AM +0200, Jan Beulich wrote:
>> On 17.08.2022 22:46, Demi Marie Obenour wrote:
>>> --- a/tools/hotplug/Linux/block
>>> +++ b/tools/hotplug/Linux/block
>>> @@ -330,7 +330,7 @@ mount it read-write in a guest domain."
>>>          else
>>>            roflag=''
>>>          fi
>>> -        do_or_die losetup $roflag "$loopdev" "$file"
>>> +        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
>>
>> I guess you want to first check (maybe in tools/configure) that losetup
>> actually supports that option. The old-ish one I'm looking at doesn't,
>> according to its --help output at least.
> 
> What version are you referring to?

The tool itself doesn't recognize --version. It originates from
util-linux 2.19.1 from all I can tell.

> In Qubes OS the current plan is to use a block “script” written in C, to
> improve performance and (when combined with kernel patches) eliminate
> race conditions.  This code could be made a wrapper for the C version.

Is this relevant here in some way I don't recognize, or did you say this
only to provide some further background info?

Jan

Re: [PATCH] Use direct I/O for loop devices

[Demi Marie Obenour]

[-- Attachment #1: Type: text/plain, Size: 1568 bytes --]

On Thu, Aug 18, 2022 at 04:12:10PM +0200, Jan Beulich wrote:
> On 18.08.2022 16:04, Demi Marie Obenour wrote:
> > On Thu, Aug 18, 2022 at 09:02:11AM +0200, Jan Beulich wrote:
> >> On 17.08.2022 22:46, Demi Marie Obenour wrote:
> >>> --- a/tools/hotplug/Linux/block
> >>> +++ b/tools/hotplug/Linux/block
> >>> @@ -330,7 +330,7 @@ mount it read-write in a guest domain."
> >>>          else
> >>>            roflag=''
> >>>          fi
> >>> -        do_or_die losetup $roflag "$loopdev" "$file"
> >>> +        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
> >>
> >> I guess you want to first check (maybe in tools/configure) that losetup
> >> actually supports that option. The old-ish one I'm looking at doesn't,
> >> according to its --help output at least.
> > 
> > What version are you referring to?
> 
> The tool itself doesn't recognize --version. It originates from
> util-linux 2.19.1 from all I can tell.

I see.  Qubes OS doesn’t need to support such old versions.  Are you
referring to 

> > In Qubes OS the current plan is to use a block “script” written in C, to
> > improve performance and (when combined with kernel patches) eliminate
> > race conditions.  This code could be made a wrapper for the C version.
> 
> Is this relevant here in some way I don't recognize, or did you say this
> only to provide some further background info?

If the C code becomes part of the toolstack it would avoid needing to
call into losetup.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] Use direct I/O for loop devices

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1413 bytes --]

On Wed, Aug 17, 2022 at 04:46:34PM -0400, Demi Marie Obenour wrote:
> This is a huge performance improvement for two reasons:
> 
> 1. It uses the filesystem’s asynchronous I/O support, rather than using
>    synchronous I/O.
> 2. It bypasses the page cache, removing a redundant layer of caching and
>    associated overhead.

Cross-posting here just in case: this patch breaks block script on
devices with 4k block size, at least with some util-linux versions.
More details at https://github.com/QubesOS/qubes-issues/issues/7828

> ---
>  tools/hotplug/Linux/block | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/hotplug/Linux/block b/tools/hotplug/Linux/block
> index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
> --- a/tools/hotplug/Linux/block
> +++ b/tools/hotplug/Linux/block
> @@ -330,7 +330,7 @@ mount it read-write in a guest domain."
>          else
>            roflag=''
>          fi
> -        do_or_die losetup $roflag "$loopdev" "$file"
> +        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
>          xenstore_write "$XENBUS_PATH/node" "$loopdev"
>          write_dev "$loopdev"
>          release_lock "block"
> -- 
> Sincerely,
> Demi Marie Obenour (she/her/hers)
> Invisible Things Lab
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]