[Xenomai-help] rtdm_mmap_to_user without MMU

[Xenomai-help] rtdm_mmap_to_user without MMU

[Waschk,Kolja]

Hi,

I'm starting to develop a RTDM driver to be used in uClinux on a Blackfin-based
device. Its main task is to receive lots of framed data (30 MBit/s) and pass
them to an application in user space, retaining the framing info. A major
concern is the performance. I'd like to avoid copying, so I looked into using
rtdm_mmap_to_user(). It seems to use the mmap op of /dev/zero, which is "return
ENOSYS"(*) in drivers/char/mem.c on systems without a MMU. In other words, it
doesn't seem to be usable in uClinux. Well, I could just forget about mmap,
there is no MMU and thus my user code can access the data anyway. But what'd be
the proper way to cope with this? I'd really like to code it so it would work
on a system with MMU as well (maybe in next year's project...)

Probably I'll move a bit more of the user space code (the decoding of the frame
into a larger structure) into the driver, turning the "copying" into
"decoding", which has to be done anyway. But ideally the kernel part should
be kept as simple as possible.

Kolja

(*) I might have missed something between, because the actual error code is ENODEV

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Jan Kiszka]

Waschk,Kolja wrote:
> Hi,
> 
> I'm starting to develop a RTDM driver to be used in uClinux on a Blackfin-based
> device. Its main task is to receive lots of framed data (30 MBit/s) and pass
> them to an application in user space, retaining the framing info. A major
> concern is the performance. I'd like to avoid copying, so I looked into using
> rtdm_mmap_to_user(). It seems to use the mmap op of /dev/zero, which is "return
> ENOSYS"(*) in drivers/char/mem.c on systems without a MMU. In other words, it

rtdm_mmap_to_user uses remap_pfn_range internally when there is no MMU.
And IIRC, it is supposed to work also in this scenario. You may just not
expect being able to specify a remap target address...

> doesn't seem to be usable in uClinux. Well, I could just forget about mmap,
> there is no MMU and thus my user code can access the data anyway. But what'd be
> the proper way to cope with this? I'd really like to code it so it would work
> on a system with MMU as well (maybe in next year's project...)
> 
> Probably I'll move a bit more of the user space code (the decoding of the frame
> into a larger structure) into the driver, turning the "copying" into
> "decoding", which has to be done anyway. But ideally the kernel part should
> be kept as simple as possible.

Yes, generally the right approach.

> 
> Kolja
> 
> (*) I might have missed something between, because the actual error code is ENODEV

Can you provide some hardware-independent test case?

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Kolja Waschk]

Hi,

> rtdm_mmap_to_user uses remap_pfn_range internally when there is no MMU.

Ah, thanks, I missed the part in rtdm_do_mmap where the
/dev/zero->f_op are temporarily replaced with rtdm_mmap_fops.

Now I was able to find the actual ENODEV source down in linux/mm/nommu.c
validate_mmap_request() where the following code path is followed:

if(file)
{
     ...
     if (mapping && mapping->backing_dev_info)
         capabilities = mapping->backing_dev_info->capabilities;
     ...
     if (!file->f_op->get_unmapped_area)
         capabilities &= ~BDI_CAP_MAP_DIRECT;
     ...
     if (flags & MAP_SHARED) {
         ....
         if (!(capabilities & BDI_CAP_MAP_DIRECT))
             return -ENODEV;
...

The mapping->backing_dev_info->capabilities is only BDI_CAP_MAP_COPY,
no BDI_CAP_MAP_DIRECT. So even if drvlib would provide a
get_unmapped_area op, validate_mmap_request would answer ENODEV.

It seems that /dev/mem instead of /dev/zero has the required
capabilities. So I changed rtdm_do_mmap to open dev/mem instead
of dev/zero and provide a get_unmapped_area:

#ifndef CONFIG_MMU
     rtdm_mmap_fops.get_unmapped_area = old_fops->get_unmapped_area;
#endif

Now the rtdm_mmap_to_user returns zero, indicating no error, but I still
do not get a usable pptr (it's NULL as on input) from it. I somehow
expect to get src_addr back in pptr.

> Can you provide some hardware-independent test case?

I'm using the code from 
http://www.captain.at/xenomai-real-time-driver-example.php
almost unmodified except for some plausibility checking around
mmappointer after calling the ioctl.

Kolja

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Kolja Waschk]

Hi,

I finally seem to have a complete solution for my case. It's probably
still missing one or the other detail, but it starts to work. I only
added a few lines to ksrc/skins/rtdm/drvlib.c in rtdm_do_mmap:

1. Because linux/mm/nommu.c/validate_mmap_request() checks the
capabilities of the backing dev, and /dev/zero lacks BDI_CAP_MAP_DIRECT
capability, the /dev/mem has to be used instead of /dev/zero.

> filp = filp_open("/dev/mem", O_RDWR, 0);

2. A get_unmapped_area() f_op has to be provided on nommu systems. It is
also checked by validate_mmap_request(). I just copy in the pointer from
old_fops: the /dev/mem f_op->get_unmapped_area simply passes pgoffs.

> rtdm_mmap_fops.get_unmapped_area = old_fops->get_unmapped_area;

3. Since /dev/mem is used, the offset passed to do_mmap() has to
point to the actual buffer (probably the physical address, for now I
just used mmap_data->src_vaddr).

> user_ptr = (void *)do_mmap(filp, (unsigned long)*pptr, len, prot,
>            MAP_SHARED, mmap_data->src_vaddr);

Kolja

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2620 bytes --]

Kolja Waschk wrote:
> Hi,
> 
> I finally seem to have a complete solution for my case. It's probably
> still missing one or the other detail, but it starts to work. I only
> added a few lines to ksrc/skins/rtdm/drvlib.c in rtdm_do_mmap:
> 
> 1. Because linux/mm/nommu.c/validate_mmap_request() checks the
> capabilities of the backing dev, and /dev/zero lacks BDI_CAP_MAP_DIRECT
> capability, the /dev/mem has to be used instead of /dev/zero.
> 
>> filp = filp_open("/dev/mem", O_RDWR, 0);
> 
> 2. A get_unmapped_area() f_op has to be provided on nommu systems. It is
> also checked by validate_mmap_request(). I just copy in the pointer from
> old_fops: the /dev/mem f_op->get_unmapped_area simply passes pgoffs.
> 
>> rtdm_mmap_fops.get_unmapped_area = old_fops->get_unmapped_area;
> 
> 3. Since /dev/mem is used, the offset passed to do_mmap() has to
> point to the actual buffer (probably the physical address, for now I
> just used mmap_data->src_vaddr).
> 
>> user_ptr = (void *)do_mmap(filp, (unsigned long)*pptr, len, prot,
>>            MAP_SHARED, mmap_data->src_vaddr);
> 

Nice analysis! I tried to fold your findings into a patch (actually,
they will become two in the end), see below. Please check if it works
for you.

In the meantime I need to verify if switching to /dev/mem comes without
side-effects for CONFIG_MMU. Anyone with a rtdm_mmap/iomap_to_user test
case at hand is welcome to check this, too!

Jan

---------->

diff --git a/ksrc/skins/rtdm/drvlib.c b/ksrc/skins/rtdm/drvlib.c
index 9f1cda8..3f37e75 100644
--- a/ksrc/skins/rtdm/drvlib.c
+++ b/ksrc/skins/rtdm/drvlib.c
@@ -1844,8 +1844,24 @@ static int rtdm_mmap_buffer(struct file *filp, struct vm_area_struct *vma)
 	return ret;
 }
 
+#ifndef CONFIG_MMU
+static unsigned long rtdm_get_unmapped_area(struct file *file,
+					    unsigned long addr,
+					    unsigned long len,
+					    unsigned long pgoff,
+					    unsigned long flags)
+{
+	struct rtdm_mmap_data *mmap_data = filp->private_data;
+
+	return mmap_data->src_vaddr;
+}
+#endif /* !CONFIG_MMU */
+
 static struct file_operations rtdm_mmap_fops = {
 	.mmap = rtdm_mmap_buffer,
+#ifndef CONFIG_MMU
+	.get_unmapped_area = rtdm_get_unmapped_area,
+#endif /* !CONFIG_MMU */
 };
 
 static int rtdm_do_mmap(rtdm_user_info_t *user_info,
@@ -1859,7 +1875,7 @@ static int rtdm_do_mmap(rtdm_user_info_t *user_info,
 
 	XENO_ASSERT(RTDM, xnpod_root_p(), return -EPERM;);
 
-	filp = filp_open("/dev/zero", O_RDWR, 0);
+	filp = filp_open("/dev/mem", O_RDWR, 0);
 	if (IS_ERR(filp))
 		return PTR_ERR(filp);
 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Kolja Waschk]

Hi Jan,

> a patch (actually, they will become two in the end), see below. Please
> check if it works for you.

Yes, the patch works for me.

Kolja

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Kolja Waschk]

> they will become two in the end), see below. Please check if it works
> for you.

Aah, I was checking too hastily and missed a compile error. Sorry!
Actually, there's a typo in the get_unmapped_area (filp <-> file), and
it does _not_ work this way, unless the address to map is specified as
the offset when calling do_mmap:

     user_ptr = (void *)do_mmap(filp, (unsigned long)*pptr, len, prot,
                        MAP_SHARED, mmap_data->src_vaddr);

Otherwise pptr will simply remain zero.

Kolja

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Kolja Waschk wrote:
>> they will become two in the end), see below. Please check if it works
>> for you.
> 
> Aah, I was checking too hastily and missed a compile error. Sorry!
> Actually, there's a typo in the get_unmapped_area (filp <-> file), and
> it does _not_ work this way, unless the address to map is specified as
> the offset when calling do_mmap:
> 
>     user_ptr = (void *)do_mmap(filp, (unsigned long)*pptr, len, prot,
>                        MAP_SHARED, mmap_data->src_vaddr);
> 
> Otherwise pptr will simply remain zero.

Hmm, thought I got all paths. Such side effects indicate that this way
is not a stable one. Ok, will try to find a better solution, but may
take some time.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] rtdm_mmap_to_user without MMU

[Kolja Waschk]

> Hmm, thought I got all paths. Such side effects indicate that this way
> is not a stable one. Ok, will try to find a better solution, but may

Hi,

One last note regarding the mmap on blackfin (nommu) thread a few days
ago.

I just noticed that reserving pages with SetPageReserved and clearing
that flag with ClearPageReserved doesn't work as expected. I blindly
copied that code from Jan's (old) RTDM driver example, as below, and now
found that the kernel will fail some seconds after the ClearPageReserved loop:

> / # Bad page state in process 'events/0'
> page:009e29c0 flags:0x00004400 mapping:00000000 mapcount:0 count:0
> Trying to fix it up, but a reboot is needed

If SetPageReserved doesn't work, it may well be the cause for the mmap
trick to fail afterwards. For now, I bypass the page reservation and
mmap and munmap, there is no MMU anyway, so there's no need to hurry.

Here's the code that I refer to:

/* Mark buffer pages as reserved so rtdm_mmap_to_user will work */
unsigned long vaddr;
for (vaddr  = (unsigned long)pool;
      vaddr <= (unsigned long)pool + UFSIZE;
      vaddr += PAGE_SIZE)
{
     SetPageReserved(virt_to_page(vaddr));
}

/* mmap... */
/* munmap... */


/* clear pages reserved */
for (vaddr = (unsigned long)pool;
      vaddr < (unsigned long)pool + BUFSIZE;
      vaddr += PAGE_SIZE)
{
     ClearPageReserved(virt_to_page(vaddr));
}

Kolja