Re: Crashes if running WordPerfect 5.1

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Bart Oldeman wrote:
> mremap is behaving consistently (i.e. if you mmap bytes 4096-8191 of a
> file with size 4000 you can't expect that file to be extended to 8192
> bytes, the same is true for shm virtual files.
I just think kernel must hide that
things from userspace and resolve
them internally. "man mmap" says:
---
  MAP_ANONYMOUS
               The mapping is not backed by any file
---
So there should be no considerations
about a file resizing on a userspace.
For userspace it must behave as if there
is no file involved, otherwise it
contradicts with docs I think.

What I was thinking about a cheap
work around, is probably to allocate
the new region the way your hack does,
but then, instead of memcpy(), still
use mremap() MREMAP_FIXED to remap the
old region to the beginning of the new one.
This will preserve the aliases, but will
probably not work reliably... at least
that looks extremely risky to do.

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Bart Oldeman wrote:
>> don't need IPC SHM, and pagemalloc perhaps doesn't really hurt?
> I think I would personnally like it best if there wasn't a pagealloc
> either, but just multiple fd's.
After all I changed my mind again and
think it is not a good idea any more.
By using the kernel's allocator, we assume
it suits whatever the DOS prog may expect,
in particular that the subsequent malloc()s
returns the higher addresses each. There
were the programs that used to crash when
the subsequent malloc() was returning the
address below the one of the previous malloc.
I was reading this:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0406.2/0874.html
In particular the following makes me to worry:
---
  0xbfxxxxxx ... _end_ of all mmaps, new mmaps go below old ones
---
So it seems the kernel guys may break that
our assumption and we'll be screwed. I don't
think we have to propagate the DOS requirements
to the kernel and ask the folks to not do this.
They should not care about what DOS progs do.
Instead we have to wrap that functionality
ourselves and provide whatever DOS progs are
expecting from us.
That's what pagemalloc did. Perhaps dropping
it was a premature decision. To stay on a safe
side, I think we have to put it back.

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Bart Oldeman wrote:
> I think I would personnally like it best if there wasn't a pagealloc
> either, but just multiple fd's.
Yes, I already recognized that as a
good idea even for mapfile (esp. now
on /dev/shm), so the first step I was
going to take, is to avoid pagemalloc
for mapfile.

> That way you can have only one mapping
> backend, it just differs in what the fd opens (i.e. /dev/zero or a
> temporary file, or perhaps shm_open), and we don't need the mremap 
> trick.
Also we may not need the multiple
mapping backends amymore then, since
the only difference would be how the
fd is being opened.

> alloc_mapping opens a /dev/zero
Was it already verified that this can
be shared without mremap?

  or tmpfile -- the fd will be remembered
> (linked list connecting it to the memory address)
Yes, somewhat resembling the tracking
of kmem mappings in mapping.c I think.
The problem is that the run-time check
for NPTL will not help, we need a
compile-time check if we want to use
open_shm(). That will mean the binary-
distributed dosemu will either have
open_shm() disabled, or not working
with linuxthreads. Or maybe dlopen()
librt? No fun... How's your ptrace patch
going? It could really help in resolving
this mess.

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Lars_BjÛrndal wrote:
>> $_mapping = "mapfile"
>  Yes, it worked! What happens when you spesify that?
Internal thing. Temporary file is
being used for the shared backing-store
(for storing the DOS memory), instead of
a shared mem. Since the file is being
created in /dev/shm, it will not
be slower than the shared mem, so you
loose nothing.
There is a hack now in CVS which will
make WP to work again even without changing
$_mapping option. That hack was posted also
in that list by Bart, so you can try it, it
works. But I don't think it is a good
fix, to me it looks more like a temporary
solution (it makes mapshm and mapfile API to
behave differently, which is IMHO a potential
source of troubles).
I'll try to come up with something more
fundamental soon, that keeps the API
consistent.
But you can just upgrade from CVS and
forget about that problem. Or stay with
$_mapping="mapfile" - that also works
properly.
-
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Bart Oldeman wrote:
>> But it still uses mremap(), although
>> not with a shared mem.
>  yes of course, so that comment should be in dosext/dpmi/memory.c then I
>  guess.
What I was (implicitly) asking is
whether it is safe to use mremap()
at least in this case. It doesn't
work reliably on 2.6 kernels even
though it is used on a private
mapping and I was wondering whether
or not it is safe for 2.4 et al.
It looks like mremap() is not something
to rely upon :(

>> New pages are not aliased, that's for
>> sure, but is this always true also for
>> the old ones? 
>  I'm not sure what you exactly mean here
What I mean is:
You have a 4K-region aliased like this:
0x40020000 -> 0xe0000
Then you resize it to 8K. With
mremap() the above alias should still
be valid (I think), and with memcpy()
you'll have to update the mapping
explicitly.

> but it looks like emm.c takes 
> care of remapping everything.
Very probably, but that was the subject
for cleanups. I think the mapping
subsystem should try to preserve the
aliases at realloc itself, and if it does
so for mapfile, I think it should do that
also for mapshm.

> a real file you'd have to ftruncate it to the new size first. But here 
> we don't have a fd to call ftruncate.
But I wonder if SIGBUS is a correct
behaviour. I would expect mremap()
(nopage handler actually) to do everything
that necessary, including ftruncate().
Otherwise it looks broken to me.

> We would only be able to use
> ftruncate+mremap with explicit /dev/zero mappings or shm_open
So should we get back to pagemalloc
for mapshm then? After all we still
don't need IPC SHM, and pagemalloc
perhaps doesn't really hurt?

Re: Crashes if running WordPerfect 5.1

[Bart Oldeman]

On Sun, 30 May 2004, Stas Sergeev wrote:

> What I mean is: You have a 4K-region aliased like this: 0x40020000 ->
> 0xe0000 Then you resize it to 8K. With mremap() the above alias should
> still be valid (I think), and with memcpy() you'll have to update the
> mapping explicitly.

Yes that's right. The first 4K still map to the same virtual file with
mremap. emm.c I think expects it only to behave as a plain "realloc",
historical reasons probably and that's why it works.

> > a real file you'd have to ftruncate it to the new size first. But here
> > we don't have a fd to call ftruncate.
> But I wonder if SIGBUS is a correct behaviour. I would expect mremap()
> (nopage handler actually) to do everything that necessary, including
> ftruncate(). Otherwise it looks broken to me.

Maybe, but that's more of a kernel issue. mmap certainly isn't guaranteed
to extend a file (it works to create a new file though), so in that way
mremap is behaving consistently (i.e. if you mmap bytes 4096-8191 of a
file with size 4000 you can't expect that file to be extended to 8192
bytes, the same is true for shm virtual files.

Look at the ftruncate example in "info libc" to see how they really need
the ftruncate.

> So should we get back to pagemalloc for mapshm then? After all we still
> don't need IPC SHM, and pagemalloc perhaps doesn't really hurt?

I think I would personnally like it best if there wasn't a pagealloc
either, but just multiple fd's. That way you can have only one mapping
backend, it just differs in what the fd opens (i.e. /dev/zero or a
temporary file, or perhaps shm_open), and we don't need the mremap trick.

That would mean that
open_mapping doesn't have to do anything at all.
alloc_mapping opens a /dev/zero or tmpfile -- the fd will be remembered
(linked list connecting it to the memory address) or it can be returned to
the caller.
realloc_mapping would have to do ftruncate and then mremap
alias_map can mmap from the memory region to the fd.

of course the drawback of this is some overhead taking care of the fd's.
But that gets rid of the overhead in pagemalloc.c and only allocates on
demand (instead of one big chunk in the beginning).

Bart

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Bart Oldeman wrote:
> The patch below also does the job.
Why didn't the mremap() do the trick?

> As far as I could see DPMI no longer
> calls this function
But it still uses mremap(), although
not with a shared mem.

> +  /* expanded new memory is apparently not shared so can't be aliased.
Cant this somehow happen that the EMS
keeps the mapping on some handle and
does the realloc on the same handle?
New pages are not aliased, that's for
sure, but is this always true also for
the old ones?

Re: Crashes if running WordPerfect 5.1

[Bart Oldeman]

Hi,

> Bart Oldeman wrote:
> > The patch below also does the job.
> Why didn't the mremap() do the trick?

see below.

> > As far as I could see DPMI no longer
> > calls this function
> But it still uses mremap(), although
> not with a shared mem.

yes of course, so that comment should be in dosext/dpmi/memory.c then I
guess.

> > +  /* expanded new memory is apparently not shared so can't be aliased.
> Cant this somehow happen that the EMS
> keeps the mapping on some handle and
> does the realloc on the same handle?
> New pages are not aliased, that's for
> sure, but is this always true also for
> the old ones?

I'm not sure what you exactly mean here but it looks like emm.c takes care
of remapping everything.

What happened:
before:
40020000-40024000 rw-s 00000000 00:04 96225      /dev/zero (deleted)

after
40705000-4070d000 rw-s 00000000 00:04 96026      /dev/zero (deleted)

but the second half of this mremapped piece of memory doesn't exist (I get
(gdb) print *(0x40705000)
$2 = 1297889860
(gdb) print *(0x40709000)
Cannot access memory at address 0x40709000
) So my comment needs to be adjusted, the aliasing works but expanding
mremaps of shared memory don't seem to work. It smells a bit like a kernel
bug: the program below gives a bus error...

Bart

#define _GNU_SOURCE 1
#include <unistd.h>
#include <sys/mman.h>
#include <stdio.h>

int main(void)
{
  char *p = mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0);
  p[0] = '1';
  printf("p = %p, p[0] = %c\n", p, p[0]);
  p = mremap(p, 4096, 8192, MREMAP_MAYMOVE);
  printf("after mremap p = %p, p[0] = %c\n", p, p[0]);
  p[4096] = '2';
  return 0;
}

Re: Crashes if running WordPerfect 5.1

[Bart Oldeman]

> It smells a bit like a kernel bug: the program below gives a bus
> error...
>
> #define _GNU_SOURCE 1
> #include <unistd.h>
> #include <sys/mman.h>
> #include <stdio.h>
>
> int main(void)
> {
>   char *p = mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0);
>   p[0] = '1';
>   printf("p = %p, p[0] = %c\n", p, p[0]);
>   p = mremap(p, 4096, 8192, MREMAP_MAYMOVE);
>   printf("after mremap p = %p, p[0] = %c\n", p, p[0]);
>   p[4096] = '2';
>   return 0;
> }

to followup, no it's not a bug.
replacing with
  int fd = open("/tmp/foo", O_RDWR);
  char *p = mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, fd, 0);
also crashes.

For shared anon mappings mmap creates a 4096 bytes sized virtual file. For
a real file you'd have to ftruncate it to the new size first. But here we
don't have a fd to call ftruncate. We would only be able to use
ftruncate+mremap with explicit /dev/zero mappings or shm_open (best
would be to use a different fd for each memory type), but of course that
would be more involving.

Bart

Re: Crashes if running WordPerfect 5.1

[Stas Sergeev]

Hello.

Lars_BjÛrndal wrote:
> Dosemu, and I picked the latest CVS (yesterday evening). Now, I can
> not run WordPerfect 5.1 any more, Dosemu crashes. It's very important
> for me to be able to run wp. What can I do to this?
You can set

$_mapping = "mapfile"

in your dosemu.conf.
-
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crashes if running WordPerfect 5.1

[Bart Oldeman]

Hi,

> Lars_BjÛrndal wrote:
> > Dosemu, and I picked the latest CVS (yesterday evening). Now, I can
> > not run WordPerfect 5.1 any more, Dosemu crashes. It's very important
> > for me to be able to run wp. What can I do to this?
> You can set
>
> $_mapping = "mapfile"
>
> in your dosemu.conf.

The patch below also does the job. As far as I could see DPMI no longer
calls this function so I guess I could remove the comment whilst I was at
it.

Bart

--- src/arch/linux/mapping/mapshm.c.~1.11.~	Sun May  9 11:39:26 2004
+++ src/arch/linux/mapping/mapshm.c	Sat May 29 21:40:08 2004
@@ -147,15 +147,23 @@
   munmap(addr, mapsize);
 }

-/*
- * NOTE: DPMI relies on realloc_mapping() _not_ changing the address ('addr'),
- *       when shrinking the memory region.
- */
 static void *realloc_mapping_shm(int cap, void *addr, int oldsize, int newsize)
 {
+  void *ret;
   Q__printf("MAPPING: realloc, cap=%s, addr=%p, oldsize=%x, newsize=%x\n",
 	cap, addr, oldsize, newsize);
-  return mremap(addr, oldsize, newsize, MREMAP_MAYMOVE);
+
+  if (newsize <= oldsize)
+    return mremap(addr, oldsize, newsize, MREMAP_MAYMOVE);
+
+  /* expanded new memory is apparently not shared so can't be aliased.
+     so we must allocate a new region and memcpy to it */
+  ret = alloc_mapping_shm(cap, newsize);
+  if (ret != MAP_FAILED) {
+    memcpy(ret, addr, oldsize);
+    free_mapping_shm(cap, addr, oldsize);
+  }
+  return ret;
 }

 static void *mmap_mapping_shm(int cap, void *target, int mapsize, int protect, void *source)

-
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crashes if running WordPerfect 5.1

[Lars Bjørndal]

Stas Sergeev <stsp@aknet.ru> writes:

> You can set
>
> $_mapping = "mapfile"

Yes, it worked! What happens when you spesify that?

Crashes if running WordPerfect 5.1

[Lars Bjørndal]

I've upgraded my Red Hat Fedora to Core 2. Then, I also had to upgrade
Dosemu, and I picked the latest CVS (yesterday evening). Now, I can
not run WordPerfect 5.1 any more, Dosemu crashes. It's very important
for me to be able to run wp. What can I do to this?

Lars

Re: Crashes if running WordPerfect 5.1

[Bart Oldeman]

On Sat, 29 May 2004, Lars Bjørndal wrote:

> I've upgraded my Red Hat Fedora to Core 2. Then, I also had to upgrade
> Dosemu, and I picked the latest CVS (yesterday evening). Now, I can
> not run WordPerfect 5.1 any more, Dosemu crashes. It's very important
> for me to be able to run wp. What can I do to this?

Now that you mention it I can only confirm that WP crashes on the current
CVS, it's not Fedora specific. It doesn't crash on the stable branch or
plain 1.2.1 though.

Bart

-
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html