[linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Michael Tokarev]

Hello!

I finally got working (at least at first view) lvm with
RedHat's 2.2.17-8 kernel (from rawhide) and lvm-0.9.new_raid.patch,
patched by Andreas Dilger (with small additional changes).

So far, so good.

As I can see, lvm patch should go with sct's rawio patch, so
I conclude that them should work together.  Isn't it? :)

My main goal is to use oracle database with raw devices on
top of lvm (using some number of disks, so that total
storage size is large and needs to be managed intelligently).
This is IMHO a great thing to have with linux -- Oracle's
best results can be achieved on raw devices, and those needs
to be managed (using disk partitions is a PITA here).

I've made simple LV, and attached raw device on top of it,
using `raw' utility.  And what I've noticied is that I
can't write 512-byte blocks to it.  The only block size
I can use is 1024, 2048, 3072, etc, i.e. 1024*n.  With
just lvm device it is ok (seemed to be), but with /dev/raw
device write/read gives "invalid argument" error message.

The bad thing is that Oracle tries to write 512 bytes
_when creating tablespace_ (I've set up it to use 4k
blocks, so it will read/write 4096*n blocks after ts
creation).  I attached some strace output from oracle
process when creating tablespace, below.

/dev/raw/raw100 bound to /dev/vg0/ora0 lv (128M).
 dd if=/dev/zero of=/dev/raw/raw100 bs=512
  dd: /dev/raw/raw100: Invalid argument
  1+0 records in
  0+0 records out

But what's interesting is that I already have set up
some databases to use raw devices, and them working
good (no glitches was found so far).  I used "plain"
disk partitions for this, and softraid-devices, e.g.
  partition => rawdevice => oracle datafile
  partition,partition => raid0 => rawdevice => oracle

/dev/raw/raw1 bound to /dev/sda2 (1G)
 dd if=/dev/zero of=/dev/raw/raw1 bs=512
 ^C
 281835+0 records in
 281834+0 records out

(I've just hit ^C here for. Process will complete
correctly).

So the question: why read/write fails with rawio
on top of lvm when requesting "incorrect" block size?

Strace excerpt below.  What I noticied is that
oracle tried to use different methods here, but
all failed.  Some of them used with 1024-multiple
sizes only, but also failed.
BTW, does anybody knows what's "pwrite()" ?

Oracle 8.1.6 EE (Oracle8iR2) for Linux.

I think that we all interested in resolving this
particular issue.  I'll be glad to try different things
here as well, and provide any additional info, or providing
all my experience for this... And just one thing -- may this
be due to strangers with lvm patch (0.9-2.2.17-new_raid +
Andreas "patch for patch" + my *minor* tweaks)? 

Thank you.

Regards,
 Michael.

SQL> create tablespace x datafile '/dev/raw/raw100' size 100m reuse;
...
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDONLY)       = 9
close(9)                                = 0
open("/dev/raw/raw100", O_RDWR)         = 9
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
fstat(408, {st_mode=S_IFREG|0640, st_size=1077248, ...}) = 0
fstat(407, 0xbfffb65c)                  = -1 EBADF (Bad file descriptor)
dup2(9, 407)                            = 407
close(9)                                = 0
fcntl(407, F_SETFD, FD_CLOEXEC)         = 0
fcntl(407, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl(407, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDWR)         = 9
lseek(9, 0, SEEK_SET)                   = 0
write(9, "\0\0\0\0\0\20\0\0\0d\0\0]\\[Z\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(9, 104861184, SEEK_SET)           = 104861184
read(9, 0x92e4800, 512)                 = -1 EINVAL (Invalid argument)
close(9)                                = 0
fcntl(407, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(407)                              = 0
old_mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40191000
old_mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x401d2000
old_mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40213000
old_mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40254000
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDONLY)       = 9
close(9)                                = 0
open("/dev/raw/raw100", O_RDWR)         = 9
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
fstat(407, 0xbfffaf58)                  = -1 EBADF (Bad file descriptor)
dup2(9, 407)                            = 407
close(9)                                = 0
fcntl(407, F_SETFD, FD_CLOEXEC)         = 0
fcntl(407, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl(407, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDWR)         = 9
lseek(9, 0, SEEK_SET)                   = 0
write(9, "\0\0\0\0\0\20\0\0\377\377\377\377]\\[Z\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(9, 4294966784, SEEK_SET)          = -1 EINVAL (Invalid argument)
close(9)                                = 0
fcntl(407, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(407)                              = 0
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDONLY)       = 9
close(9)                                = 0
gettimeofday({975703953, 657435}, NULL) = 0
open("/dev/raw/raw100", O_RDWR)         = 9
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024}) = 0
fstat(407, 0xbfffb378)                  = -1 EBADF (Bad file descriptor)
dup2(9, 407)                            = 407
close(9)                                = 0
fcntl(407, F_SETFD, FD_CLOEXEC)         = 0
pwrite(407, "\0\2\0\0\1\0\0\0\0\0\0\0\0\0\1\1\0\0\0\0\0\0\0\0\0\0\0"..., 262144, 4096) = -1 EINVAL (Invalid argument)
pwrite(407, "\0\2\0\0A\0\0\0\0\0\0\0\0\0\1\1\0\0\0\0\0\0\0\0\0\0\0\0"..., 262144, 266240) = -1 EINVAL (Invalid argument)
pwrite(407, "\0\2\0\0\201\0\0\0\0\0\0\0\0\0\1\1\0\0\0\0\0\0\0\0\0\0"..., 262144, 528384) = -1 EINVAL (Invalid argument)
pwrite(407, "\0\2\0\0\301\0\0\0\0\0\0\0\0\0\1\1\0\0\0\0\0\0\0\0\0\0"..., 262144, 790528) = -1 EINVAL (Invalid argument)
stat("/dev/raw/raw100", {st_mode=S_IFCHR|0660, st_rdev=makedev(162, 100), ...}) = 0
open("/dev/raw/raw100", O_RDWR)         = 9
lseek(9, 0, SEEK_SET)                   = 0
write(9, "\0\0\0\0\0\20\0\0\0\1\0\0]\\[Z\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(9, 1052160, SEEK_SET)             = 1052160
read(9, 0x92e4800, 512)                 = -1 EINVAL (Invalid argument)
close(9)                                = 0
fcntl(407, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(407)                              = 0
gettimeofday({975703953, 665286}, NULL) = 0
gettimeofday({975703953, 665470}, NULL) = 0
close(6)                                = 0
open("/usr/oracle/dbs/orcl/bgdump/alert_orcl.log", O_WRONLY|O_APPEND|O_CREAT, 0664) = 6
... "ORA-19502 signalled during: crea"

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Andreas Dilger]

Michael Tokarev writes:
> The bad thing is that Oracle tries to write 512 bytes
> _when creating tablespace_ (I've set up it to use 4k
> blocks, so it will read/write 4096*n blocks after ts
> creation).  I attached some strace output from oracle
> process when creating tablespace, below.

One "hack" you could try when creating the tablespaces initially is
to symlink /dev/raw/raw100 to the block device you are using
(in this case /dev/vg0/ora0), and then when you are done with
the tabelspace creation remove the symlink and set up the raw
device as before.

I'm not 100% sure this will work, however.  Is the /dev/raw directory
a "virtual filesystem" like proc or /dev/shm?  The other possibility
is that Oracle will treat it differently because it is a block device
and not a character device, but I'm not sure of that either.

As to the real problem, I have no idea.  There were a couple of changes
that Heinz made to LVM w.r.t. block sizes, but I think that was limited
to removing constants, and not changing the actual block handling.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Michael Ju. Tokarev]

Andreas Dilger wrote:
> 
> Michael Tokarev writes:
> > The bad thing is that Oracle tries to write 512 bytes
> > _when creating tablespace_ (I've set up it to use 4k
> > blocks, so it will read/write 4096*n blocks after ts
> > creation).  I attached some strace output from oracle
> > process when creating tablespace, below.
> 
> One "hack" you could try when creating the tablespaces initially is
> to symlink /dev/raw/raw100 to the block device you are using
> (in this case /dev/vg0/ora0), and then when you are done with
> the tabelspace creation remove the symlink and set up the raw
> device as before.

What's *still* unknown to me is a difference between character and
block specials.  In principle, block device should only allow
block access (i.e. multiple of 512 or 1024 or whatether size),
while character devs should allow read of 1 byte.  For example,
solaris doesn't allow "any-size" i/o on block devices -- things
will be bad here (when I read one byte, I got it, but next byte
read will actually be 513'th one, not 2nd etc).  Linux sometimes
gives us errors (invalid argument) in such a cases, and sometimes
not (lvm devices doesn't generate this errors).  The whole rawio
thing as it seemed *to me* to be, is to provide some layer between
block and char i/o, so it should be possible to use any read/write
size with it.  At the other hand, rawio (as stated at sgi oss site
at least), while uses character devices, requires even more
restrictions to be meet -- also about aligning memory buffers
used for read/write requests etc.  I'm confused here... :(

> I'm not 100% sure this will work, however.

This will not work.  I don't know if it's an oracle bug or not
(seemed to be), but if I give to it *block* special file (like
lvm's lv), it will try to remove that file (device) first and
than complain that "file already exists" (funny ;).  But this
triggers something in my mind -- I can create tablespace in
regular file and than copy that file to lv.  For this I don't
shure if it will work or not (will check later - interesting),
but for shure this isn't a solution, since we'll not be able
to extend that datafile from oracle (like extending filesystem
on top of lv), since this again will require writing/reading
like at initial creation time, -- so almost all lvm work will
go away.

> Is the /dev/raw directory
> a "virtual filesystem" like proc or /dev/shm?  The other possibility

No, it's just a plain subdirectory with a bunch (254) files named
raw1..raw254.  It looks like RedHat uses this layout (since there
is no standard layout for rawdevs like for sda hdb etc) to be
prepared for 2.4 kernel (I use 2.2.17).  Linux have no character
devices for disks etc for now, so the rawio patch is for that
purpose (e.g. solaris always had /dev/dsk/xxx devices - that are
character specials, and /dev/rdsk/xxx, block specials (or the
opposite, I don't remember) that have the same major/minor but
different type; linux lacks this).  For now, linux's way is a
ugly -- one should "bound" block device to one of /dev/raw/xxx
device using supplied `raw' utility, and after that can open
that /dev/raw/xxx and use it.  One thing that stops me trying
lvm in "my real life" is that I didn't know if it can work together
with rawio patch.  For now, lvm patch requires rawio patch, so
I concluded that them works together well and tried that...

> is that Oracle will treat it differently because it is a block device
> and not a character device, but I'm not sure of that either.

Shure it will, and will refuse to do something... ;)

Seriously, for me it's not a critical to *not* to use lvm for this
sort of things (yet), there are another working solutions already
exists.  With lvm, things should be far easily.  The main goal
of my original post is to enshure (or make it that) that linux works
in this environment (there should be high demand for raw partitions
on top of lvm in enterprise level, and oracle is the main consumer
for that raw devices).

> 
> As to the real problem, I have no idea.  There were a couple of changes
> that Heinz made to LVM w.r.t. block sizes, but I think that was limited
> to removing constants, and not changing the actual block handling.

This again reminds my "unknowlege" of block/char specials -- should
*block* lvm devices have some *block* limits or not ?! ;) But ok.

I need to check if "stock" lvm patches for stock kernel will work
here (using "retired" lvm-0.9-2.2.17-new_raid.patch and a patch
for that patch isn't a clean experiment ;) ), and will post results
here (but this will be at least at monday, as I won't not reboot
machines at work from home -- all my experiments are at work, I
have no hardware to test things at home machine).  If things will
go, I'll see what's different in two situations...

BTW, here, we are using both lvm and softraid together.  The
test machine have hardware raid controller (5x18Gb, level 5,
total ~72Gb, 68Gb used for one pv that I want to manage for
oracle datafiles), and another four 18G disks on different
controller, used as some number of softraid1-pairs (again,
with raw devices on top and oracle usage).  So it is important
that all subsystem works -- lvm, md and rawio.  Md together with
rawio works pretty well, but not lvm+rawio.  I don't want to
try lvm+md... :)

BTW, one (little?) question.  There was a thread on the list
recently subjected "lvm 0.9 - make_req_fn", for that I don't
have beginning (it was a time when mailinglist switched from
msede.com to sistina.com).  Is there any place where this
can be found ? (sistina's archives are quiet about this).
I ask because I got the problem when tried to apply (really
adopt) patches to redhat-patched kernel exactly with this
(make_req_fn), and wanted to ask list too about this (and
found end of that thread...).

Thanks!

Regards,
 Michael.

P.S.  I like lvm tools -- the first look was *very* nice
tool collection, with very friendly interface/concepts...

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Christoph Hellwig]

On Sat, Dec 02, 2000 at 03:35:18AM +0300, Michael Ju. Tokarev wrote:
> What's *still* unknown to me is a difference between character and
> block specials.  In principle, block device should only allow
> block access (i.e. multiple of 512 or 1024 or whatether size),
> while character devs should allow read of 1 byte.  For example,
> solaris doesn't allow "any-size" i/o on block devices -- things
> will be bad here (when I read one byte, I got it, but next byte
> read will actually be 513'th one, not 2nd etc).  Linux sometimes
> gives us errors (invalid argument) in such a cases, and sometimes
> not (lvm devices doesn't generate this errors).  The whole rawio
> thing as it seemed *to me* to be, is to provide some layer between
> block and char i/o, so it should be possible to use any read/write
> size with it.  At the other hand, rawio (as stated at sgi oss site
> at least), while uses character devices, requires even more
> restrictions to be meet -- also about aligning memory buffers
> used for read/write requests etc.  I'm confused here... :(

Ok.  The blockdevices itself don't have a direct interface to the
userspace.  They are only called by ll_rw_block.  The blockdevice
specials are implemented in fs/block_dev.c, and allow cached access
to the block devices (currently through the buffer cache, but it may
move to the pagecache soon).  The rawio stuff on the other hand allows
raw, uncached access to the block devices.  It has another advantage:
by using kiobufs the data does not have to be copied from user- to
kernelspace or vice versa.

> > Is the /dev/raw directory
> > a "virtual filesystem" like proc or /dev/shm?  The other possibility
> 
> No, it's just a plain subdirectory with a bunch (254) files named
> raw1..raw254.  It looks like RedHat uses this layout (since there
> is no standard layout for rawdevs like for sda hdb etc) to be
> prepared for 2.4 kernel (I use 2.2.17).

Nope.  You can use whatever /dev layout you want with every kernel you
want unless you use devfs.  The newer raw versions just use the /dev/raw
directory instead of having the files directly in /dev.

> Linux have no character
> devices for disks etc for now, so the rawio patch is for that
> purpose (e.g. solaris always had /dev/dsk/xxx devices - that are
> character specials, and /dev/rdsk/xxx, block specials (or the
> opposite, I don't remember)

Yepp. The opposite... (rdsk = raw disk).

> that have the same major/minor but
> different type; linux lacks this).  For now, linux's way is a
> ugly -- one should "bound" block device to one of /dev/raw/xxx
> device using supplied `raw' utility, and after that can open
> that /dev/raw/xxx and use it.  One thing that stops me trying
> lvm in "my real life" is that I didn't know if it can work together
> with rawio patch.  For now, lvm patch requires rawio patch, so
> I concluded that them works together well and tried that..

LVM doesn't really need rawio but the kiobuf infrastructure included
in the rawio patch.  Without snapshot LVs it should be usable without
that patch.  And yes, LVM works with rawio.  Rawio works with every
blockdvice, because it needs no special support on the block device
layer.

> > 
> > As to the real problem, I have no idea.  There were a couple of changes
> > that Heinz made to LVM w.r.t. block sizes, but I think that was limited
> > to removing constants, and not changing the actual block handling.
> 
> This again reminds my "unknowlege" of block/char specials -- should
> *block* lvm devices have some *block* limits or not ?! ;) But ok.

Block lvm devices shouldn't have a limit on the number of read/written
chars.  Actually block special devices is a very confusing name.  These
are just device files that allow cached access to the block devices.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Jorg de Jong]

Hi,

your mail trigged me to try this on my system! 

SQL> create tablespace x datafile '/dev/raw/raw1' ;
create tablespace x datafile '/dev/raw/raw1'

Tablespace created.

and I did not have any problems what so ever!
my setup is : 
- rh7.0 
- 2.4.11 stock kernel with 
- plain old LVM 8.0 (usertools patched)
- Oracle is 8.1.6 SE

regards 
-- 
Jorg de Jong
Work : mailto:jorg.de.jong@ict.nl 
Play : mailto:j.e.s.de.jong@freeler.nl

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Jorg de Jong]

Hi,

your mail trigged me to try this on my system! 

SQL> create tablespace x datafile '/dev/raw/raw1' ;
create tablespace x datafile '/dev/raw/raw1'

Tablespace created.

and I did not have any problems what so ever!
my setup is : 
- rh7.0 
- 2.4.11 stock kernel with 
- plain old LVM 8.0 (usertools patched)
- Oracle is 8.1.6 SE

regards 
-- 
Jorg de Jong
Work : mailto:jorg.de.jong@ict.nl 
Play : mailto:j.e.s.de.jong@freeler.nl

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Michael Ju. Tokarev]

Jorg de Jong wrote:
> 
> Hi,
> 
> your mail trigged me to try this on my system!
> 
> SQL> create tablespace x datafile '/dev/raw/raw1' ;
> create tablespace x datafile '/dev/raw/raw1'
> 
> Tablespace created.
> 
> and I did not have any problems what so ever!
> my setup is :
> - rh7.0
> - 2.4.11 stock kernel with

What's the "2.4.11" kernel?  I think that there is no
such beast exists, and will not exists for at least
some months...  There is only 2.4testN, i.e. pre-2.4.
Redhat 7.0 ships with something like 2.2.16-22, and
included 2.4-0.something -- maybe this is what you
have?  And maybe things go well on 2.4-tobe kernel?
(I tried with 2.2.17, 2.4 is still not an option for
our applications).

> - plain old LVM 8.0 (usertools patched)
> - Oracle is 8.1.6 SE
> 

Regards,
 Michael.

Re: [linux-lvm] Using Oracle with lvm AND rawio: read(512) from /dev/raw/...

[Jorg de Jong]

"Michael Ju. Tokarev" wrote:
> 
> Jorg de Jong wrote:
> >
> > my setup is :
> > - rh7.0
> > - 2.4.11 stock kernel with
> 
> What's the "2.4.11" kernel?  I think that there is no
> such beast exists, and will not exists for at least
> some months...  There is only 2.4testN, i.e. pre-2.4.
> Redhat 7.0 ships with something like 2.2.16-22, and
> included 2.4-0.something -- maybe this is what you
> have?  And maybe things go well on 2.4-tobe kernel?
> (I tried with 2.2.17, 2.4 is still not an option for
> our applications).

your are ride offcource,
what I ment was 
- 2.4.0-test11 stock kernel ( downloaded from ftp.kernel.org ) 

> 

-- 
Jorg de Jong
Work : mailto:jorg.de.jong@ict.nl 
Play : mailto:j.e.s.de.jong@freeler.nl