[dm-crypt] out of memory

[dm-crypt] out of memory

[Paul Dugas]

I'm working with an up-to-date CENTOS-5 x86_64 machine connected via
ATA-over-Ethernet to a Coraid array.  I have one drive in the array
that I want to use for manually transported offsite backups and I want
the drives to be encrypted.  I've done a fair bit of reading but am
having trouble.  Since I can't really find a single source of the
trouble, I thought it was time to ask for some input.

As I mentioned earlier, the drive is in an AoE array.  It's setup as a
single drive (JBOD, no raid) and it properly appears as
/dev/etherd/e1.12; 12th drive in shelf 1.  The drive capacity is 2TB.
I started like so:

     # cryptsetup -y luksFormat /dev/etherd/e1.12
         confirm and set password
     # cryptsetup luksOpen /dev/etherd/e1.12 offsitefs
         give it the password
     # cryptsetup status offsitefs
         just checking

No trouble up to here.  Then, based on guidance found online, I ran dd like so.

     # dd if=/dev/urandom of=/dev/mapper/offsitefs
         come back tomorrow for the dd to be finished...

The first time I tried this, the machine almost immediately went to
its knees.  Console and terminal sessions staggered to a crawl.
Messages on the console indicated the machine was out of memory.  I
eventually gave up on it and cycled power.

I tried it again but was prepared with some terminals running top and
other monitors before I started the dd.  I saw the dd process
consuming some CPU but not terribly much.  This time, the dd evenually
completed.  It took a couple days but it did eventually complete.

Next, I setup the filesystem like so:

     # mke2fs -j -O dir_index /dev/mapper/offsitefs
         create ext2+journal filesystem

This also brought the machine to its knees but after a couple
power-cycle iterations and a couple more days, it completed and I was
able to mount the drive and use it to a point.  I was able copy most
of our backup files (tar.gz files) to the drive but the largest one
(over 300GB) failed.  I didn't catch why as I'd let it chug along
without me again.

My first question is whether or not I've made some fundamental error
in the approach I'm taking.  Should this work?

My second question is if I just need to add more RAM to the machine
(currently on 2GB).

Third, what additional information can I collect to narrow down the
issues I'm having?

Thanks in advance,

Paul
--
Paul Dugas • Dugas Enterprises, LLC • Computer Engineer •
Paul@DugasEnterprises.com

Re: [dm-crypt] out of memory

[Arno Wagner]

On Wed, Oct 06, 2010 at 08:48:24PM -0400, Paul Dugas wrote:
> I'm working with an up-to-date CENTOS-5 x86_64 machine connected via
> ATA-over-Ethernet to a Coraid array.  I have one drive in the array
> that I want to use for manually transported offsite backups and I want
> the drives to be encrypted.  I've done a fair bit of reading but am
> having trouble.  Since I can't really find a single source of the
> trouble, I thought it was time to ask for some input.
> 
> As I mentioned earlier, the drive is in an AoE array.  It's setup as a
> single drive (JBOD, no raid) and it properly appears as
> /dev/etherd/e1.12; 12th drive in shelf 1.  The drive capacity is 2TB.
> I started like so:
> 
>      # cryptsetup -y luksFormat /dev/etherd/e1.12
>          confirm and set password
>      # cryptsetup luksOpen /dev/etherd/e1.12 offsitefs
>          give it the password
>      # cryptsetup status offsitefs
>          just checking
> 
> No trouble up to here.  Then, based on guidance found online, I ran dd like so.
> 
>      # dd if=/dev/urandom of=/dev/mapper/offsitefs
>          come back tomorrow for the dd to be finished...
> 
> The first time I tried this, the machine almost immediately went to
> its knees.  Console and terminal sessions staggered to a crawl.
> Messages on the console indicated the machine was out of memory.  I
> eventually gave up on it and cycled power.
> 
> I tried it again but was prepared with some terminals running top and
> other monitors before I started the dd.  I saw the dd process
> consuming some CPU but not terribly much.  This time, the dd evenually
> completed.  It took a couple days but it did eventually complete.
> 
> Next, I setup the filesystem like so:
> 
>      # mke2fs -j -O dir_index /dev/mapper/offsitefs
>          create ext2+journal filesystem
> 
> This also brought the machine to its knees but after a couple
> power-cycle iterations and a couple more days, it completed and I was
> able to mount the drive and use it to a point.  I was able copy most
> of our backup files (tar.gz files) to the drive but the largest one
> (over 300GB) failed.  I didn't catch why as I'd let it chug along
> without me again.
> 
> My first question is whether or not I've made some fundamental error
> in the approach I'm taking.  Should this work?

It should. I don't see any obvious problem.
 
> My second question is if I just need to add more RAM to the machine
> (currently on 2GB).

Not really. LUKS does not need a lot of memory.

> Third, what additional information can I collect to narrow down the
> issues I'm having?

The question would be where the memory actually goes. Is there maybe
some bus or disk (i.e. hardware) problem that slows down writes to
the disk and the memory all goes to the buffer? Or does it go 
to some process? top or ps -auxwww + free should bring insights
into that.

One thing you can try is to use dd_rescue instead of dd, as it
gives progress statistics. Then you can try to zero the raw disk
and see whether that goes with the expected speed. This will kill
what you already have, sorry. But there seems to be something
fundamentally wrong on writing. 

BTW, you can just zero /dev/mapper/offsitefs, reading from urandom
is slow and completely unnecessary, read from /dev/zero instead.


Arno

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] out of memory

[Milan Broz]

On 10/07/2010 05:31 AM, Arno Wagner wrote:
> On Wed, Oct 06, 2010 at 08:48:24PM -0400, Paul Dugas wrote:
>> I'm working with an up-to-date CENTOS-5 x86_64 machine connected via
>> ATA-over-Ethernet to a Coraid array.  I have one drive in the array

I would say that the problem will be more caused by network
layer (ATAoE) than dm-crypt (or combination of these).

Anyway, dmcrypt in RHEL5 is not upstream, if it is reproducible,
please report it to Red Hat Bugzilla with all used packages
and configuration. Ideally with full OOM log also.

It will not get the highest priority (well, it is Centos :)
but I would like to check it later.
(It will be assigned to me anyway:-)

You can try iSCSI instead to check if it helps.
But it should basically work for any block device without OOM,
even with very low RAM.

In any case this is kernel problem, not cryptsetup one.

(And I hope I will have more time for dmcrypt/cryptsetup in next weeks.)

Milan