Re: 'dd' local works, but not over net, help as to why?

Re: 'dd' local works, but not over net, help as to why?

[Pete Zaitcev]

> dd if=/dev/hda2 bs=1M|rsh other-system of=/dev/sda2 bs=1M

That does not work on all unixes. Perhaps it does on IRIX,
but certainly not on Solaris. The remote dd is going to
get short reads from the socket. The telltale sign is the
number to short records:

itcev@niphredil /boot]$ dd if=vmlinux-2.4.7-niph bs=8k | ssh pentabug dd of=xxx bs=8k
333+1 records in
333+1 records out
7+653 records in		<------- almost all are shorts
7+653 records out
[zaitcev@niphredil /boot]$ 

There is no good way to deal with it, unless you use some external
filter (perhaps netcat?). tar has a flag -B to deal with this.
An oldtimer's trick is to use compress, which works because
zcat uses stdio for output and its blocksize is fixed.

[zaitcev@niphredil /boot]$ dd if=vmlinux-2.4.7-niph bs=8k | compress | ssh pentabug "zcat |dd of=xxx bs=8k"
333+1 records in
333+1 records out
333+1 records in
333+1 records out
[zaitcev@niphredil /boot]$ 

This behaviour of sockets on Linux is entirely normal, so deal with it.
My pet peeve is how DaveM insists that WRITES to a socket may return
short. That stupidity just makes me furious each time. A socket has
a posix-schmozix "right" to return short, I agree.  But it is extremely
rude and in the right implementation, it should not. It breaks just
about _every_ application that uses printk.

-- Pete

Re: 'dd' local works, but not over net, help as to why?

[L A Walsh]

Hmmm....well both are linux machines with kernels 2.4.10 and 2.4.9.
So I'm not specifying the count -- if the dd on the receiving machine gets
a short read for an 8M block, does that imply that it would write the
short data and pad with 0's?  If that's the case, I can see why it would
run out of space. I know Solaris had the known bug of non-integrity
of binary data transfer over rsh, but I know IRIX doesn't have that and
I've never run into it with linux using 'tar' (w/o the B option).  It
may be the first time I've tried this with 'dd' over the net and with 
such a large dataset.  Is the below output from linux2linux?  I thought
of compress/zcat, but it's a 100Mps local LAN w/little to no traffic, so
I know that almost any compression was going to slow down the xfer...

Y'd think that 'dd' would do what it is told and read the correct block
size rather than returning a 'short'.  Would one consider this a
possible "undesired feature" [bug] in 'dd', if that proves to be the
problem?  Seems like there should at least be a way to say for it to wait
for short blocks.  Hmph.

oh well.
thanks for all the pointers.
Linda



Pete Zaitcev wrote:
> 
> > dd if=/dev/hda2 bs=1M|rsh other-system of=/dev/sda2 bs=1M
> 
> That does not work on all unixes. Perhaps it does on IRIX,
> but certainly not on Solaris. The remote dd is going to
> get short reads from the socket. The telltale sign is the
> number to short records:
> 
> itcev@niphredil /boot]$ dd if=vmlinux-2.4.7-niph bs=8k | ssh pentabug dd of=xxx bs=8k
> 333+1 records in
> 333+1 records out
> 7+653 records in                <------- almost all are shorts
> 7+653 records out
> [zaitcev@niphredil /boot]$
> 
> There is no good way to deal with it, unless you use some external
> filter (perhaps netcat?). tar has a flag -B to deal with this.
> An oldtimer's trick is to use compress, which works because
> zcat uses stdio for output and its blocksize is fixed.
> 
> [zaitcev@niphredil /boot]$ dd if=vmlinux-2.4.7-niph bs=8k | compress | ssh pentabug "zcat |dd of=xxx bs=8k"
> 333+1 records in
> 333+1 records out
> 333+1 records in
> 333+1 records out
> [zaitcev@niphredil /boot]$
> 
> This behaviour of sockets on Linux is entirely normal, so deal with it.
> My pet peeve is how DaveM insists that WRITES to a socket may return
> short. That stupidity just makes me furious each time. A socket has
> a posix-schmozix "right" to return short, I agree.  But it is extremely
> rude and in the right implementation, it should not. It breaks just
> about _every_ application that uses printk.
> 
> -- Pete

--  -    _    -    _    -    _    -    _    -    _    -    _    -    _    -
L A Walsh, law at sgi dot com     | Senior Engineer
01-650-933-5338                   | Trust Technology, Core Linux, SGI

'dd' local works, but not over net, help as to why?

[LA Walsh]

I'm sure there's an obvious answer to this, but it is eluding me.

If I am on my local laptop, I can 'dd' an 8G partition to a
removable HD of the same or slightly larger size (slightly large
because of geometry differences).

If I am on my desktop, "I can 'dd' the same size partition to
a slightly larger one -- again, no problem.

But if I use:

dd if=/dev/hda2 bs=1M|rsh other-system of=/dev/sda2 bs=1M, I
get failures of running out of room on target.  I've tried
a variety of block size ranging from 1K->64G, but no luck.

Is there something in the networking code that's preventing me
from transferring more than a 2 or 4 G limit?

I just wanted an exact image off onto another system.  Would
seem to have been straight forwared. but I guess not?  

Thanks in advance for any work-arounds and explanations...

-linda

Re: 'dd' local works, but not over net, help as to why?

[Matti Aarnio]

On Tue, Oct 02, 2001 at 12:52:48AM -0700, LA Walsh wrote:
> I'm sure there's an obvious answer to this, but it is eluding me.
    Probably...

> If I am on my local laptop, I can 'dd' an 8G partition to a
> removable HD of the same or slightly larger size (slightly large
> because of geometry differences).
> 
> If I am on my desktop, "I can 'dd' the same size partition to
> a slightly larger one -- again, no problem.
> 
> But if I use:
> 
> dd if=/dev/hda2 bs=1M|rsh other-system of=/dev/sda2 bs=1M, I
> get failures of running out of room on target.  I've tried
> a variety of block size ranging from 1K->64G, but no luck.

   You are missing one 'dd' from the other system side, but
   are you also sure that the remote system can support large
   files, and that the dd in there does support large files ?

   Verify that you can 'dd if=/dev/zero of=/dev/sda2 bs=1M'
   at that remote system -- then we know that dd in there
   (and the system) can do it, and the bug-sphere shrinks.

> Is there something in the networking code that's preventing me
> from transferring more than a 2 or 4 G limit?

   There should not be anything such.
   (But of course things like RSH could have some nasty surprises
    in them...)

> I just wanted an exact image off onto another system.  Would
> seem to have been straight forwared. but I guess not?  
> 
> Thanks in advance for any work-arounds and explanations...
> 
> -linda

/Matti Aarnio

RE: 'dd' local works, but not over net, help as to why?

[LA Walsh]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Matti Aarnio
> Sent: Tuesday, October 02, 2001 1:20 AM
> To: LA Walsh
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 'dd' local works, but not over net, help as to why?
>
>
> On Tue, Oct 02, 2001 at 12:52:48AM -0700, LA Walsh wrote:
> > I'm sure there's an obvious answer to this, but it is eluding me.
>     Probably...
>
> > If I am on my local laptop, I can 'dd' an 8G partition to a
> > removable HD of the same or slightly larger size (slightly large
> > because of geometry differences).
> >
> > If I am on my desktop, "I can 'dd' the same size partition to
> > a slightly larger one -- again, no problem.
> >
> > But if I use:
> >
> > dd if=/dev/hda2 bs=1M|rsh other-system of=/dev/sda2 bs=1M, I
> > get failures of running out of room on target.  I've tried
> > a variety of block size ranging from 1K->64G, but no luck.
>
>    You are missing one 'dd' from the other system side, but
>    are you also sure that the remote system can support large
>    files, and that the dd in there does support large files ?
---
	Missing 'dd' typo.  It's on the other system I tried copying the 8G
to a slightly large 9G partition -- that worked.  On the source system
I can copy 8G to another 8G partition.  Just running them over rsh seems to
be a problem.  Same version of 'dd' on each side (SuSE 7.2).

	Maybe I can fool ftp with symlinks tomorrow into doing the copy and see
if that works.  Just for fun I tried 'cat' as well -- same error -- out of
space on target.

It transfers a lot of data -- right around 2G the first time I tried it, so
it looked awfully suspicious.

Linda

Re: 'dd' local works, but not over net, help as to why?

[Matti Aarnio]

On Tue, Oct 02, 2001 at 01:35:28AM -0700, LA Walsh wrote:
> >    You are missing one 'dd' from the other system side, but
> >    are you also sure that the remote system can support large
> >    files, and that the dd in there does support large files ?
> ---
> 	Missing 'dd' typo.  It's on the other system I tried copying the 8G
> to a slightly large 9G partition -- that worked.  On the source system
> I can copy 8G to another 8G partition.  Just running them over rsh seems to
> be a problem.  Same version of 'dd' on each side (SuSE 7.2).

   There could be something amiss in the dd when feeding into a pipe
   ( 2G limit should not matter, but...)  

   Try following:

	dd if=/dev/sdX | cat > /dev/null

   Hmm..  And also:

	dd if=/dev/sdX | rsh other-host dd of=/dev/null


> 	Maybe I can fool ftp with symlinks tomorrow into doing the copy and see
> if that works.  Just for fun I tried 'cat' as well -- same error -- out of
> space on target.

	'cat' local or remote ?
 
> It transfers a lot of data -- right around 2G the first time I tried it, so
> it looked awfully suspicious.

	Yes, suspicuous about something in the rsh barfing.

> Linda

/Matti Aarnio

Re: 'dd' local works, but not over net, help as to why?

[Alistair Riddell]

On Tue, 2 Oct 2001, LA Walsh wrote:

> Thanks in advance for any work-arounds and explanations...

not sure about explanations, but have you tried using netpipes (faucet /
hose) as a work-around...?

-- 
Alistair Riddell - BOFH
IT Manager, George Watson's College, Edinburgh
Tel: +44 131 447 7931 Ext 176       Fax: +44 131 452 8594
Microsoft - because god hates us