[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Hi!

I'm using buildroot revision 23203. After all seems to work on my 
at91sam9263 based board, I figured out that scp transferes not more than 
800kByte in one file. This is not acceptable and I tried openssh instead 
of dropbear. Still, the maximum transferable file size is less than 800k.

Any idea where the limit is bound to?

Regards
	akuehn

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Peter Korsgaard]

>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:

 Andreas> Hi!

 Andreas> I'm using buildroot revision 23203. After all seems to work
 Andreas> on my at91sam9263 based board, I figured out that scp
 Andreas> transferes not more than 800kByte in one file. This is not
 Andreas> acceptable and I tried openssh instead of dropbear. Still,
 Andreas> the maximum transferable file size is less than 800k.

Works here:

scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/
root at 150.158.210.109's password:
binutils-2.18.tar.bz2                         100%   14MB   2.0MB/s   00:07

 Andreas> Any idea where the limit is bound to?

Maybe you're running out of disk (flash/ramdisk) space?

-- 
Bye, Peter Korsgaard

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Peter Korsgaard wrote:
>>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:
> 
>  Andreas> Hi!
> 
>  Andreas> I'm using buildroot revision 23203. After all seems to work
>  Andreas> on my at91sam9263 based board, I figured out that scp
>  Andreas> transferes not more than 800kByte in one file. This is not
>  Andreas> acceptable and I tried openssh instead of dropbear. Still,
>  Andreas> the maximum transferable file size is less than 800k.
> 
> Works here:
> 
> scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/
> root at 150.158.210.109's password:
> binutils-2.18.tar.bz2                         100%   14MB   2.0MB/s   00:07
> 
>  Andreas> Any idea where the limit is bound to?
> 
> Maybe you're running out of disk (flash/ramdisk) space?
> 
Hey Peter!
Fine hearing that it should work as expected.

About the memory capacity:
Mem: 8612K used, 118892K free, 0K shrd, 320K buff, 5168K cached
rootfs: 119.5M used, 56.0M free

I still expect something like that ext2-fs has some limitation or uClibc
has some buffer length to be adjusted. But if not, it sounds like my
buildroot needs some "make dirclean" and hopefully all the quirks
disappear. Any other ideas?

Not knowing the cause is somewhat uncomfortable. (I hate keeping fingers
crossed. Last time I dropped my coffee pot!)


Regards
	akuehn

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Hinko Kocevar]

Andreas Kuehn wrote:
> 
> Peter Korsgaard wrote:
>>>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:
>>  Andreas> Hi!
>>
>>  Andreas> I'm using buildroot revision 23203. After all seems to work
>>  Andreas> on my at91sam9263 based board, I figured out that scp
>>  Andreas> transferes not more than 800kByte in one file. This is not
>>  Andreas> acceptable and I tried openssh instead of dropbear. Still,
>>  Andreas> the maximum transferable file size is less than 800k.
>>
>> Works here:
>>
>> scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/
>> root at 150.158.210.109's password:
>> binutils-2.18.tar.bz2                         100%   14MB   2.0MB/s   00:07
>>

Here too.

> disappear. Any other ideas?
> 
> Not knowing the cause is somewhat uncomfortable. (I hate keeping fingers
> crossed. Last time I dropped my coffee pot!)

Try running scp and/or dropbear on target under strace or better gdbserver.
Does it also limit file size if you transfer file from target to host or only other way around?

Regards,
Hinko

-- 
Hinko Ko?evar, OSS developer
?ETRTA POT, d.o.o.
Planina 3, 4000 Kranj, SI EU
tel     ++386 (0) 4 280 66 03
e-mail  hinko.kocevar at cetrtapot.si
http    www.cetrtapot.si

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Hinko Kocevar wrote:
> Andreas Kuehn wrote:
>> Peter Korsgaard wrote:
>>>>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:
>>>  Andreas> Hi!
>>>
>>>  Andreas> I'm using buildroot revision 23203. After all seems to work
>>>  Andreas> on my at91sam9263 based board, I figured out that scp
>>>  Andreas> transferes not more than 800kByte in one file. This is not
>>>  Andreas> acceptable and I tried openssh instead of dropbear. Still,
>>>  Andreas> the maximum transferable file size is less than 800k.
>>>
>>> Works here:
>>>
>>> scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/
>>> root at 150.158.210.109's password:
>>> binutils-2.18.tar.bz2                         100%   14MB   2.0MB/s   00:07
>>>
> 
> Here too.
> 
>> disappear. Any other ideas?
>>
>> Not knowing the cause is somewhat uncomfortable. (I hate keeping fingers
>> crossed. Last time I dropped my coffee pot!)
> 
> Try running scp and/or dropbear on target under strace or better gdbserver.
> Does it also limit file size if you transfer file from target to host or only other way around?
> 
> Regards,
> Hinko
> 
First, scp (sftp) segfaults in both directions.

  - getting a file from the host to the box ends at 880KB
  - putting a file from the box to the host ends at 992KB


  1. Here is the debug output of "scp user at host:gimp.ps ."
Right in the middle you see a "segmentation fault"

  2. Further down you can find the not very enlighting result from
"strace scp user at host:gimp.ps . "

  3. I tried it the opposite way. Sitting on the host and scp a file 
from the box. The result and -vvv output is even more scary!

Well, summing up all, it looks as there are different aspects 
interfering each other but all of it is around the openssh system. 
Configuration is one part but currently I redo the buildroot system with 
my current configuration to get rid of leftovers from previous 
configurations. (That'll take a while...)
Nevertheless, can someone read tea leaves from the below?

Regards
	akuehn



Output of "scp user at host:gimp.ps ."

Sending file modes: C0644 4146862 gimp.ps
debug2: channel 0: written 42 to efd 6
Sink: C0644 4146862 gimp.ps
gimp.ps                                         0%    0     0.0KB/s 
--:-- ETAdebug2: channel 0: window 65472 sent adjust 65600
debug2: channel 0: window 49152 sent adjust 81920
debug2: channel 0: window 49152 sent adjust 81920
    * akuehn: this message comes multiple times *
debug2: channel 0: window 49152 sent adjust 81920
gimp.ps                                        22%  896KB 896.0KB/s 
00:01    debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: output open -> closed
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: window 49152 sent adjust 65536
Segmentation fault

debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
    * akuehn: this message comes multiple times *

debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

debug3: channel 0: close_fds r -1 w -1 e 6 c -1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 42 seconds
debug1: Bytes per second: stdin 00, stdout 00, stderr 00
debug1: Exit status 1


strace output "strace scp user at host:gimp.ps . "

read(7, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
write(3, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
read(7, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
write(3, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
read(7, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
write(3, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
--- SIGALRM (Alarm clock) @ 0 (0) ---
getpgrp()                               = 884
ioctl(1, TIOCGPGRP, [884])              = 0
time(NULL)                              = 6388
gimp.ps                                         8%  324KB 324.0KB/s 
00:01    ) = 80
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++



Doing "scp user at box:openocd ."
   ...

debug3: Ignored env LESSCLOSE
debug3: Ignored env _
debug1: Sending command: scp -v -f openocd
debug2: channel 0: request exec confirm 0
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
   #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

debug3: channel 0: close_fds r -1 w -1 e 7 c -1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status -1

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Okay, I did a complete rebuild of buildroot with the same configuration 
as befor. The transfer limitation is still there. Unfortunately, a have 
no alternative host system to check if the hosts configuration is what 
causes the trouble.
Host: i686/Debian Edge; SSH-2.0-OpenSSH_4.3p2 Debian-9etch3
Box : at91sam9263/buildroot; SSH-2.0-OpenSSH_4.6

Questions, Ideas?

Regards
	akuehn




Hinko Kocevar wrote:
> Andreas Kuehn wrote:
>> Peter Korsgaard wrote:
>>>>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:
>>>  Andreas> Hi!
>>>
>>>  Andreas> I'm using buildroot revision 23203. After all seems to work
>>>  Andreas> on my at91sam9263 based board, I figured out that scp
>>>  Andreas> transferes not more than 800kByte in one file. This is not
>>>  Andreas> acceptable and I tried openssh instead of dropbear. Still,
>>>  Andreas> the maximum transferable file size is less than 800k.
>>>
>>> Works here:
>>>
>>> scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/
>>> root at 150.158.210.109's password:
>>> binutils-2.18.tar.bz2                         100%   14MB   2.0MB/s   00:07
>>>
> 
> Here too.
> 
>> disappear. Any other ideas?
>>
>> Not knowing the cause is somewhat uncomfortable. (I hate keeping fingers
>> crossed. Last time I dropped my coffee pot!)
> 
> Try running scp and/or dropbear on target under strace or better gdbserver.
> Does it also limit file size if you transfer file from target to host or only other way around?
> 
> Regards,
> Hinko
> 

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Peter Korsgaard]

>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:

Hi,

 Andreas> Fine hearing that it should work as expected.

 Andreas> About the memory capacity:
 Andreas> Mem: 8612K used, 118892K free, 0K shrd, 320K buff, 5168K cached
 Andreas> rootfs: 119.5M used, 56.0M free

 Andreas> I still expect something like that ext2-fs has some
 Andreas> limitation or uClibc

Ext2? Are you using an initrd? How much space do you have free?

-- 
Bye, Peter Korsgaard

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Peter Korsgaard wrote:
>>>>>> "Andreas" == Andreas Kuehn <Andreas.Kuehn@gin.de> writes:
> 
> Hi,
> 
>  Andreas> Fine hearing that it should work as expected.
> 
>  Andreas> About the memory capacity:
>  Andreas> Mem: 8612K used, 118892K free, 0K shrd, 320K buff, 5168K cached
>  Andreas> rootfs: 119.5M used, 56.0M free
> 
>  Andreas> I still expect something like that ext2-fs has some
>  Andreas> limitation or uClibc
> 
> Ext2? Are you using an initrd? How much space do you have free?
> 
Good Morning Peter!

No, I do *not* use initrd.

The rootfs is an ext2 on a CompactFlash card (Industial Type).
I tried cards from different brands but without result. Moreover, Linux 
uses these cards through its IDE interface. I can copy 5MB files around 
the disk without trouble. To me, the storage system works fine.

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Hinko Kocevar]

Andreas Kuehn wrote:
> 
> First, scp (sftp) segfaults in both directions.
> 
>  - getting a file from the host to the box ends at 880KB
>  - putting a file from the box to the host ends at 992KB
> 
> 
>  1. Here is the debug output of "scp user at host:gimp.ps ."
> Right in the middle you see a "segmentation fault"
> 

I would suggest to run the segfaulting process under
gdb(server) and backtrace when segfault occurs. You'll
probably need to rebuild uClibc and busybox with
debugging symbols.

...
> 
> read(7, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
> write(3, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
> read(7, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
> write(3, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
> read(7, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
> write(3, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
> --- SIGALRM (Alarm clock) @ 0 (0) ---
> getpgrp()                               = 884
> ioctl(1, TIOCGPGRP, [884])              = 0
> time(NULL)                              = 6388
> gimp.ps                                         8%  324KB 324.0KB/s
> 00:01    ) = 80
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> +++ killed by SIGSEGV +++
> 

Segfault at 0, hmm, and in first second. Maybe time(NULL)
is the culprit here. Another thing - can you verify that
segfault occurs at the first second (eg. when time(NULL)
is first called), or does it happen on random elapsed time?

IIRC, latest buildroot uses uclibc-0.9.30, do early
versions work, try uClibc-0.9.29 perhaps?

HTH,
Hinko

-- 
Hinko Ko?evar, OSS developer
?ETRTA POT, d.o.o.
Planina 3, 4000 Kranj, SI EU
tel     ++386 (0) 4 280 66 03
e-mail  hinko.kocevar at cetrtapot.si
http    www.cetrtapot.si

[Buildroot] dropbear & openssh transfers are limited to about 800k !?

[Andreas Kuehn]

Hi there...

it took a while but I did a complete rebuild of my buildroot 
environment. And for sure, I tried both versions of the uClibc. Finally 
I did an strace on dropbear and got these news:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
   7676     0011853          72       164         1 read
   1256     0001940        1940         1           execve
   1068     0001649          14       118           write
    000     0000000           0         1           fork
    000     0000000           0        12         2 open
    000     0000000           0        13           close
    000     0000000           0         3           time
    000     0000000           0         1           alarm
    000     0000000           0         3           pipe
    000     0000000           0         4           brk
    000     0000000           0         8         2 ioctl
    000     0000000           0         2           umask
    000     0000000           0         2           getpgrp
    000     0000000           0         4           munmap
    000     0000000           0         1           stat
    000     0000000           0         7           fstat
    000     0000000           0         4           mprotect
    000     0000000           0         6           rt_sigaction
    000     0000000           0        19           mmap2
    000     0000000           0         2           stat64
    000     0000000           0         1           getuid32
------ ----------- ----------- --------- --------- ----------------
100.00     0015442                   376         5 total


To me it looks like uClibc calls are not to happy?!
To keep this posting as short as possible I added some *interesting* 
sniplets from a normal strace run. Does that give somebody a clue about 
the problem? Is that a configuration issue?

    ...

brk(0x3e000)                           = 0x3e000
getuid32()                              = 0
open("/etc/passwd", O_RDONLY)           = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbedec990) = -1 ENOTTY 
(Inappropriate ioctl for device)
read(3, "root:x:0:0:root:/root:/bin/sh\ndae"..., 4096) = 562
close(3)                                = 0
ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, {B115200 opost isig icanon echo 
...}) = 0
rt_sigaction(SIGPIPE, {0x16c9c, [PIPE], SA_RESTART|0x4000000}, {SIG_DFL, 
[], 0}, 8) = 0
pipe([3, 4])                            = 0
pipe([5, 6])                            = 0

   ... and finally

write(3, 
"W\260\23\341\246\330\v\tU\34\202\302b at JH\275CY`ywt\264\r\213D\264\331\31\267}m"..., 
4096) = 4096
read(7, 
"\3V\226\226\340\337R\261\270\214\317\253+\370,\336,\225\227K\213\177)\225J\345\322\352\322\352R\251\370"..1
read(7, 0x3d1af, 25)                    = ? ERESTARTSYSTo be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---
getpgrp()                               = 826
ioctl(1, TIOCGPGRP, [826])              = 0
time(NULL)                              = 2444
write(1, "\rbigPayLoad                      "..., 80) = 80
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++





Hinko Kocevar wrote:
> Andreas Kuehn wrote:
>> First, scp (sftp) segfaults in both directions.
>>
>>  - getting a file from the host to the box ends at 880KB
>>  - putting a file from the box to the host ends at 992KB
>>
>>
>>  1. Here is the debug output of "scp user at host:gimp.ps ."
>> Right in the middle you see a "segmentation fault"
>>
> 
> I would suggest to run the segfaulting process under
> gdb(server) and backtrace when segfault occurs. You'll
> probably need to rebuild uClibc and busybox with
> debugging symbols.
> 
> ...
>> read(7, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
>> write(3, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096
>> read(7, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
>> write(3, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096
>> read(7, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
>> write(3, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096
>> --- SIGALRM (Alarm clock) @ 0 (0) ---
>> getpgrp()                               = 884
>> ioctl(1, TIOCGPGRP, [884])              = 0
>> time(NULL)                              = 6388
>> gimp.ps                                         8%  324KB 324.0KB/s
>> 00:01    ) = 80
>> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
>> +++ killed by SIGSEGV +++
>>
> 
> Segfault at 0, hmm, and in first second. Maybe time(NULL)
> is the culprit here. Another thing - can you verify that
> segfault occurs at the first second (eg. when time(NULL)
> is first called), or does it happen on random elapsed time?
> 
> IIRC, latest buildroot uses uclibc-0.9.30, do early
> versions work, try uClibc-0.9.29 perhaps?
> 
> HTH,
> Hinko
> 


-- 
  aKuehn

--------------> Never trust a computer you can't lift. -- Stan Masor