st.c block limits

st.c block limits

[ael]

I have a scsi tape drive:-

Host: scsi0 Channel: 00 Id: 00 Lun: 00
     Vendor: AIWA     Model: TD-20001         Rev: 0159
     Type:   Sequential-Access                ANSI SCSI revision: 02

It has worked perfectly for several years, but from kernel 2.4.10 st.c
started to report a 24-bit block limit: st0: Block limits 1 - 16777215 
bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".

The above drive is a NS-20 drive with default block size of 512 and 20G 
blocks: so a range of 25 bits for the block number is needed.

So when I attempt to write to the tape beyond block 2^24,
I get an error like:

# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
tar: /dev/ntape: Wrote only 0 of 10240 bytes
tar: Error is not recoverable: exiting now
At block 17832312.

With:
    kernel: st0: Block limits 1 - 16777215 bytes.
in the kernel log.


Now I *think* that I used to write to the end of tapes without hitting this
2^24 block limit. I guess that I can work around by setting a larger block 
size. But is there a fix? Or is the 24-bit limit built into the scsi tape 
protocol? Since I used to fill 2 tapes to complete a backup, I must have 
written beyond block (2^24 -1).

Sorry if this is an FAQ. I did google and found nothing directly relevant.
When did st.c change and why? I did turn on DEBUG, and I append a few 
entries, although I am not sure that it helps much:

Jul 30 16:49:28 conquest3 kernel: st: Version 20020205, bufsize 32768, wrt
30720, max init. bufs 4, s/g segs 16
Jul 30 16:49:28 conquest3 kernel: Attached scsi tape st0 at scsi0, channel
0, id 0, lun 0
Jul 30 16:49:28 conquest3 kernel: st: Allocated tape buffer 0 (32768 bytes,
1 segments, dma: 1, a: c01e8000).
Jul 30 16:49:28 conquest3 kernel: st: segment sizes: first 32768, last
32768 bytes.
drwx------    2 ael      ael          4096 Nov  3  2001 Mail
Jul 30 16:49:28 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 16:49:28 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 16:49:28 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 16:49:28 conquest3 kernel: st0: Block size: 512, buffer size: 32768
(64 blocks).

After switching to 1K blocks and fsf'ing past existing data:-
--------------------------------------------------------------


Jul 30 17:28:50 conquest3 kernel: st0: Setting block size to 1024 bytes.
Jul 30 17:28:50 conquest3 kernel: st0: Rewinding tape.
Jul 30 17:28:52 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:28:52 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:28:52 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:28:52 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:28:52 conquest3 kernel: st0: Rewinding tape.
Jul 30 17:29:36 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:29:36 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:29:36 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:29:36 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:29:36 conquest3 kernel: st0: Spacing tape forward over 12 filemarks.
Jul 30 17:32:28 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:32:28 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:32:28 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:32:28 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:32:28 conquest3 kernel: st0: Got tape pos. blk 17832312 part 0.

Thanks in advance for any help

ael

Re: st.c block limits

[Kai Makisara]

On Tue, 30 Jul 2002, ael wrote:

> I have a scsi tape drive:-
>
> Host: scsi0 Channel: 00 Id: 00 Lun: 00
>      Vendor: AIWA     Model: TD-20001         Rev: 0159
>      Type:   Sequential-Access                ANSI SCSI revision: 02
>
> It has worked perfectly for several years, but from kernel 2.4.10 st.c
> started to report a 24-bit block limit: st0: Block limits 1 - 16777215
> bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".
>
> The above drive is a NS-20 drive with default block size of 512 and 20G
> blocks: so a range of 25 bits for the block number is needed.
>
The block limits tell the block size limits the drive reports, not limits
to the amount of data. The driver writes data until the drive tells that
there is no space. I.e., the driver does not limit the amount of data
written to the tape. No changes affecting this behaviour has been made for
a very long time.

> So when I attempt to write to the tape beyond block 2^24,
> I get an error like:
>
> # tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
> tar: /dev/ntape: Wrote only 0 of 10240 bytes
> tar: Error is not recoverable: exiting now
> At block 17832312.
>
Looks like you are at end of tape. I think the 20 GB capacity is
compressed and not real. This means that the actual amount of data you can
write on the tape depends on the compressibility of data. 17.8 GB is
actually not too bad in this case.

	Kai

Re: st.c block limits

[ael]

Kai Makisara wrote:
> On Tue, 30 Jul 2002, ael wrote:
> 
> 
>>I have a scsi tape drive:-
>>
>>Host: scsi0 Channel: 00 Id: 00 Lun: 00
>>     Vendor: AIWA     Model: TD-20001         Rev: 0159
>>     Type:   Sequential-Access                ANSI SCSI revision: 02
>>
>>It has worked perfectly for several years, but from kernel 2.4.10 st.c
>>started to report a 24-bit block limit: st0: Block limits 1 - 16777215
>>bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".
>>
>>The above drive is a NS-20 drive with default block size of 512 and 20G
>>blocks: so a range of 25 bits for the block number is needed.
>>
> 
> The block limits tell the block size limits the drive reports, not limits
> to the amount of data. The driver writes data until the drive tells that
> there is no space. I.e., the driver does not limit the amount of data
> written to the tape. No changes affecting this behaviour has been made for
> a very long time.
> 
> 
>>So when I attempt to write to the tape beyond block 2^24,
>>I get an error like:
>>
>># tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
>>tar: /dev/ntape: Wrote only 0 of 10240 bytes
>>tar: Error is not recoverable: exiting now
>>At block 17832312.
>>
> 
> Looks like you are at end of tape. I think the 20 GB capacity is
> compressed and not real. This means that the actual amount of data you can
> write on the tape depends on the compressibility of data. 17.8 GB is
> actually not too bad in this case.

Thanks for the reply.

The tape has a real 10GB capacity. (I believe.)  The 20BG blocks is correct 
for the default block size of 512. So compression is not an issue here. And 
the report above is actually 17.8/2 = 8.9 GB :-(

I did wonder whether some bug in decoding what the drive reports had crept 
in. I will see if I have kept any logs old enough to see what used to 
happen...  Or maybe I have some substandard tapes. It looks as if I had 
better do some more checks.

Thanks anyway,

ael

RE: st.c block limits

[Tony Battersby]

> >># tar -cf /dev/ntape blah_home.tar.bz2; mt  tell

> The tape has a real 10GB capacity. (I believe.)  The 20BG
> blocks is correct
> for the default block size of 512. So compression is not an
> issue here. And
> the report above is actually 17.8/2 = 8.9 GB :-(

Some compression algorithms will actually expand data by a significant
amount.  For example, many implementations of ALDC compression will give a
compression ratio of 0.8 on random data.  In your example above you are
writing a .bz2 file (which is already compressed), so it would probably
expand if compressed again.  Try turning drive internal compression off:

mt -f /dev/ntape compression off

This might help squeeze a little more data on your tape.

Anthony Battersby
Cybernetics

Re: st.c block limits

[Rogier Wolff]

Tony Battersby wrote:
[Charset iso-8859-1 unsupported, filtering to ASCII...]
> > >># tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
> 
> > The tape has a real 10GB capacity. (I believe.)  The 20BG
> > blocks is correct
> > for the default block size of 512. So compression is not an
> > issue here. And
> > the report above is actually 17.8/2 = 8.9 GB :-(
> 
> Some compression algorithms will actually expand data by a significant
> amount.  For example, many implementations of ALDC compression will give a
> compression ratio of 0.8 on random data.  In your example above you are
> writing a .bz2 file (which is already compressed), so it would probably
> expand if compressed again.  Try turning drive internal compression off:

It's easy to prevent this from costing too much: Simply prepend a
block of data with a "1" compressed or "0" uncompressed bit. 

Max expansion is then 1 bit per block. (At a cost of one extra bit per
block).

This technique was maybe unacceptable in the old days when you
couldn't afford to buffe a WHOLE block in RAM while you were
compressing to see if it would expand...... 

Nowadays you can easily buffer the data, and still get acceptable
speed, at an acceptable cost in the form of a little higher RAM usage. 

			Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots. 

Re: st.c block limits

[ael]

Tony Battersby wrote:
>>>># tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
>>>
> 
>>The tape has a real 10GB capacity. (I believe.)  The 20BG
>>blocks is correct
>>for the default block size of 512. So compression is not an
>>issue here. And
>>the report above is actually 17.8/2 = 8.9 GB :-(
> 
> 
> Some compression algorithms will actually expand data by a significant
> amount.  For example, many implementations of ALDC compression will give a
> compression ratio of 0.8 on random data.  In your example above you are
> writing a .bz2 file (which is already compressed), so it would probably
> expand if compressed again.  Try turning drive internal compression off:
> 
> mt -f /dev/ntape compression off
> 
> This might help squeeze a little more data on your tape.

Thanks for the suggestion. Various such thoughts had crossed what passes 
for my mind, including whether 10GB was before formatting. But I did assume 
that the tape drive/driver was reporting actual bytes on the tape 
regardless of compression.

I was indeed recording largely bz2 files, and I did realize that the 
default drive streaming compression would almost certainly expand again. I 
sometimes do remember to turn the drive compression off in such 
circumstances. But as I say, I assume that the actual data on the tape 
after compression was being reported. That has always seemed to be the case 
in the past, although I am not sure whether I checked very thoroughly.

I guess that I should browse the Imation web site and see exactly they 
specify for the tape itself. The packaging is clearly marked 10GB 
uncompressed. The drive has a read-while-write-and-rerecord-on-error 
circuit, but the tapes have not had much use, and I don't think that the
data is being expanded by repeated blocks. I could be wrong, of course.
And in any case, I would expect the drive to report the total blaoc count 
including repeats.

ael

Re: st.c block limits

[ael]

Kai Makisara wrote:
> On Tue, 30 Jul 2002, ael wrote:
> 
> 
>>I have a scsi tape drive:-
>>
>>Host: scsi0 Channel: 00 Id: 00 Lun: 00
>>     Vendor: AIWA     Model: TD-20001         Rev: 0159
>>     Type:   Sequential-Access                ANSI SCSI revision: 02
>>
>>It has worked perfectly for several years, but from kernel 2.4.10 st.c
>>started to report a 24-bit block limit: st0: Block limits 1 - 16777215
>>bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".
>>
>>The above drive is a NS-20 drive with default block size of 512 and 20G
>>blocks: so a range of 25 bits for the block number is needed.
>>
> 
> The block limits tell the block size limits the drive reports, not limits
> to the amount of data. The driver writes data until the drive tells that
> there is no space. I.e., the driver does not limit the amount of data
> written to the tape. No changes affecting this behaviour has been made for
> a very long time.
> 
> 
>>So when I attempt to write to the tape beyond block 2^24,
>>I get an error like:
>>
>># tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
>>tar: /dev/ntape: Wrote only 0 of 10240 bytes
>>tar: Error is not recoverable: exiting now
>>At block 17832312.
>>
> 
> Looks like you are at end of tape. I think the 20 GB capacity is
> compressed and not real. This means that the actual amount of data you can
> write on the tape depends on the compressibility of data. 17.8 GB is
> actually not too bad in this case.
> 
> 	Kai

Following up my previous reply where I noted that I had only written 8.8Gb 
to a tape with uncompressed capacity of 10Gb, it then seems that the tape 
firmware is reporting a maximum drive capacity of ~ 16GB.

And perhaps a real tape limit of 17832312 /2 = 8.9GB which is supposedly on 
a 10GB tape : Imation NS-20, specified as 10GB uncompressed. Should I ask 
Imation for replacements :-) ?



ael

Re: st.c block limits

[Bryan Henderson]

>It's easy to prevent this from costing too much: Simply prepend a
>block of data with a "1" compressed or "0" uncompressed bit.
>
>Max expansion is then 1 bit per block. (At a cost of one extra bit per
>block).

Could you elaborate on this?  What does "'1' compressed or '0' uncompressed
bit" mean?  Where exactly do you do this prepending?  Is this according to
some protocol that has a name?

Re: st.c block limits

[Bryan Henderson]

>I mean that if the compression doesn't have "blocks" (like a gzip or
>compress algorithm), then you chose a more or less arbitrary block size.
>Say 1024 bytes.
>
>Then when compression of a block of data leads to an expansion, you
>just store a "0" bit, followed by the block. If the block size is 1024
>bytes, then this costs 1024.125 bytes of storage (8193 bits). If the
>compression /is/ succesful, then you just store a "1" bit followed by
>the compressed data (less than 1024 bytes....).

Oh, so you're talking about a way the tape drive could be engineered so we
wouldn't have to worry about expansion.  It sounded like you might be
talking about something a user of an existing tape drive could do to avoid
the problem.


BTW, one other reason that you wouldn't be able to fit 10GB on a nominally
10GB tape is defects in the medium, particularly those that arose after
manufacture.  The drive will automatically skip bad parts of the tape.