mount: Function not implemented

mount: Function not implemented

[KE Liew]

[-- Attachment #1.1: Type: text/plain, Size: 2496 bytes --]

Hi all,

I was in #xfs with sandeen in discussing on the above issue. In the
nutshell, I can't mount /dev/hdb due to the above error: mount -t xfs
/dev/hdb /path/to

It all started after I did a sequence of things to the new hdd I purchased.
First, I created an msdos disklabel on a new xfs partition /dev/hdb1. It
uses the full capacity of the hdd. Then I transferred files from one hdd to
another using mv -v /stuff-to-transfer /hdb-mount-point Upon completion, I
umount both devices and rebooted. On boot, I was not able to mount neither
/dev/hdb1 nor /dev/hdb. /dev/hdb1 returns mount: special device /dev/hdb1
does not exist and /dev/hdb returns the above error message, no relevant
messages are present in dmesg | tail

The hexdump:
===================
# dd if=/dev/hdb bs=512 count=1 | hexdump -C
1+0 records in
1+0 records out
00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0
|XFSB.........:8.|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
|................|
00000020  82 fd 56 0c 80 92 4f 4a  87 8b 5e 5d 5e 55 89 aa
|..V...OJ..^]^U..|
00000030  00 00 00 00 00 20 00 05  00 00 00 00 00 00 09 00  |.....
..........|
00000040  00 00 00 00 00 00 09 01  00 00 00 00 00 00 09 02
|................|
00000050  00 00 00 01 00 03 a3 8b  00 00 00 10 00 00 00 00
|................|
00000060  00 00 07 47 3c 04 80 00  01 00 01 00 00 00 00 00
|...G<...........|
00000070  00 00 00 00 00 00 00 00  10 0f 08 08 12 00 00 19
|................|
00000080  00 00 00 00 00 00 01 00  00 00 00 00 00 00 00 fd
|................|
00000090  00 00 00 00 00 3a 31 18  00 00 00 00 00 00 00 00
|.....:1.........|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
|................|
*
000000c0  00 0f 80 00 00 01 00 00  00 00 00 00 00 00 00 00
|................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
|................|
*
00000200
512 bytes (512 B) copied, 0.00695137 seconds, 73.7 kB/s
===================

The xfs_db:
===================
# xfs_db -r -c "sb 0" -c p /dev/hdb
cache_node_purge: refcount was 1, not zero (node=0x80ca410)
xfs_db: cannot read root inode (22)
cache_node_purge: refcount was 1, not zero (node=0x80da608)
xfs_db: cannot read realtime bitmap inode (22)
Segmentation fault
===================

Distro: Debian etch
uname -a : 2.6.18-5-686
xfsprogs version 2.9.4-2 (from Sid)

You will find the chatlog attached. I just hope that it's recoverable even
though I have backups that aren't exactly up to date.

Thanks!


Best regards,

KwangErn

[-- Attachment #1.2: Type: text/html, Size: 2893 bytes --]

[-- Attachment #2: XFS-SGI.txt --]
[-- Type: text/plain, Size: 8196 bytes --]

<ktwilight> i've got an XFS partition, it doesn't mount 'cuz the superblock can't be found, which was weird. i tried xfs_repair but it didn't give me any good results. i tried with xfs_repair -o assume_xfs /dev/sda and it still doesn't seem to work. so how can i mount this partition when it can't find the superblock? i know there are contents 'cuz i transfered files to it and umount it before rebooting.
<sandeen> what happened between working and not working?
<ktwilight> sandeen, i transfered files to it, umount it, then reboot
<sandeen> ktwilight, how big is the partition
<sandeen> ktwilight, what kernel, which partition, what type of block device, etc
<ktwilight> 2.6.18-5, it's the entire hdd, /dev/hdb
<ktwilight> what do you mean by block device?
<sandeen> that's enough
<sandeen> what does file -s /dev/sdb say
<sandeen> er
<sandeen> hdb :)
<ktwilight> /dev/hdb: SGI XFS filesystem data (blksz 65536, inosz 256, v2 dirs)
<ktwilight> it should have a /dev/hdb1
<sandeen> huh, ok, I thought maybe something overwrote it
<ktwilight> it's umount, and i try my best not to do anything to do :)
<sandeen> nothing wrong with using the whole bdev
<ktwilight> hm, but i can't mount either hdb or hdb1
<sandeen> which one *should* it be
<sandeen> what used to mount :)
<sandeen> i.e .what's in fstab
<ktwilight> nothing in fstab
<ktwilight> but i used hdb1 to mount previously
<sandeen> what does file -s /dev/hdb1 say then?
<sandeen> or fdisk -l /dev/sdb?
<ktwilight> /dev/hdb1: ERROR: cannot open `/dev/hdb1' (No such file or directory)
<ktwilight> Disk /dev/hdb: 250.0 GB, 250059350016 bytes
<ktwilight> 255 heads, 63 sectors/track, 30401 cylinders
<ktwilight> Units = cylinders of 16065 * 512 = 8225280 bytes
<ktwilight> Disk /dev/hdb doesn't contain a valid partition table
<ktwilight> the last sentence is quite... :/
<sandeen> er, so, you no longer have a partition table, but rather it is an xfs filesystem on the whole /dev/hdb... which is fine, but on the last boot you mounted hdb1?
<sandeen> I'm having a hard time believing that nothing happened except file copies and a reboot :)
<sandeen> looks to me like you mkfs'd /dev/hdb
<sandeen> can you mount /dev/hdb?
<ktwilight> can't be...
<ktwilight> nope, i can't mount it
<sandeen> what does dmesg say when you try?
<xorAxAx> ktwilight: hexdump -C the first 512 bytes for us
<xorAxAx> and use paste.pocoo.org
<sandeen> a) you have no partition table, thus no /dev/hdb1, and b) "file" finds an XFS superblock directly on hdb, so...
<ktwilight> dmesg doesn't say anything, but mounting gives me "mount: Function not implemented"
* sandeen thinks xfs should store mkfs time for cases like this :)
<xorAxAx> sandeen: hehe
<sandeen> lsmod | grep xfs ? :)
<ktwilight> um, so hexdump -C /dev/hdb?
<xorAxAx> ktwilight: dd ... count=1
<sandeen> dd if=/dev/hdb bs=512 count=1 | hexdump -C
<ktwilight> xfs is loaded, i ave other xfs partitions mounted
<ktwilight> http://rafb.net/p/WLxJzd90.html
<sandeen> I have a hard time believing that mount /dev/hdb /some/where fails but generates no kernel messages
<sandeen> try mount -t xfs?
<ktwilight> http://rafb.net/p/ghKCgI36.html
<sandeen> xfs_db -r -c "sb 0" -c p /dev/sdb 
<sandeen> and mount -t xfs /dev/hdb /some/where; dmesg | tail -n 10
<ktwilight> http://rafb.net/p/GL95be48.html
<sandeen> sorry s/sdb/hdb/ :)
<ktwilight> np ;)
<sandeen> ew
<ktwilight> http://rafb.net/p/AcilLz30.html
<ktwilight> not a good ew i guess
<ktwilight> that segfault isn't nice
<sandeen> errr
<sandeen> XFS mounting filesystem sdb1
<sandeen> Ending clean XFS mount for filesystem: sdb1
<sandeen> oh sdb
<sandeen> geesh
<ktwilight> different ;)
<sandeen> you'd think I'd have this straight by now :)
<sandeen> *sigh*
<ktwilight> it's hdb, NOT sdb
<ktwilight> though i have an sdb and it's working perfectly.
<ktwilight> let's not touch it :)
<ktwilight> so dmesg doesn't say anything when trying to mount hdb
<sandeen> weird that xfs_db segfaults
<sandeen> is this latest xfsprogs?
<ktwilight> 2.8.11-1 
<ktwilight> debian etch.
<ktwilight> it's "stable" :)
<sandeen> might try very latest xfsprogs for fun
<ktwilight> ...
<sandeen> just to see if we can examine it more
<ktwilight> so i should grab it off sid?
<ktwilight> tell me what you know so far or its assumptions
<sandeen> or use xfs_metadump to get an fs image, and provide it to the sgi guys
<sandeen> at a minimum xfs_db should not segfault
<ktwilight> hm
<sandeen> well, so far, bare hdb has at least the beginning of an xfs filesystem signature, XFSB is supe block magic
<sandeen> and, you have no partition table on hdb
<ktwilight> so basically i have no files on hdb?
<ktwilight> no way of recovering?
<sandeen> not sure yet
<ktwilight> great :)
<sandeen> but, I don't quite believe that you had /dev/hdb1 mounted, copied files, umounted, and suddenly you had no more partition table
<sandeen> something else happened in between whether you did it yourself or something did it to you I think :)
<ktwilight> i don't believe it either, but that's the truth.
<ktwilight> well, rather than copy, it was a move. does that make any difference? :P
<sandeen> what kind of partition table did you have?
<sandeen> dos or gpt?
<ktwilight> dos
<sandeen> your root inode nr from the hexdump is a little funky
<ktwilight> bad funky?
<ktwilight> k, finally gettin' sid's xfsprogs
<ktwilight> great, xfs_db segfaults too :)
<ktwilight> xfsprogs 2.9.4-2
<sandeen> hmm
<sandeen> other same messages?
<sandeen> cache_node stuff or no
<ktwilight> not really
<ktwilight> http://rafb.net/p/e4uYNC58.html
<sandeen> same stuff
<ktwilight> o ok :|
<ktwilight> node is different. that helps?
<ktwilight> haha
<ktwilight> sandeen, would xfs_fsr help?
<sandeen> no
<sandeen> you have to mount it first :)
<ktwilight> :(
<ktwilight> how 'bout xfs_check or xfs_admin?
<sandeen> not that it'd help anyway
<ktwilight> hm
<ktwilight> so what are my options?
<sandeen> if xfs_db can't even recognize the filesystem things aren't so good
<ktwilight> would dd help? let's say i skip the first few bytes
<ktwilight> then mount it as an iso.
<hannesd> no backup?
<ktwilight> hannesd, yes, but not that up to date :/
<ktwilight> if i can recover, it'd be better.
<sandeen> just for fun, does dd if=/dev/hdb bs=512 count=32 | hexdump -C | grep XFSB turn up more than 1 XFSB?
* sandeen guesses at the count=32, but hdb1 probably started before 16k
<ktwilight> http://rafb.net/p/tHqWyb88.html
<ktwilight> i only see one.
<sandeen> nope
<ktwilight> hm, so it's not good.
<ktwilight> so should i just forget about it? or is there a glimpse of hope somewhere? :)
<sandeen> was gonna look over the first hexdump a bit
<ktwilight> ok
<sandeen> 00 00 00 00 00 00 09 00 should be your root inode, which is not 22, so a little confused about why it is ssaying rootino 22 when you try xfs_db
<sandeen> unless that's EINVAL that got lost
<sandeen> can you use xfs_metadump to make an fs image?   maybe the sgi guys can help further
<ktwilight> some monster ate it i see.
<ktwilight> how big would the fs image be? as big as the partition?
<sandeen> at least bnaujok would probably like to see what's going on with xfs_db
<sandeen> no it only dumps metadata no data
<sandeen> so it should be significantly smaller
<ktwilight> hm
<sandeen> but if db can't read it, metadump might not either
<ktwilight> ya gotta see this
<ktwilight> http://rafb.net/p/ojVANy73.html
<sandeen> yeah, libxfs problem, I was sorta afraid of that
<sandeen> well.  underlying problem is what's on your disk, but then libxfs isn't coping well either
<sandeen> maybe the first few K of the fs would be enough for them to dig in further
<ktwilight> hm
<sandeen> I should be working on another fs right now ;-)
<ktwilight> :)
<ktwilight> k, so i will send xfs_metadump, and xfs_db to sgi?
<ktwilight> what else should i include in the email?
<hannesd> the chat log maybe :)
<ktwilight> :)
<ktwilight> a heck load of stuff then
<ktwilight> email? support@sgi.com?
<sandeen> xfs@sgi.com
<sandeen> open source xfs list
<sandeen> hexdump of hdb would be a place to start, along w/ xfs_db results
<ktwilight> nm, found. xfs@oss.sgi.com
<sandeen> and a sworn statement that oh really this used to be hdb1 but it magically moved :)

Re: mount: Function not implemented

[David Chinner]

On Thu, Dec 13, 2007 at 04:46:18PM +0100, KE Liew wrote:
> Hi all,
> 
> I was in #xfs with sandeen in discussing on the above issue. In the
> nutshell, I can't mount /dev/hdb due to the above error: mount -t xfs
> /dev/hdb /path/to

#define ENOSYS          38      /* Function not implemented */

Exactly what is mount being told there is not system call support?
What kernel are you running?

Can you strace the mount process and find out where the error is
coming from?

> It all started after I did a sequence of things to the new hdd I purchased.
> First, I created an msdos disklabel on a new xfs partition /dev/hdb1. It
> uses the full capacity of the hdd.

Did you then write out the partition table? Did the kernel warn you that it
couldn't reread the partition table and so you should reboot before doing
anything else? Did you make the xfs filesystem on /dev/hdb or hdb1?

> Then I transferred files from one hdd to
> another using mv -v /stuff-to-transfer /hdb-mount-point Upon completion, I
> umount both devices and rebooted. On boot, I was not able to mount neither
> /dev/hdb1 nor /dev/hdb. /dev/hdb1 returns mount: special device /dev/hdb1
> does not exist and /dev/hdb returns the above error message, no relevant
> messages are present in dmesg | tail
> 
> The hexdump:
> ===================
> # dd if=/dev/hdb bs=512 count=1 | hexdump -C
> 1+0 records in
> 1+0 records out
> 00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0
> |XFSB.........:8.|

This kind of indicates that you did a 'mkfs.xfs /dev/hdb' not
the partition you created. If that is the case, then rebooting
may have written who-knows-what to disk.

> The xfs_db:
> ===================
> # xfs_db -r -c "sb 0" -c p /dev/hdb
> cache_node_purge: refcount was 1, not zero (node=0x80ca410)
> xfs_db: cannot read root inode (22)

EINVAL.

> cache_node_purge: refcount was 1, not zero (node=0x80da608)
> xfs_db: cannot read realtime bitmap inode (22)
> Segmentation fault
> ===================

That tends to indicate that the filesystem superblock is ok but
the contents are not.

Without knowing exactly what you did and what errors came up, it's
going to be hard reconstructing what went wrong. Perhaps a metadump
of the filesysetm woul dbe useful in working out how it is broken....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: mount: Function not implemented

[Eric Sandeen]

David Chinner wrote:
> On Thu, Dec 13, 2007 at 04:46:18PM +0100, KE Liew wrote:

>> The hexdump:
>> ===================
>> # dd if=/dev/hdb bs=512 count=1 | hexdump -C
>> 1+0 records in
>> 1+0 records out
>> 00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0
>> |XFSB.........:8.|
> 
> This kind of indicates that you did a 'mkfs.xfs /dev/hdb' not
> the partition you created. If that is the case, then rebooting
> may have written who-knows-what to disk.

although if it was a) new and b) he made a whole-disk partition, it was
probably not a busy disk and I wouldn't expect a reboot was necessary.

>> The xfs_db:
>> ===================
>> # xfs_db -r -c "sb 0" -c p /dev/hdb
>> cache_node_purge: refcount was 1, not zero (node=0x80ca410)
>> xfs_db: cannot read root inode (22)
> 
> EINVAL.

Hm, that should probably use perror or something, silly me read that as
the inode *number* and then thought "hmm I bet something stuck EINVAL
into the inode number, oops!"  :)

>> cache_node_purge: refcount was 1, not zero (node=0x80da608)
>> xfs_db: cannot read realtime bitmap inode (22)
>> Segmentation fault
>> ===================
> 
> That tends to indicate that the filesystem superblock is ok but
> the contents are not.
> 
> Without knowing exactly what you did and what errors came up, it's
> going to be hard reconstructing what went wrong. Perhaps a metadump
> of the filesysetm woul dbe useful in working out how it is broken....

Metadump failed with the same errors as xfs_db

-Eric

Re: mount: Function not implemented

[David Chinner]

On Thu, Dec 13, 2007 at 10:06:32PM -0600, Eric Sandeen wrote:
> David Chinner wrote:
> > On Thu, Dec 13, 2007 at 04:46:18PM +0100, KE Liew wrote:
> 
> >> The hexdump:
> >> ===================
> >> # dd if=/dev/hdb bs=512 count=1 | hexdump -C
> >> 1+0 records in
> >> 1+0 records out
> >> 00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0
> >> |XFSB.........:8.|
> > 
> > This kind of indicates that you did a 'mkfs.xfs /dev/hdb' not
> > the partition you created. If that is the case, then rebooting
> > may have written who-knows-what to disk.
> 
> although if it was a) new and b) he made a whole-disk partition, it was
> probably not a busy disk and I wouldn't expect a reboot was necessary.

Agreed. But something went wrong so best to ask....

> >> The xfs_db:
> >> ===================
> >> # xfs_db -r -c "sb 0" -c p /dev/hdb
> >> cache_node_purge: refcount was 1, not zero (node=0x80ca410)
> >> xfs_db: cannot read root inode (22)
> > 
> > EINVAL.
> 
> Hm, that should probably use perror or something, silly me read that as
> the inode *number* and then thought "hmm I bet something stuck EINVAL
> into the inode number, oops!"  :)

Yeah, it probably should perror then tell you the root inode number
it failed to read....

Looking at it a bit deeper, libxfs_iget() returned EINVAL and that
is most likely from:

        ip->i_ino = ino;
        ip->i_mount = mp;
        if ((error = xfs_itobp(mp, tp, ip, &dip, &bp, bno)))
                return error;
        if (INT_GET(dip->di_core.di_magic, ARCH_CONVERT) != XFS_DINODE_MAGIC) {
                xfs_trans_brelse(tp, bp);
>>>>>>          return EINVAL;
        }

The inode number being toast.

Looking back at superblock dump, the root inode number is 0x900,
which is extremely strange. Normal root inode number is 0x80 which
indicates a block number of 0x40 (512 byte blocks), whereas
0x900 indicates a block number of 0x480 (1152) which is a long
way away from where it should be.

00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0
|XFSB.........:8.|

Hmmmm. Filesystem block size = 0x10000 = 64k and data blocks = 0x3a38b0
which gives a filesystem size of 232GB.

<ktwilight> Disk /dev/hdb: 250.0 GB, 250059350016 bytes

Still the root inode is in the wrong place. They should
be at 0x800 for 64k block size. (yes, I just checked this out by
making a 64k block size filesystem).

So, key questions:

	1. What platform is this?
	2. What page size is being used?
	3. what kernel is being run?
	4. What was the mkfs command used to make the filesystem?
	5. how did you mount this filesystem in the first place?

> > Without knowing exactly what you did and what errors came up, it's
> > going to be hard reconstructing what went wrong. Perhaps a metadump
> > of the filesysetm woul dbe useful in working out how it is broken....
> 
> Metadump failed with the same errors as xfs_db

Oh, I didn't notice that. But seeing as the above error is in libxfs
it's not surprising that they both fail there....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: mount: Function not implemented

[KE Liew]

On Dec 14, 2007 6:28 AM, David Chinner <dgc@sgi.com> wrote:

> So, key questions:
>
>        1. What platform is this?


Debian Etch.


>
>        2. What page size is being used?


Pardon my lack of knowledge, but what do you mean by page size?


>
>        3. what kernel is being run?


Stock kernel 2.6.18-5-686


>
>        4. What was the mkfs command used to make the filesystem?


from .bash_history - mkfs.xfs -f /dev/hdb1


>
>        5. how did you mount this filesystem in the first place?


As mentioned, the usual mount -t xfs /dev/hdb1 /path/to Tried both /dev/hdb
and /dev/hdb1, where hdb1 was the first I tried.


KwangErn


[[HTML alternate version deleted]]

Re: mount: Function not implemented

[KE Liew]

Just for document purposes...

sandeen mentioned to do,

dd if=/dev/hdb bs=512 count=128  | hexdump -C | grep XFSB

00000000  58 46 53 42 00 01 00 00  00 00 00 00 00 3a 38 b0  |XFSB.........:8.|
00007e00  58 46 53 42 00 00 10 00  00 00 00 00 03 a3 88 a0  |XFSB............|
128+0 records in
128+0 records out
65536 bytes (66 kB) copied, 0.0283236 seconds, 2.3 MB/s


So recreating the partition should work. First we backup the sectors.

dd if=/dev/hdb bs=512 count=256 of=backup_sectors

Then the partition is recreated using fdisk /dev/hdb, option n -> p (primary
-> 1 for partition 1 -> default first cylinder 1 -> default last cylinder
30401 -> (w)rite table to disk and exit. :)

Running xfs_check /dev/hdb1 returns no errors, which was hoped. Mounting the
partition shows ALL files present! So everything is working back to normal.

Thanks to XFS team and sandeen! :)


KwangErn


[[HTML alternate version deleted]]