ext4 unknown problem

ext4 unknown problem

[LuVar]

Hi,
I have seriouse issue here. I have copyied about 7GB files to one of my ext4 partitions and from that point of time there is sustained write to that partition from jbd2/dm-3-8 process. I have lunched "sudo sync" command (about day ago :) and a few hours a after sync start, I have tryied to use umount on that partition... Booth of them are still running...

I have no serious data there, but if I could be helpfull in bugreport, give me some info what to do.

# uname -a
Linux blacktroja 3.2.12-gentoo #1 SMP Wed Jun 27 15:09:02 CEST 2012 x86_64 Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz GenuineIntel GNU/Linux

# df -h /var/lib/mpd/music/local
Súborový systém       Veľk  Použ  Dost Pou% Pripojený na
/dev/mapper/vg-music   24G   17G  5,8G  75% /var/lib/mpd/music/local

I am ready to do anything with current state of that partition to provide valuable informations for development.

PS: that partition is on top of lvm which is on top of sw raid (mdraid) level 5.

Bye,
--
LuVar
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4 unknown problem

[Theodore Ts'o]

On Mon, Jul 30, 2012 at 08:26:20PM +0100, LuVar wrote:
> Hi,
> I have seriouse issue here. I have copyied about 7GB files to one of my ext4 partitions and from that point of time there is sustained write to that partition from jbd2/dm-3-8 process. I have lunched "sudo sync" command (about day ago :) and a few hours a after sync start, I have tryied to use umount on that partition... Booth of them are still running...
> 
> I have no serious data there, but if I could be helpfull in bugreport, give me some info what to do.

Try running the dmesg command, and see if there's anything interesting
there.  If you're not sure if it's interesting, send it to us and we
can take a quick look at it.

Regards,

					- Ted

Re: ext4 unknown problem

[luvar]

Hi,
dmesg is probably useles:

Probably some logs after mounting...:

# dmesg | grep EXT
EXT3-fs (md2): mounted filesystem with ordered data mode
EXT3-fs (md2): using internal journal
EXT3-fs (dm-2): using internal journal
EXT3-fs (dm-2): mounted filesystem with ordered data mode
EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
EXT4-fs (dm-12): mounted filesystem without journal. Opts: (null)
EXT4-fs (dm-8): re-mounted. Opts: commit=0
EXT4-fs (dm-3): re-mounted. Opts: commit=0
EXT4-fs (dm-12): re-mounted. Opts: commit=0
EXT4-fs (dm-28): mounted filesystem with ordered data mode. Opts: (null)
EXT3-fs (dm-10): using internal journal
EXT3-fs (dm-10): mounted filesystem with ordered data mode

On last few lines, there is some info about usb key used on thic pc:

# dmesg | tail -n 30
sd 14:0:0:0: [sdg] Asking for cache data failed
sd 14:0:0:0: [sdg] Assuming drive cache: write through
 sdg: sdg1
sd 14:0:0:0: [sdg] Asking for cache data failed
sd 14:0:0:0: [sdg] Assuming drive cache: write through
sd 14:0:0:0: [sdg] Attached SCSI disk
mdadm: sending ioctl 1261 to a partition!
mdadm: sending ioctl 1261 to a partition!
usb 2-4.4: USB disconnect, device number 10
hda-intel: IRQ timing workaround is activated for card #0. Suggest a bigger bdl_pos_adj.
usb 1-1: new high-speed USB device number 2 using ehci_hcd
usb 1-1: New USB device found, idVendor=0951, idProduct=1687
usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1: Product: DT R400
usb 1-1: Manufacturer: Kingston
usb 1-1: SerialNumber: 0018F30C7FX10B000XX029
scsi15 : usb-storage 1-1:1.0
scsi 15:0:0:0: Direct-Access     Kingston DT R400          PMAP PQ: 0 ANSI: 0 CCS
sd 15:0:0:0: Attached scsi generic sg6 type 0
sd 15:0:0:0: [sdg] 7774208 512-byte logical blocks: (3.98 GB/3.70 GiB)
sd 15:0:0:0: [sdg] Write Protect is off
sd 15:0:0:0: [sdg] Mode Sense: 23 00 00 00
sd 15:0:0:0: [sdg] No Caching mode page present
sd 15:0:0:0: [sdg] Assuming drive cache: write through
sd 15:0:0:0: [sdg] No Caching mode page present
sd 15:0:0:0: [sdg] Assuming drive cache: write through
 sdg: sdg1 sdg2
sd 15:0:0:0: [sdg] No Caching mode page present
sd 15:0:0:0: [sdg] Assuming drive cache: write through
sd 15:0:0:0: [sdg] Attached SCSI removable disk

Other lines from dmesg seems not relevant. Should I post them too?

PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it still on top of I/O activity).

LuVar

----- "Theodore Ts'o" <tytso@mit.edu> wrote:

> On Mon, Jul 30, 2012 at 08:26:20PM +0100, LuVar wrote:
> > Hi,
> > I have seriouse issue here. I have copyied about 7GB files to one of
> my ext4 partitions and from that point of time there is sustained
> write to that partition from jbd2/dm-3-8 process. I have lunched "sudo
> sync" command (about day ago :) and a few hours a after sync start, I
> have tryied to use umount on that partition... Booth of them are still
> running...
> > 
> > I have no serious data there, but if I could be helpfull in
> bugreport, give me some info what to do.
> 
> Try running the dmesg command, and see if there's anything
> interesting
> there.  If you're not sure if it's interesting, send it to us and we
> can take a quick look at it.
> 
> Regards,
> 
> 					- Ted

Re: ext4 unknown problem

[Ľubomír Varga]

I have used strace on described process (sync and umount):

This is "sudo sync" proces:

$ sudo strace -p 24933
Process 24933 attached
select(6, [3 5], [], NULL, NULL

This one is umount:

$ sudo strace -p 12103
Process 12103 attached

By for now.

----- luvar@plaintext.sk wrote:

> Hi,
> dmesg is probably useles:
> 
> Probably some logs after mounting...:
> 
> # dmesg | grep EXT
> EXT3-fs (md2): mounted filesystem with ordered data mode
> EXT3-fs (md2): using internal journal
> EXT3-fs (dm-2): using internal journal
> EXT3-fs (dm-2): mounted filesystem with ordered data mode
> EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts:
> (null)
> EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
> (null)
> EXT4-fs (dm-12): mounted filesystem without journal. Opts: (null)
> EXT4-fs (dm-8): re-mounted. Opts: commit=0
> EXT4-fs (dm-3): re-mounted. Opts: commit=0
> EXT4-fs (dm-12): re-mounted. Opts: commit=0
> EXT4-fs (dm-28): mounted filesystem with ordered data mode. Opts:
> (null)
> EXT3-fs (dm-10): using internal journal
> EXT3-fs (dm-10): mounted filesystem with ordered data mode
> 
> On last few lines, there is some info about usb key used on thic pc:
> 
> # dmesg | tail -n 30
> sd 14:0:0:0: [sdg] Asking for cache data failed
> sd 14:0:0:0: [sdg] Assuming drive cache: write through
>  sdg: sdg1
> sd 14:0:0:0: [sdg] Asking for cache data failed
> sd 14:0:0:0: [sdg] Assuming drive cache: write through
> sd 14:0:0:0: [sdg] Attached SCSI disk
> mdadm: sending ioctl 1261 to a partition!
> mdadm: sending ioctl 1261 to a partition!
> usb 2-4.4: USB disconnect, device number 10
> hda-intel: IRQ timing workaround is activated for card #0. Suggest a
> bigger bdl_pos_adj.
> usb 1-1: new high-speed USB device number 2 using ehci_hcd
> usb 1-1: New USB device found, idVendor=0951, idProduct=1687
> usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> usb 1-1: Product: DT R400
> usb 1-1: Manufacturer: Kingston
> usb 1-1: SerialNumber: 0018F30C7FX10B000XX029
> scsi15 : usb-storage 1-1:1.0
> scsi 15:0:0:0: Direct-Access     Kingston DT R400          PMAP PQ: 0
> ANSI: 0 CCS
> sd 15:0:0:0: Attached scsi generic sg6 type 0
> sd 15:0:0:0: [sdg] 7774208 512-byte logical blocks: (3.98 GB/3.70
> GiB)
> sd 15:0:0:0: [sdg] Write Protect is off
> sd 15:0:0:0: [sdg] Mode Sense: 23 00 00 00
> sd 15:0:0:0: [sdg] No Caching mode page present
> sd 15:0:0:0: [sdg] Assuming drive cache: write through
> sd 15:0:0:0: [sdg] No Caching mode page present
> sd 15:0:0:0: [sdg] Assuming drive cache: write through
>  sdg: sdg1 sdg2
> sd 15:0:0:0: [sdg] No Caching mode page present
> sd 15:0:0:0: [sdg] Assuming drive cache: write through
> sd 15:0:0:0: [sdg] Attached SCSI removable disk
> 
> Other lines from dmesg seems not relevant. Should I post them too?
> 
> PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it
> still on top of I/O activity).
> 
> LuVar
> 
> ----- "Theodore Ts'o" <tytso@mit.edu> wrote:
> 
> > On Mon, Jul 30, 2012 at 08:26:20PM +0100, LuVar wrote:
> > > Hi,
> > > I have seriouse issue here. I have copyied about 7GB files to one
> of
> > my ext4 partitions and from that point of time there is sustained
> > write to that partition from jbd2/dm-3-8 process. I have lunched
> "sudo
> > sync" command (about day ago :) and a few hours a after sync start,
> I
> > have tryied to use umount on that partition... Booth of them are
> still
> > running...
> > > 
> > > I have no serious data there, but if I could be helpfull in
> > bugreport, give me some info what to do.
> > 
> > Try running the dmesg command, and see if there's anything
> > interesting
> > there.  If you're not sure if it's interesting, send it to us and
> we
> > can take a quick look at it.
> > 
> > Regards,
> > 
> > 					- Ted

-- 
Ing. Ľubomír Varga
+421 (0)908 541 700
varga@plaintext.sk
www.plaintext.sk
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4 unknown problem

[Theodore Ts'o]

On Tue, Jul 31, 2012 at 08:18:42AM +0100, luvar@plaintext.sk wrote:
> usb 2-4.4: USB disconnect, device number 10

So what's this?  Is this at all related to how you've created your
RAID device?  You haven't told us what your RAID device on dm-3 is
composed of (i.e., what are the underlying devices).

And there were no OOPS or BUG or WARN messages in your dmesg?

> PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it still on top of I/O activity).

I'm not sure if your kernel has trace debugging enabled, or if you
have debugfs mounted, but try this:

echo 1 > /sys/kernel/debug/tracing/events/jbd2/jbd2_run_stats/enable
cat /sys/kernel/debug/tracing/trace_pipe

If these commands don't work try running this command and see if it
works then:

mount -t debugfs debugfs /sys/kernel/debug

Let us know if you see jbd2 activity.  It's possible that sync isn't
returning because some process is still writing to the file system,
and if I recall correctly, Linux 3.2 didn't have some of the sync
livelock fixes that could cause sync to not return until the file
system had actually quieted down completely.

Regards,

					- Ted

Re: ext4 unknown problem

[Eric Sandeen]

On 7/31/12 2:18 AM, luvar@plaintext.sk wrote:
> Hi,
> dmesg is probably useles:
> 
> Probably some logs after mounting...:
> 
> # dmesg | grep EXT
> EXT3-fs (md2): mounted filesystem with ordered data mode
> EXT3-fs (md2): using internal journal
> EXT3-fs (dm-2): using internal journal
> EXT3-fs (dm-2): mounted filesystem with ordered data mode
> EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null)
> EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
> EXT4-fs (dm-12): mounted filesystem without journal. Opts: (null)
> EXT4-fs (dm-8): re-mounted. Opts: commit=0
> EXT4-fs (dm-3): re-mounted. Opts: commit=0
> EXT4-fs (dm-12): re-mounted. Opts: commit=0

^^^ um, what's that all about?

In theory, according to the docs, that's the same as the default, I guess.

and:

                } else if (token == Opt_commit) {
                        if (arg == 0)
                                arg = JBD2_DEFAULT_MAX_COMMIT_AGE;

ok, I guess it is.

> EXT4-fs (dm-28): mounted filesystem with ordered data mode. Opts: (null)
> EXT3-fs (dm-10): using internal journal
> EXT3-fs (dm-10): mounted filesystem with ordered data mode
> 
> On last few lines, there is some info about usb key used on thic pc:
> 
> # dmesg | tail -n 30
> sd 14:0:0:0: [sdg] Asking for cache data failed
> sd 14:0:0:0: [sdg] Assuming drive cache: write through
...

> Other lines from dmesg seems not relevant. Should I post them too?

Usually best to post all the data and let us sort it out, yes.

-Eric

> PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it still on top of I/O activity).
> 
> LuVar
> 
> ----- "Theodore Ts'o" <tytso@mit.edu> wrote:
> 
>> On Mon, Jul 30, 2012 at 08:26:20PM +0100, LuVar wrote:
>>> Hi,
>>> I have seriouse issue here. I have copyied about 7GB files to one of
>> my ext4 partitions and from that point of time there is sustained
>> write to that partition from jbd2/dm-3-8 process. I have lunched "sudo
>> sync" command (about day ago :) and a few hours a after sync start, I
>> have tryied to use umount on that partition... Booth of them are still
>> running...
>>>
>>> I have no serious data there, but if I could be helpfull in
>> bugreport, give me some info what to do.
>>
>> Try running the dmesg command, and see if there's anything
>> interesting
>> there.  If you're not sure if it's interesting, send it to us and we
>> can take a quick look at it.
>>
>> Regards,
>>
>> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: ext4 unknown problem

[LuVar]

Hi,

----- "Theodore Ts'o" <tytso@mit.edu> wrote:

> On Tue, Jul 31, 2012 at 08:18:42AM +0100, luvar@plaintext.sk wrote:
> > usb 2-4.4: USB disconnect, device number 10
> 
> So what's this?  Is this at all related to how you've created your
> RAID device?  You haven't told us what your RAID device on dm-3 is
> composed of (i.e., what are the underlying devices).
> 

My raid is on top of five physical sata disc. That usb messages are related to usb keys which I have used on that PC.

> And there were no OOPS or BUG or WARN messages in your dmesg?
> 

My full dmesg is here:
http://dawn.ynet.sk/~luvar/asdf/20120801_1024_dmesg

> > PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it
> still on top of I/O activity).
> 
> I'm not sure if your kernel has trace debugging enabled, or if you
> have debugfs mounted, but try this:
> 
> echo 1 > /sys/kernel/debug/tracing/events/jbd2/jbd2_run_stats/enable

File does not exist... (if there is something to turn on in kernel config, I could do that for future...)

> cat /sys/kernel/debug/tracing/trace_pipe

again, file does not exist.

> 
> If these commands don't work try running this command and see if it
> works then:
> 
> mount -t debugfs debugfs /sys/kernel/debug

file does not exist (debug directory), but when I have tryied to mount it to /mnt/usbsticke dir, it has failed with unknown filesystem type (again, should I for future turn something on in kernel config?).

> 
> Let us know if you see jbd2 activity.  It's possible that sync isn't
> returning because some process is still writing to the file system,
> and if I recall correctly, Linux 3.2 didn't have some of the sync
> livelock fixes that could cause sync to not return until the file
> system had actually quieted down completely.
> 

Yes, still there is activity (first few lines from iotop):

Total DISK READ: 0.00 B/s | Total DISK WRITE: 134.97 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
17873 be/3 root        0.00 B/s    0.00 B/s  0.00 % 64.84 % [jbd2/dm-3-8]

> Regards,
> 
> 					- Ted

Have a nice day,
--
LuVar

Re: ext4 unknown problem

[LuVar]

Hi,

----- "Eric Sandeen" <sandeen@redhat.com> wrote:

> On 7/31/12 2:18 AM, luvar@plaintext.sk wrote:
> > Hi,
> > dmesg is probably useles:
> > 
> > Probably some logs after mounting...:
> > 
> > # dmesg | grep EXT
> > EXT3-fs (md2): mounted filesystem with ordered data mode
> > EXT3-fs (md2): using internal journal
> > EXT3-fs (dm-2): using internal journal
> > EXT3-fs (dm-2): mounted filesystem with ordered data mode
> > EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts:
> (null)
> > EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
> (null)
> > EXT4-fs (dm-12): mounted filesystem without journal. Opts: (null)
> > EXT4-fs (dm-8): re-mounted. Opts: commit=0
> > EXT4-fs (dm-3): re-mounted. Opts: commit=0
> > EXT4-fs (dm-12): re-mounted. Opts: commit=0
> 
> ^^^ um, what's that all about?

That is some info about startup mounted filesystems. I have about 10-15 mountpoints.

> 
> In theory, according to the docs, that's the same as the default, I
> guess.
> 
> and:
> 
>                 } else if (token == Opt_commit) {
>                         if (arg == 0)
>                                 arg = JBD2_DEFAULT_MAX_COMMIT_AGE;
> 
> ok, I guess it is.
> 
> > EXT4-fs (dm-28): mounted filesystem with ordered data mode. Opts:
> (null)
> > EXT3-fs (dm-10): using internal journal
> > EXT3-fs (dm-10): mounted filesystem with ordered data mode
> > 
> > On last few lines, there is some info about usb key used on thic
> pc:
> > 
> > # dmesg | tail -n 30
> > sd 14:0:0:0: [sdg] Asking for cache data failed
> > sd 14:0:0:0: [sdg] Assuming drive cache: write through
> ...
> 
> > Other lines from dmesg seems not relevant. Should I post them too?
> 
> Usually best to post all the data and let us sort it out, yes.

I have posted it a few minutes ago, but for comfort, my full dmesg is here:
http://dawn.ynet.sk/~luvar/asdf/20120801_1024_dmesg


> 
> -Eric
> 
> > PS: Writing to disk from jbd2/dm-3-8 still active (iotop shows it
> still on top of I/O activity).
> > 
> > LuVar
> > 
> > ----- "Theodore Ts'o" <tytso@mit.edu> wrote:
> > 
> >> On Mon, Jul 30, 2012 at 08:26:20PM +0100, LuVar wrote:
> >>> Hi,
> >>> I have seriouse issue here. I have copyied about 7GB files to one
> of
> >> my ext4 partitions and from that point of time there is sustained
> >> write to that partition from jbd2/dm-3-8 process. I have lunched
> "sudo
> >> sync" command (about day ago :) and a few hours a after sync start,
> I
> >> have tryied to use umount on that partition... Booth of them are
> still
> >> running...
> >>>
> >>> I have no serious data there, but if I could be helpfull in
> >> bugreport, give me some info what to do.
> >>
> >> Try running the dmesg command, and see if there's anything
> >> interesting
> >> there.  If you're not sure if it's interesting, send it to us and
> we
> >> can take a quick look at it.
> >>
> >> Regards,
> >>
> >> 					- Ted
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

By,
--
LuVar

Re: ext4 unknown problem

[Theodore Ts'o]

On Wed, Aug 01, 2012 at 09:30:39AM +0100, LuVar wrote:
> 
> File does not exist... (if there is something to turn on in kernel config, I could do that for future...)

The options you need to enable to use ftrace are:

    CONFIG_FUNCTION_TRACER
    CONFIG_FUNCTION_GRAPH_TRACER
    CONFIG_STACK_TRACER
    CONFIG_DYNAMIC_FTRACE    

There's a pretty good tutorial here: http://lwn.net/Articles/365835/

If you are willing to compile your own kernel, can you try going to
the latest 3.5 kernel and see if the problem still affects you there?

    	       	      	      	 - Ted

Re: ext4 unknown problem

[luvar]

I will compile my kernel with these options, but I have question what to do now. Should I ignore current state (locked journaling proccess) and just sync and reboot? (I have no problem, but I would like to try to investigate current problem if it is possible with my current kernel config).

By LuVar.


----- "Theodore Ts'o" <tytso@mit.edu> wrote:

> On Wed, Aug 01, 2012 at 09:30:39AM +0100, LuVar wrote:
> > 
> > File does not exist... (if there is something to turn on in kernel
> config, I could do that for future...)
> 
> The options you need to enable to use ftrace are:
> 
>     CONFIG_FUNCTION_TRACER
>     CONFIG_FUNCTION_GRAPH_TRACER
>     CONFIG_STACK_TRACER
>     CONFIG_DYNAMIC_FTRACE    
> 
> There's a pretty good tutorial here: http://lwn.net/Articles/365835/
> 
> If you are willing to compile your own kernel, can you try going to
> the latest 3.5 kernel and see if the problem still affects you there?
> 
>     	       	      	      	 - Ted