HDD problem, software bug, bios bug, or hardware ?

[Adko Branil <adkobranil@yahoo.com>]

My system hangs from time to time, after few hours work(which differs from minutes to 8-9 hours), with kernel panic. Before that begins it worked fine for about 6 years - no software or hardware changes during this period.
I have some photos of the screen after panic, the first two are with the old linux kernel 2.6.16.27:

http://picpaste.com/pics/img00005-73m0unO0.1345852235.jpg
http://picpaste.com/pics/P170812_12.01-MeZrs3zv.1345817375.jpg

-they can enlarge on click.


Then i installed slackware-current with their default kernel "huge.s" and the crashes continued:

http://picpaste.com/pics/P210812_15.34-3NSTEV8f.1345816730.jpg

then i swithced off the swap:

http://picpaste.com/pics/P230812_15.06-hB12169n.1345812390.jpg

after that i managed to save one message with netconsole (swap is off):

	1. [13330.042569] BUG: unable to handle kernel paging request at 000060ff80001f1c
	2. [13330.043554] IP: [<ffffffff810b17e0>] no_action+0x10/0x10
	3. [13330.043554] PGD 0 
	4. [13330.043554] Oops: 0002 [#1] SMP 
	5. [13330.043554] CPU 1 
	6. [13330.043554] Modules linked in: ipv6 
lp netconsole snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq 
snd_pcm_oss snd_mixer_oss fuse nouveau mxm_wmi wmi video ttm 
drm_kms_helper drm amd64_agp processor thermal_sys k8temp agpgart hwmon 
snd_via82xx snd_ac97_codec snd_mpu401_uart snd_rawmidi snd_seq_device 
snd_pcm snd_page_alloc snd_timer snd soundcore ac97_bus ppdev parport_pc i2c_algo_bit gameport evdev shpchp button i2c_viapro i2c_core loop skge parport [last unloaded: lp]
	7. [13330.043554] 
	8. [13330.043554] Pid: 0, comm: swapper/1 Not tainted 3.2.27 #2 To Be Filled By O.E.M. To Be Filled By O.E.M./A8V Deluxe
	9. [13330.043554] RIP: 0010:[<ffffffff810b17e0>]  [<ffffffff810b17e0>] no_action+0x10/0x10
	10. [13330.043554] RSP: 0018:ffff88007fd03f10  EFLAGS: 00010086
	11. [13330.043554] RAX: 000060ff80001f1c RBX: ffff88007aef2c00 RCX: 00000000fffffffa
	12. [13330.043554] RDX: 00000000000000d0 RSI: ffff88007ae93f80 RDI: ffff88007aef2c00
	13. [13330.043554] RBP: ffff88007fd03f38 R08: ffff88007aef2c00 R09: ffff88007cc00000
	14. [13330.043554] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88007aef2c8c
	15. [13330.043554] R13: 0000000000000011 R14: 0000000000000000 R15: 0000000000000000
	16. [13330.043554] FS:  00007f674b3e6740(0000) GS:ffff88007fd00000(0000) knlGS:00000000f7369700
	17. [13330.043554] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
	18. [13330.043554] CR2: 000060ff80001f1c CR3: 000000006f115000 CR4: 00000000000006e0
	19. [13330.043554] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
	20. [13330.043554] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
	21. [13330.043554] Process swapper/1 (pid: 0, threadinfo ffff88007bd18000, task ffff88007d0ec4c0)
	22. [13330.043554] Stack:
	23. [13330.043554]  ffffffff810b1a10 ffff88007fd03f58 ffff88007aef2c00 0000000000000051
	24. [13330.043554]  0000000000000011 ffff88007fd03f58 ffffffff810b4879 ffff88007fd03f58
	25. [13330.043554]  0000000000000011 ffff88007fd03f78 ffffffff81003d12 ffff88007fd03f78
	26. [13330.043554] Call Trace:
	27. [13330.043554]  <IRQ> 
	28. [13330.043554]  [<ffffffff810b1a10>] ? handle_irq_event+0x40/0x70
	29. [13330.043554]  [<ffffffff810b4879>] handle_fasteoi_irq+0x59/0x100
	30. [13330.043554]  [<ffffffff81003d12>] handle_irq+0x22/0x40
	31. [13330.043554]  [<ffffffff81b3158a>] do_IRQ+0x5a/0xe0
	32. [13330.043554]  [<ffffffff81b2e82b>] common_interrupt+0x6b/0x6b
	33. [13330.043554]  <EOI>

 here is link to dmesg, before that last crash: http://pastebin.com/Af7bb34x

And at the end i noticed scary messages in the syslog:

[31770.094556] REISERFS warning (device sda1): clm-6006 reiserfs_dirty_inode: writing inode 347717 on readonly FS
[31770.472848] REISERFS warning (device sda1): clm-6006 reiserfs_dirty_inode: writing inode 347740 on readonly FS
[31790.796117] REISERFS warning (device sda1): clm-6006 reiserfs_dirty_inode: writing inode 426162 on readonly FS


after which i have done reiserfsck immediately - no corruption were found.
Never seen such messages before, i have syslogs for 17 days before that  - no messages like this.

I have done some tests with smartmontools before - when it was the old linux (2.6.16.27) - the result of "smartctl -s on -a /dev/sda" is:

smartctl 5.43 2012-06-30 r3573 [x86_64-linux-3.5.2] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.7 and 7200.7 Plus
Device Model:     ST3200822AS
Serial Number:    4LJ221BB
Firmware Version: 3.01
User Capacity:    200,049,647,616 bytes [200 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Sat Aug 25 03:09:01 2012 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  430) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        No General Purpose Logging support.
Short self-test routine 
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 111) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   050   046   006    Pre-fail  Always       -       179699255
  3 Spin_Up_Time            0x0003   097   096   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       123
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       6
  7 Seek_Error_Rate         0x000f   078   060   030    Pre-fail  Always       -       81170784
  9 Power_On_Hours          0x0032   039   039   000    Old_age   Always       -       53553
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       142
194 Temperature_Celsius     0x0022   037   054   000    Old_age   Always       -       37
195 Hardware_ECC_Recovered  0x001a   050   046   000    Old_age   Always       -       179699255
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   198   000    Old_age   Always       -       2
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 Data_Address_Mark_Errs  0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 2
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 13784 hours (574 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 00 7a 7d 1d e0  Error: ICRC, ABRT at LBA = 0x001d7d7a = 1932666

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 7b 7c 1d e0 00      22:14:23.595  READ DMA EXT
  25 00 00 7b 7b 1d e0 00      22:14:23.593  READ DMA EXT
  25 00 00 7b 7a 1d e0 00      22:14:23.576  READ DMA EXT
  25 00 00 7b 79 1d e0 00      22:14:23.567  READ DMA EXT
  25 00 00 7b 78 1d e0 00      22:14:23.566  READ DMA EXT

Error 1 occurred at disk power-on lifetime: 13784 hours (574 days + 8 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 00 fa 0e 01 e0  Error: ICRC, ABRT at LBA = 0x00010efa = 69370

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 fb 0d 01 e0 00      22:13:03.489  READ DMA EXT
  25 00 00 fb 0c 01 e0 00      22:13:03.487  READ DMA EXT
  25 00 00 fb 0b 01 e0 00      22:13:03.701  READ DMA EXT
  25 00 00 fb 09 01 e0 00      22:13:03.682  READ DMA EXT
  25 00 00 fb 07 01 e0 00      22:13:03.681  READ DMA EXT

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     53153         -
# 2  Short offline       Completed without error       00%     53152         -
# 3  Short offline       Completed without error       00%     53152         -
# 4  Short offline       Completed without error       00%     53152         -
# 5  Short offline       Completed without error       00%     53152         -
# 6  Short offline       Completed without error       00%     53148         -
# 7  Short offline       Completed without error       00%     53148         -
# 8  Short offline       Completed without error       00%     53148         -
# 9  Extended offline    Aborted by host               80%     53148         -
#10  Short offline       Completed without error       00%     53147         -
#11  Short offline       Completed without error       00%     53147         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


And soon after that (you can see the time of the messages)" i just succeed to to capture one whole panic message(i am hoping it is):

[32874.215014] BUG: unable to handle kernel NULL pointer dereference at 0000000000000086
[32874.215192] IP: [<ffffffff819f9440>] start_show+0x30/0x30
[32874.215192] PGD 7afe0067 PUD 7497e067 PMD 0 
[32874.215192] Oops: 0002 [#1] SMP 
[32874.215192] CPU 1 
[32874.215192] Modules linked in: netconsole ipt_REJECT xt_tcpudp iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_filter ip_tables x_tables ipv6 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss fuse nouveau mxm_wmi wmi video ttm drm_kms_helper snd_via82xx snd_ac97_codec snd_mpu401_uart snd_rawmidi snd_seq_device snd_pcm snd_page_alloc drm snd_timer amd64_agp processor i2c_algo_bit snd shpchp k8temp agpgart thermal_sys i2c_viapro hwmon i2c_core skge soundcore ac97_bus gameport evdev ppdev button parport_pc parport loop [last unloaded: lp]
[32874.215192] 
[32874.215192] Pid: 0, comm: swapper/1 Not tainted 3.2.27 #2 To Be Filled By O.E.M. To Be Filled By O.E.M./A8V Deluxe
[32874.215192] RIP: 0010:[<ffffffff819f9440>]  [<ffffffff819f9440>] start_show+0x30/0x30
[32874.215192] RSP: 0018:ffff88007fd03eb0  EFLAGS: 00010006
[32874.215192] RAX: 0000000000000086 RBX: ffffffff820c2fc0 RCX: 0000000000000001
[32874.215192] RDX: 00001de61fe84bdb RSI: 0000000000000000 RDI: ffffffff820c2fc0
[32874.215192] RBP: ffff88007fd03ed8 R08: 0000000000000000 R09: 0000000000000001
[32874.215192] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000008069
[32874.215192] R13: 00000000484f99af R14: 0000000000ab2476 R15: 0000000000000000
[32874.215192] FS:  00007f61bddf4740(0000) GS:ffff88007fd00000(0000) knlGS:00000000f75fc6c0
[32874.215192] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[32874.215192] CR2: 0000000000000086 CR3: 00000000746e8000 CR4: 00000000000006e0
[32874.215192] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[32874.215192] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[32874.215192] Process swapper/1 (pid: 0, threadinfo ffff88007bd18000, task ffff88007d0ec4c0)
[32874.215192] Stack:
[32874.215192]  ffffffff8107df04 ffff88007fd12680 0000000000000001 000000000000d300
[32874.215192]  0000000000000000 ffff88007fd03ef8 ffffffff8107ab80 ffff88007fd0d300
[32874.215192]  0000000000000001 ffff88007fd03f08 ffffffff8107abe9 ffff88007fd03f28
[32874.215192] Call Trace:
[32874.215192]  <IRQ> 
[32874.215192]  [<ffffffff8107df04>] ? ktime_get+0x64/0xe0
[32874.215192]  [<ffffffff8107ab80>] sched_clock_tick+0x40/0x90
[32874.215192]  [<ffffffff8107abe9>] sched_clock_idle_wakeup_event+0x19/0x20
[32874.215192]  [<ffffffff8108538e>] tick_nohz_stop_idle+0x3e/0x50
[32874.215192]  [<ffffffff81085b77>] tick_check_idle+0xb7/0xd0
[32874.215192]  [<ffffffff8105a749>] irq_enter+0x69/0x70
[32874.215192]  [<ffffffff81b31653>] smp_apic_timer_interrupt+0x43/0x99
[32874.215192]  [<ffffffff81b2f9cb>] apic_timer_interrupt+0x6b/0x70
[32874.215192]  <EOI> 
[32874.215192]  [<ffffffff8107aa58>] ? sched_clock_cpu+0xa8/0x120
[32874.215192]  [<ffffffff8100a89a>] ? default_idle+0x5a/0x180
[32874.215192]  [<ffffffff810009b6>] cpu_idle+0xf6/0x110
[32874.215192]  [<ffffffff81b146ea>] start_secondary+0x1cf/0x1d6
[32874.215192] Code: 66 66 66 90 48 8b 0f 48 c7 c2 0d 46 dc 81 48 89 f0 be 00 10 00 00 48 89 c7 31 c0 e8 5b 71 b9 ff 5d 48 98 c3 0f 1f 80 00 00 00 00 <55> 48 89 e5 66 66 66 66 90 8b 15 39 31 6f 00 ed 25 ff ff ff 00 
[32874.215192] RIP  [<ffffffff819f9440>] start_show+0x30/0x30
[32874.215192]  RSP <ffff88007fd03eb0>
[32874.215192] CR2: 0000000000000086
[32874.215192] [drm] nouveau 0000:01:00.0: Setting dpms mode 0 on vga encoder (output 0)
[32874.215192] ---[ end trace 90aad159d8ed7c1e ]---
[32874.215192] Kernel panic - not syncing: Fatal exception in interrupt
[32874.215192] Pid: 0, comm: swapper/1 Tainted: G      D      3.2.27 #2
[32874.215192] Call Trace:
[32874.215192]  <IRQ>  [<ffffffff81b1aeea>] panic+0x91/0x189
[32874.215192]  [<ffffffff81005491>] oops_end+0x91/0xa0
[32874.215192]  [<ffffffff81b1a85f>] no_context+0x1fa/0x225
[32874.215192]  [<ffffffff81b1aa3b>] __bad_area_nosemaphore+0x1b1/0x1d0
[32874.215192]  [<ffffffff81b1aa6d>] bad_area_nosemaphore+0x13/0x15
[32874.215192]  [<ffffffff81028794>] do_page_fault+0x2b4/0x480
[32874.215192]  [<ffffffff8104aa6c>] ? load_balance+0xac/0x780
[32874.215192]  [<ffffffff81a1b1e0>] ? skb_release_head_state+0x60/0x100
[32874.215192]  [<ffffffff81a1affe>] ? __kfree_skb+0x1e/0xa0
[32874.215192]  [<ffffffff81a1b0b1>] ? consume_skb+0x31/0x70
[32874.215192]  [<ffffffff81b2ea2f>] page_fault+0x1f/0x30
[32874.215192]  [<ffffffff819f9440>] ? start_show+0x30/0x30
[32874.215192]  [<ffffffff8107df04>] ? ktime_get+0x64/0xe0
[32874.215192]  [<ffffffff8107ab80>] sched_clock_tick+0x40/0x90
[32874.215192]  [<ffffffff8107abe9>] sched_clock_idle_wakeup_event+0x19/0x20
[32874.215192]  [<ffffffff8108538e>] tick_nohz_stop_idle+0x3e/0x50
[32874.215192]  [<ffffffff81085b77>] tick_check_idle+0xb7/0xd0
[32874.215192]  [<ffffffff8105a749>] irq_enter+0x69/0x70
[32874.215192]  [<ffffffff81b31653>] smp_apic_timer_interrupt+0x43/0x99
[32874.215192]  [<ffffffff81b2f9cb>] apic_timer_interrupt+0x6b/0x70
[32874.215192]  <EOI>  [<ffffffff8107aa58>] ? sched_clock_cpu+0xa8/0x120
[32874.215192]  [<ffffffff8100a89a>] ? default_idle+0x5a/0x180
[32874.215192]  [<ffffffff810009b6>] cpu_idle+0xf6/0x110
[32874.215192]  [<ffffffff81b146ea>] start_secondary+0x1cf/0x1d6
[32874.215192] panic occurred, switching back to text console.

swap is off again.

After that i ran the machine with the newest kernel - 3.5.2, and if it happens again i will try "nosmp" option.Any ideas of what should be the reason, or how to catch it, will be welcome.

Is that the right place to ask, or should i send it to kernel@vger.kernel.org, or somewhere else ?
Thanks in advance !

Adko.