list_del corruption ... kernel BUG at lib/list_debug.c:62!

list_del corruption ... kernel BUG at lib/list_debug.c:62!

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 8573 bytes --]

Hello Andrew,

Alexis Huxley reported a bug against the Debian kernel where his x86_64
system died on a list corruption in rmqueue_bulk running a 6.1.129
kernel. Find below the output of the first BUG. Further details and logs
are available at https://bugs.debian.org/1101944.

The problem isn't reproducible, so Alexis cannot reliably confirm if
newer kernels are fixed. Alexis ran a memtest on that machine over-night
without any reported problems.

Does this ring a bell for you?

Best regards
Uwe

2025-04-02T11:16:47.035411+02:00 pici kernel: [747711.533523] ------------[ cut here ]------------
2025-04-02T11:16:47.035411+02:00 pici kernel: [747711.540294] kernel BUG at lib/list_debug.c:62!
2025-04-02T11:16:47.035412+02:00 pici kernel: [747711.546872] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
2025-04-02T11:16:47.035412+02:00 pici kernel: [747711.554212] CPU: 10 PID: 6770 Comm: python3 Not tainted 6.1.0-32-amd64 #1  Debian 6.1.129-1
2025-04-02T11:16:47.035413+02:00 pici kernel: [747711.564696] Hardware name: Gigabyte Technology Co., Ltd. Z790 AERO G/Z790 AERO G, BIOS F13b 12/04/2024
2025-04-02T11:16:47.035413+02:00 pici kernel: [747711.576153] RIP: 0010:__list_del_entry_valid.cold+0x23/0x6f
2025-04-02T11:16:47.035413+02:00 pici kernel: [747711.583892] Code: e8 0d 9f fe ff 0f 0b 48 89 fe 48 c7 c7 98 62 fa bc e8 fc 9e fe ff 0f 0b 48 89 d1 48 c7 c7 b8 63 fa bc 48 89 c2 e8 e8 9e fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 68 63 fa bc e8 d4 9e fe ff 0f 0b
2025-04-02T11:16:47.035414+02:00 pici kernel: [747711.604949] RSP: 0000:ffffa9fba3247b60 EFLAGS: 00010046
2025-04-02T11:16:47.035414+02:00 pici kernel: [747711.612451] RAX: 000000000000006d RBX: ffff9ba2bf6b78e8 RCX: 0000000000000000
2025-04-02T11:16:47.035414+02:00 pici kernel: [747711.621863] RDX: 0000000000000000 RSI: ffff9ba2bf6a03a0 RDI: ffff9ba2bf6a03a0
2025-04-02T11:16:47.035414+02:00 pici kernel: [747711.631254] RBP: 0000000000000010 R08: 0000000000000000 R09: ffffa9fba32479f8
2025-04-02T11:16:47.035415+02:00 pici kernel: [747711.640657] R10: 0000000000000003 R11: ffff9ba2ffb75fe8 R12: 0000000000000000
2025-04-02T11:16:47.035415+02:00 pici kernel: [747711.650042] R13: ffffd11bca09ac88 R14: ffff9ba2ffbd5c00 R15: 0000000000000000
2025-04-02T11:16:47.035416+02:00 pici kernel: [747711.659410] FS:  00007f9245392740(0000) GS:ffff9ba2bf680000(0000) knlGS:0000000000000000
2025-04-02T11:16:47.035416+02:00 pici kernel: [747711.669723] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2025-04-02T11:16:47.035416+02:00 pici kernel: [747711.677683] CR2: 00007f92449af024 CR3: 000000074b884000 CR4: 0000000000752ee0
2025-04-02T11:16:47.035416+02:00 pici kernel: [747711.687062] PKRU: 55555554
2025-04-02T11:16:47.035417+02:00 pici kernel: [747711.691986] Call Trace:
2025-04-02T11:16:47.035417+02:00 pici kernel: [747711.696619]  <TASK>
2025-04-02T11:16:47.035417+02:00 pici kernel: [747711.700866]  ? __die_body.cold+0x1a/0x1f
2025-04-02T11:16:47.035418+02:00 pici kernel: [747711.706918]  ? die+0x2a/0x50
2025-04-02T11:16:47.035418+02:00 pici kernel: [747711.711930]  ? do_trap+0xc5/0x110
2025-04-02T11:16:47.035418+02:00 pici kernel: [747711.717348]  ? __list_del_entry_valid.cold+0x23/0x6f
2025-04-02T11:16:47.035419+02:00 pici kernel: [747711.724402]  ? do_error_trap+0x6a/0x90
2025-04-02T11:16:47.035419+02:00 pici kernel: [747711.730211]  ? __list_del_entry_valid.cold+0x23/0x6f
2025-04-02T11:16:47.035419+02:00 pici kernel: [747711.737229]  ? exc_invalid_op+0x4c/0x60
2025-04-02T11:16:47.035420+02:00 pici kernel: [747711.743122]  ? __list_del_entry_valid.cold+0x23/0x6f
2025-04-02T11:16:47.035420+02:00 pici kernel: [747711.750119]  ? asm_exc_invalid_op+0x16/0x20
2025-04-02T11:16:47.035420+02:00 pici kernel: [747711.756313]  ? __list_del_entry_valid.cold+0x23/0x6f
2025-04-02T11:16:47.035420+02:00 pici kernel: [747711.763278]  rmqueue_bulk+0x124/0x5d0
2025-04-02T11:16:47.035421+02:00 pici kernel: [747711.768917]  get_page_from_freelist+0xbae/0x11b0
2025-04-02T11:16:47.035421+02:00 pici kernel: [747711.775519]  ? page_add_file_rmap+0xba/0x2f0
2025-04-02T11:16:47.035421+02:00 pici kernel: [747711.781757]  ? do_set_pte+0x189/0x1d0
2025-04-02T11:16:47.035421+02:00 pici kernel: [747711.787377]  __alloc_pages+0x1dc/0x330
2025-04-02T11:16:47.035422+02:00 pici kernel: [747711.793095]  __folio_alloc+0x17/0x50
2025-04-02T11:16:47.035422+02:00 pici kernel: [747711.798651]  ? policy_node+0x51/0x70
2025-04-02T11:16:47.035422+02:00 pici kernel: [747711.804195]  vma_alloc_folio+0x9c/0x370
2025-04-02T11:16:47.035423+02:00 pici kernel: [747711.809989]  __handle_mm_fault+0x92f/0xfa0
2025-04-02T11:16:47.035423+02:00 pici kernel: [747711.816041]  handle_mm_fault+0xdb/0x2d0
2025-04-02T11:16:47.035423+02:00 pici kernel: [747711.821822]  do_user_addr_fault+0x191/0x550
2025-04-02T11:16:47.035423+02:00 pici kernel: [747711.827943]  exc_page_fault+0x70/0x170
2025-04-02T11:16:47.035424+02:00 pici kernel: [747711.833619]  asm_exc_page_fault+0x22/0x30
2025-04-02T11:16:47.035424+02:00 pici kernel: [747711.839553] RIP: 0033:0x7f9245143628
2025-04-02T11:16:47.035424+02:00 pici kernel: [747711.845040] Code: fe ff ff 4c 89 f6 48 2b 35 95 52 20 00 49 8b 46 08 48 b9 ab aa aa aa aa aa aa aa 48 c1 fe 04 48 0f af f1 48 8d b8 00 10 00 00 <c7> 40 24 ff ff 00 00 89 70 20 49 89 7e 08 45 89 56 10 45 85 d2 0f
2025-04-02T11:16:47.035424+02:00 pici kernel: [747711.865857] RSP: 002b:00007ffe256b2510 EFLAGS: 00010a07
2025-04-02T11:16:47.035425+02:00 pici kernel: [747711.873138] RAX: 00007f92449af000 RBX: 0000000000000004 RCX: aaaaaaaaaaaaaaab
2025-04-02T11:16:47.035425+02:00 pici kernel: [747711.882354] RDX: 000000000000002f RSI: 0000000000000018 RDI: 00007f92449b0000
2025-04-02T11:16:47.035425+02:00 pici kernel: [747711.891593] RBP: 0000000000000008 R08: 000000000000002f R09: 00007f9245348680
2025-04-02T11:16:47.035425+02:00 pici kernel: [747711.900839] R10: 000000000000002e R11: 0000000019483a70 R12: 00007f92453215c0
2025-04-02T11:16:47.035426+02:00 pici kernel: [747711.910096] R13: 0000000000000018 R14: 000000001934aaa0 R15: 000000000000002e
2025-04-02T11:16:47.035426+02:00 pici kernel: [747711.919372]  </TASK>
2025-04-02T11:16:47.035426+02:00 pici kernel: [747711.923698] Modules linked in: sunrpc vhost_net vhost vhost_iotlb tap tun veth ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc qrtr binfmt_misc snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_hda_codec_hdmi soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci nls_ascii snd_sof_xtensa_dsp snd_hda_codec_realtek nls_cp437 snd_sof vfat snd_hda_codec_generic ledtrig_audio snd_sof_utils fat snd_soc_hdac_hda btusb snd_hda_ext_core snd_soc_acpi_intel_match intel_rapl_msr intel_rapl_common btrtl snd_soc_acpi btbcm x86_pkg_temp_thermal snd_soc_core intel_powerclamp iwlmvm coretemp btintel snd_compress mac80211 soundwire_bus btmtk kvm_intel i915 libarc4 kvm bluetooth snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass ghash_clmulni_intel snd_hda_codec jitterentropy_rng iwlwifi drm_buddy sha256_ssse3 sha1_ssse3 drm_display_helper sha512_ssse3
2025-04-02T11:16:47.035427+02:00 pici kernel: [747711.923723]  aesni_intel cec snd_hda_core sha512_generic ctr rc_core cfg80211 drbg crypto_simd iTCO_wdt snd_hwdep ttm pl2303 snd_pcm intel_pmc_bxt drm_kms_helper ansi_cprng iTCO_vendor_support mei_hdcp usbserial ecdh_generic ecc evdev i2c_algo_bit cryptd snd_timer rapl intel_cstate snd rfkill mei_me watchdog soundcore intel_hid gigabyte_wmi wmi_bmof intel_uncore intel_pmc_core pcspkr sparse_keymap acpi_tad acpi_pad sg mei button ib_ipoib rdma_ucm rdma_cm iw_cm ib_cm ib_umad mlx5_ib mlx5_core mlxfw psample pci_hyperv_intf ib_uverbs ib_core drbd lru_cache loop fuse efi_pstore drm configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 multipath linear hid_generic usbhid hid dm_mod raid0 md_mod uas usb_storage sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci libata crct10dif_pclmul crct10dif_common scsi_mod usbcore
2025-04-02T11:16:47.035427+02:00 pici kernel: [747712.022525]  igc e1000e crc32_pclmul crc32c_intel intel_lpss_pci i2c_i801 intel_lpss i2c_smbus video usb_common idma64 scsi_common fan wmi pinctrl_alderlake
2025-04-02T11:16:47.035428+02:00 pici kernel: [747712.142019] ---[ end trace 0000000000000000 ]---

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: list_del corruption ... kernel BUG at lib/list_debug.c:62!

[Matthew Wilcox]

On Wed, Apr 23, 2025 at 10:52:27AM +0200, Uwe Kleine-König wrote:

[trimmed all the irrelevant garbage from the start of each line]

>  ------------[ cut here ]------------
>  kernel BUG at lib/list_debug.c:62!

That's:

            CHECK_DATA_CORRUPTION(next->prev != entry,
                        "list_del corruption. next->prev should be %px, but was %px. (next=%px)\n",
                        entry, next->prev, next))
                return false;

Unfortunately, I don't see that rather useful information in this
report.

>  invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
>  CPU: 10 PID: 6770 Comm: python3 Not tainted 6.1.0-32-amd64 #1  Debian 6.1.129-1
>  Hardware name: Gigabyte Technology Co., Ltd. Z790 AERO G/Z790 AERO G, BIOS F13b 12/04/2024
>  RIP: 0010:__list_del_entry_valid.cold+0x23/0x6f
>  Code: e8 0d 9f fe ff 0f 0b 48 89 fe 48 c7 c7 98 62 fa bc e8 fc 9e fe ff 0f 0b 48 89 d1 48 c7 c7 b8 63 fa bc 48 89 c2 e8 e8 9e fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 68 63 fa bc e8 d4 9e fe ff 0f 0b
>  RSP: 0000:ffffa9fba3247b60 EFLAGS: 00010046
>  RAX: 000000000000006d RBX: ffff9ba2bf6b78e8 RCX: 0000000000000000
>  RDX: 0000000000000000 RSI: ffff9ba2bf6a03a0 RDI: ffff9ba2bf6a03a0
>  RBP: 0000000000000010 R08: 0000000000000000 R09: ffffa9fba32479f8
>  R10: 0000000000000003 R11: ffff9ba2ffb75fe8 R12: 0000000000000000
>  R13: ffffd11bca09ac88 R14: ffff9ba2ffbd5c00 R15: 0000000000000000
>  FS:  00007f9245392740(0000) GS:ffff9ba2bf680000(0000) knlGS:0000000000000000
>  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>  CR2: 00007f92449af024 CR3: 000000074b884000 CR4: 0000000000752ee0
>  PKRU: 55555554
>  Call Trace:
>   <TASK>
>   ? __die_body.cold+0x1a/0x1f
>   ? die+0x2a/0x50
>   ? do_trap+0xc5/0x110
>   ? __list_del_entry_valid.cold+0x23/0x6f
>   ? do_error_trap+0x6a/0x90
>   ? __list_del_entry_valid.cold+0x23/0x6f
>   ? exc_invalid_op+0x4c/0x60
>   ? __list_del_entry_valid.cold+0x23/0x6f
>   ? asm_exc_invalid_op+0x16/0x20
>   ? __list_del_entry_valid.cold+0x23/0x6f
>   rmqueue_bulk+0x124/0x5d0
>   get_page_from_freelist+0xbae/0x11b0

We're in the page allocator.  There's nothing particularly exciting to
be found in this backtrace, but in future, running it through
scripts/decode_stacktrace.sh can be very helpful.

>   ? page_add_file_rmap+0xba/0x2f0
>   ? do_set_pte+0x189/0x1d0
>   __alloc_pages+0x1dc/0x330
>   __folio_alloc+0x17/0x50
>   ? policy_node+0x51/0x70
>   vma_alloc_folio+0x9c/0x370
>   __handle_mm_fault+0x92f/0xfa0
>   handle_mm_fault+0xdb/0x2d0
>   do_user_addr_fault+0x191/0x550
>   exc_page_fault+0x70/0x170
>   asm_exc_page_fault+0x22/0x30
>  RIP: 0033:0x7f9245143628
>  Code: fe ff ff 4c 89 f6 48 2b 35 95 52 20 00 49 8b 46 08 48 b9 ab aa aa aa aa aa aa aa 48 c1 fe 04 48 0f af f1 48 8d b8 00 10 00 00 <c7> 40 24 ff ff 00 00 89 70 20 49 89 7e 08 45 89 56 10 45 85 d2 0f
>  RSP: 002b:00007ffe256b2510 EFLAGS: 00010a07
>  RAX: 00007f92449af000 RBX: 0000000000000004 RCX: aaaaaaaaaaaaaaab
>  RDX: 000000000000002f RSI: 0000000000000018 RDI: 00007f92449b0000
>  RBP: 0000000000000008 R08: 000000000000002f R09: 00007f9245348680
>  R10: 000000000000002e R11: 0000000019483a70 R12: 00007f92453215c0
>  R13: 0000000000000018 R14: 000000001934aaa0 R15: 000000000000002e
>   </TASK>
>  Modules linked in: sunrpc vhost_net vhost vhost_iotlb tap tun veth ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc qrtr binfmt_misc snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_hda_codec_hdmi soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci nls_ascii snd_sof_xtensa_dsp snd_hda_codec_realtek nls_cp437 snd_sof vfat snd_hda_codec_generic ledtrig_audio snd_sof_utils fat snd_soc_hdac_hda btusb snd_hda_ext_core snd_soc_acpi_intel_match intel_rapl_msr intel_rapl_common btrtl snd_soc_acpi btbcm x86_pkg_temp_thermal snd_soc_core intel_powerclamp iwlmvm coretemp btintel snd_compress mac80211 soundwire_bus btmtk kvm_intel i915 libarc4 kvm bluetooth snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi irqbypass ghash_clmulni_intel snd_hda_codec jitterentropy_rng iwlwifi drm_buddy sha256_ssse3 sha1_ssse3 drm_display_helper sha512_ssse3
>   aesni_intel cec snd_hda_core sha512_generic ctr rc_core cfg80211 drbg crypto_simd iTCO_wdt snd_hwdep ttm pl2303 snd_pcm intel_pmc_bxt drm_kms_helper ansi_cprng iTCO_vendor_support mei_hdcp usbserial ecdh_generic ecc evdev i2c_algo_bit cryptd snd_timer rapl intel_cstate snd rfkill mei_me watchdog soundcore intel_hid gigabyte_wmi wmi_bmof intel_uncore intel_pmc_core pcspkr sparse_keymap acpi_tad acpi_pad sg mei button ib_ipoib rdma_ucm rdma_cm iw_cm ib_cm ib_umad mlx5_ib mlx5_core mlxfw psample pci_hyperv_intf ib_uverbs ib_core drbd lru_cache loop fuse efi_pstore drm configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 multipath linear hid_generic usbhid hid dm_mod raid0 md_mod uas usb_storage sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci libata crct10dif_pclmul crct10dif_common scsi_mod usbcore
>   igc e1000e crc32_pclmul crc32c_intel intel_lpss_pci i2c_i801 intel_lpss i2c_smbus video usb_common idma64 scsi_common fan wmi pinctrl_alderlake
>  ---[ end trace 0000000000000000 ]---

Re: Bug#1101944: list_del corruption ... kernel BUG at lib/list_debug.c:62!

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

Hello Alexis,

On Wed, Apr 23, 2025 at 01:19:21PM +0100, Matthew Wilcox wrote:
> On Wed, Apr 23, 2025 at 10:52:27AM +0200, Uwe Kleine-König wrote:
> 
> [trimmed all the irrelevant garbage from the start of each line]

As Matthew pointed out the output of the CHECK_DATA_CORRUPTION would be
interesting. Can you please check if you can find this line still in
your journal?

> >  ------------[ cut here ]------------
> >  kernel BUG at lib/list_debug.c:62!
> 
> That's:
> 
>             CHECK_DATA_CORRUPTION(next->prev != entry,
>                         "list_del corruption. next->prev should be %px, but was %px. (next=%px)\n",
>                         entry, next->prev, next))
>                 return false;
> 
> Unfortunately, I don't see that rather useful information in this
> report.
> 
> >  invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> >  CPU: 10 PID: 6770 Comm: python3 Not tainted 6.1.0-32-amd64 #1  Debian 6.1.129-1
> >  Hardware name: Gigabyte Technology Co., Ltd. Z790 AERO G/Z790 AERO G, BIOS F13b 12/04/2024
> >  RIP: 0010:__list_del_entry_valid.cold+0x23/0x6f
> >  Code: e8 0d 9f fe ff 0f 0b 48 89 fe 48 c7 c7 98 62 fa bc e8 fc 9e fe ff 0f 0b 48 89 d1 48 c7 c7 b8 63 fa bc 48 89 c2 e8 e8 9e fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 68 63 fa bc e8 d4 9e fe ff 0f 0b
> >  RSP: 0000:ffffa9fba3247b60 EFLAGS: 00010046
> >  RAX: 000000000000006d RBX: ffff9ba2bf6b78e8 RCX: 0000000000000000
> >  RDX: 0000000000000000 RSI: ffff9ba2bf6a03a0 RDI: ffff9ba2bf6a03a0
> >  RBP: 0000000000000010 R08: 0000000000000000 R09: ffffa9fba32479f8
> >  R10: 0000000000000003 R11: ffff9ba2ffb75fe8 R12: 0000000000000000
> >  R13: ffffd11bca09ac88 R14: ffff9ba2ffbd5c00 R15: 0000000000000000
> >  FS:  00007f9245392740(0000) GS:ffff9ba2bf680000(0000) knlGS:0000000000000000
> >  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >  CR2: 00007f92449af024 CR3: 000000074b884000 CR4: 0000000000752ee0
> >  PKRU: 55555554
> >  Call Trace:
> >   <TASK>
> >   ? __die_body.cold+0x1a/0x1f
> >   ? die+0x2a/0x50
> >   ? do_trap+0xc5/0x110
> >   ? __list_del_entry_valid.cold+0x23/0x6f
> >   ? do_error_trap+0x6a/0x90
> >   ? __list_del_entry_valid.cold+0x23/0x6f
> >   ? exc_invalid_op+0x4c/0x60
> >   ? __list_del_entry_valid.cold+0x23/0x6f
> >   ? asm_exc_invalid_op+0x16/0x20
> >   ? __list_del_entry_valid.cold+0x23/0x6f
> >   rmqueue_bulk+0x124/0x5d0
> >   get_page_from_freelist+0xbae/0x11b0
> 
> We're in the page allocator.  There's nothing particularly exciting to
> be found in this backtrace, but in future, running it through
> scripts/decode_stacktrace.sh can be very helpful.

I can do that. Will follow up when I find the time.

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Bug#1101944: list_del corruption ... kernel BUG at lib/list_debug.c:62!

[Alexis Huxley]

Hi Uwe,

> On Wed, Apr 23, 2025 at 01:19:21PM +0100, Matthew Wilcox wrote:
> > On Wed, Apr 23, 2025 at 10:52:27AM +0200, Uwe Kleine-König wrote:
> > 
> > [trimmed all the irrelevant garbage from the start of each line]
> 
> As Matthew pointed out the output of the CHECK_DATA_CORRUPTION would be
> interesting. Can you please check if you can find this line still in
> your journal?

I have nightly backups of the machine covering the period when the
crash happened, but I do not find 'corrupt' in a case insensitive
search of uncompressed or compressed files in /var/log of the backup
of the next night or the night after that.

journalctl on the machine itself does not go back that far.

At this point it's fine for me if you want to mark this wonfix or
cantfix or whatever: the bug is recorded and searchable and I'm not
able to further help you to help me; if anybody else encounters the 
problem then the logs might be useful reference.

Alexis