Re: [PATCH 0/5] btrfs: snapshot deletion via readahead

[Arne Jansen <sensille@gmx.net>]

On 13.04.2012 09:10, Liu Bo wrote:
> On 04/13/2012 02:53 PM, Arne Jansen wrote:
> 
>> On 13.04.2012 05:40, Liu Bo wrote:
>>>> On 04/12/2012 11:54 PM, Arne Jansen wrote:
>>>>>> This patchset reimplements snapshot deletion with the help of the readahead
>>>>>> framework. For this callbacks are added to the framework. The main idea is
>>>>>> to traverse many snapshots at once at read many branches at once. This way
>>>>>> readahead get many requests at once (currently about 50000), giving it the
>>>>>> chance to order disk accesses properly. On a single disk, the effect is
>>>>>> currently spoiled by sync operations that still take place, mainly checksum
>>>>>> deletion. The most benefit can be gained with multiple devices, as all devices
>>>>>> can be fully utilized. It scales quite well with the number of devices.
>>>>>> For more details see the commit messages of the individual patches and the
>>>>>> source code comments.
>>>>>>
>>>>>> How it is tested:
>>>>>> I created a test volume using David Sterba's stress-subvol-git-aging.sh. It
>>>>>> checks out randoms version of the kernel git tree, creating a snapshot from it
>>>>>> from time to time and checks out other versions there, and so on. In the end
>>>>>> the fs had 80 subvols with various degrees of sharing between them. The
>>>>>> following tests were conducted on it:
>>>>>>  - delete a subvol using droptree and check the fs with btrfsck afterwards
>>>>>>    for consistency
>>>>>>  - delete all subvols and verify with btrfs-debug-tree that the extent
>>>>>>    allocation tree is clean
>>>>>>  - delete 70 subvols, and in parallel empty the other 10 with rm -rf to get
>>>>>>    a good pressure on locking
>>>>>>  - add various degrees of memory pressure to the previous test to get pages
>>>>>>    to expire early from page cache
>>>>>>  - enable all relevant kernel debugging options during all tests
>>>>>>
>>>>>> The performance gain on a single drive was about 20%, on 8 drives about 600%.
>>>>>> It depends vastly on the maximum parallelity of the readahead, that is
>>>>>> currently hardcoded to about 50000. This number is subject to 2 factors, the
>>>>>> available RAM and the size of the saved state for a commit. As the full state
>>>>>> has to be saved on commit, a large parallelity leads to a large state.
>>>>>>
>>>>>> Based on this I'll see if I can add delayed checksum deletions and running
>>>>>> the delayed refs via readahead, to gain a maximum ordering of I/O ops.
>>>>>>
>>>>
>>>> Hi Arne,
>>>>
>>>> Can you please show us some user cases in this, or can we get some extra benefits from it? :)
>>
>> The case I'm most concerned with is having large filesystems (like 20x3T)
>> with thousands of users on it in thousands of subvolumes and making
>> hourly snapshots. Creating these snapshots is relatively easy, getting rid
>> of them is not.
>> But there are already reports where deleting a single snapshot can take
>> several days. So we really need a huge speedup here, and this is only
>> the beginning :)
> 
> 
> I see.
> 
> I've just tested it on 3.4-rc2, I cannot get it through due to the following, could you take a look?
> 
> Apr  8 14:58:08 kvm kernel: ------------[ cut here ]------------
> Apr  8 14:58:08 kvm kernel: kernel BUG at fs/btrfs/droptree.c:418!

might be out of memory. How much does this vm (?) have?
Can you try to reduce the constants in disk-io.c:2003-2005?

Thanks,
Arne

> Apr  8 14:58:08 kvm kernel: invalid opcode: 0000 [#1] SMP
> Apr  8 14:58:08 kvm kernel: CPU 1
> Apr  8 14:58:08 kvm kernel: Modules linked in: btrfs(O) zlib_deflate libcrc32c ip6table_filter ip6_tables ebtable_nat ebtables iptable_filter ipt_REJECT ip_tables bridge stp llc nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ext3 jbd dm_mirror dm_region_hash dm_log dm_mod kvm_intel kvm ppdev sg parport_pc parport coretemp hwmon pcspkr i2c_i801 iTCO_wdt iTCO_vendor_support sky2 snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_
 generic ata_piix i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: btrfs]
> Apr  8 14:58:08 kvm kernel:
> Apr  8 14:58:08 kvm kernel: Pid: 532, comm: btrfs-readahead Tainted: G        W  O 3.4.0-rc1+ #10 LENOVO QiTianM7150/To be filled by O.E.M.
> Apr  8 14:58:08 kvm kernel: RIP: 0010:[<ffffffffa082f800>]  [<ffffffffa082f800>] droptree_fetch_ref+0x4b0/0x4c0 [btrfs]
> Apr  8 14:58:08 kvm kernel: RSP: 0018:ffff88003418bda0  EFLAGS: 00010286
> Apr  8 14:58:08 kvm kernel: RAX: 00000000ffffffff RBX: ffff88007ab74348 RCX: 0000000105585190
> Apr  8 14:58:08 kvm kernel: RDX: 000000000000003a RSI: ffffffff81ade6a0 RDI: 0000000000000286
> Apr  8 14:58:08 kvm kernel: RBP: ffff88003418be10 R08: 000000000000003f R09: 0000000000000003
> Apr  8 14:58:08 kvm kernel: R10: 0000000000000002 R11: 0000000000008340 R12: ffff880004194718
> Apr  8 14:58:08 kvm kernel: R13: ffff88004004e000 R14: ffff880034b9b000 R15: ffff88000c64a820
> Apr  8 14:58:08 kvm kernel: FS:  0000000000000000(0000) GS:ffff88007da80000(0000) knlGS:0000000000000000
> Apr  8 14:58:08 kvm kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr  8 14:58:08 kvm kernel: CR2: 0000003842d454a4 CR3: 000000003d0a0000 CR4: 00000000000407e0
> Apr  8 14:58:08 kvm kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr  8 14:58:08 kvm kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr  8 14:58:08 kvm kernel: Process btrfs-readahead (pid: 532, threadinfo ffff88003418a000, task ffff880076d3a040)
> Apr  8 14:58:08 kvm kernel: Stack:
> Apr  8 14:58:08 kvm kernel: ffff880078f06d40 ffff88007690ecd8 ffff88000c605758 ffff880078f06d80
> Apr  8 14:58:08 kvm kernel: ffff880036916740 ffff88007ab742c0 0000000000000002 000000000000064f
> Apr  8 14:58:08 kvm kernel: ffff88003418be10 ffff88007690ecc0 ffff88007690ed10 ffff88007690ecd8
> Apr  8 14:58:08 kvm kernel: Call Trace:
> Apr  8 14:58:08 kvm kernel: [<ffffffffa08032df>] worker_loop+0x14f/0x5a0 [btrfs]
> Apr  8 14:58:08 kvm kernel: [<ffffffffa0803190>] ? btrfs_queue_worker+0x300/0x300 [btrfs]
> Apr  8 14:58:08 kvm kernel: [<ffffffffa0803190>] ? btrfs_queue_worker+0x300/0x300 [btrfs]
> Apr  8 14:58:08 kvm kernel: [<ffffffff8106f1ae>] kthread+0x9e/0xb0
> Apr  8 14:58:08 kvm kernel: [<ffffffff814fbea4>] kernel_thread_helper+0x4/0x10
> Apr  8 14:58:08 kvm kernel: [<ffffffff8106f110>] ? kthread_freezable_should_stop+0x70/0x70
> Apr  8 14:58:08 kvm kernel: [<ffffffff814fbea0>] ? gs_change+0x13/0x13
> Apr  8 14:58:08 kvm kernel: Code: fe 0f 0b 0f 1f 84 00 00 00 00 00 eb f6 0f 0b eb fe be fe 01 00 00 48 c7 c7 58 7b 83 a0 e8 e9 dd 81 e0 44 8b 55 a8 e9 77 ff ff ff <0f> 0b eb fe 0f 0b eb fe 90 90 90 90 90 90 90 90 55 48 89 e5 41
> Apr  8 14:58:08 kvm kernel: RIP  [<ffffffffa082f800>] droptree_fetch_ref+0x4b0/0x4c0 [btrfs]
> 
> 
> thanks,