* Re: [PATCH] md: avoid warning for 32-bit sector_t
From: NeilBrown @ 2015-12-16 4:02 UTC (permalink / raw)
To: Arnd Bergmann, linux-raid; +Cc: linux-kernel, linux-arm-kernel
In-Reply-To: <65226011.NWtA6TWNA3@wuerfel>
[-- Attachment #1: Type: text/plain, Size: 2301 bytes --]
On Tue, Nov 24 2015, Arnd Bergmann wrote:
> When CONFIG_LBDAF is not set, sector_t is only 32-bits wide, which
> means we cannot have devices with more than 2TB, and the code that
> is trying to handle compatibility support for large devices in
> md version 0.90 is meaningless but also causes a compile-time warning:
>
> drivers/md/md.c: In function 'super_90_load':
> drivers/md/md.c:1029:19: warning: large integer implicitly truncated to unsigned type [-Woverflow]
> drivers/md/md.c: In function 'super_90_rdev_size_change':
> drivers/md/md.c:1323:17: warning: large integer implicitly truncated to unsigned type [-Woverflow]
>
> This adds a check for CONFIG_LBDAF to avoid even getting into this
> code path, and also adds an explicit cast to let the compiler know
> it doesn't have to warn about the truncation.
>
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> ---
> Noticed on ARM randconfig builds with recent gcc versions.
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 0b48c5d7c489..ee9a7ab44f32 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -1025,8 +1025,9 @@ static int super_90_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor
> * (not needed for Linear and RAID0 as metadata doesn't
> * record this size)
> */
> - if (rdev->sectors >= (2ULL << 32) && sb->level >= 1)
> - rdev->sectors = (2ULL << 32) - 2;
> + if (IS_ENABLED(CONFIG_LBDAF) && (u64)rdev->sectors >= (2ULL << 32) &&
> + sb->level >= 1)
> + rdev->sectors = (sector_t)(2ULL << 32) - 2;
>
> if (rdev->sectors < ((sector_t)sb->size) * 2 && sb->level >= 1)
> /* "this cannot possibly happen" ... */
> @@ -1319,8 +1320,9 @@ super_90_rdev_size_change(struct md_rdev *rdev, sector_t num_sectors)
> /* Limit to 4TB as metadata cannot record more than that.
> * 4TB == 2^32 KB, or 2*2^32 sectors.
> */
> - if (num_sectors >= (2ULL << 32) && rdev->mddev->level >= 1)
> - num_sectors = (2ULL << 32) - 2;
> + if (IS_ENABLED(CONFIG_LBDAF) && (u64)num_sectors >= (2ULL << 32) &&
> + rdev->mddev->level >= 1)
> + num_sectors = (sector_t)(2ULL << 32) - 2;
> md_super_write(rdev->mddev, rdev, rdev->sb_start, rdev->sb_size,
> rdev->sb_page);
> md_super_wait(rdev->mddev);
applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] drivers: md: use ktime_get_real_seconds()
From: NeilBrown @ 2015-12-16 4:18 UTC (permalink / raw)
To: Deepa Dinamani, linux-kernel, linux-raid, Arnd Bergmann
In-Reply-To: <1447384577-30866-1-git-send-email-deepa.kernel@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6213 bytes --]
On Fri, Nov 13 2015, Deepa Dinamani wrote:
> get_seconds() API is not y2038 safe on 32 bit systems and the API
> is deprecated. Replace it with calls to ktime_get_real_seconds()
> API instead. Change mddev structure types to time64_t accordingly.
>
> 32 bit signed timestamps will overflow in the year 2038.
>
> Change the user interface mdu_array_info_s structure timestamps:
> ctime and utime values used in ioctls GET_ARRAY_INFO and
> SET_ARRAY_INFO to unsigned int. This will extend the field to last
> until the year 2106.
> The long term plan is to get rid of ctime and utime values in
> this structure as this information can be read from the on-disk
> meta data directly.
>
> Clamp the tim64_t timestamps to positive values with a max of U32_MAX
> when returning from GET_ARRAY_INFO ioctl to accommodate above changes
> in the data type of timestamps to unsigned int.
>
> v0.90 on disk meta data uses u32 for maintaining time stamps.
> So this will also last until year 2106.
> Assumption is that the usage of v0.90 will be deprecated by
> year 2106.
>
> Timestamp fields in the on disk meta data for v1.0 version already
> use 64 bit data types. Remove the truncation of the bits while
> writing to or reading from these from the disk.
>
> Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com>
> Reviewed-by: Arnd Bergmann <arnd@arndb.de>
> ---
>
> Adding the maintainer lists for md.
>
> Notes:
> A separate patch will update mdadm to obtain times from the metadata,
> and to give a deprecation warning for use of v0.90 arrays
>
> drivers/md/md.c | 18 +++++++++---------
> drivers/md/md.h | 2 +-
> include/uapi/linux/raid/md_u.h | 4 ++--
> 3 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 7ab9ed9..20763ea 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -1196,13 +1196,13 @@ static void super_90_sync(struct mddev *mddev, struct md_rdev *rdev)
> memcpy(&sb->set_uuid2, mddev->uuid+8, 4);
> memcpy(&sb->set_uuid3, mddev->uuid+12,4);
>
> - sb->ctime = mddev->ctime;
> + sb->ctime = clamp_t(time64_t, mddev->ctime, 0, U32_MAX);
> sb->level = mddev->level;
> sb->size = mddev->dev_sectors / 2;
> sb->raid_disks = mddev->raid_disks;
> sb->md_minor = mddev->md_minor;
> sb->not_persistent = 0;
> - sb->utime = mddev->utime;
> + sb->utime = clamp_t(time64_t, mddev->utime, 0, U32_MAX);
> sb->state = 0;
> sb->events_hi = (mddev->events>>32);
> sb->events_lo = (u32)mddev->events;
> @@ -1542,8 +1542,8 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
> mddev->patch_version = 0;
> mddev->external = 0;
> mddev->chunk_sectors = le32_to_cpu(sb->chunksize);
> - mddev->ctime = le64_to_cpu(sb->ctime) & ((1ULL << 32)-1);
> - mddev->utime = le64_to_cpu(sb->utime) & ((1ULL << 32)-1);
> + mddev->ctime = le64_to_cpu(sb->ctime);
> + mddev->utime = le64_to_cpu(sb->utime);
> mddev->level = le32_to_cpu(sb->level);
> mddev->clevel[0] = 0;
> mddev->layout = le32_to_cpu(sb->layout);
> @@ -2331,7 +2331,7 @@ repeat:
>
> spin_lock(&mddev->lock);
>
> - mddev->utime = get_seconds();
> + mddev->utime = ktime_get_real_seconds();
>
> if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
> force_change = 1;
> @@ -5828,7 +5828,7 @@ static int get_array_info(struct mddev *mddev, void __user *arg)
> info.major_version = mddev->major_version;
> info.minor_version = mddev->minor_version;
> info.patch_version = MD_PATCHLEVEL_VERSION;
> - info.ctime = mddev->ctime;
> + info.ctime = clamp_t(time64_t, mddev->ctime, 0, U32_MAX);
> info.level = mddev->level;
> info.size = mddev->dev_sectors / 2;
> if (info.size != mddev->dev_sectors / 2) /* overflow */
> @@ -5838,7 +5838,7 @@ static int get_array_info(struct mddev *mddev, void __user *arg)
> info.md_minor = mddev->md_minor;
> info.not_persistent= !mddev->persistent;
>
> - info.utime = mddev->utime;
> + info.utime = clamp_t(time64_t, mddev->utime, 0, U32_MAX);
> info.state = 0;
> if (mddev->in_sync)
> info.state = (1<<MD_SB_CLEAN);
> @@ -6338,13 +6338,13 @@ static int set_array_info(struct mddev *mddev, mdu_array_info_t *info)
> /* ensure mddev_put doesn't delete this now that there
> * is some minimal configuration.
> */
> - mddev->ctime = get_seconds();
> + mddev->ctime = ktime_get_real_seconds();
> return 0;
> }
> mddev->major_version = MD_MAJOR_VERSION;
> mddev->minor_version = MD_MINOR_VERSION;
> mddev->patch_version = MD_PATCHLEVEL_VERSION;
> - mddev->ctime = get_seconds();
> + mddev->ctime = ktime_get_real_seconds();
>
> mddev->level = info->level;
> mddev->clevel[0] = 0;
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index f5b9aad..237b507 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -261,7 +261,7 @@ struct mddev {
> * managed externally */
> char metadata_type[17]; /* externally set*/
> int chunk_sectors;
> - time_t ctime, utime;
> + time64_t ctime, utime;
> int level, layout;
> char clevel[16];
> int raid_disks;
> diff --git a/include/uapi/linux/raid/md_u.h b/include/uapi/linux/raid/md_u.h
> index 1cb8aa6..36cd821 100644
> --- a/include/uapi/linux/raid/md_u.h
> +++ b/include/uapi/linux/raid/md_u.h
> @@ -80,7 +80,7 @@ typedef struct mdu_array_info_s {
> int major_version;
> int minor_version;
> int patch_version;
> - int ctime;
> + unsigned int ctime;
> int level;
> int size;
> int nr_disks;
> @@ -91,7 +91,7 @@ typedef struct mdu_array_info_s {
> /*
> * Generic state information
> */
> - int utime; /* 0 Superblock update time */
> + unsigned int utime; /* 0 Superblock update time */
> int state; /* 1 State bits (clean, ...) */
> int active_disks; /* 2 Number of currently active disks */
> int working_disks; /* 3 Number of working disks */
> --
> 1.9.1
Applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Start reshape failed
From: Kev Dorman @ 2015-12-16 7:47 UTC (permalink / raw)
To: linux-raid
Hi,
Xiao Ni posted back in September about a problem where mdadm --grow fails with:
mdadm: Failed to initiate reshape!
I'm seeing the same problem, with mdadm 3.3.4, Kernel 4.3.0. I can
reproduce this reliably
when adding a device to a 2-device raid0 array. Instead of switching
to raid4 for the
reshape, then switching back to raid0 when done, it ends up in raid4
in recovery mode.
I tried the suggested change of adding a call to md_check_recovery()
in action_store(),
and that does seem to help most of the time.
I was curious if there is a better solution, and/or if this or another
change will show up in
4.3.x soon.
Thanks.
Kev
^ permalink raw reply
* Re: [PATCH v2 0/2] Introduce the bulk IV mode for improving the crypto engine efficiency
From: Milan Broz @ 2015-12-16 8:08 UTC (permalink / raw)
To: Baolin Wang, axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel
In-Reply-To: <cover.1450233399.git.baolin.wang@linaro.org>
On 12/16/2015 04:18 AM, Baolin Wang wrote:
> From the dm-crypt performance report, we found it shows low efficiency
> with crypto engine for some mode (like ecb or xts mode). Because in dm
> crypt, it will map the IO data buffer with scatterlist, and send the
> scatterlist of one bio to the encryption engine, if send more scatterlists
> with bigger size at one time, that helps the engine palys best performance,
> which means a high encryption speed.
>
> But now the dm-crypt only map one segment (always one sector) of one bio
> with one scatterlist to the crypto engine at one time. which is more
> time-consuming and ineffective for the crypto engine. Especially for some
> modes which don't need different IV for each sector, we can map the whole
> bio with multiple scatterlists to improve the engine performance.
>
> But this optimization is not support some ciphers and IV modes which should
> do sector by sector and need different IV for each sector.
>
> Change since v1:
> - Introduce one different IV mode.
> - Change the conditions for bulk mode.
I tried the patchset on 32bit Intel VM and kernel immediately OOPsed (just tried aes-ecb)...
Crash log below.
Milan
[ 40.989759] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 40.990736] IP: [<f8710d26>] crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.990800] *pde = 00000000
[ 40.990844] Oops: 0000 [#1] PREEMPT SMP
[ 40.990961] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 40.991412] CPU: 2 PID: 6 Comm: kworker/u8:0 Not tainted 4.4.0-rc5+ #44
[ 40.991460] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 40.991531] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 40.991587] task: f4c04180 ti: f4c06000 task.ti: f4c06000
[ 40.991629] EIP: 0060:[<f8710d26>] EFLAGS: 00010246 CPU: 2
[ 40.991672] EIP is at crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.991725] EAX: 00001000 EBX: 00001000 ECX: f73e85c0 EDX: 00000000
[ 40.991772] ESI: 00000000 EDI: 00001000 EBP: f4c07e28 ESP: f4c07de8
[ 40.991819] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 40.991862] CR0: 8005003b CR2: 00000000 CR3: 018c1000 CR4: 001406d0
[ 40.991949] Stack:
[ 40.991976] f49967c8 00000000 00000000 00000000 00000000 01000058 00000000 f73e85c0
[ 40.992173] 00000000 f4baf170 00001000 f4ba7290 00000000 f4baf030 f4baf170 f4baf0c0
[ 40.992387] f4c07e60 f8710e8a f4baf170 f4c07e4c f4baf170 f4baf114 f4baf160 f8713fe8
[ 40.992599] Call Trace:
[ 40.992647] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 40.992715] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 40.992781] [<c1056e23>] process_one_work+0x153/0x420
[ 40.992842] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 40.992905] [<c1057127>] worker_thread+0x37/0x470
[ 40.992964] [<c10570f0>] ? process_one_work+0x420/0x420
[ 40.993026] [<c105c326>] kthread+0x96/0xb0
[ 40.993083] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 40.993146] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 40.993207] Code: c2 01 31 f6 85 db 75 d1 89 55 e0 85 ff 0f 85 41 ff ff ff 8b 55 d8 8d 65 f4 89 d0 5b 5e 5f 5d c3 90 8d 74 26 00 8b 55 c8 8b 4d dc <8b> 02 89 4d dc 89 45 c8 c1 e8 1a c1 e0 04 8b 80 80 a2 0b c2 83
[ 40.995405] EIP: [<f8710d26>] crypt_sg_entry+0x186/0x270 [dm_crypt] SS:ESP 0068:f4c07de8
[ 40.995604] CR2: 0000000000000000
[ 40.995703] ---[ end trace d78b89aae913dc1f ]---
[ 40.995825] ------------[ cut here ]------------
[ 40.995930] WARNING: CPU: 2 PID: 6 at kernel/softirq.c:150 __local_bh_enable_ip+0x88/0xd0()
[ 40.996118] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul
[ 40.996352] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 40.996354] IP: [<f8710d26>] crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.996357] *pde = 00000000
[ 40.996359] Oops: 0000 [#2] PREEMPT SMP
[ 40.996361] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 40.996368] CPU: 3 PID: 53 Comm: kworker/u8:1 Tainted: G D 4.4.0-rc5+ #44
[ 40.996369] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 40.996371] Workqueue: kcryptd kcryptd_crypt [dm_crypt]
[ 40.996372] task: f489c0c0 ti: f489e000 task.ti: f489e000
[ 40.996373] EIP: 0060:[<f8710d26>] EFLAGS: 00010246 CPU: 3
[ 40.996375] EIP is at crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.996375] EAX: 00001000 EBX: 00001000 ECX: f71e3a20 EDX: 00000000
[ 40.996376] ESI: 00000000 EDI: 00010000 EBP: f489fe28 ESP: f489fde8
[ 40.996377] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 40.996377] CR0: 8005003b CR2: 00000000 CR3: 33c26000 CR4: 001406d0
[ 40.996410] Stack:
[ 40.996411] f49967c8 00000000 00000000 00000000 00000000 01000058 00000000 f71e3a20
[ 40.996413] 00000000 f4bed370 00001000 f4b2e7c0 00000000 f4bed230 f4bed370 f4bed2c0
[ 40.996415] f489fe60 f8710e8a f4bed370 f489fe4c f4bed370 f4bed314 f4bed360 f8713fe8
[ 40.996418] Call Trace:
[ 40.996421] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 40.996423] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 40.996426] [<c1056e23>] process_one_work+0x153/0x420
[ 40.996428] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 40.996430] [<c1057127>] worker_thread+0x37/0x470
[ 40.996432] [<c10570f0>] ? process_one_work+0x420/0x420
[ 40.996433] [<c105c326>] kthread+0x96/0xb0
[ 40.996436] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 40.996438] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 40.996439] Code: c2 01 31 f6 85 db 75 d1 89 55 e0 85 ff 0f 85 41 ff ff ff 8b 55 d8 8d 65 f4 89 d0 5b 5e 5f 5d c3 90 8d 74 26 00 8b 55 c8 8b 4d dc <8b> 02 89 4d dc 89 45 c8 c1 e8 1a c1 e0 04 8b 80 80 a2 0b c2 83
[ 40.996453] EIP: [<f8710d26>] crypt_sg_entry+0x186/0x270 [dm_crypt] SS:ESP 0068:f489fde8
[ 40.996455] CR2: 0000000000000000
[ 40.996456] ---[ end trace d78b89aae913dc20 ]---
[ 40.996459] ------------[ cut here ]------------
[ 40.996461] WARNING: CPU: 3 PID: 53 at kernel/softirq.c:150 __local_bh_enable_ip+0x88/0xd0()
[ 40.996461] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 40.996465] CPU: 3 PID: 53 Comm: kworker/u8:1 Tainted: G D 4.4.0-rc5+ #44
[ 40.996466] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 40.996469] 00000000 00000000 f489fc64 c12a98f2 00000000 f489fc7c c1042757 c10455a8
[ 40.996471] 00000201 c10bf91e f489c0c0 f489fc8c c10427ff 00000009 00000000 f489fc9c
[ 40.996473] c10455a8 c16d07e0 c16d0660 f489fca8 c14e49ca f489c0c0 f489fcbc c10bf91e
[ 40.996475] Call Trace:
[ 40.996477] [<c12a98f2>] dump_stack+0x4b/0x79
[ 40.996479] [<c1042757>] warn_slowpath_common+0x67/0xa0
[ 40.996480] [<c10455a8>] ? __local_bh_enable_ip+0x88/0xd0
[ 40.996482] [<c10bf91e>] ? cgroup_exit+0x4e/0xc0
[ 40.996484] [<c10427ff>] warn_slowpath_null+0xf/0x20
[ 40.996486] [<c10455a8>] __local_bh_enable_ip+0x88/0xd0
[ 40.996488] [<c14e49ca>] _raw_spin_unlock_bh+0x2a/0x30
[ 40.996490] [<c10bf91e>] cgroup_exit+0x4e/0xc0
[ 40.996491] [<c1043674>] do_exit+0x224/0x920
[ 40.996494] [<c1092065>] ? kmsg_dump+0x105/0x180
[ 40.996496] [<c1004f21>] oops_end+0x61/0x90
[ 40.996498] [<c1038965>] no_context+0xf5/0x210
[ 40.996500] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 40.996501] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 40.996502] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 40.996504] [<c1064022>] ? try_to_wake_up+0x182/0x340
[ 40.996505] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996507] [<c103924b>] do_page_fault+0xb/0x10
[ 40.996508] [<c14e601f>] error_code+0x5f/0x64
[ 40.996509] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996510] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.996511] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 40.996512] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 40.996514] [<c1056e23>] process_one_work+0x153/0x420
[ 40.996515] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 40.996516] [<c1057127>] worker_thread+0x37/0x470
[ 40.996517] [<c10570f0>] ? process_one_work+0x420/0x420
[ 40.996518] [<c105c326>] kthread+0x96/0xb0
[ 40.996519] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 40.996520] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 40.996521] ---[ end trace d78b89aae913dc21 ]---
[ 40.996547] BUG: unable to handle kernel paging request at ffffffc8
[ 40.996548] IP: [<c105c75a>] kthread_data+0xa/0x10
[ 40.996551] *pde = 018c3067 *pte = 00000000
[ 40.996552] Oops: 0000 [#3] PREEMPT SMP
[ 40.996554] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 40.996560] CPU: 3 PID: 53 Comm: kworker/u8:1 Tainted: G D W 4.4.0-rc5+ #44
[ 40.996560] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 40.996564] task: f489c0c0 ti: f489e000 task.ti: f489e000
[ 40.996565] EIP: 0060:[<c105c75a>] EFLAGS: 00010002 CPU: 3
[ 40.996566] EIP is at kthread_data+0xa/0x10
[ 40.996567] EAX: 00000000 EBX: 00000003 ECX: 18a6743d EDX: 00000003
[ 40.996567] ESI: f489c0c0 EDI: c18b6e00 EBP: f489fc80 ESP: f489fc78
[ 40.996568] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 40.996568] CR0: 8005003b CR2: 00000014 CR3: 33c26000 CR4: 001406d0
[ 40.996594] Stack:
[ 40.996595] c1057e5b f489c478 f489fcb0 c14e0083 00000003 f5fb6e10 f5fb6e00 c129283a
[ 40.996597] 00000000 f5fb6e00 f489c0c0 f48a0000 f489f9b8 f489fce0 f489fcbc c14e0612
[ 40.996598] f489c0c0 f489fcf4 c1043a2e f489c3cc f57e8040 00000001 00000002 f489c3cc
[ 40.996600] Call Trace:
[ 40.996602] [<c1057e5b>] ? wq_worker_sleeping+0xb/0x90
[ 40.996603] [<c14e0083>] __schedule+0x6a3/0xad0
[ 40.996605] [<c129283a>] ? put_io_context_active+0xaa/0xd0
[ 40.996607] [<c14e0612>] schedule+0x32/0x80
[ 40.996609] [<c1043a2e>] do_exit+0x5de/0x920
[ 40.996610] [<c1004f21>] oops_end+0x61/0x90
[ 40.996612] [<c1038965>] no_context+0xf5/0x210
[ 40.996614] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 40.996616] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 40.996617] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 40.996619] [<c1064022>] ? try_to_wake_up+0x182/0x340
[ 40.996621] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996622] [<c103924b>] do_page_fault+0xb/0x10
[ 40.996623] [<c14e601f>] error_code+0x5f/0x64
[ 40.996625] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996627] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.996629] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 40.996630] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 40.996631] [<c1056e23>] process_one_work+0x153/0x420
[ 40.996632] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 40.996633] [<c1057127>] worker_thread+0x37/0x470
[ 40.996634] [<c10570f0>] ? process_one_work+0x420/0x420
[ 40.996635] [<c105c326>] kthread+0x96/0xb0
[ 40.996636] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 40.996637] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 40.996638] Code: 00 3b 5f 34 74 0f 89 f8 e8 04 83 48 00 83 c4 78 5b 5e 5f 5d c3 8b 4f 28 eb bf 8d b4 26 00 00 00 00 55 8b 80 64 03 00 00 89 e5 5d <8b> 40 c8 c3 66 90 55 b9 04 00 00 00 89 e5 83 ec 04 8b 90 64 03
[ 40.996652] EIP: [<c105c75a>] kthread_data+0xa/0x10 SS:ESP 0068:f489fc78
[ 40.996654] CR2: 00000000ffffffc8
[ 40.996655] ---[ end trace d78b89aae913dc22 ]---
[ 40.996655] Fixing recursive fault but reboot is needed!
[ 40.996656] BUG: scheduling while atomic: kworker/u8:1/53/0x00000003
[ 40.996657] INFO: lockdep is turned off.
[ 40.996657] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 40.996660] irq event stamp: 11388
[ 40.996661] hardirqs last enabled at (11387): [<c14e4a62>] _raw_spin_unlock_irq+0x22/0x50
[ 40.996662] hardirqs last disabled at (11388): [<c14e489d>] _raw_spin_lock_irq+0xd/0x60
[ 40.996664] softirqs last enabled at (11314): [<c10fcddc>] wb_wakeup_delayed+0x4c/0x60
[ 40.996666] softirqs last disabled at (11310): [<c10fcdb6>] wb_wakeup_delayed+0x26/0x60
[ 40.996667] Preemption disabled at:[<c1004f21>] oops_end+0x61/0x90
[ 40.996668]
[ 40.996669] CPU: 3 PID: 53 Comm: kworker/u8:1 Tainted: G D W 4.4.0-rc5+ #44
[ 40.996670] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 40.996674] 00000000 00000000 f489fb00 c12a98f2 f489c0c0 f489fb0c c106059f f48a0000
[ 40.996677] f489fb3c c14e0168 00000003 00000009 f5fb6e00 f489fb40 00000000 f5fb6e00
[ 40.996680] f489c0c0 f48a0000 00000009 f489c0c0 f489fb48 c14e0612 f489c0c0 f489fb84
[ 40.996683] Call Trace:
[ 40.996685] [<c12a98f2>] dump_stack+0x4b/0x79
[ 40.996687] [<c106059f>] __schedule_bug+0x5f/0xb0
[ 40.996688] [<c14e0168>] __schedule+0x788/0xad0
[ 40.996690] [<c14e0612>] schedule+0x32/0x80
[ 40.996691] [<c1043b7b>] do_exit+0x72b/0x920
[ 40.996692] [<c1092065>] ? kmsg_dump+0x105/0x180
[ 40.996693] [<c1004f21>] oops_end+0x61/0x90
[ 40.996694] [<c1038965>] no_context+0xf5/0x210
[ 40.996695] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 40.996697] [<c106a54a>] ? update_curr+0x9a/0x130
[ 40.996698] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996699] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 40.996700] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 40.996701] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996702] [<c103924b>] do_page_fault+0xb/0x10
[ 40.996703] [<c14e601f>] error_code+0x5f/0x64
[ 40.996704] [<c106007b>] ? ttwu_stat+0x5b/0x200
[ 40.996705] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996706] [<c105c75a>] ? kthread_data+0xa/0x10
[ 40.996707] [<c1057e5b>] ? wq_worker_sleeping+0xb/0x90
[ 40.996708] [<c14e0083>] __schedule+0x6a3/0xad0
[ 40.996709] [<c129283a>] ? put_io_context_active+0xaa/0xd0
[ 40.996710] [<c14e0612>] schedule+0x32/0x80
[ 40.996711] [<c1043a2e>] do_exit+0x5de/0x920
[ 40.996712] [<c1004f21>] oops_end+0x61/0x90
[ 40.996713] [<c1038965>] no_context+0xf5/0x210
[ 40.996715] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 40.996716] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 40.996717] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 40.996718] [<c1064022>] ? try_to_wake_up+0x182/0x340
[ 40.996719] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996720] [<c103924b>] do_page_fault+0xb/0x10
[ 40.996721] [<c14e601f>] error_code+0x5f/0x64
[ 40.996722] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 40.996723] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 40.996724] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 40.996726] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 40.996727] [<c1056e23>] process_one_work+0x153/0x420
[ 40.996728] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 40.996729] [<c1057127>] worker_thread+0x37/0x470
[ 40.996730] [<c10570f0>] ? process_one_work+0x420/0x420
[ 40.996731] [<c105c326>] kthread+0x96/0xb0
[ 40.996733] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 40.996735] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 41.023333] crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 41.023559] CPU: 2 PID: 6 Comm: kworker/u8:0 Tainted: G D W 4.4.0-rc5+ #44
[ 41.023649] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 41.023757] 00000000 00000000 f4c07c64 c12a98f2 00000000 f4c07c7c c1042757 c10455a8
[ 41.023963] 00000201 c10bf91e f4c04180 f4c07c8c c10427ff 00000009 00000000 f4c07c9c
[ 41.024168] c10455a8 c16d07e0 c16d0660 f4c07ca8 c14e49ca f4c04180 f4c07cbc c10bf91e
[ 41.024373] Call Trace:
[ 41.024421] [<c12a98f2>] dump_stack+0x4b/0x79
[ 41.024477] [<c1042757>] warn_slowpath_common+0x67/0xa0
[ 41.024537] [<c10455a8>] ? __local_bh_enable_ip+0x88/0xd0
[ 41.024599] [<c10bf91e>] ? cgroup_exit+0x4e/0xc0
[ 41.024655] [<c10427ff>] warn_slowpath_null+0xf/0x20
[ 41.024714] [<c10455a8>] __local_bh_enable_ip+0x88/0xd0
[ 41.024775] [<c14e49ca>] _raw_spin_unlock_bh+0x2a/0x30
[ 41.024834] [<c10bf91e>] cgroup_exit+0x4e/0xc0
[ 41.024890] [<c1043674>] do_exit+0x224/0x920
[ 41.024945] [<c1092065>] ? kmsg_dump+0x105/0x180
[ 41.025002] [<c1004f21>] oops_end+0x61/0x90
[ 41.025057] [<c1038965>] no_context+0xf5/0x210
[ 41.025113] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 41.025176] [<c10b5f0b>] ? __module_text_address+0xb/0x60
[ 41.025236] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 41.025295] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 41.025353] [<c1004d0e>] ? print_context_stack+0x4e/0xb0
[ 41.025412] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.025471] [<c103924b>] do_page_fault+0xb/0x10
[ 41.025527] [<c14e601f>] error_code+0x5f/0x64
[ 41.025582] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.025642] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 41.025705] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 41.025768] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 41.025831] [<c1056e23>] process_one_work+0x153/0x420
[ 41.026756] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 41.026816] [<c1057127>] worker_thread+0x37/0x470
[ 41.026873] [<c10570f0>] ? process_one_work+0x420/0x420
[ 41.026932] [<c105c326>] kthread+0x96/0xb0
[ 41.026987] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 41.027047] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 41.027106] ---[ end trace d78b89aae913dc23 ]---
[ 41.027229] BUG: unable to handle kernel paging request at ffffffc8
[ 41.027349] IP: [<c105c75a>] kthread_data+0xa/0x10
[ 41.027455] *pde = 018c3067 *pte = 00000000
[ 41.027568] Oops: 0000 [#4] PREEMPT SMP
[ 41.027755] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 41.029586] CPU: 2 PID: 6 Comm: kworker/u8:0 Tainted: G D W 4.4.0-rc5+ #44
[ 41.029826] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 41.030943] task: f4c04180 ti: f4c06000 task.ti: f4c06000
[ 41.031137] EIP: 0060:[<c105c75a>] EFLAGS: 00010002 CPU: 2
[ 41.031341] EIP is at kthread_data+0xa/0x10
[ 41.031536] EAX: 00000000 EBX: 00000002 ECX: 18c2e78b EDX: 00000002
[ 41.031687] ESI: f4c04180 EDI: c18b6e00 EBP: f4c07c80 ESP: f4c07c78
[ 41.031802] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 41.031955] CR0: 8005003b CR2: 00000014 CR3: 018c1000 CR4: 001406d0
[ 41.032207] Stack:
[ 41.032410] c1057e5b f4c04538 f4c07cb0 c14e0083 00000002 f5e65e10 f5e65e00 c129283a
[ 41.032772] 00000000 f5e65e00 f4c04180 f4c08000 f4c079b8 f4c07ce0 f4c07cbc c14e0612
[ 41.035807] f4c04180 f4c07cf4 c1043a2e f4c0448c f57e8040 00000001 00000002 f4c0448c
[ 41.037146] Call Trace:
[ 41.037353] [<c1057e5b>] ? wq_worker_sleeping+0xb/0x90
[ 41.037556] [<c14e0083>] __schedule+0x6a3/0xad0
[ 41.037759] [<c129283a>] ? put_io_context_active+0xaa/0xd0
[ 41.038744] [<c14e0612>] schedule+0x32/0x80
[ 41.038883] [<c1043a2e>] do_exit+0x5de/0x920
[ 41.039067] [<c1004f21>] oops_end+0x61/0x90
[ 41.039263] [<c1038965>] no_context+0xf5/0x210
[ 41.039455] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 41.039655] [<c10b5f0b>] ? __module_text_address+0xb/0x60
[ 41.039850] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 41.040025] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 41.040222] [<c1004d0e>] ? print_context_stack+0x4e/0xb0
[ 41.040373] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.040543] [<c103924b>] do_page_fault+0xb/0x10
[ 41.040765] [<c14e601f>] error_code+0x5f/0x64
[ 41.040984] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.041210] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 41.041435] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 41.041650] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 41.042746] [<c1056e23>] process_one_work+0x153/0x420
[ 41.042928] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 41.043123] [<c1057127>] worker_thread+0x37/0x470
[ 41.043337] [<c10570f0>] ? process_one_work+0x420/0x420
[ 41.043450] [<c105c326>] kthread+0x96/0xb0
[ 41.043529] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 41.043653] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
[ 41.043831] Code: 00 3b 5f 34 74 0f 89 f8 e8 04 83 48 00 83 c4 78 5b 5e 5f 5d c3 8b 4f 28 eb bf 8d b4 26 00 00 00 00 55 8b 80 64 03 00 00 89 e5 5d <8b> 40 c8 c3 66 90 55 b9 04 00 00 00 89 e5 83 ec 04 8b 90 64 03
[ 41.055486] EIP: [<c105c75a>] kthread_data+0xa/0x10 SS:ESP 0068:f4c07c78
[ 41.055692] CR2: 00000000ffffffc8
[ 41.055803] ---[ end trace d78b89aae913dc24 ]---
[ 41.055917] Fixing recursive fault but reboot is needed!
[ 41.056038] BUG: scheduling while atomic: kworker/u8:0/6/0x00000003
[ 41.056162] INFO: lockdep is turned off.
[ 41.056353] Modules linked in: dm_crypt loop rpcsec_gss_krb5 dm_mod crc32_pclmul crc32c_intel ata_piix aesni_intel aes_i586 lrw ablk_helper cryptd
[ 41.056783] irq event stamp: 17002
[ 41.056833] hardirqs last enabled at (17001): [<c108d686>] __raw_spin_lock_init+0x16/0x50
[ 41.056944] hardirqs last disabled at (17002): [<c14e601b>] error_code+0x5b/0x64
[ 41.057103] softirqs last enabled at (16962): [<c10bf868>] cgroup_post_fork+0x68/0xd0
[ 41.057335] softirqs last disabled at (16960): [<c10bf822>] cgroup_post_fork+0x22/0xd0
[ 41.057522] Preemption disabled at:[<c1004f21>] oops_end+0x61/0x90
[ 41.057719]
[ 41.057819] CPU: 2 PID: 6 Comm: kworker/u8:0 Tainted: G D W 4.4.0-rc5+ #44
[ 41.058887] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
[ 41.059066] 00000000 00000000 f4c07b00 c12a98f2 f4c04180 f4c07b0c c106059f f4c08000
[ 41.059774] f4c07b3c c14e0168 00000002 00000009 f5e65e00 f4c07b40 00000000 f5e65e00
[ 41.060519] f4c04180 f4c08000 00000009 f4c04180 f4c07b48 c14e0612 f4c04180 f4c07b84
[ 41.060728] Call Trace:
[ 41.060776] [<c12a98f2>] dump_stack+0x4b/0x79
[ 41.060833] [<c106059f>] __schedule_bug+0x5f/0xb0
[ 41.060891] [<c14e0168>] __schedule+0x788/0xad0
[ 41.060979] [<c14e0612>] schedule+0x32/0x80
[ 41.061067] [<c1043b7b>] do_exit+0x72b/0x920
[ 41.061148] [<c1092065>] ? kmsg_dump+0x105/0x180
[ 41.061250] [<c1004f21>] oops_end+0x61/0x90
[ 41.061306] [<c1038965>] no_context+0xf5/0x210
[ 41.061362] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 41.061425] [<c106a54a>] ? update_curr+0x9a/0x130
[ 41.061482] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.061542] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 41.061602] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 41.061661] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.061720] [<c103924b>] do_page_fault+0xb/0x10
[ 41.061789] [<c14e601f>] error_code+0x5f/0x64
[ 41.062866] [<c106007b>] ? ttwu_stat+0x5b/0x200
[ 41.063053] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.063247] [<c105c75a>] ? kthread_data+0xa/0x10
[ 41.063434] [<c1057e5b>] ? wq_worker_sleeping+0xb/0x90
[ 41.063625] [<c14e0083>] __schedule+0x6a3/0xad0
[ 41.063818] [<c129283a>] ? put_io_context_active+0xaa/0xd0
[ 41.064043] [<c14e0612>] schedule+0x32/0x80
[ 41.064762] [<c1043a2e>] do_exit+0x5de/0x920
[ 41.064944] [<c1004f21>] oops_end+0x61/0x90
[ 41.065133] [<c1038965>] no_context+0xf5/0x210
[ 41.065313] [<c1038b1c>] __bad_area_nosemaphore+0x9c/0x150
[ 41.065505] [<c10b5f0b>] ? __module_text_address+0xb/0x60
[ 41.065692] [<c1038bdd>] bad_area_nosemaphore+0xd/0x10
[ 41.066731] [<c1038e0f>] __do_page_fault+0x6f/0x4a0
[ 41.066848] [<c1004d0e>] ? print_context_stack+0x4e/0xb0
[ 41.067012] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.067231] [<c103924b>] do_page_fault+0xb/0x10
[ 41.067500] [<c14e601f>] error_code+0x5f/0x64
[ 41.067689] [<c1039240>] ? __do_page_fault+0x4a0/0x4a0
[ 41.067881] [<f8710d26>] ? crypt_sg_entry+0x186/0x270 [dm_crypt]
[ 41.068069] [<f8710e8a>] crypt_convert_io+0x7a/0x360 [dm_crypt]
[ 41.068259] [<f87131e5>] kcryptd_crypt+0x395/0x3da [dm_crypt]
[ 41.068417] [<c1056e23>] process_one_work+0x153/0x420
[ 41.068534] [<c1056dd0>] ? process_one_work+0x100/0x420
[ 41.068709] [<c1057127>] worker_thread+0x37/0x470
[ 41.068917] [<c10570f0>] ? process_one_work+0x420/0x420
[ 41.069118] [<c105c326>] kthread+0x96/0xb0
[ 41.069307] [<c14e5389>] ret_from_kernel_thread+0x21/0x38
[ 41.069434] [<c105c290>] ? kthread_worker_fn+0xf0/0xf0
^ permalink raw reply
* Re: [PATCH v2 0/2] Introduce the bulk IV mode for improving the crypto engine efficiency
From: Baolin Wang @ 2015-12-16 8:31 UTC (permalink / raw)
To: Milan Broz
Cc: Jens Axboe, Alasdair G Kergon, Mike Snitzer, dm-devel, neilb,
dan.j.williams, martin.petersen, sagig, Kent Overstreet,
keith.busch, tj, Mark Brown, Arnd Bergmann, linux-block,
linux-raid, LKML
In-Reply-To: <56711C0F.8030105@gmail.com>
On 16 December 2015 at 16:08, Milan Broz <gmazyland@gmail.com> wrote:
> On 12/16/2015 04:18 AM, Baolin Wang wrote:
>> From the dm-crypt performance report, we found it shows low efficiency
>> with crypto engine for some mode (like ecb or xts mode). Because in dm
>> crypt, it will map the IO data buffer with scatterlist, and send the
>> scatterlist of one bio to the encryption engine, if send more scatterlists
>> with bigger size at one time, that helps the engine palys best performance,
>> which means a high encryption speed.
>>
>> But now the dm-crypt only map one segment (always one sector) of one bio
>> with one scatterlist to the crypto engine at one time. which is more
>> time-consuming and ineffective for the crypto engine. Especially for some
>> modes which don't need different IV for each sector, we can map the whole
>> bio with multiple scatterlists to improve the engine performance.
>>
>> But this optimization is not support some ciphers and IV modes which should
>> do sector by sector and need different IV for each sector.
>>
>> Change since v1:
>> - Introduce one different IV mode.
>> - Change the conditions for bulk mode.
>
> I tried the patchset on 32bit Intel VM and kernel immediately OOPsed (just tried aes-ecb)...
>
I'm sorry for that. I'll check why it will crash though it can work
well on my beaglebone black board. Thanks.
> Crash log below.
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-16 15:31 UTC (permalink / raw)
To: John Stoffel; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZUQNBPNXFs69JRU0Q82TQ4RjAgpsc7voMgEzuhSZhmDjig@mail.gmail.com>
Phil, the 16k chunk size has really given a boost to my RAID 5
sequential write performance measured with fio, bs=1408k.
This is what I was getting with a 128k chunk size:
iodepth=4 => 605 MB/s
iodepth=8 => 589 MB/s
iodepth=16 => 634 MB/s
iodepth=32 => 635 MB/s
But this is what I'm getting with a 16k chunk size:
iodepth=4 => 825 MB/s
iodepth=8 => 810 MB/s
iodepth=16 => 851 MB/s
iodepth=32 => 866 MB/s
^ permalink raw reply
* [PATCH 1/2] mdadm: do not try to hold dlm lock in free_super1
From: Guoqing Jiang @ 2015-12-16 17:54 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
Since free_super1 actually doesn't change the sb, it
just free the addr space of sb. Also free_super1 is
called in lots of place within mdadm, so remove dlm
lock code since the func doesn't need the protection
and also reduce latency.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
super1.c | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/super1.c b/super1.c
index 7a1156d..38362e4 100644
--- a/super1.c
+++ b/super1.c
@@ -2421,15 +2421,6 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
static void free_super1(struct supertype *st)
{
- int rv, lockid;
- if (is_clustered(st)) {
- rv = cluster_get_dlmlock(st, &lockid);
- if (rv) {
- pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
- cluster_release_dlmlock(st, lockid);
- return;
- }
- }
if (st->sb)
free(st->sb);
@@ -2441,8 +2432,6 @@ static void free_super1(struct supertype *st)
free(di);
}
st->sb = NULL;
- if (is_clustered(st))
- cluster_release_dlmlock(st, lockid);
}
#ifndef MDASSEMBLE
--
2.1.4
^ permalink raw reply related
* [PATCH 2/2] mdadm: improve the safeguard for change cluster raid's sb
From: Guoqing Jiang @ 2015-12-16 17:54 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1450288466-21168-1-git-send-email-gqjiang@suse.com>
This commit does the following jobs:
1. rename is_clustered to dlm_funs_ready since it match the
function better.
2. st->cluster_name can't be use to identify the raid is a
clustered or not, we should check the bitmap's version to
perform the identification.
3. for cluster_get_dlmlock/cluster_release_dlmlock funcs, both
of them just need the lockid as parameter since the cluster
name can get by get_cluster_name().
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
mdadm.h | 6 +++---
super1.c | 33 ++++++++++++++++++---------------
util.c | 30 ++++++++++++++++++------------
3 files changed, 39 insertions(+), 30 deletions(-)
diff --git a/mdadm.h b/mdadm.h
index aad0fa8..be5dc7f 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -1484,9 +1484,9 @@ struct dlm_hooks {
};
extern int get_cluster_name(char **name);
-extern int is_clustered(struct supertype *st);
-extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
-extern int cluster_release_dlmlock(struct supertype *st, int lockid);
+extern int dlm_funs_ready(void);
+extern int cluster_get_dlmlock(int *lockid);
+extern int cluster_release_dlmlock(int lockid);
extern void set_dlm_hooks(void);
#define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
diff --git a/super1.c b/super1.c
index 38362e4..c7ff634 100644
--- a/super1.c
+++ b/super1.c
@@ -1100,12 +1100,13 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
int rv = 0;
int lockid;
struct mdp_superblock_1 *sb = st->sb;
+ bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb) + MAX_SB_SIZE);
- if (is_clustered(st)) {
- rv = cluster_get_dlmlock(st, &lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready()) {
+ rv = cluster_get_dlmlock(&lockid);
if (rv) {
pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
- cluster_release_dlmlock(st, lockid);
+ cluster_release_dlmlock(lockid);
return rv;
}
}
@@ -1368,8 +1369,8 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
rv = -1;
sb->sb_csum = calc_sb_1_csum(sb);
- if (is_clustered(st))
- cluster_release_dlmlock(st, lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready())
+ cluster_release_dlmlock(lockid);
return rv;
}
@@ -1474,13 +1475,14 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
struct mdp_superblock_1 *sb = st->sb;
__u16 *rp = sb->dev_roles + dk->number;
struct devinfo *di, **dip;
+ bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb) + MAX_SB_SIZE);
int rv, lockid;
- if (is_clustered(st)) {
- rv = cluster_get_dlmlock(st, &lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready()) {
+ rv = cluster_get_dlmlock(&lockid);
if (rv) {
pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
- cluster_release_dlmlock(st, lockid);
+ cluster_release_dlmlock(lockid);
return rv;
}
}
@@ -1513,8 +1515,8 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
di->next = NULL;
*dip = di;
- if (is_clustered(st))
- cluster_release_dlmlock(st, lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready())
+ cluster_release_dlmlock(lockid);
return 0;
}
@@ -1529,13 +1531,14 @@ static int store_super1(struct supertype *st, int fd)
struct align_fd afd;
int sbsize;
unsigned long long dsize;
+ bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb) + MAX_SB_SIZE);
int rv, lockid;
- if (is_clustered(st)) {
- rv = cluster_get_dlmlock(st, &lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready()) {
+ rv = cluster_get_dlmlock(&lockid);
if (rv) {
pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
- cluster_release_dlmlock(st, lockid);
+ cluster_release_dlmlock(lockid);
return rv;
}
}
@@ -1599,8 +1602,8 @@ static int store_super1(struct supertype *st, int fd)
}
}
fsync(fd);
- if (is_clustered(st))
- cluster_release_dlmlock(st, lockid);
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && dlm_funs_ready())
+ cluster_release_dlmlock(lockid);
return 0;
}
diff --git a/util.c b/util.c
index 8217e11..f1b0b95 100644
--- a/util.c
+++ b/util.c
@@ -92,13 +92,9 @@ struct dlm_lock_resource {
struct dlm_lksb lksb;
};
-int is_clustered(struct supertype *st)
+int dlm_funs_ready(void)
{
- /* is it a cluster md or not */
- if (is_dlm_hooks_ready && st->cluster_name)
- return 1;
- else
- return 0;
+ return is_dlm_hooks_ready ? 1 : 0;
}
/* Using poll(2) to wait for and dispatch ASTs */
@@ -128,17 +124,24 @@ static void dlm_ast(void *arg)
ast_called = 1;
}
+static char *cluster_name = NULL;
/* Create the lockspace, take bitmapXXX locks on all the bitmaps. */
-int cluster_get_dlmlock(struct supertype *st, int *lockid)
+int cluster_get_dlmlock(int *lockid)
{
int ret = -1;
char str[64];
int flags = LKF_NOQUEUE;
+ ret = get_cluster_name(&cluster_name);
+ if (ret) {
+ pr_err("The md can't get cluster name\n");
+ return -1;
+ }
+
dlm_lock_res = xmalloc(sizeof(struct dlm_lock_resource));
- dlm_lock_res->ls = dlm_hooks->create_lockspace(st->cluster_name, O_RDWR);
+ dlm_lock_res->ls = dlm_hooks->create_lockspace(cluster_name, O_RDWR);
if (!dlm_lock_res->ls) {
- pr_err("%s failed to create lockspace\n", st->cluster_name);
+ pr_err("%s failed to create lockspace\n", cluster_name);
goto out;
}
@@ -146,7 +149,7 @@ int cluster_get_dlmlock(struct supertype *st, int *lockid)
if (flags & LKF_CONVERT)
dlm_lock_res->lksb.sb_lkid = *lockid;
- snprintf(str, 64, "bitmap%04d", st->nodes);
+ snprintf(str, 64, "bitmap%s", cluster_name);
/* if flags with LKF_CONVERT causes below return ENOENT which means
* "No such file or directory" */
ret = dlm_hooks->ls_lock(dlm_lock_res->ls, LKM_PWMODE, &dlm_lock_res->lksb,
@@ -171,10 +174,13 @@ out:
return ret;
}
-int cluster_release_dlmlock(struct supertype *st, int lockid)
+int cluster_release_dlmlock(int lockid)
{
int ret = -1;
+ if (!cluster_name)
+ return -1;
+
/* if flags with LKF_CONVERT causes below return EINVAL which means
* "Invalid argument" */
ret = dlm_hooks->ls_unlock(dlm_lock_res->ls, lockid, 0,
@@ -195,7 +201,7 @@ int cluster_release_dlmlock(struct supertype *st, int lockid)
goto out;
}
- ret = dlm_hooks->release_lockspace(st->cluster_name, dlm_lock_res->ls, 1);
+ ret = dlm_hooks->release_lockspace(cluster_name, dlm_lock_res->ls, 1);
if (ret) {
pr_err("error %d happened when release lockspace\n", errno);
/* XXX make sure the lockspace is released eventually */
--
2.1.4
^ permalink raw reply related
* Re: [PATCH 00/17] mdadm: SCSI enclosure based array
From: Dan Williams @ 2015-12-16 18:54 UTC (permalink / raw)
To: NeilBrown; +Cc: Song Liu, linux-raid, Shaohua Li
In-Reply-To: <87d1u7b188.fsf@notabene.neil.brown.name>
On Tue, Dec 15, 2015 at 6:01 PM, NeilBrown <neilb@suse.com> wrote:
> On Wed, Dec 02 2015, Song Liu wrote:
>
>> These patches by Dan Williams enable creating RAID array based on
>> SCSI enclosures.
>
> Is there any background reading available so I can have some idea what
> these patches are supposed to do.
"Blink the light when the disk dies", is the terse summary.
> I have a vague idea what an "enclosure" is but I didn't know that Linux
> knew about them. Is there a link to the LWN article about them or some
> other useful documentation? "git grep -i enclosur Documentation/"
> didn't show anything likely to be useful.
The interface these patches use is presented by the "ses" driver
(drivers/scsi/ses.c) which leverages the sysfs interface presented by
the "enclosure" driver (drivers/misc/enclosure.c). I don't recall
their being much in the way of documentation.
For each disk in an enclosure the driver exposes an enclosure object
with the following attributes:
fault
status
active
locate
power_status
type
slot
Where 'fault', 'active', and 'locate' select different led signal
patterns. The 'power_status' attribute allows power to be turned off
to the slot. The 'slot' attribute is meant to match the slot
identifier that is on the board's silk screen or some physical label a
technician can observe.
> And what is "/dev/disk/by-slot"?? I've never come across it before
> and google doesn't help much. Just these patches and
> http://comments.gmane.org/gmane.linux.scsi/93809
/dev/disk/by-slot are new proposed symlinks for udev. The new udev
rules for this are in patch1.
When the scsi core is probing devices and finds an enclosure target
device it associates enclosure slot objects with the scsi_target.
With that association we can create a symlink for the resulting disk
based on its physical slot identifier.
At the time mdadm was chosen to house these rules for convenience, but
I think they should be submitted to upstream udev.
^ permalink raw reply
* [PATCH 1/2] MD: change journal disk role to disk 0
From: Shaohua Li @ 2015-12-16 21:13 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, neilb
Neil pointed out setting journal disk role to raid_disks will confuse
reshape if we support reshape eventually. Switching the role to 0 (we
should be fine as long as the value >=0) and skip sysfs file creation to
avoid error.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 2 +-
drivers/md/md.h | 8 ++++++--
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 807095f..874c843 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1652,7 +1652,7 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
rdev->journal_tail = le64_to_cpu(sb->journal_tail);
if (mddev->recovery_cp == MaxSector)
set_bit(MD_JOURNAL_CLEAN, &mddev->flags);
- rdev->raid_disk = mddev->raid_disks;
+ rdev->raid_disk = 0;
break;
default:
rdev->saved_raid_disk = role;
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 2bea51e..ca0b643 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -566,7 +566,9 @@ static inline char * mdname (struct mddev * mddev)
static inline int sysfs_link_rdev(struct mddev *mddev, struct md_rdev *rdev)
{
char nm[20];
- if (!test_bit(Replacement, &rdev->flags) && mddev->kobj.sd) {
+ if (!test_bit(Replacement, &rdev->flags) &&
+ !test_bit(Journal, &rdev->flags) &&
+ mddev->kobj.sd) {
sprintf(nm, "rd%d", rdev->raid_disk);
return sysfs_create_link(&mddev->kobj, &rdev->kobj, nm);
} else
@@ -576,7 +578,9 @@ static inline int sysfs_link_rdev(struct mddev *mddev, struct md_rdev *rdev)
static inline void sysfs_unlink_rdev(struct mddev *mddev, struct md_rdev *rdev)
{
char nm[20];
- if (!test_bit(Replacement, &rdev->flags) && mddev->kobj.sd) {
+ if (!test_bit(Replacement, &rdev->flags) &&
+ !test_bit(Journal, &rdev->flags) &&
+ mddev->kobj.sd) {
sprintf(nm, "rd%d", rdev->raid_disk);
sysfs_remove_link(&mddev->kobj, nm);
}
--
2.4.6
^ permalink raw reply related
* [PATCH 2/2] raid5-cache: add journal hot add/remove support
From: Shaohua Li @ 2015-12-16 21:13 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, neilb
In-Reply-To: <2f750625c6c016754fabfab98b84e6f2719bb0e5.1450300354.git.shli@fb.com>
Add support for journal disk hot add/remove. Mostly trival checks in md
part. The raid5 part is a little tricky. For hot-remove, we can't wait
pending write as it's called from raid5d. The wait will cause deadlock.
We simplily fail the hot-remove. A hot-remove retry can success
eventually since if journal disk is faulty all pending write will be
failed and finish. For hot-add, since an array supporting journal but
without journal disk will be marked read-only, we are safe to hot add
journal without stopping IO (should be read IO, while journal only
handles write IO).
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 42 ++++++++++++++++++++++++++++++------------
drivers/md/raid5-cache.c | 16 ++++++++++++----
drivers/md/raid5.c | 34 ++++++++++++++++++++++++++--------
3 files changed, 68 insertions(+), 24 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 874c843..7e0a7c6 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2052,8 +2052,9 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
return -EEXIST;
/* make sure rdev->sectors exceeds mddev->dev_sectors */
- if (rdev->sectors && (mddev->dev_sectors == 0 ||
- rdev->sectors < mddev->dev_sectors)) {
+ if (!test_bit(Journal, &rdev->flags) &&
+ rdev->sectors &&
+ (mddev->dev_sectors == 0 || rdev->sectors < mddev->dev_sectors)) {
if (mddev->pers) {
/* Cannot change size, so fail
* If mddev->level <= 0, then we don't care
@@ -2084,7 +2085,8 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
}
}
rcu_read_unlock();
- if (mddev->max_disks && rdev->desc_nr >= mddev->max_disks) {
+ if (!test_bit(Journal, &rdev->flags) &&
+ mddev->max_disks && rdev->desc_nr >= mddev->max_disks) {
printk(KERN_WARNING "md: %s: array is limited to %d devices\n",
mdname(mddev), mddev->max_disks);
return -EBUSY;
@@ -6031,8 +6033,23 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
else
clear_bit(WriteMostly, &rdev->flags);
- if (info->state & (1<<MD_DISK_JOURNAL))
+ if (info->state & (1<<MD_DISK_JOURNAL)) {
+ struct md_rdev *rdev2;
+ bool has_journal = false;
+
+ /* make sure no existing journal disk */
+ rdev_for_each(rdev2, mddev) {
+ if (test_bit(Journal, &rdev2->flags)) {
+ has_journal = true;
+ break;
+ }
+ }
+ if (has_journal) {
+ export_rdev(rdev);
+ return -EBUSY;
+ }
set_bit(Journal, &rdev->flags);
+ }
/*
* check whether the device shows up in other nodes
*/
@@ -8162,19 +8179,20 @@ static int remove_and_add_spares(struct mddev *mddev,
continue;
if (test_bit(Faulty, &rdev->flags))
continue;
- if (test_bit(Journal, &rdev->flags))
- continue;
- if (mddev->ro &&
- ! (rdev->saved_raid_disk >= 0 &&
- !test_bit(Bitmap_sync, &rdev->flags)))
- continue;
+ if (!test_bit(Journal, &rdev->flags)) {
+ if (mddev->ro &&
+ ! (rdev->saved_raid_disk >= 0 &&
+ !test_bit(Bitmap_sync, &rdev->flags)))
+ continue;
- rdev->recovery_offset = 0;
+ rdev->recovery_offset = 0;
+ }
if (mddev->pers->
hot_add_disk(mddev, rdev) == 0) {
if (sysfs_link_rdev(mddev, rdev))
/* failure here is OK */;
- spares++;
+ if (!test_bit(Journal, &rdev->flags))
+ spares++;
md_new_event(mddev);
set_bit(MD_CHANGE_DEVS, &mddev->flags);
}
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index b887e04..9cd32c2 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -806,10 +806,18 @@ void r5l_quiesce(struct r5l_log *log, int state)
bool r5l_log_disk_error(struct r5conf *conf)
{
+ struct r5l_log *log;
+ bool ret;
/* don't allow write if journal disk is missing */
- if (!conf->log)
- return test_bit(MD_HAS_JOURNAL, &conf->mddev->flags);
- return test_bit(Faulty, &conf->log->rdev->flags);
+ rcu_read_lock();
+ log = rcu_dereference(conf->log);
+
+ if (!log)
+ ret = test_bit(MD_HAS_JOURNAL, &conf->mddev->flags);
+ else
+ ret = test_bit(Faulty, &log->rdev->flags);
+ rcu_read_unlock();
+ return ret;
}
struct r5l_recovery_ctx {
@@ -1172,7 +1180,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (r5l_load_log(log))
goto error;
- conf->log = log;
+ rcu_assign_pointer(conf->log, log);
return 0;
error:
md_unregister_thread(&log->reclaim_thread);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 704ef7f..783376b 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7141,14 +7141,19 @@ static int raid5_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
struct disk_info *p = conf->disks + number;
print_raid5_conf(conf);
- if (test_bit(Journal, &rdev->flags)) {
+ if (test_bit(Journal, &rdev->flags) && conf->log) {
+ struct r5l_log *log;
/*
- * journal disk is not removable, but we need give a chance to
- * update superblock of other disks. Otherwise journal disk
- * will be considered as 'fresh'
+ * we can't wait pending write here, as this is called in
+ * raid5d, wait will deadlock.
*/
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- return -EINVAL;
+ if (atomic_read(&mddev->writes_pending))
+ return -EBUSY;
+ log = conf->log;
+ conf->log = NULL;
+ synchronize_rcu();
+ r5l_exit_log(log);
+ return 0;
}
if (rdev == p->rdev)
rdevp = &p->rdev;
@@ -7212,8 +7217,21 @@ static int raid5_add_disk(struct mddev *mddev, struct md_rdev *rdev)
int first = 0;
int last = conf->raid_disks - 1;
- if (test_bit(Journal, &rdev->flags))
- return -EINVAL;
+ if (test_bit(Journal, &rdev->flags)) {
+ char b[BDEVNAME_SIZE];
+ if (conf->log)
+ return -EBUSY;
+
+ rdev->raid_disk = 0;
+ /*
+ * The array is in readonly mode if journal is missing, so no
+ * write requests running. We should be safe
+ */
+ r5l_init_log(conf, rdev);
+ printk(KERN_INFO"md/raid:%s: using device %s as journal\n",
+ mdname(mddev), bdevname(rdev->bdev, b));
+ return 0;
+ }
if (mddev->recovery_disabled == conf->recovery_disabled)
return -EBUSY;
--
2.4.6
^ permalink raw reply related
* Re: [PATCH 00/17] mdadm: SCSI enclosure based array
From: NeilBrown @ 2015-12-16 21:35 UTC (permalink / raw)
To: Dan Williams; +Cc: Song Liu, linux-raid, Shaohua Li
In-Reply-To: <CAPcyv4g47ZUZfCEeLRtwZkXudArRVRyFLar4RrDg1iM=kjK6iQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]
On Thu, Dec 17 2015, Dan Williams wrote:
> On Tue, Dec 15, 2015 at 6:01 PM, NeilBrown <neilb@suse.com> wrote:
>> On Wed, Dec 02 2015, Song Liu wrote:
>>
>>> These patches by Dan Williams enable creating RAID array based on
>>> SCSI enclosures.
>>
>> Is there any background reading available so I can have some idea what
>> these patches are supposed to do.
>
> "Blink the light when the disk dies", is the terse summary.
>
>> I have a vague idea what an "enclosure" is but I didn't know that Linux
>> knew about them. Is there a link to the LWN article about them or some
>> other useful documentation? "git grep -i enclosur Documentation/"
>> didn't show anything likely to be useful.
>
> The interface these patches use is presented by the "ses" driver
> (drivers/scsi/ses.c) which leverages the sysfs interface presented by
> the "enclosure" driver (drivers/misc/enclosure.c). I don't recall
> their being much in the way of documentation.
>
> For each disk in an enclosure the driver exposes an enclosure object
> with the following attributes:
> fault
> status
> active
> locate
> power_status
> type
> slot
>
> Where 'fault', 'active', and 'locate' select different led signal
> patterns. The 'power_status' attribute allows power to be turned off
> to the slot. The 'slot' attribute is meant to match the slot
> identifier that is on the board's silk screen or some physical label a
> technician can observe.
cat > Documentation/scsi/ses.txt ???
Thanks. That is useful background.
>
>> And what is "/dev/disk/by-slot"?? I've never come across it before
>> and google doesn't help much. Just these patches and
>> http://comments.gmane.org/gmane.linux.scsi/93809
>
> /dev/disk/by-slot are new proposed symlinks for udev. The new udev
> rules for this are in patch1.
If this is proposed, then there must be a proposal - right?
Written?? (I doubt it).
So "by-path" is about the data-path that leads to the device, but
"by-slot" is about the physical location of the device (which may
correlate with the path, but is conceptually different). I guess I can
accept that.
I don't suppose "by-location" was considered?
Can a "disk" have other locations that "slots"?
I guess 'sd' cards live in slots.
But USB sticks plug into sockets (or are sockets just slots?)
CDROMs live in trays.
Is there some convention for naming slots?
enclosure-rank-row-column ??
Is there some convention for naming enclosures? Bob, Jane, Ahmed?
>
> When the scsi core is probing devices and finds an enclosure target
> device it associates enclosure slot objects with the scsi_target.
> With that association we can create a symlink for the resulting disk
> based on its physical slot identifier.
>
> At the time mdadm was chosen to house these rules for convenience, but
> I think they should be submitted to upstream udev.
Yes, I would definitely like to see this "by-location" ;-) proposal
accepted by upstream udev before committing to it in mdadm.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* RE: [PATCH 00/17] mdadm: SCSI enclosure based array
From: Song Liu @ 2015-12-16 21:49 UTC (permalink / raw)
To: NeilBrown, Dan Williams; +Cc: linux-raid, Shaohua Li
In-Reply-To: <87h9ji9ivl.fsf@notabene.neil.brown.name>
> So "by-path" is about the data-path that leads to the device, but "by-slot" is
> about the physical location of the device (which may correlate with the path,
> but is conceptually different). I guess I can accept that.
> I don't suppose "by-location" was considered?
> Can a "disk" have other locations that "slots"?
> I guess 'sd' cards live in slots.
> But USB sticks plug into sockets (or are sockets just slots?) CDROMs live in trays.
>
> Is there some convention for naming slots?
> enclosure-rank-row-column ??
> Is there some convention for naming enclosures? Bob, Jane, Ahmed?
The enclosure:slots name comes from SCSI Enclosure Service (SES) definitions,
and it is just enclosure-slot (no rank, row, column).
Here, we use the SAS address of the SCSI Enclosure Processor (SEP) to identify
the enclosure, while the slot is just numbers. For example:
enclosure-0x570e2840203080ff-slot0 -> /dev/sdl
enclosure-0x570e2840203080ff-slot1 -> /dev/sdj
enclosure-0x570e2840203080ff-slot10 -> /dev/sdk
At this time, this concept only applies to drives in SCSI enclosure. Other drives
do not have this symlink.
Thanks,
Song
^ permalink raw reply
* Re: [PATCH 2/2] raid5-cache: add journal hot add/remove support
From: NeilBrown @ 2015-12-16 22:50 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: Kernel-team, songliubraving, hch
In-Reply-To: <2b10b1d8988431c43f013fecc28008fb4585c888.1450300354.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 750 bytes --]
On Thu, Dec 17 2015, Shaohua Li wrote:
> Add support for journal disk hot add/remove. Mostly trival checks in md
> part. The raid5 part is a little tricky. For hot-remove, we can't wait
> pending write as it's called from raid5d. The wait will cause deadlock.
> We simplily fail the hot-remove. A hot-remove retry can success
> eventually since if journal disk is faulty all pending write will be
> failed and finish. For hot-add, since an array supporting journal but
> without journal disk will be marked read-only, we are safe to hot add
> journal without stopping IO (should be read IO, while journal only
> handles write IO).
This and previous patch both applied - thanks.
(will push things out to for-next next later today I hope)
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 2/2] mdadm: improve the safeguard for change cluster raid's sb
From: NeilBrown @ 2015-12-16 22:53 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1450288466-21168-2-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 598 bytes --]
On Thu, Dec 17 2015, Guoqing Jiang wrote:
> This commit does the following jobs:
>
> 1. rename is_clustered to dlm_funs_ready since it match the
> function better.
> 2. st->cluster_name can't be use to identify the raid is a
> clustered or not, we should check the bitmap's version to
> perform the identification.
> 3. for cluster_get_dlmlock/cluster_release_dlmlock funcs, both
> of them just need the lockid as parameter since the cluster
> name can get by get_cluster_name().
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
This and other patch applied, thanks.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-17 5:57 UTC (permalink / raw)
To: Mark Knecht; +Cc: John Stoffel, Phil Turmel, Linux-RAID
In-Reply-To: <CAK2H+eeD2k4yzuvL4uF_qKycp6A=XPe8pVF_J-7Agi8Ze89PPQ@mail.gmail.com>
On Wed, Dec 16, 2015 at 6:24 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Wed, Dec 16, 2015 at 7:31 AM, Dallas Clement <dallas.a.clement@gmail.com>
> wrote:
>>
>> Phil, the 16k chunk size has really given a boost to my RAID 5
>> sequential write performance measured with fio, bs=1408k.
>>
>> This is what I was getting with a 128k chunk size:
>>
>> iodepth=4 => 605 MB/s
>> iodepth=8 => 589 MB/s
>> iodepth=16 => 634 MB/s
>> iodepth=32 => 635 MB/s
>>
>> But this is what I'm getting with a 16k chunk size:
>>
>> iodepth=4 => 825 MB/s
>> iodepth=8 => 810 MB/s
>> iodepth=16 => 851 MB/s
>> iodepth=32 => 866 MB/s
>
>
> Dallas,
> Hi. Just for kicks I tried Phil's idea (I think it was Phil) and sampled
> stripe_cache_active
> by putting this command in a 1 second loop and running it today while I
> worked.
>
> cat /sys/block/md3/md/stripe_cache_active >> testCacheResults
>
> My workload is _very_ different from what you're working on. This is a
> high-end desktop
> machine (Intel 980i Extreme processor, 24GB DRAM, RAID6) running 2 Windows 7
> VMs
> while I watch the stock market and program in MatLab. None the less I was
> somewhat
> surprise at the spread in the number of active lines. The test ran for about
> 10 hours with
> about 94% of the results being 0, but numbers ranging from 1 line to 2098
> lines active
> at a single time. Also interesting to me was when that 2098 value hit it was
> apparently
> all clear in less than 1 second as the values immediately following where
> back to 0.
>
> Note that this is 5 disk RAID6 set up with a chunk size of 16k and an
> internal
> intent bitmap. I did no work like you're doing when I set the machine up. I
> just picked
> some numbers and built it so that I could get to work.
>
> I've not done any real speed testing but a quick run of dd suggested
> maybe
> 160MB/S-180MB/S which sounds about right to me.
>
> Anyway, just thought it was interesting.
>
> - Mark
>
> mark@c2RAID6 ~ $ sort -g testCacheResults | uniq -c
> 33316 0
> 127 1
> 98 2
> 105 3
> 141 4
> 71 5
> 48 6
> 38 7
> 39 8
> 36 9
> 31 10
> 23 11
> 30 12
> 26 13
> 17 14
> 12 15
> 20 16
> 14 17
> 17 18
> 23 19
> 19 20
> 12 21
> 13 22
> 14 23
> 16 24
> 15 25
> 14 26
> 8 27
> 11 28
> 16 29
> 10 30
> 3 31
> 9 32
> 3 33
> 5 34
> 13 35
> 7 36
> 7 37
> 3 38
> 7 39
> 6 40
> 9 41
> 5 42
> 6 43
> 7 44
> 12 45
> 7 46
> 7 47
> 6 48
> 6 49
> 5 50
> 4 51
> 8 52
> 2 53
> 6 54
> 10 55
> 3 56
> 7 57
> 7 58
> 9 59
> 3 60
> 5 61
> 8 62
> 1 63
> 5 64
> 4 65
> 9 66
> 3 67
> 3 68
> 2 69
> 2 70
> 5 71
> 2 72
> 3 73
> 3 74
> 3 75
> 3 76
> 3 77
> 1 78
> 4 79
> 1 80
> 3 81
> 2 82
> 1 83
> 4 84
> 1 85
> 4 86
> 1 87
> 2 89
> 2 90
> 1 91
> 2 92
> 1 93
> 4 94
> 2 95
> 5 96
> 2 97
> 2 98
> 2 99
> 5 100
> 2 101
> 1 102
> 6 103
> 5 104
> 1 105
> 3 106
> 3 107
> 2 108
> 3 109
> 3 110
> 4 111
> 3 112
> 1 113
> 4 114
> 1 115
> 1 116
> 1 117
> 3 118
> 4 119
> 3 120
> 3 121
> 2 122
> 3 123
> 4 124
> 2 125
> 3 126
> 1 127
> 2 128
> 2 129
> 1 130
> 3 131
> 2 132
> 2 133
> 2 134
> 3 135
> 1 136
> 2 137
> 3 138
> 5 140
> 3 141
> 3 142
> 1 143
> 1 144
> 5 145
> 1 146
> 6 147
> 3 148
> 1 149
> 1 150
> 1 152
> 2 153
> 1 154
> 1 155
> 1 156
> 4 157
> 3 158
> 1 159
> 3 160
> 1 161
> 6 162
> 1 163
> 2 164
> 1 165
> 1 166
> 4 167
> 2 168
> 5 169
> 2 170
> 3 172
> 5 173
> 4 174
> 4 175
> 4 176
> 3 177
> 2 178
> 2 179
> 6 180
> 2 181
> 3 182
> 3 184
> 2 185
> 3 186
> 4 187
> 2 188
> 5 190
> 4 192
> 3 193
> 2 194
> 6 196
> 1 197
> 1 198
> 1 199
> 2 200
> 4 201
> 2 203
> 2 204
> 4 206
> 1 207
> 2 208
> 5 209
> 2 210
> 3 211
> 6 212
> 3 213
> 3 214
> 4 215
> 4 216
> 6 217
> 8 218
> 1 219
> 5 220
> 6 221
> 4 222
> 6 223
> 6 224
> 5 225
> 2 226
> 3 227
> 5 228
> 2 229
> 1 230
> 5 231
> 6 232
> 6 233
> 3 234
> 4 235
> 6 236
> 5 237
> 1 238
> 5 239
> 2 240
> 5 241
> 4 242
> 2 244
> 2 245
> 2 246
> 2 247
> 3 248
> 2 249
> 4 250
> 3 251
> 6 252
> 2 253
> 2 254
> 5 255
> 3 256
> 4 257
> 3 258
> 3 259
> 6 260
> 2 261
> 3 262
> 3 263
> 1 264
> 3 265
> 1 266
> 4 267
> 4 268
> 4 269
> 3 270
> 4 271
> 2 272
> 1 273
> 1 275
> 1 276
> 5 277
> 6 278
> 2 279
> 2 280
> 1 281
> 6 282
> 5 283
> 8 284
> 1 285
> 5 286
> 4 287
> 2 288
> 2 289
> 3 290
> 2 291
> 1 292
> 2 293
> 1 294
> 3 295
> 2 296
> 2 297
> 1 298
> 3 299
> 2 300
> 1 301
> 2 303
> 3 305
> 3 306
> 1 307
> 1 308
> 2 309
> 2 310
> 1 311
> 1 312
> 1 313
> 2 314
> 1 315
> 1 317
> 1 318
> 2 320
> 1 321
> 2 322
> 2 323
> 2 324
> 1 325
> 1 326
> 2 327
> 3 328
> 2 329
> 1 331
> 1 335
> 1 336
> 2 337
> 1 338
> 1 339
> 1 340
> 3 341
> 1 343
> 1 344
> 1 346
> 1 347
> 1 348
> 2 349
> 1 350
> 1 352
> 2 353
> 1 357
> 1 359
> 1 360
> 1 365
> 1 368
> 1 369
> 2 372
> 2 373
> 1 378
> 1 380
> 1 388
> 2 392
> 1 409
> 1 410
> 1 414
> 1 425
> 1 444
> 1 455
> 2 460
> 1 465
> 1 469
> 1 484
> 1 485
> 1 492
> 1 499
> 1 503
> 1 504
> 1 509
> 1 518
> 1 534
> 1 540
> 1 541
> 1 543
> 1 546
> 1 572
> 1 575
> 1 586
> 1 591
> 1 592
> 1 602
> 1 637
> 1 661
> 1 674
> 1 732
> 1 770
> 1 780
> 1 905
> 2 927
> 1 928
> 1 1036
> 1 1146
> 1 1151
> 1 1157
> 1 1314
> 1 1974
> 1 2098
> mark@c2RAID6 ~ $
Hi Mark. This is quite fascinating. Now I really want to try it with
my workloads. How big is your stripe cache btw?
^ permalink raw reply
* [PATCH v3 0/2] Introduce the bulk IV mode for improving the crypto engine efficiency
From: Baolin Wang @ 2015-12-17 7:32 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
From the dm-crypt performance report, we found it shows low efficiency
with crypto engine for some mode (like ecb or xts mode). Because in dm
crypt, it will map the IO data buffer with scatterlist, and send the
scatterlist of one bio to the encryption engine, if send more scatterlists
with bigger size at one time, that helps the engine palys best performance,
which means a high encryption speed.
But now the dm-crypt only map one segment (always one sector) of one bio
with one scatterlist to the crypto engine at one time. which is more
time-consuming and ineffective for the crypto engine. Especially for some
modes which don't need different IV for each sector, we can map the whole
bio with multiple scatterlists to improve the engine performance.
But this optimization is not support some ciphers and IV modes which should
do sector by sector and need different IV for each sector.
Change since v2:
- Introduce blk_bio_map_sg() function to map one bio.
- Do some modifications for checking sg entry.
Baolin Wang (2):
block: Introduce blk_bio_map_sg() to map one bio
md: dm-crypt: Introduce the bulk IV mode for bulk crypto
block/blk-merge.c | 45 +++++++
drivers/md/dm-crypt.c | 333 +++++++++++++++++++++++++++++++++++++++++++++++-
include/linux/blkdev.h | 3 +
3 files changed, 375 insertions(+), 6 deletions(-)
--
1.7.9.5
^ permalink raw reply
* [PATCH v3 1/2] block: Introduce blk_bio_map_sg() to map one bio
From: Baolin Wang @ 2015-12-17 7:32 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
In-Reply-To: <cover.1450336551.git.baolin.wang@linaro.org>
In dm-crypt, it need to map one bio to scatterlist for improving the
encryption efficiency. Thus this patch introduces the blk_bio_map_sg()
function to map one bio with scatterlists.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
block/blk-merge.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
include/linux/blkdev.h | 3 +++
2 files changed, 48 insertions(+)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index de5716d8..281b9e5 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -374,6 +374,51 @@ single_segment:
}
/*
+ * map a bio to scatterlist, return number of sg entries setup.
+ */
+int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg)
+{
+ struct bio_vec bvec, bvprv = { NULL };
+ struct bvec_iter iter;
+ int nsegs, cluster;
+
+ nsegs = 0;
+ cluster = blk_queue_cluster(q);
+
+ if (bio->bi_rw & REQ_DISCARD) {
+ /*
+ * This is a hack - drivers should be neither modifying the
+ * biovec, nor relying on bi_vcnt - but because of
+ * blk_add_request_payload(), a discard bio may or may not have
+ * a payload we need to set up here (thank you Christoph) and
+ * bi_vcnt is really the only way of telling if we need to.
+ */
+
+ if (bio->bi_vcnt)
+ goto single_segment;
+
+ return 0;
+ }
+
+ if (bio->bi_rw & REQ_WRITE_SAME) {
+single_segment:
+ *sg = sglist;
+ bvec = bio_iovec(bio);
+ sg_set_page(*sg, bvec.bv_page, bvec.bv_len, bvec.bv_offset);
+ return 1;
+ }
+
+ bio_for_each_segment(bvec, bio, iter)
+ __blk_segment_map_sg(q, &bvec, sglist, &bvprv, sg,
+ &nsegs, &cluster);
+
+ return nsegs;
+}
+EXPORT_SYMBOL(blk_bio_map_sg);
+
+/*
* map a request to scatterlist, return number of sg entries setup. Caller
* must make sure sg can hold rq->nr_phys_segments entries
*/
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 3fe27f8..3ca90ac 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1004,6 +1004,9 @@ extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable);
extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev);
extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *);
+extern int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg);
extern void blk_dump_rq_flags(struct request *, char *);
extern long nr_blockdev_pages(void);
--
1.7.9.5
^ permalink raw reply related
* [PATCH v3 2/2] md: dm-crypt: Introduce the bulk IV mode for bulk crypto
From: Baolin Wang @ 2015-12-17 7:32 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
In-Reply-To: <cover.1450336551.git.baolin.wang@linaro.org>
In now dm-crypt code, it is ineffective to map one segment (always one
sector) of one bio with just only one scatterlist at one time for hardware
crypto engine. Especially for some encryption mode (like ecb or xts mode)
cooperating with the crypto engine, they just need one initial IV or null
IV instead of different IV for each sector. In this situation We can consider
to use multiple scatterlists to map the whole bio and send all scatterlists
of one bio to crypto engine to encrypt or decrypt, which can improve the
hardware engine's efficiency.
With this optimization, On my test setup (beaglebone black board) using 64KB
I/Os on an eMMC storage device I saw about 60% improvement in throughput for
encrypted writes, and about 100% improvement for encrypted reads. But this
is not fit for other modes which need different IV for each sector.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/md/dm-crypt.c | 333 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 327 insertions(+), 6 deletions(-)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 917d47e..003d2e9 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -32,6 +32,7 @@
#include <linux/device-mapper.h>
#define DM_MSG_PREFIX "crypt"
+#define DM_MAX_SG_LIST 1024
/*
* context holding the current state of a multi-part conversion
@@ -68,6 +69,8 @@ struct dm_crypt_request {
struct convert_context *ctx;
struct scatterlist sg_in;
struct scatterlist sg_out;
+ struct sg_table sgt_in;
+ struct sg_table sgt_out;
sector_t iv_sector;
};
@@ -140,6 +143,7 @@ struct crypt_config {
char *cipher;
char *cipher_string;
+ int bulk_crypto;
struct crypt_iv_operations *iv_gen_ops;
union {
struct iv_essiv_private essiv;
@@ -238,6 +242,9 @@ static struct crypto_ablkcipher *any_tfm(struct crypt_config *cc)
*
* plumb: unimplemented, see:
* http://article.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/454
+ *
+ * bulk: the initial vector is the 64-bit little-endian version of the sector
+ * number, which is used as just one initial IV for one bulk data.
*/
static int crypt_iv_plain_gen(struct crypt_config *cc, u8 *iv,
@@ -755,6 +762,15 @@ static int crypt_iv_tcw_post(struct crypt_config *cc, u8 *iv,
return r;
}
+static int crypt_iv_bulk_gen(struct crypt_config *cc, u8 *iv,
+ struct dm_crypt_request *dmreq)
+{
+ memset(iv, 0, cc->iv_size);
+ *(__le64 *)iv = cpu_to_le64(dmreq->iv_sector);
+
+ return 0;
+}
+
static struct crypt_iv_operations crypt_iv_plain_ops = {
.generator = crypt_iv_plain_gen
};
@@ -799,6 +815,10 @@ static struct crypt_iv_operations crypt_iv_tcw_ops = {
.post = crypt_iv_tcw_post
};
+static struct crypt_iv_operations crypt_iv_bulk_ops = {
+ .generator = crypt_iv_bulk_gen
+};
+
static void crypt_convert_init(struct crypt_config *cc,
struct convert_context *ctx,
struct bio *bio_out, struct bio *bio_in,
@@ -833,6 +853,11 @@ static u8 *iv_of_dmreq(struct crypt_config *cc,
crypto_ablkcipher_alignmask(any_tfm(cc)) + 1);
}
+static int crypt_is_bulk_mode(struct crypt_config *cc)
+{
+ return cc->bulk_crypto;
+}
+
static int crypt_convert_block(struct crypt_config *cc,
struct convert_context *ctx,
struct ablkcipher_request *req)
@@ -881,24 +906,40 @@ static int crypt_convert_block(struct crypt_config *cc,
static void kcryptd_async_done(struct crypto_async_request *async_req,
int error);
+static void kcryptd_async_all_done(struct crypto_async_request *async_req,
+ int error);
static void crypt_alloc_req(struct crypt_config *cc,
struct convert_context *ctx)
{
unsigned key_index = ctx->cc_sector & (cc->tfms_count - 1);
+ struct dm_crypt_request *dmreq;
if (!ctx->req)
ctx->req = mempool_alloc(cc->req_pool, GFP_NOIO);
+ dmreq = dmreq_of_req(cc, ctx->req);
+ dmreq->sgt_in.orig_nents = 0;
+ dmreq->sgt_out.orig_nents = 0;
+
ablkcipher_request_set_tfm(ctx->req, cc->tfms[key_index]);
/*
* Use REQ_MAY_BACKLOG so a cipher driver internally backlogs
* requests if driver request queue is full.
*/
- ablkcipher_request_set_callback(ctx->req,
- CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
- kcryptd_async_done, dmreq_of_req(cc, ctx->req));
+ if (crypt_is_bulk_mode(cc))
+ ablkcipher_request_set_callback(ctx->req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG
+ | CRYPTO_TFM_REQ_MAY_SLEEP,
+ kcryptd_async_all_done,
+ dmreq_of_req(cc, ctx->req));
+ else
+ ablkcipher_request_set_callback(ctx->req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG
+ | CRYPTO_TFM_REQ_MAY_SLEEP,
+ kcryptd_async_done,
+ dmreq_of_req(cc, ctx->req));
}
static void crypt_free_req(struct crypt_config *cc,
@@ -911,6 +952,221 @@ static void crypt_free_req(struct crypt_config *cc,
}
/*
+ * Check how many sg entry numbers are needed when map one bio
+ * with scatterlists in advance.
+ */
+static unsigned int crypt_sg_entry(struct bio *bio_t)
+{
+ struct request_queue *q = bdev_get_queue(bio_t->bi_bdev);
+ int cluster = blk_queue_cluster(q);
+ struct bio_vec bvec, bvprv = { NULL };
+ struct bvec_iter biter;
+ unsigned long nbytes = 0, sg_length = 0;
+ unsigned int sg_cnt = 0, first_bvec = 0;
+
+ if (bio_t->bi_rw & REQ_DISCARD) {
+ if (bio_t->bi_vcnt)
+ return 1;
+ return 0;
+ }
+
+ if (bio_t->bi_rw & REQ_WRITE_SAME)
+ return 1;
+
+ bio_for_each_segment(bvec, bio_t, biter) {
+ nbytes = bvec.bv_len;
+
+ if (!cluster) {
+ sg_cnt++;
+ continue;
+ }
+
+ if (!first_bvec) {
+ first_bvec = 1;
+ goto new_segment;
+ }
+
+ if (sg_length + nbytes > queue_max_segment_size(q))
+ goto new_segment;
+
+ if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bvec))
+ goto new_segment;
+
+ if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bvec))
+ goto new_segment;
+
+ sg_length += nbytes;
+ continue;
+
+new_segment:
+ memcpy(&bvprv, &bvec, sizeof(struct bio_vec));
+ sg_length = nbytes;
+ sg_cnt++;
+ }
+
+ return sg_cnt;
+}
+
+static int crypt_convert_all_blocks(struct crypt_config *cc,
+ struct convert_context *ctx,
+ struct ablkcipher_request *req)
+{
+ struct dm_crypt_io *io =
+ container_of(ctx, struct dm_crypt_io, ctx);
+ struct dm_crypt_request *dmreq = dmreq_of_req(cc, req);
+ u8 *iv = iv_of_dmreq(cc, dmreq);
+ struct bio *orig_bio = io->base_bio;
+ struct bio *bio_in = ctx->bio_in;
+ struct bio *bio_out = ctx->bio_out;
+ unsigned int total_bytes = orig_bio->bi_iter.bi_size;
+ struct scatterlist *sg_in = NULL;
+ struct scatterlist *sg_out = NULL;
+ struct scatterlist *sg = NULL;
+ unsigned int total_sg_len_in = 0;
+ unsigned int total_sg_len_out = 0;
+ unsigned int sg_in_max = 0, sg_out_max = 0;
+ int ret;
+
+ dmreq->iv_sector = ctx->cc_sector;
+ dmreq->ctx = ctx;
+
+ /*
+ * Need to calculate how many sg entry need to be used
+ * for this bio.
+ */
+ sg_in_max = crypt_sg_entry(bio_in) + 1;
+ if (sg_in_max > DM_MAX_SG_LIST || sg_in_max <= 0) {
+ DMERR("%s sg entry too large or none %d\n",
+ __func__, sg_in_max);
+ return -EINVAL;
+ } else if (sg_in_max == 2) {
+ sg_in = &dmreq->sg_in;
+ }
+
+ if (!sg_in) {
+ ret = sg_alloc_table(&dmreq->sgt_in, sg_in_max, GFP_KERNEL);
+ if (ret) {
+ DMERR("%s sg in allocation failed\n", __func__);
+ return -ENOMEM;
+ }
+
+ sg_in = dmreq->sgt_in.sgl;
+ }
+
+ total_sg_len_in = blk_bio_map_sg(bdev_get_queue(bio_in->bi_bdev),
+ bio_in, sg_in, &sg);
+ if ((total_sg_len_in <= 0)
+ || (total_sg_len_in > sg_in_max)) {
+ DMERR("%s in sg map error %d, sg_in_max[%d]\n",
+ __func__, total_sg_len_in, sg_in_max);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_in.bi_size -= total_bytes;
+
+ if (bio_data_dir(orig_bio) == READ)
+ goto set_crypt;
+
+ sg_out_max = crypt_sg_entry(bio_out) + 1;
+ if (sg_out_max > DM_MAX_SG_LIST || sg_out_max <= 0) {
+ DMERR("%s sg entry too large or none %d\n",
+ __func__, sg_out_max);
+ return -EINVAL;
+ } else if (sg_out_max == 2) {
+ sg_out = &dmreq->sg_out;
+ }
+
+ if (!sg_out) {
+ ret = sg_alloc_table(&dmreq->sgt_out, sg_out_max, GFP_KERNEL);
+ if (ret) {
+ DMERR("%s sg out allocation failed\n", __func__);
+ return -ENOMEM;
+ }
+
+ sg_out = dmreq->sgt_out.sgl;
+ }
+
+ sg = NULL;
+ total_sg_len_out = blk_bio_map_sg(bdev_get_queue(bio_out->bi_bdev),
+ bio_out, sg_out, &sg);
+ if ((total_sg_len_out <= 0) ||
+ (total_sg_len_out > sg_out_max)) {
+ DMERR("%s out sg map error %d, sg_out_max[%d]\n",
+ __func__, total_sg_len_out, sg_out_max);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_out.bi_size -= total_bytes;
+set_crypt:
+ if (cc->iv_gen_ops) {
+ ret = cc->iv_gen_ops->generator(cc, iv, dmreq);
+ if (ret < 0) {
+ DMERR("%s generator iv error %d\n", __func__, ret);
+ return ret;
+ }
+ }
+
+ if (bio_data_dir(orig_bio) == WRITE) {
+ ablkcipher_request_set_crypt(req, sg_in,
+ sg_out, total_bytes, iv);
+
+ ret = crypto_ablkcipher_encrypt(req);
+ } else {
+ ablkcipher_request_set_crypt(req, sg_in,
+ sg_in, total_bytes, iv);
+
+ ret = crypto_ablkcipher_decrypt(req);
+ }
+
+ if (!ret && cc->iv_gen_ops && cc->iv_gen_ops->post)
+ ret = cc->iv_gen_ops->post(cc, iv, dmreq);
+
+ return ret;
+}
+
+/*
+ * Encrypt / decrypt data from one whole bio at one time.
+ */
+static int crypt_convert_io(struct crypt_config *cc,
+ struct convert_context *ctx)
+{
+ int r;
+
+ atomic_set(&ctx->cc_pending, 1);
+ crypt_alloc_req(cc, ctx);
+ atomic_inc(&ctx->cc_pending);
+
+ r = crypt_convert_all_blocks(cc, ctx, ctx->req);
+ switch (r) {
+ case -EBUSY:
+ /*
+ * Lets make this synchronous bio by waiting on
+ * in progress as well.
+ */
+ case -EINPROGRESS:
+ wait_for_completion(&ctx->restart);
+ ctx->req = NULL;
+ break;
+ case 0:
+ atomic_dec(&ctx->cc_pending);
+ cond_resched();
+ break;
+ /* There was an error while processing the request. */
+ default:
+ atomic_dec(&ctx->cc_pending);
+ return r;
+ }
+
+ return 0;
+}
+
+/*
* Encrypt / decrypt data from one bio to another one (can be the same one)
*/
static int crypt_convert(struct crypt_config *cc,
@@ -1070,12 +1326,18 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
struct crypt_config *cc = io->cc;
struct bio *base_bio = io->base_bio;
int error = io->error;
+ struct dm_crypt_request *dmreq;
if (!atomic_dec_and_test(&io->io_pending))
return;
- if (io->ctx.req)
+ if (io->ctx.req) {
+ dmreq = dmreq_of_req(cc, io->ctx.req);
+ sg_free_table(&dmreq->sgt_out);
+ sg_free_table(&dmreq->sgt_in);
+
crypt_free_req(cc, io->ctx.req, base_bio);
+ }
base_bio->bi_error = error;
bio_endio(base_bio);
@@ -1312,7 +1574,11 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
sector += bio_sectors(clone);
crypt_inc_pending(io);
- r = crypt_convert(cc, &io->ctx);
+ if (crypt_is_bulk_mode(cc))
+ r = crypt_convert_io(cc, &io->ctx);
+ else
+ r = crypt_convert(cc, &io->ctx);
+
if (r)
io->error = -EIO;
crypt_finished = atomic_dec_and_test(&io->ctx.cc_pending);
@@ -1342,7 +1608,11 @@ static void kcryptd_crypt_read_convert(struct dm_crypt_io *io)
crypt_convert_init(cc, &io->ctx, io->base_bio, io->base_bio,
io->sector);
- r = crypt_convert(cc, &io->ctx);
+ if (crypt_is_bulk_mode(cc))
+ r = crypt_convert_io(cc, &io->ctx);
+ else
+ r = crypt_convert(cc, &io->ctx);
+
if (r < 0)
io->error = -EIO;
@@ -1387,6 +1657,40 @@ static void kcryptd_async_done(struct crypto_async_request *async_req,
kcryptd_crypt_write_io_submit(io, 1);
}
+static void kcryptd_async_all_done(struct crypto_async_request *async_req,
+ int error)
+{
+ struct dm_crypt_request *dmreq = async_req->data;
+ struct convert_context *ctx = dmreq->ctx;
+ struct dm_crypt_io *io = container_of(ctx, struct dm_crypt_io, ctx);
+ struct crypt_config *cc = io->cc;
+
+ if (error == -EINPROGRESS)
+ return;
+
+ if (!error && cc->iv_gen_ops && cc->iv_gen_ops->post)
+ error = cc->iv_gen_ops->post(cc, iv_of_dmreq(cc, dmreq), dmreq);
+
+ if (error < 0)
+ io->error = error;
+
+ sg_free_table(&dmreq->sgt_out);
+ sg_free_table(&dmreq->sgt_in);
+
+ crypt_free_req(cc, req_of_dmreq(cc, dmreq), io->base_bio);
+
+ if (!atomic_dec_and_test(&ctx->cc_pending)) {
+ complete(&io->ctx.restart);
+ return;
+ }
+
+ complete(&io->ctx.restart);
+ if (bio_data_dir(io->base_bio) == READ)
+ kcryptd_crypt_read_done(io);
+ else
+ kcryptd_crypt_write_io_submit(io, 1);
+}
+
static void kcryptd_crypt(struct work_struct *work)
{
struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
@@ -1633,6 +1937,21 @@ static int crypt_ctr_cipher(struct dm_target *ti,
goto bad_mem;
}
+ /*
+ * Here we need to check if it can be encrypted or decrypted with
+ * bulk block, which means these encryption modes don't need IV or
+ * just need one initial IV. For bulk mode, we can expand the
+ * scatterlist entries to map the bio, then send all the scatterlists
+ * to the hardware engine at one time to improve the crypto engine
+ * efficiency. But it does not fit for other IV modes, it has to do
+ * encryption and decryption sector by sector because every sector
+ * has different IV.
+ */
+ if (!ivmode || !strcmp(ivmode, "bulk") || !strcmp(ivmode, "null"))
+ cc->bulk_crypto = 1;
+ else
+ cc->bulk_crypto = 0;
+
/* Allocate cipher */
ret = crypt_alloc_tfms(cc, cipher_api);
if (ret < 0) {
@@ -1680,6 +1999,8 @@ static int crypt_ctr_cipher(struct dm_target *ti,
cc->iv_gen_ops = &crypt_iv_tcw_ops;
cc->key_parts += 2; /* IV + whitening */
cc->key_extra_size = cc->iv_size + TCW_WHITENING_SIZE;
+ } else if (strcmp(ivmode, "bulk") == 0) {
+ cc->iv_gen_ops = &crypt_iv_bulk_ops;
} else {
ret = -EINVAL;
ti->error = "Invalid IV mode";
--
1.7.9.5
^ permalink raw reply related
* Re: [PATCH v2 0/2] Introduce the bulk IV mode for improving the crypto engine efficiency
From: Baolin Wang @ 2015-12-17 7:37 UTC (permalink / raw)
To: Milan Broz
Cc: Jens Axboe, Alasdair G Kergon, Mike Snitzer, dm-devel, neilb,
dan.j.williams, martin.petersen, sagig, Kent Overstreet,
keith.busch, tj, Mark Brown, Arnd Bergmann, linux-block,
linux-raid, LKML
In-Reply-To: <56711C0F.8030105@gmail.com>
Hi Milan,
On 16 December 2015 at 16:08, Milan Broz <gmazyland@gmail.com> wrote:
> On 12/16/2015 04:18 AM, Baolin Wang wrote:
>> From the dm-crypt performance report, we found it shows low efficiency
>> with crypto engine for some mode (like ecb or xts mode). Because in dm
>> crypt, it will map the IO data buffer with scatterlist, and send the
>> scatterlist of one bio to the encryption engine, if send more scatterlists
>> with bigger size at one time, that helps the engine palys best performance,
>> which means a high encryption speed.
>>
>> But now the dm-crypt only map one segment (always one sector) of one bio
>> with one scatterlist to the crypto engine at one time. which is more
>> time-consuming and ineffective for the crypto engine. Especially for some
>> modes which don't need different IV for each sector, we can map the whole
>> bio with multiple scatterlists to improve the engine performance.
>>
>> But this optimization is not support some ciphers and IV modes which should
>> do sector by sector and need different IV for each sector.
>>
>> Change since v1:
>> - Introduce one different IV mode.
>> - Change the conditions for bulk mode.
>
> I tried the patchset on 32bit Intel VM and kernel immediately OOPsed (just tried aes-ecb)...
>
I've checked the code and I guess some macros I used with different
definitions on different arch. Could you please try the new patchset
with some optimization on your platform? It can work well on my arm
board. Thanks.
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-17 13:41 UTC (permalink / raw)
To: Mark Knecht, Dallas Clement; +Cc: John Stoffel, Linux-RAID
In-Reply-To: <CAK2H+eeD2k4yzuvL4uF_qKycp6A=XPe8pVF_J-7Agi8Ze89PPQ@mail.gmail.com>
Hi Mark,
On 12/16/2015 07:24 PM, Mark Knecht wrote:
> On Wed, Dec 16, 2015 at 7:31 AM, Dallas Clement
> <dallas.a.clement@gmail.com <mailto:dallas.a.clement@gmail.com>> wrote:
>
> Phil, the 16k chunk size has really given a boost to my RAID 5
> sequential write performance measured with fio, bs=1408k.
>
> This is what I was getting with a 128k chunk size:
>
> iodepth=4 => 605 MB/s
> iodepth=8 => 589 MB/s
> iodepth=16 => 634 MB/s
> iodepth=32 => 635 MB/s
>
> But this is what I'm getting with a 16k chunk size:
>
> iodepth=4 => 825 MB/s
> iodepth=8 => 810 MB/s
> iodepth=16 => 851 MB/s
> iodepth=32 => 866 MB/s
Very interesting. Good to see hypotheses supported by results.
> Dallas,
> Hi. Just for kicks I tried Phil's idea (I think it was Phil) and
:-)
> sampled stripe_cache_active
> by putting this command in a 1 second loop and running it today while I
> worked.
>
> cat /sys/block/md3/md/stripe_cache_active >> testCacheResults
>
> My workload is _very_ different from what you're working on. This is a
> high-end desktop
> machine (Intel 980i Extreme processor, 24GB DRAM, RAID6) running 2
> Windows 7 VMs
> while I watch the stock market and program in MatLab. None the less I
> was somewhat
> surprise at the spread in the number of active lines. The test ran for
> about 10 hours with
> about 94% of the results being 0, but numbers ranging from 1 line to
> 2098 lines active
> at a single time. Also interesting to me was when that 2098 value hit it
> was apparently
> all clear in less than 1 second as the values immediately following
> where back to 0.
Yeah, latencies are pretty low. One-second samples will be fairly
random snapshots under most conditions. Consider sampling much faster,
but building one-minute histograms and recording those.
Phil
^ permalink raw reply
* [PATCH] Fix remove_and_add_spares removes drive added as spare in slot_store
From: rgoldwyn @ 2015-12-17 13:45 UTC (permalink / raw)
To: linux-raid, neilb; +Cc: pawel.baldysiak, Goldwyn Rodrigues
From: Goldwyn Rodrigues <rgoldwyn@suse.com>
Commit 2910ff17d154baa5eb50e362a91104e831eb2bb6
introduced a regression which would remove a recently added spare via
slot_store. Revert part of the patch which touches slot_store() and add
the disk directly using pers->hot_add_disk()
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Pawel Baldysiak <pawel.baldysiak@intel.com>
---
drivers/md/md.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index a71b36f..0444afa 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2726,6 +2726,7 @@ slot_store(struct md_rdev *rdev, const char *buf, size_t len)
/* Activating a spare .. or possibly reactivating
* if we ever get bitmaps working here.
*/
+ int err;
if (rdev->raid_disk != -1)
return -EBUSY;
@@ -2747,9 +2748,15 @@ slot_store(struct md_rdev *rdev, const char *buf, size_t len)
rdev->saved_raid_disk = -1;
clear_bit(In_sync, &rdev->flags);
clear_bit(Bitmap_sync, &rdev->flags);
- remove_and_add_spares(rdev->mddev, rdev);
- if (rdev->raid_disk == -1)
- return -EBUSY;
+ err = rdev->mddev->pers->
+ hot_add_disk(rdev->mddev, rdev);
+ if (err) {
+ rdev->raid_disk = -1;
+ return err;
+ } else
+ sysfs_notify_dirent_safe(rdev->sysfs_state);
+ if (sysfs_link_rdev(rdev->mddev, rdev))
+ /* failure here is OK */;
/* don't wakeup anyone, leave that to userspace. */
} else {
if (slot >= rdev->mddev->raid_disks &&
--
2.6.2
^ permalink raw reply related
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-17 21:08 UTC (permalink / raw)
To: Phil Turmel; +Cc: Mark Knecht, John Stoffel, Linux-RAID
In-Reply-To: <5672BB7A.4050808@turmel.org>
I am still in the process of collecting a bunch of performance data.
But so far, it is shocking to see the throughput difference when
blocks written are stripe aligned. However, in the non-ideal world it
is not always possible to ensure that clients are writing blocks of
data which are stripe aligned. If the goal is to reduce the # of RMWs
it seems like writing big blocks would also help for sequential
workloads where large quantities of data are being written. Can any
of you think of anything else that can be tuned in the kernel to
reduce # of RMWs in the case where blocks are not stripe aligned? Is
it a bad idea to mess with the timing of the stripe cache?
^ permalink raw reply
* raid5-cache: avoid GFP_NOFAIL allocation
From: Christoph Hellwig @ 2015-12-17 22:09 UTC (permalink / raw)
To: neilb, shli; +Cc: linux-raid
bio_sets, mempools and high level retries are our friends:
^ permalink raw reply
* [PATCH 1/3] raid5-cache: use a bio_set
From: Christoph Hellwig @ 2015-12-17 22:09 UTC (permalink / raw)
To: neilb, shli; +Cc: linux-raid
In-Reply-To: <1450390197-19479-1-git-send-email-hch@lst.de>
This allows us to make guaranteed forward progress.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 668e973..0a64e97 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -34,6 +34,12 @@
#define RECLAIM_MAX_FREE_SPACE (10 * 1024 * 1024 * 2) /* sector */
#define RECLAIM_MAX_FREE_SPACE_SHIFT (2)
+/*
+ * We only need 2 bios per I/O unit to make progress, but ensure we
+ * have a few more available to not get too tight.
+ */
+#define R5L_POOL_SIZE 4
+
struct r5l_log {
struct md_rdev *rdev;
@@ -70,6 +76,7 @@ struct r5l_log {
struct bio flush_bio;
struct kmem_cache *io_kc;
+ struct bio_set *bs;
struct md_thread *reclaim_thread;
unsigned long reclaim_target; /* number of space that need to be
@@ -248,7 +255,7 @@ static void r5l_submit_current_io(struct r5l_log *log)
static struct bio *r5l_bio_alloc(struct r5l_log *log)
{
- struct bio *bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
+ struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_PAGES, log->bs);
bio->bi_rw = WRITE;
bio->bi_bdev = log->rdev->bdev;
@@ -1153,6 +1160,10 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (!log->io_kc)
goto io_kc;
+ log->bs = bioset_create(R5L_POOL_SIZE, 0);
+ if (!log->bs)
+ goto io_bs;
+
log->reclaim_thread = md_register_thread(r5l_reclaim_thread,
log->rdev->mddev, "reclaim");
if (!log->reclaim_thread)
@@ -1170,6 +1181,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
error:
md_unregister_thread(&log->reclaim_thread);
reclaim_thread:
+ bioset_free(log->bs);
+io_bs:
kmem_cache_destroy(log->io_kc);
io_kc:
kfree(log);
@@ -1179,6 +1192,7 @@ io_kc:
void r5l_exit_log(struct r5l_log *log)
{
md_unregister_thread(&log->reclaim_thread);
+ bioset_free(log->bs);
kmem_cache_destroy(log->io_kc);
kfree(log);
}
--
1.9.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox