* You have received fax, document 00000961727
From: Interfax @ 2015-11-14 10:26 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 339 bytes --]
You have a new fax!
Please, download fax document attached to this email.
Fax name: scanned-00000961727.doc
Sender: Victor Bryant
Scanned: Sat, 14 Nov 2015 11:20:16 +0300
Scan duration: 5 seconds
Filesize: 228 Kb
Pages: 12
Resolution: 100 DPI
Thank you for using Interfax!
[-- Attachment #2: scanned-00000961727.zip --]
[-- Type: application/zip, Size: 1643 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Mark Brown @ 2015-11-13 11:51 UTC (permalink / raw)
To: Jens Axboe
Cc: Baolin Wang, Mike Snitzer, Alasdair G Kergon, dm-devel, neilb,
linux-raid, jack, Arnd Bergmann, LKML, keith.busch, jmoyer, tj,
bart.vanassche, Garg, Dinesh
In-Reply-To: <5644AFA2.6040201@kernel.dk>
[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]
On Thu, Nov 12, 2015 at 08:26:26AM -0700, Jens Axboe wrote:
> On 11/12/2015 03:04 AM, Mark Brown wrote:
> >Android now wants to encrypt phones and tablets by default and have been
> >seeing substantial performance hits as a result, we can try to get
> >people to share performance data from productionish systems but it might
> >be difficult.
> Well, shame on them for developing out-of-tree, looks like they are reaping
> all the benefits of that.
> Guys, we need some numbers, enough with the hand waving. There's no point
> discussing this further until we know how much of a difference it makes to
> handle X MB chunks instead of Y MB chunks. As was previously stated, unless
> there's a _substantial_ performance benefit, this patchset isn't going
> anywhere.
Yeah, what I'm saying here is that there will issues getting the numbers
from relevant production systems - we are most likely to be looking at
proxies which are hopefully reasonably representative but there's likely
to be more divergence than you'd see just running benchmark workloads on
similar systems to those used in production.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-13 11:37 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Jan Kara, Christoph Hellwig, Jens Axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Garg, Dinesh, LKML
In-Reply-To: <4015766.7tjD0ocbRK@wuerfel>
On 13 November 2015 at 17:05, Arnd Bergmann <arnd@arndb.de> wrote:
> On Friday 13 November 2015 10:05:28 Baolin Wang wrote:
>>
>> Well, I did a simple test with dd reading, cause my engine limitation is 1M,
>> (1) so the time like below when handle 1M at one time.
>> 1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s
>>
>> (2) These handle 64K at one time * 16 times
>> 1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
>> 1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s
>>
>> Here is a 10ms level difference, try to image if the hardware engine's
>> throughput is bigger than that. But like Jens said, we can measure it
>> by the performance data.
>
> The absolute numbers look really low. Does this include writing to
> a hard drive? That would certainly make the difference appear
> less significant.
>
OK, I'll supply the complete performance data to measure it.
> Could you try backing this with a ram disk backing for comparison,
> and also use 'time dd' to show the CPU utilization for all cases?
> For completeness, including cpu-only performance might also help
> put this into perspective.
>
OK, I'll try. Thanks.
> Arnd
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Arnd Bergmann @ 2015-11-13 9:05 UTC (permalink / raw)
To: Baolin Wang
Cc: Jan Kara, Christoph Hellwig, axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Garg, Dinesh, LKML
In-Reply-To: <CAMz4ku+NfrPG9PEwOHb-dLxn+jWhKEtwF2UoBnJes7Ok2MOiVQ@mail.gmail.com>
On Friday 13 November 2015 10:05:28 Baolin Wang wrote:
>
> Well, I did a simple test with dd reading, cause my engine limitation is 1M,
> (1) so the time like below when handle 1M at one time.
> 1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s
>
> (2) These handle 64K at one time * 16 times
> 1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
> 1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
> 1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s
>
> Here is a 10ms level difference, try to image if the hardware engine's
> throughput is bigger than that. But like Jens said, we can measure it
> by the performance data.
The absolute numbers look really low. Does this include writing to
a hard drive? That would certainly make the difference appear
less significant.
Could you try backing this with a ram disk backing for comparison,
and also use 'time dd' to show the CPU utilization for all cases?
For completeness, including cpu-only performance might also help
put this into perspective.
Arnd
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-13 3:27 UTC (permalink / raw)
To: Mark Brown
Cc: Arnd Bergmann, Jan Kara, Christoph Hellwig, Jens Axboe,
Alasdair G Kergon, Mike Snitzer, dm-devel, neilb, tj, jmoyer,
keith.busch, bart.vanassche, linux-raid, Garg, Dinesh, LKML
In-Reply-To: <20151112150238.GO12392@sirena.org.uk>
On 12 November 2015 at 23:02, Mark Brown <broonie@kernel.org> wrote:
> On Thu, Nov 12, 2015 at 01:57:27PM +0100, Arnd Bergmann wrote:
>> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
>
>> > But it maybe not enough for HW engine which can handle maybe 10M/20M
>> > at one time.
>
>> Given that you have already done measurements, can you find out how much
>> you lose in overall performance with your existing patch if you artificially
>> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
>
> It's probably also worth looking at the impact on CPU utilisation as
> well as throughput in your benchmarking since the system will often not
> be idle when it's doing a lot of I/O - I know you've done some
> measurements in that area before, including them when looking at block
> sizes might be interesting.
Make sense.
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-13 3:25 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Jan Kara, Christoph Hellwig, Jens Axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Garg, Dinesh, LKML
In-Reply-To: <4436790.rotQ3a7v5c@wuerfel>
On 12 November 2015 at 20:57, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
>> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> >> > Well, one question is "can handle" and other question is how big gain in
>> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> >> > constant overhead to issue a request to the crypto hw and by the time it is
>> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> >> > the encryption itself which is in principle linear in the size of the
>> >> > block. That's why I'd like to get idea of the real numbers...
>> >>
>> >> Please correct me if I misunderstood your point. Let's suppose the AES
>> >> engine can handle 16M at one time. If we give the size of data is less
>> >> than 16M, the engine can handle it at one time. But if the data size
>> >> is 20M (more than 16M), the engine driver will split the data with 16M
>> >> and 4M to deal with. I can not say how many numbers, but I think the
>> >> engine is like to big chunks than small chunks which is the hardware
>> >> engine's advantage.
>> >
>> > No, I meant something different. I meant that if HW can encrypt 1M in say
>> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
>> > gives you some advantage it becomes diminishingly small.
>> >
>>
>> But if it encrypts 16M with 1M one by one, it will be much more than
>> 16.05ms (should be consider the SW submits bio one by one).
>
> The example that Jan gave was meant to illustrate the case where it's not
> much more than 16.05ms, just slightly more.
>
> The point is that we need real numbers to show at what size we stop
> getting significant returns from increased block sizes.
>
Got it. Thanks.
>> >> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> >> > struct bio_vec. Is that not enough?
>> >> >>
>> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> >> or some other small chunks. But request can combine some sequential
>> >> >> small bios to be a big block and it is better than bio at least.
>> >> >
>> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> >> > you check it?
>> >>
>> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>> >
>> > On 4.3 it is 1 MB (which should be enough because requests are limited to
>> > 512 KB by default anyway). Previously the maximum bio size depended on the
>> > queue parameters such as max number of segments etc.
>>
>> But it maybe not enough for HW engine which can handle maybe 10M/20M
>> at one time.
>
> Given that you have already done measurements, can you find out how much
> you lose in overall performance with your existing patch if you artificially
> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
>
Cause my board AES engine throughput is 1M, I just did a simple dd
test with small chunks. Results are in last email.
> Arnd
--
Baolin.wang
Best Regards
^ permalink raw reply
* [PATCH] drivers: md: use ktime_get_real_seconds()
From: Deepa Dinamani @ 2015-11-13 3:16 UTC (permalink / raw)
To: linux-kernel, linux-raid, Neil Brown, Arnd Bergmann
get_seconds() API is not y2038 safe on 32 bit systems and the API
is deprecated. Replace it with calls to ktime_get_real_seconds()
API instead. Change mddev structure types to time64_t accordingly.
32 bit signed timestamps will overflow in the year 2038.
Change the user interface mdu_array_info_s structure timestamps:
ctime and utime values used in ioctls GET_ARRAY_INFO and
SET_ARRAY_INFO to unsigned int. This will extend the field to last
until the year 2106.
The long term plan is to get rid of ctime and utime values in
this structure as this information can be read from the on-disk
meta data directly.
Clamp the tim64_t timestamps to positive values with a max of U32_MAX
when returning from GET_ARRAY_INFO ioctl to accommodate above changes
in the data type of timestamps to unsigned int.
v0.90 on disk meta data uses u32 for maintaining time stamps.
So this will also last until year 2106.
Assumption is that the usage of v0.90 will be deprecated by
year 2106.
Timestamp fields in the on disk meta data for v1.0 version already
use 64 bit data types. Remove the truncation of the bits while
writing to or reading from these from the disk.
Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com>
Reviewed-by: Arnd Bergmann <arnd@arndb.de>
---
Adding the maintainer lists for md.
Notes:
A separate patch will update mdadm to obtain times from the metadata,
and to give a deprecation warning for use of v0.90 arrays
drivers/md/md.c | 18 +++++++++---------
drivers/md/md.h | 2 +-
include/uapi/linux/raid/md_u.h | 4 ++--
3 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 7ab9ed9..20763ea 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1196,13 +1196,13 @@ static void super_90_sync(struct mddev *mddev, struct md_rdev *rdev)
memcpy(&sb->set_uuid2, mddev->uuid+8, 4);
memcpy(&sb->set_uuid3, mddev->uuid+12,4);
- sb->ctime = mddev->ctime;
+ sb->ctime = clamp_t(time64_t, mddev->ctime, 0, U32_MAX);
sb->level = mddev->level;
sb->size = mddev->dev_sectors / 2;
sb->raid_disks = mddev->raid_disks;
sb->md_minor = mddev->md_minor;
sb->not_persistent = 0;
- sb->utime = mddev->utime;
+ sb->utime = clamp_t(time64_t, mddev->utime, 0, U32_MAX);
sb->state = 0;
sb->events_hi = (mddev->events>>32);
sb->events_lo = (u32)mddev->events;
@@ -1542,8 +1542,8 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
mddev->patch_version = 0;
mddev->external = 0;
mddev->chunk_sectors = le32_to_cpu(sb->chunksize);
- mddev->ctime = le64_to_cpu(sb->ctime) & ((1ULL << 32)-1);
- mddev->utime = le64_to_cpu(sb->utime) & ((1ULL << 32)-1);
+ mddev->ctime = le64_to_cpu(sb->ctime);
+ mddev->utime = le64_to_cpu(sb->utime);
mddev->level = le32_to_cpu(sb->level);
mddev->clevel[0] = 0;
mddev->layout = le32_to_cpu(sb->layout);
@@ -2331,7 +2331,7 @@ repeat:
spin_lock(&mddev->lock);
- mddev->utime = get_seconds();
+ mddev->utime = ktime_get_real_seconds();
if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
force_change = 1;
@@ -5828,7 +5828,7 @@ static int get_array_info(struct mddev *mddev, void __user *arg)
info.major_version = mddev->major_version;
info.minor_version = mddev->minor_version;
info.patch_version = MD_PATCHLEVEL_VERSION;
- info.ctime = mddev->ctime;
+ info.ctime = clamp_t(time64_t, mddev->ctime, 0, U32_MAX);
info.level = mddev->level;
info.size = mddev->dev_sectors / 2;
if (info.size != mddev->dev_sectors / 2) /* overflow */
@@ -5838,7 +5838,7 @@ static int get_array_info(struct mddev *mddev, void __user *arg)
info.md_minor = mddev->md_minor;
info.not_persistent= !mddev->persistent;
- info.utime = mddev->utime;
+ info.utime = clamp_t(time64_t, mddev->utime, 0, U32_MAX);
info.state = 0;
if (mddev->in_sync)
info.state = (1<<MD_SB_CLEAN);
@@ -6338,13 +6338,13 @@ static int set_array_info(struct mddev *mddev, mdu_array_info_t *info)
/* ensure mddev_put doesn't delete this now that there
* is some minimal configuration.
*/
- mddev->ctime = get_seconds();
+ mddev->ctime = ktime_get_real_seconds();
return 0;
}
mddev->major_version = MD_MAJOR_VERSION;
mddev->minor_version = MD_MINOR_VERSION;
mddev->patch_version = MD_PATCHLEVEL_VERSION;
- mddev->ctime = get_seconds();
+ mddev->ctime = ktime_get_real_seconds();
mddev->level = info->level;
mddev->clevel[0] = 0;
diff --git a/drivers/md/md.h b/drivers/md/md.h
index f5b9aad..237b507 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -261,7 +261,7 @@ struct mddev {
* managed externally */
char metadata_type[17]; /* externally set*/
int chunk_sectors;
- time_t ctime, utime;
+ time64_t ctime, utime;
int level, layout;
char clevel[16];
int raid_disks;
diff --git a/include/uapi/linux/raid/md_u.h b/include/uapi/linux/raid/md_u.h
index 1cb8aa6..36cd821 100644
--- a/include/uapi/linux/raid/md_u.h
+++ b/include/uapi/linux/raid/md_u.h
@@ -80,7 +80,7 @@ typedef struct mdu_array_info_s {
int major_version;
int minor_version;
int patch_version;
- int ctime;
+ unsigned int ctime;
int level;
int size;
int nr_disks;
@@ -91,7 +91,7 @@ typedef struct mdu_array_info_s {
/*
* Generic state information
*/
- int utime; /* 0 Superblock update time */
+ unsigned int utime; /* 0 Superblock update time */
int state; /* 1 State bits (clean, ...) */
int active_disks; /* 2 Number of currently active disks */
int working_disks; /* 3 Number of working disks */
--
1.9.1
^ permalink raw reply related
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-13 2:07 UTC (permalink / raw)
To: Jens Axboe
Cc: Mark Brown, Mike Snitzer, Alasdair G Kergon, dm-devel, neilb,
linux-raid, Jan Kara, Arnd Bergmann, LKML, keith.busch, jmoyer,
tj, bart.vanassche, Garg, Dinesh
In-Reply-To: <5644AFA2.6040201@kernel.dk>
On 12 November 2015 at 23:26, Jens Axboe <axboe@kernel.dk> wrote:
> On 11/12/2015 03:04 AM, Mark Brown wrote:
>>
>> On Thu, Nov 12, 2015 at 04:20:41PM +0800, Baolin Wang wrote:
>>
>>> 3. perforamence data
>>> It is just a simple dd test result, and will provide the formal report
>>> in future. But from the simple test, we can see the improvement.
>>
>>
>> It's probably also worth pointing out that Qualcomm have been shipping
>> an out of tree implementation of this as a separate module in their BSP
>> (originally written by Danesh Garg who's on this thread):
>>
>>
>> https://android.googlesource.com/kernel/msm/+/android-msm-dory-3.10-kitkat-wear/drivers/md/dm-req-crypt.c
>>
>> Android now wants to encrypt phones and tablets by default and have been
>> seeing substantial performance hits as a result, we can try to get
>> people to share performance data from productionish systems but it might
>> be difficult.
>
>
> Well, shame on them for developing out-of-tree, looks like they are reaping
> all the benefits of that.
>
> Guys, we need some numbers, enough with the hand waving. There's no point
> discussing this further until we know how much of a difference it makes to
> handle X MB chunks instead of Y MB chunks. As was previously stated, unless
> there's a _substantial_ performance benefit, this patchset isn't going
> anywhere.
That's fair enough and we will provide the performance data to measure
the patchset.
>
> If there is a huge benefit, we can look into ways of making it actually
> work. That may not even be a request interface, it could just be proper
> utilization of plugging for in-dm bio merging.
Make sense. Thanks.
>
> --
> Jens Axboe
>
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-13 2:05 UTC (permalink / raw)
To: Jan Kara
Cc: Christoph Hellwig, axboe, Alasdair G Kergon, Mike Snitzer,
dm-devel, neilb, tj, jmoyer, keith.busch, bart.vanassche,
linux-raid, Mark Brown, Arnd Bergmann, Garg, Dinesh, LKML
In-Reply-To: <20151112125901.GD27454@quack.suse.cz>
On 12 November 2015 at 20:59, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 20:51:10, Baolin Wang wrote:
>> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> >> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
>> >> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
>> >> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> >> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> >> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> >> >> >> one bio must use the sequential physical address and it also has a limitation
>> >> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> >> >> >> hardware support the big block data encryption.
>> >> >> >> >>
>> >> >> >> >> This patch series introduc the 'based-request' method to handle the data
>> >> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> >> >> >> handle big block data to improve the efficiency.
>> >> >> >> >
>> >> >> >> > NAK for more request based stacking or DM drivers. They are a major
>> >> >> >> > pain to deal with, and adding more with different requirements then
>> >> >> >> > dm-multipath is not helping in actually making that one work properly.
>> >> >> >>
>> >> >> >> But now many vendors supply the hardware engine to handle the
>> >> >> >> encyrtion/decryption. The hardware really need a big block to indicate
>> >> >> >> its performance with request based things. Another thing is now the
>> >> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> >> >> >> so on) to improve their performance and there's a real performance
>> >> >> >> requirement here (I can show the performance result later).
>> >> >> >
>> >> >> > So you've mentioned several times that hardware needs big blocks. How big
>> >> >> > those blocks need to be? Ideally, can you give some numbers on how the
>> >> >> > throughput of the encryption hw grows with the block size?
>> >> >>
>> >> >> It depends on the hardware design. My beaglebone black board's AES
>> >> >> engine can handle 1M at one time which is not big. As I know some
>> >> >> other AES engine can handle 16M data at one time or more.
>> >> >
>> >> > Well, one question is "can handle" and other question is how big gain in
>> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> >> > constant overhead to issue a request to the crypto hw and by the time it is
>> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> >> > the encryption itself which is in principle linear in the size of the
>> >> > block. That's why I'd like to get idea of the real numbers...
>> >>
>> >> Please correct me if I misunderstood your point. Let's suppose the AES
>> >> engine can handle 16M at one time. If we give the size of data is less
>> >> than 16M, the engine can handle it at one time. But if the data size
>> >> is 20M (more than 16M), the engine driver will split the data with 16M
>> >> and 4M to deal with. I can not say how many numbers, but I think the
>> >> engine is like to big chunks than small chunks which is the hardware
>> >> engine's advantage.
>> >
>> > No, I meant something different. I meant that if HW can encrypt 1M in say
>> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
>> > gives you some advantage it becomes diminishingly small.
>> >
>>
>> But if it encrypts 16M with 1M one by one, it will be much more than
>> 16.05ms (should be consider the SW submits bio one by one).
>
> Really? In my example, it would take 16.8 ms if we encrypted 16M in 1M
> chunks and 16.05 ms if done in one chunk. That is a difference for which I
> would not be willing to bend over backwards. Now these numbers are
> completely made up and that's why I wanted to see the real numbers...
>
Well, I did a simple test with dd reading, cause my engine limitation is 1M,
(1) so the time like below when handle 1M at one time.
1048576 bytes (1.0 MB) copied, 0.0841235 s, 12.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0836294 s, 12.5 MB/s
1048576 bytes (1.0 MB) copied, 0.0836526 s, 12.5 MB/s
(2) These handle 64K at one time * 16 times
1048576 bytes (1.0 MB) copied, 0.0937223 s, 11.2 MB/s
1048576 bytes (1.0 MB) copied, 0.097205 s, 10.8 MB/s
1048576 bytes (1.0 MB) copied, 0.0935884 s, 11.2 MB/s
Here is a 10ms level difference, try to image if the hardware engine's
throughput is bigger than that. But like Jens said, we can measure it
by the performance data. Thanks.
>> >> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> >> > struct bio_vec. Is that not enough?
>> >> >>
>> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> >> or some other small chunks. But request can combine some sequential
>> >> >> small bios to be a big block and it is better than bio at least.
>> >> >
>> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> >> > you check it?
>> >>
>> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>> >
>> > On 4.3 it is 1 MB (which should be enough because requests are limited to
>> > 512 KB by default anyway). Previously the maximum bio size depended on the
>> > queue parameters such as max number of segments etc.
>>
>> But it maybe not enough for HW engine which can handle maybe 10M/20M
>> at one time.
>
> Currently, you would not be able to create larger than 512K / 1M chunks
> even with request based dm-crypt since requests have limits on number of
> data they can carry as well... So this is kind of abstract discussion.
>
OK. But I think if it is that it should change the default limitation
for the DM device.
> Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: Raid server Motherboard recommendation.
From: Ram Ramesh @ 2015-11-13 1:54 UTC (permalink / raw)
To: Michael Munger, Linux Raid
In-Reply-To: <46FCA36BC7E1D546AC28870158F9C25B03037AB9EC@ABRAHAM.highpoweredhelp.local>
On 11/12/2015 10:04 AM, Michael Munger wrote:
>>> All others seem to be X99 or other server chipsets which I am not fond of. I like desktop boards as the server also doubles as mythtv backend/frontend.
> I have three of these (1 X79 Sabertooh Tuff and 2 X99 Sabertooth Tuff boards) all running like a champ with Debian. I highly recommend them.
>
> If you want something cheaper, I have one box (rather old) that is an Intel H61M-DS2. So, whatever the current version of that family of boards is. And I have another that is a Biostar T41 HD.
Thanks for the suggestion. I have not built any with X99. May be I
should try. I will think about it.
Ramesh
^ permalink raw reply
* Re: Raid server Motherboard recommendation.
From: Ram Ramesh @ 2015-11-13 1:51 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Roman Mamedov, Linux Raid
In-Reply-To: <20151112151105.GA32385@EIS>
On 11/12/2015 09:11 AM, Andreas Klauer wrote:
> On Thu, Nov 12, 2015 at 08:15:01AM -0600, Ram Ramesh wrote:
>> Thanks. I did see that. But, in newegg, there are reviews dated much
>> later than above complaining about ubuntu 14.04 installs failing with
>> ASMedia enabled. So, I concluded that it is iffy. I could not see any
>> consistent message of positive result.
> I use ASMedia controllers myself, onboard and as standalone card.
> They were cheap (much cheaper than getting a board with additional
> ports using the same chipset really) and I don't need the PCIe slots
> for anything else.
>
> They work fine in AHCI mode for HDD, with mdadm software raid.
>
> If you have a CD drive hooked to it the kernel will crash.
> Until you add libata.atapi_passthru16=0 to your cmdline.
>
> Regards
> Andreas
Andreas,
Thanks a lot. Now I feel a lot more confident trying out extreme6. I
usually put all my boot related (cd, primary sdd) on intel any way. So,
CD drive should not be a problem.
Ramesh
^ permalink raw reply
* Re: Problem w/ commit ac8fa4196d20 on older, slower hardware
From: Andreas Klauer @ 2015-11-13 0:03 UTC (permalink / raw)
To: Joshua Kinard; +Cc: Neil Brown, linux-raid
In-Reply-To: <56451299.6090107@gentoo.org>
On Thu, Nov 12, 2015 at 05:28:41PM -0500, Joshua Kinard wrote:
> running MD RAID5 and the XFS filesystem. I have /, /home, /usr, /var,
> and /tmp on separate partitions, each a RAID5 setup.
Hi, sorry for butting in,
I have the same issue, on a regular consumer Haswell i5 box,
with a setup very very similar to yours:
7x2TB disks, multiple partitions, for each: RAID-5, LUKS, LVM, XFS.
The issue occurs during regular RAID check which I run daily
(different partition/RAID each day, so it's more like a
evenly distributed weekly check).
I have an application that uses `find -size +100M` on a directory
tree with ~3k subdirs and ~6k files in total. It doesn't do anything
with the find result, it's purely informal. So no big data involved,
even though the files themselves aren't small.
Yet, it's slooow. The following tests were on a completely idle box,
apart from a running RAID check on the same /dev/mdX device.
Kernel 4.2.3, unpatched:
real 0m53.555s
user 0m0.013s
sys 0m0.037s
real 1m3.777s
user 0m0.013s
sys 0m0.037s
real 1m3.453s
user 0m0.014s
sys 0m0.036s
Kernel 4.2.3, reverted ac8fa4196d20:
real 0m3.206s
user 0m0.010s
sys 0m0.030s
real 0m0.450s
user 0m0.003s
sys 0m0.014s
real 0m0.375s
user 0m0.003s
sys 0m0.012s
I did echo 3 > /proc/sys/vm/drop_caches between each find.
For some reason, subsequent calls in the reverted kernel are
considerably faster regardless. In the original kernel it
stays slow... if I don't drop_caches, the time is 0.006s.
I don't normally reboot (while a RAID sync or check is
running) but while switching between kernels I noticed
the shutdown was very slow also in the original kernel.
Are small requests getting delayed a lot or something?
Regards
Andreas Klauer
^ permalink raw reply
* Re: [PATCH] drivers: md: use ktime_get_real_seconds()
From: Deepa Dinamani @ 2015-11-12 23:14 UTC (permalink / raw)
To: Arnd Bergmann, linux-raid, linux-kernel, Neil Brown; +Cc: y2038
In-Reply-To: <4073088.alOdrNRecE@wuerfel>
On Thu, Nov 12, 2015 at 1:56 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 11 November 2015 19:23:33 Deepa Dinamani wrote:
> > get_seconds() API is not y2038 safe on 32 bit systems and the API
> > is deprecated. Replace it with calls to ktime_get_real_seconds()
> > API instead. Change mddev structure types to time64_t accordingly.
> >
> > 32 bit signed timestamps will overflow in the year 2038.
> >
> > Change the user interface mdu_array_info_s structure timestamps:
> > ctime and utime values used in ioctls GET_ARRAY_INFO and
> > SET_ARRAY_INFO to unsigned int. This will extend the field to last
> > until the year 2106.
> > The long term plan is to get rid of ctime and utime values in
> > this structure as this information can be read from the on-disk
> > meta data directly.
> >
> > Clamp the tim64_t timestamps to positive values with a max of U32_MAX
> > when returning from GET_ARRAY_INFO ioctl to accommodate above changes
> > in the data type of timestamps to unsigned int.
> >
> > v0.90 on disk meta data uses u32 for maintaining time stamps.
> > So this will also last until year 2106.
> > Assumption is that the usage of v0.90 will be deprecated by
> > year 2106.
> >
> > Timestamp fields in the on disk meta data for v1.0 version already
> > use 64 bit data types. Remove the truncation of the bits while
> > writing to or reading from these from the disk.
> >
> > Signed-off-by: Deepa Dinamani <deepa.kernel@gmail.com>
>
> Reviewed-by: Arnd Bergmann <arnd@arndb.de>
>
> The patch looks great, and the changelog is perfect.
>
> I didn't even know we had a clamp_t() macro for this.
>
>
Adding the maintainer lists for md.
_______________________________________________
Y2038 mailing list
Y2038@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/y2038
^ permalink raw reply
* Re: Problem w/ commit ac8fa4196d20 on older, slower hardware
From: Joshua Kinard @ 2015-11-12 22:28 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
In-Reply-To: <87twq0527k.fsf@notabene.neil.brown.name>
On 10/08/2015 20:13, Neil Brown wrote:
>
>> Per commit ac8fa4196d20:
>>
>>> md: allow resync to go faster when there is competing IO.
>>>
>>> When md notices non-sync IO happening while it is trying to resync (or
>>> reshape or recover) it slows down to the set minimum.
>>>
>>> The default minimum might have made sense many years ago but the drives have
>>> become faster. Changing the default to match the times isn't really a long
>>> term solution.
>>
>> This holds true for modern hardware, but this commit is causing problems on
>> older hardware, like SGI MIPS platforms, that use mdraid. Namely, while trying
>> to chase down an unrelated hardlock bug on an Onyx2, one of the arrays got out
>> of sync, so on the next reboot, mdraid's attempt to resync at full speed
>> absolutely murdered interactivity. It took close to 30mins for the system to
>> finally reach the login prompt.
>>
>> Revert this patch was working to mitigate the problem at first, but it appears
>> that in recent kernels, this is no longer the case, and reverting this commit
>> has no noticeable effect anymore. I assume I'd have to hunt down newer commits
>> to revert, but it's probably saner to just highlight the problem and test any
>> proposed solutions.
>>
>> Is there some way to resolve this in such a way that old hardware maintains
>> some level of interactivity during a resync, but that won't inconvenience the
>> more modern systems?
>>
>> http://git.linux-mips.org/cgit/ralf/linux.git/commit/?id=ac8fa4196d20
>>
>> Thanks!,
>>
>
> Hmmm... this change shouldn't have that effect.
> It should allow resync to soak up a bit more of the idle time, but when
> there is any other IO, resync should still back off.
>
> I wonder if there is some other change which has confused the event
> counting for the particular hardware you are using.
>
> How did you identify this commit as a possible cause?
Sorry for the late response. I pinned down this particular commit as the cause
on an SGI Onyx2 (IP27), which is a MIPS big-endian platform that supports
ccNUMA. The SCSI chip is a QLogic ISP1040B. It's been supported in the
mainline kernel for a long time, but has suffered from bit-rot over the years.
There's an unidentified bug somewhere in the architecture code that, under
heavy disk I/O or memory operations (I am not sure which, yet), the machine
will completely lock up hard.
I have three ~50GB SCA SCSI drives plugged into it, running MD RAID5 and the
XFS filesystem. I have /, /home, /usr, /var, and /tmp on separate partitions,
each a RAID5 setup. After one of these hard lockups, on the next reboot, the
kernel detected that my largest partition, /usr, needed to be rebuilt, so it
launched a background resync. The other partitions were fine.
I noticed after several minutes that the kernel had still not proceeded to
execute /init, and that XFS hadn't even mounted the rootfs yet. I thought the
machine had hardlocked again. The lockup bug normally does not happen with a
resync (which takes place entirely within the kernel), but more so when running
commands from userspace. Physically checking the machine, the disk lights were
showing drive activity, so I let it sit for a good half-hour, and when I later
checked the serial console out, it had gotten most of the way through the
bootup process and was still bringing up runlevel 3 services.
Logging into the root console several minutes later showed the resync was
almost complete, but interactivity remained very sluggish until the resync
finished. So I dug into gitweb on linux-mips.org and looked for any recent
commits to md.c that might have something to do with resync operations, and
this one stood out the most. Reverting it, then forcing the lockup bug to
happen several times until another background resync took place showed
drastically-improved bootup speed. The machine was able to boot to userland
within ~4-6 mins with the background resync happening on /usr.
I think this was on 3.19 or 4.0 (I forget). It was on the next version up that
I noticed the revert was no longer having an effect, and a resync slowed I/O
down enough that booting to userland was back into the ~30min range. I have
also noticed that the lockup bug is also happening, randomly, during a resync
now too. I suspect whatever issue is causing the lockup is getting worse.
The last kernel I booted on this platform was a 4.2-rcX release. I have not
had time to test 4.3.x out.
I have also reproduced the same issue on an SGI Octane (IP30), which needs
out-of-tree patches to work. It's basically the smaller cousin of an
Origin/Onyx2, using the same CPU, SCSI chip, same partition layout, same
filesystem. Only the disks, 3x 73GB SCA SCSI disks, and some internal hardware
architecture, are different between the two. It does not suffer from any
lockup bugs whatsoever, and I only triggered a background resync when I got
frustrated at an unrelated issue and powered the machine off out of annoyance.
Per hdparm -tT, the average I/O speed is ~160MB/sec reading from cache, and
~18.3MB/sec reading from the /dev/mdX devices. Reading from the individual
/dev/sdX drives is slightly faster at ~18.5MB/sec. This is true for both machines.
> The fact that reverting it no longer helps strongly suggests that some
> other change is implicated. I don't think there have been other changes
> in md which could affect this.
The changes to the code that this commit affected seems to play some role in
the issue, but I agree that it does not appear to be the sole participant anymore.
> Have you tried adjusting /proc/sys/dev/raid/speed_limit_m{ax,in} ??
> Did that have any noticeable effect?
Hard to do when your kernel takes 30+ minutes to boot up :) Once I got to
userland in one instance, though, I did touch one of the /proc parameters (I
for get which one, but it had something to do w/ the minimum background I/O
speed) and dropped it down to 1,000K/sec, the machine's responsiveness improved
dramatically.
The real issue of what's causing the lockups in the first place ultimately
needs to be chased down, but I lack the debugging skills necessary to do that.
I tend to stop for the night when the resync needs to take place and power the
machine down, as it drinks ~700W+, and I save the long resync for a day when
utility rates are low.
--J
^ permalink raw reply
* Re: [PATCH 1/6] md-cluster: Protect communication with mutexes
From: NeilBrown @ 2015-11-12 21:59 UTC (permalink / raw)
To: Goldwyn Rodrigues, linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <5641631F.2080000@suse.de>
[-- Attachment #1: Type: text/plain, Size: 9182 bytes --]
On Tue, Nov 10 2015, Goldwyn Rodrigues wrote:
> On 11/09/2015 05:31 PM, NeilBrown wrote:
>> On Fri, Nov 06 2015, rgoldwyn@suse.de wrote:
>>
>>> From: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>>
>>> Communication can happen through multiple threads. It is possible that
>>> one thread steps over another threads sequence. So, we use mutexes to
>>> protect both the send and receive sequences.
>>>
>>> We use a new flag CLUSTER_MD_COMM_LOCKED which is used to detect
>>> if communication is already locked. This is useful for cases where we need to
>>> take and hold the token DLM lock for a while. This bit is set only
>>> after locking communication.
>>
>> I don't understand what this flag is trying to achieve, but I'm fairly
>> sure it doesn't achieve it.
>
> Lets consider three specific cases of locking communication channels to
> show the conflict:
>
> 1. resync_info_update(): communication is locked and release for sending
> a message.
> 2. A regular md_update_sb(): communication is locked in
> metadata_update_start() and unlocked in metadata_update_finish() after
> writing to disk. In metadata_update_finish(), the sendmsg is called to
> send METADATA_UPDATED.
> 3. An add operation which culminates in a md_update_sb(): Here the
> communication is locked before initiating add. If the add is successful,
> it results in md_update_sb(). In md_update_sb(), metadata_update_start()
> has to check if the communication is already locked. If locked, it
> should not lock again.
Oh, I get it now - thanks.
Going back and looking at the original commit I can now see that it does
say that, but I didn't understand the implication at the time.
I might be wrong again, but I think this approach is broken.
The 'add disk' sequence does:
1/ ->add_new_disk which takes the lock
2/ other stuff, protected by the lock
3/ schedule a metadata update
4/ metadata update is initiated, which doesn't take the lock
because the flag is set
5/ metadata update completes, lock is dropped.
What if some other event causes the metadata update to happen during
'2'?
Maybe if you delayed setting MD_CLUSTER_COMM_LOCKED until after '2'.
i.e. set it when scheduling the update. So the flag means:
"lock has been taken for metadata update"
One tricky bit there would be if metadata_update_start() found the bit
wasn't set, and then entered mutex_lock() in lock_comm().
When MD_CLUSTER_COMM_LOCKED gets set we want that code to stop waiting
and start doing useful things. But it won't.
It might be easiest to make our own 'mutex' with a flag bit and a wait
queue.
Then calls to mutex_lock become
wait_event(mddev->wait, !test_and_set_bit(bit, &cinfo->state));
mutex_unlock becoems
clear_bit(bit, &cinfo->state);
and the mutex_lock used from metadata_update_start is
wait_event(mddev->wait, !test_and_set_bit(bit, &cinfo->state) ||
test_and_clear_bit(MD_CLUSTER_COMM_LOCKED,
....));
That might work.... providing it is well documented.
(and you are right - mutex_trylock wouldn't have helped, I was
completely misunderstanding).
Thanks,
NeilBrown
>
> The flag is used only for case 3. If the communication is already
> locked, it should not lock again. This flag is set only after
> lock_comm() has executed, but is checked for in metadata_update_start().
> This should insure that any of the operations 1, 2 or 3 do not interfere
> with each other.
>
> I am not sure if I have made the best effort to explain this. I had a
> tough time getting it right (which may or may not be complete).
>
>>
>> Maybe if it was test_and_set_bit in metadata_update_start, it might make
>> sense. But then I would suggest that clearing the bit be moved to
>> unlock_comm()
>>
>> Do you just want to use mutex_try_lock() to detect if communication is
>> already locked?
>
> Consider a race between md_update_sb() and sendmsg() [Case 1. and 2]
>
> mutex_try_lock() may not work in this situation because lock_comm()
> could have been called by sendmsg(), which will release it as soon as
> the message is sent. In the meantime (while the lock is locked), if a
> metadata_update_sb() operation executes. It will not relock the
> communication. This will result in the WARN_ON in unlock_comm() since
> sendmsg() sequence had already unlocked it.
>
>
>>
>>>
>>> Also, Remove stray/ununsed sb_mutex.
>>
>> I already removed that it mainline - I should have mentioned.
>>
>> Thanks,
>> NeilBrown
>>
>>
>>>
>>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>>> ---
>>> drivers/md/md-cluster.c | 26 +++++++++++++++++++++-----
>>> 1 file changed, 21 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
>>> index b73729b..a93734e 100644
>>> --- a/drivers/md/md-cluster.c
>>> +++ b/drivers/md/md-cluster.c
>>> @@ -47,6 +47,7 @@ struct resync_info {
>>> #define MD_CLUSTER_WAITING_FOR_NEWDISK 1
>>> #define MD_CLUSTER_SUSPEND_READ_BALANCING 2
>>> #define MD_CLUSTER_BEGIN_JOIN_CLUSTER 3
>>> +#define MD_CLUSTER_COMM_LOCKED 4
>>>
>>>
>>> struct md_cluster_info {
>>> @@ -54,7 +55,8 @@ struct md_cluster_info {
>>> dlm_lockspace_t *lockspace;
>>> int slot_number;
>>> struct completion completion;
>>> - struct mutex sb_mutex;
>>> + struct mutex recv_mutex;
>>> + struct mutex send_mutex;
>>> struct dlm_lock_resource *bitmap_lockres;
>>> struct dlm_lock_resource *resync_lockres;
>>> struct list_head suspend_list;
>>> @@ -503,6 +505,7 @@ static void recv_daemon(struct md_thread *thread)
>>> struct cluster_msg msg;
>>> int ret;
>>>
>>> + mutex_lock(&cinfo->recv_mutex);
>>> /*get CR on Message*/
>>> if (dlm_lock_sync(message_lockres, DLM_LOCK_CR)) {
>>> pr_err("md/raid1:failed to get CR on MESSAGE\n");
>>> @@ -529,6 +532,7 @@ static void recv_daemon(struct md_thread *thread)
>>> ret = dlm_unlock_sync(message_lockres);
>>> if (unlikely(ret != 0))
>>> pr_info("unlock msg failed return %d\n", ret);
>>> + mutex_unlock(&cinfo->recv_mutex);
>>> }
>>>
>>> /* lock_comm()
>>> @@ -542,20 +546,22 @@ static int lock_comm(struct md_cluster_info *cinfo)
>>> {
>>> int error;
>>>
>>> - if (cinfo->token_lockres->mode == DLM_LOCK_EX)
>>> - return 0;
>>> + mutex_lock(&cinfo->send_mutex);
>>>
>>> error = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
>>> if (error)
>>> pr_err("md-cluster(%s:%d): failed to get EX on TOKEN (%d)\n",
>>> __func__, __LINE__, error);
>>> + mutex_lock(&cinfo->recv_mutex);
>>> return error;
>>> }
>>>
>>> static void unlock_comm(struct md_cluster_info *cinfo)
>>> {
>>> WARN_ON(cinfo->token_lockres->mode != DLM_LOCK_EX);
>>> + mutex_unlock(&cinfo->recv_mutex);
>>> dlm_unlock_sync(cinfo->token_lockres);
>>> + mutex_unlock(&cinfo->send_mutex);
>>> }
>>>
>>> /* __sendmsg()
>>> @@ -709,7 +715,8 @@ static int join(struct mddev *mddev, int nodes)
>>> init_completion(&cinfo->completion);
>>> set_bit(MD_CLUSTER_BEGIN_JOIN_CLUSTER, &cinfo->state);
>>>
>>> - mutex_init(&cinfo->sb_mutex);
>>> + mutex_init(&cinfo->send_mutex);
>>> + mutex_init(&cinfo->recv_mutex);
>>> mddev->cluster_info = cinfo;
>>>
>>> memset(str, 0, 64);
>>> @@ -839,7 +846,12 @@ static int slot_number(struct mddev *mddev)
>>>
>>> static int metadata_update_start(struct mddev *mddev)
>>> {
>>> - return lock_comm(mddev->cluster_info);
>>> + struct md_cluster_info *cinfo = mddev->cluster_info;
>>> + int ret;
>>> + if (test_and_clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state))
>>> + return 0;
>>> + ret = lock_comm(cinfo);
>>> + return ret;
>>> }
>>>
>>> static int metadata_update_finish(struct mddev *mddev)
>>> @@ -864,6 +876,7 @@ static int metadata_update_finish(struct mddev *mddev)
>>> ret = __sendmsg(cinfo, &cmsg);
>>> } else
>>> pr_warn("md-cluster: No good device id found to send\n");
>>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>>> unlock_comm(cinfo);
>>> return ret;
>>> }
>>> @@ -871,6 +884,7 @@ static int metadata_update_finish(struct mddev *mddev)
>>> static void metadata_update_cancel(struct mddev *mddev)
>>> {
>>> struct md_cluster_info *cinfo = mddev->cluster_info;
>>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>>> unlock_comm(cinfo);
>>> }
>>>
>>> @@ -945,6 +959,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
>>> memcpy(cmsg.uuid, uuid, 16);
>>> cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
>>> lock_comm(cinfo);
>>> + set_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>>> ret = __sendmsg(cinfo, &cmsg);
>>> if (ret)
>>> return ret;
>>> @@ -964,6 +979,7 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
>>> static void add_new_disk_cancel(struct mddev *mddev)
>>> {
>>> struct md_cluster_info *cinfo = mddev->cluster_info;
>>> + clear_bit(MD_CLUSTER_COMM_LOCKED, &cinfo->state);
>>> unlock_comm(cinfo);
>>> }
>>>
>>> --
>>> 1.8.5.6
>
> --
> Goldwyn
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Raid server Motherboard recommendation.
From: Piergiorgio Sartor @ 2015-11-12 18:46 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Linux Raid
In-Reply-To: <5643FD4A.4070709@gmail.com>
On Wed, Nov 11, 2015 at 08:45:30PM -0600, Ram Ramesh wrote:
> All,
>
> It is time for my server to be upgraded and I am looking for advice. I
> usually build my machine so, I am looking primarily for a motherboard that
> has several sata ports that is compatible with linux.
>
> I looked at Asrock extreme6, but apparently it has 4 of its 10 sata ports
> are on ASMedia ASM 1061 chipset which apparently is not supported on linux
> per what I can find on various comments/information from the net
>
> All others seem to be X99 or other server chipsets which I am not fond of. I
> like desktop boards as the server also doubles as mythtv backend/frontend.
I usually use Supermicro MB, for storage server.
Lately you can find them for quite reasoneble
price, and fully featured for servers.
You'll have to choose carefully, since they've
many similar boards, with just minor differences.
bye,
pg
>
> Let me know if this is the wrong thread and I should ask some where else.
>
> Regards
> Ramesh
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* RE: Raid server Motherboard recommendation.
From: Michael Munger @ 2015-11-12 16:04 UTC (permalink / raw)
To: 'Ram Ramesh', Linux Raid
In-Reply-To: <5643FD4A.4070709@gmail.com>
>>All others seem to be X99 or other server chipsets which I am not fond of. I like desktop boards as the server also doubles as mythtv backend/frontend.
I have three of these (1 X79 Sabertooh Tuff and 2 X99 Sabertooth Tuff boards) all running like a champ with Debian. I highly recommend them.
If you want something cheaper, I have one box (rather old) that is an Intel H61M-DS2. So, whatever the current version of that family of boards is. And I have another that is a Biostar T41 HD.
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Jens Axboe @ 2015-11-12 15:26 UTC (permalink / raw)
To: Mark Brown, Baolin Wang
Cc: Mike Snitzer, Alasdair G Kergon, dm-devel, neilb, linux-raid,
jack, Arnd Bergmann, LKML, keith.busch, jmoyer, tj,
bart.vanassche, Garg, Dinesh
In-Reply-To: <20151112100422.GM12392@sirena.org.uk>
On 11/12/2015 03:04 AM, Mark Brown wrote:
> On Thu, Nov 12, 2015 at 04:20:41PM +0800, Baolin Wang wrote:
>
>> 3. perforamence data
>> It is just a simple dd test result, and will provide the formal report
>> in future. But from the simple test, we can see the improvement.
>
> It's probably also worth pointing out that Qualcomm have been shipping
> an out of tree implementation of this as a separate module in their BSP
> (originally written by Danesh Garg who's on this thread):
>
> https://android.googlesource.com/kernel/msm/+/android-msm-dory-3.10-kitkat-wear/drivers/md/dm-req-crypt.c
>
> Android now wants to encrypt phones and tablets by default and have been
> seeing substantial performance hits as a result, we can try to get
> people to share performance data from productionish systems but it might
> be difficult.
Well, shame on them for developing out-of-tree, looks like they are
reaping all the benefits of that.
Guys, we need some numbers, enough with the hand waving. There's no
point discussing this further until we know how much of a difference it
makes to handle X MB chunks instead of Y MB chunks. As was previously
stated, unless there's a _substantial_ performance benefit, this
patchset isn't going anywhere.
If there is a huge benefit, we can look into ways of making it actually
work. That may not even be a request interface, it could just be proper
utilization of plugging for in-dm bio merging.
--
Jens Axboe
^ permalink raw reply
* Re: Raid server Motherboard recommendation.
From: Andreas Klauer @ 2015-11-12 15:11 UTC (permalink / raw)
To: Ram Ramesh; +Cc: Roman Mamedov, Linux Raid
In-Reply-To: <56449EE5.3010002@gmail.com>
On Thu, Nov 12, 2015 at 08:15:01AM -0600, Ram Ramesh wrote:
> Thanks. I did see that. But, in newegg, there are reviews dated much
> later than above complaining about ubuntu 14.04 installs failing with
> ASMedia enabled. So, I concluded that it is iffy. I could not see any
> consistent message of positive result.
I use ASMedia controllers myself, onboard and as standalone card.
They were cheap (much cheaper than getting a board with additional
ports using the same chipset really) and I don't need the PCIe slots
for anything else.
They work fine in AHCI mode for HDD, with mdadm software raid.
If you have a CD drive hooked to it the kernel will crash.
Until you add libata.atapi_passthru16=0 to your cmdline.
Regards
Andreas
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Mark Brown @ 2015-11-12 15:02 UTC (permalink / raw)
To: Arnd Bergmann
Cc: Baolin Wang, Jan Kara, Christoph Hellwig, axboe,
Alasdair G Kergon, Mike Snitzer, dm-devel, neilb, tj, jmoyer,
keith.busch, bart.vanassche, linux-raid, Garg, Dinesh, LKML
In-Reply-To: <4436790.rotQ3a7v5c@wuerfel>
[-- Attachment #1: Type: text/plain, Size: 733 bytes --]
On Thu, Nov 12, 2015 at 01:57:27PM +0100, Arnd Bergmann wrote:
> On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
> > But it maybe not enough for HW engine which can handle maybe 10M/20M
> > at one time.
> Given that you have already done measurements, can you find out how much
> you lose in overall performance with your existing patch if you artificially
> limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
It's probably also worth looking at the impact on CPU utilisation as
well as throughput in your benchmarking since the system will often not
be idle when it's doing a lot of I/O - I know you've done some
measurements in that area before, including them when looking at block
sizes might be interesting.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply
* Re: Raid server Motherboard recommendation.
From: Ram Ramesh @ 2015-11-12 14:15 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux Raid
In-Reply-To: <20151112122908.0406d170@natsu>
On 11/12/2015 01:29 AM, Roman Mamedov wrote:
> On Wed, 11 Nov 2015 20:45:30 -0600
> Ram Ramesh <rramesh2400@gmail.com> wrote:
>
>> ports are on ASMedia ASM 1061 chipset which apparently is not supported
>> on linux per what I can find on various comments/information from the net
> What.
>
> http://superuser.com/questions/363144/is-asmedia-asm1061-sata3-controller-supported-under-linux
>
>
Roman,
Thanks. I did see that. But, in newegg, there are reviews dated much
later than above complaining about ubuntu 14.04 installs failing with
ASMedia enabled. So, I concluded that it is iffy. I could not see any
consistent message of positive result. In fact, the one you quote (Nov
11) is about the only positve I could find. Many others, (more recent),
are not that positive. This worried me. Have you used it or some one
who you know did and it worked fine? If so, please let me know the linux
distro that is successful.
Thanks
Ramesh
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Jan Kara @ 2015-11-12 12:59 UTC (permalink / raw)
To: Baolin Wang
Cc: Jan Kara, Christoph Hellwig, axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Arnd Bergmann,
Garg, Dinesh, LKML
In-Reply-To: <CAMz4kuLmGfyae=PXb8keuCpkEFDdXZzvuBtUiJHRAz0P55mxXg@mail.gmail.com>
On Thu 12-11-15 20:51:10, Baolin Wang wrote:
> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> >> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
> >> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> >> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> >> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> >> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> >> >> >> one bio must use the sequential physical address and it also has a limitation
> >> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> >> >> >> hardware support the big block data encryption.
> >> >> >> >>
> >> >> >> >> This patch series introduc the 'based-request' method to handle the data
> >> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
> >> >> >> >> handle big block data to improve the efficiency.
> >> >> >> >
> >> >> >> > NAK for more request based stacking or DM drivers. They are a major
> >> >> >> > pain to deal with, and adding more with different requirements then
> >> >> >> > dm-multipath is not helping in actually making that one work properly.
> >> >> >>
> >> >> >> But now many vendors supply the hardware engine to handle the
> >> >> >> encyrtion/decryption. The hardware really need a big block to indicate
> >> >> >> its performance with request based things. Another thing is now the
> >> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
> >> >> >> so on) to improve their performance and there's a real performance
> >> >> >> requirement here (I can show the performance result later).
> >> >> >
> >> >> > So you've mentioned several times that hardware needs big blocks. How big
> >> >> > those blocks need to be? Ideally, can you give some numbers on how the
> >> >> > throughput of the encryption hw grows with the block size?
> >> >>
> >> >> It depends on the hardware design. My beaglebone black board's AES
> >> >> engine can handle 1M at one time which is not big. As I know some
> >> >> other AES engine can handle 16M data at one time or more.
> >> >
> >> > Well, one question is "can handle" and other question is how big gain in
> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
> >> > constant overhead to issue a request to the crypto hw and by the time it is
> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
> >> > the encryption itself which is in principle linear in the size of the
> >> > block. That's why I'd like to get idea of the real numbers...
> >>
> >> Please correct me if I misunderstood your point. Let's suppose the AES
> >> engine can handle 16M at one time. If we give the size of data is less
> >> than 16M, the engine can handle it at one time. But if the data size
> >> is 20M (more than 16M), the engine driver will split the data with 16M
> >> and 4M to deal with. I can not say how many numbers, but I think the
> >> engine is like to big chunks than small chunks which is the hardware
> >> engine's advantage.
> >
> > No, I meant something different. I meant that if HW can encrypt 1M in say
> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> > gives you some advantage it becomes diminishingly small.
> >
>
> But if it encrypts 16M with 1M one by one, it will be much more than
> 16.05ms (should be consider the SW submits bio one by one).
Really? In my example, it would take 16.8 ms if we encrypted 16M in 1M
chunks and 16.05 ms if done in one chunk. That is a difference for which I
would not be willing to bend over backwards. Now these numbers are
completely made up and that's why I wanted to see the real numbers...
> >> >> > You mentioned that you use requests because of size limitations on bios - I
> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> >> > struct bio_vec. Is that not enough?
> >> >>
> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> >> or some other small chunks. But request can combine some sequential
> >> >> small bios to be a big block and it is better than bio at least.
> >> >
> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> >> > you check it?
> >>
> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
> >
> > On 4.3 it is 1 MB (which should be enough because requests are limited to
> > 512 KB by default anyway). Previously the maximum bio size depended on the
> > queue parameters such as max number of segments etc.
>
> But it maybe not enough for HW engine which can handle maybe 10M/20M
> at one time.
Currently, you would not be able to create larger than 512K / 1M chunks
even with request based dm-crypt since requests have limits on number of
data they can carry as well... So this is kind of abstract discussion.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Arnd Bergmann @ 2015-11-12 12:57 UTC (permalink / raw)
To: Baolin Wang
Cc: Jan Kara, Christoph Hellwig, axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Garg, Dinesh, LKML
In-Reply-To: <CAMz4kuLmGfyae=PXb8keuCpkEFDdXZzvuBtUiJHRAz0P55mxXg@mail.gmail.com>
On Thursday 12 November 2015 20:51:10 Baolin Wang wrote:
> On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> >> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> >> > Well, one question is "can handle" and other question is how big gain in
> >> > throughput it will bring compared to say 1M chunks. I suppose there's some
> >> > constant overhead to issue a request to the crypto hw and by the time it is
> >> > encrypting 1M it may be that this overhead is well amortized by the cost of
> >> > the encryption itself which is in principle linear in the size of the
> >> > block. That's why I'd like to get idea of the real numbers...
> >>
> >> Please correct me if I misunderstood your point. Let's suppose the AES
> >> engine can handle 16M at one time. If we give the size of data is less
> >> than 16M, the engine can handle it at one time. But if the data size
> >> is 20M (more than 16M), the engine driver will split the data with 16M
> >> and 4M to deal with. I can not say how many numbers, but I think the
> >> engine is like to big chunks than small chunks which is the hardware
> >> engine's advantage.
> >
> > No, I meant something different. I meant that if HW can encrypt 1M in say
> > 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> > gives you some advantage it becomes diminishingly small.
> >
>
> But if it encrypts 16M with 1M one by one, it will be much more than
> 16.05ms (should be consider the SW submits bio one by one).
The example that Jan gave was meant to illustrate the case where it's not
much more than 16.05ms, just slightly more.
The point is that we need real numbers to show at what size we stop
getting significant returns from increased block sizes.
> >> >> > You mentioned that you use requests because of size limitations on bios - I
> >> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> >> > struct bio_vec. Is that not enough?
> >> >>
> >> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> >> or some other small chunks. But request can combine some sequential
> >> >> small bios to be a big block and it is better than bio at least.
> >> >
> >> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> >> > you check it?
> >>
> >> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
> >
> > On 4.3 it is 1 MB (which should be enough because requests are limited to
> > 512 KB by default anyway). Previously the maximum bio size depended on the
> > queue parameters such as max number of segments etc.
>
> But it maybe not enough for HW engine which can handle maybe 10M/20M
> at one time.
Given that you have already done measurements, can you find out how much
you lose in overall performance with your existing patch if you artificially
limit the maximum size to sizes like 256kb, 1MB, 4MB, ...?
Arnd
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-11-12 12:51 UTC (permalink / raw)
To: Jan Kara
Cc: Christoph Hellwig, axboe, Alasdair G Kergon, Mike Snitzer,
dm-devel, neilb, tj, jmoyer, keith.busch, bart.vanassche,
linux-raid, Mark Brown, Arnd Bergmann, Garg, Dinesh, LKML
In-Reply-To: <20151112122400.GB27454@quack.suse.cz>
On 12 November 2015 at 20:24, Jan Kara <jack@suse.cz> wrote:
> On Thu 12-11-15 19:46:26, Baolin Wang wrote:
>> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
>> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
>> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
>> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
>> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
>> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
>> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
>> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
>> >> >> >> one bio must use the sequential physical address and it also has a limitation
>> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
>> >> >> >> hardware support the big block data encryption.
>> >> >> >>
>> >> >> >> This patch series introduc the 'based-request' method to handle the data
>> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
>> >> >> >> handle big block data to improve the efficiency.
>> >> >> >
>> >> >> > NAK for more request based stacking or DM drivers. They are a major
>> >> >> > pain to deal with, and adding more with different requirements then
>> >> >> > dm-multipath is not helping in actually making that one work properly.
>> >> >>
>> >> >> But now many vendors supply the hardware engine to handle the
>> >> >> encyrtion/decryption. The hardware really need a big block to indicate
>> >> >> its performance with request based things. Another thing is now the
>> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
>> >> >> so on) to improve their performance and there's a real performance
>> >> >> requirement here (I can show the performance result later).
>> >> >
>> >> > So you've mentioned several times that hardware needs big blocks. How big
>> >> > those blocks need to be? Ideally, can you give some numbers on how the
>> >> > throughput of the encryption hw grows with the block size?
>> >>
>> >> It depends on the hardware design. My beaglebone black board's AES
>> >> engine can handle 1M at one time which is not big. As I know some
>> >> other AES engine can handle 16M data at one time or more.
>> >
>> > Well, one question is "can handle" and other question is how big gain in
>> > throughput it will bring compared to say 1M chunks. I suppose there's some
>> > constant overhead to issue a request to the crypto hw and by the time it is
>> > encrypting 1M it may be that this overhead is well amortized by the cost of
>> > the encryption itself which is in principle linear in the size of the
>> > block. That's why I'd like to get idea of the real numbers...
>>
>> Please correct me if I misunderstood your point. Let's suppose the AES
>> engine can handle 16M at one time. If we give the size of data is less
>> than 16M, the engine can handle it at one time. But if the data size
>> is 20M (more than 16M), the engine driver will split the data with 16M
>> and 4M to deal with. I can not say how many numbers, but I think the
>> engine is like to big chunks than small chunks which is the hardware
>> engine's advantage.
>
> No, I meant something different. I meant that if HW can encrypt 1M in say
> 1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
> gives you some advantage it becomes diminishingly small.
>
But if it encrypts 16M with 1M one by one, it will be much more than
16.05ms (should be consider the SW submits bio one by one).
>> >> > You mentioned that you use requests because of size limitations on bios - I
>> >> > had a look and current struct bio can easily describe 1MB requests (that's
>> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
>> >> > struct bio_vec. Is that not enough?
>> >>
>> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
>> >> or some other small chunks. But request can combine some sequential
>> >> small bios to be a big block and it is better than bio at least.
>> >
>> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
>> > you check it?
>>
>> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
>
> On 4.3 it is 1 MB (which should be enough because requests are limited to
> 512 KB by default anyway). Previously the maximum bio size depended on the
> queue parameters such as max number of segments etc.
>
But it maybe not enough for HW engine which can handle maybe 10M/20M
at one time.
> Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Jan Kara @ 2015-11-12 12:24 UTC (permalink / raw)
To: Baolin Wang
Cc: Jan Kara, Christoph Hellwig, axboe, Alasdair G Kergon,
Mike Snitzer, dm-devel, neilb, tj, jmoyer, keith.busch,
bart.vanassche, linux-raid, Mark Brown, Arnd Bergmann,
Garg, Dinesh, LKML
In-Reply-To: <CAMz4kuKmU8E3O6pvE0VXGzJ53uobFzGkNYien0_OXXBssafo3Q@mail.gmail.com>
On Thu 12-11-15 19:46:26, Baolin Wang wrote:
> On 12 November 2015 at 19:06, Jan Kara <jack@suse.cz> wrote:
> > On Thu 12-11-15 17:40:59, Baolin Wang wrote:
> >> On 12 November 2015 at 17:17, Jan Kara <jack@suse.cz> wrote:
> >> > On Thu 12-11-15 10:15:32, Baolin Wang wrote:
> >> >> On 11 November 2015 at 17:48, Christoph Hellwig <hch@infradead.org> wrote:
> >> >> > On Wed, Nov 11, 2015 at 05:31:43PM +0800, Baolin Wang wrote:
> >> >> >> Now the dm-crypt code only implemented the 'based-bio' method to encrypt/
> >> >> >> decrypt block data, which can only hanle one bio at one time. As we know,
> >> >> >> one bio must use the sequential physical address and it also has a limitation
> >> >> >> of length. Thus it may limit the big block encyrtion/decryption when some
> >> >> >> hardware support the big block data encryption.
> >> >> >>
> >> >> >> This patch series introduc the 'based-request' method to handle the data
> >> >> >> encryption/decryption. One request can contain multiple bios, so it can
> >> >> >> handle big block data to improve the efficiency.
> >> >> >
> >> >> > NAK for more request based stacking or DM drivers. They are a major
> >> >> > pain to deal with, and adding more with different requirements then
> >> >> > dm-multipath is not helping in actually making that one work properly.
> >> >>
> >> >> But now many vendors supply the hardware engine to handle the
> >> >> encyrtion/decryption. The hardware really need a big block to indicate
> >> >> its performance with request based things. Another thing is now the
> >> >> request based things is used by many vendors (Qualcomm, Spreadtrum and
> >> >> so on) to improve their performance and there's a real performance
> >> >> requirement here (I can show the performance result later).
> >> >
> >> > So you've mentioned several times that hardware needs big blocks. How big
> >> > those blocks need to be? Ideally, can you give some numbers on how the
> >> > throughput of the encryption hw grows with the block size?
> >>
> >> It depends on the hardware design. My beaglebone black board's AES
> >> engine can handle 1M at one time which is not big. As I know some
> >> other AES engine can handle 16M data at one time or more.
> >
> > Well, one question is "can handle" and other question is how big gain in
> > throughput it will bring compared to say 1M chunks. I suppose there's some
> > constant overhead to issue a request to the crypto hw and by the time it is
> > encrypting 1M it may be that this overhead is well amortized by the cost of
> > the encryption itself which is in principle linear in the size of the
> > block. That's why I'd like to get idea of the real numbers...
>
> Please correct me if I misunderstood your point. Let's suppose the AES
> engine can handle 16M at one time. If we give the size of data is less
> than 16M, the engine can handle it at one time. But if the data size
> is 20M (more than 16M), the engine driver will split the data with 16M
> and 4M to deal with. I can not say how many numbers, but I think the
> engine is like to big chunks than small chunks which is the hardware
> engine's advantage.
No, I meant something different. I meant that if HW can encrypt 1M in say
1.05 ms and it can encrypt 16M in 16.05 ms, then although using 16 M blocks
gives you some advantage it becomes diminishingly small.
> >> > You mentioned that you use requests because of size limitations on bios - I
> >> > had a look and current struct bio can easily describe 1MB requests (that's
> >> > assuming 64-bit architecture, 4KB pages) when we have 1 page worth of
> >> > struct bio_vec. Is that not enough?
> >>
> >> Usually one bio does not always use the full 1M, maybe some 1k/2k/8k
> >> or some other small chunks. But request can combine some sequential
> >> small bios to be a big block and it is better than bio at least.
> >
> > As Christoph mentions 4.3 should be better in submitting larger bios. Did
> > you check it?
>
> I'm sorry I didn't check it. What's the limitation of one bio on 4.3?
On 4.3 it is 1 MB (which should be enough because requests are limited to
512 KB by default anyway). Previously the maximum bio size depended on the
queue parameters such as max number of segments etc.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox