PG: all requests stuck when acting set < min_size

PG: all requests stuck when acting set < min_size

[GuangYang]

Hi there,
Is there any reason we stuck read only requests as well for a PG when the acting set size is less than min_size?

Thanks,
Guang 		 	   		  

Re: PG: all requests stuck when acting set < min_size

[Gregory Farnum]

On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
> Hi there,
> Is there any reason we stuck read only requests as well for a PG when the acting set size is less than min_size?

A few.
The most important reason: PGs don't have any concept of a read-only
mode in the code. They are "active" or not, and an active PG handles
writes. (The full flags and other things which block writes but allow
reads are at the OSD level, not the PG level, and are handled when ops
come in before they reach the PG.) Allowing read requests against a PG
to complete even when we aren't taking writes on a per-PG level would
take some doing.
Also: it would be weird from several different levels. We'd need to
keep track of client streams because we wouldn't want to let through a
read that is ordered after a write. How would we handle the memory
pressure implied by that? While I can imagine it being useful for some
stuff like RGW reads, in general making data available for read but
not write is a pretty complicated thing to explain to users — how do
we expose that in a useful way?
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PG: all requests stuck when acting set < min_size

[Samuel Just]

Actually, we really can't accept reads below min_size and still keep
the properties we want it to have.  Suppose we have 3 osds (a, b, and
c) which see writes 0...1000.  min_size is 2.  If a and b are then
powered off only having committed up to 900 (therefore the client
could only have seen up to 900 commit), then c would be able to serve
reads based on updates up to 1000 with a and b stopped (no way to know
a and b only committed to 900).  If c then stops and a and b are
restarted, they would begin serving reads and writes only based on
commits up to 900 even though we would have exposed the writes up to
1000 to the client.
-Sam

On Tue, Oct 27, 2015 at 12:47 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
>> Hi there,
>> Is there any reason we stuck read only requests as well for a PG when the acting set size is less than min_size?
>
> A few.
> The most important reason: PGs don't have any concept of a read-only
> mode in the code. They are "active" or not, and an active PG handles
> writes. (The full flags and other things which block writes but allow
> reads are at the OSD level, not the PG level, and are handled when ops
> come in before they reach the PG.) Allowing read requests against a PG
> to complete even when we aren't taking writes on a per-PG level would
> take some doing.
> Also: it would be weird from several different levels. We'd need to
> keep track of client streams because we wouldn't want to let through a
> read that is ordered after a write. How would we handle the memory
> pressure implied by that? While I can imagine it being useful for some
> stuff like RGW reads, in general making data available for read but
> not write is a pretty complicated thing to explain to users — how do
> we expose that in a useful way?
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: PG: all requests stuck when acting set < min_size

[GuangYang]

Thanks Greg.

----------------------------------------
> Date: Tue, 27 Oct 2015 12:47:34 -0700
> Subject: Re: PG: all requests stuck when acting set < min_size
> From: gfarnum@redhat.com
> To: yguang11@outlook.com
> CC: ceph-devel@vger.kernel.org
>
> On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
>> Hi there,
>> Is there any reason we stuck read only requests as well for a PG when the acting set size is less than min_size?
>
> A few.
> The most important reason: PGs don't have any concept of a read-only
> mode in the code. They are "active" or not, and an active PG handles
> writes. (The full flags and other things which block writes but allow
> reads are at the OSD level, not the PG level, and are handled when ops
> come in before they reach the PG.) Allowing read requests against a PG
> to complete even when we aren't taking writes on a per-PG level would
> take some doing.
[yguang] Yeah that is what I thought as well, I didn't see a read-only PG state, they are either active or inactive (incomplete).
> Also: it would be weird from several different levels. We'd need to
> keep track of client streams because we wouldn't want to let through a
> read that is ordered after a write. How would we handle the memory
> pressure implied by that? 
[yguang] I am not aware of this part, could you elaborate a little bit?
While I can imagine it being useful for some
> stuff like RGW reads, in general making data available for read but
> not write is a pretty complicated thing to explain to users — how do
> we expose that in a useful way?
[yguang] The motivation is to improve availability upon failures (yay we are using radosgw:), and some use cases of object storage like S3,
most commonly object are written once, read many, and this could help with the availability.
> -Greg
 		 	   		  --
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: PG: all requests stuck when acting set < min_size

[GuangYang]

Thanks Sam.

----------------------------------------
> Date: Tue, 27 Oct 2015 14:05:42 -0700
> Subject: Re: PG: all requests stuck when acting set < min_size
> From: sjust@redhat.com
> To: gfarnum@redhat.com
> CC: yguang11@outlook.com; ceph-devel@vger.kernel.org
>
> Actually, we really can't accept reads below min_size and still keep
> the properties we want it to have. Suppose we have 3 osds (a, b, and
> c) which see writes 0...1000. min_size is 2. If a and b are then
> powered off only having committed up to 900 (therefore the client
> could only have seen up to 900 commit), then c would be able to serve
> reads based on updates up to 1000 with a and b stopped (no way to know
> a and b only committed to 900). If c then stops and a and b are
> restarted, they would begin serving reads and writes only based on
> commits up to 900 even though we would have exposed the writes up to
> 1000 to the client.
[yguang] Thanks for the example, that is true. What about for EC pool? Looks like for EC pool we don't have this problem.
> -Sam
>
> On Tue, Oct 27, 2015 at 12:47 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
>>> Hi there,
>>> Is there any reason we stuck read only requests as well for a PG when the acting set size is less than min_size?
>>
>> A few.
>> The most important reason: PGs don't have any concept of a read-only
>> mode in the code. They are "active" or not, and an active PG handles
>> writes. (The full flags and other things which block writes but allow
>> reads are at the OSD level, not the PG level, and are handled when ops
>> come in before they reach the PG.) Allowing read requests against a PG
>> to complete even when we aren't taking writes on a per-PG level would
>> take some doing.
>> Also: it would be weird from several different levels. We'd need to
>> keep track of client streams because we wouldn't want to let through a
>> read that is ordered after a write. How would we handle the memory
>> pressure implied by that? While I can imagine it being useful for some
>> stuff like RGW reads, in general making data available for read but
>> not write is a pretty complicated thing to explain to users ― how do
>> we expose that in a useful way?
>> -Greg
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
 		 	   		  

Re: PG: all requests stuck when acting set < min_size

[Brad Hubbard]

----- Original Message -----
> From: "Samuel Just" <sjust@redhat.com>
> To: "Gregory Farnum" <gfarnum@redhat.com>
> Cc: "GuangYang" <yguang11@outlook.com>, ceph-devel@vger.kernel.org
> Sent: Wednesday, 28 October, 2015 7:05:42 AM
> Subject: Re: PG: all requests stuck when acting set < min_size
> 
> Actually, we really can't accept reads below min_size and still keep
> the properties we want it to have.  Suppose we have 3 osds (a, b, and
> c) which see writes 0...1000.  min_size is 2.  If a and b are then
> powered off only having committed up to 900 (therefore the client
> could only have seen up to 900 commit), then c would be able to serve
> reads based on updates up to 1000 with a and b stopped (no way to know
> a and b only committed to 900).  If c then stops and a and b are
> restarted, they would begin serving reads and writes only based on
> commits up to 900 even though we would have exposed the writes up to
> 1000 to the client.

If a and/or b then accept a write you have a recipe for split-brain and no one
wants to see that in Ceph.

Cheers,
Brad

> -Sam
> 
> On Tue, Oct 27, 2015 at 12:47 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> > On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
> >> Hi there,
> >> Is there any reason we stuck read only requests as well for a PG when the
> >> acting set size is less than min_size?
> >
> > A few.
> > The most important reason: PGs don't have any concept of a read-only
> > mode in the code. They are "active" or not, and an active PG handles
> > writes. (The full flags and other things which block writes but allow
> > reads are at the OSD level, not the PG level, and are handled when ops
> > come in before they reach the PG.) Allowing read requests against a PG
> > to complete even when we aren't taking writes on a per-PG level would
> > take some doing.
> > Also: it would be weird from several different levels. We'd need to
> > keep track of client streams because we wouldn't want to let through a
> > read that is ordered after a write. How would we handle the memory
> > pressure implied by that? While I can imagine it being useful for some
> > stuff like RGW reads, in general making data available for read but
> > not write is a pretty complicated thing to explain to users — how do
> > we expose that in a useful way?
> > -Greg
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PG: all requests stuck when acting set < min_size

[Samuel Just]

It's actually worse for an ec pool.  EC pools need to be able to roll
back divergent entries in order to keep the durability requirements
implied by min_size.  If you have 10 data blocks and 4 parity blocks
with min_size set to 12, then one would expect that as long as one can
recover any 12 shards, you can always recover the pg regardless of
intermediate pg degradation states.  However, if we accept a read with
only 10 replicas, the most recent log entries cannot ever be
considered divergent and we might become unable to recover without
exactly those 10 osds if the other 4 happened to not have committed
those updates.
-Sam

On Tue, Oct 27, 2015 at 5:19 PM, Brad Hubbard <bhubbard@redhat.com> wrote:
> ----- Original Message -----
>> From: "Samuel Just" <sjust@redhat.com>
>> To: "Gregory Farnum" <gfarnum@redhat.com>
>> Cc: "GuangYang" <yguang11@outlook.com>, ceph-devel@vger.kernel.org
>> Sent: Wednesday, 28 October, 2015 7:05:42 AM
>> Subject: Re: PG: all requests stuck when acting set < min_size
>>
>> Actually, we really can't accept reads below min_size and still keep
>> the properties we want it to have.  Suppose we have 3 osds (a, b, and
>> c) which see writes 0...1000.  min_size is 2.  If a and b are then
>> powered off only having committed up to 900 (therefore the client
>> could only have seen up to 900 commit), then c would be able to serve
>> reads based on updates up to 1000 with a and b stopped (no way to know
>> a and b only committed to 900).  If c then stops and a and b are
>> restarted, they would begin serving reads and writes only based on
>> commits up to 900 even though we would have exposed the writes up to
>> 1000 to the client.
>
> If a and/or b then accept a write you have a recipe for split-brain and no one
> wants to see that in Ceph.
>
> Cheers,
> Brad
>
>> -Sam
>>
>> On Tue, Oct 27, 2015 at 12:47 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> > On Tue, Oct 27, 2015 at 11:47 AM, GuangYang <yguang11@outlook.com> wrote:
>> >> Hi there,
>> >> Is there any reason we stuck read only requests as well for a PG when the
>> >> acting set size is less than min_size?
>> >
>> > A few.
>> > The most important reason: PGs don't have any concept of a read-only
>> > mode in the code. They are "active" or not, and an active PG handles
>> > writes. (The full flags and other things which block writes but allow
>> > reads are at the OSD level, not the PG level, and are handled when ops
>> > come in before they reach the PG.) Allowing read requests against a PG
>> > to complete even when we aren't taking writes on a per-PG level would
>> > take some doing.
>> > Also: it would be weird from several different levels. We'd need to
>> > keep track of client streams because we wouldn't want to let through a
>> > read that is ordered after a write. How would we handle the memory
>> > pressure implied by that? While I can imagine it being useful for some
>> > stuff like RGW reads, in general making data available for read but
>> > not write is a pretty complicated thing to explain to users — how do
>> > we expose that in a useful way?
>> > -Greg
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html