What to do?

What to do?

[Nigel Cunningham]

Hi all.

A few months back, I posted TuxOnIce for something like the third time,
seeking to see if we could make some progress on getting it merged. The
outcome of that was an agreement between Rafael and I that we'd work on
getting TuxOnIce functionality (if not the actual code) merged bit by
bit, improving swsusp rather than doing an outright merge.

Unfortunately, as the intervening months have shown, Rafael is busy with
other things and so am I. The few patches I did prepare and post have
had no review, and I haven't had a chance to build on or rework them
since. (I'd actually like them not to be merged, because I've realised
that there are more foundational changes that need to go in first.)

Given that this has been the outcome so far, I see no reason to imagine
that we're going to make any serious progress any time soon.

I'd therefore like to ask whether there's anyone out there who could do
one or more of:

1) Step up to the plate and help improve swsusp, without relying on any
help from me but with my blessing if they choose to copy wholesale bits
of TuxOnIce;
2) Take over maintaining TuxOnIce from me;
3) Some other way forward that hasn't occurred to me.

Thoughts, input, flames, whatever else welcome. A lack of deadly silence
would be especially appreciated :)

Regards,

Nigel

Re: What to do?

[Rafael J. Wysocki]

On Thursday 20 August 2009, Nigel Cunningham wrote:
> Hi all.

Hi,

> A few months back, I posted TuxOnIce for something like the third time,
> seeking to see if we could make some progress on getting it merged. The
> outcome of that was an agreement between Rafael and I that we'd work on
> getting TuxOnIce functionality (if not the actual code) merged bit by
> bit, improving swsusp rather than doing an outright merge.
> 
> Unfortunately, as the intervening months have shown, Rafael is busy with
> other things and so am I. The few patches I did prepare and post have
> had no review, and I haven't had a chance to build on or rework them
> since. (I'd actually like them not to be merged, because I've realised
> that there are more foundational changes that need to go in first.)
> 
> Given that this has been the outcome so far, I see no reason to imagine
> that we're going to make any serious progress any time soon.

That unfortunately is probably right.

The problem for my part is that new things are still appearing that I need to
take care of rather urgently.

In fact the user space part of (u)swsusp also has been suffering from the lack
of attention recently, so to speak.

> I'd therefore like to ask whether there's anyone out there who could do
> one or more of:
> 
> 1) Step up to the plate and help improve swsusp, without relying on any
> help from me but with my blessing if they choose to copy wholesale bits
> of TuxOnIce;
> 2) Take over maintaining TuxOnIce from me;
> 3) Some other way forward that hasn't occurred to me.
> 
> Thoughts, input, flames, whatever else welcome. A lack of deadly silence
> would be especially appreciated :)

Well, the code is out there.  If there's time to try to get it in, I think it's
worth doing.

Best,
Rafael

Re: [TuxOnIce-devel] What to do?

[Xavier Gnata]

Rafael J. Wysocki wrote:
> On Thursday 20 August 2009, Nigel Cunningham wrote:
>   
>> Hi all.
>>     
>
> Hi,
>
>   
>> A few months back, I posted TuxOnIce for something like the third time,
>> seeking to see if we could make some progress on getting it merged. The
>> outcome of that was an agreement between Rafael and I that we'd work on
>> getting TuxOnIce functionality (if not the actual code) merged bit by
>> bit, improving swsusp rather than doing an outright merge.
>>
>> Unfortunately, as the intervening months have shown, Rafael is busy with
>> other things and so am I. The few patches I did prepare and post have
>> had no review, and I haven't had a chance to build on or rework them
>> since. (I'd actually like them not to be merged, because I've realised
>> that there are more foundational changes that need to go in first.)
>>
>> Given that this has been the outcome so far, I see no reason to imagine
>> that we're going to make any serious progress any time soon.
>>     
>
> That unfortunately is probably right.
>
> The problem for my part is that new things are still appearing that I need to
> take care of rather urgently.
>
> In fact the user space part of (u)swsusp also has been suffering from the lack
> of attention recently, so to speak.
>
>   
>> I'd therefore like to ask whether there's anyone out there who could do
>> one or more of:
>>
>> 1) Step up to the plate and help improve swsusp, without relying on any
>> help from me but with my blessing if they choose to copy wholesale bits
>> of TuxOnIce;
>> 2) Take over maintaining TuxOnIce from me;
>> 3) Some other way forward that hasn't occurred to me.
>>
>> Thoughts, input, flames, whatever else welcome. A lack of deadly silence
>> would be especially appreciated :)
>>     
>
> Well, the code is out there.  If there's time to try to get it in, I think it's
> worth doing.
>
> Best,
> Rafael
>   

Well I have no time to jump into the details of the code but I can test 
whatever you want :)

Xavier

Re: What to do?

[Jiri Slaby]

On 08/20/2009 06:22 AM, Nigel Cunningham wrote:
> I'd therefore like to ask whether there's anyone out there who could do
> one or more of:
> 
> 1) Step up to the plate and help improve swsusp, without relying on any
> help from me but with my blessing if they choose to copy wholesale bits
> of TuxOnIce;

Hi, I think I can help here. Is there any TODO or similar resources?

Re: What to do?

[Rafael J. Wysocki]

On Sunday 23 August 2009, Jiri Slaby wrote:
> On 08/20/2009 06:22 AM, Nigel Cunningham wrote:
> > I'd therefore like to ask whether there's anyone out there who could do
> > one or more of:
> > 
> > 1) Step up to the plate and help improve swsusp, without relying on any
> > help from me but with my blessing if they choose to copy wholesale bits
> > of TuxOnIce;
> 
> Hi, I think I can help here.

Great, thanks!

> Is there any TODO or similar resources?

Not that I know of, but I think we should start with the non-controversial
things, like image compression etc.

Best,
Rafael

Re: What to do?

[Nigel Cunningham]

Hi.

Rafael J. Wysocki wrote:
> On Sunday 23 August 2009, Jiri Slaby wrote:
>> On 08/20/2009 06:22 AM, Nigel Cunningham wrote:
>>> I'd therefore like to ask whether there's anyone out there who could do
>>> one or more of:
>>>
>>> 1) Step up to the plate and help improve swsusp, without relying on any
>>> help from me but with my blessing if they choose to copy wholesale bits
>>> of TuxOnIce;
>> Hi, I think I can help here.
> 
> Great, thanks!

Yes, thanks for volunteering, Jiri!

We should start by asking where you're at as far as knowledge of swsusp
and TuxOnIce goes. I know you've been around the TuxOnIce lists a bit,
but don't know how much you know about C programming or the inards of
the kernel, swsusp or TuxOnIce (ie how much help do you need to get up
to speed?).

>> Is there any TODO or similar resources?
> 
> Not that I know of, but I think we should start with the non-controversial
> things, like image compression etc.

No, I have a bit of a mental to-do list, but haven't written anything
down yet. Here's a quick, high-level list (some of these things are big):

- Rework swap allocation (as per below) and freeing.
- Rework i/o to use bmapping and lay the foundations for use multiple
block devices.
- Add modular design (will simplify the following additions)
- Add multithreaded I/O
- Add compression support
- Add support for multiple swap devices
- Add support for generic files (file allocator)
- Do storage allocation etc ('image preparation') prior to the atomic
copy instead of afterwards, making the only remaining variable how much
ram drivers' pm routines need (and that might not be an issue by then).
- Add userspace helper support (storage manager/userui).

I'd really like to start with adding the modular design - it will make
things a lot cleaner, but I think we need to do some disentanglement
first (TuxOnIce has all the support for compression in one file that
simply isn't compiled if we don't want compression support. It would be
good to get swsusp set up like that). This is why I have above the
initial thought of handling doing the swap & compression stuff first.

When I prepared those compression patches earlier, it became apparent
that we really need to begin with reworking how the swap is allocated,
remembered and stored in the image header.

The current swsusp scheme of having a page of swap addresses preceding
the pages that are written doesn't work well for compression, because
when rereading the image, you can only do readahead in batches of that
size. Everything will have to grind to a halt while we wait for the next
'index' page.

In TuxOnIce, we allocate all of the swap at the start of preparing to
hibernate, which has other advantages I won't go into now. The pages
allocated are stored as extents in memory. For doing the actual I/O, we
bmap the swap that's allocated right after allocating it, storing the
sector numbers in extents as well (remembering the block sizes that are
relevant as well). We then use them directly in writing the image,
storing major and minor numbers, block sizes and the extents in the
image header for use at resume time.

I'd suggest that the place to begin would be converting swsusp to use
this sort of scheme.

Regarding the non-controversial vs controversial thing, let's not worry
about that. We should do things in the order that's logical, sensible
and lays the foundation for the improvements we're planning to have
follow. If an improvement is really an improvement, it shouldn't be a
big problem to show that it has technical merits that make it
worthwhile. As always in Linux, if someone wants to nak something
without good reason, they'll just end up getting themselves ignored.

Regards,

Nigel

Re: What to do?

[Rafael J. Wysocki]

On Monday 24 August 2009, Nigel Cunningham wrote:
> Hi.

Hi,

> Rafael J. Wysocki wrote:
> > On Sunday 23 August 2009, Jiri Slaby wrote:
> >> On 08/20/2009 06:22 AM, Nigel Cunningham wrote:
> >>> I'd therefore like to ask whether there's anyone out there who could do
> >>> one or more of:
> >>>
> >>> 1) Step up to the plate and help improve swsusp, without relying on any
> >>> help from me but with my blessing if they choose to copy wholesale bits
> >>> of TuxOnIce;
> >> Hi, I think I can help here.
> > 
> > Great, thanks!
> 
> Yes, thanks for volunteering, Jiri!
> 
> We should start by asking where you're at as far as knowledge of swsusp
> and TuxOnIce goes. I know you've been around the TuxOnIce lists a bit,
> but don't know how much you know about C programming or the inards of
> the kernel, swsusp or TuxOnIce (ie how much help do you need to get up
> to speed?).
> 
> >> Is there any TODO or similar resources?
> > 
> > Not that I know of, but I think we should start with the non-controversial
> > things, like image compression etc.
> 
> No, I have a bit of a mental to-do list, but haven't written anything
> down yet. Here's a quick, high-level list (some of these things are big):
> 
> - Rework swap allocation (as per below) and freeing.
> - Rework i/o to use bmapping and lay the foundations for use multiple
> block devices.
> - Add modular design (will simplify the following additions)
> - Add multithreaded I/O
> - Add compression support
> - Add support for multiple swap devices
> - Add support for generic files (file allocator)
> - Do storage allocation etc ('image preparation') prior to the atomic
> copy instead of afterwards, making the only remaining variable how much
> ram drivers' pm routines need (and that might not be an issue by then).
> - Add userspace helper support (storage manager/userui).
> 
> I'd really like to start with adding the modular design - it will make
> things a lot cleaner, but I think we need to do some disentanglement
> first (TuxOnIce has all the support for compression in one file that
> simply isn't compiled if we don't want compression support. It would be
> good to get swsusp set up like that). This is why I have above the
> initial thought of handling doing the swap & compression stuff first.
> 
> When I prepared those compression patches earlier, it became apparent
> that we really need to begin with reworking how the swap is allocated,
> remembered and stored in the image header.
> 
> The current swsusp scheme of having a page of swap addresses preceding
> the pages that are written doesn't work well for compression, because
> when rereading the image, you can only do readahead in batches of that
> size. Everything will have to grind to a halt while we wait for the next
> 'index' page.
> 
> In TuxOnIce, we allocate all of the swap at the start of preparing to
> hibernate, which has other advantages I won't go into now. The pages
> allocated are stored as extents in memory. For doing the actual I/O, we
> bmap the swap that's allocated right after allocating it, storing the
> sector numbers in extents as well (remembering the block sizes that are
> relevant as well). We then use them directly in writing the image,
> storing major and minor numbers, block sizes and the extents in the
> image header for use at resume time.
> 
> I'd suggest that the place to begin would be converting swsusp to use
> this sort of scheme.
> 
> Regarding the non-controversial vs controversial thing, let's not worry
> about that. We should do things in the order that's logical, sensible
> and lays the foundation for the improvements we're planning to have
> follow. If an improvement is really an improvement, it shouldn't be a
> big problem to show that it has technical merits that make it
> worthwhile. As always in Linux, if someone wants to nak something
> without good reason, they'll just end up getting themselves ignored.

Well, if you start from something that people don't like, it may be difficult
to push the rest.

Still, I think Jiri is more than capable of handling that, so more or less
everything that is a clear improvement and doesn't introduce regressions of any
kind will probably work for me.

Thanks,
Rafael

Re: [TuxOnIce-devel] What to do?

[Nigel Cunningham]

Hi.

Xavier Gnata wrote:
> Well I have no time to jump into the details of the code but I can test
> whatever you want :)

Thanks. Perhaps with Jiri's help we'll have some code for testing sooner
rather than later :)

Regards,

Nigel

Re: What to do?

[Jiri Slaby]

Hi again.

On 08/24/2009 01:20 AM, Rafael J. Wysocki wrote:
> On Monday 24 August 2009, Nigel Cunningham wrote:
>> We should start by asking where you're at as far as knowledge of swsusp
>> and TuxOnIce goes. 

Well, I know swsups a bit and don't know about toi much. Still I think
it's not a problem at all. I learn quickly ;).

>> I know you've been around the TuxOnIce lists a bit,
>> but don't know how much you know about C programming or the inards of
>> the kernel, swsusp or TuxOnIce (ie how much help do you need to get up
>> to speed?).

I think C is no problem at all, so the kernel. The two I mentioned
above. I think I'll need no or low help from your side regarding the
code site. From what I've read already I would say it's well documented.

> Well, if you start from something that people don't like, it may be difficult
> to push the rest.
> 
> Still, I think Jiri is more than capable of handling that, so more or less
> everything that is a clear improvement and doesn't introduce regressions of any
> kind will probably work for me.

Thank you guys. I needed to cope with other things (a suspend
regression, writable /proc/<pid>/limits etc.) which took me a long time,
sorry.

Then I also went through the last "discussion" around the last merge try
mainly to see what others think about kernel-side implementation of
compression etc.

I hope I'll come up with something early.

Thanks.