Re: Multi Core Support for compression in compression.c

[Nick Krause <xerofoify@gmail.com>]

On Mon, Jul 28, 2014 at 11:13 AM, Nick Krause <xerofoify@gmail.com> wrote:
> On Mon, Jul 28, 2014 at 6:10 AM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 07/27/2014 11:21 PM, Nick Krause wrote:
>>> On Sun, Jul 27, 2014 at 10:56 PM, Austin S Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>>>> On 07/27/2014 04:47 PM, Nick Krause wrote:
>>>>> This may be a bad idea , but compression in brtfs seems to be only
>>>>> using one core to compress.
>>>>> Depending on the CPU used and the amount of cores in the CPU we can
>>>>> make this much faster
>>>>> with multiple cores. This seems bad by my reading at least I would
>>>>> recommend for writing compression
>>>>> we write a function to use a certain amount of cores based on the load
>>>>> of the system's CPU not using
>>>>> more then 75% of the system's CPU resources as my system when idle has
>>>>> never needed more
>>>>> then one core of my i5 2500k to run when with interrupts for opening
>>>>> eclipse are running. For reading
>>>>> compression on good core seems fine to me as testing other compression
>>>>> software for reads , it's
>>>>> way less CPU intensive.
>>>>> Cheers Nick
>>>> We would probably get a bigger benefit from taking an approach like
>>>> SquashFS has recently added, that is, allowing multi-threaded
>>>> decompression fro reads, and decompressing directly into the pagecache.
>>>>  Such an approach would likely make zlib compression much more scalable
>>>> on large systems.
>>>>
>>>>
>>>
>>> Austin,
>>> That seems better then my idea as you seem to be more up to date on
>>> brtfs devolopment.
>>> If you and the other developers of brtfs are interested in adding this
>>> as a feature please let
>>> me known as I would like to help improve brtfs as the file system as
>>> an idea is great just
>>> seems like it needs a lot of work :).
>>> Nick
>> I wouldn't say that I am a BTRFS developer (power user maybe?), but I
>> would definitely say that parallelizing compression on writes would be a
>> good idea too (especially for things like lz4, which IIRC is either in
>> 3.16 or in the queue for 3.17).  Both options would be a lot of work,
>> but almost any performance optimization would.  I would almost say that
>> it would provide a bigger performance improvement to get BTRFS to
>> intelligently stripe reads and writes (at the moment, any given worker
>> thread only dispatches one write or read to a single device at a time,
>> and any given write() or read() syscall gets handled by only one worker).
>>
>
> I will look into this idea and see if I can do this for writes.
> Regards Nick

Austin,
Seems since we don't want to release the cache for inodes in order to
improve writes if
are going to use the page cache. We seem to be doing this for writes in
end_compressed_bio_write for standard pages and in end_compressed_bio_write.
If we want to cache write pages why are we removing then ? Seems like this needs
to be removed in order to start off.
Regards Nick