Re: Sets update

["Kerin Millar" <kfm@plushkava.net>]

On Sun, 21 Jul 2024, at 3:38 PM, Slavko wrote:
> Hi,
>
> Dňa 21. júla 2024 12:29:44 UTC používateľ Kerin Millar 
> <kfm@plushkava.net> napísal:
>
>>You have a point. If on a severely RAM-constrained system, you may prefer to avoid the use of asynchronous pipelines and generate temporary files instead. Doing so would minimise RAM consumption.
>
> Correct me, if i am wrong, please...
>
> AFAIK memory is not problem in pipes. Yes, any pipe consumes some
> memory, but AFAIK it is somewhat limited (i don't know exact numbers)
> and if full, the writing process is paused, thus stops to produce more
> output, and thus not consumes more memory.

Yes, the default size of a pipe buffer is just 16 pages in Linux. However, one must also take into account that pipelines are fundamentally asynchronous. That is, all of the commands that comprise a shell pipeline are executed at approximately the same time and run in parallel to one another. In that particular respect, there is a potential bearing on memory consumption. I must emphasise that I am definitely not trying to discourage the use of pipelines - far from it! I only brought it up because I had the impression that you might be contending with a system that is severely short on RAM, in which case running programs serially can be helpful as a deliberate memory-saving measure.

>
> On other side, nowadays linux systems uses tmpfs for /tmp, thus storing
> output in temp file can be even worse (in mean of RAM) than using pipes,
> as whole output is in that file. Storing temporary file on disk is possible too,
> but doing that too often can drain Flash's lifetime, as modern Flash storage
> are really limited in that (in comparison with magnetic disks). IIRC the disk
> was used for pipes in old Unixes...

It depends on the exact circumstances but yes, it could easily be worse. If you have a mktemp(1) utility, it is probable that it responds to the TMPDIR variable, in which case you are free to point it to something other than a tmpfs-backed filesystem such as /var/tmp.

>
> But yes, without temp file, one cannot produce the same IP list multiple
> times (reading it into variable is the same), but at least second temp file
> have to be avoided...

It is not quite the same but I understand your point. Assuming that TMPDIR is backed by a tmpfs, both cases would result in non-disk-backed pages being allocated that may later be paged out to a swap - if any - under sufficient memory pressure.

--
Kerin Millar