[cocci] Merging results from parallel data processing?

[cocci] Merging results from parallel data processing?

[Markus Elfring]

Hello,

A programming interface is used by the Coccinelle software for parallel data processing.
https://gitlab.inria.fr/search?search=Parmap.parfold&nav_source=navbar&project_id=18465&group_id=10912&search_code=true&repository_ref=master
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/enter.ml#L1283-1299

Corresponding results are concatenated so far.
I would appreciate if data can occasionally be combined by other algorithms.

I imagine that unique constraints can be taken better into account accordingly.
Some information would be usable as key data on demand then, wouldn't it?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Julia Lawall]

On Sat, 9 May 2026, Markus Elfring wrote:

> Hello,
>
> A programming interface is used by the Coccinelle software for parallel data processing.
> https://gitlab.inria.fr/search?search=Parmap.parfold&nav_source=navbar&project_id=18465&group_id=10912&search_code=true&repository_ref=master
> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/enter.ml#L1283-1299
>
> Corresponding results are concatenated so far.
> I would appreciate if data can occasionally be combined by other algorithms.
>
> I imagine that unique constraints can be taken better into account accordingly.
> Some information would be usable as key data on demand then, wouldn't it?

I don't know exactly what you are reffing to, but you can for example,
concatenation the set of hash tables into a list of hash tables and then
reprocess the data however you want.  I don't think that Coccinelle can
anticipate what structure the user will want.  Making a list preserves all
the information.

julia

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> A programming interface is used by the Coccinelle software for parallel data processing.
>> https://gitlab.inria.fr/search?search=Parmap.parfold&nav_source=navbar&project_id=18465&group_id=10912&search_code=true&repository_ref=master
>> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/enter.ml#L1283-1299
>>
>> Corresponding results are concatenated so far.
>> I would appreciate if data can occasionally be combined by other algorithms.
>>
>> I imagine that unique constraints can be taken better into account accordingly.
>> Some information would be usable as key data on demand then, wouldn't it?
> 
> I don't know exactly what you are reffing to,

???


>                                               but you can for example,
> concatenation the set of hash tables into a list of hash tables

* Would you like to distinguish item concatenation better from
  the more advanced application of a merge algorithm?

* Where is the information presented and explained that such data structures
  would be involved here?


>                                                                 and then
> reprocess the data however you want.  I don't think that Coccinelle can
> anticipate what structure the user will want.

I hope that data processing requirements can be clarified accordingly.


>                                                Making a list preserves all
> the information.
Will any other structures become more appropriate for data merging?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> * Would you like to distinguish item concatenation better from
>>   the more advanced application of a merge algorithm?
> 
> There is no more advanced applicationof the merge algorithm.  There is
> only one merge algorithm.  It does only one thing.

How do you think about to point the circumstances out under which this one
would be applied?


>> * Where is the information presented and explained that such data structures
>>   would be involved here?
> 
> What data structures?

Would you become more interested in the clarification of corresponding case distinctions?


>> I hope that data processing requirements can be clarified accordingly.
> 
> The requirements are whatever requirements you have.
Do you care for the ordering of generated data anyhow?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> How do you think about to point the circumstances out under which this one
>> would be applied?
> 
> When the semantic patch requests it.  Please see tests/merge_vars.cocci or
> merco_vars_python.cocci

Thanks for another hint.

https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars.cocci
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars_python.cocci

Do you distinguish “merging” from data concatenation in such code?


>> Would you become more interested in the clarification of corresponding case distinctions?
> 
> There are no case distinctions.

How will such a view evolve further according to the use case “parallel data processing”?


>> Do you care for the ordering of generated data anyhow?
> 
> No.
I find this feedback strange.
Data users tend to prefer some information ordering, don't they?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Julia Lawall]

[-- Attachment #1: Type: text/plain, Size: 1640 bytes --]



On Sun, 10 May 2026, Markus Elfring wrote:

> >> How do you think about to point the circumstances out under which this one
> >> would be applied?
> >
> > When the semantic patch requests it.  Please see tests/merge_vars.cocci or
> > merco_vars_python.cocci
>
> Thanks for another hint.
>
> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars.cocci
> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars_python.cocci

No idea what is the point of these links.

> Do you distinguish “merging” from data concatenation in such code?

No idea what this question means.

You can declare variable in OCaml or Python to be the result of
concatenating the results from the different threads.  There is nothing
more to discuss about it.

It's true that the above tests are overly simplistic.  Maybe
tests/names.cocci would be a better example.  But it doesn't have a python
counterpart.

>
> >> Would you become more interested in the clarification of corresponding case distinctions?
> >
> > There are no case distinctions.
>
> How will such a view evolve further according to the use case “parallel data processing”?

No idea what this means.

>
> >> Do you care for the ordering of generated data anyhow?
> >
> > No.
> I find this feedback strange.
> Data users tend to prefer some information ordering, don't they?

You look at the data and order it however you want.  The user has no
control over what files get handled by what tasks, so it doesn't make
sense for Coccinelle to provide any ordering guarantees.

julia

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars.cocci
>> https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/merge_vars_python.cocci
> 
> No idea what is the point of these links.

You offered another bit of background information.

I find it still unclear how much these code examples can help with the desired
clarification for parallel data processing.
They are not referenced by the SmPL manual so far.


>> Do you distinguish “merging” from data concatenation in such code?
> 
> No idea what this question means.

???


> You can declare variable in OCaml or Python to be the result of
> concatenating the results from the different threads.

Would an other algorithm than “concatenation” be occasionally preferred here?


>                                                        There is nothing
> more to discuss about it.
> 
> It's true that the above tests are overly simplistic.  Maybe
> tests/names.cocci would be a better example.  But it doesn't have a python
> counterpart.

Will such information trigger further collateral evolution?


>> How will such a view evolve further according to the use case “parallel data processing”?
> 
> No idea what this means.

???


>> Data users tend to prefer some information ordering, don't they?
> 
> You look at the data and order it however you want.

Will additional development approaches be needed to achieve a preferred order?


>                                                      The user has no
> control over what files get handled by what tasks, so it doesn't make
> sense for Coccinelle to provide any ordering guarantees.
I came along further application requirements through the years.
Data are usually processed by the involved computation units in a known way,
aren't they?

Do you distinguish tasks from threads (or background processes) anyhow?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

> You can declare variable in OCaml or Python to be the result of
> concatenating the results from the different threads.  There is nothing
> more to discuss about it.
> 
> It's true that the above tests are overly simplistic.  Maybe
> tests/names.cocci would be a better example.  But it doesn't have a python
> counterpart.

How should users of the Coccinelle software get to know
that such a special programming interface is supported according to
parallel data processing anyhow?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> How should users of the Coccinelle software get to know
>> that such a special programming interface is supported according to
>> parallel data processing anyhow?
> 
> It seems that Thierry should have written some documentation for the
> manual when he implemented it.  Since he did not, someone probably should.
> Perhaps all the effort you have put into complaining about it could have
> been used for that.
Are there any agreements needed for involved technical details
before a better software documentation can be attempted?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> Are there any agreements needed for involved technical details
>> before a better software documentation can be attempted?
> 
> No.

Will development views evolve further accordingly?

How many contributors can achieve more desirable adjustments
also for the affected documentation?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[Julia Lawall]

On Sun, 10 May 2026, Markus Elfring wrote:

> >> Are there any agreements needed for involved technical details
> >> before a better software documentation can be attempted?
> >
> > No.
>
> Will development views evolve further accordingly?

No/.  I see no reason for the default behavior to change.  It works fine,
and Coccinelle has no more information about what the user wants.

julia

>
> How many contributors can achieve more desirable adjustments
> also for the affected documentation?
>
> Regards,
> Markus
>

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

>> Will development views evolve further accordingly?
> No/.  I see no reason for the default behavior to change.

Will further development concerns become more interesting also for this use case?


>                                                            It works fine,

Partly, yes.


> and Coccinelle has no more information about what the user wants.

Do you find the presented clarification approach improvable?

Regards,
Markus

Re: [cocci] Merging results from parallel data processing?

[loopily_]

Good day all,

>> Will development views evolve further accordingly?
> No/.  I see no reason for the default behavior to change.  It works fine,
> and Coccinelle has no more information about what the user wants.

https://boinc.berkeley.edu/forum_thread.php?id=15619&postid=117251

Thank you very much.

Re: [cocci] Merging results from parallel data processing?

[Markus Elfring]

…
> It's true that the above tests are overly simplistic.  Maybe
> tests/names.cocci would be a better example.…

Under which circumstances will the chances grow to explain an information context
like “merge” better (also according to the mentioned SmPL script)?
https://gitlab.inria.fr/coccinelle/coccinelle/-/blob/03a1af0a3316ddfc2b54d827b8b9627a841c85c9/tests/names.cocci#L25-31

https://en.wikipedia.org/wiki/Test_case_(software)#Typical_written_test_case_format

Regards,
Markus