[Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

Hi,

I have looked through the CUPS 3.x planning and hit into the following 
problem:

Filters
-------

CUPS 3.x, independent whether local or sharing server, has to convert 
formats to turn the input format into one of the 4 output formats. So we 
need filter, not a way to add filters, but we need filters.

I am working on cups-filter 2.x for longer time now, with the principal 
intention to better integrate with the New Architecture, of driverless 
IPP-only CUPS and Printer Applications.

Especially now when I was looking about handling dependencies and moving 
PPD support completely only into the retro-fitting Printer Applications 
I have started to change libcupsfilters in a way so that it has no PPD 
support at all, especially also not by conditional compiling.

The filter functions (named "cfFilter...()" will accept printer 
properties only as an IPP message (like the response to a 
get-printer-attributes request) and job options only by an IPP message 
containing job IPP attributes (like they come with jobs via IPP) or a 
CUPS options list data structure.

All PPD handling for retro-fits and for CUPS filters to be used with 
CUPS 2.5.x or older will get moved into libppd. For this libppd will get 
some new API functions, for example for loading PPDs and converting all 
properties and options into a printer IPP attributes message compatible 
to a get-printer-attributes IPP response, and especially a set of 
PPD-supporting filter functions (named "ppdFilter...()", name 
corresponding to the appropriate original filter function in 
libcupsfilters) which load and convert the PPD, do PPD-specific action 
and then call the original filter function doing the dirty work in 
libcupsfilters. So the filter functions in libppd are wrappers around 
the ones in libcupsfilters to add PPD file support.

The CUPS filter stubs in the filter/ directory will call the filter 
functions of libppd, as they are for CUPS 2.5.x or older.

Now I am thinking about how to get filter functionality into CUPS 3.x. A 
pur IPP, PPD- and driver-free CUPS still needs filtering. CUPS 3.x will 
accept the same input formats as 2.x, PDF, PostScript, Apple/PWG Raster, 
JPG, PNG, plain text at least and as output formats it has to provide 
PDF, Apple Raster, PWG Raster, and PCLm at least, to cover all 
driverless printers. So we need to convert formats if needed and most 
probably we also want to do page management (N-up, even/odd, reverse 
order, booklet, ...) in an inbetween PDF step.

To do this there are the following possibilities:

1. Continue with external filter executables from cups-filters, but now 
they do not need support PPDs, they can be based on the original filter 
functions in libcupsfilters. The "universal" CUPS filter, based on the 
"cfFilterUniversal()" filter function, can be used to call filter chains 
from a single executable to reduce external executable calls).

2. Let CUPS use libcupsfilters to call its filter functions, makes cupsd 
(both local and sharing) depending on libcupsfilters which in turn 
depends on libcups.

3. Re-merge libcupsfilters into libcups: All functions in libcupsfilters 
get moved into libcups, their names change from "cf...()" to 
"cups...()". The CUPS project will get independent of cups-filters, both 
for build and for use. the project cups-filters will end then, libppd 
and cups-browsed will be projects on their own and both only depend on 
libcups,

4. (PLEASE DO NOT) Re-implement all the filtering independently in 
libcups. So we re-invent the wheel after more than 2 decades of filter 
experience. Note that software is like a human being, it gets experience 
throughout its life, it gets better and better ...

Independent of this, cups-browsed will get turned into a Printer 
Application whose purpose it is to create and manage printer clusters, 
especially if such functionality is not provided by the sharing CUPS 
server. The PPD dependecy of cups-browsed will get completely removed.

libppd, residing in its own project will solely be used for 
retro-fitting purposes, especially Printer Applications using 
libpappl-retrofit or old CUPS versions in enterprise distros.

cups-filters 2.x will provide the non-PPD-supporting libcupsfilters. 
libppd and cups-browsed can be already split off, but I am not yet 100% 
sure.

Now I want to ask you all how we should solve the need of the filtering 
capability of CUPS 3.x: (1), (2), (3), or ...???

WDYT?

    Till

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 2135 bytes --]

Till,

For CUPS 3.0 I'd like us to limit "filtered" printing to a smaller set of common formats: plain (Unicode) text, JPEG and PNG images, and PDF.  Then we "only" need to support converting from those formats to PWG/Apple Raster and PDF using the standard IPP Everywhere Job Template attributes. Obviously we can still allow printer-specific formats (PCL, PostScript, ESC/P, ZPL, etc.) sent to compatible printers, but limiting the formats we process locally will greatly simplify development and testing.

> On Apr 26, 2022, at 11:42 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> ...
> 1. Continue with external filter executables from cups-filters, but now they do not need support PPDs, they can be based on the original filter functions in libcupsfilters. The "universal" CUPS filter, based on the "cfFilterUniversal()" filter function, can be used to call filter chains from a single executable to reduce external executable calls).

I think that isolating the filtering in a separate process is an important security and reliability choice - a filter crashing stops a single job, but crashing the local spooler will stop (and potentially lose) everything, not to mention likely causing issues for the applications that are talking to it.

The model I used in the PWG ippserver and ippeveprinter sample code (with an external transform command) is probably the way we want to go, with that executable handling all of the basic filtering functionality in one program.  I really don't want to see a proliferation of filters like we have in the current CUPS since that has historically been a great source of pain/support calls.  Better to have a smaller set of file formats we support consistently.

WRT filtering functionality, we've recently had new requests to be able to add watermarking and header/footer text to pages, in addition to the current IPP Everywhere-based filter options.  Doing so is important for replicating common printer driver options, and is also useful in enterprise environments where printing is under greater control/restrictions.

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

Seems that your plans are to discontinue cups-filters and to include a 
new from-scratch filter approach in CUPS 3.x. Am I right?

I think this is not a good idea. cups-filters contains 2 decades of hard 
work and even if we do not make use of its full functionality we should 
not throw it overboard.

First, to the formats we will need.

PostScript and all these old, exotic image formats which older CUPS 
versions and cups-filters support we indeed do not need. PDF, JPEG, PNG, 
and text is good enough, especially as desktop applications usually send 
PDF. Output should be PDF, Apple Raster. PWG Raster, and PCLm. PCLm is 
also actually needed as there are PCLm-only printers which fulfill the 
driverless printing standards. We discovered such printers recently. I 
really do not want to have to provide a Printer Application for printers 
which fulfill driverless printing standards.

Also there is some useful functionality compared to tools/ipptransform.c 
which I like to keep:

- Ghostscript is the PDF renderer best optimized for printing. The 
interpreter got recently converted to be completely written in C, 
improving its performance and resource usage. So calling Ghostscript as 
external executable (to not need to worry about license compatibility) 
could improve things compared to MuPDF (and AFAIR MuPDF and Ghostscript 
have the same license).

- Ghostscript produces all the 4 driverless printing output formats: PDF 
(not actually needed in our use case), Apple Raster (I have added it in 
9.56.0), PWG Raster, and PCLm, so no post-filtering after Ghostscript 
needed, the output of Ghostscript can be directly passed on to the 
printer. For Snaps and other sandboxed packages or for IoT Ghostscript 
can be compiled with only the needed output devices (I do this in the 
Printer Application Snaps).

- CUPS provides some extra functions in page management: N-up, even/odd, 
mirror, ... via the pdftopdf filter. We should conserve them.

- The pdftopdf filter flattens filled PDF forms and annotations, which 
assures that the filled in text actually gets printed by the printer, 
independent whether the printer prints PDF directly or when need to to 
convert incoming PDF to some raster format.

- Thorough page geometry tests and fixes have been performed on all 
filters of cups-filters recently. All combinations of the print-scaling 
and number-up attributes should work, and with both standard portrait 
and also landscape-oriented (long-edge-first) paper size settings.


Now on the finalization of the 2.x release I will make libcupsfilters 
fully free of PPD dependency. I have already worked out a design for 
this and only need to implement it.

I have already written up my idea and reported some progress on it:

 From my 18th-April weekly report at Canonical:

https://discourse.ubuntu.com/t/desktop-team-updates-monday-18th-april-2022/27715/8

----------
How do we have a libcupsfilters without any PPD file support to use as 
distro package in a system with CUPS 3.x? But how still be able to 
install a (non-snapped, DEB package) retro-fitting Printer Application 
in such a system without needing to replace libcupsfilters by a 
differently built variant (is this possible in distros?) which supports 
PPDs, without needing a libcupsfilters generally supporting PPDs (and we 
all wanted to get rid of them)?

So I thought out a design change in cups-filters: Currently the filter 
functions support PPDs, libcupsfilters depends on libppd, I will remove 
the PPD support from the filter functions, making libcupsfilters 
independent of PPDs and libppd, create wrapper filter functions in 
libppd overtaking the PPD support and calling the original filter 
function. So dependencies get reversed, libppd depends on libcupsfilters 
and libppd provides the PPD-supporting set of filter functions, 
libcupsfilters the PPD-free set, and that without code duplications. Now 
I am turning this idea into code.
----------

 From my 25th-April weekly report at Canonical:

https://discourse.ubuntu.com/t/desktop-team-updates-monday-25th-april-2022/27880/7

----------
I started to do the restructuring to remove the PPD file support 
completely from libcupsfilters and move it into libppd, making libppd 
depending on libcupsfilters and libcupsfilters not depending on libppd. 
Created a first wrapper filter function ppdFilterPDFToPDF() calling 
cfFilterPDFToPDF() with the latter not having PPD file support. I let 
the PPD file loader function directly translate the PPD file into 
printer IPP attributes for the call of the original filter function in 
libcupsfilters now and I am currently creating a function to extract the 
requested paper size dimensions from job IPP attributes and options. 
This will be the last major code change before the release.
----------

This way CUPS 3.x could use libcupsfilters without pulling any 
dependency of PPD file support.

For Snaps and other types of sandboxed packages I will also add 
conditionals to libcupsfilters to skip building the support for unneeded 
data formats.

Even that libcupsfilters provides the filters as library functions 
(filter functions) we can let CUPS 3.x call them as external executable. 
We only need to write a little executable stub which calls the needed 
filter function (or a chain of them) and has the command line we need, 
for example being the same as of ipptransform. Due to the fact that we 
can easily chain filter functions we can always reduce to a single 
external executable call for any pair of input and output formats.

The pdftopdf filter is currently in C++ as it uses QPDF. Here we can 
switch to pDFio in the future as soon as PDFio provides all required 
features (especially form-flattening). But this is not a blocker for the 
cups-filters 2.0 release. The switchover does not change any APIs.

The filters in cups-filters have matured well in the years. Bug reports 
of the type "I cannot print file XXX" got extremely rare.

Watermarking could probably be easily added to pdftopdf, header/footer 
text (labels) are perhaps already in there.

WDYT?

    Till

On 27/04/2022 14:43, Michael Sweet wrote:
> Till,
> 
> For CUPS 3.0 I'd like us to limit "filtered" printing to a smaller set of common formats: plain (Unicode) text, JPEG and PNG images, and PDF.  Then we "only" need to support converting from those formats to PWG/Apple Raster and PDF using the standard IPP Everywhere Job Template attributes. Obviously we can still allow printer-specific formats (PCL, PostScript, ESC/P, ZPL, etc.) sent to compatible printers, but limiting the formats we process locally will greatly simplify development and testing.
> 
> 
> I think that isolating the filtering in a separate process is an important security and reliability choice - a filter crashing stops a single job, but crashing the local spooler will stop (and potentially lose) everything, not to mention likely causing issues for the applications that are talking to it.
> 
> The model I used in the PWG ippserver and ippeveprinter sample code (with an external transform command) is probably the way we want to go, with that executable handling all of the basic filtering functionality in one program.  I really don't want to see a proliferation of filters like we have in the current CUPS since that has historically been a great source of pain/support calls.  Better to have a smaller set of file formats we support consistently.
> 
> WRT filtering functionality, we've recently had new requests to be able to add watermarking and header/footer text to pages, in addition to the current IPP Everywhere-based filter options.  Doing so is important for replicating common printer driver options, and is also useful in enterprise environments where printing is under greater control/restrictions.
> 
> ________________________
> Michael Sweet
> 
> 
> 

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 4329 bytes --]

Till,

> On May 6, 2022, at 11:14 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> 
> Seems that your plans are to discontinue cups-filters and to include a new from-scratch filter approach in CUPS 3.x. Am I right?

I'm not sure at this point.  I *do* know that I'd like to see us focus out efforts on core file formats and a minimal set of dependencies for converting them - IMHO there are too many combinations to test and support properly.

> I think this is not a good idea. cups-filters contains 2 decades of hard work and even if we do not make use of its full functionality we should not throw it overboard.

I don't think I suggested that.

> First, to the formats we will need.
> 
> PostScript and all these old, exotic image formats which older CUPS versions and cups-filters support we indeed do not need.

Well, certainly the PS printer application needs PostScript support.  The "exotic" image formats are already supported by desktop (Gimp, etc.) and command-line (Imagemagick, etc.) applications that can handle printing.

> PDF, JPEG, PNG, and text is good enough, especially as desktop applications usually send PDF. Output should be PDF, Apple Raster. PWG Raster, and PCLm. PCLm is also actually needed as there are PCLm-only printers which fulfill the driverless printing standards.

PCLm only satisfies Wi-Fi Direct and Mopria.  Can you name a single PCLm printer that also doesn't support AirPrint or IPP Everywhere?

> We discovered such printers recently. I really do not want to have to provide a Printer Application for printers which fulfill driverless printing standards.

Make and model please.

> Also there is some useful functionality compared to tools/ipptransform.c which I like to keep:

ippsample's prototyping tool?  That's effectively my model for the new 3.0 filtering/transform front-end, although I will shortly be eliminating its usage of MuPDF - that library's API is far too unstable to depend on and uses an incompatible license for production.

> - Ghostscript is the PDF renderer best optimized for printing. The interpreter got recently converted to be completely written in C, improving its performance and resource usage. So calling Ghostscript as external executable (to not need to worry about license compatibility) could improve things compared to MuPDF (and AFAIR MuPDF and Ghostscript have the same license).

We CANNOT rely on Artifex software anymore.  AGPL3 makes it a non-starter for anyone using CUPS on embedded or cloud platforms - unlike GPL(2/3) you can't just pipe the output, since AGPL applies to the work as a whole (total overreach IMHO, but it's their software...)

I have no problem with Ghostscript being an option, but we need to make sure CUPS can be used freely by everyone, not just the True GNU.

> - Ghostscript produces all the 4 driverless printing output formats: PDF (not actually needed in our use case), Apple Raster (I have added it in 9.56.0), PWG Raster, and PCLm, so no post-filtering after Ghostscript needed, the output of Ghostscript can be directly passed on to the printer. For Snaps and other sandboxed packages or for IoT Ghostscript can be compiled with only the needed output devices (I do this in the Printer Application Snaps).

Ghostscript may be able to rasterize PDF content, but it doesn't handle the transform/imposition/watermarking functionality that is often used.

> - CUPS provides some extra functions in page management: N-up, even/odd, mirror, ... via the pdftopdf filter. We should conserve them.

Again, I wasn't suggesting that functionality would go away.

> - The pdftopdf filter flattens filled PDF forms and annotations, which assures that the filled in text actually gets printed by the printer, independent whether the printer prints PDF directly or when need to to convert incoming PDF to some raster format.

Right, but the current QPDF-based filtering seems to cause problems for some printers.

> - Thorough page geometry tests and fixes have been performed on all filters of cups-filters recently. All combinations of the print-scaling and number-up attributes should work, and with both standard portrait and also landscape-oriented (long-edge-first) paper size settings.

I'm glad to hear that.

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

On 06/05/2022 20:22, Michael Sweet wrote:
> Till,
> 
>> On May 6, 2022, at 11:14 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
>>
>> Seems that your plans are to discontinue cups-filters and to include a new from-scratch filter approach in CUPS 3.x. Am I right?
> 
> I'm not sure at this point.  I *do* know that I'd like to see us focus out efforts on core file formats and a minimal set of dependencies for converting them - IMHO there are too many combinations to test and support properly.
> 

I understand. Therefore I want to give the possibility to compile 
libcupsfilters with only support for the input and output formats needed 
by CUPS 3.x, so that one can use this minimum version in Snaps and other 
sandboxed packages.

>> I think this is not a good idea. cups-filters contains 2 decades of hard work and even if we do not make use of its full functionality we should not throw it overboard.
> 
> I don't think I suggested that.
> 

I do not say that you suggested this, I only see that you use something 
else which makes the impression for me that you re-create the filters 
(at least the actually needed ones for the New Architecture) from scratch.

>> First, to the formats we will need.
>>
>> PostScript and all these old, exotic image formats which older CUPS versions and cups-filters support we indeed do not need.
> 
> Well, certainly the PS printer application needs PostScript support.

For that I use the pdftops() filter function of libcupsfilters 2.x. I 
could move this piece of code into ps-printer-app Or I can leave it in 
libcupsfilters 2.x but make it optionally compiled, only if requested, 
and let the PostScript Printer Application Snap build cups-filters with 
PS support. But as distributions want to provide the PostScript Printer 
Application as distro package and then will all build libcupsfilters 
with PostScript support. So perhaps better move the PostScript support 
code into the PostScript Printer Application. WDYT?

> The "exotic" image formats are already supported by desktop (Gimp, etc.) and command-line (Imagemagick, etc.) applications that can handle printing.
>

The support for them is still lurking around in libcupsfilters 2.x. 
Probably they will never get used. I could remove them now, making use 
that we are starting a new generation (2). WDYT?

>> PDF, JPEG, PNG, and text is good enough, especially as desktop applications usually send PDF. Output should be PDF, Apple Raster. PWG Raster, and PCLm. PCLm is also actually needed as there are PCLm-only printers which fulfill the driverless printing standards.
> 
> PCLm only satisfies Wi-Fi Direct and Mopria.  Can you name a single PCLm printer that also doesn't support AirPrint or IPP Everywhere?
> 

See this issue: https://github.com/apple/cups/issues/6022

>> We discovered such printers recently. I really do not want to have to provide a Printer Application for printers which fulfill driverless printing standards.
> 
> Make and model please.
> 

The printers are the HP LaserJet M14, M15, M16, M17

----------
avahi-browse -rt _ipp._tcp
- enp2s0 IPv4 bbox Ippos Printer Internet Printer local
- lo IPv4 HP LaserJet M15a (B6676D) (USB)               Internet Printer 
     local
= enp2s0 IPv4 bbox Ippos Printer Internet Printer local
hostname = [bbox.local]
address = [192.168.1.254]
port = [631]
txt = []
= lo IPv4 HP LaserJet M15a (B6676D) (USB) Internet Printer local
hostname = [bruno-TOSH.local]
address = [127.0.0.1]
port = [60000]
txt = ["air=none" "mopria-certified=1.3" "rp=ipp/print" "priority=50" 
"kind=document,envelope,photo" "PaperMax=legal-A4" 
"UUID=564e4333-4234-3434-3231-c8d9d2b6676d" "Color=F" "Duplex=F" "note=" 
"qtotal=1" "usb_MDL=HP LaserJet M14-M17" "usb_MFG=HP" 
"usb_CMD=PJL,PML,PWG_RASTER,URP" "ty=HP LaserJet M14-M17" "product=(HP 
LaserJet M14-M17)" "pdl=application/PCLm,application/octet-stream" 
"txtvers=1" "adminurl=http://localhost:60000" "Fax=T" "rfo=ipp/faxout" 
"Scan=F"]
----------

So it is a certain series of HP printers, not many compared to the whole 
market but as they fulfill driverless printing standards we should 
consider them driverless and not require the user to install a Printer 
Application (like the HPLIP Printer Application).

>> Also there is some useful functionality compared to tools/ipptransform.c which I like to keep:
> 
> ippsample's prototyping tool?  That's effectively my model for the new 3.0 filtering/transform front-end, although I will shortly be eliminating its usage of MuPDF - that library's API is far too unstable to depend on and uses an incompatible license for production.

Which sufficiently free PDF interpreter will you put at MuPDF's place?

> 
>> - Ghostscript is the PDF renderer best optimized for printing. The interpreter got recently converted to be completely written in C, improving its performance and resource usage. So calling Ghostscript as external executable (to not need to worry about license compatibility) could improve things compared to MuPDF (and AFAIR MuPDF and Ghostscript have the same license).
> 
> We CANNOT rely on Artifex software anymore.  AGPL3 makes it a non-starter for anyone using CUPS on embedded or cloud platforms - unlike GPL(2/3) you can't just pipe the output, since AGPL applies to the work as a whole (total overreach IMHO, but it's their software...)
> 
> I have no problem with Ghostscript being an option, but we need to make sure CUPS can be used freely by everyone, not just the True GNU.
> 

So let us use Ghostscript as an option as the Linux distributions 
usually use it. I will also continue with Ghostscript in the Printer 
Application Snaps in the Snap Store.

>> - Ghostscript produces all the 4 driverless printing output formats: PDF (not actually needed in our use case), Apple Raster (I have added it in 9.56.0), PWG Raster, and PCLm, so no post-filtering after Ghostscript needed, the output of Ghostscript can be directly passed on to the printer. For Snaps and other sandboxed packages or for IoT Ghostscript can be compiled with only the needed output devices (I do this in the Printer Application Snaps).
> 
> Ghostscript may be able to rasterize PDF content, but it doesn't handle the transform/imposition/watermarking functionality that is often used.
> 

OK.

Which is the PDF rasterizer which should be used then? Or do we need a 
PDF pre-processor like pdftopdf for that?

>> - CUPS provides some extra functions in page management: N-up, even/odd, mirror, ... via the pdftopdf filter. We should conserve them.
> 
> Again, I wasn't suggesting that functionality would go away.
> 

How is it planned to get implemented in CUPS 3.x.

>> - The pdftopdf filter flattens filled PDF forms and annotations, which assures that the filled in text actually gets printed by the printer, independent whether the printer prints PDF directly or when need to to convert incoming PDF to some raster format.
> 
> Right, but the current QPDF-based filtering seems to cause problems for some printers.
> 

OK, what is your alternative?

Nothing against your ideas to get a more lightweight and more 
software-licence-compliant CUPS. I also do not want to impose the use of 
cups-filters or any given PDF renderer in CUPS. If we do something 
completely different in 3.x, no problem for me.

I only want to find out how we will exactly do CUPS 3.x and on what 
tasks I will have to concentrate in the next months/years.

For example I have invested a lot of time into cups-filters to make it 
more suitable with the New Architecture, to make it easier to use it 
from Printer Applications, both native and PPD-retro-fitting ones, to 
let it easily go through the transition into a PPD-free world, to test 
it more and find and fix some bugs, ...

Now the question naturally is whether it really is needed in the future, 
and if so which changes on its inner workings it should receive in the 
2.n versions coming after the 2.0 API change?

Distros are currently eagerly waiting for me to release cups-filters 2.x 
so that they can distro-package the retro-fitting Printer Applications.

So I need to know whether it is still worthwhile to invest in 
cups-filters or whether I should perhaps put it into maintenance mode 
and move to for example working on PDFio to make it flatten forms and do 
other things to replace the pdftopdf filter?

I also hope not to have given to many false promises working on 
cups-filters and announcing my progress in the monthly news posts.

And I want to know what I will tell on the OpenPrinting Summit what I/we 
will do in the next months.

    Till

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 3989 bytes --]

Till,

> On May 6, 2022, at 3:18 PM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> ...
> and then will all build libcupsfilters with PostScript support. So perhaps better move the PostScript support code into the PostScript Printer Application. WDYT?

Long term that is probably the best approach.  But I suspect CUPS 2.x will be around for a while yet so I wouldn't jump at moving things around until we are done supporting cups-filters.

>> The "exotic" image formats are already supported by desktop (Gimp, etc.) and command-line (Imagemagick, etc.) applications that can handle printing.
>> 
> 
> The support for them is still lurking around in libcupsfilters 2.x. Probably they will never get used. I could remove them now, making use that we are starting a new generation (2). WDYT?

Up to you, but it would make sense.

>>> PDF, JPEG, PNG, and text is good enough, especially as desktop applications usually send PDF. Output should be PDF, Apple Raster. PWG Raster, and PCLm. PCLm is also actually needed as there are PCLm-only printers which fulfill the driverless printing standards.
>> PCLm only satisfies Wi-Fi Direct and Mopria.  Can you name a single PCLm printer that also doesn't support AirPrint or IPP Everywhere?
> 
> See this issue: https://github.com/apple/cups/issues/6022
> 
>>> We discovered such printers recently. I really do not want to have to provide a Printer Application for printers which fulfill driverless printing standards.
>> Make and model please.
> 
> The printers are the HP LaserJet M14, M15, M16, M17

LaserJet *Pro* M14-M17, and they do not appear to be manufactured anymore (looks like there are few still available from Amazon, just not from HP directly that I can see) and have been replaced by the M110 series which *does* support AirPrint and Apple Raster.

Anyways, if that is the extent of the PCLm-only printers we need to support, I am not convinced we need to add the complexity of supporting PCLm - it's not just a matter of the format, you also have to support the banding, etc. attributes and values.

> ...
>>> Also there is some useful functionality compared to tools/ipptransform.c which I like to keep:
>> ippsample's prototyping tool?  That's effectively my model for the new 3.0 filtering/transform front-end, although I will shortly be eliminating its usage of MuPDF - that library's API is far too unstable to depend on and uses an incompatible license for production.
> 
> Which sufficiently free PDF interpreter will you put at MuPDF's place?

Xpdf/Popper are GPL2, which fit the bill.

> ...
> Which is the PDF rasterizer which should be used then? Or do we need a PDF pre-processor like pdftopdf for that?

It would be better to preprocess the PDF so that the PDF interpreter only has to rasterize content.

>>> - CUPS provides some extra functions in page management: N-up, even/odd, mirror, ... via the pdftopdf filter. We should conserve them.
>> Again, I wasn't suggesting that functionality would go away.
> 
> How is it planned to get implemented in CUPS 3.x.

In the transform program.

>>> - The pdftopdf filter flattens filled PDF forms and annotations, which assures that the filled in text actually gets printed by the printer, independent whether the printer prints PDF directly or when need to to convert incoming PDF to some raster format.
>> Right, but the current QPDF-based filtering seems to cause problems for some printers.
> 
> OK, what is your alternative?

PDFio.

> ...
> So I need to know whether it is still worthwhile to invest in cups-filters or whether I should perhaps put it into maintenance mode and move to for example working on PDFio to make it flatten forms and do other things to replace the pdftopdf filter?

You'll need to support cups-filters for a while yet, obviously, and it would be useful (even for cups-filters) to use PDFio instead of QPDF for PDF filtering...

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

On 07/05/2022 00:32, Michael Sweet wrote:
> Till,
> 
>> On May 6, 2022, at 3:18 PM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
>> ...
>> and then will all build libcupsfilters with PostScript support. So perhaps better move the PostScript support code into the PostScript Printer Application. WDYT?
> 
> Long term that is probably the best approach.  But I suspect CUPS 2.x will be around for a while yet so I wouldn't jump at moving things around until we are done supporting cups-filters.
> 

So I will keep it. It is only a small piece of code, as the dirty work 
is done by external executables (one of Ghostscript, Poppler's pdftops, 
MuPDF). Users who do not need PostScript output support do not need to 
install these executables.

>>> The "exotic" image formats are already supported by desktop (Gimp, etc.) and command-line (Imagemagick, etc.) applications that can handle printing.
>>>
>>
>> The support for them is still lurking around in libcupsfilters 2.x. Probably they will never get used. I could remove them now, making use that we are starting a new generation (2). WDYT?
> 
> Up to you, but it would make sense.
> 

Removing them would CUPS 2.x not supporting them any more, but in 
practise no one sends these directly to the printer any more. I will 
remove them for maintainability reasons.

>> The printers are the HP LaserJet M14, M15, M16, M17
> 
> LaserJet *Pro* M14-M17, and they do not appear to be manufactured anymore (looks like there are few still available from Amazon, just not from HP directly that I can see) and have been replaced by the M110 series which *does* support AirPrint and Apple Raster.
> 
> Anyways, if that is the extent of the PCLm-only printers we need to support, I am not convinced we need to add the complexity of supporting PCLm - it's not just a matter of the format, you also have to support the banding, etc. attributes and values.
> 

Once, free software code for PCLm is in cups-filters and Ghostscript, 
the former is the license of CUPS from cups-filters 2.x on, but second 
it is really exotic.

I am currently managing a project of the printer setup tool in GNOME 
Control Center being replaced by a new one for the New Architecture. At 
one point it must determine whether a discovered printer is driverless 
or not (to decide whether to look up a Printer Application). Usually I 
would strictly follow the standards here, which is IPP 2.x + at least 
one of PDF, Apple Raster, PWG Raster, PCLm. This would make PCLm-only 
printers considered driverless and CUPS 3.x would fail with them.

I can change this check into IPP 2.x + at least one of PDF, Apple 
Raster, PWG Raster, making a PCLm-only printer considered not 
driverless. With current market of printers these would be that HP's and 
the printer setup tool would install the HPLIP Printer Application, 
making this printer work. Only if we discover PCLm-only printers which 
are not from HP we would need a PCLm Printer Application.

>> Which sufficiently free PDF interpreter will you put at MuPDF's place?
> 
> Xpdf/Popper are GPL2, which fit the bill.
>

So GPL2 is OK and AGPL of Ghostscript/Artifex not? Why?

So I can make sure that libcupsfilters will always support Poppler-based 
PDF rendering (but offer the others as alternative).

> It would be better to preprocess the PDF so that the PDF interpreter only has to rasterize content.
>

Sop something pdftopdf-alike?

>>> Again, I wasn't suggesting that functionality would go away.
>>
>> How is it planned to get implemented in CUPS 3.x.
> 
> In the transform program.
> 
>>>> - The pdftopdf filter flattens filled PDF forms and annotations, which assures that the filled in text actually gets printed by the printer, independent whether the printer prints PDF directly or when need to to convert incoming PDF to some raster format.
>>> Right, but the current QPDF-based filtering seems to cause problems for some printers.
>>
>> OK, what is your alternative?
> 
> PDFio.
>

Means that after the release of cups-filters 2.x, one of the first new 
features in 2.n will be a pdftopdf based on PDFio, as soon as PDFio 
supports flattening PDF forms.

By the way, we have seen some printers which do not like PDF output of 
QPDF (I got one bug report in the last days). Are there many more? 
Should we report a bug on QPDF? Or has it some general fault making 
reporting a bug on it not worthwhile?

Note that PDF interpreters in printers can have quirks like PostScript 
interpreters. They also do not necessarily only hit QPDF output, but can 
also hit the ourput of Ghostscript, GTK, Qt, even PDFio.

So in my opinion we should not necessarily rush on getting rid of QPDF 
(we report bugs on QPDF compatibility issues to QPDF though) but better 
make sure that if a driverless printer is not able to print a PDF file 
that we smoothly fall back to raster, what you are currently fixing/have 
already fixed.

I agree with replacing of QPDF by PDFio for sake of simplicity and 
getting rid of C++ though. As you probably I am much more comfortable 
with regular C.

>> ...
>> So I need to know whether it is still worthwhile to invest in cups-filters or whether I should perhaps put it into maintenance mode and move to for example working on PDFio to make it flatten forms and do other things to replace the pdftopdf filter?
> 
> You'll need to support cups-filters for a while yet, obviously, and it would be useful (even for cups-filters) to use PDFio instead of QPDF for PDF filtering...

Yes, one of the 2.n releases will have a PDFio-based pdftopdf() filter 
function, depends on PDFio's feature-completeness, especially flattening 
PDF forms.

    Till

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 3244 bytes --]

Till,

> On May 7, 2022, at 3:05 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> ...
>>> Which sufficiently free PDF interpreter will you put at MuPDF's place?
>> Xpdf/Popper are GPL2, which fit the bill.
>> 
> 
> So GPL2 is OK and AGPL of Ghostscript/Artifex not? Why?

The GPL2 terms still allow companies, as well as individuals, to use and combine software freely.

The AGPL terms are specifically written to prohibit combining of software without either providing the combined software under the GPL/AGPL or seeking alternative licensing from the developer. Unfortunately, what defines a combined work is somewhat ambiguous, so one could argue that making use of non-free Wi-Fi firmware on a system that accepts print jobs over Wi-Fi violates the AGPL of Ghostscript.  I doubt that Artifex would push for such a legal interpretation, but they've chosen the AGPL and do license Ghostscript commercially.

As an organization under the Linux Foundation, I believe OpenPrinting has a responsibility to ensure that all users, including businesses, can use our work freely.  Unfortunately, the AGPL goes beyond *software* freedom and community (sharing) by explicitly limiting what you can do do with the software and effectively declaring that "all your base are belong to us."

(FWIW, I feel similarly about using GPL3 in OpenPrinting - its use is toxic to commercial distribution of OSS...)

> ...
>> It would be better to preprocess the PDF so that the PDF interpreter only has to rasterize content.
>> 
> 
> Sop something pdftopdf-alike?

Correct.

> ...
> By the way, we have seen some printers which do not like PDF output of QPDF (I got one bug report in the last days). Are there many more? Should we report a bug on QPDF? Or has it some general fault making reporting a bug on it not worthwhile?

Definitely need to report a bug, but it will be good to determine what the problem is...

> Note that PDF interpreters in printers can have quirks like PostScript interpreters. They also do not necessarily only hit QPDF output, but can also hit the ourput of Ghostscript, GTK, Qt, even PDFio.

Of course.

> So in my opinion we should not necessarily rush on getting rid of QPDF (we report bugs on QPDF compatibility issues to QPDF though) but better make sure that if a driverless printer is not able to print a PDF file that we smoothly fall back to raster, what you are currently fixing/have already fixed.

Right, and the support for raster fallback was designed into IPP Everywhere/AirPrint from the beginning, specifically because of our experiences with buggy interpreters.

(FWIW, I don't know whether Mopria talks about this at all, and Wi-Fi Direct doesn't address this...)

> I agree with replacing of QPDF by PDFio for sake of simplicity and getting rid of C++ though. As you probably I am much more comfortable with regular C.

Personally I'm fine with C++ code/syntax, but I also know that libstdc++ and the ABI/name mangling algorithms have gone through a number of incompatible changes over the years that has made distribution of compiled printer drivers/applications very difficult.  C just doesn't have those issues...

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

On 07/05/2022 15:09, Michael Sweet wrote:
> Till,
> 
>> On May 7, 2022, at 3:05 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
>> ...
>>>> Which sufficiently free PDF interpreter will you put at MuPDF's place?
>>> Xpdf/Popper are GPL2, which fit the bill.
>>>
>>
>> So GPL2 is OK and AGPL of Ghostscript/Artifex not? Why?
> 
> The GPL2 terms still allow companies, as well as individuals, to use and combine software freely.
>

Does the GPL2 not require them to then publish the source code of the 
combined software?

> The AGPL terms are specifically written to prohibit combining of software without either providing the combined software under the GPL/AGPL or seeking alternative licensing from the developer. Unfortunately, what defines a combined work is somewhat ambiguous, so one could argue that making use of non-free Wi-Fi firmware on a system that accepts print jobs over Wi-Fi violates the AGPL of Ghostscript.  I doubt that Artifex would push for such a legal interpretation, but they've chosen the AGPL and do license Ghostscript commercially.
> 

OK, they want to sell their licenses, so for the users who want to use 
cups-filters in IoT, printers, ... it is better to use GPL2-ed Poppler then.

So I will continue supporting both Poppler and Ghostscript. Poppler as 
the choice for embedded/IoT and Ghostscript as the choice for the Linux 
distributions.

> As an organization under the Linux Foundation, I believe OpenPrinting has a responsibility to ensure that all users, including businesses, can use our work freely.  Unfortunately, the AGPL goes beyond *software* freedom and community (sharing) by explicitly limiting what you can do do with the software and effectively declaring that "all your base are belong to us."
> 

OK, I will continue to support Poppler, Ghostscript optional for the 
Linux distributions.

> (FWIW, I feel similarly about using GPL3 in OpenPrinting - its use is toxic to commercial distribution of OSS...)
> 

I have switched cups-filters 2.x to the same license as CUPS, Apache 2.0 
with exception. I recommend this license also to other, new OpenPrinting 
projects.

>> ...
>>> It would be better to preprocess the PDF so that the PDF interpreter only has to rasterize content.
>>>
>>
>> So something pdftopdf-alike? >
> Correct.
> 
>> ...
>> By the way, we have seen some printers which do not like PDF output of QPDF (I got one bug report in the last days). Are there many more? Should we report a bug on QPDF? Or has it some general fault making reporting a bug on it not worthwhile?
> 
> Definitely need to report a bug, but it will be good to determine what the problem is...
> 

I will do. Perhaps Jay will then interact with the reporter and find out 
what is going wrong.

>> Note that PDF interpreters in printers can have quirks like PostScript interpreters. They also do not necessarily only hit QPDF output, but can also hit the ourput of Ghostscript, GTK, Qt, even PDFio.
> 
> Of course.
> 
>> So in my opinion we should not necessarily rush on getting rid of QPDF (we report bugs on QPDF compatibility issues to QPDF though) but better make sure that if a driverless printer is not able to print a PDF file that we smoothly fall back to raster, what you are currently fixing/have already fixed.
> 
> Right, and the support for raster fallback was designed into IPP Everywhere/AirPrint from the beginning, specifically because of our experiences with buggy interpreters.
> 

It is a great piece of design from you. This was really missing in the 
many decades of PostScript (but PostScript printers did not have a 
common raster fallback format (or perhaps PCL?), driverless IPP printers 
usually do Apple Raster).

> (FWIW, I don't know whether Mopria talks about this at all, and Wi-Fi Direct doesn't address this...)
> 

We have implemented it in CUPS, so we do the better driverless printing 
on the client side, and it even works on Mopria and Wi-Fi Direct printers.

>> I agree with replacing of QPDF by PDFio for sake of simplicity and getting rid of C++ though. As you probably I am much more comfortable with regular C.
> 
> Personally I'm fine with C++ code/syntax, but I also know that libstdc++ and the ABI/name mangling algorithms have gone through a number of incompatible changes over the years that has made distribution of compiled printer drivers/applications very difficult.  C just doesn't have those issues...
> 

OK, this is a good argument, so we need to get the form-flattening into 
PDFio soonish.

    Till

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Till,

> On May 7, 2022, at 10:07 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> 
> On 07/05/2022 15:09, Michael Sweet wrote:
>> Till,
>>> On May 7, 2022, at 3:05 AM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
>>> ...
>>>>> Which sufficiently free PDF interpreter will you put at MuPDF's place?
>>>> Xpdf/Popper are GPL2, which fit the bill.
>>>> 
>>> 
>>> So GPL2 is OK and AGPL of Ghostscript/Artifex not? Why?
>> The GPL2 terms still allow companies, as well as individuals, to use and combine software freely.
>> 
> 
> Does the GPL2 not require them to then publish the source code of the combined software?

Only if the software is linked together as a single executable.

>> The AGPL terms are specifically written to prohibit combining of software without either providing the combined software under the GPL/AGPL or seeking alternative licensing from the developer. Unfortunately, what defines a combined work is somewhat ambiguous, so one could argue that making use of non-free Wi-Fi firmware on a system that accepts print jobs over Wi-Fi violates the AGPL of Ghostscript.  I doubt that Artifex would push for such a legal interpretation, but they've chosen the AGPL and do license Ghostscript commercially.
> 
> OK, they want to sell their licenses, so for the users who want to use cups-filters in IoT, printers, ... it is better to use GPL2-ed Poppler then.
> 
> So I will continue supporting both Poppler and Ghostscript. Poppler as the choice for embedded/IoT and Ghostscript as the choice for the Linux distributions.

Honestly, using it in a Linux distribution has the same risks since you are deploying cloud services (SaaS, etc.) using those distributions.  But that's far outside the scope of this group here and as long as it is possible to build without Ghostscript I am happy.

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

On 07/05/2022 16:45, Michael Sweet wrote:
>>
>> Does the GPL2 not require them to then publish the source code of the combined software?
> 
> Only if the software is linked together as a single executable.
>

OK, if it is used only by a filter executable, a piece of code of CUPS 
or cups-filters which is free software anyway it is no problem. The 
caller, like printer firmware, can be closed-source.

>> So I will continue supporting both Poppler and Ghostscript. Poppler as the choice for embedded/IoT and Ghostscript as the choice for the Linux distributions.
> 
> Honestly, using it in a Linux distribution has the same risks since you are deploying cloud services (SaaS, etc.) using those distributions.  But that's far outside the scope of this group here and as long as it is possible to build without Ghostscript I am happy.

OK, we will offer both options. It is not our responsibility when the 
distros accept including Ghostscript, letting the printing stack use 
Ghostscript, and then use the distro for cloud services (and up to now 
no one complains about it). I hope it is at least no AGPL violation to 
use Ghostscript on a regular real-iron computer due to its closed-source 
BIOS.

    Till

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Michael Sweet]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

Till,

> On May 7, 2022, at 12:07 PM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
> ...
> OK, we will offer both options. It is not our responsibility when the distros accept including Ghostscript, letting the printing stack use Ghostscript, and then use the distro for cloud services (and up to now no one complains about it). I hope it is at least no AGPL violation to use Ghostscript on a regular real-iron computer due to its closed-source BIOS.

Like I said, the AGPL is not specific enough to exclude this... :/

________________________
Michael Sweet




[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Printing-architecture] CUPS 3.x: How we make it converting job data formats?

[Till Kamppeter]

On 07/05/2022 21:04, Michael Sweet wrote:
> Till,
> 
>> On May 7, 2022, at 12:07 PM, Till Kamppeter <till.kamppeter@gmail.com> wrote:
>> ...
>> OK, we will offer both options. It is not our responsibility when the distros accept including Ghostscript, letting the printing stack use Ghostscript, and then use the distro for cloud services (and up to now no one complains about it). I hope it is at least no AGPL violation to use Ghostscript on a regular real-iron computer due to its closed-source BIOS.
> 
> Like I said, the AGPL is not specific enough to exclude this... :/

By all distros including Ghostscript and having it used by their 
printing stack and no one complaining we can assume they have no problem 
with that, but for outside the distro world I will continue to support 
Poppler. Ghostscript delivering high print quality in distros is a good 
advertising for them. CUPS 3.x should also support the two, GHostscript 
to make distros happy, and Poppler for correct commercial use.

    Till