[Printing-architecture] Bi-di plug-in functionarities

[Printing-architecture] Bi-di plug-in functionarities

[Yasumasa TORATANI]

Hello Mark, Mike, Peter and All,

I appreciate your comment, and I reply this time into the architecture
ML, and please forgive me if it's not appropriate.

In our bi-di plug-in group's opinion, we hope we could design
the bi-di plug-in module and its API like following;

1) Make it simple and small for being used frexibly from any upper
modules, such as PAPI with CUPS backend modules or other
filters, or any daemon modules with Omni / PDC, etc.

2) Make the generic plug-in API  to keep the plug-ins independent
from the implementations.

3) Each bi-di plug-in module for each printer vendor has the
    following functionarities;
  a) Handle the vendor dependent protocol to communicate with
      each vendor's printer.
  b) Obtain the printer status, including error, e.g. ink level,
      paper jam, no cartridge, etc.
  c) Obtain the information of the printer options, e.g. sorter,
      stapler, extended memory, etc.
  d) Printer discovery for both local and network.
  e) No UI. to be simple and generic.
  f) In some cases, receive the printing data from the upper system,
     translate it for eachr device or protocol, and send it to the
     appropriate target.
  g) Send the printer information to the upper system.

All of the above idea is TBD, and I appreciate any comment.
We think we should make a list of all functionarities, use cases and
senario to consider how we should deal with the printer information
and send them back to the upper system.

Regards,
Yasumasa TORATANI.

Re: [Printing-architecture] Bi-di plug-in functionarities

[Michael Sweet]

Yasumasa TORATANI wrote:
> ...
> 3) Each bi-di plug-in module for each printer vendor has the
>     following functionarities;
>   a) Handle the vendor dependent protocol to communicate with
>       each vendor's printer.
>   b) Obtain the printer status, including error, e.g. ink level,
>       paper jam, no cartridge, etc.
>   c) Obtain the information of the printer options, e.g. sorter,
>       stapler, extended memory, etc.
>   d) Printer discovery for both local and network.
>   e) No UI. to be simple and generic.
>   f) In some cases, receive the printing data from the upper system,
>      translate it for eachr device or protocol, and send it to the
>      appropriate target.
>   g) Send the printer information to the upper system.

FWIW, in CUPS #3 is normally handled by the backend (in CUPS 1.1
and earlier) and/or by the driver (in CUPS 1.2), the idea being
that the same commands/protocols will generally be used over
serial, parallel, USB, and network connections, and since the
driver has a copy of the device URI (the device for the printer)
it can tailor its input/output accordingly.

-- 
______________________________________________________________________
Michael Sweet, Easy Software Products                  mike@easysw.com
Printing Software for UNIX                       http://www.easysw.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Pete Zannucci]

Michael Sweet wrote:

> FWIW, in CUPS #3 is normally handled by the backend (in CUPS 1.1
> and earlier) and/or by the driver (in CUPS 1.2), the idea being
> that the same commands/protocols will generally be used over
> serial, parallel, USB, and network connections, and since the
> driver has a copy of the device URI (the device for the printer)
> it can tailor its input/output accordingly.

With using backends, how do we handle spool information that may
be needed to do recovery, restart a job, rerender a page, and possibly
manage other asynchronous events that may need to be mananged?

I realize there may be potential security issues but in my diagrams
I attempted to allow for management of the spool system and rendering
along with async. data manipulation.  Is there any easy way with CUPS
to provide similar function so that we don't end up with issues such
as jobs disappearing along with providing robust recovery from errors?

This direction, especially error recovery and job tracking, has been
a hot button with corporate customers in the past and generated into
a much more integrated solution for bi-di support.

Thanks,


Peter Zannucci

IBM Linux Technology Center
Open Source Software Development
Omni Print Project http://sourceforge.net/projects/omniprint
Austin, TX - Tel. 512-838-4687 (t/l 678) pzaan@us.ibm.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Michael Sweet]

Pete Zannucci wrote:
> Michael Sweet wrote:
> 
> 
>>FWIW, in CUPS #3 is normally handled by the backend (in CUPS 1.1
>>and earlier) and/or by the driver (in CUPS 1.2), the idea being
>>that the same commands/protocols will generally be used over
>>serial, parallel, USB, and network connections, and since the
>>driver has a copy of the device URI (the device for the printer)
>>it can tailor its input/output accordingly.
> 
> 
> With using backends, how do we handle spool information that may
> be needed to do recovery, restart a job, rerender a page, and possibly
> manage other asynchronous events that may need to be mananged?

The backend would have to understand the PDL, which is one of the
reasons we've added backchannel support in CUPS 1.2 so that the
driver (which is generating the PDL) can handle these details.

> I realize there may be potential security issues but in my diagrams
> I attempted to allow for management of the spool system and rendering
> along with async. data manipulation.  Is there any easy way with CUPS
> to provide similar function so that we don't end up with issues such
> as jobs disappearing along with providing robust recovery from errors?

There isn't any reason that you couldn't create a backend that
communicated with a common interface/daemon/server to provide
concurrent access to a device; the HP OfficeJet drivers do this
IIRC...

The main issue is that print data is generated by the filters
for the whole job as a stream, so drivers/backends need to save
the whole print file if they want to support recovery.  Also,
you may need to add a common set of initialization commands
to the front of any recovery data to get the printer into the
right mode(s).

What we're concentrating on for CUPS 1.2 is better error tracking,
page accounting, and event notification which will allow for more
robust drivers to be written.

-- 
______________________________________________________________________
Michael Sweet, Easy Software Products                  mike@easysw.com
Printing Software for UNIX                       http://www.easysw.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Yasumasa TORATANI]

On Fri, 06 Dec 2002 13:11:32 -0500
Michael Sweet <mike@easysw.com> wrote:

> > Michael Sweet wrote:
> > 
> > 
> >>FWIW, in CUPS #3 is normally handled by the backend (in CUPS 1.1
> >>and earlier) and/or by the driver (in CUPS 1.2), the idea being
> >>that the same commands/protocols will generally be used over
> >>serial, parallel, USB, and network connections, and since the
> >>driver has a copy of the device URI (the device for the printer)
> >>it can tailor its input/output accordingly.

Yes, and I think CUPS has well hierarchized, quite flexible architecture
with several filters which can handle data transformation from one type
to another, including printer command generation, and backends which
can deal with several devices and/or protocols operation.

And, as you mentioned, under the CUPS 1.1.x, if we prepare each
backend for each vendor's printers, and if each vendor's backend can
handle each vendor's protocol, we'll be able to deal with all of #3 issues. 

Additionally, if we prepare the common API between the CUPS
standard backends and each vendor's bi-di plug-in module to handle
each vendor's specific protorol, I suppose we'll be able to reuse the
common functionarities of each CUPS standard backend to control
each device port, for instance open it, close it, return the list of it, etc,
as well as to handle each vendor's functionarities for supporting #3
at the bi-di plug-in module.

The dataflow is as following;

[CUPS filters]-->[CUPS standard backend]<-->[bi-di plug-in]<-->[prt]

    or
                                   +----[bi-di plug-in]--------+
                                   |                                         |
                                  V                                        |
[CUPS filters]-->[CUPS standard backend]---->[prt]

Furthermore, if we make the bi-di plug-in API generic, everyone will
be able to reuse these modules for any other printing system if they
want.

The plug-in might be a shared library linked with the upper system,
or the individual process forked by the upper system. The target file
discriptor will be passed from the upper system to the plug-in, but all
interfaces between the upper system and the bi-di plug-in are TBD.

> The backend would have to understand the PDL, which is one of the
> reasons we've added backchannel support in CUPS 1.2 so that the
> driver (which is generating the PDL) can handle these details.

This is the point we thought.
The bi-di plug-in module would handle the vendor specific PDL
as well as to obtain the printer info from the printer with the vendor's
protocol, and the drivers also handle the info obtained from the
backchannel to make the PDL properly.

I'd appreciate any comment.
Yasumasa TORATANI.

Re: [Printing-architecture] Bi-di plug-in functionarities

[Michael Sweet]

Yasumasa TORATANI wrote:
> ...
> The dataflow is as following;
> 
> [CUPS filters]-->[CUPS standard backend]<-->[bi-di plug-in]<-->[prt]
> 
>     or
>                                    +----[bi-di plug-in]--------+
>                                    |                                         |
>                                   V                                        |
> [CUPS filters]-->[CUPS standard backend]---->[prt]
> 
> Furthermore, if we make the bi-di plug-in API generic, everyone will
> be able to reuse these modules for any other printing system if they
> want.

As I've posted before, the CUPS 1.2 dataflow will include a
pipe going from the backend to the filters, so the backend can
send the backchannel data to a printer-specific printer driver
for processing (so the intelligence would be in the printer driver
and not in the backend).  Your implementation is basically making
the CUPS backend a printer driver which talks with a separate
daemon/process to communicate with the device.  This is certainly
doable, but limits the use of your driver to interfaces that the
bi-di plug-in API (instead of all devices); we wouldn't do that
for regular consumer devices, but if you have a custom interface
already (high-speed serial card/whatever) to the device, this isn't
so much of an issue...

-- 
______________________________________________________________________
Michael Sweet, Easy Software Products                  mike@easysw.com
Printing Software for UNIX                       http://www.easysw.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Yasumasa TORATANI]

On Mon, 09 Dec 2002 09:19:36 -0500
Michael Sweet <mike@easysw.com> wrote:
> As I've posted before, the CUPS 1.2 dataflow will include a
> pipe going from the backend to the filters, so the backend can
> send the backchannel data to a printer-specific printer driver

Sorry for my diagram based on the current CUPS stuff.
I rewrote it for CUPS 1.2, and changed the [CUPS filters] to
[driver].

     +----------+
      |                 |
     V                |
[driver]-->[CUPS standard backend]<-->[bi-di plug-in]<-->[prt]

or

     +----------+   +----[bi-di plug-in]--------+
      |                |     |                                         |
     V               |    V                                        |
[driver]-->[CUPS standard backend]---->[prt]

> Your implementation is basically making
> the CUPS backend a printer driver which talks with a separate
> daemon/process to communicate with the device.

Yes, and I suppose it might be better, for instance,  in case of
the printing data being blocked due to the buffer full or some
error, such as no paper, while the driver writes the printing
data into the pipe. I suppose the driver can't read the printer
info from the backchannel under the single process when it's
blocked, but the driver doesn't have to read any printer info
from the backchannel pipe with another process or thread of
the driver, since the printer info which affects the printer data
generation should be obtained once before each printing, and
probably, printer error and other info should be obtained by
another process separately.

For instance, the bi-di plug-in module, if it is a separated
process from the driver and backend, the bi-di plug-in can
read the printer info from the printer and send it back to the
upper system periodically via another path, such as stderr,
and user can see the printer info.

So, one process for generating and writing the printing data,
and another process for reading the printer status is the point.

As you mentioned, we can do like this with the customized
backend and/or customised daemon which communicate with
the printer, but I suppose preparing the bi-di plug-in has some
benefit for CUPS and other printing systems.

I appreciate any comment, and this week, I'll summarize
the matter of discussion we had, then report to the Japanese
members and discuss about it.

Thanks,
Yasumasa TORATANI.

Re: [Printing-architecture] Bi-di plug-in functionarities

[Michael Sweet]

Yasumasa TORATANI wrote:
> ...
> Yes, and I suppose it might be better, for instance,  in case of
> the printing data being blocked due to the buffer full or some
> error, such as no paper, while the driver writes the printing
> data into the pipe. I suppose the driver can't read the printer
> info from the backchannel under the single process when it's
> blocked, but the driver doesn't have to read any printer info
> from the backchannel pipe with another process or thread of
> the driver, since the printer info which affects the printer data
> generation should be obtained once before each printing, and
> probably, printer error and other info should be obtained by
> another process separately.

Actually, a driver can use select() to determine when it is
"safe" to write a buffer of data and when there is backchannel
data available, so threading is not needed.  Also, threading
can introduce performance and portability issues on some
platforms, and isn't always the correct solution under UNIX.
(unfortunately, threads are the only way to implement some
things under Windows...)

> ...
> So, one process for generating and writing the printing data,
> and another process for reading the printer status is the point.

That is one possible implementation, and it is used for the
current EPSON and Canon backends for the GIMP-print project to
provide ink level information to the user.

Again, it isn't the only possible implementation, and CUPS can
support any kind of backend/driver architecture.  The *optimal*
one from the CUPS standpoint is one where the driver handles
the encoding/decoding of printer data, and the backend handles
the communications with the printer.

-- 
______________________________________________________________________
Michael Sweet, Easy Software Products                  mike@easysw.com
Printing Software for UNIX                       http://www.easysw.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Yasumasa TORATANI]

On Tue, 10 Dec 2002 10:30:58 -0500
Michael Sweet <mike@easysw.com> wrote:

> Actually, a driver can use select() to determine when it is
> "safe" to write a buffer of data and when there is backchannel
> data available, so threading is not needed.  Also, threading

That makes sense to me. I've got how we should do on the
CUPS 1.2 driver, and I suppose we should consider PIPE_BUF 
to handle writing, reading and timeout events effectively.

Thanks,
Yasumasa TORATANI.

Re: [Printing-architecture] Bi-di plug-in functionarities

[Pete Zannucci]

Yasumasa TORATANI wrote:
> ...
> The dataflow is as following;
>
> [CUPS filters]-->[CUPS standard backend]<-->[bi-di plug-in]<-->[prt]
>
>     or
>                                   +----[bi-di plug-in]--------+
>                                   |                           |
>                                   V                           |
> [CUPS filters]-->[CUPS standard backend]------------------>[prt]
>
> Furthermore, if we make the bi-di plug-in API generic, everyone will
> be able to reuse these modules for any other printing system if they
> want.

Michael Sweet Wrote:

> Your implementation is basically making
> the CUPS backend a printer driver which talks with a separate
> daemon/process to communicate with the device.  This is certainly
> doable, but limits the use of your driver to interfaces that the
> bi-di plug-in API (instead of all devices);

I would like to see that separate process due to a couple of issues:

1.  Customer's like to have true status and info on the printer.
    This would make it easier to manage async. events and alerts
    coming from the printer (power off/on, jam, etc.)
2.  That process would likely have the ability be be more robust
    in error handling because if you do have two pipes and that
    would imply separate threads to do all the management function
    and I have see many times that a particular api call has
    the ability to get hung which will force multiple threading.

I will give you that having a separate plugin would be more complex
to manage from a driver perspective but since the implementor would
want to handle the bidi case no matter what, the non-bidi (simplistic)
case should be easy and with less overhead.

If we can adequately handle async. events, true printer status, job
tracking and error reporting/recovery with the simpler implementation
than it is fine.  I don't think it is just an interface issue if we
need to do job/page management also.


Peter Zannucci

IBM Linux Technology Center
Open Source Software Development
Omni Print Project http://sourceforge.net/projects/omniprint
Austin, TX - Tel. 512-838-4687 (t/l 678) pzaan@us.ibm.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Michael Sweet]

Pete Zannucci wrote:
> ...
> I would like to see that separate process due to a couple of issues:
> 
> 1.  Customer's like to have true status and info on the printer.
>     This would make it easier to manage async. events and alerts
>     coming from the printer (power off/on, jam, etc.)

A printer driver can query (poll) a printer for its status, but
most asynchronous notifications should be done via SNMP or similar
mechanisms outside the normal print data stream; otherwise you tie
up the printer with polling which causes problems when you have
multiple (possibly redundant) servers connected to the same network
devices.  This is a problem that plagues Windows print servers, and
we regularly have to support customers that have accessbility
problems in mixed environments.

Another issue is security/access control.  Not every admin wants
to allow an arbitrary user to access status information for a
printer, and while such security measures go beyond what many
would feel is an adequate level, they are required in sensitive
environments where such information could be used to attack,
eavesdrop, or otherwise compromise the security of the print
job.

Again, I'm not saying that it can't be done with CUPS (in fact
it is already being done for specific types of devices), but I'm
just pointing out some of the issues that will come up in
implementation that need to be designed into any interface you
cook up.  A good "network printing proxy" implementation could
use multiple protocols to get the "most bang for the buck"
without adversely affecting the accessibility of that device
from other network hosts while providing the kind of security/
access control options that will be required.

> 2.  That process would likely have the ability be be more robust
>     in error handling because if you do have two pipes and that
>     would imply separate threads to do all the management function
>     and I have see many times that a particular api call has
>     the ability to get hung which will force multiple threading.

Having two pipes does not imply the use of threads unless you are
running Windows which lacks some of the basic functionality
needed to efficiently manage multiple input/output sources from
a single thread.

Any program that does asynchronous event handling will need to
be event driven in nature and properly time-out when sending or
waiting for data.  Threading is one way of implementing this,
where each thread is doing one task and syncronizing with the
other thread(s), however an ordinary FSM works exceedingly
well as a threading replacement when responding to async IO,
and in fact that is what CUPS uses in many places, including
the scheduler and HTTP/IPP APIs.

In any case, I will reiterate once more that both a direct and
indirect backend interface will work with CUPS without difficulty.
The standard CUPS backends will (obviously) be implementing the
CUPS backchannel interface.

-- 
______________________________________________________________________
Michael Sweet, Easy Software Products                  mike@easysw.com
Printing Software for UNIX                       http://www.easysw.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Pete Zannucci]

Yasumasa TORATANI wrote:

> Yes, and I suppose it might be better, for instance,  in case of
> the printing data being blocked due to the buffer full or some
> error, such as no paper, while the driver writes the printing
> data into the pipe. I suppose the driver can't read the printer
> info from the backchannel under the single process when it's
> blocked, but the driver doesn't have to read any printer info
> from the backchannel pipe with another process or thread of
> the driver, since the printer info which affects the printer data
> generation should be obtained once before each printing, and
> probably, printer error and other info should be obtained by
> another process separately.

> For instance, the bi-di plug-in module, if it is a separated
> process from the driver and backend, the bi-di plug-in can
> read the printer info from the printer and send it back to the
> upper system periodically via another path, such as stderr,
> and user can see the printer info.

> So, one process for generating and writing the printing data,
> and another process for reading the printer status is the point.

Blocking issues and async. communications to a print job were
precisely what I am worried about.  That is why I stated previously
that multiple threads or processes would be very helpful.  Printer
alerts and other info could be gathered async. to the print job.
The issue that really would need to be addressed is the "send it
back to the upper system periodically via another path".  That path
would need to be designed into the print and rendering system.

As far as a bidirectional plug-in, it would be nice to have a single
interface for plugging in protocol support so that the same info.
and interfaces are used in the system.  The underlying printer
communications protocol can be hidden.  A single backend (transport)
and spool interface could be defined so that smaller components
could be used (would be nice).  The more modularized the architecture
can be made, the less individual code and testing will be required.
Perhaps it could be started with a custom backend grouping of interfaces
for a bidi plug-in along with defining what data might be needed where,
such as error recover, status, power off/on alerts, etc.  Once we
are sure what components would need what particular pieces of data
then we what would really need to be added to the system.

Since CUPS has a major portion of what is already needed, we should
start looking what would need to be extended, if anything.

Much of the work would be designing a backend architecture to support
plugins.  That would be a good beginning utilizing the basic CUPS
infrastructure.


Peter Zannucci

IBM Linux Technology Center
Open Source Software Development
Omni Print Project http://sourceforge.net/projects/omniprint
Austin, TX - Tel. 512-838-4687 (t/l 678) pzaan@us.ibm.com

Re: [Printing-architecture] Bi-di plug-in functionarities

[Yasumasa TORATANI]

On Tue, 10 Dec 2002 09:18:59 -0600
"Pete Zannucci" <pzaan@us.ibm.com> wrote:

> Since CUPS has a major portion of what is already needed, we should
> start looking what would need to be extended, if anything.
> 
> Much of the work would be designing a backend architecture to support
> plugins.  That would be a good beginning utilizing the basic CUPS
> infrastructure.

I appreciate your comment, and that would be a nice and reasonable
thing we could begin first, if we could have a consensus of how we
should support the bi-di channel in the whole printing framework.
And I'd like to ask Peter and All about the current consensus or
the plan of how the US people would tackle the Notification API or
the Capabilities API including the dynamic capabilities that might be
firmly related to the bi-di channel.

Thanks,
Yasumasa TORATANI.