SysEx overflow when using the MIDI sequencer event interface

SysEx overflow when using the MIDI sequencer event interface

[Haakon Riiser]

I'm trying to fix a problem with the DOSBox emulator that occurs
when it's sending SysEx events to an external Roland MIDI device
(CM-32L in my case).

After a little investigation, I discovered that the problem is that
SysEx messages sent via the sequencer event interface are pushed out
faster than my Roland CM-32L can process them.

This bug only affects the event interface:  Sending the SysEx
commands using the rawmidi interface (for example by using 'amidi
-s <sysex-file>') works perfectly.  I can reproduce DOSBox's
problems using a toy version of amidi that uses the event interface.
Here's what my implementation does:

  snd_seq_open(&handle, "hw", SND_SEQ_OPEN_DUPLEX, 0);
  snd_seq_create_simple_port(handle, NULL, SND_SEQ_PORT_CAP_WRITE |
                                           SND_SEQ_PORT_CAP_SUBS_WRITE |
                                           SND_SEQ_PORT_CAP_READ,
                             SND_SEQ_PORT_TYPE_MIDI_GENERIC);
  snd_seq_connect_to(handle, 0, 16, 0);

  while (get_next_sysex_command(sysex_data, &sysex_len)) {
          snd_seq_event_t ev;
          snd_seq_ev_clear(&ev);
          snd_seq_ev_set_source(&ev, 0);
          snd_seq_ev_set_subs(&ev);
          snd_seq_ev_set_direct(&ev);
          snd_seq_ev_set_sysex(&ev, sysex_len, sysex_data);
          snd_seq_event_output(handle, &ev);
          snd_seq_drain_output(handle);
          snd_seq_sync_output_queue(handle);
  } 

As an example of how much data is lost, consider that it took less
than 1.5 seconds to send 18 kB of SysEx messages using my program,
while it takes amidi over 6 seconds.

If I insert a delay of, say, 100 ms between each sent SysEx
command, my program appears to work, but there _has_ to be a
better way to ensure that the SysEx message is processed before
starting to send a new one.

Is this a bug in the ALSA code, or is there a synchronization
call or something like that I can use?

-- 
 Haakon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Haakon Riiser]

A little followup to my previous post:  I just placed a printk()
in snd_emu10k1_midi_output_trigger() that prints the number of
bytes just sent to the SB Live's MPU-401.  I also also increased
the dmesg size to 512 KiB to make sure that nothing was lost.

I had expected that by summing the entries in dmesg after having
run 'amidi -s <sysex-file>', I would get the same number as the
number of in bytes of the sysex file, but that didn't happen.

When I try to insert a 18584 byte long sysex file using amidi,
dmesg shows a byte count of approx. 10868 (+/- a few bytes).
My own toy-version of amidi that uses the event interface gets
consistently less than ten(!) bytes in the dmesg log.  If I insert
a delay between each sysex event of about 100 ms, the byte count
increases to almost 300.

That is _far_ too low a number to explain how the MIDI tunes
can sound the same as when I load the sysex data with amidi.
If only 300 sysex bytes were actually sent to the MIDI device,
I would obviously hear the difference between that and amidi's
method that got over 10000 bytes through.

None of this makes any sense to me.  I thought all bytes pushed onto
the MPU-401 were sent through snd_emu10k1_midi_output_trigger(), but
this experiment contradicts that.  I'm having trouble getting the
big picture of how the MIDI parts in ALSA works, so a little help
would be appreciated. :-)

-- 
 Haakon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Haakon Riiser]

> [...] I'm having trouble getting the big picture of how the MIDI
> parts in ALSA works, so a little help would be appreciated. :-)

Never mind, I finally sucked it up, and forced myself to do the
tedious task of tracking the MIDI events all the way from user
space to hardware, and discovered what the problem was:

Turns out that the buffer that overflows isn't a hardware buffer
in the CM-32L -- it's a software buffer in ALSA that is fixed to
a single page of memory (4096 bytes on x86).

This wouldn't be a problem if there were a mechanism that allowed
incoming events to wait until there's room in the output buffer,
or at least to fail with EAGAIN, allowing the application to retry.

What happens here is that the function in question returns ENOMEM
and prints a warning (but only if you enable ALSA debugging,
which, of course, I had not) saying that events are lost due to
buffer overflow.  Unfortunately, the return code is ignored by
the calling function, so the error is silently ignored.

I _really_ hope this isn't intentional, because it makes the
event interface too fragile to be useful.  I can fix this, but
there are many ways to go about doing it (for example waiting
for the buffer to drain, dynamically increasing buffer size,
or getting the error code propagated up the stack).  I'd like to
know what the authors think is the best solution, so that I can
get the fix into the official sources.

-- 
 Haakon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Clemens Ladisch]

Haakon Riiser wrote:
> Turns out that the buffer that overflows isn't a hardware buffer
> in the CM-32L -- it's a software buffer in ALSA that is fixed to
> a single page of memory (4096 bytes on x86).
> 
> This wouldn't be a problem if there were a mechanism that allowed
> incoming events to wait until there's room in the output buffer,
> or at least to fail with EAGAIN, allowing the application to retry.

The sequencer interface is designed to send small messages
asynchronously in real time (or at specified scheduled times).  Not
doing any bandwidth management is part of the current design.

The rawmidi interface is better suited to send big SysEx messages.

You said you're trying to fix DOSBox.  If it emulates a hardware MIDI
interface, it shouldn't accept MIDI data at a faster rate than a real
MIDI interface would.


Regards,
Clemens

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Haakon Riiser]

[Clemens Ladisch]

> The sequencer interface is designed to send small messages
> asynchronously in real time (or at specified scheduled times).  Not
> doing any bandwidth management is part of the current design.
> 
> The rawmidi interface is better suited to send big SysEx messages.

Right, but is it possible to use rawmidi while the sequencer is opened?
I tried, but open() blocks until I close the sequencer.

-- 
 Haakon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Clemens Ladisch]

Haakon Riiser wrote:
> > The sequencer interface is designed to send small messages
> > asynchronously in real time (or at specified scheduled times).  Not
> > doing any bandwidth management is part of the current design.
> > 
> > The rawmidi interface is better suited to send big SysEx messages.
> 
> Right, but is it possible to use rawmidi while the sequencer is opened?

No.  The rawmidi interface allows too read/write raw MIDI bytes which
means that it is not possible to properly merge multiple streams.  The
sequencer can do it because its messages are self-contained.


Regards,
Clemens

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: SysEx overflow when using the MIDI sequencer event interface

[Haakon Riiser]

[Clemens Ladisch]

>> Right, but is it possible to use rawmidi while the sequencer is opened?
> 
> No.  The rawmidi interface allows too read/write raw MIDI bytes
> which means that it is not possible to properly merge multiple
> streams.  The sequencer can do it because its messages are
> self-contained.

OK, so implementing bandwidth throttling in DOSBox is the only
correct solution for ALSA.  In the meantime, I've simply increased
ALSA's output buffer size to 128 KiB.

-- 
 Haakon

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642