Re: [Asterisk] DTMF noise

Re: [Asterisk] DTMF noise

[Thomas Tonino]

Mark Spencer wrote:

> The DTMF detector in the linux kernel is fairly simplistic and doesn't do
> many relative energy tests.  The Zapata library has a much better tone
> detector, but it is FP, and so would have to be made fixed point.  If
> nothing else, it may provide some lessons for the ISDN folks.

I remember that a good DTMF decoder can be very simplistic: DTMF was designed 
for that.

The idea is:

- separate the high tones from the low tones.
- amplify clip the high band and the low band separately
- run the tone decoders on the clipped signals

The clipping stage would make sure that only relatively pure tones will trigger 
the detector.

See also http://groups.google.com/groups?selm=7462%40accuvax.nwu.edu


Thomas

Re: [Asterisk] DTMF noise

[Roy Sigurd Karlsbakk]

>> The DTMF detector in the linux kernel is fairly simplistic and 
>> doesn't do
>> many relative energy tests.  The Zapata library has a much better tone
>> detector, but it is FP, and so would have to be made fixed point.  If
>> nothing else, it may provide some lessons for the ISDN folks.
>
> I remember that a good DTMF decoder can be very simplistic: DTMF was 
> designed for that.
>
> The idea is:
>
> - separate the high tones from the low tones.
> - amplify clip the high band and the low band separately
> - run the tone decoders on the clipped signals
>
> The clipping stage would make sure that only relatively pure tones 
> will trigger the detector.
>
> See also http://groups.google.com/groups?selm=7462%40accuvax.nwu.edu

but the problem here, is just what you're describing.

The DTMF decoder is finding DTMF signals in normal speech, so talking 
with someone with Asterisk (dot org) using isdn4linux is a true 
nightmare. Linux's DTMF detector finds DTMF signals all the time, 
making asterisk signal them as sounds.

so - we DO NOT need a 'simplistic' DTMF decoder. not for this purpose, 
that is

roy

Re: [Asterisk] DTMF noise

[Thomas Tonino]

Roy Sigurd Karlsbakk wrote:

> so - we DO NOT need a 'simplistic' DTMF decoder.

You need a good one. But good can be simplistic, is what I'm saying.

DTMF was designed to be easy to decode reliably. Complex doesn't automatically 
mean better.

I remember reading a more specific version of the message I pointed a pointer 
to, but couldn't find it back. But it came down to the devil being in the details.

It probably pays to have someone look at this with old hardware experience in 
this. Telco newsgroup perhaps.


Thomas

Re: [Asterisk] DTMF noise

[David D. Hagood]

Thomas Tonino wrote:
> Roy Sigurd Karlsbakk wrote:
> 
>> so - we DO NOT need a 'simplistic' DTMF decoder.
> 
> 
> You need a good one. But good can be simplistic, is what I'm saying.
> 
> DTMF was designed to be easy to decode reliably. Complex doesn't 
> automatically mean better.
> 

I haven't looked at the code, but I'd recommend using a bank of Goertzel 
filters -


http://www.google.com/search?hl=en&lr=&ie=ISO-8859-1&q=Goertzel+filter+DTMF&btnG=Google+Search

The basic idea is that you have 8 filters (for the 4 row and 4 column 
frequencies), as well as 8 filters looking at the first harmonic of the 
8 frequencies. You then compare the energies in each frequency - if you 
see significant energy in the harmonic filter bank, discard the signal. 
That prevents you from detecting speech as DTMF, since speech will 
usually have harmonics that a good DTMF signal won't.

Since the Goertzel filters are simple, they can be implemented in fixed 
point math rather than floating point. At work, we've done this on a 
Motorola 56301 DSP, which is a fixed-point DSP. I think there's an app 
note from Moto on this - I'll check when I get into work today.

Re: [Asterisk] DTMF noise

[Matti Aarnio]

On Wed, Jan 08, 2003 at 06:43:10AM -0600, David D. Hagood wrote:
> Thomas Tonino wrote:
> >Roy Sigurd Karlsbakk wrote:
> >>so - we DO NOT need a 'simplistic' DTMF decoder.
> >
> >You need a good one. But good can be simplistic, is what I'm saying.
> >
> >DTMF was designed to be easy to decode reliably. Complex doesn't 
> >automatically mean better.
> 
> I haven't looked at the code, but I'd recommend using a bank of Goertzel 
> filters -

  Do look into   drivers/isdn/isdn_audio.c     That does use  Görtzel 
  filter, but does not do complete energy comparisons -> false detections.

/Matti Aarnio

Re: [Asterisk] DTMF noise

[Wolfgang Fritz]

David D. Hagood wrote:
 > Thomas Tonino wrote:
 >
 >> Roy Sigurd Karlsbakk wrote:
 >>
 >>> so - we DO NOT need a 'simplistic' DTMF decoder.
 >>
 >>
 >>
 >> You need a good one. But good can be simplistic, is what I'm saying.
 >>
 >> DTMF was designed to be easy to decode reliably. Complex doesn't
 >> automatically mean better.
 >>
 >
 > I haven't looked at the code, but I'd recommend using a bank of Goertzel
 > filters -
 >
 >
 > 
http://www.google.com/search?hl=en&lr=&ie=ISO-8859-1&q=Goertzel+filter+DTMF&btnG=Google+Search 

 >
 >
 > The basic idea is that you have 8 filters (for the 4 row and 4 column
 > frequencies), as well as 8 filters looking at the first harmonic of the
 > 8 frequencies. You then compare the energies in each frequency - if you
 > see significant energy in the harmonic filter bank, discard the signal.
 > That prevents you from detecting speech as DTMF, since speech will
 > usually have harmonics that a good DTMF signal won't.

That is done in the isdn_audio DTMF detection but did not work very well
with a number of phone sets I tested which seem to generate DTMF tones with
strong harmonics. They may be out of spec but they exist.

I have a patch which adds a simple energy comparison and some
plausablility checks to the DTMF eval code but does not look at the
harmonics. That improved the detection with above phone sets.

Maybe it would be better to reenable harmonic checks but comparing
harmonic levels to the level of the fundamental instead of using
absolute values as in the present implementation.

OTOH I don't think its a good approach to check harmonics anyway but to
check other non DTMF frequencies in the main speech band and only accept
a DTMF if a DTMF frequency pair is present but no signal on the non DTMF
frequencies (no signal = xxx dB below the detected DTMF levels).

There exists a long text about DTMF detection somewhere on the net (I 
may have the link in the office but I'm on vacation now). What I 
remember is that a "correct" DTMF detection requires much more computing 
power as the present i4l implementation needs (much longer audio samples 
for the goertzel filter, a larger number of frequencies to check) and a 
standard test procedure with a lot of test cases which are not available 
to mortal humans (audio tapes from Bellcore IIRC)

Wolfgang
 >
 > Since the Goertzel filters are simple, they can be implemented in fixed
 > point math rather than floating point. At work, we've done this on a
 > Motorola 56301 DSP, which is a fixed-point DSP. I think there's an app
 > note from Moto on this - I'll check when I get into work today.
 >
 > -
 > To unsubscribe from this list: send the line "unsubscribe 
linux-kernel" in
 > the body of a message to majordomo@vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html
 > Please read the FAQ at  http://www.tux.org/lkml/
 >

Re: [Asterisk] DTMF noise

[Wolfgang Fritz]

Wolfgang Fritz wrote:

[cut]

> 
> There exists a long text about DTMF detection somewhere on the net (I 
> may have the link in the office but I'm on vacation now). What I 
> remember is that a "correct" DTMF detection requires much more computing 
> power as the present i4l implementation needs (much longer audio samples 
> for the goertzel filter, a larger number of frequencies to check) and a 
> standard test procedure with a lot of test cases which are not available 
> to mortal humans (audio tapes from Bellcore IIRC)
>

The link below is not the text I mean above but shows an approach which 
is similar to the present i4l implementation but has some improvements 
which may be good enough for us.

http://www.mitsubishichips.com/press/dtmf0199e.pdf

> Wolfgang
>  >
[cut]

Re: [Asterisk] DTMF noise

[Andrew McGregor]

--On Wednesday, January 08, 2003 16:49:06 +0100 Wolfgang Fritz 
<wolfgang.fritz@gmx.net> wrote:


> That is done in the isdn_audio DTMF detection but did not work very well
> with a number of phone sets I tested which seem to generate DTMF tones
> with strong harmonics. They may be out of spec but they exist.

Distortion :-)  Unfortunately, this stuff is cheap analog and even harmonic 
distortion will create havoc with that algorithm.

By the way, if you happened to be trying this with Quicknet hardware, there 
is a major overhaul to the driver coming that reduces the distortion levels 
in the analog stages of the hardware immensely.  (I suspect not as the 
thread is about isdn)

> I have a patch which adds a simple energy comparison and some
> plausablility checks to the DTMF eval code but does not look at the
> harmonics. That improved the detection with above phone sets.
>
> Maybe it would be better to reenable harmonic checks but comparing
> harmonic levels to the level of the fundamental instead of using
> absolute values as in the present implementation.

It certainly would.  And be relatively generous about the relative amount 
of harmonic allowed; something like 30%.  If you use absolute levels, 
you're at the mercy of noise and level calibration errors, both of which 
you have to assume are present.  If you require the relative level to be 
too low, you're at the mercy of distortion.

> OTOH I don't think its a good approach to check harmonics anyway but to
> check other non DTMF frequencies in the main speech band and only accept
> a DTMF if a DTMF frequency pair is present but no signal on the non DTMF
> frequencies (no signal = xxx dB below the detected DTMF levels).
>
> There exists a long text about DTMF detection somewhere on the net (I may
> have the link in the office but I'm on vacation now). What I remember is
> that a "correct" DTMF detection requires much more computing power as the
> present i4l implementation needs (much longer audio samples for the
> goertzel filter, a larger number of frequencies to check) and a standard
> test procedure with a lot of test cases which are not available to mortal
> humans (audio tapes from Bellcore IIRC)

There is a pretty good text linked in the source :-)

It's also near the top of a google for goertzel filter dtmf.

What I don't get is why the kernel links to this text, but implements one 
of the algorithms that the conclusion of that paper rejects as unable to 
satisfy the standard for DTMF detection?  Maybe the original implementor 
wanted to avoid doing matrix math in the kernel, or couldn't understand 
what to do.  The best algorithm was only twice as expensive in CPU, for 
dramatically better reliability and standards compliance.

Andrew

Re: [Asterisk] DTMF noise

[Jamie Lokier]

Wolfgang Fritz wrote:
> There exists a long text about DTMF detection somewhere on the net (I 
> may have the link in the office but I'm on vacation now). What I 
> remember is that a "correct" DTMF detection requires much more computing 
> power as the present i4l implementation needs (much longer audio samples 
> for the goertzel filter, a larger number of frequencies to check) and a 
> standard test procedure with a lot of test cases which are not available 
> to mortal humans (audio tapes from Bellcore IIRC)

Take a look at this:

	http://www-s.ti.com/sc/psheets/spra096a/spra096a.pdf

It describes an algorithm, plus test results.  It was tested on a TI
DSP using those very Bellcore tapes, plus another set of tests, and
passes both tests very well.

Of course your ISDN hardware + phone handset may have much worse
analogue circuitry, but I would hope the Bellcore tapes represent that
to some degree.

Unfortunately, TI have removed the version of their application node
which includes DSP source code.  It can be found here instead:

	http://sulcata6.cs.ccu.edu.tw/~vlsi/data/c54x/spra096.pdf

I guess if that _exact_ DSP algorithm were recoded in C, you could be
reasonably confident that the C implementation would pass those
Bellcore and MITEL tests with reasonable analogue hardware.  That's
probably the best you can do on the digital side.

enjoy,
-- Jamie

Re: [Asterisk] DTMF noise

[David D. Hagood]

Wolfgang Fritz wrote:

> Maybe it would be better to reenable harmonic checks but comparing
> harmonic levels to the level of the fundamental instead of using
> absolute values as in the present implementation.

You mean the code DOESN'T normalize the signal to the total energy 
first?!?!? YEEP!

The very FIRST thing you do is compute the total signal energy in the 
sample period, trivially reject if Etotal < MinETotal, then normalize 
all other signal energies to Etotal - that is a basic tenant of DSP.


> standard test procedure with a lot of test cases which are not available 
> to mortal humans (audio tapes from Bellcore IIRC)

I think we may have the test cases as WAVs at work, and I think they are 
freely distributable - I'll kick a reminder to my work account off to 
check later today.

Re: [Asterisk] DTMF noise

[Wolfgang Fritz]

David D. Hagood wrote:
 > Wolfgang Fritz wrote:
 >
 >> Maybe it would be better to reenable harmonic checks but comparing
 >> harmonic levels to the level of the fundamental instead of using
 >> absolute values as in the present implementation.
 >
 >
 > You mean the code DOESN'T normalize the signal to the total energy
 > first?!?!? YEEP!

No. The original code used _absolute_ thresholds for the DTMF tones and
the harmonics. That did not work very well.

My simple patch added a relative energy comparision of the DTMF tones
and a simple plausibiltity check (DTMF is only accepted if there is
exactly one DTNF pair and no/low signal level on the other DTMF
frequencies. That worked with my (very limited) tests.

 >
 > The very FIRST thing you do is compute the total signal energy in the
 > sample period, trivially reject if Etotal < MinETotal, then normalize
 > all other signal energies to Etotal - that is a basic tenant of DSP.
 >

My patch did a first step in that direction, but took only the energy
on the DTMF frequencies. That does not seem to be sufficient.

Another thing which may improve resistance against false DTMF detection
would be to require more than one consecutive samples to contain a valid
DTMF tone. See the link in one of my posts on lkml.

 >
 >> standard test procedure with a lot of test cases which are not
 >> available to mortal humans (audio tapes from Bellcore IIRC)
 >
 >
 > I think we may have the test cases as WAVs at work, and I think they are
 > freely distributable - I'll kick a reminder to my work account off to
 > check later today.
 >

That would be nice. But that must be a rather big chunk of data - the
Mitel tape alone contains 30 minutes of speech, the Bellcore tapes even
more. Too much for my dialup line, I'm afraid.

Wolfgang

Re: [Asterisk] DTMF noise

[David D. Hagood]

Well, I found out that while we have the DTMF test tape at work, it is 
exactly that - a cassette tape that is copyrighted. So, no easy/legal 
way to make it available for testing...

Re: [Asterisk] DTMF noise

[Matti Aarnio]

On Thu, Jan 09, 2003 at 05:32:51PM -0600, David D. Hagood wrote:
> Well, I found out that while we have the DTMF test tape at work, it is 
> exactly that - a cassette tape that is copyrighted. So, no easy/legal 
> way to make it available for testing...

What does such tape contain ?
 - DTMF tones buried in various degrees of distortions,
   which should be decodable ?
 - DTMF tones buried in varying noises which should not be
   decodable ?
 - Other multi-tone signals which should not decode ?

For the last, I know of cases where test was done by playing a radio 
station on the decoder for 2-3 days, and seeing when does it trigger
DTMFs (if ever).

For that matter, the Linux kernel ISDN audio DTMF detection is exactly
of the last variant.

/Matti Aarnio

Re: [Asterisk] DTMF noise

[David D. Hagood]

Matti Aarnio wrote:

> What does such tape contain ?
>  - DTMF tones buried in various degrees of distortions,
>    which should be decodable ?
>  - DTMF tones buried in varying noises which should not be
>    decodable ?
>  - Other multi-tone signals which should not decode ?

All that, plus voice (to detect falsing).

It's about 30 minutes long.

Re: [Asterisk] DTMF noise

[Roy Sigurd Karlsbakk]

> My simple patch added a relative energy comparision of the DTMF tones
> and a simple plausibiltity check (DTMF is only accepted if there is
> exactly one DTNF pair and no/low signal level on the other DTMF
> frequencies. That worked with my (very limited) tests.

I'm not sure if we're at the source of the problem. I mean - it should be 
possible to set a minimum length as well, so just touching a key won't be 
accepted. I beleive this'll remove most of the falsly detected dtmf signals 
as well, as noone really holds the same tone for a long time while speaking

roy
-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: [Asterisk] DTMF noise

[Thomas Tonino]

David D. Hagood wrote:

> The basic idea is that you have 8 filters (for the 4 row and 4 column 
> frequencies), as well as 8 filters looking at the first harmonic of the 
> 8 frequencies. You then compare the energies in each frequency - if you 
> see significant energy in the harmonic filter bank, discard the signal. 
> That prevents you from detecting speech as DTMF, since speech will 
> usually have harmonics that a good DTMF signal won't.

The original idea does one better by splitting high and low bands first. If that 
is combined with Goertzel it might be even better: by looking at how much low 
band energy there is total versus the low detected tone, and the same for the 
high band total versus the high band detected tone. Only if the detected tone is 
sufficiently strong compared to the total band it is in should the tone be 
triggered.

But it may be more expensive computationally than doing twice the number of 
Goertzel filters.

Harmonics seem like a bad idea. In between frequencies are better, but using the 
total band energy must be the most sure way to detect interference.


Thomas

Re: [Asterisk] DTMF noise

[David D. Hagood]

Thomas Tonino wrote:

> The original idea does one better by splitting high and low bands first. 

<snip>

> But it may be more expensive computationally than doing twice the number 
> of Goertzel filters.

Not really - the Goertzel filter is a fairly cheap filter. A filter to 
split the low band frequencies off from the high band, then doing the 8 
Goertzel filters for tone detection burns more MIPS than just doing the 
8 Goertzel filters. Furthurmore, the result is the same - you get the 
energies in the 8 tones.


> Harmonics seem like a bad idea. In between frequencies are better, but 
> using the total band energy must be the most sure way to detect 
> interference.
> 

Not really. The problem with the total energy approach is that the least 
amount of real noise will prevent tone detection, and if you set the 
threshold high enough that white noise does not prevent tone detection 
then you get falsing in voice.

If you have 1mW/Hz of white noise, then that is 2700 mW of noise power 
across the band. With voice, you get one of two types of "noise" - 
either voiced signals with lots of energy at harmonicly related 
frequencies, or unvoiced fricatives ("s", "f") that are basically white 
noise. A voiced signal with a total power of 2000 mW of power and a 
fundemental at one of the tone frequencies might have 1000 mW of power 
at the tone energy, and the remaining 1000 mW on the harmonics. Yet, if 
your noise threshold is set to not be blocked by the white noise case 
(2700 mW of power), then it would accept the voice case (1000 mW of power).

The 16 filter algorithm is the one we use in radio - it lets us pick a 
tone out of a staticy signal without falsing on voice.

It helps to think of looking at the signal on an audio spectrum analyzer 
- a DTMF tone looks like 2 peaks. If the signal in question has more 
than 2 peaks, it isn't DTMF.

DTMF noise

[Roy Sigurd Karlsbakk]

hi

when dialing out from the D-link MGCP phone (they actually work now - most of 
the time), I get lots of DTMF noise whenever the other person talks. I only 
get this from the MGCP phone - not with MSN messenger. This seems to be an 
error in isdn4linux falsely detecting DTMF in speech with Asterisk creating 
the actual noise.

This testing has been done on hhe following cards (from lspci)

02:09.0 Network controller: Cologne Chip Designs GmbH ISDN network controller 
[HFC-PCI] (rev 02)
02:0a.0 Network controller: Cologne Chip Designs GmbH ISDN network controller 
[HFC-PCI] (rev 02)
02:0b.0 Network controller: Cologne Chip Designs GmbH ISDN network controller 
[HFC-PCI] (rev 02)

roy

-- 
Roy Sigurd Karlsbakk, Datavaktmester
ProntoTV AS - http://www.pronto.tv/
Tel: +47 9801 3356

Computers are like air conditioners.
They stop working when you open Windows.

Re: [Asterisk] DTMF noise

[Mark Spencer]

The DTMF detector in the linux kernel is fairly simplistic and doesn't do
many relative energy tests.  The Zapata library has a much better tone
detector, but it is FP, and so would have to be made fixed point.  If
nothing else, it may provide some lessons for the ISDN folks.

Mark

On Tue, 7 Jan 2003, Roy Sigurd Karlsbakk wrote:

> hi
>
> when dialing out from the D-link MGCP phone (they actually work now - most of
> the time), I get lots of DTMF noise whenever the other person talks. I only
> get this from the MGCP phone - not with MSN messenger. This seems to be an
> error in isdn4linux falsely detecting DTMF in speech with Asterisk creating
> the actual noise.
>
> This testing has been done on hhe following cards (from lspci)
>
> 02:09.0 Network controller: Cologne Chip Designs GmbH ISDN network controller
> [HFC-PCI] (rev 02)
> 02:0a.0 Network controller: Cologne Chip Designs GmbH ISDN network controller
> [HFC-PCI] (rev 02)
> 02:0b.0 Network controller: Cologne Chip Designs GmbH ISDN network controller
> [HFC-PCI] (rev 02)
>
> roy
>
> --
> Roy Sigurd Karlsbakk, Datavaktmester
> ProntoTV AS - http://www.pronto.tv/
> Tel: +47 9801 3356
>
> Computers are like air conditioners.
> They stop working when you open Windows.
>