[Xenomai-help] NI 6251 "wrong" samples

[Xenomai-help] NI 6251 "wrong" samples

[Daniele Nicolodi]

Hello.

I'm working with a NI PCI 6251 ADC board. I configured it to sample
channels 6 and 7, configured as differential analog inputs, at 50 kHz
with a conversion time of 2000 ns. I'm using a Xenomai kernel and the
Analogy drivers, a port of comedi drivers over the Xenomai real time
device driver abstraction layer.

The systems works well (I do not fully understand the properties fo the
noise spectrum of the sample data, see my other email to the mailing
list, but this is quite mynor) except that now and then I receive a
clearly "wrong" sample from the board.

I think I receive a wrong sample because I'm testing the system
acquiring a simple sine wave generated from a function generator, and
sometimes I obtain a sample that is clearly not where expected and I I
hardly believe that the discrepancy is due to physical noise.

I'm unable to identify a pattern of when those "wrong" samples appear in
my output, so I do not believe the issue is due to wrong FIFO
synchronization.

Has someone ever observed a similar behavior? Does someone has a
suggestion on how I can try to investigate it?

Thanks. Cheers,
-- 
Daniele

Re: [Xenomai-help] [comedi] NI 6251 "wrong" samples

[Daniele Nicolodi]

Hello. Sorry for the long delay, I had some real work to get done with
the acquisition system and I had to implement a work-around for this
problem...

I have been investigating the issue further, but its random nature makes
this quite difficult.

> The observation that the "wrong" sample occurs on the last sample of the
> scan sounds useful.  Does it occur randomly or on every scan?

I must retire this statement. The "wrong" samples can occur on any
channel. Some are just more probable than other. The "wrong" samples do
NOT occur at each scan, and they look to be completely random in their
occurrence, I'm trying to find a pattern but without luck up to now.

> Also, are you using the TRIG_WAKE_EOS command flag (or equivalently,
> setting stop_arg to TRIG_COUNT and stop_count to 1)?  If so, does the
> wrong sample still occur when the TRIG_WAKE_EOS flag is not set?  (You
> might need to look harder for the wrong samples when TRIG_WAKE_EOS is
> not set, as they might occur much less frequently and they might occur
> at any position in the scan.

No. I'm running a continuous memory mapped acquisition. The only maybe
uncommon thing I'm doing is that I'm reading the mmapped buffer in fixed
chunks, rather than reading all the samples available each time. The
number of samples I read each time is larger than the 4095 samples after
which the hardware sends the FIFO half full trigger event.

Suggestions?

Thanks. Cheers,
-- 
Daniele

Re: [Xenomai-help] [comedi] NI 6251 "wrong" samples

[Daniele Nicolodi]

On 17/06/10 11:07, Daniele Nicolodi wrote:
> No. I'm running a continuous memory mapped acquisition. The only maybe
> uncommon thing I'm doing is that I'm reading the mmapped buffer in fixed
> chunks, rather than reading all the samples available each time. The
> number of samples I read each time is larger than the 4095 samples after
> which the hardware sends the FIFO half full trigger event.

I modified my code from a loop like:

unsigned int required = <compute required number of bytes>;
unsigned int read = 0;
void * buffer = a4l_mmap(dsc, ...);
while (1) {
  while (read < required) {
    read = a4l_pool(...);
    clock_nanosleep(...);
  }
  process(buffer, ...);
  a4l_buff_markrw(dsc, subdevice, required, ...);
}

to something like:

unsigned int required = <compute required number of bytes>;
unsigned int read = 0;
void *buffer = malloc(required);
while (1) {
  while (read < required) {
    a4l_sys_read(dsc, buffer + read, required - read);
  }
  process(buffer, ...);
}

I tested this setup over the weekend and the wrong samples went away!

Therefore i suspect that the problem is not related to FIFO
synchronization but to kernel space buffer handling.

I have tried to look at Analogy buffer handling, but the lack of any
comment in the source code makes understanding how it is supposed to
work a complex task.

Alexis, can you see any case where there may be a synchronization issue
like the one I'm seeing in your code? Any suggestion in where to look
for the bug?

Thanks. Cheers,
-- 
Daniele

Re: [Xenomai-help] [comedi] NI 6251 "wrong" samples

[Alexis Berlemont]

Hi,

On Mon, Jun 21, 2010 at 11:03 AM, Daniele Nicolodi <daniele@domain.hid> wrote:
> On 17/06/10 11:07, Daniele Nicolodi wrote:
>> No. I'm running a continuous memory mapped acquisition. The only maybe
>> uncommon thing I'm doing is that I'm reading the mmapped buffer in fixed
>> chunks, rather than reading all the samples available each time. The
>> number of samples I read each time is larger than the 4095 samples after
>> which the hardware sends the FIFO half full trigger event.
>
> I modified my code from a loop like:
>
> unsigned int required = <compute required number of bytes>;
> unsigned int read = 0;
> void * buffer = a4l_mmap(dsc, ...);
> while (1) {
>  while (read < required) {
>    read = a4l_pool(...);
>    clock_nanosleep(...);
>  }
>  process(buffer, ...);
>  a4l_buff_markrw(dsc, subdevice, required, ...);
> }
>
> to something like:
>
> unsigned int required = <compute required number of bytes>;
> unsigned int read = 0;
> void *buffer = malloc(required);
> while (1) {
>  while (read < required) {
>    a4l_sys_read(dsc, buffer + read, required - read);
>  }
>  process(buffer, ...);
> }
>
> I tested this setup over the weekend and the wrong samples went away!
>
> Therefore i suspect that the problem is not related to FIFO
> synchronization but to kernel space buffer handling.
>
> I have tried to look at Analogy buffer handling, but the lack of any
> comment in the source code makes understanding how it is supposed to
> work a complex task.
>
Please, tell me where comments are lacking, I will fix that.

> Alexis, can you see any case where there may be a synchronization issue
> like the one I'm seeing in your code? Any suggestion in where to look
> for the bug?
>

You told you that the bug was occuring randomly and that there was no
pattern in the repetition of the bad samples. But are you sure that
the wrong samples do not occur every modulo x * 64KB ?

If the bug is located in Analogy buffer handling, it should not be
specific to a driver. So, I will try to reproduce it with the testing
drivers. I will compare the mapped and unmapped acquisition results.
If I remember well, the "fake" driver outputs some deterministic
content.  If nothing comes out, I will try to use a NI device.

Many thanks for reporting that.

> Thanks. Cheers,
> --
> Daniele
>

Alexis.

Re: [Xenomai-help] [comedi] NI 6251 "wrong" samples

[Daniele Nicolodi]

On 21/06/10 13:59, Alexis Berlemont wrote:
>> I have tried to look at Analogy buffer handling, but the lack of any
>> comment in the source code makes understanding how it is supposed to
>> work a complex task.
>>
> Please, tell me where comments are lacking, I will fix that.

I find buffer.c and buffer.h to be under documented to be easily
understood. For example it is not clear what the *_count members of the
a4l_buf_t struct are for without a quite tedious inspection of the code.

>> Alexis, can you see any case where there may be a synchronization issue
>> like the one I'm seeing in your code? Any suggestion in where to look
>> for the bug?
>>
> 
> You told you that the bug was occuring randomly and that there was no
> pattern in the repetition of the bad samples. But are you sure that
> the wrong samples do not occur every modulo x * 64KB ?

I convinced myself that this is not the case. For sure the error is not
there at any ring buffer wrap around. I increased the buffer size to
128KB and there hasn't been any noticeable difference. Am I right
supposing that those 64KB you are referring to are the buffer size?

> If the bug is located in Analogy buffer handling, it should not be
> specific to a driver. So, I will try to reproduce it with the testing
> drivers. I will compare the mapped and unmapped acquisition results.
> If I remember well, the "fake" driver outputs some deterministic
> content.  If nothing comes out, I will try to use a NI device.

To detect the problem you need to feed a deterministic signal to your
device. It looks like every now and then an old sample is returned (I
say so because the wrong value is always in the signal range). If you
feed a constant value you would be unable to distinguish the wrong
sample from the good ones. If you are missing a function generator the
DAC channels of the NI ADC board work just fine, as long as you provide
proper low pass filters.

My test case is a phase-meter implementation. I'm testing the phase
noise of the phase-meter and thus I acquire a zero mean sine wave. Every
now and then the average of a fixed number of samples, containing an
integer number of sine waves, is far from zero. When I detect this
situation I dump the content of the current buffer, and in each dump
there is always a single wrong sample.

My code is embedded into a data acquisition framework I'm developing,
but I can provide the code, however I can take a little bit to have it
to compile stand alone.

The bug shows usually after some minutes to half an hour of data taking
at 10 to 50 kHz. It would be unpractical to save all that data to disk
for inspection.

Thanks. Cheers,
-- 
Daniele

Re: [Xenomai-help] [comedi] NI 6251 "wrong" samples

[Alexis Berlemont]

Hi,

Daniele Nicolodi wrote:
> On 21/06/10 13:59, Alexis Berlemont wrote:
> >> I have tried to look at Analogy buffer handling, but the lack of any
> >> comment in the source code makes understanding how it is supposed to
> >> work a complex task.
> >>
> > Please, tell me where comments are lacking, I will fix that.
> 
> I find buffer.c and buffer.h to be under documented to be easily
> understood. For example it is not clear what the *_count members of the
> a4l_buf_t struct are for without a quite tedious inspection of the code.
> 
> >> Alexis, can you see any case where there may be a synchronization issue
> >> like the one I'm seeing in your code? Any suggestion in where to look
> >> for the bug?
> >>
> > 
> > You told you that the bug was occuring randomly and that there was no
> > pattern in the repetition of the bad samples. But are you sure that
> > the wrong samples do not occur every modulo x * 64KB ?
> 
> I convinced myself that this is not the case. For sure the error is not
> there at any ring buffer wrap around. I increased the buffer size to
> 128KB and there hasn't been any noticeable difference. Am I right
> supposing that those 64KB you are referring to are the buffer size?
> 

Yes.

> > If the bug is located in Analogy buffer handling, it should not be
> > specific to a driver. So, I will try to reproduce it with the testing
> > drivers. I will compare the mapped and unmapped acquisition results.
> > If I remember well, the "fake" driver outputs some deterministic
> > content.  If nothing comes out, I will try to use a NI device.
> 
> To detect the problem you need to feed a deterministic signal to your
> device. It looks like every now and then an old sample is returned (I
> say so because the wrong value is always in the signal range). If you
> feed a constant value you would be unable to distinguish the wrong
> sample from the good ones. If you are missing a function generator the
> DAC channels of the NI ADC board work just fine, as long as you provide
> proper low pass filters.
> 

I was just starting to define how to debug it with the testing driver
"fake" but your additional comment made me doubt: especially the fact
that I will not notice it with a constant value as the wrong acquired
sample is an "old one".

Regarding that, would you consider possible that the user application
could read the data too early? Maybe there is a synchronization issue
between the interrupt and the DMA transfer. That would explain why
unoptimized accesses (unmapped buffer and read syscalls) make the bug
vanish. 

I may have messed something up with the mite transfer counters. I will
double-check the code in this location (mite.c).

If you re-read a faulting value after a while, does it get updated ?
You might find this question silly but what strikes me is that an
unmapped acquisition gives you no wrong sample; that means that the
data are properly copied by the DMA controller into the kernel memory
buffer, the very same buffer you map in user space in the optimized mode.

> My test case is a phase-meter implementation. I'm testing the phase
> noise of the phase-meter and thus I acquire a zero mean sine wave. Every
> now and then the average of a fixed number of samples, containing an
> integer number of sine waves, is far from zero. When I detect this
> situation I dump the content of the current buffer, and in each dump
> there is always a single wrong sample.
> 
> My code is embedded into a data acquisition framework I'm developing,
> but I can provide the code, however I can take a little bit to have it
> to compile stand alone.
> 
> The bug shows usually after some minutes to half an hour of data taking
> at 10 to 50 kHz. It would be unpractical to save all that data to disk
> for inspection.
> 
> Thanks. Cheers,
> -- 
> Daniele

-- 
Alexis.