[PATCH 0/2] RFC: support for audio wall clock

[PATCH 0/2] RFC: support for audio wall clock

[Pierre-Louis Bossart]

Attached a set of patches to show how the audio wall clock (eg. the
HDAudio one) can be used to provide precise audio timestamps and track
the drift between audio time and system time. On my laptop I am able
to track a 7ppm delta.  I will explain in more details how this can be
used during LPC in San Diego, but with the summer coming I'd like to
get feedback as there are changes to the core/alsa-lib that haven't
been modified in years.

Pierre-Louis Bossart (2):
  ALSA: core: add hooks for audio timestamps read from WALLCLOCK
  ALSA: hda: support for wallclock timestamps

 include/sound/asound.h    |    7 +++-
 include/sound/pcm.h       |    2 +
 sound/core/pcm_lib.c      |   15 ++++++-
 sound/core/pcm_native.c   |    2 +
 sound/pci/hda/hda_intel.c |  106 ++++++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 128 insertions(+), 4 deletions(-)

-- 
1.7.6.5

[PATCH 0/2] RFC: support for audio wall clock

[Pierre-Louis Bossart]

V2 of patches to show how the audio wall clock (eg. the
HDAudio one) can be used to provide precise audio timestamps and track
the drift between audio time and system time. On my laptop I am able
to track a 7ppm delta.  
I will explain in more details how this can be
used during LPC in San Diego, but with the summer coming I'd like to
get feedback as there are changes to the core/alsa-lib that haven't
been modified in years.

TODO:
- 64-bit alignment, verification of pcm_compat.c changes
- handling of digital inputs, need to disable wall-clock timestamps in that case (feedback from Clemens)
- check differences in protocol
- change timecounter to avoid accumulation of rounding errors in cycle->ns conversions
- check overflows
- rework the timecounter init on trigger

Pierre-Louis Bossart (2):
  ALSA: core: add hooks for audio timestamps read from WALLCLOCK
  ALSA: hda: support for wallclock timestamps

 include/sound/asound.h    |    7 ++++-
 include/sound/pcm.h       |    2 +
 sound/core/pcm_compat.c   |   13 +++++++++-
 sound/core/pcm_lib.c      |   15 ++++++++++-
 sound/core/pcm_native.c   |    2 +
 sound/pci/hda/hda_intel.c |   54 ++++++++++++++++++++++++++++++++++++++++++++-
 6 files changed, 86 insertions(+), 7 deletions(-)

-- 
1.7.6.5

[PATCH 1/2] ALSA: core: add hooks for audio timestamps read from WALLCLOCK

[Pierre-Louis Bossart]

Add new .audio_wallclock routine to enable fine-grain synchronization
between monotonic system time and audio hardware time.
Prior to this patch, the clock drift estimation was handled in
user-space by comparing frames and system time.
Using the wallclock, if supported in hardware, allows for a
much better sub-microsecond precision and a common drift tracking for
all devices sharing the same wall clock (master clock).

TODO: handle 64-bit alignment

Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
---
 include/sound/asound.h  |    7 +++++--
 include/sound/pcm.h     |    2 ++
 sound/core/pcm_compat.c |   13 +++++++++++--
 sound/core/pcm_lib.c    |   15 +++++++++++++--
 sound/core/pcm_native.c |    2 ++
 5 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/include/sound/asound.h b/include/sound/asound.h
index 0876a1e..b889585 100644
--- a/include/sound/asound.h
+++ b/include/sound/asound.h
@@ -152,7 +152,7 @@ struct snd_hwdep_dsp_image {
  *                                                                           *
  *****************************************************************************/
 
-#define SNDRV_PCM_VERSION		SNDRV_PROTOCOL_VERSION(2, 0, 10)
+#define SNDRV_PCM_VERSION		SNDRV_PROTOCOL_VERSION(2, 0, 11)
 
 typedef unsigned long snd_pcm_uframes_t;
 typedef signed long snd_pcm_sframes_t;
@@ -274,6 +274,7 @@ typedef int __bitwise snd_pcm_subformat_t;
 #define SNDRV_PCM_INFO_JOINT_DUPLEX	0x00200000	/* playback and capture stream are somewhat correlated */
 #define SNDRV_PCM_INFO_SYNC_START	0x00400000	/* pcm support some kind of sync go */
 #define SNDRV_PCM_INFO_NO_PERIOD_WAKEUP	0x00800000	/* period wakeup can be disabled */
+#define SNDRV_PCM_INFO_HAS_WALL_CLOCK   0x01000000      /* has audio wall clock for audio/system time sync */
 #define SNDRV_PCM_INFO_FIFO_IN_FRAMES	0x80000000	/* internal kernel flag - FIFO size is in frames */
 
 typedef int __bitwise snd_pcm_state_t;
@@ -422,7 +423,8 @@ struct snd_pcm_status {
 	snd_pcm_uframes_t avail_max;	/* max frames available on hw since last status */
 	snd_pcm_uframes_t overrange;	/* count of ADC (capture) overrange detections from last status */
 	snd_pcm_state_t suspended_state; /* suspended stream state */
-	unsigned char reserved[60];	/* must be filled with zero */
+	struct timespec audio_tstamp;	/* audio wall clock timestamp */
+	unsigned char reserved[60-sizeof(struct timespec)]; /* must be filled with zero */
 };
 
 struct snd_pcm_mmap_status {
@@ -431,6 +433,7 @@ struct snd_pcm_mmap_status {
 	snd_pcm_uframes_t hw_ptr;	/* RO: hw ptr (0...boundary-1) */
 	struct timespec tstamp;		/* Timestamp */
 	snd_pcm_state_t suspended_state; /* RO: suspended stream state */
+	struct timespec audio_tstamp;	/* audio wall clock timestamp */
 };
 
 struct snd_pcm_mmap_control {
diff --git a/include/sound/pcm.h b/include/sound/pcm.h
index 68372bc..d4d65bf 100644
--- a/include/sound/pcm.h
+++ b/include/sound/pcm.h
@@ -71,6 +71,8 @@ struct snd_pcm_ops {
 	int (*prepare)(struct snd_pcm_substream *substream);
 	int (*trigger)(struct snd_pcm_substream *substream, int cmd);
 	snd_pcm_uframes_t (*pointer)(struct snd_pcm_substream *substream);
+	int (*wall_clock)(struct snd_pcm_substream *substream,
+			  struct timespec *audio_ts);
 	int (*copy)(struct snd_pcm_substream *substream, int channel,
 		    snd_pcm_uframes_t pos,
 		    void __user *buf, snd_pcm_uframes_t count);
diff --git a/sound/core/pcm_compat.c b/sound/core/pcm_compat.c
index 91cdf94..ab4c953 100644
--- a/sound/core/pcm_compat.c
+++ b/sound/core/pcm_compat.c
@@ -190,7 +190,8 @@ struct snd_pcm_status32 {
 	u32 avail_max;
 	u32 overrange;
 	s32 suspended_state;
-	unsigned char reserved[60];
+	struct timespec audio_tstamp;
+	unsigned char reserved[60-sizeof(struct timespec)];
 } __attribute__((packed));
 
 
@@ -215,7 +216,9 @@ static int snd_pcm_status_user_compat(struct snd_pcm_substream *substream,
 	    put_user(status.avail, &src->avail) ||
 	    put_user(status.avail_max, &src->avail_max) ||
 	    put_user(status.overrange, &src->overrange) ||
-	    put_user(status.suspended_state, &src->suspended_state))
+	    put_user(status.suspended_state, &src->suspended_state) ||
+	    put_user(status.audio_tstamp.tv_sec, &src->audio_tstamp.tv_sec) ||
+	    put_user(status.audio_tstamp.tv_nsec, &src->audio_tstamp.tv_nsec))
 		return -EFAULT;
 
 	return err;
@@ -364,6 +367,7 @@ struct snd_pcm_mmap_status32 {
 	u32 hw_ptr;
 	struct compat_timespec tstamp;
 	s32 suspended_state;
+	struct compat_timespec audio_tstamp;
 } __attribute__((packed));
 
 struct snd_pcm_mmap_control32 {
@@ -426,12 +430,17 @@ static int snd_pcm_ioctl_sync_ptr_compat(struct snd_pcm_substream *substream,
 	sstatus.hw_ptr = status->hw_ptr % boundary;
 	sstatus.tstamp = status->tstamp;
 	sstatus.suspended_state = status->suspended_state;
+	sstatus.audio_tstamp = status->audio_tstamp;
 	snd_pcm_stream_unlock_irq(substream);
 	if (put_user(sstatus.state, &src->s.status.state) ||
 	    put_user(sstatus.hw_ptr, &src->s.status.hw_ptr) ||
 	    put_user(sstatus.tstamp.tv_sec, &src->s.status.tstamp.tv_sec) ||
 	    put_user(sstatus.tstamp.tv_nsec, &src->s.status.tstamp.tv_nsec) ||
 	    put_user(sstatus.suspended_state, &src->s.status.suspended_state) ||
+	    put_user(sstatus.audio_tstamp.tv_sec,
+		    &src->s.status.audio_tstamp.tv_sec) ||
+	    put_user(sstatus.audio_tstamp.tv_nsec,
+		    &src->s.status.audio_tstamp.tv_nsec) ||
 	    put_user(scontrol.appl_ptr, &src->c.control.appl_ptr) ||
 	    put_user(scontrol.avail_min, &src->c.control.avail_min))
 		return -EFAULT;
diff --git a/sound/core/pcm_lib.c b/sound/core/pcm_lib.c
index 8f312fa..beece7d 100644
--- a/sound/core/pcm_lib.c
+++ b/sound/core/pcm_lib.c
@@ -315,6 +315,7 @@ static int snd_pcm_update_hw_ptr0(struct snd_pcm_substream *substream,
 	unsigned long jdelta;
 	unsigned long curr_jiffies;
 	struct timespec curr_tstamp;
+	struct timespec audio_tstamp;
 
 	old_hw_ptr = runtime->status->hw_ptr;
 
@@ -326,9 +327,17 @@ static int snd_pcm_update_hw_ptr0(struct snd_pcm_substream *substream,
 	 */
 	pos = substream->ops->pointer(substream);
 	curr_jiffies = jiffies;
-	if (runtime->tstamp_mode == SNDRV_PCM_TSTAMP_ENABLE)
+	if (runtime->tstamp_mode == SNDRV_PCM_TSTAMP_ENABLE) {
 		snd_pcm_gettime(runtime, (struct timespec *)&curr_tstamp);
 
+		if ((runtime->hw.info & SNDRV_PCM_INFO_HAS_WALL_CLOCK) &&
+			(substream->ops->wall_clock))
+			substream->ops->wall_clock(substream, &audio_tstamp);
+		else
+			/* no audio tstamp available, initialize anyway */
+			audio_tstamp = curr_tstamp;
+	}
+
 	if (pos == SNDRV_PCM_POS_XRUN) {
 		xrun(substream);
 		return -EPIPE;
@@ -506,8 +515,10 @@ static int snd_pcm_update_hw_ptr0(struct snd_pcm_substream *substream,
 	runtime->hw_ptr_base = hw_base;
 	runtime->status->hw_ptr = new_hw_ptr;
 	runtime->hw_ptr_jiffies = curr_jiffies;
-	if (runtime->tstamp_mode == SNDRV_PCM_TSTAMP_ENABLE)
+	if (runtime->tstamp_mode == SNDRV_PCM_TSTAMP_ENABLE) {
 		runtime->status->tstamp = curr_tstamp;
+		runtime->status->audio_tstamp = audio_tstamp;
+	}
 
 	return snd_pcm_update_state(substream, runtime);
 }
diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c
index 53b5ada..c5206f4 100644
--- a/sound/core/pcm_native.c
+++ b/sound/core/pcm_native.c
@@ -594,6 +594,8 @@ int snd_pcm_status(struct snd_pcm_substream *substream,
 		snd_pcm_update_hw_ptr(substream);
 		if (runtime->tstamp_mode == SNDRV_PCM_TSTAMP_ENABLE) {
 			status->tstamp = runtime->status->tstamp;
+			status->audio_tstamp =
+				runtime->status->audio_tstamp;
 			goto _tstamp_end;
 		}
 	}
-- 
1.7.6.5

[PATCH 2/2] ALSA: hda: support for wallclock timestamps

[Pierre-Louis Bossart]

reuse code from clocksource to handle wall clock counter.
Since wrapparound occurs, the timestamps are reinitialized with
monotonic time on a trigger.

TODO:
- only re-init timecounter if there was no device active.
- Keep the same timestamp for all devices on same chip.
- make sure no overflow occurs in the 125/3 scaling implementation

Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
---
 sound/pci/hda/hda_intel.c |   54 ++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 53 insertions(+), 1 deletions(-)

diff --git a/sound/pci/hda/hda_intel.c b/sound/pci/hda/hda_intel.c
index 86758dd..9bb795b 100644
--- a/sound/pci/hda/hda_intel.c
+++ b/sound/pci/hda/hda_intel.c
@@ -46,6 +46,10 @@
 #include <linux/mutex.h>
 #include <linux/reboot.h>
 #include <linux/io.h>
+
+#include <linux/clocksource.h>
+#include <linux/time.h>
+
 #ifdef CONFIG_X86
 /* for snoop control */
 #include <asm/pgtable.h>
@@ -496,6 +500,10 @@ struct azx {
 
 	/* reboot notifier (for mysterious hangup problem at power-down) */
 	struct notifier_block reboot_notifier;
+
+	struct timecounter  azx_tc;
+	struct cyclecounter azx_cc;
+
 };
 
 /* driver types */
@@ -1700,6 +1708,45 @@ static inline void azx_release_device(struct azx_dev *azx_dev)
 	azx_dev->opened = 0;
 }
 
+static cycle_t azx_cc_read(const struct cyclecounter *cc)
+{
+	struct azx *chip = container_of(cc, struct azx, azx_cc);
+
+	return azx_readl(chip, WALLCLK);
+}
+
+static void azx_timecounter_init(struct azx *chip)
+{
+	struct timecounter *tc = &chip->azx_tc;
+	struct cyclecounter *cc = &chip->azx_cc;
+	struct timespec ts;
+	u64 nsec;
+
+	cc->read = azx_cc_read;
+	cc->mask = CLOCKSOURCE_MASK(32);
+
+#define NBITS_NS 12 /* accuracy issues in cyc2ns with lower number of bits */
+	cc->mult = clocksource_khz2mult(24000, NBITS_NS);
+	cc->shift = NBITS_NS;
+
+	ktime_get_ts(&ts);
+	nsec = timespec_to_ns(&ts);
+	timecounter_init(tc, cc, nsec);
+}
+
+static int azx_get_wallclock_tstamp(struct snd_pcm_substream *substream,
+				struct timespec *ts)
+{
+	struct azx_pcm *apcm = snd_pcm_substream_chip(substream);
+	struct azx *chip = apcm->chip;
+	u64 nsec;
+
+	nsec = timecounter_read(&chip->azx_tc);
+	*ts = ns_to_timespec(nsec);
+
+	return 0;
+}
+
 static struct snd_pcm_hardware azx_pcm_hw = {
 	.info =			(SNDRV_PCM_INFO_MMAP |
 				 SNDRV_PCM_INFO_INTERLEAVED |
@@ -1709,6 +1756,7 @@ static struct snd_pcm_hardware azx_pcm_hw = {
 				 /* SNDRV_PCM_INFO_RESUME |*/
 				 SNDRV_PCM_INFO_PAUSE |
 				 SNDRV_PCM_INFO_SYNC_START |
+				 SNDRV_PCM_INFO_HAS_WALL_CLOCK |
 				 SNDRV_PCM_INFO_NO_PERIOD_WAKEUP),
 	.formats =		SNDRV_PCM_FMTBIT_S16_LE,
 	.rates =		SNDRV_PCM_RATE_48000,
@@ -1982,8 +2030,10 @@ static int azx_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
 	}
 	spin_unlock(&chip->reg_lock);
 	if (start) {
-		if (nsync == 1)
+		if (nsync == 1) {
+			azx_timecounter_init(chip);
 			return 0;
+		}
 		/* wait until all FIFOs get ready */
 		for (timeout = 5000; timeout; timeout--) {
 			nwait = 0;
@@ -1999,6 +2049,7 @@ static int azx_pcm_trigger(struct snd_pcm_substream *substream, int cmd)
 				break;
 			cpu_relax();
 		}
+		azx_timecounter_init(chip);
 	} else {
 		/* wait until all RUN bits are cleared */
 		for (timeout = 5000; timeout; timeout--) {
@@ -2240,6 +2291,7 @@ static struct snd_pcm_ops azx_pcm_ops = {
 	.prepare = azx_pcm_prepare,
 	.trigger = azx_pcm_trigger,
 	.pointer = azx_pcm_pointer,
+	.wall_clock =  azx_get_wallclock_tstamp,
 	.mmap = azx_pcm_mmap,
 	.page = snd_pcm_sgbuf_ops_page,
 };
-- 
1.7.6.5

Re: [PATCH 0/2] RFC: support for audio wall clock

[Raymond Yau]

2012/6/29 Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>:
> V2 of patches to show how the audio wall clock (eg. the
> HDAudio one) can be used to provide precise audio timestamps and track
> the drift between audio time and system time. On my laptop I am able
> to track a 7ppm delta.
> I will explain in more details how this can be
> used during LPC in San Diego, but with the summer coming I'd like to
> get feedback as there are changes to the core/alsa-lib that haven't
> been modified in years.
>
> TODO:
> - 64-bit alignment, verification of pcm_compat.c changes
> - handling of digital inputs, need to disable wall-clock timestamps in that case (feedback from Clemens)
> - check differences in protocol
> - change timecounter to avoid accumulation of rounding errors in cycle->ns conversions
> - check overflows
> - rework the timecounter init on trigger

Do suspsend/resume affect the usage of the wall clock ?

Those au88x0 also have a 32 bit register which return the ticks from a
12.288 Mz oscillator (i.e. 256 x 48000Hz) , this register wrap around
in around 349.5 seconds

Can this clock be used to determine the drift too ?

Re: [PATCH 0/2] RFC: support for audio wall clock

[Takashi Iwai]

At Thu, 28 Jun 2012 16:12:33 -0500,
Pierre-Louis Bossart wrote:
> 
> V2 of patches to show how the audio wall clock (eg. the
> HDAudio one) can be used to provide precise audio timestamps and track
> the drift between audio time and system time. On my laptop I am able
> to track a 7ppm delta.  
> I will explain in more details how this can be
> used during LPC in San Diego, but with the summer coming I'd like to
> get feedback as there are changes to the core/alsa-lib that haven't
> been modified in years.

Sorry for the late reply.  I've looked over the things around my
vacation...

This version looks better, so if other people have no objection, I can
merge it.  The extension of mmap_status seems still fitting with 64bit
char size, so it should be OK.

But, one thing -- can't we reuse the normal tstamp field for the audio
wallclock?  That is, add a new tstamp mode, and then put wallclock
tstamp there instead of the normal system tstamp.  Just an idea.


thanks,

Takashi

> 
> TODO:
> - 64-bit alignment, verification of pcm_compat.c changes
> - handling of digital inputs, need to disable wall-clock timestamps in that case (feedback from Clemens)
> - check differences in protocol
> - change timecounter to avoid accumulation of rounding errors in cycle->ns conversions
> - check overflows
> - rework the timecounter init on trigger
> 
> Pierre-Louis Bossart (2):
>   ALSA: core: add hooks for audio timestamps read from WALLCLOCK
>   ALSA: hda: support for wallclock timestamps
> 
>  include/sound/asound.h    |    7 ++++-
>  include/sound/pcm.h       |    2 +
>  sound/core/pcm_compat.c   |   13 +++++++++-
>  sound/core/pcm_lib.c      |   15 ++++++++++-
>  sound/core/pcm_native.c   |    2 +
>  sound/pci/hda/hda_intel.c |   54 ++++++++++++++++++++++++++++++++++++++++++++-
>  6 files changed, 86 insertions(+), 7 deletions(-)
> 
> -- 
> 1.7.6.5
> 
> _______________________________________________
> Alsa-devel mailing list
> Alsa-devel@alsa-project.org
> http://mailman.alsa-project.org/mailman/listinfo/alsa-devel
> 

Re: [PATCH 0/2] RFC: support for audio wall clock

[Pierre-Louis Bossart]

On 7/18/2012 12:10 PM, Takashi Iwai wrote:
> Sorry for the late reply. I've looked over the things around my 
> vacation... This version looks better, so if other people have no 
> objection, I can merge it. The extension of mmap_status seems still 
> fitting with 64bit char size, so it should be OK. But, one thing -- 
> can't we reuse the normal tstamp field for the audio wallclock? That 
> is, add a new tstamp mode, and then put wallclock tstamp there instead 
> of the normal system tstamp. Just an idea. thanks, Takashi

Thanks for reviewing the updated patches. No, the audio wall clock needs 
to come in addition to the regular system tstamp. This is needed for 
applications such as PulseAudio who try to estimate the drift between 
system time and audio time. If you remove the system time information, 
they can't program a timer any longer.
Better wait a bit to merge, I need to work on all the TODOs, complete 
some additional validation for corner cases and get your feedback live 
at LPC.
-Pierre

Re: [PATCH 0/2] RFC: support for audio wall clock

[Pierre-Louis Bossart]

On 6/29/2012 8:03 PM, Raymond Yau wrote:
> Do suspsend/resume affect the usage of the wall clock ? Those au88x0 
> also have a 32 bit register which return the ticks from a 12.288 Mz 
> oscillator (i.e. 256 x 48000Hz) , this register wrap around in around 
> 349.5 seconds Can this clock be used to determine the drift too ? 

In theory any clock can be used, as long as it's synchronous with the 
bitclock used in the serial stream. I am planning to use our 19.2 MHz 
clock on non HDAudio platforms, a 12.288 would work at well. The cycle 
count will be translated to ns, the initial clock base has a marginal 
impact on the result.
I haven't validated anything related to suspend/resume. It should work 
if the trigger is invoked on resume where the audio wallclock is 
reinitialized with the system time
-Pierre

Re: [PATCH 0/2] RFC: support for audio wall clock

[Takashi Iwai]

At Fri, 20 Jul 2012 14:47:24 -0500,
Pierre-Louis Bossart wrote:
> 
> On 7/18/2012 12:10 PM, Takashi Iwai wrote:
> > Sorry for the late reply. I've looked over the things around my 
> > vacation... This version looks better, so if other people have no 
> > objection, I can merge it. The extension of mmap_status seems still 
> > fitting with 64bit char size, so it should be OK. But, one thing -- 
> > can't we reuse the normal tstamp field for the audio wallclock? That 
> > is, add a new tstamp mode, and then put wallclock tstamp there instead 
> > of the normal system tstamp. Just an idea. thanks, Takashi
> 
> Thanks for reviewing the updated patches. No, the audio wall clock needs 
> to come in addition to the regular system tstamp. This is needed for 
> applications such as PulseAudio who try to estimate the drift between 
> system time and audio time. If you remove the system time information, 
> they can't program a timer any longer.
> Better wait a bit to merge, I need to work on all the TODOs, complete 
> some additional validation for corner cases and get your feedback live 
> at LPC.

Alright.  Let's cook a bit longer.


Takashi