Optimal synchronization of audio output

[cxrodgers]

Hey all, I'm embarking on a project to get my synchronization as precise as possible between the neural data I record, the video data of the behavior, and behavioral timestamps from Autopilot. The goal is to make PSTHs from auditory cortical neurons in response to sound, so I definitely need 5 ms precision, but 1 ms precision or better would be ideal. I re-read section 4.1 of the preprint, but what I'm trying to do is a little different, because for this, I don't need to minimize latency, I just want to have as precise an estimate as possible of when the sounds actually come out of the speaker.

I'm wondering if there's a way to get the EXACT tick time at which jack audio plays a frame? As I understand it, jackd calls process every frame time.

autopilot/autopilot/stim/sound/jackclient.py

Line 358 in 04b5968

def process(self, frames):

I think a frame is 1024 sample at 192 kHz, so that's every 5.3 ms. These frames get loaded into a buffer of length 3 frames (nper). So presumably they get played about 15.9 ms after they get loaded. I also think I understand that the frame size and buffer size are configurable options when booting jackd.

autopilot/autopilot/agents/pilot.py

Line 101 in 04b5968

jackd -P75 -p16 -t2000 -dalsa -dhw:sndrpihifiberry -P -rfs -n3 -s &

So I'm wondering if jackd outputs the tick time that it plays each frame in some way. If that's not possible, I could just take a tick time (or datetime.datetime.now) at the time that the sound is loaded into the buffer by process. I think this will precede the time it actually plays by 15.9 ms or so, right? And also, there will be a bit of random variability here, because process is not guaranteed to be called at exactly the same time as the current frame is being played by jackd, it could be delayed by other things, which is why we need that 3-frame buffer in the first place.

On a somewhat related note, Jonny can you say a bit more about the choice to have your pigpio fork "return isoformatted timestamps rather than tick numbers"?

autopilot/autopilot/hardware/gpio.py

Line 7 in 04b5968

returns isoformatted timestamps rather than tick numbers in callbacks. See the ``setup_pilot.sh`` script.

I assume that the raspberry pi occasionally synchronizes its clock with network time using ntp or similar, and so the relationship between ticks and isoformatted timestamps might change during an experiment, right? I will also need to synchronize times from Parent and Child pis, so I need to figure this out at some point. Is there a big performance difference between using datetime.datetime.now() and using something like pigpio's get_current_tick?

thanks!!

[cxrodgers]

I think I figured out something that might work! There are these properties of the jack client called last_frame_time and frames_since_cycle_start. If I store these params on every process call, alongside datetime.datetime.now, I think I can work out the relationship between frame times and clock time. Then I think I can assume that the sound comes out of the speakers NPER blocks after the process call.
https://jackclient-python.readthedocs.io/en/0.5.4/api.html#jack.Client.last_frame_time

[cxrodgers]

It would still be cool if I could figure out how to pulse a pin on the exact start of each block though. I can't quite make sense of the whole "master timebase" thing and where it comes from, even whether it's a software clock or a hardware clock

[sneakers-the-rat]

Jackd timing

Yes! so there are some calls to get specific transport timing in the client, eg

autopilot/autopilot/stim/sound/jackclient.py

Line 372 in 04b5968

state, pos = self.client.transport_query()

I used this to do exactly that - pulse a pin exactly when a sound starts:

autopilot/autopilot/stim/sound/jackclient.py

Line 455 in 04b5968

self.wait_until = self.client.last_frame_time+(self.blocksize*self.alsa_nperiods)

autopilot/autopilot/stim/sound/jackclient.py

Line 578 in 04b5968

def _wait_for_end(self):

and that gets it within 1ms (that's also using pigpio scripts which have a bit of startup latency). From those values you can tell when the frame you feed into the buffer will get pushed to the card.

Ideally what i had wanted to do was to be able to directly connect a jack port to pigpiod so you could just have jack flip the pin, but i never quite got there.

pigpio timestamps

The whole deal there is that pigpio sends up the event and the ticks for every callback. the tick is the 32 bits "seconds" part of a system timestamp, and so you need to send another 32 bit int that has the microseconds part. So the strategy is if you get a paired tick and timestamp, then you can use the offset between them to convert between ticks and timestamps.

you can see how those work by comparing the fork to the base branch: joan2937/pigpio@master...sneakers-the-rat:pigpio:master

where when the pi class is instantiated it calls this synchronize method:
https://github.com/sneakers-the-rat/pigpio/blob/3c237159e5995ec58cd673579bdd66a8d819b269/pigpio.py#L5312

which gets several pairs of ticks and system timestamps
https://github.com/sneakers-the-rat/pigpio/blob/3c237159e5995ec58cd673579bdd66a8d819b269/pigpio.py#L2115
by calling
https://github.com/sneakers-the-rat/pigpio/blob/3c237159e5995ec58cd673579bdd66a8d819b269/pigpio.c#L2373
which either gets the time or the ticks first, alternatingly, in case there was some delay between those, and then averages. I doubt that's the best way to do that, but it seemed to work pretty well.

Then if there is some unexpected tick (eg. the ticks wrap around, as they do) then the synchronize call is made again:
https://github.com/sneakers-the-rat/pigpio/blob/3c237159e5995ec58cd673579bdd66a8d819b269/pigpio.py#L5327

I assume that the raspberry pi occasionally synchronizes its clock with network time using ntp or similar, and so the relationship between ticks and isoformatted timestamps might change during an experiment, right?

It depends! You can configure your NTP client to behave in different ways( https://wiki.auto-pi-lot.com/index.php/NTP ) probably the best thing to do would be to synchronize before the experiment and then turn off NTP during it to avoid that. +1 to the eternal need to actually refactor the task running logic to add hooks for the different phases like before_run after_run so that would be easy to do in a provenance preserving way. Another thing would be to just periodically call the synchronize method on the pigpio.pi object

I will also need to synchronize times from Parent and Child pis, so I need to figure this out at some point.

here's a guide! https://wiki.auto-pi-lot.com/index.php/NTP

Is there a big performance difference between using datetime.datetime.now() and using something like pigpio's get_current_tick?

I assume yes but haven't measured. the pigpio method has to do a whole socket-based message exchange with the daemon so i assume that would be slower than directly calling python's time methods (datetime.now() is a very thin wrapper around time.time() https://github.com/python/cpython/blob/ecad802e3f9ef7063e7d7fe9f0c037561396ef8e/Lib/datetime.py#L1807 )

[jerlich]

In the past we have used hardware (the chips used in sound equipment to show levels) to convert a volume into a ttl and send that back into a digital IO to get the exact time that a sound passed some DB level.

The "problem" with asking jackd for the onset is that if the sound ramps or has a silent period at the beginning the sound onset will not be when the animal hears it.

[sneakers-the-rat]

yes we have done that as well. it should be possible to correct for any ramp or whatevs since the sound server makes its own version of the sound object:

autopilot/autopilot/stim/sound/jackclient.py

Line 388 in 04b5968

self._continuous_sound = autopilot.hydrate(self._continuous_dehydrated)

so any offset or ramp or whatever you are using should be something the sound server could correct for. i can't think of a principled way to do that for the general case, since eg. with a ramp obvs the point where the animal hears it is a function of its audiogram and the amplitude and whatnot.

but! @cxrodgers you have also worked with multichannel sound, and if that is easier then i'd be happy to help generalize the sound system so it's easier to combine sounds on different channels. i'm overdue for maintenance here anyway

[jerlich]

totally agree that it isn't principled.... the time when the animal hears the sound depends on so many things...

[cxrodgers]

Thanks @sneakers-the-rat ! I'm still processing a lot of what you wrote. Before you wrote back, I also posted a related question on the jackclient-python github page. spatialaudio/jackclient-python#118 (comment)

Concerning audio, I think the best way for now is just for me to log last_frame_time, similar to what your code does, and I can figure out how to convert this to real time. In the long-term it would be better if I could drive a gpio port directly at the time each period is played, like you said. As a workaround I might also be able to sample the speaker voltage itself, similar to what @jerlich suggested, and capture the tick on which this gpio pin goes nonzero.

To synchronize my raspberry pi with my neural data acquisition system, I'm thinking the ideal way would be if I could sample a hardware clock on the pi. Do you know if there's a way to expose a system clock (or downsampled version of it) on a GPIO pin? But for this to be useful, I would need a way to log events from Autopilot according to their time on the very same clock. If there's the potential for drift between the hardware clock and the values returned by get_current_tick or time.now, then this isn't useful.
https://pinout.xyz/pinout/gpclk

If sampling the hardware clock doesn't make sense, then what I would do is raise a gpio at every trial start, capture these times with the neural data acquisition system, and linearly fit them to the isoformatted trial start times in the autopilot logs. This is basically how I've handled this before.

For synchronizing between the Parent and Child, NTP sounds good, I didn't realize it could be that accurate! I could also connect them on a shared gpio line and store the time of pulses on each one. It looks like pigpiod checks the GPIO lines every 5 us, even though callbacks are only generated every 1 ms, so presumably these tick times are pretty accurate.

[sneakers-the-rat]

I actually hate how it's implemented, but here's an example of GPIO at the same time as a sound ends, and doing it at the same time as a sound starts would be pretty similar:
wiki: https://wiki.auto-pi-lot.com/index.php/Wehrlab
callback called when stim ends: https://github.com/auto-pi-lot/autopilot-plugin-wehrlab/blob/29a7d04c7f0b6dc4234dc0f9e5f00d2edc102eb4/gap/nafc_gap.py#L303
which overrides the base class method, given to the stimulus:

autopilot/autopilot/tasks/nafc.py

Line 215 in 55cd82e

self.stim_manager.set_triggers(self.stim_end)

which is called when the event is set by the sound server, which adjusts for the timing of the jack client:

autopilot/autopilot/stim/sound/jackclient.py

Line 578 in 55cd82e

def _wait_for_end(self):

That makes me cringe with how implicit that is, but it should give an example - we got pretty good, low-jitter alignment with that.

I have wanted to put a clock signal on a GPIO pin - basically sending out some 64-bit timestamp periodically and then running a clock signal in between - but i have not done so. I think system timestamps would probably be good enough, but always a fan of redundancy - the gpio pin signal as well as the timestamps in case one goes wonky. also when in doubt, do it and measure rather than preoptimizing yno?

Another thing I have wanted to do is to use an external RTC module that's wired into multiple pis so their system times can be really synchronized. in that case system timestamps would be the best bet.

+1 for me wishing I would have started this project now after knowing all that i have learned by doing it lmao because a lot of this jank comes from me not knowing how to write the low-level code to do a good clock, and now that i've been learning Rust I am like "microsecond precision should not be a problem." Looking forward to returning to this project after building a p2p engine for it.