Synchronization of external events with audio

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Synchronization of external events with audio


I am relatively new to gstreamer and started working with it over the last few months.  For a project I am working on i have an audio stream (from file or live) that contains data on one channel and audio on the other.  I wrote a custom plugin that decodes the data channel (AMI) and then duplicates the audio channel over the data so that left and right have the same audio and sends the buffer on.  Afterwards i have a volume plugin, equalizer, and then alsasink.  My sources are either alsasrc (line-in) or filesrc.

Now for the data I decode I need to spawn an external event to happen at the time when the audio will occur.  So what i do is try and calculate when the event should occur based on the current time, buffer time, sample number in current buffer, and the base time.  This gets me close, especially when playing a file and not changing the playback rate at at all:

now = gst_clock_get_time(element->clock)/1000000;
eventTime = (gst_element_get_base_time(element)/1000000) + (((GST_BUFFER_TIMESTAMP(buf)/1000000) + (sampleInBuffer/SAMPLE_RATE_MS))/playRate);
msToEvent = eventTime-now;

2 things really cause synchronization of audio and event to be off though:

1.  Live Source (alsasrc) - This causes syncronization to really be off over time.  I am thinking i need to handle latency somehow?  I also had to add sync=false to alsasink in live streams or the audio would just stop playing after about 1 second.  Maybe this param is causing issues?  But the audio will not play at all without it!  Here is my live pipeline:  filesrc location=XXX ! mad ! audiocovnert ! CustomDecoder ! equalizer-3bands ! volume ! alsasink

2.  Changing playback rate (via seek) - It is sometimes desired to play a file back at a slower or faster rate.  When doing this my calculation above seems to correctly modify the event speeds, but the event and audio or not in sync, they seem to be consistently off by some constant amount, like changing the rate ads some constant delay to the audio play time?  Here is my file pipeline:  alsasrc ! audiocovnert ! CustomDecoder ! equalizer-3bands ! volume ! alsasink sync=false

Maybe I am going about this the wrong way all together, but any advice as to how i should accomplish this synchronization would be greatly appreciated!!!  I have read through all the gstreamer clock documentation and i thought i was understanding it all correctly, but i am still not able to achieve correct synchronization : /.