correct way to combine audio and video stream by frame timestamp
This is what I'm trying to do:
audio stream ----->\
-----> appsink -> custom code to package audio
video into mkv
video stream ----->/
The audio, video stream could be coming from live sources or demuxed from
files. I know that appsink only has one sink pad. Right now I wrote two
on_new_sample callbacks and created two independent pipelines, one for audio
and one for video. But then I ran into issue where audio and video frames
can be out of order, so I have to write custom logic to sync the frames. I
feel that this should not be a rare use case. I know there is matroskamux
but it doesnt fit my use case. I need to get the buffers in timestamp order.
I did some research and found that there exists elements like input-select
and multiqueue, but I cant figure out how to use them correctly here or if
it is possible at all. Any help is greatly appreciated.