Sync Video and Audio

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Sync Video and Audio

killerrats
gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
max-files=360 muxer=avimux location=video%06d.avi name=mux source. !
rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0

I put together an application but somehow the video and audio are off by
0.5-1sec. It will hit a hickup sometime and then after that the video and
audio will be off quite a bit. It has got up to 1min off where the audio
will show up before movement.

h264parse error:
the only thing that happened was h264parse error:
"couldn't find associated picture parameter set with id: 2

How can you adjust the video and audio sync? I have put sync=true on the
filesink in the splitmuxsink. any ideas?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
max-files=360 location=video%06d.avi name=mux source. !
rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0

Okay so i used the mp4mux element and the video and audio are spot on when
watching them.

Error that comes up and doesn't let it go anywhere after that. Any ideas?

13:15:20.737978710  5888   02ED4EB0 WARN qtmux
gstqtmux.c:4553:gst_qt_mux_add_buffer:muxer> error: Buffer has no PTS.
ERROR: from element
/GstPipeline:pipeline/GstSplitMuxSinkNpc:splitmuxsink/GstMP4
Mux:muxer: Could not multiplex stream.Additional debug info:
gstqtmux.c(4553): gst_qt_mux_add_buffer ():
/GstPipeline:pipeline/GstSplitMuxSin
kNpc:splitmuxsink/GstMP4Mux:muxer:Buffer has no PTS.



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

Nicolas Dufresne-5
Le mercredi 22 août 2018 à 17:20 -0500, killerrats a écrit :

> gst-launch-1.0 --gst-debug=3 -e rtspsrc location=[IP] name=source !
> rtpjitterbuffer ! rtph264depay ! h264parse config-interval=1 ! tee name=vtee
> ! mux.video splitmuxsink async-handling=true max-size-bytes=1048576
> max-files=360 location=video%06d.avi name=mux source. !
> rtpjitterbuffer ! rtpmp4gdepay ! aacparse ! tee name=atee ! mux.audio_0
>
> Okay so i used the mp4mux element and the video and audio are spot on when
> watching them.
>
> Error that comes up and doesn't let it go anywhere after that. Any ideas?
>
> 13:15:20.737978710  5888   02ED4EB0 WARN qtmux
> gstqtmux.c:4553:gst_qt_mux_add_buffer:muxer> error: Buffer has no PTS.
> ERROR: from element
> /GstPipeline:pipeline/GstSplitMuxSinkNpc:splitmuxsink/GstMP4
> Mux:muxer: Could not multiplex stream.Additional debug info:
> gstqtmux.c(4553): gst_qt_mux_add_buffer ():
> /GstPipeline:pipeline/GstSplitMuxSin
> kNpc:splitmuxsink/GstMP4Mux:muxer:Buffer has no PTS.
We removed the naive PTS generator (copying DTS into PTS) and added
this error, so that GStreamer stop producing invalid ISOMP4. So this
error is telling you that something before the muxer is not setting
PTS. We know some work is needed, someone need to find the time to fix
this properly.

>
>
>
> -----
> ------------------------------
> Gstreamer 1.14.1
> ------------------------------
> Windows
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
will the qtmux do the same thing as well if i use it?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

Mandar Joshi
In reply to this post by Nicolas Dufresne-5
Nicolas,
Is this workaround not valid any more?
https://lists.freedesktop.org/archives/gstreamer-devel/2017-October/065833.html

Regards
Mandar Joshi
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
I at least found the equivalent to the other method you showed. the only thing is that haven't been able to figure out how to apply it. gst_base_parse_set_pts_interpolation(GstBaseParse,gboolean); GstBaseParse
------------------------------
Gstreamer 1.14.1
------------------------------
Windows


Sent from the GStreamer-devel mailing list archive at Nabble.com.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
In reply to this post by Nicolas Dufresne-5
I searched for gstreamer pts dts or something like that and came across
multiqueue that I had used awhile back. In the gstreamer 1.7.1 says
"multiqueue: Use buffer DTS if present, else PTS". could this still be
active and useful?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

Nicolas Dufresne-5
In reply to this post by Mandar Joshi


Le jeu. 23 août 2018 22:12, Mandar Joshi <[hidden email]> a écrit :
Nicolas,
Is this workaround not valid any more?
https://lists.freedesktop.org/archives/gstreamer-devel/2017-October/065833.html

Sure you can hack the parser like in this example, or you could just copy the DTS into the PTS with a pad probe, assuming you don't have B-Frames. We should implement this heuristic in h264parse, just need someone to have the time.



Regards
Mandar Joshi
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
Is this the correct way of doing this?

GstPadProbeType ProbeType = GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM;

gulong id = gst_pad_add_probe(gst_element_get_static_pad(h264parse, "sink"),
ProbeType, Check_buffer_pts_probe, NULL, NULL);

GstPadProbeReturn Check_buffer_pts_probe(GstPad* pad, GstPadProbeInfo* info,
gpointer user_data)
{
        if (GST_EVENT_TYPE(GST_PAD_PROBE_INFO_DATA(info)) == GST_EVENT_EOS)
        {
                return GST_PAD_PROBE_OK;
        }

        GstBuffer * buffer = gst_pad_probe_info_get_buffer(info);
       
        GST_BUFFER_PTS(buffer) = GST_BUFFER_DTS(buffer);

        return GST_PAD_PROBE_OK;
}



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
In reply to this post by Nicolas Dufresne-5
Is this the correct way of doing this?

GstPadProbeType ProbeType = GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM;

gulong id = gst_pad_add_probe(gst_element_get_static_pad(h264parse, "sink"),
ProbeType, Check_buffer_pts_probe, NULL, NULL);

GstPadProbeReturn Check_buffer_pts_probe(GstPad* pad, GstPadProbeInfo* info,
gpointer user_data)
{
        if (GST_EVENT_TYPE(GST_PAD_PROBE_INFO_DATA(info)) == GST_EVENT_EOS)
        {
                return GST_PAD_PROBE_OK;
        }

        GstBuffer * buffer = gst_pad_probe_info_get_buffer(info);

        GST_BUFFER_PTS(buffer) = GST_BUFFER_DTS(buffer);

        return GST_PAD_PROBE_OK;


}



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
sorry for the repeat. i thought it didn't submit the other reply yesterday.

well it looks like this works.
GstElement* parse = gst_element_factory_make("h264parse",NULL);
gst_base_parse_set_pts_interpolation(GST_BASE_PARSE(parse), TRUE);

In the long run it seems to have not underrun for the splitmuxsink like it
did without apply that gst_base_parse_set_pts_interpolation().



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
That h264parse definitely works. the only thing is that the video and audio
are off from each other after several hours. I put sync = true on the mp4mux
in my application before adding to the splitmuxsink. any ideas?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
does anybody know if filesink ignores sync=true?

I read this  sync ignored
<https://lists.freedesktop.org/archives/gstreamer-bugs/2010-June/065302.html>
. is this still true or has this been fixed?




-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

Nicolas Dufresne-5


Le mar. 18 sept. 2018 19 h 12, killerrats <[hidden email]> a écrit :
does anybody know if filesink ignores sync=true?

I read this  sync ignored
<https://lists.freedesktop.org/archives/gstreamer-bugs/2010-June/065302.html>
. is this still true or has this been fixed?

This ancient bug is marked as resolved. Unless there is a regression, it should still be this way.





-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
It takes about a day before I notice the video and audio start to go out of
sync. I set the value qos to true as well on the mp4mux. Do I have to do
anything to check if video and audio are out of sync along the pipeline?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
In reply to this post by Nicolas Dufresne-5
okay I'm trying a new approach by creating two sources to separate between
audio and video in the same bin and try to see if that syncs it better than
having one source give out the both medias.

unless anybody thinks that it doesn't matter which way result is the same?



-----
------------------------------
Gstreamer 1.14.1
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows
Reply | Threaded
Open this post in threaded view
|

Re: Sync Video and Audio

killerrats
gst-launch-1.0 -e rtspsrc location=[IP] latency=66 name=source !
rtpjitterbuffer do-lost=true ! rtph264depay ! identity ! h264parse
config-interval=1 ! avdec_h264 ! openh264enc ! h264parse ! tee name=vtee !
mq.sink_1 multiqueue name=mq ! mux.video splitmuxsink async-handling=true
max-size-bytes=4194304 max-files=360 muxer=mp4mux location=video%06d.avi
name=mux mq.src_2 ! mux.audio_1 source. ! rtpjitterbuffer do-lost=true !
rtpmp4gdepay ! identity ! aacparse ! avdec_aac ! avenc_aac ! aacparse ! tee
name=atee ! mq.sink_2

found out if I do this for each side of the branches for audio and video.

depay -> parse -> decode -> encode -> parse -> tee

if i leave the decode -> encode -> parse out of it the pipeline won't go out
of sync. i apparently don't need to have two sources.



-----
------------------------------
Gstreamer 1.14.2
------------------------------
Windows
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
------------------------------
Gstreamer 1.14.4
------------------------------
Windows