Skipped frames when displaying RTP / H.264 stream

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Skipped frames when displaying RTP / H.264 stream

jean-philippe
Hi,

I'm sending live video (30FPS @ 1920x1080 YUY2) from an iMX8MM to a Linux
host that receives and displays it. Both are connected via a point-to-point
connection (not through a network).

The video displayed stutters and I'm hoping you kind people may be able to
help me understand the cause.

I ran tests with GSstreamer v1.14.5 & v1.17.0 (built from sources)

Master pipeline (using gst-rtsp-server):
v4l2src name=camera !
video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! queue !
vpuenc_h264 gop-size=30 bitrate=0 ! queue ! h264parse ! identity
name=identity ! rtph264pay name=pay0

Client pipeline:
rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
application/x-rtp,media=video ! queue name=q1 ! rtph264depay !
video/x-h264,stream-format=avc ! h264parse ! identity name=identity !
avdec_h264 ! videoconvert ! queue name=q2 ! glimagesink sync=true

On both server and client, I'm using the 'identity' element to look at the
PTS of buffers and check for a jump of more than 8ms between consecutive
buffers's PTS.

On the client, libav prints the following errors every 4.2sec:
ERROR                  libav :0:: Frame num change from 8 to 9
ERROR                  libav :0:: decode_slice_header error

When'h264parse' is in the pipeline, I detect a jump in the PTS of 33ms
(using the 'identity' element), every 4.2ms matching with the errors.

Using a fakesink and removing the decoding of the H.264, I still detect the
33ms jump in the PTS every 4.2ms, but only with 'h264parse' in the pipeline:
rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
application/x-rtp,media=video ! queue name=q1 ! rtph264depay ! h264parse !
identity name=identity ! fakesink sync=true

I detect no jump in the PTS on the server.

Pssoibly relevant the following two warnings on the client side:
FIXME        rtpjitterbuffer
gstrtpjitterbuffer.c:1535:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
Unsupported timestamp reference clock
FIXME        rtpjitterbuffer
gstrtpjitterbuffer.c:1543:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
Unsupported media clock

If I stream the H.264 received from the RTP stream to disk instead of
rendering it, the file plays back perfectly and libav does not print the
errors shown above.
gst-launch-1.0 -e rtspsrc location=rtsp://172.20.1.129:8554/video3 latency=0
! "application/x-rtp,media=video" ! queue ! rtph264depay !
video/x-h264,stream-format=byte-stream ! filesink
location=~/gstreamer_graphs/camera.h264
gst-launch-1.0 filesrc location=~/gstreamer_graphs/camera.h264 ! h264parse !
decodebin ! autovideoconvert ! autovideosink

This is my first streaming app so I lack experience, but I presume I should
be able to receive and display a near perfect video given that I'm not
streaming on the network (point-to-point) and both client and server CPU
usage is low (<20%)?

Why do I detect jumps in the PTS every 4.2ms when using h264parse on the
client?

Given the H.264 stream captured to file replays smoothly with no missing
frames and no error message from libav, what may be causing the live display
to stutter?

Thanks
JP



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

Nicolas Dufresne-5
Le vendredi 31 janvier 2020 à 06:40 -0600, jean-philippe a écrit :

> Hi,
>
> I'm sending live video (30FPS @ 1920x1080 YUY2) from an iMX8MM to a Linux
> host that receives and displays it. Both are connected via a point-to-point
> connection (not through a network).
>
> The video displayed stutters and I'm hoping you kind people may be able to
> help me understand the cause.
>
> I ran tests with GSstreamer v1.14.5 & v1.17.0 (built from sources)
>
> Master pipeline (using gst-rtsp-server):
> v4l2src name=camera !
> video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! queue !
> vpuenc_h264 gop-size=30 bitrate=0 ! queue ! h264parse ! identity
> name=identity ! rtph264pay name=pay0

Careful with this bitrate=0 setting, not using CBR with UDP streaming is looking
for troubles. You might also want to make sure you have no B-Frames.

>
> Client pipeline:
> rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
> application/x-rtp,media=video ! queue name=q1 ! rtph264depay !
> video/x-h264,stream-format=avc ! h264parse ! identity name=identity !
> avdec_h264 ! videoconvert ! queue name=q2 ! glimagesink sync=true
>
> On both server and client, I'm using the 'identity' element to look at the
> PTS of buffers and check for a jump of more than 8ms between consecutive
> buffers's PTS.
>
> On the client, libav prints the following errors every 4.2sec:
> ERROR                  libav :0:: Frame num change from 8 to 9
> ERROR                  libav :0:: decode_slice_header error

This will happen on packet lost. Even on perfect network you can loose packets
with UDP. This can happen for two reasons, a) the server stream is not perfectly
CBR (specially on keyframes). That will cause burst on the line, and may exhaust
the receiver socket buffer size. b) the receiver is spending too much time
processing and it's socket buffer get exhausted.

The main configuration is the udp-buffer-size on rtspsrc element. Though, if
that is not sufficient, you should work on your sender to smooth the
transmission. These days, I configure max-bitrate on udpsink to be twice the
bandwidth configured on my encoder (as my default goto settings).

>
> When'h264parse' is in the pipeline, I detect a jump in the PTS of 33ms
> (using the 'identity' element), every 4.2ms matching with the errors.
>
> Using a fakesink and removing the decoding of the H.264, I still detect the
> 33ms jump in the PTS every 4.2ms, but only with 'h264parse' in the pipeline:
> rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
> application/x-rtp,media=video ! queue name=q1 ! rtph264depay ! h264parse !
> identity name=identity ! fakesink sync=true
>
> I detect no jump in the PTS on the server.
>
> Pssoibly relevant the following two warnings on the client side:
> FIXME        rtpjitterbuffer
> gstrtpjitterbuffer.c:1535:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
> Unsupported timestamp reference clock
> FIXME        rtpjitterbuffer
> gstrtpjitterbuffer.c:1543:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
> Unsupported media clock
>
> If I stream the H.264 received from the RTP stream to disk instead of
> rendering it, the file plays back perfectly and libav does not print the
> errors shown above.
> gst-launch-1.0 -e rtspsrc location=rtsp://172.20.1.129:8554/video3 latency=0
> ! "application/x-rtp,media=video" ! queue ! rtph264depay !
> video/x-h264,stream-format=byte-stream ! filesink
> location=~/gstreamer_graphs/camera.h264
> gst-launch-1.0 filesrc location=~/gstreamer_graphs/camera.h264 ! h264parse !
> decodebin ! autovideoconvert ! autovideosink
>
> This is my first streaming app so I lack experience, but I presume I should
> be able to receive and display a near perfect video given that I'm not
> streaming on the network (point-to-point) and both client and server CPU
> usage is low (<20%)?
>
> Why do I detect jumps in the PTS every 4.2ms when using h264parse on the
> client?
>
> Given the H.264 stream captured to file replays smoothly with no missing
> frames and no error message from libav, what may be causing the live display
> to stutter?
>
> Thanks
> JP
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

jean-philippe
Hi,

Re-sent as yesterday's email was not replying to latest email in thread.

Thanks for your comments about packet loss, I have adjusted my server
pipeline to only produce I-Frames with a low CBR of 5kbps.

This made no difference whatsoever.

For memory, my pipelines:

Server: v4l2src name=camera !
video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! queue !
vpuenc_h264 gop-size=1 bitrate=5000 ! queue ! h264parse ! identity
name=identity ! rtph264pay name=pay0

Client: rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
application/x-rtp,media=video ! queue name=q1 ! rtph264depay !
video/x-h264,stream-format=avc ! h264parse ! identity name=identity !
fakesink sync=false

*I’m still detecting a jump in PTS of 33ms (one frame) every 4.2s which is
128 frames or just over 10MB*; very suspicious!
I’ve increased the size of the queue on the client, but to no avail. Is
there a queue in ‘rtspsrc’? If so, can I / should I change its size? For
some reason, the macro GST_DEBUG_BIN_TO_DOT_FILE() does not dump the content
of the ‘rtspsrc’ bin in the pipeline above. The pipeline in the image
generates starts at the queue. Any idea why?

I only detect those PTS jumps when h264parse is in the pipeline. Any idea
why this may be?.. this could be a clue?

At the same time, libav prints the following errors:
ERROR                  libav :0:: Frame num change from 8 to 9
ERROR                  libav :0:: decode_slice_header error

As mentioned before, I’m not seeing PTS discrepancies on the server side.

BTW, there was a typo in my previous post, the frame drop occurs every 4.2
seconds, not milliseconds.

For memory, I’m using the ‘identity’ element to look at the PTS of each
buffer and detect a jump in PTS between consecutive buffers. I do this on
the server side just before the ‘rtppayh264’ element and on the client just
before ‘avdec_h264’.

Could the FIXME warnings issued by rtpjitterbuffer have something to do with
the problem? I’m looking at the code, but don’t’ yet understand what they
mean. Any pointers someone can give?

FIXME        rtpjitterbuffer
gstrtpjitterbuffer.c:1535:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
Unsupported timestamp reference clock
FIXME        rtpjitterbuffer
gstrtpjitterbuffer.c:1543:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0>
Unsupported media clock

One of the things that puzzle me is why the H.264 stream saved to file seems
to playback perfectly when the live display loses frames?
Is it possible the decoder interpolates the missing frame when playing back
from file? But if so, how does it know the difference between a stream read
from file and one received live?

JP



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

jean-philippe
Update.

I've managed to narrow the problem down to the 'rtph264pay' and/or
'rtph264depay' elements.

With the following pipeline measures a framerate of 29.77Hz which
corresponds to 1 frame being lost every 128 frames:

videotestsrc is-live=true !
video/x-raw,width=640,height=480,format=I420,framerate=30/1 ! queue !
vpuenc_h264 ! queue ! rtph264pay ! queue ! rtph264depay ! fpsdisplaysink
text-overlay=false video-sink=fakesink sync=false

With either of the following pipelines, the framerate is 30Hz:

videotestsrc is-live=true !
video/x-raw,width=640,height=480,format=I420,framerate=30/1 ! queue !
rtpvrawpay ! queue ! rtpvrawdepay ! fpsdisplaysink text-overlay=false
video-sink=fakesink sync=false

videotestsrc is-live=true !
video/x-raw,width=640,height=480,format=I420,framerate=30/1 ! queue !
vpuenc_h264 ! queue ! fpsdisplaysink text-overlay=false video-sink=fakesink
sync=false

I've run the same tests on my Ubuntu 18.04 with x264enc, and got 30Hz for
all configurations.

Could this be showing a problem rtph264pay? I'll be looking at the code in
details tomorrow.

And/or the H.264 stream produced from the i.MX8MM VPU encoder is not 'right'
and rtph264pay as a result drops some data. So far the only strange thing I
have noticed is that the coded slices that correspond to he location of the
problem have a slice_qp_delta that is a very smaller negative number (-1906
and -1842) whereas other slices have values around 0 (1, 2, ...). I'll be
looking into what this means tomorrow.

If any of this rings a bell for anyone, please drop me a line. Could save me
day or weeks of analysis / debugging.

Anyone here doing live streaming from NXP's i.MX8MMini?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

jean-philippe
Me again... I thought I'd post my latest findings in case it rings a bell
with anyone on the mailing list.

To recap, I detect one lost frame every 128 with the following pipeline:
videotestsrc is-live=true !
video/x-raw,width=640,height=480,format=I420,framerate=30/1 ! queue !
vpuenc_h264 ! queue ! rtph264pay ! queue ! rtph264depay ! fpsdisplaysink
text-overlay=false video-sink=fakesink sync=false

This is with the i.MX8MMini. So a 30Hz video stream comes out at 29.77Hz.

I've now found out that the H.264 stream contains one Filler NALU every 128
NALUs.
The NALUs surrounding the Filler look fine, no different from the other VCL
NALUs in the stream.

So I think rtph264pay and/or rtph264depay may be dropping a frame when it
encounters a Filler NALU.

I'm now reading the RTP spec + looking at the code, but any suggestion
appreciated...



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

Nicolas Dufresne-5
On jeu, 2020-02-06 at 07:45 -0600, jean-philippe wrote:

> Me again... I thought I'd post my latest findings in case it rings a bell
> with anyone on the mailing list.
>
> To recap, I detect one lost frame every 128 with the following pipeline:
> videotestsrc is-live=true !
> video/x-raw,width=640,height=480,format=I420,framerate=30/1 ! queue !
> vpuenc_h264 ! queue ! rtph264pay ! queue ! rtph264depay ! fpsdisplaysink
> text-overlay=false video-sink=fakesink sync=false
>
> This is with the i.MX8MMini. So a 30Hz video stream comes out at 29.77Hz.
>
> I've now found out that the H.264 stream contains one Filler NALU every 128
> NALUs.
> The NALUs surrounding the Filler look fine, no different from the other VCL
> NALUs in the stream.
>
> So I think rtph264pay and/or rtph264depay may be dropping a frame when it
> encounters a Filler NALU.

Can you capture a GOP with such filler and share ? Seems like something we can
capture in a unit test if this is the case.

>
> I'm now reading the RTP spec + looking at the code, but any suggestion
> appreciated...
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

jean-philippe
Attached a package with H.264 streams captured when going through the RTP
payloader and when not going through it.

There is a text file describing the content which I copy here for
convenience. Lost-Frames.7z
<http://gstreamer-devel.966125.n4.nabble.com/file/t379103/Lost-Frames.7z>  

Files cbr_rtp.h264 and cbr_nortp.h264 were captured on the i.MX8MMini using
the pipelines shown below.
Files cbr_rtp.txt and cbr_nortp.txt were generated using FFmpeg using the
command lines below.
Each contains 150 frames which is 5 full GOPs, enough to show the problem,
which occurs on frame 128.

Location of problem:
cbr-nortp.txt, Line 2705 is NALU #128 and is followed by Filler Data.
cbr-rtp.txt, Line 4169 is NALU #128 and is followed by Filler Data.

gst-launch-1.0 -v videotestsrc num-buffers=150 is-live=true
animation-mode=frames motion=sweep pattern=ball !
video/x-raw,width=640,height=480,pixelformat=YUY2,framerate=30/1 !
vpuenc_h264 gop-size=30 bitrate=5000 ! queue ! rtph264pay ! queue !
rtph264depay ! queue ! h264parse ! video/x-h264,stream-format=byte-stream !
filesink location=/usr/bin/recordings/cbr_rtp.h264

gst-launch-1.0 -v videotestsrc num-buffers=150 is-live=true
animation-mode=frames motion=sweep pattern=ball !
video/x-raw,width=640,height=480,pixelformat=YUY2,framerate=30/1 !
vpuenc_h264 gop-size=30 bitrate=5000 ! queue ! filesink
location=/usr/bin/recordings/cbr_nortp.h264

.\ffmpeg-4.2.2-win32-static\bin\ffmpeg.exe -i cbr_rtp.h264 -c copy -bsf:v
trace_headers -f null - 2> cbr-rtp.txt
.\ffmpeg-4.2.2-win32-static\bin\ffmpeg.exe -i cbr_nortp.h264 -c copy -bsf:v
trace_headers -f null - 2> cbr-nortp.txt




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

jean-philippe
Update.

It seems the problem is with 'rtph264pay' when the h.264 pushed into it is
'avc'.

Modifying the pipeline like this seems to make the problem go away:
... ! vpuenc_h264 gop-size=30 bitrate=5000 ! queue ! h264parse !
video/x-h264,stream-format=*byte-stream* ! rtph264pay ! queue ! rtph264depay
! ...




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

Nicolas Dufresne-5


Le ven. 7 févr. 2020 09 h 00, jean-philippe <[hidden email]> a écrit :
Update.

It seems the problem is with 'rtph264pay' when the h.264 pushed into it is
'avc'.

Modifying the pipeline like this seems to make the problem go away:
... ! vpuenc_h264 gop-size=30 bitrate=5000 ! queue ! h264parse !
video/x-h264,stream-format=*byte-stream* ! rtph264pay ! queue ! rtph264depay
! ...

Good finding, that will help. I'm surprise NXP decided to produce AVCc by default. It's quite assymetric, since the Hantro decoder they use requires annex B / byte-stream. I'll try to make sure we don't reproduce this with the upstream drivers  and GST elements ;-)

Thanks for the data, I'm saving this locally and will analyse this to find out what need fixing. But I may not be able to look at this this week.





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Skipped frames when displaying RTP / H.264 stream

Stephenwei
In reply to this post by jean-philippe
----------------------------------------------------
Master pipeline (using gst-rtsp-server):
v4l2src name=camera !
video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! queue !
vpuenc_h264 gop-size=30 bitrate=0 ! queue ! h264parse ! identity
name=identity ! rtph264pay name=pay0

Client pipeline:
rtspsrc name=videosrc location=rtsp://172.20.1.129:8554/video3 !
application/x-rtp,media=video ! queue name=q1 ! rtph264depay !
video/x-h264,stream-format=avc ! h264parse ! identity name=identity !
avdec_h264 ! videoconvert ! queue name=q2 ! glimagesink sync=true
------------------------------------------------------------------------------
 I'm not sure, but why don't you try vpu decode?



-----
GStreamer is a convenient multimedia platform, I like it.
Develop the NVR system on ARM/x86(c/python)
Use python to generate NVR is crazy, of course works fine.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
GStreamer is a convenient multimedia platform, I like it.
Develop the NVR system on ARM/x86(c/python)
Use python to generate NVR is crazy, of course works fine.