appsrc: minimum latency bigger than maximum latency

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

appsrc: minimum latency bigger than maximum latency

Milian Wolff
Hey all,

I'm running into a strange problem that I don't know the solution to. I'm
using appsrc with the following pipeline:

```
glvideomixer name=m ! tee name=t
    t. ! queue name=display_queue ! qmlglsink name=sink
    t. ! queue name=record_queue ! glcolorconvert ! gldownload ! video/x-
raw,format=ARGB ! videoconvert ! omxh264enc control-rate=1 target-
bitrate=1145000 ! video/x-h264,profile=baseline ! h264parse ! avimux !
filesink location=test.avi
    rpicamsrc preview=false ! video/x-h264,width=640,height=480,framerate=25/1
! h264parse ! omxh264dec ! video/x-raw(memory:GLMemory) ! queue ! glupload !
m.
    appsrc name=overlaysrc blocksize=1228800 stream-type=0 is-live=true
format=time ! rawvideoparse format=rgba width=640 height=480 frame-
size=1228800 framerate=5/1 ! video/x-
raw,format=RGBA,width=640,height=480,framerate=5/1 ! glupload ! m.
```

I'm essentially trying to overlay a video feed with a (changing) custom image
supplied through appsrc. But running the above yields:

```
0:00:01.316311614  1043 0x7300fe60 ERROR               GST_PADS gstpad.c:
3352:gst_pad_query_latency_default:<mixer:src> minimum latency bigger than
maximum latency
0:00:01.316615832  1043 0x7300fe60 WARN              aggregator
gstaggregator.c:1783:gst_aggregator_query_latency_unlocked:<mixer> warning:
Impossible to configure latency: max 0:00:00.000000000 < min
0:00:00.040000000. Add queues or other buffering elements.
```

How do I solve this?

When I leave appsrc out of the picture, then this message disappears - so I'm
apparently doing something wrong in my appsrc?

This "works" (warning doesn't appear, but performance is bad - probably a
different topic):

```
glvideomixer name=m ! tee name=t
    t. ! queue name=display_queue ! qmlglsink name=sink
    t. ! queue name=record_queue ! glcolorconvert ! gldownload ! video/x-
raw,format=ARGB ! videoconvert ! omxh264enc control-rate=1 target-
bitrate=1145000 ! video/x-h264,profile=baseline ! h264parse ! avimux !
filesink location=test.avi
    rpicamsrc preview=false ! video/x-h264,width=640,height=480,framerate=25/1
! h264parse ! omxh264dec ! video/x-raw(memory:GLMemory) ! queue ! glupload !
m.
    videotestsrc ! video/x-raw,format=RGBA,width=640,height=480 ! glupload !
glalpha method=green ! m.
```

What's odd is that the following works too? Here, I keep the appsrc but don't
force h264 decoding through omxh264dec but instead configure the rpicamsrc to
give me raw frames:

```
glvideomixer name=m ! tee name=t    t. ! queue name=display_queue ! qmlglsink
name=sink    t. ! queue name=record_queue ! glcolorconvert ! gldownload !
video/x-raw,format=ARGB ! videoconvert ! omxh264enc control-rate=1 target-
bitrate=1145000 ! video/x-h264,profile=baseline ! h264parse ! avimux !
filesink location=test.avi     rpicamsrc preview=false ! video/x-
raw,format=RGBA,width=640,height=480,framerate=25/1 ! glupload ! m.    appsrc
name=overlaysrc blocksize=1228800 stream-type=0 is-live=true format=time !
rawvideoparse format=rgba width=640 height=480 frame-size=1228800
framerate=5/1 ! video/x-raw,format=RGBA,width=640,height=480,framerate=5/1 !
glupload ! m.
```

Thanks

--
Milian Wolff | [hidden email] | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: appsrc: minimum latency bigger than maximum latency

Tim Müller
On Tue, 2019-11-26 at 13:09 +0100, Milian Wolff wrote:

Hi Milian,

> 0:00:01.316311614  1043 0x7300fe60 ERROR               GST_PADS
> gstpad.c:
> 3352:gst_pad_query_latency_default:<mixer:src> minimum latency bigger
> than
> maximum latency
> 0:00:01.316615832  1043 0x7300fe60 WARN              aggregator
> gstaggregator.c:1783:gst_aggregator_query_latency_unlocked:<mixer>
> warning:
> Impossible to configure latency: max 0:00:00.000000000 < min
> 0:00:00.040000000. Add queues or other buffering elements.
>
> How do I solve this?

This usually means there is not enough buffering in one part of the
pipeline to cater for the worst case latency config that might happen
in another part of the pipeline. To solve this, add a "queue max-size-
bytes=0 max-size-buffers=0" in the other branch.

Problem here is perhaps that you put appsrc into live mode but didn't
configure latency settings on appsrc? In any case, a queue might help
here. It will provide buffering. It won't introduce latency/delay
itself.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: appsrc: minimum latency bigger than maximum latency

Milian Wolff
On Mittwoch, 27. November 2019 10:58:52 CET Tim Müller wrote:

> On Tue, 2019-11-26 at 13:09 +0100, Milian Wolff wrote:
>
> Hi Milian,
>
> > 0:00:01.316311614  1043 0x7300fe60 ERROR               GST_PADS
> > gstpad.c:
> > 3352:gst_pad_query_latency_default:<mixer:src> minimum latency bigger
> > than
> > maximum latency
> > 0:00:01.316615832  1043 0x7300fe60 WARN              aggregator
> > gstaggregator.c:1783:gst_aggregator_query_latency_unlocked:<mixer>
> > warning:
> > Impossible to configure latency: max 0:00:00.000000000 < min
> > 0:00:00.040000000. Add queues or other buffering elements.
> >
> > How do I solve this?
>
> This usually means there is not enough buffering in one part of the
> pipeline to cater for the worst case latency config that might happen
> in another part of the pipeline. To solve this, add a "queue max-size-
> bytes=0 max-size-buffers=0" in the other branch.
Indeed, that removes the warning - thanks!

> Problem here is perhaps that you put appsrc into live mode but didn't
> configure latency settings on appsrc? In any case, a queue might help
> here. It will provide buffering. It won't introduce latency/delay
> itself.

Yes - that's probably the issue. I tried adding explicit latenc values but
that didn't help. Quite frankly, the documentation on that topic is quasi
impossible to understand for me. GstAppSrc just references latency but doesn't
explain when one should use it and what values one should use. Then the
overall topic on Latency just talks about the time between two points which is
a metric I can understand - but how does one apply this concept to a single
point (AppSrc)? If my AppSrc is fed with a 25Hz timer, is the max-latency then
40ms?

Thanks

--
Milian Wolff | [hidden email] | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: appsrc: minimum latency bigger than maximum latency

Tim Müller
On Wed, 2019-11-27 at 12:03 +0100, Milian Wolff wrote:

> Quite frankly, the documentation on that topic is quasi
> impossible to understand for me. GstAppSrc just references latency
> but doesn't explain when one should use it and what values one should
> use. Then the overall topic on Latency just talks about the time
> between two points which is a metric I can understand - but how does
> one apply this concept to a single point (AppSrc)? If my AppSrc is
> fed with a 25Hz timer, is the max-latency then  40ms?

There's some more detailed documentation here:

https://gstreamer.freedesktop.org/documentation/additional/design/latency.html?gi-language=c

You would typically set min-latency (which is "the latency") and not
set max-latency. max-latency indicates how much internal buffering
there is.

For example: an audio source might push out data in 20ms chunks, so
would introduce 20ms latency, but might have an internal ring buffer of
10 chunks, so 200ms. Now if another branch (going into the same
audiomixer, say) advertised 500ms latency that would be a problem,
because it means the audio source branch does not have enough buffering
to accommodate the 500ms of latency/delay/wait-time caused by the other
branch's latency. To fix that you'd have to add a queue of at least
300ms after the audio source or configure the audio source to have a
larger ring buffer.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: appsrc: minimum latency bigger than maximum latency

Milian Wolff
On Mittwoch, 27. November 2019 12:49:17 CET Tim Müller wrote:

> On Wed, 2019-11-27 at 12:03 +0100, Milian Wolff wrote:
> > Quite frankly, the documentation on that topic is quasi
> > impossible to understand for me. GstAppSrc just references latency
> > but doesn't explain when one should use it and what values one should
> > use. Then the overall topic on Latency just talks about the time
> > between two points which is a metric I can understand - but how does
> > one apply this concept to a single point (AppSrc)? If my AppSrc is
> > fed with a 25Hz timer, is the max-latency then  40ms?
>
> There's some more detailed documentation here:
>
> https://gstreamer.freedesktop.org/documentation/additional/design/latency.ht
> ml?gi-language=c
>
> You would typically set min-latency (which is "the latency") and not
> set max-latency. max-latency indicates how much internal buffering
> there is.
>
> For example: an audio source might push out data in 20ms chunks, so
> would introduce 20ms latency, but might have an internal ring buffer of
> 10 chunks, so 200ms. Now if another branch (going into the same
> audiomixer, say) advertised 500ms latency that would be a problem,
> because it means the audio source branch does not have enough buffering
> to accommodate the 500ms of latency/delay/wait-time caused by the other
> branch's latency. To fix that you'd have to add a queue of at least
> 300ms after the audio source or configure the audio source to have a
> larger ring buffer.
Thanks a lot for that explanation. If one then has a source which produces
image data (to be used as video frame overlays) in a somewhat unpredictable
manner but roughly at ~25Hz, without any internal buffering, would the min-
latency then be 40ms? Or should one still introduce a videorate or queue
afterwards to get a well-behaved source?

Thanks

--
Milian Wolff | [hidden email] | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: appsrc: minimum latency bigger than maximum latency

Tim Müller
On Wed, 2019-11-27 at 13:39 +0100, Milian Wolff wrote:

Hi,

> Thanks a lot for that explanation. If one then has a source which
> produces image data (to be used as video frame overlays) in a
> somewhat unpredictable manner but roughly at ~25Hz, without any
> internal buffering, would the min-latency then be 40ms? Or should one
> still introduce a videorate or queue afterwards to get a well-behaved
> source?

That sounds fine, yes, especially if the other branch advertises 40ms
as well anyway. You shouldn't need a videorate if you're feeding into
glvideomixer/compositor, it will only look at the timestamps, and a
videorate will introduce one frame latency (with default settings). You
may still need the queue to make the warning go away.
glvideomixer/compositor should output a "well-behaved" fixed-
framerate/constant-frame-duration stream.

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel