shmsink changes from 1.0.10 to 1.1.4

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
Hi.

Apparently there are some changes in plugin shmsink between 1.0.10 and 1.1.4 that breaks how external programs should use the shmsink. I tried to look in the release announcements for 1.1.1-1.1.4, but only smaller changes mostly wrt to how gstreamer handles shmsinks without a connected process are listed. Nevertheless, there appear to me to be changes that goes beyond that.

I have a program, namely Snowmix that uses the shm of GStreamer heavily and it works quite well up till 1.0.10, but for 1.1.4 the shmsink doesn't seem to understand when I signal that I have read the buffer it has send. I do receive frames send via shmsink until the allocated shm area has been used, and then no more, even though I signal over the control connection, that I have released the buffers sent.

Has the binary layout format for the control connection messages changed? If yes, in what way? Is it documented somewhere?

If the binary format has changed, we really need for it to be able to tell its version over the control connection so applications can detect the gstreamer version and act accordingly.

The pipeline used for testing is this simple pipeline

RAWVIDEO='video/x-raw, format=(string)BGRA, bpp=(int)32, depth=(int)32, endianness=(int)4321, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false'

gst-launch-1.0 -v videotestsrc is-live=true !\
    $RAWVIDEO !\
    queue !\
    identity !\
    shmsink socket-path=/tmp/feed1-control-pipe shm-size=10000000 wait-for-connection=1 sync=true

Kind regards
Peter MM

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Gst.inti(None) bugs for gstreamer-1.0 ( using PyGobject introspection)

nar6du14
I use PyGobject on windows 7 for gstreamer-1.0 and it crashes.

Gst.init(None)  crashes my application.


De : Peter Maersk-Moller <[hidden email]>
À : Discussion of the development of and with GStreamer <[hidden email]>
Envoyé le : Mercredi 18 septembre 2013 18h07
Objet : shmsink changes from 1.0.10 to 1.1.4

Hi.

Apparently there are some changes in plugin shmsink between 1.0.10 and 1.1.4 that breaks how external programs should use the shmsink. I tried to look in the release announcements for 1.1.1-1.1.4, but only smaller changes mostly wrt to how gstreamer handles shmsinks without a connected process are listed. Nevertheless, there appear to me to be changes that goes beyond that.

I have a program, namely Snowmix that uses the shm of GStreamer heavily and it works quite well up till 1.0.10, but for 1.1.4 the shmsink doesn't seem to understand when I signal that I have read the buffer it has send. I do receive frames send via shmsink until the allocated shm area has been used, and then no more, even though I signal over the control connection, that I have released the buffers sent.

Has the binary layout format for the control connection messages changed? If yes, in what way? Is it documented somewhere?

If the binary format has changed, we really need for it to be able to tell its version over the control connection so applications can detect the gstreamer version and act accordingly.

The pipeline used for testing is this simple pipeline

RAWVIDEO='video/x-raw, format=(string)BGRA, bpp=(int)32, depth=(int)32, endianness=(int)4321, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false'

gst-launch-1.0 -v videotestsrc is-live=true !\
    $RAWVIDEO !\
    queue !\
    identity !\
    shmsink socket-path=/tmp/feed1-control-pipe shm-size=10000000 wait-for-connection=1 sync=true

Kind regards
Peter MM

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel



_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
In reply to this post by Peter Maersk-Moller-2
Problem solved.

It turns out that GStreamer 1.1.4 is a bit more pedantic about timing compared to 1.0.x and 0.10. The following pipeline will use the shared memory area and then freeze in 1.1.4 while working in 1.0.x and 0.10

gst-launch-1.0 -v videotestsrc is-live=true ! $RAWVIDEO ! queue ! identity ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=20000000 wait-for-connection=0 sync=true

However if you add 'do-timestamp=true' to videotestsrc, it will flow again, also in 1.1.4.

gst-launch-1.0 -v videotestsrc is-live=true do-timestamp=true ! $RAWVIDEO ! queue ! identity ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=20000000 wait-for-connection=0 sync=true

I guess that is an overall change in 1.1.x not only affecting shmsink? Probably a right decision, but it will for sure be difficult for many current implementations.

Best regards
Peter MM


On Wed, Sep 18, 2013 at 7:07 PM, Peter Maersk-Moller <[hidden email]> wrote:
Hi.

Apparently there are some changes in plugin shmsink between 1.0.10 and 1.1.4 that breaks how external programs should use the shmsink. I tried to look in the release announcements for 1.1.1-1.1.4, but only smaller changes mostly wrt to how gstreamer handles shmsinks without a connected process are listed. Nevertheless, there appear to me to be changes that goes beyond that.

I have a program, namely Snowmix that uses the shm of GStreamer heavily and it works quite well up till 1.0.10, but for 1.1.4 the shmsink doesn't seem to understand when I signal that I have read the buffer it has send. I do receive frames send via shmsink until the allocated shm area has been used, and then no more, even though I signal over the control connection, that I have released the buffers sent.

Has the binary layout format for the control connection messages changed? If yes, in what way? Is it documented somewhere?

If the binary format has changed, we really need for it to be able to tell its version over the control connection so applications can detect the gstreamer version and act accordingly.

The pipeline used for testing is this simple pipeline

RAWVIDEO='video/x-raw, format=(string)BGRA, bpp=(int)32, depth=(int)32, endianness=(int)4321, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false'

gst-launch-1.0 -v videotestsrc is-live=true !\
    $RAWVIDEO !\
    queue !\
    identity !\
    shmsink socket-path=/tmp/feed1-control-pipe shm-size=10000000 wait-for-connection=1 sync=true

Kind regards
Peter MM


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Olivier Crête-3
Hi,

On Wed, 2013-09-18 at 22:03 +0200, Peter Maersk-Moller wrote:

> It turns out that GStreamer 1.1.4 is a bit more pedantic about timing
> compared to 1.0.x and 0.10. The following pipeline will use the shared
> memory area and then freeze in 1.1.4 while working in 1.0.x and 0.10
>
> gst-launch-1.0 -v videotestsrc is-live=true ! $RAWVIDEO ! queue !
> identity ! shmsink socket-path=/tmp/feed1-control-pipe
> shm-size=20000000 wait-for-connection=0 sync=true
>
>
> However if you add 'do-timestamp=true' to videotestsrc, it will flow
> again, also in 1.1.4.
>
> gst-launch-1.0 -v videotestsrc is-live=true do-timestamp=true !
> $RAWVIDEO ! queue ! identity ! shmsink
> socket-path=/tmp/feed1-control-pipe shm-size=20000000
> wait-for-connection=0 sync=true

I'm curious to know how your pipeline is different from these ones which
work fine for me using git master:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=(string)I420,
width=(int)320, height=(int)240, framerate=(fraction)30/1' ! queue !
identity ! shmsink wait-for-connection=1 socket-path=/tmp/tmpsock
shm-size=20000000 sync=true

gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! 'video/x-raw,
format=(string)I420, width=(int)320, height=(int)240,
framerate=(fraction)30/1' ! xvimagesink


--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
Hi Olivier

Tried your pipelines and it works (ie. not locking). However experimented a bit further and ended up with these two pipelines

gst-launch-1.0 -v videotestsrc ! $RAWVIDEO ! queue ! identity ! shmsink wait-for-connection=1 socket-path=/tmp/tmpsock shm-size=15000000 sync=true

gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! $RAWVIDEO ! queue ! fakesink

Now if I first define RAWVIDEO as shown below:

RAWVIDEO='video/x-raw, format=(string)I420, width=(int)1280, height=(int)720, framerate=(fraction)30/1'

Then around 9-10 frames give or take are acknowledged in shmpipe.c before freezing. 15000000 is 10.8 frames in that size in I420.

If i increase shm-size to shm-size=20000000, then it will run without freezing. And I can't see why.


On Wed, Sep 18, 2013 at 11:29 PM, Olivier Crête <[hidden email]> wrote:
Hi,

On Wed, 2013-09-18 at 22:03 +0200, Peter Maersk-Moller wrote:

> It turns out that GStreamer 1.1.4 is a bit more pedantic about timing
> compared to 1.0.x and 0.10. The following pipeline will use the shared
> memory area and then freeze in 1.1.4 while working in 1.0.x and 0.10
>
> gst-launch-1.0 -v videotestsrc is-live=true ! $RAWVIDEO ! queue !
> identity ! shmsink socket-path=/tmp/feed1-control-pipe
> shm-size=20000000 wait-for-connection=0 sync=true
>
>
> However if you add 'do-timestamp=true' to videotestsrc, it will flow
> again, also in 1.1.4.
>
> gst-launch-1.0 -v videotestsrc is-live=true do-timestamp=true !
> $RAWVIDEO ! queue ! identity ! shmsink
> socket-path=/tmp/feed1-control-pipe shm-size=20000000
> wait-for-connection=0 sync=true

I'm curious to know how your pipeline is different from these ones which
work fine for me using git master:

gst-launch-1.0 -v videotestsrc ! 'video/x-raw, format=(string)I420,
width=(int)320, height=(int)240, framerate=(fraction)30/1' ! queue !
identity ! shmsink wait-for-connection=1 socket-path=/tmp/tmpsock
shm-size=20000000 sync=true

gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! 'video/x-raw,
format=(string)I420, width=(int)320, height=(int)240,
framerate=(fraction)30/1' ! xvimagesink


--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Josh Doe
On Wed, Sep 18, 2013 at 7:26 PM, Peter Maersk-Moller <[hidden email]> wrote:

> Hi Olivier
>
> Tried your pipelines and it works (ie. not locking). However experimented a
> bit further and ended up with these two pipelines
>
> gst-launch-1.0 -v videotestsrc ! $RAWVIDEO ! queue ! identity ! shmsink
> wait-for-connection=1 socket-path=/tmp/tmpsock shm-size=15000000 sync=true
>
> gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! $RAWVIDEO ! queue !
> fakesink
>
> Now if I first define RAWVIDEO as shown below:
>
> RAWVIDEO='video/x-raw, format=(string)I420, width=(int)1280,
> height=(int)720, framerate=(fraction)30/1'
>
> Then around 9-10 frames give or take are acknowledged in shmpipe.c before
> freezing. 15000000 is 10.8 frames in that size in I420.
>
> If i increase shm-size to shm-size=20000000, then it will run without
> freezing. And I can't see why.

I've experienced this as well, having to bump the shm-size up
significantly to avoid any blocking. I'd think a warning would be
better than blocking if the receiving end can't keep up with the
sending side, but I'd have to look at the implementation to see if/how
this could be done.

-Josh
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
Hi Josh

Nothing wrong in the receiving end not keeping up with the sending end. In that case the sender should eventually stop handing out shmbuf upstream, which I believe it does. Adding a leaky queue upstream will then lead to dropping buffers, which is predictable. However, what seems to happen is that shmsink stops sending shmbuf into shm area or stop signal them as sent over its control pipe, if shm-size is below some currently not understood threshold, even though the receiving end acknowledge the buffers sent and the buffers becomes available again.

(To the Gst developers)
BTW, are shmsrc/sink now available for the OS X platform? If not, do you have any plans for it?

BTW2. A neat feature of shmsink, especially when used by external programs, would be if it had 2 extra commands added. These should be:

  1) Stop handing out more buffers upstream.
  2) Start handing out more buffers upstream.
  3) Drop all buffers handed out upstream until now, but not yet sent.

A video mixer application like Snowmix could then much better do resource control. Imagine you have a video mixer with 10-20 gstreamer pipeline inputs, it can really save a lot of CPU and memmove bandwidth, if the mixer can temporarily halt pipeline not currently being mixed. Now that is already possible, but not  before the shmsink has used up all available shm area. Just a thought. Limiting the amount of unnecessary memmove becomes important when doing many pipelines of 1920x1080 @60fps.

best regards
Peter MM



Best regards

On Thu, Sep 19, 2013 at 3:05 PM, Josh Doe <[hidden email]> wrote:
On Wed, Sep 18, 2013 at 7:26 PM, Peter Maersk-Moller <[hidden email]> wrote:
> Hi Olivier
>
> Tried your pipelines and it works (ie. not locking). However experimented a
> bit further and ended up with these two pipelines
>
> gst-launch-1.0 -v videotestsrc ! $RAWVIDEO ! queue ! identity ! shmsink
> wait-for-connection=1 socket-path=/tmp/tmpsock shm-size=15000000 sync=true
>
> gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! $RAWVIDEO ! queue !
> fakesink
>
> Now if I first define RAWVIDEO as shown below:
>
> RAWVIDEO='video/x-raw, format=(string)I420, width=(int)1280,
> height=(int)720, framerate=(fraction)30/1'
>
> Then around 9-10 frames give or take are acknowledged in shmpipe.c before
> freezing. 15000000 is 10.8 frames in that size in I420.
>
> If i increase shm-size to shm-size=20000000, then it will run without
> freezing. And I can't see why.

I've experienced this as well, having to bump the shm-size up
significantly to avoid any blocking. I'd think a warning would be
better than blocking if the receiving end can't keep up with the
sending side, but I'd have to look at the implementation to see if/how
this could be done.

-Josh
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Olivier Crête-3
On Thu, 2013-09-19 at 15:42 +0200, Peter Maersk-Moller wrote:

> (To the Gst developers)
>
> BTW, are shmsrc/sink now available for the OS X platform? If not, do
> you have any plans for it?

They should be? If it doesn't work, please file a bug (and patches
welcome, I don't have a Mac).

> BTW2. A neat feature of shmsink, especially when used by external
> programs, would be if it had 2 extra commands added. These should be:
>
>
>   1) Stop handing out more buffers upstream.
>   2) Start handing out more buffers upstream.
>   3) Drop all buffers handed out upstream until now, but not yet sent.
>
>
> A video mixer application like Snowmix could then much better do
> resource control. Imagine you have a video mixer with 10-20 gstreamer
> pipeline inputs, it can really save a lot of CPU and memmove
> bandwidth, if the mixer can temporarily halt pipeline not currently
> being mixed. Now that is already possible, but not  before the shmsink
> has used up all available shm area. Just a thought. Limiting the
> amount of unnecessary memmove becomes important when doing many
> pipelines of 1920x1080 @60fps.

Can't your just halt it directly ? Why do you wait for all of the buffer
space to be used up ?

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Olivier Crête-3
In reply to this post by Josh Doe
On Thu, 2013-09-19 at 09:05 -0400, Josh Doe wrote:

> On Wed, Sep 18, 2013 at 7:26 PM, Peter Maersk-Moller <[hidden email]> wrote:
> > Hi Olivier
> >
> > Tried your pipelines and it works (ie. not locking). However experimented a
> > bit further and ended up with these two pipelines
> >
> > gst-launch-1.0 -v videotestsrc ! $RAWVIDEO ! queue ! identity ! shmsink
> > wait-for-connection=1 socket-path=/tmp/tmpsock shm-size=15000000 sync=true
> >
> > gst-launch-1.0 shmsrc socket-path=/tmp/tmpsock ! $RAWVIDEO ! queue !
> > fakesink
> >
> > Now if I first define RAWVIDEO as shown below:
> >
> > RAWVIDEO='video/x-raw, format=(string)I420, width=(int)1280,
> > height=(int)720, framerate=(fraction)30/1'
> >
> > Then around 9-10 frames give or take are acknowledged in shmpipe.c before
> > freezing. 15000000 is 10.8 frames in that size in I420.
> >
> > If i increase shm-size to shm-size=20000000, then it will run without
> > freezing. And I can't see why.
>
> I've experienced this as well, having to bump the shm-size up
> significantly to avoid any blocking. I'd think a warning would be
> better than blocking if the receiving end can't keep up with the
> sending side, but I'd have to look at the implementation to see if/how
> this could be done.

The explanation is that I re-enabled zero-copy in shmsink, so that means
that now the buffers are being allocated and sent up the pipeline, so
they end up being used longer so you need more.

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
In reply to this post by Olivier Crête-3
Hi Oliver.

See answer inline after your question.

On Thu, Sep 19, 2013 at 8:05 PM, Olivier Crête <[hidden email]> wrote:
> BTW2. A neat feature of shmsink, especially when used by external
> programs, would be if it had 2 extra commands added. These should be:
>
>
>   1) Stop handing out more buffers upstream.
>   2) Start handing out more buffers upstream.
>   3) Drop all buffers handed out upstream until now, but not yet sent.
>
>
> A video mixer application like Snowmix could then much better do
> resource control. Imagine you have a video mixer with 10-20 gstreamer
> pipeline inputs, it can really save a lot of CPU and memmove
> bandwidth, if the mixer can temporarily halt pipeline not currently
> being mixed. Now that is already possible, but not  before the shmsink
> has used up all available shm area. Just a thought. Limiting the
> amount of unnecessary memmove becomes important when doing many
> pipelines of 1920x1080 @60fps.

Can't your just halt it directly ? Why do you wait for all of the buffer
space to be used up ?

No I can not halt it directly.

One of the standard use scenario for shmsink is when you have a GStreamer pipeline in one process (process 1) sending buffers to another process (process 2) running whatever that process runs. There is no easy way for process 2 to tell process 1 to halt unless it keeps all the buffers sent by process 1. Please note that halt is bit unclear. In my scenario halt means stop sending buffers (and hopefully stop using so many ressources, CPU, disk, memmove bandwidth, network etc.). It does NOT mean terminate.

Now if process 2 keeps all the buffers it receive, process 1 will eventually reach a state, where it temporarily halts or alternatively leaks buffers in a queue. However there is a lag and as we have seen, the lag has to  be of a certain size due to the freeze thing we discussed in another thread.

Hmmmmmm that give rise to a problem. Assume you have the following:

  somesrc ! queue leaky=2 ! shsmsink  ----------- process 2

Then how can the queue leak if process 2 keeps all the buffers ? In that case, somsrc will freeze too. How can that be fixed?

 best regards








_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Olivier Crête-3
See below

On Thu, 2013-09-19 at 22:23 +0200, Peter Maersk-Moller wrote:

> On Thu, Sep 19, 2013 at 8:05 PM, Olivier Crête
> <[hidden email]> wrote:
>         > BTW2. A neat feature of shmsink, especially when used by
>         external
>         > programs, would be if it had 2 extra commands added. These
>         should be:
>         >
>         >
>         >   1) Stop handing out more buffers upstream.
>         >   2) Start handing out more buffers upstream.
>         >   3) Drop all buffers handed out upstream until now, but not
>         yet sent.
>         >
>         >
>         > A video mixer application like Snowmix could then much
>         better do
>         > resource control. Imagine you have a video mixer with 10-20
>         gstreamer
>         > pipeline inputs, it can really save a lot of CPU and memmove
>         > bandwidth, if the mixer can temporarily halt pipeline not
>         currently
>         > being mixed. Now that is already possible, but not  before
>         the shmsink
>         > has used up all available shm area. Just a thought. Limiting
>         the
>         > amount of unnecessary memmove becomes important when doing
>         many
>         > pipelines of 1920x1080 @60fps.
>        
>        
>         Can't your just halt it directly ? Why do you wait for all of
>         the buffer
>         space to be used up ?
>
>
> No I can not halt it directly.
>
> One of the standard use scenario for shmsink is when you have a
> GStreamer pipeline in one process (process 1) sending buffers to
> another process (process 2) running whatever that process runs. There
> is no easy way for process 2 to tell process 1 to halt unless it keeps
> all the buffers sent by process 1. Please note that halt is bit
> unclear. In my scenario halt means stop sending buffers (and hopefully
> stop using so many ressources, CPU, disk, memmove bandwidth, network
> etc.). It does NOT mean terminate.
>
>
> Now if process 2 keeps all the buffers it receive, process 1 will
> eventually reach a state, where it temporarily halts or alternatively
> leaks buffers in a queue. However there is a lag and as we have seen,
> the lag has to  be of a certain size due to the freeze thing we
> discussed in another thread.
>
> Hmmmmmm that give rise to a problem. Assume you have the following:
>
>
>   somesrc ! queue leaky=2 ! shsmsink  ----------- process 2
>
>
> Then how can the queue leak if process 2 keeps all the buffers ? In
> that case, somsrc will freeze too. How can that be fixed?

One thing we could do is add properties to limit the size of "queue"
that's inside shmsink by time or bytes or buffers (like the queue
element). I'd certainly be open to something like that.

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Olivier Crête-3
On Thu, 2013-09-19 at 18:50 -0400, Olivier Crête wrote:
> One thing we could do is add properties to limit the size of "queue"
> that's inside shmsink by time or bytes or buffers (like the queue
> element). I'd certainly be open to something like that.

Actually, there is already a buffer-time property to limit the size of
the "queue", but it is only triggered when the buffer is free on the
receiving side, probably not exactly what you want, but that could be
fixed.

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
Hi Olivier

I'm afraid that's not exactly what I'm looking for. Also my pipeline example didn't detailed the problem clear enough, so please allow me to try again. Assuming you have the following pipeline:

  gst-launch udpsrc ! 'video/mpgets' !\
    queue name=qts leaky=2 ! decodebin !\
    'video/x-raw' ! queue name=qvid leaky=2 ! shmsink name=shmvid \
    'audio/x-raw' ! queue name=qaud leaky=2 ! shmsink name=shmaud

Also assume we have two other processes that process the buffers passed by the two shmsinks and acknowledge the buffers when read.

The leaky queue 'qts' is included to point out, that if decodebin stops or don't process buffers received from upstream fast enough, TS packets are lost. In fact that would happen anyway as an UDP connection in principle is a queue anyway and that queue is leaky in case data is not read fast enough. This is a hard fact and not something we can control. In fact I would prefer I could guarantee that decodebin always receives all packets.

The queues 'qvid' and 'qaud' are included to illustrate that I would like raw video or raw audio to be dropped frame by frame if the processes that shmsink delivers buffers are not fast enough or decide it is desired that raw frames are dropped. However I believe decodebin may also include queues on its src pads (output), but they are harder to control individually.

Now I think the pipeline works this way, that the shmsinks passes shmbufs upstream to decodebin for decodebin to fill with raw video and raw audio frames. The shmsinks receives these buffers and signal to the two external processes that shmbuffers are ready to be read.

Lets now assume that the process reading the raw video frames in the pipeline would like to signal to the gstreamer pipeline, that the pipeline temporarily can halt sending decoded frames. Right now the only way to do that is by not sending Acks through the control pipe for buffers it has received. However, the shmsink will continue handing out shmbuffers upstream and signal received buffers ready to be read until it has no more buffers. Now when the shmsink 'shmvid' has no more buffers to hand out, I believe decodebin will stop decode video data. I fear this will also stop the decoder from sending audio data upstream which can be fatal. I also fear that if shmsink 'shmvid' stops, decodebin will stop sending buffers upstream leading to an unpredictable loss of packets.

My questions are now these:

1) Am I correct that decodebin will stop decode video if it stops receiving ready shmbuffers from downstream?
2) If shmsink 'shmvid' stops sending shmbuffers upstream, will it also stop audio decoding ?
3) If decodebin stops receiving shmbuf from downstream shmsink 'shmvid', will it eventually stop decoding TS packets?
4) If decodebin stops receiving shmbuf from downstream, shmsink 'shmvid', what is the state of decoding when it starts receive buffers again?

Oberservations and opinions:

1) It can be fatal if stopping video also stops audio, so that is to be avoided.
2) It can be fatal for the quality of the decoded streams, if TS packets are dropped before decoded, so that is to be avoided.
3) It can be fatal for the quality of the decoded video stream, if encoded video data is dropped before decoded because the the encoder otherwise will be in an unpredictable state and when started again (receiving buffers) we may get large pixelation, green or gray screen and weird moving blocks which is unacceptable for production of professional TV, though personal player may have this behaviour.

So here is what I would like. I would like that the external process could send a message through the control pipe and instruct the shmsink that it temporarily is instructed not to signal more buffers ready for the process. That it self is not a very big difference, but the shmsink could use the information to signal upstream, that while it may be handing out shmbuffers upstream, it will temporarily not accept any more buffers downstream and a leaky queue upstream to shmsink can then leak(drop) buffers. Even better would it be if a decoder upstream can somehow use this information to understand, that it should not send decoded frames downstream, but it should continue to decode to keep an updated state. The latter may be unobtainable with the GStreamer (and decoder) design, but it would be nice to have to optimize resource usage. Even better would it be, if the decoder can stop decode while maintain a state and inspecting the encoeded stream so it when requested as soon as possible can start deliver a new decoded full frame.

Sorry for the lengthy text, but it's a rather complicated subject.

Best regards
Peter Maersk-Moller



On Fri, Sep 20, 2013 at 12:58 AM, Olivier Crête <[hidden email]> wrote:
On Thu, 2013-09-19 at 18:50 -0400, Olivier Crête wrote:
> One thing we could do is add properties to limit the size of "queue"
> that's inside shmsink by time or bytes or buffers (like the queue
> element). I'd certainly be open to something like that.

Actually, there is already a buffer-time property to limit the size of
the "queue", but it is only triggered when the buffer is free on the
receiving side, probably not exactly what you want, but that could be
fixed.

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: shmsink changes from 1.0.10 to 1.1.4

Peter Maersk-Moller-2
Experimenting a bit further with gstreamer-1.1.90 and shm, I have found through trial and error the following.

1) If the the shmsink option 'wait-for-connection' is true (1), the shmsink option shm-size must be a size equal to or greater than 6 frames. If the shm-size is less than that, the pipeline with the shmsink may and quite often will freeze whether or not another process connects in the other end. If the shmsink does not have a queue element upstream, the shm-size must be equal to or greater than 3 frames.

2) If the the shmsink option 'wait-for-connection' is false (0) and the shmsink has a queue element upstream, the shmsink option shm-size must be a size equal to or greater than 3 frames. If the shm-size is less than that, the pipeline with the shmsink may and quite often will freeze whether or not another process connects in the other end. If the shmsink does not have a queue element upstream, the shm-size must be equal to or greater than 2 frames.

Is this what we want ? Especially the 6 frames can be a bit excessive.

Here is how it was tested.

Sender side with queue element:
gst-launch-1.0 -v videotestsrc is-live=true do-timestamp=false ! $RAWVIDEO ! queue ! identity silent=false ! shmsink socket-path=/tmp/control-pipe shm-size=22118400 wait-for-connection=1 sync=true

Sender side without queue element
gst-launch-1.0 -v videotestsrc is-live=true do-timestamp=false ! $RAWVIDEO ! identity silent=false ! shmsink socket-path=/tmp/control-pipe shm-size=22118400 wait-for-connection=1 sync=true

RAWVIDEO was defined as
$ RAWVIDEO='video/x-raw, format=(string)BGRA, width=(int)1280, height=(int)720, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false'

Receiver side was defined as
while true ; do gst-launch-1.0 -v shmsrc do-timestamp=true is-live=true socket-path=/tmp/control-pipe ! identity silent=false ! fakesink sync=true ; sleep 2 ;done

With the defined frame type and size, the sizes tested was as follows
1 frame = 3686400
2 frames = 7372800
3 frames = 11059200
4 frames = 14745600
6 frames = 22118400
7 frames = 25804800

In the above example, the videotesrc element was used as a video src. In a more complex example shown below, the number of frames/buffers needed to avoid freezing, EVEN with wait-for-connection=0, is 6 or approx 38MB. Here is the example:

/usr/local/bin/gst-launch-1.0 -v filesrc location=../test/LES_TDS_launch.mp4 do-timestamp=true ! decodebin name=decoder ! videoconvert ! videorate ! videoscale ! videoconvert ! 'video/x-raw, format=(string)BGRA, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, width=(int)1280, height=(int)720, framerate=(fraction)25/1' ! identity silent=false ! queue ! shmsink socket-path=/tmp/control-pipe shm-size=36864000 wait-for-connection=0 sync=true decoder. ! queue ! audioconvert ! audioresample ! 'audio/x-raw, rate=(int)44100, channels=(int)1' ! fdsink fd=3 sync=true

If we remove the queue upstream to shmsink, the minimum required shm-size is 2 frames/buffers. If the consuming side keep one or more buffers, you for some cases have to add that number plus one to the buffer pool (shm-size).

In this more advance example, using wait-for-connection=1 does not seem to increase the need for buffers.

Olivier, is this what you expect with the changes you made to shmsink?

Kind regards
Peter MM



On Fri, Sep 20, 2013 at 11:53 AM, Peter Maersk-Moller <[hidden email]> wrote:
Hi Olivier

I'm afraid that's not exactly what I'm looking for. Also my pipeline example didn't detailed the problem clear enough, so please allow me to try again. Assuming you have the following pipeline:

  gst-launch udpsrc ! 'video/mpgets' !\
    queue name=qts leaky=2 ! decodebin !\
    'video/x-raw' ! queue name=qvid leaky=2 ! shmsink name=shmvid \
    'audio/x-raw' ! queue name=qaud leaky=2 ! shmsink name=shmaud

Also assume we have two other processes that process the buffers passed by the two shmsinks and acknowledge the buffers when read.

The leaky queue 'qts' is included to point out, that if decodebin stops or don't process buffers received from upstream fast enough, TS packets are lost. In fact that would happen anyway as an UDP connection in principle is a queue anyway and that queue is leaky in case data is not read fast enough. This is a hard fact and not something we can control. In fact I would prefer I could guarantee that decodebin always receives all packets.

The queues 'qvid' and 'qaud' are included to illustrate that I would like raw video or raw audio to be dropped frame by frame if the processes that shmsink delivers buffers are not fast enough or decide it is desired that raw frames are dropped. However I believe decodebin may also include queues on its src pads (output), but they are harder to control individually.

Now I think the pipeline works this way, that the shmsinks passes shmbufs upstream to decodebin for decodebin to fill with raw video and raw audio frames. The shmsinks receives these buffers and signal to the two external processes that shmbuffers are ready to be read.

Lets now assume that the process reading the raw video frames in the pipeline would like to signal to the gstreamer pipeline, that the pipeline temporarily can halt sending decoded frames. Right now the only way to do that is by not sending Acks through the control pipe for buffers it has received. However, the shmsink will continue handing out shmbuffers upstream and signal received buffers ready to be read until it has no more buffers. Now when the shmsink 'shmvid' has no more buffers to hand out, I believe decodebin will stop decode video data. I fear this will also stop the decoder from sending audio data upstream which can be fatal. I also fear that if shmsink 'shmvid' stops, decodebin will stop sending buffers upstream leading to an unpredictable loss of packets.

My questions are now these:

1) Am I correct that decodebin will stop decode video if it stops receiving ready shmbuffers from downstream?
2) If shmsink 'shmvid' stops sending shmbuffers upstream, will it also stop audio decoding ?
3) If decodebin stops receiving shmbuf from downstream shmsink 'shmvid', will it eventually stop decoding TS packets?
4) If decodebin stops receiving shmbuf from downstream, shmsink 'shmvid', what is the state of decoding when it starts receive buffers again?

Oberservations and opinions:

1) It can be fatal if stopping video also stops audio, so that is to be avoided.
2) It can be fatal for the quality of the decoded streams, if TS packets are dropped before decoded, so that is to be avoided.
3) It can be fatal for the quality of the decoded video stream, if encoded video data is dropped before decoded because the the encoder otherwise will be in an unpredictable state and when started again (receiving buffers) we may get large pixelation, green or gray screen and weird moving blocks which is unacceptable for production of professional TV, though personal player may have this behaviour.

So here is what I would like. I would like that the external process could send a message through the control pipe and instruct the shmsink that it temporarily is instructed not to signal more buffers ready for the process. That it self is not a very big difference, but the shmsink could use the information to signal upstream, that while it may be handing out shmbuffers upstream, it will temporarily not accept any more buffers downstream and a leaky queue upstream to shmsink can then leak(drop) buffers. Even better would it be if a decoder upstream can somehow use this information to understand, that it should not send decoded frames downstream, but it should continue to decode to keep an updated state. The latter may be unobtainable with the GStreamer (and decoder) design, but it would be nice to have to optimize resource usage. Even better would it be, if the decoder can stop decode while maintain a state and inspecting the encoeded stream so it when requested as soon as possible can start deliver a new decoded full frame.

Sorry for the lengthy text, but it's a rather complicated subject.

Best regards
Peter Maersk-Moller



On Fri, Sep 20, 2013 at 12:58 AM, Olivier Crête <[hidden email]> wrote:
On Thu, 2013-09-19 at 18:50 -0400, Olivier Crête wrote:
> One thing we could do is add properties to limit the size of "queue"
> that's inside shmsink by time or bytes or buffers (like the queue
> element). I'd certainly be open to something like that.

Actually, there is already a buffer-time property to limit the size of
the "queue", but it is only triggered when the buffer is free on the
receiving side, probably not exactly what you want, but that could be
fixed.

--
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel



_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel