Appsink Vs. Last-Sample

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Appsink Vs. Last-Sample

Eric-Stone
Hello all,

I am using Gstreamer to render live video with Unity3D. Currently, I append a fakesink to my pipeline, and at every frame of the Unity application the fakesink's last-sample is queried and used to update a texture.

This works, but I feel that the performance of the plugin is a bit lacking. I'm considering rewriting the plugin to instead use an appsink.

My question is: does querying appsink samples have any different performance characteristics relative to querying a fakesink's last-sample?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Appsink Vs. Last-Sample

Nirbheek Chauhan
On Thu, Mar 16, 2017 at 8:49 AM, Brendan Lockhart <[hidden email]> wrote:

> I am using Gstreamer to render live video with Unity3D. Currently, I append
> a fakesink to my pipeline, and at every frame of the Unity application the
> fakesink's last-sample is queried and used to update a texture.
>
> This works, but I feel that the performance of the plugin is a bit lacking.
> I'm considering rewriting the plugin to instead use an appsink.
>
> My question is: does querying appsink samples have any different performance
> characteristics relative to querying a fakesink's last-sample?
>

Are you connecting to the last-sample signal and running a callback?
If so, there is a noticeable difference between using a gobject signal
for running a callback vs using a function call in libgstapp. You can
try using gst_app_sink_pull_sample() and friends.

However, I think you should do some performance measurements first. I
suspect that since you're doing GL rendering, most of your overhead is
in copying video buffers around and the function call vs. signal
callback overhead is minor in comparison. In that case, you should be
using glimagesink to write directly to a GL texture. Here's some
examples that do this with SDL2:
https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/sdl

Cheers,
Nirbheek
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Appsink Vs. Last-Sample

Eric-Stone
>> I am using Gstreamer to render live video with Unity3D. Currently, I append
>> a fakesink to my pipeline, and at every frame of the Unity application the
>> fakesink's last-sample is queried and used to update a texture.
>>
>> This works, but I feel that the performance of the plugin is a bit lacking.
>> I'm considering rewriting the plugin to instead use an appsink.
>>
>> My question is: does querying appsink samples have any different performance
>> characteristics relative to querying a fakesink's last-sample?
>>
>Are you connecting to the last-sample signal and running a callback?
>If so, there is a noticeable difference between using a gobject signal
>for running a callback vs using a function call in libgstapp. You can
>try using gst_app_sink_pull_sample() and friends.

Thanks for the reply!

Every frame of the application, the plugin get's the fakesink's "last-sample" property. From the returned GstSample it gets the buffer and copies that to a target texture, so it's not using any callbacks.

>However, I think you should do some performance measurements first. I
>suspect that since you're doing GL rendering, most of your overhead is
>in copying video buffers around and the function call vs. signal
>callback overhead is minor in comparison. In that case, you should be
>using glimagesink to write directly to a GL texture. Here's some
>examples that do this with SDL2:
>https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/sdl

You're probably right that the major bottleneck is most likely copying the data from the CPU to the GPU. I looked into it, and I think I can have gstamcvideodec decode directly into GL memory.
Unfortunately when I try to do this gstamcvideodec logs:
"ERR amcvideodec gstamcvideodec.c:1977:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0> Codec only supports GL output but downstream does not"

This is despite that fact that, before that error, gstamcvideodec logs:

"gstamcvideodec.c:1904:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0> Available downstream caps: video/x-raw(memory:GLMemory), format=(string)RGBA, texture-target=(string)external-oes"

Which I believe should intersect properly with amcvideodec's required GL output sink caps.

Could this be a bug, or am I doing something wrong here?
This is with Gstreamer 1.10.4, and I'm setting playbin's videosink to "video/x-raw(memory:GLMemory),texture-target=2D,format=RGBA ! fakesink sync=1 qos=1 name=sink"

On Thu, Mar 16, 2017 at 9:12 AM, Nirbheek Chauhan <[hidden email]> wrote:
On Thu, Mar 16, 2017 at 8:49 AM, Brendan Lockhart <[hidden email]> wrote:
> I am using Gstreamer to render live video with Unity3D. Currently, I append
> a fakesink to my pipeline, and at every frame of the Unity application the
> fakesink's last-sample is queried and used to update a texture.
>
> This works, but I feel that the performance of the plugin is a bit lacking.
> I'm considering rewriting the plugin to instead use an appsink.
>
> My question is: does querying appsink samples have any different performance
> characteristics relative to querying a fakesink's last-sample?
>

Are you connecting to the last-sample signal and running a callback?
If so, there is a noticeable difference between using a gobject signal
for running a callback vs using a function call in libgstapp. You can
try using gst_app_sink_pull_sample() and friends.

However, I think you should do some performance measurements first. I
suspect that since you're doing GL rendering, most of your overhead is
in copying video buffers around and the function call vs. signal
callback overhead is minor in comparison. In that case, you should be
using glimagesink to write directly to a GL texture. Here's some
examples that do this with SDL2:
https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/sdl

Cheers,
Nirbheek
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Appsink Vs. Last-Sample

Eric-Stone
>You're probably right that the major bottleneck is most likely copying the
>data from the CPU to the GPU. I looked into it, and I think I can have
>gstamcvideodec decode directly into GL memory.
>Unfortunately when I try to do this gstamcvideodec logs:
>"ERR amcvideodec
>gstamcvideodec.c:1977:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Codec only supports GL output but downstream does not"
>
>This is despite that fact that, before that error, gstamcvideodec logs:
>
>"gstamcvideodec.c:1904:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Available downstream caps: video/x-raw(memory:GLMemory),
>format=(string)RGBA, texture-target=(string)external-oes"
>
>Which I *believe *should intersect properly with amcvideodec's required GL
>output sink caps.
>
>Could this be a bug, or am I doing something wrong here?
>This is with Gstreamer 1.10.4, and I'm setting playbin's videosink to
>"video/x-raw(memory:GLMemory),texture-target=2D,format=RGBA ! fakesink
>sync=1 qos=1 name=sink"

Does anyone have any suggestions on how I can figure out if this is a bug in my application code, or a bug in gstamcvideodec?
Any advice or feedback would be more than appreciated.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: Appsink Vs. Last-Sample

Matthew Waters



On 22/03/17 09:56, Brendan Lockhart wrote:
>You're probably right that the major bottleneck is most likely copying the
>data from the CPU to the GPU. I looked into it, and I think I can have
>gstamcvideodec decode directly into GL memory.
>Unfortunately when I try to do this gstamcvideodec logs:
>"ERR amcvideodec
>gstamcvideodec.c:1977:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Codec only supports GL output but downstream does not"
>
>This is despite that fact that, before that error, gstamcvideodec logs:
>
>"gstamcvideodec.c:1904:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Available downstream caps: video/x-raw(memory:GLMemory),
>format=(string)RGBA, texture-target=(string)external-oes"
>
>Which I *believe *should intersect properly with amcvideodec's required GL
>output sink caps.
>Could this be a bug, or am I doing something wrong here?
>This is with Gstreamer 1.10.4, and I'm setting playbin's videosink to
>"video/x-raw(memory:GLMemory),texture-target=2D,format=RGBA ! fakesink
>sync=1 qos=1 name=sink"

These caps don't intersect with amcvideodec's caps as they have different texture-target fields.  amcvideodec only ever does texture-target=external-oes.

Cheers
-Matt

Does anyone have any suggestions on how I can figure out if this is a bug in my application code, or a bug in gstamcvideodec?
Any advice or feedback would be more than appreciated.



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (527 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Appsink Vs. Last-Sample

Eric-Stone

>>You're probably right that the major bottleneck is most likely copying the
>>data from the CPU to the GPU. I looked into it, and I think I can have
>>gstamcvideodec decode directly into GL memory.
>>Unfortunately when I try to do this gstamcvideodec logs:
>>"ERR amcvideodec
>>gstamcvideodec.c:1977:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Codec only supports GL output but downstream does not">These caps don't intersect with amcvideodec's caps as they have different texture-target fields.  amcvideodec only ever does texture-target=external-oes.

I switched "fakesink" with "glimagesink". I no longer get the "GL Output disabled" error, and performance is much better.

Thank you guys for taking the time to help.

On Wed, Mar 22, 2017 at 3:58 AM, Matthew Waters <[hidden email]> wrote:



On 22/03/17 09:56, Brendan Lockhart wrote:
>You're probably right that the major bottleneck is most likely copying the
>data from the CPU to the GPU. I looked into it, and I think I can have
>gstamcvideodec decode directly into GL memory.
>Unfortunately when I try to do this gstamcvideodec logs:
>"ERR amcvideodec
>gstamcvideodec.c:1977:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Codec only supports GL output but downstream does not"
>
>This is despite that fact that, before that error, gstamcvideodec logs:
>
>"gstamcvideodec.c:1904:gst_amc_video_dec_set_format:<amcvideodec-omxexynosavcdec0>
>Available downstream caps: video/x-raw(memory:GLMemory),
>format=(string)RGBA, texture-target=(string)external-oes"
>
>Which I *believe *should intersect properly with amcvideodec's required GL
>output sink caps.
>Could this be a bug, or am I doing something wrong here?
>This is with Gstreamer 1.10.4, and I'm setting playbin's videosink to
>"video/x-raw(memory:GLMemory),texture-target=2D,format=RGBA ! fakesink
>sync=1 qos=1 name=sink"

These caps don't intersect with amcvideodec's caps as they have different texture-target fields.  amcvideodec only ever does texture-target=external-oes.

Cheers
-Matt

Does anyone have any suggestions on how I can figure out if this is a bug in my application code, or a bug in gstamcvideodec?
Any advice or feedback would be more than appreciated.




_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel