glimagesink and how it likes to do things

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

glimagesink and how it likes to do things

Lasse Laursen
Hey y'all,

I know this is kind of a beaten horse, but given the recent release of the 1.4 version of gStreamer I thought I'd pull out a stick and give it a few more whacks.

Today I finally got around to working out how to properly use an appsink to copy gstreamers video output directly into an OpenGL texture. This works pretty well for now - but I'm constantly trying to work out, if there's a way of bypassing having to copy the data manually and just have gstreamer throw it into an OpenGL texture for me.

The glimagesink looked promising, despite it's current 'bad' status and... 0.10 reference manual? I take it, this is the most up-to-date doc currently:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-gl-plugins/html/
(although it probably should eventually be merged into the 'bad' folder given it's status?)

Anyway - I had a closer look at the following example:

http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/generic/cube/main.cpp

It seems as though the client callback receives a texture reference... Is there anyway of specifying a texture to which one would like the glimagesink to write, instead of receiving a reference to one in every callback? If not - is it possible to somehow intercept the texture ID earlier - and are there any guarantees that it doesn't change at some point down the line?

Regards,
Lasse

--
Lasse Farnung Laursen
Post.doc at the University of Tokyo
www.lasselaursen.com
FB: lasse.laursen
Twitter: @PMP_J

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: glimagesink and how it likes to do things

Lasse Laursen
Looking closer at available plug-ins, there's also something called GstGLUpload - which says that it uploads to GL Textures... Is this supposed to be used as a sink? Can it?

I've also just learned that if I wanted to use the GstGLShader object, I am forced to make use of the GstGLImageSink? Would GstGLUpload also be acceptable here?

Regards,
Lasse

On 08-08-2014 19:09, Lasse Laursen wrote:
Hey y'all,

I know this is kind of a beaten horse, but given the recent release of the 1.4 version of gStreamer I thought I'd pull out a stick and give it a few more whacks.

Today I finally got around to working out how to properly use an appsink to copy gstreamers video output directly into an OpenGL texture. This works pretty well for now - but I'm constantly trying to work out, if there's a way of bypassing having to copy the data manually and just have gstreamer throw it into an OpenGL texture for me.

The glimagesink looked promising, despite it's current 'bad' status and... 0.10 reference manual? I take it, this is the most up-to-date doc currently:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-gl-plugins/html/
(although it probably should eventually be merged into the 'bad' folder given it's status?)

Anyway - I had a closer look at the following example:

http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/generic/cube/main.cpp

It seems as though the client callback receives a texture reference... Is there anyway of specifying a texture to which one would like the glimagesink to write, instead of receiving a reference to one in every callback? If not - is it possible to somehow intercept the texture ID earlier - and are there any guarantees that it doesn't change at some point down the line?

Regards,
Lasse

--
Lasse Farnung Laursen
Post.doc at the University of Tokyo
www.lasselaursen.com
FB: lasse.laursen
Twitter: @PMP_J


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Lasse Farnung Laursen
Post.doc at the University of Tokyo
www.lasselaursen.com
FB: lasse.laursen
Twitter: @PMP_J

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: glimagesink and how it likes to do things

Matthew Waters
On 08/08/14 20:27, Lasse Laursen wrote:
> Looking closer at available plug-ins, there's also something called
> GstGLUpload - which says that it uploads to GL Textures... Is this
> supposed to be used as a sink? Can it?

GstGLImageSink is the only sink currently provided by the opengl
plugins.  In 1.4, GstGLUpload is now a helper object (not an element)
available within libgstgl for elements/applications to use.  All opengl
elements implicitly upload/download as required by the pipeline using it
and its counterpart, GstGLDownload.

> I've also just learned that if I wanted to use the GstGLShader object,
> I am forced to make use of the GstGLImageSink? Would GstGLUpload also
> be acceptable here?

GstGLShader (the element), like all opengl elements will upload/download
data as required by the surrounding elements.  So you could have any
other non-GL element after it and it should still work.

> Regards,
> Lasse
>
> On 08-08-2014 19:09, Lasse Laursen wrote:
>> Hey y'all,
>>
>> I know this is kind of a beaten horse, but given the recent release
>> of the 1.4 version of gStreamer I thought I'd pull out a stick and
>> give it a few more whacks.
>>
>> Today I finally got around to working out how to properly use an
>> appsink to copy gstreamers video output directly into an OpenGL
>> texture. This works pretty well for now - but I'm constantly trying
>> to work out, if there's a way of bypassing having to copy the data
>> manually and just have gstreamer throw it into an OpenGL texture for me.
GstGLUpload is designed for this use case :)

>> The glimagesink looked promising, despite it's current 'bad' status
>> and... 0.10 reference manual? I take it, this is the most up-to-date
>> doc currently:
>>
>> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-gl-plugins/html/
>>
>> (although it probably should eventually be merged into the 'bad'
>> folder given it's status?)

There are docs in the -bad repository however they don't seem to have
been updated on/uploaded to the server.

>> Anyway - I had a closer look at the following example:
>>
>> http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/generic/cube/main.cpp
>>
>>
>> It seems as though the client callback receives a texture
>> reference... Is there anyway of specifying a texture to which one
>> would like the glimagesink to write, instead of receiving a reference
>> to one in every callback? If not - is it possible to somehow
>> intercept the texture ID earlier - and are there any guarantees that
>> it doesn't change at some point down the line?
The exact behaviour will depend on the specific elements involved.  The
two possibilities are
1. glimagesink uploads the incoming video frame -> texture id will
change every draw call (performance gain is possible here although only
to the second case)
2. glimagesink displays upstream's GL texture and is using a bufferpool
-> texture id will loop between a certain amount of texture id's
depending on the lifetime of the buffers.  This is so that it is
possible to upload the previous texture while the next texture is being
displayed/used.  The other catch is that the basesink class by default
holds on to the last buffer rendered until the next buffer arrives.

Any existence of the texture id outside of the callback is not really
guaranteed.  Is there any particular reason you would like to set the
texture id for upload?

>> Regards,
>> Lasse
>>
>> --
>> Lasse Farnung Laursen
>> Post.doc at the University of Tokyo
>> www.lasselaursen.com <http://www.lasselaursen.com>
>> FB: lasse.laursen <https://www.facebook.com/lasse.laursen>
>> Twitter: @PMP_J <https://twitter.com/PMP_J>
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: glimagesink and how it likes to do things

Lasse Laursen
Thanks a ton for the answer Matthew - although your answers opens up a few more questions, if you don't mind?

On 08-08-2014 20:12, Matthew Waters wrote:
On 08/08/14 20:27, Lasse Laursen wrote:
Looking closer at available plug-ins, there's also something called
GstGLUpload - which says that it uploads to GL Textures... Is this
supposed to be used as a sink? Can it?
GstGLImageSink is the only sink currently provided by the opengl
plugins.  In 1.4, GstGLUpload is now a helper object (not an element)
available within libgstgl for elements/applications to use.  All opengl
elements implicitly upload/download as required by the pipeline using it
and its counterpart, GstGLDownload.

Ah - so if I understand you correctly - it's completely possible (and even ok) to use an opengl element (such as the GstGLShader element) in a pipeline, and then have the final result arrive at an appsink, and not the gstglimagesink?

You say all elements implicitly upload/download as required, so am I to understand that I could define a pipeline as follows "uridecodebin uri=<some file> ! GlShader ! appsink name=sink caps=<some caps>", and pass it to 'gst_parse_launch', and it'd figure out to insert the upload/download elements around the shader as required?

      
I've also just learned that if I wanted to use the GstGLShader object,
I am forced to make use of the GstGLImageSink? Would GstGLUpload also
be acceptable here?
GstGLShader (the element), like all opengl elements will upload/download
data as required by the surrounding elements.  So you could have any
other non-GL element after it and it should still work.

Regards,
Lasse

On 08-08-2014 19:09, Lasse Laursen wrote:
Hey y'all,

I know this is kind of a beaten horse, but given the recent release
of the 1.4 version of gStreamer I thought I'd pull out a stick and
give it a few more whacks.

Today I finally got around to working out how to properly use an
appsink to copy gstreamers video output directly into an OpenGL
texture. This works pretty well for now - but I'm constantly trying
to work out, if there's a way of bypassing having to copy the data
manually and just have gstreamer throw it into an OpenGL texture for me.
GstGLUpload is designed for this use case :)

In your first reply, you mention that GlImageSink is currently the *only* sink provided by the opengl plugins, does this mean that I'm not allowed to have a pipeline end in a GstGLUpload, if all I want is the data to end in a texture I am managing? Looking at the documentation, for GstGLUpload, I'm in doubt if this is even possible? That is to say - that I specify the texture ID?

You ask at the end why I want to set the texture ID - yes - essentially I'm using the Cinder framework, and I'd like to integrate GStreamers output in a simple GUI I'm building. To have it play well with Cinder, I'd need to use cinders own Texture object, which - as far as I can tell either requires me to give a fixed texture ID to gstreamer where I'd like the data to end up, or retrieve a texture ID from GStreamer and build the cinder texture object using it.

----

Just to reiterate - My current implementation scenario is, I use gstreamer to build a pipeline which currently includes 0 GL elements. It ends with an Appsink, from which I directly copy data into a Texture for every received sample. Seems to work ok for now.

2 Potential scenarios:

1 - A pipeline with 0 GL elements - except potentially the last one. Is GstGLUpload the way to go to have the data end up in a texture for me to use? Any example code?

2 - A pipeline with 1 or more GL elements, most likely the GstGLShader object. Is it possible to access the handled data that's stored in a texture on the GPU? Or is it much easier to let it get handed back to the CPU, just to upload it again?


The glimagesink looked promising, despite it's current 'bad' status
and... 0.10 reference manual? I take it, this is the most up-to-date
doc currently:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-gl-plugins/html/

(although it probably should eventually be merged into the 'bad'
folder given it's status?)
There are docs in the -bad repository however they don't seem to have
been updated on/uploaded to the server.

Anyway - I had a closer look at the following example:

http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/generic/cube/main.cpp


It seems as though the client callback receives a texture
reference... Is there anyway of specifying a texture to which one
would like the glimagesink to write, instead of receiving a reference
to one in every callback? If not - is it possible to somehow
intercept the texture ID earlier - and are there any guarantees that
it doesn't change at some point down the line?
The exact behaviour will depend on the specific elements involved.  The
two possibilities are
1. glimagesink uploads the incoming video frame -> texture id will
change every draw call (performance gain is possible here although only
to the second case)
2. glimagesink displays upstream's GL texture and is using a bufferpool
-> texture id will loop between a certain amount of texture id's
depending on the lifetime of the buffers.  This is so that it is
possible to upload the previous texture while the next texture is being
displayed/used.  The other catch is that the basesink class by default
holds on to the last buffer rendered until the next buffer arrives.

Any existence of the texture id outside of the callback is not really
guaranteed.  Is there any particular reason you would like to set the
texture id for upload?

Regards,
Lasse

-- 
Lasse Farnung Laursen
Post.doc at the University of Tokyo
www.lasselaursen.com <http://www.lasselaursen.com>
FB: lasse.laursen <https://www.facebook.com/lasse.laursen>
Twitter: @PMP_J <https://twitter.com/PMP_J>


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

      

      

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Lasse Farnung Laursen
Post.doc at the University of Tokyo
www.lasselaursen.com
FB: lasse.laursen
Twitter: @PMP_J

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Re: glimagesink and how it likes to do things

Matthew Waters

On 08/08/14 21:41, Lasse Laursen wrote:

> Thanks a ton for the answer Matthew - although your answers opens up a
> few more questions, if you don't mind?
>
> On 08-08-2014 20:12, Matthew Waters wrote:
>> On 08/08/14 20:27, Lasse Laursen wrote:
>>> Looking closer at available plug-ins, there's also something called
>>> GstGLUpload - which says that it uploads to GL Textures... Is this
>>> supposed to be used as a sink? Can it?
>> GstGLImageSink is the only sink currently provided by the opengl
>> plugins.  In 1.4, GstGLUpload is now a helper object (not an element)
>> available within libgstgl for elements/applications to use.  All opengl
>> elements implicitly upload/download as required by the pipeline using it
>> and its counterpart, GstGLDownload.
>
> Ah - so if I understand you correctly - it's completely possible (and
> even ok) to use an opengl element (such as the GstGLShader element) in
> a pipeline, and then have the final result arrive at an appsink, and
> not the gstglimagesink?
Correct.

> You say all elements implicitly upload/download as required, so am I
> to understand that I could define a pipeline as follows "uridecodebin
> uri=<some file> ! GlShader ! appsink name=sink caps=<some caps>", and
> pass it to 'gst_parse_launch', and it'd figure out to insert the
> upload/download elements around the shader as required?

Correct although the upload/download are not in separate elements
anymore, it is provided by the GstGLUpload object in the libgstgl
library.  The docs for the libgstgl library that the opengl elements use
is
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/gl.html
which are up to date.

>>> I've also just learned that if I wanted to use the GstGLShader object,
>>> I am forced to make use of the GstGLImageSink? Would GstGLUpload also
>>> be acceptable here?
>> GstGLShader (the element), like all opengl elements will upload/download
>> data as required by the surrounding elements.  So you could have any
>> other non-GL element after it and it should still work.
>>
>>> Regards,
>>> Lasse
>>>
>>> On 08-08-2014 19:09, Lasse Laursen wrote:
>>>> Hey y'all,
>>>>
>>>> I know this is kind of a beaten horse, but given the recent release
>>>> of the 1.4 version of gStreamer I thought I'd pull out a stick and
>>>> give it a few more whacks.
>>>>
>>>> Today I finally got around to working out how to properly use an
>>>> appsink to copy gstreamers video output directly into an OpenGL
>>>> texture. This works pretty well for now - but I'm constantly trying
>>>> to work out, if there's a way of bypassing having to copy the data
>>>> manually and just have gstreamer throw it into an OpenGL texture
>>>> for me.
>> GstGLUpload is designed for this use case :)
>
> In your first reply, you mention that GlImageSink is currently the
> *only* sink provided by the opengl plugins, does this mean that I'm
> not allowed to have a pipeline end in a GstGLUpload, if all I want is
> the data to end in a texture I am managing? Looking at the
> documentation, for GstGLUpload, I'm in doubt if this is even possible?
> That is to say - that I specify the texture ID?
The 0.10 GstGLUpload was a filter element that transformed from raw
video frames into texture handles so it is/was not possible to use
gstglupload as a sink.  As for specifying the texture id, that is
currently not possible with the current API.

> You ask at the end why I want to set the texture ID - yes -
> essentially I'm using the Cinder framework, and I'd like to integrate
> GStreamers output in a simple GUI I'm building. To have it play well
> with Cinder, I'd need to use cinders own Texture object, which - as
> far as I can tell either requires me to give a fixed texture ID to
> gstreamer where I'd like the data to end up, or retrieve a texture ID
> from GStreamer and build the cinder texture object using it.

One option is to use
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/gst-plugins-bad-libs-gstglmemory.html#gst-gl-memory-copy-into-texture
however that involves a gpu side copy that will not work for all
textures on all platforms (RGBA is fine though).

> ----
>
> Just to reiterate - My current implementation scenario is, I use
> gstreamer to build a pipeline which currently includes 0 GL elements.
> It ends with an Appsink, from which I directly copy data into a
> Texture for every received sample. Seems to work ok for now.
>
> 2 Potential scenarios:
>
> 1 - A pipeline with 0 GL elements - except potentially the last one.
> Is GstGLUpload the way to go to have the data end up in a texture for
> me to use? Any example code?
I would build a pipeline looking like:
... ! glcolorscale ! appsink

The upload/download would be done by the glcolorscale element and the
specific caps (with features) on either side of it.  The appsink would
upload the buffer using map and wrap the GstGLMemory inside the
GstBuffer's you receive into Cinder's Texture object.

If you run 'gst-inspect-1.0 glcolorscale', you'll see that one of the
caps supported on the src and sinkpads include the 'memory:GLMemory'
caps feature.  That means that the GstBuffers that glcolorscale
consumes/produces will contain GstGLMemory's which contain GL texture
handles.

> 2 - A pipeline with 1 or more GL elements, most likely the GstGLShader
> object. Is it possible to access the handled data that's stored in a
> texture on the GPU? Or is it much easier to let it get handed back to
> the CPU, just to upload it again?

... ! glshader ! appsink
Exactly the same idea as in 1.  You would have an appsink that would
transform the GstGLMemory into Cinder's Texture.

>>
>>>> The glimagesink looked promising, despite it's current 'bad' status
>>>> and... 0.10 reference manual? I take it, this is the most up-to-date
>>>> doc currently:
>>>>
>>>> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-gl-plugins/html/
>>>>
>>>>
>>>> (although it probably should eventually be merged into the 'bad'
>>>> folder given it's status?)
>> There are docs in the -bad repository however they don't seem to have
>> been updated on/uploaded to the server.
>>
>>>> Anyway - I had a closer look at the following example:
>>>>
>>>> http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/generic/cube/main.cpp
>>>>
>>>>
>>>>
>>>> It seems as though the client callback receives a texture
>>>> reference... Is there anyway of specifying a texture to which one
>>>> would like the glimagesink to write, instead of receiving a reference
>>>> to one in every callback? If not - is it possible to somehow
>>>> intercept the texture ID earlier - and are there any guarantees that
>>>> it doesn't change at some point down the line?
>> The exact behaviour will depend on the specific elements involved.  The
>> two possibilities are
>> 1. glimagesink uploads the incoming video frame -> texture id will
>> change every draw call (performance gain is possible here although only
>> to the second case)
>> 2. glimagesink displays upstream's GL texture and is using a bufferpool
>> -> texture id will loop between a certain amount of texture id's
>> depending on the lifetime of the buffers.  This is so that it is
>> possible to upload the previous texture while the next texture is being
>> displayed/used.  The other catch is that the basesink class by default
>> holds on to the last buffer rendered until the next buffer arrives.
>>
>> Any existence of the texture id outside of the callback is not really
>> guaranteed.  Is there any particular reason you would like to set the
>> texture id for upload?
>>
>>>> Regards,
>>>> Lasse
>>>>
>>>> --
>>>> Lasse Farnung Laursen
>>>> Post.doc at the University of Tokyo
>>>> www.lasselaursen.com <http://www.lasselaursen.com>
>>>> FB: lasse.laursen <https://www.facebook.com/lasse.laursen>
>>>> Twitter: @PMP_J <https://twitter.com/PMP_J>
>>>>
>>>>
>>>> _______________________________________________
>>>> gstreamer-devel mailing list
>>>> [hidden email]
>>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

signature.asc (484 bytes) Download Attachment