Reading streaming video frame by frame

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Reading streaming video frame by frame

isak3t
This post was updated on .
Hello all,

I am extremely new to gstreamer.

I am trying to read frame data for a live streaming video through gstreamer in C to create an opengl texture.

Currently going through online repositories and tutorials I could come up with code below, which is still not working. I was hoping if somebody could point me in the right direction  or point out errors in code or errors in the approach I am using. Any help would be appreciated.

The code below is not creating texture. I am just accessing frame data as a first milestone.
Code lines which I feel are important to approach are marked as bold.

    #include <gst/gst.h>
    #include <string.h>
    #include <stdio.h>

    typedef struct _CustomData {
      GstElement *pipeline,*sink,*bin;
      GstPad *pad,*ghost_pad;
      GstBus *bus;
      GstMessage *msg;
      GMainLoop *loop;
      gboolean is_live;
    } CustomData;

    //signal handling function
    void frame_func(GstElement *sink, CustomData *data){
      printf("frame signal\n");
      GstSample *sample;
      GstBuffer *buf;
      g_signal_emit_by_name(sink,"pull_sample",&sample);
      buf=gst_sample_get_buffer(sample);
      printf("%c\n",buf[0]);
      if(sample){
        gst_sample_unref(sample);
      }
    }

    int main(int argc, char *argv[]) {
      GstElement *pipeline,*sink;
      GstBus *bus;
      GMainLoop *main_loop;
      GstStateChangeReturn ret;
      CustomData data;

    //initialize
      gst_init (&argc, &argv);
      memset (&data, 0, sizeof (data));

    //setup pipeline
      pipeline = gst_parse_launch ("uridecodebin uri=http://10.61.36.75/sac/SacwebContent/CVAW/CVAW.mp4! decodebin ! appsink name=video-sink caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"", NULL);

     //access sink to create signal
      sink=gst_bin_get_by_name(GST_BIN(pipeline),"video-sink");

    //setting video to play
      ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
      if (ret == GST_STATE_CHANGE_FAILURE)  {
        g_printerr ("Unable to set the pipeline to the playing state.\n");
        gst_object_unref (pipeline);
        return -1;
      } else{
         if (ret == GST_STATE_CHANGE_NO_PREROLL)
          printf("for live sources, state changed but data could not be produced\n");
        else if (ret == GST_STATE_CHANGE_ASYNC)
          printf("state will change async\n");
        else if (ret == GST_STATE_CHANGE_SUCCESS)
          printf("successfully played\n");
        else
          printf("unknown %d\n",ret);
        data.is_live=TRUE;
      }
      bus = gst_element_get_bus (pipeline);

    //setting up signals from sink
      g_object_set(G_OBJECT(sink),"emit-signals",TRUE,NULL);
      g_signal_connect (sink, "new-sample", G_CALLBACK(frame_func), &data);


    //setting up loop
      main_loop = g_main_loop_new (NULL, FALSE);
      data.loop = main_loop;
      data.pipeline = pipeline;
      data.sink = sink;

      g_main_loop_run (main_loop);

    //cleanup
      g_main_loop_unref (main_loop);
      gst_object_unref (bus);
      gst_element_set_state (pipeline, GST_STATE_NULL);
      gst_object_unref (pipeline);
      return 0;
    }

_______________________________________________
gstreamer-devel mailing list
gstreamer-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

isak3t
This post was updated on .
bumping up the thread becuase i have yet been unable to solve the problem.

Any help or pointing in right direction would be highly appreciated

the key lines i feel to read a video frame data are

pipeline = gst_parse_launch ("uridecodebin uri=http://10.61.36.75/sac/SacwebContent/CVAW/CVAW.mp4! decodebin ! appsink name=video-sink caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"", NULL);

sink=gst_bin_get_by_name(GST_BIN(pipeline),"video-sink");

g_object_set(G_OBJECT(sink),"emit-signals",TRUE,NULL);
g_signal_connect (sink, "new-sample", G_CALLBACK(frame_func), &data);

void frame_func(GstElement *sink, CustomData *data){
 g_signal_emit_by_name(sink,"pull_sample",&sample);
 buf=gst_sample_get_buffer(sample);
 printf("%c\n",buf[0]);
}

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

omer.tal
This post was updated on .
Hello sir,

In your callback, try this:

  GstMapInfo info;
  sample = gst_app_sink_pull_sample((GstAppSink*)sink);
  if (sample) {
    buffer = gst_sample_get_buffer(sample);
    gst_buffer_map(buffer, &info, GST_MAP_READ);
    gst_sample_unref (sample);


Your data should be stored in *info.data*

Let me know if it works out for you...



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

isak3t
Hello Sir,

Currently the callback function itself is not being called.
I have included a line *printf("frame signal\n");* in beginnning of
frame_func fucntion but it is not printed.

The only output on console is :
state will change async



Additonally,
Trying your code in frame_func give the error as *GstAppSink  undeclared*
Is there a specific header to be included for it?





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

omer.tal
Hello again,

So your issue was the return type of the callback:

    //signal handling function
static GstFlowReturn new_sample (GstElement *sink, CustomData *data){
      printf("Hello\n");
      printf("frame signal\n");
      GstSample *sample;
      GstBuffer *buf;
      g_signal_emit_by_name(sink,"pull_sample",&sample);
      buf=gst_sample_get_buffer(sample);
//      printf("%c\n",buf[0]);
      if(sample){
        gst_sample_unref(sample);
        return GST_FLOW_OK;
      }
    return GST_FLOW_ERROR;
    }


Hope this works for you. I've tested your program with a sample pipeline and
it worked for me.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

isak3t
This post was updated on .
Hello Omer,

First thank you so much for effort and help. I was really stuck at this.

However I still could not run with the modification.
These are 2 codes with the link to download the code file directly

1)
modified_as_suggested.c   :  
https://drive.google.com/file/d/19yKnsADr6Yqze9g-pnxvzG2fDsemH9-k/view?usp=sharing
<https://drive.google.com/file/d/19yKnsADr6Yqze9g-pnxvzG2fDsemH9-k/view?usp=sharing

This file is with the modification as suggested.
I do not get any print of "Hello" on terminal and only get "state will change async"

2)
modified_with_msg_handling.c :
https://drive.google.com/file/d/1FCv_pmo-vhATyq41NJPwaX-ZJhV0CtWm/view?usp=sharing
<https://drive.google.com/file/d/1FCv_pmo-vhATyq41NJPwaX-ZJhV0CtWm/view?usp=sharing

This file is with modification as suggested and includes bus message handling.
Here I get following message on terminal :
state will change async
Error : Internal data stream error

Can you run either of these codes with pipeline as it is and let me know if you get same output as mine so I can check if it is code issue or environment issue.

PS : I tested the video with pipleline built with playbin and the video plays, so it is not an error with the video data





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

omer.tal
Hey... Sorry for the late response.

So I downloaded the file modified_as_suggested.c..

Firstly, I tried to ping the IP address which is in the pipeline. I got no
response, so obviously I could not use your pipeline.
So I took a pipeline which I use for my own development (and kept your
appsink element as is):

*rtspsrc location=rtsp://127.0.0.1:8554/test ! rtph264depay name=depay !
h264parse ! avdec_h264 ! videoconvert ! videoscale ! appsink name=video-sink
caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"", NULL*

With this pipeline everything worked fine.

Please try the following pipeline and see if you get anything on standard
output:
*videotestsrc ! appsink name=video-sink
caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"*

I've tested this (just to be sure) on my machine and it worked just fine. If
it doesn't work for you it must be environment issues (though I highly doubt
if that's the case).



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Reading streaming video frame by frame

isak3t
This post was updated on .
Hello Omer,

Thank you so much for the help. I was finally able to get things working with error handling from the bus.
Will now implement in actual application and evaluate performance.

Also I found the code to read frame data from the following link :  <https://stackoverflow.com/questions/60512213/how-to-process-frames-from-video-received-from-a-file

------------Below this only for reference of anybody who might require this later--------------------

The code for reading a video data frame by frame is attached code.c

I compiled using following command. You can modify accordingly as per your environment path

.\gcc.exe code.c -I "D:\gstreamer\1.0\x86\include\gdk-pixbuf-2.0\" -I "D:\gstreamer\1.0\x86\include\gstreamer-1.0\gst\app\" -I "D:\gstreamer\1.0\x86\include\gstreamer-1.0\" -I "D:\gstreamer\1.0\x86\include\glib-2.0\" -I "D:\gstreamer\1.0\x86\lib\glib-2.0\include\" -L "D:\gstreamer\1.0\x86\bin\" -L "D:\gstreamer\1.0\x86\lib" -l glib-2.0-0 -l gstreamer-1.0-0 -l gobject-2.0-0 -l gthread-2.0-0  -l gdk_pixbuf-2.0-0 -l gstapp-1.0 -o "D:\test.exe"





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel