python code for muxing klv and video

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view

python code for muxing klv and video

Arturo Flores

I have some code for muxing klv and video (included at the end of this email).

This code is a proof of concept to see if I am doing things correctly.  I have an mpg video with klv and video streams.  I first dumped the klv into a file using the following pipeline:

gst-launch-1.0 filesrc=input.ts ! tsdemux ! meta/x-klv ! filesink location=klv.bin

By inspecting the klv, I determined how many klv packets are in the stream and the klv frame rate.

I then extracted the video frames (using ffmpeg) into a folder.  I also used ffprobe to determine things like the video frame rate.

What I attempted to do is remux the video frames and klv and end up with something identical to the original file.  The code mostly works though there are some minor differences that I am curious about.

One difference I don't understand, is the ffprobe output of the original file has:

$ ffprobe input.ts
  Duration: 00:01:05.15, start: 271.125322, bitrate: 5535 kb/s

Whereas the ffprobe output of the remuxed file has:

$ ffprobe remuxed.ts
  Duration: 00:01:04.57, start: 3600.033367, bitrate: 1745 kb/s

Note the difference in start time.  What controls the start time here?  I tried setting the start-time property of x264enc, but that didn't do the trick.

Is there anything in this code that jumps out as the wrong way to do things, or is there a better way to mux together video and klv streams?



import matplotlib.pyplot as plt
import os
import sys
import gi
import json

gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

if len(sys.argv) != 2:
    print("Usage: python [config file]")

config_file = sys.argv[1]
fh = open(config_file)
config_json = json.load(fh)

vid_frames_folder = config_json["vid_frames_folder"]
vid_frame_rate = config_json["vid_frame_rate"]
vid_width = config_json["vid_width"]
vid_height = config_json["vid_height"]
klv_file = config_json["klv_file"]
klv_frame_rate = config_json["klv_frame_rate"]
klv_packet_size = config_json["klv_packet_size"]
out_file = config_json["out_file"]


# video source elements
vsrc = Gst.ElementFactory.make("appsrc", "vidsrc")
vqueue = Gst.ElementFactory.make("queue")
vtee = Gst.ElementFactory.make("tee")

# klv source elements
appsrc = Gst.ElementFactory.make("appsrc")
queue_klv = Gst.ElementFactory.make("queue")

# display elements
queue_display = Gst.ElementFactory.make("queue")
vcvt = Gst.ElementFactory.make("videoconvert", "vidcvt")
vsink = Gst.ElementFactory.make("autovideosink", "vidsink")

# recording elements
queue_record = Gst.ElementFactory.make("queue")
vcvt_encoder = Gst.ElementFactory.make("videoconvert")
encoder = Gst.ElementFactory.make("x264enc")
muxer = Gst.ElementFactory.make("mpegtsmux")
filesink = Gst.ElementFactory.make("filesink")

# configure video element
caps_str = "video/x-raw"
caps_str += ",format=(string)RGB,width={},height={}".format(vid_width,vid_height)
caps_str += ",framerate={}/1".format(int(vid_frame_rate))
vcaps = Gst.Caps.from_string(caps_str)
vsrc.set_property("caps", vcaps);
vsrc.set_property("format", Gst.Format.TIME)

# configure appsrc element
caps_str = "meta/x-klv"
caps_str += ",parsed=True"
caps = Gst.Caps.from_string(caps_str)
appsrc.set_property("caps", caps)
# appsrc.connect("need-data", klv_need_data)
appsrc.set_property("format", Gst.Format.TIME)

# configure encoder
encoder.set_property("noise-reduction", 1000)
encoder.set_property("threads", 4)
encoder.set_property("bitrate", 1755)

# configure filesink
filesink.set_property("location", out_file)
filesink.set_property("async", 0)

pipeline = Gst.Pipeline()

# link video elements

# link display elements

# link recording elements

# link klv elements


fh = open(klv_file, 'rb')
timestamp = 0

klv_done = False
vid_done = False
vid_frame_counter = 0
klv_frame_counter = 0

t = 0
last_klv_t = 0
last_vid_t = 0
while True:
    if vid_done and klv_done:
    if t - last_klv_t >= 1.0 / klv_frame_rate:
        if not klv_done:
            klv_bytes =
            if klv_bytes:
                klvbuf = Gst.Buffer.new_allocate(None, klv_packet_size, None)
                klvbuf.fill(0, klv_bytes)
                klvbuf.pts = int(t * 1e9)
                klvbuf.dts = int(t * 1e9)

                appsrc.emit("push-buffer", klvbuf)
                klv_frame_counter += 1
                last_klv_t = t
                print("klv {} {}".format(klv_frame_counter, last_klv_t))
                klv_done = True

    if t - last_vid_t >= 1.0 / vid_frame_rate:
        if not vid_done:
            frame_filename = '%s/%04d.jpg' % (vid_frames_folder, vid_frame_counter + 1)
            if os.path.isfile(frame_filename):
                vid_frame = plt.imread(frame_filename)
                data = vid_frame.tostring()
                vidbuf = Gst.Buffer.new_allocate(None, len(data), None)
                vidbuf.fill(0, data)
                vidbuf.pts = int(t * 1e9)
                vidbuf.dts = int(t * 1e9)

                vsrc.emit("push-buffer", vidbuf)
                vid_frame_counter += 1
                last_vid_t = t
                print("vid {} {}".format(vid_frame_counter, last_vid_t))
                vid_done = True

    t += 0.000001


bus = pipeline.get_bus()

while True:
    msg = bus.poll(Gst.MessageType.ANY, Gst.CLOCK_TIME_NONE)
    t = msg.type
    if t == Gst.MessageType.EOS:
    elif t == Gst.MessageType.ERROR:
        err, debug = msg.parse_error()
        # print("Error: %s" % err, debug)
    elif t == Gst.MessageType.WARNING:
        err, debug = msg.parse_warning()
        # print("Warning: %s" % err, debug)
    elif t == Gst.MessageType.STATE_CHANGED:
    elif t == Gst.MessageType.STREAM_STATUS:
        # print(t)
        # print("Unknown message: %s" % msg)



gstreamer-devel mailing list
[hidden email]
Reply | Threaded
Open this post in threaded view

Re: python code for muxing klv and video

Fran Raga
This is great. I've been trying for years to figure out how to ingest
telemetry into a video, to no avail.

I've tried your code but the resulting video is slowed down, it could be
because the telemetry is not right.

I made a version some time ago to generate the klv from a csv to use it with
drones like a DJI and from those packages I'm using your code.

Did you make any improvements to your code?Any ideas? If you want videos to
try, I have a public repo.

Great work

Sent from:
gstreamer-devel mailing list
[hidden email]
Reply | Threaded
Open this post in threaded view

Re: python code for muxing klv and video

Fran Raga
In reply to this post by Arturo Flores
I have made some changes to your code.

If I use your code with all the frames of the video in a video that for
example 1 min in my case, the pipeline dies, I don't know what happens.I
need cut video for example the first 10 seconds.

Other point is the video result gives me an error when I play it
"meta/x-klv" , but the video show without problems.

I have made some changes to avoid the variable "t" I think it is not
necessary to do that step and neither to extract the frames, we can do that
with opencv to read the frames in real time and ingest data, is my only my

I would be happy to make this code work well with your help.


Sent from:
gstreamer-devel mailing list
[hidden email]