Streaming RTP/RTSP: sync/timestamp problems

gstreamerh.264rtpstreamingvideo

I'm having some trouble streaming H.264 video over RTSP. The goal is to live-stream a camera image to an RTSP client (ideally a browser plugin in the end). This has been working pretty well so far, except for one problem: the video will lag on startup, stutter every few seconds, and has a ~4-second delay. This is bad.

Our setup is to encode with x264 (w/ zerolatency & ultrafast) and packed into RTSP/RTP with libavformat from ffmpeg 0.6.5. For testing, I'm receiving the stream with a GStreamer pipeline with gst-launch when connecting to an RTSP server. However, I've been able to reproduce the same issue when streaming straight from another GStreamer instance with just RTP.

Sending machine:

gst-launch videotestsrc ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=10.89.6.3

Receiving machine:

gst-launch udpsrc ! application/x-rtp,payload=96 ! rtph264depay ! decodebin ! xvimagesink

You can also run these both on the same machine, just change the host to 127.0.0.1 on the sender. On the receiving end, you should notice stuttering and generally poor-performing video, along with repeated warnings on the console:

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.

One commonly-suggested "fix" that I've seen all over the Internet is to use sync=false with xvimagesink:

gst-launch udpsrc ! application/x-rtp,payload=96 ! rtph264depay ! decodebin ! xvimagesink sync=false

The video will then play back with near-zero latency, even when tested with our camera software. This is useful for testing, but is not very useful for deployment, as it won't work with Totem, VLC, or their browser plugin embeds.

I'd like to try to solve the issue at the source; I'm suspicious that there's some sort of timestamp info missing on the H.264 stream by x264 or perhaps on the RTP payloads. Is there any way to modify the source gst pipeline so that I do not need to use sync=false on the receiver?

If that's not possible, how can I tell clients (via SDP or otherwise) that the stream should not be synchronized? Ultimately, we'd embed this in the browser using a VLC plugin of sorts, so a solution that would work there would be even better.

Best Answer

As root.ctrlc posted, you can use sync=FALSE. However, you might notice a huge increase in CPU usage on the sender's end. The reason is that sync=FALSE tells the sink to just push out buffers as soon as it receives them. The sink drives the whole pipeline. Therefore, sync=FALSE will cause the pipeline to encode video and push it to UDP as fast as possible; it will use 100% CPU.

What you need is the gstrtpjitterbuffer. It also takes care of the timestamps, which are broken here.

Example sender:

gst-launch-0.10 -v videotestsrc ! videorate ! video/x-raw-yuv, framerate=30/1 ! ffmpegcolorspace ! x264enc ! rtph264pay ! udpsink port=50000 host=<sender IP>

Example receiver:

gst-launch-0.10 udpsrc port=50000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000 , encoding-name=(string)H264 , payload=(int)96" ! gstrtpjitterbuffer ! rtph264depay ! ffdec_h264 ! ffmpegcolorspace ! videoscale ! "video/x-raw-yuv, width=320, height=240" ! xvimagesink

Technically 'Yes'

(but not really...)

HTML 5's <video> tag is protocol agnostic—it does not care. You place the protocol in the src attribute as part of the URL. E.g.:

<video src="rtp://myserver.com/path/to/stream">
    Your browser does not support the VIDEO tag and/or RTP streams.
</video>

or maybe

<video src="http://myserver.com:1935/path/to/stream/myPlaylist.m3u8">
    Your browser does not support the VIDEO tag and/or RTP streams.
</video>

That said, the implementation of the <video> tag is browser specific. Since it is early days for HTML 5, I expect frequently changing support (or lack of support).

From the W3C's HTML5 spec (The video element):

User agents may support any video and audio codecs and container formats

Streaming audio and video in sync for mp4 container using Gstreamer framework

There seem to be issues with AAC in RTP as well as other RTP payloaders in gstreamer.

As mentioned in the other answer, it is anyway the best strategy to not demux and split video and audio if it is desired to have a synchronized playback on the receiver side. Additionally, a container format that is streamable can also improve overall functionality.

My proposal is therefore to remux the data streams into MPEG2-TS and transport that via RTP. This has several advantages, like good re-synchronization after packet loss and that A/V always stays sync.

Example

This works for me with gstreamer 1.2.4:

Sender

gst-launch-1.0 -v filesrc location=/foo/bar.mp4 ! \
  qtdemux name=demux demux. ! h264parse config-interval=10 ! queue ! mux. \
  mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=10.0.0.1 port=5003 \
  demux. ! aacparse ! queue ! mux.

Receiver (file)

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse !  filesink location=/foo/bar.ts

Receiver (output)

This command is based on the commands in the question, could not test it with ALSA:

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse ! tsdemux name=demux \
  demux. ! queue ! audioconvert ! alsasink \
  demux. ! queue ! xvimagesink