OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?

Question

Goal and problem

I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.

I'm looking for a way to keep up to date with a live source even if it means dropping frames.

Current approach

import cv2

cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')

while (True):
    _, frame = cap.read()
    cv2.imshow("camCapture", frame)
    cv2.waitKey(1)

I've validated the quality of the stream on VLC and it seems to work fine there.

cv2 speed

.

realistic/expected speed

Questions:

• What am I doing wrong here?
• Why is it so slow?
• How do I sync it to real-time speeds?

Solution 1

My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:

1. Set maximum buffer size

We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.

2. Set frame retrieval delay

Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.

Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.

---

If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.

from threading import Thread
import cv2, time

class ThreadedCamera(object):
    def __init__(self, src=0):
        self.capture = cv2.VideoCapture(src)
        self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
       
        # FPS = 1/X
        # X = desired FPS
        self.FPS = 1/30
        self.FPS_MS = int(self.FPS * 1000)
        
        # Start frame retrieval thread
        self.thread = Thread(target=self.update, args=())
        self.thread.daemon = True
        self.thread.start()
        
    def update(self):
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()
            time.sleep(self.FPS)
            
    def show_frame(self):
        cv2.imshow('frame', self.frame)
        cv2.waitKey(self.FPS_MS)

if __name__ == '__main__':
    src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
    threaded_camera = ThreadedCamera(src)
    while True:
        try:
            threaded_camera.show_frame()
        except AttributeError:
            pass

---

Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts

1. Python OpenCV streaming from camera - multithreading, timestamps

2. Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture

3. How to capture multiple camera streams with OpenCV?

4. OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?

5. Storing RTSP stream as video file with OpenCV VideoWriter

6. OpenCV video saving

7. Python OpenCV multiprocessing cv2.VideoCapture mp4

Solution 2

Attempt at threading

I've attempted this solution from nathancy with minor success.

It involves:

• creating a separate thread for image capture from the source
• using the main thread exclusively for display.

Code:

import cv2
from threading import Thread

class ThreadedCamera(object):
    def __init__(self, source = 0):

        self.capture = cv2.VideoCapture(source)

        self.thread = Thread(target = self.update, args = ())
        self.thread.daemon = True
        self.thread.start()

        self.status = False
        self.frame  = None

    def update(self):
        while True:
            if self.capture.isOpened():
                (self.status, self.frame) = self.capture.read()

    def grab_frame(self):
        if self.status:
            return self.frame
        return None  

if __name__ == '__main__':
    stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
    streamer = ThreadedCamera(stream_link)

    while True:
        frame = streamer.grab_frame()
        if frame is not None:
            cv2.imshow("Context", frame)
        cv2.waitKey(1) 

Jittery, but real-time results

.

The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.

Room for improvement

The real-time stream can be found here.

https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet

This site is scraped for the m3u8 using python's streamlink stream scraper.

import streamlink

streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)

which yeilds:

OrderedDict([

('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),

('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),

('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),

('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)

])

The possibility that the streams are being read wrong.

Comments

[Conic]

Can you tell me why the waiting (FPS syncing) is needed? Isn't the point of the buffer to capture frames as they come? It feels like it wouldn't matter what is grabbed from the top of the buffer. Building an artificial delay feels counterintuitive for me. Would that be about resource sharing in threads or something? Or is it that the images don't come in real-time exactly, but as they're able to be reconstructed or re-delivered. Either way, thank you for the detailed response. This works very well.

[nathancy]

@Conic The FPS syncing is needed because generally you want to sync with the source. If you are polling too slow, you will have periods where you don't have a new frame so you will be displaying the old frame. If you poll too fast (like before), you will display all the frames instantly. In both cases, this may cause the jitter/freeze so by inserting a delay, you can even out the frame distribution as it is displayed. Since there are two separate threads, once a frame is received, it is immediately displayed so that's another reason why we need a delay.

[nathancy]

Here's an analogy, say you're watching a Youtube video on 2x speed but you are using dial-up internet. The buffer rate will load slower than the video actually playing so eventually, the frames will catch up to the buffer and the video will freeze. This is what happens when the FPS is not synced. If we revert back to 1x speed, then the frames will be evened out and match the speed of the buffer rate so we don't get any freezing/jitter. The video then looks smooth

[Christoph Rackwitz]

OpenCV VideoCapture isn't polling at all. could you explain what you mean?

[sur0k]

@ChristophRackwitz actually it's polling in the update() which is called inside started thread: Thread(target=self.update, args=())

[Kenji Noguchi]

yah capture.read() inside of infinite loop is polling.

[Arunava]

Did you find a solution to this?

[Conic]

Yes I did. It's the answer with the green checkmark next to it from nathancy.

[Arunava]

Oops sorry. Since you said nathancy's answer was a minor success. I thought you meant the answer to this question (didnt check the link). Anyway. Thanks.

[Conic]

I can see how that might be misleading. I was referring to a different solution of his from another stack overflow post. He then responded with an updated answer that did FPS syncing. That ended up being the solution that worked for me.

[Arunava]

Yeah. Thanks :)

[nathancy]

It's constantly polling. The entire operation of capturing frames is put inside the thread so that if the program is exited, the capture operation will not hang as a zombie process. If we don't set daemon to True, it will be a zombie process when your main program dies. We set daemon True so that when our parent process dies, this will also kill all child processes. See threading documentation for more information

[Ben Zayed]

Could you please explain to me why you used a threaded camera. I set a maximum buffer size and a frame retrieval delay as you suggested and that was enough to get a smooth streaming.

[nathancy]

@BenZayed For a single camera just setting buffer size and delay should work but if you have multiple cameras running simultaneously you may need to use threading (I use this technique with 8 simultaneous camera streams)