Looking for temporal upsampling / motion interpolation software

video images graphics animation

6,171

Solution 1

The MVTools plugin for AVISynth performs frame interpolation. AviSynth (sourceforge) is a script based video editing tool. You define operations to perform in a script text file.

The script i had to playback Charlie Wilson's War at 120fps:

cyia.avs:

source = DirectShowSource("Cyia Batten - Charlie Wilson's War.avi")

# assume progressive NTSC Film 23.976 source

# Fast Search
#============
# we use explicit idx for more fast processing
#backward_vec = source.MVAnalyse(blksize=16, isb = true, chroma=false, pel=1, searchparam=1, idx=1)
#forward_vec = source.MVAnalyse(blksize=16, isb = false, chroma=false, pel=1, searchparam=1, idx=1)


# Slow Search
# ===========
# Use block overlap, halfpixel accuracy and Exhaustive search
backward_vec = source.MVAnalyse(overlap=4, isb = true, pel=2, search=3, idx=1)
forward_vec = source.MVAnalyse(overlap=4, isb = false, pel=2, search=3, idx=1)


#double framerate
#source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), den=FramerateDenominator(source), idx=1)

#triple framerate
#source.MVFlowFps(backward_vec, forward_vec, num=3*FramerateNumerator(source), den=FramerateDenominator(source), mask=0, idx=1)

#120fps
source.MVFlowFps(backward_vec, forward_vec, num=120, den=1, mask=0, idx=1)

Yes, it does destroy the movie quality when converted to anything higher than 24fps.

Solution 2

Short answer:

It takes a lot of manual work with current technology to make it work.

Longer answer:

The usual method of getting a "in-between" image (to so-called optical flow) is by mapping points in both pictures (source, destination) and calculating the transform between the two pictures. Of course, we do have edge-finding algorithms or predictive algorithms that tracks different objects (Blob, kernel-based, contour, etc, etc.) But a full automation of a transform of "real life" motion picture is at best experimental.

Solution 3

slowmoVideo

slowmoVideo is an OpenSource program that creates slow-motion videos from your footage.

But it does not simply make your videos play at 0.01× speed. You can smoothly slow down and speed up your footage, optionally with motion blur.

How does slow motion work? slowmoVideo tries to find out where pixels move in the video (this information is called Optical Flow), and then uses this information to calculate the additional frames.

Gstreamer slowmo plugin

The plugin uses code from slowmovideo, porting it from QT and its own internal decoding "framework" based on ffmpeg to gstreamer.

6,171

timday

"The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry." - Henry Petroski "What if we didn't take it to our limit...wouldn't we be forever dissatisfied ?" - Doug Scott "Problems are inevitable. Problems are soluble." - David Deutsch "The incremental increase in systemic complexity is rarely if ever recognized as a problem that additional complexity can't solve." - Charles Hugh Smith (OfTwoMinds blog) "If you don’t make mistakes, you’re not working on hard enough problems. And that’s a big mistake." - Frank Wilczek "Only those that risk going too far can possibly find out how far one can go." – T.S. Eliot "Engineers turn dreams into reality" - Giovanni Caproni (in Hayao Miyazaki's The Wind Rises) "When an Oxford man walks into the room, he walks in like he owns it. When a Cambridge man walks into the room, he walks in like he doesn't care who owns it." - my grandmother "The greatest scientific discovery was the discovery of ignorance" - Yuval Noah Harari. "Always train your doubt most strongly on those ideas that you really want to be true." - Sean Carroll "The first principle is that you must not fool yourself — and you are the easiest person to fool" - Richard Feynman "On the plains of hesitation lie the blackened bones of countless millions who at the dawn of victory lay down to rest, and in resting died." - Adlai E. Stevenson "Therefore Simplicio, come either with arguments and demonstrations and bring us no more Texts and authorities, for our disputes are about the Sensible World, and not one of Paper." - Salviati to Simplicio in Galileo's Dialogue On Two World Systems (1632) "The larger the island of knowledge, the longer the shoreline of wonder." - Ralph W. Sockman "I never enlighten anyone who has not been driven to distraction by trying to understand a difficulty or who has not got into a frenzy trying to put his ideas into words. When I have pointed out one corner of a square to anyone and he does not come back with the other three, I will not point it out to him a second time." - Confucius "The way to bring about the new age of peace and enlightenment is to assume it has already started" - ?

Updated on September 17, 2022

Comments

timday over 1 year

I'm looking for something (prereably FOSS software) which can take an animation with N images as input, and which will output an animation with M frames, where M is in the range 2N to 5N or so.

I believe the general technique is called "temporal upsampling" or possibly "inbetweening" (or "'tweening" for short). Note that it does need to make some effort to do motion tracking of things in the scene ("optical flow"); just fading ("dissolve") between keyframes isn't going to cut it. Googling "temporal upsampling" turns up any number of papers on the subject, but I've yet to discover any code/software (a gstreamer plugin would be perfect) I can just use to try the technique out.

Any suggestions ?

Additional information: "Motion interpolation" seems to be the more widely used name in the TV world for what I'm looking for. See this and the video here too. Interestingly, some cinephiles seem to hate what these technologies do to 24FPS film: e.g a comment "makes Gone With The Wind look like a soap opera shot on video".
Hugh Allen over 13 years

Experimental? Some TV sets do this (in real time!) to make 24/25/30 FPS video look smoother. The results sometimes even look good.
Hans Kilian over 13 years

@Hugh Allen - I don't think any TV set does a lot of motion tracking/optical flow to get "in-between" images in their usual lacklustre hardware. To be honest, even with recent SIGGRAPH entries the motion tracking technology is still not yet very mature...
Hugh Allen over 13 years

They must be doing motion tracking. The best you can do to interpolate frames otherwise is just to fade, which would only make motion look blurry.
timday over 13 years

I was also under the general impression the tech was used in some (many?) TV sets. Certainly it's a computationally hard problem; I seem to remember it being touted as a reason for putting Cell processors in some models. I've have updated the question with some relevant links. Given the TV manufacturers seem to have gotten this to work reasonably well, I don't feel I can accept this answer as is.
Hans Kilian over 13 years

This depend largely on how good you think the current implementation is. Computationally the algorithm is hard enough, and even on the frontline of computer graphics research, tracking objects with no markers manually placed is still an area of development. Personally, I don't see algorithms on the TV sets being good. (You can look at the advertisements and most ad showed simulated images only - which wouldn't be the case if the technology were that good.)
timday over 13 years

Thanks! That's exactly the sort of thing I was hoping to get hold of!