Image sequence to video stream?

Question

Like many people already seem to have (there are several threads on this subject here) I am looking for ways to create video from a sequence of images.

I want to implement my functionality in C#!

Here is what I wan't to do:

/*Pseudo code*/
void CreateVideo(List<Image> imageSequence, long durationOfEachImageMs, string outputVideoFileName, string outputFormat)
{
    // Info: imageSequence.Count will be > 30 000 images
    // Info: durationOfEachImageMs will be < 300 ms

    if (outputFormat = "mpeg")
    {
    }
    else if (outputFormat = "avi")
    {      
    }
    else
    {
    }

    //Save video file do disk
}

I know there's a project called Splicer (http://splicer.codeplex.com/) but I can't find suitable documentation or clear examples that I can follow (these are the examples that I found).

The closest I want to do, which I find here on CodePlex is this: How can I create a video from a directory of images in C#?

I have also read a few threads about ffmpeg (for example this: C# and FFmpeg preferably without shell commands? and this: convert image sequence using ffmpeg) but I find no one to help me with my problem and I don't think ffmpeg-command-line-style is the best solution for me (because of the amount of images).

I believe that I can use the Splicer-project in some way (?).

In my case, it is about about > 30 000 images where each image should be displayed for about 200 ms (in the videostream that I want to create).

(What the video is about? Plants growing ...)

Can anyone help me complete my function?

Solution 1

Well, this answer comes a bit late, but since I have noticed some activity with my original question lately (and the fact that there was not provided a working solution) I would like to give you what finally worked for me.

I'll split my answer into three parts:

• Background
• Problem
• Solution

---

Background

(this section is not important for the solution)

My original problem was that I had a lot of images (i.e. a huge amount), images that were individually stored in a database as byte arrays. I wanted to make a video sequence with all these images.

My equipment setup was something like this general drawing:

The images depicted growing tomato plants in different states. All images were taken every 1 minute under daytime.

/*pseudo code for taking and storing images*/
while (true)
{
    if (daylight)
    {
        //get an image from the camera
        //store the image as byte array to db
    }
    //wait 1 min
}

I had a very simple db for storing images, there were only one table (the table ImageSet) in it:

---

Problem

I had read many articles about ffmpeg (please see my original question) but I couldn't find any on how to go from a collection of images to a video.

---

Solution

Finally, I got a working solution! The main part of it comes from the open source project AForge.NET. In short, you could say that AForge.NET is a computer vision and artificial intelligence library in C#. (If you want a copy of the framework, just grab it from http://www.aforgenet.com/)

In AForge.NET, there is this VideoFileWriter class (a class for writing videofiles with help of ffmpeg). This did almost all of the work. (There is also a very good example here)

This is the final class (reduced) which I used to fetch and convert image data into a video from my image database:

public class MovieMaker
{

    public void Start()
    {
        var startDate = DateTime.Parse("12 Mar 2012");
        var endDate = DateTime.Parse("13 Aug 2012");

        CreateMovie(startDate, endDate);
    }    
    

    /*THIS CODE BLOCK IS COPIED*/

    public Bitmap ToBitmap(byte[] byteArrayIn)
    {
        var ms = new System.IO.MemoryStream(byteArrayIn);
        var returnImage = System.Drawing.Image.FromStream(ms);
        var bitmap = new System.Drawing.Bitmap(returnImage);

        return bitmap;
    }

    public Bitmap ReduceBitmap(Bitmap original, int reducedWidth, int reducedHeight)
    {
        var reduced = new Bitmap(reducedWidth, reducedHeight);
        using (var dc = Graphics.FromImage(reduced))
        {
            // you might want to change properties like
            dc.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.HighQualityBicubic;
            dc.DrawImage(original, new Rectangle(0, 0, reducedWidth, reducedHeight), new Rectangle(0, 0, original.Width, original.Height), GraphicsUnit.Pixel);
        }

        return reduced;
    }

    /*END OF COPIED CODE BLOCK*/


    private void CreateMovie(DateTime startDate, DateTime endDate)
    {
        int width = 320;
        int height = 240;
        var framRate = 200;

        using (var container = new ImageEntitiesContainer())
        {
            //a LINQ-query for getting the desired images
            var query = from d in container.ImageSet
                        where d.Date >= startDate && d.Date <= endDate
                        select d;

            // create instance of video writer
            using (var vFWriter = new VideoFileWriter())
            {
                // create new video file
                vFWriter.Open("nameOfMyVideoFile.avi", width, height, framRate, VideoCodec.Raw);

                var imageEntities = query.ToList();

                //loop throught all images in the collection
                foreach (var imageEntity in imageEntities)
                {
                    //what's the current image data?
                    var imageByteArray = imageEntity.Data;
                    var bmp = ToBitmap(imageByteArray);
                    var bmpReduced = ReduceBitmap(bmp, width, height);

                    vFWriter.WriteVideoFrame(bmpReduced);
                }
                vFWriter.Close();
            }
        }

    }
}

Update 2013-11-29 (how to) (Hope this is what you asked for @Kiquenet?)

1. Download AForge.NET Framework from the downloads page (Download full ZIP archive and you will find many interesting Visual Studio solutions with projects, like Video, in the AForge.NET Framework-2.2.5\Samples folder...)
2. Namespace: AForge.Video.FFMPEG (from the documentation)
3. Assembly: AForge.Video.FFMPEG (in AForge.Video.FFMPEG.dll) (from the documentation) (you can find this AForge.Video.FFMPEG.dll in the AForge.NET Framework-2.2.5\Release folder)

If you want to create your own solution, make sure you have a reference to AForge.Video.FFMPEG.dll in your project. Then it should be easy to use the VideoFileWriter class. If you follow the link to the class you will find a very good (and simple example). In the code, they are feeding the VideoFileWriter with Bitmap image in a for-loop

---

Solution 2

I found this code in the slicer samples, looks pretty close to to what you want:

string outputFile = "FadeBetweenImages.wmv";
using (ITimeline timeline = new DefaultTimeline())
{
    IGroup group = timeline.AddVideoGroup(32, 160, 100);
    ITrack videoTrack = group.AddTrack();
    IClip clip1 = videoTrack.AddImage("image1.jpg", 0, 2); // play first image for a little while
    IClip clip2 = videoTrack.AddImage("image2.jpg", 0, 2); // and the next
    IClip clip3 = videoTrack.AddImage("image3.jpg", 0, 2); // and finally the last
    IClip clip4 = videoTrack.AddImage("image4.jpg", 0, 2); // and finally the last
}

  double halfDuration = 0.5;

  // fade out and back in
  group.AddTransition(clip2.Offset - halfDuration, halfDuration, StandardTransitions.CreateFade(), true);
  group.AddTransition(clip2.Offset, halfDuration, StandardTransitions.CreateFade(), false);

  // again
  group.AddTransition(clip3.Offset - halfDuration, halfDuration, StandardTransitions.CreateFade(), true);
  group.AddTransition(clip3.Offset, halfDuration, StandardTransitions.CreateFade(), false);

  // and again
  group.AddTransition(clip4.Offset - halfDuration, halfDuration, StandardTransitions.CreateFade(), true);
  group.AddTransition(clip4.Offset, halfDuration, StandardTransitions.CreateFade(), false);

  // add some audio
  ITrack audioTrack = timeline.AddAudioGroup().AddTrack();

  IClip audio =
     audioTrack.AddAudio("testinput.wav", 0, videoTrack.Duration);

  // create an audio envelope effect, this will:
  // fade the audio from 0% to 100% in 1 second.
  // play at full volume until 1 second before the end of the track
  // fade back out to 0% volume
  audioTrack.AddEffect(0, audio.Duration,
                 StandardEffects.CreateAudioEnvelope(1.0, 1.0, 1.0, audio.Duration));

  // render our slideshow out to a windows media file
  using (
     IRenderer renderer =
        new WindowsMediaRenderer(timeline, outputFile, WindowsMediaProfiles.HighQualityVideo))
  {
     renderer.Render();
  }
}

Solution 3

I could not manage to get the above example to work. However I did find another library that works amazingly well once. Try via NuGet "accord.extensions.imaging.io", then I wrote the following little function:

    private void makeAvi(string imageInputfolderName, string outVideoFileName, float fps = 12.0f, string imgSearchPattern = "*.png")
    {   // reads all images in folder 
        VideoWriter w = new VideoWriter(outVideoFileName, 
            new Accord.Extensions.Size(480, 640), fps, true);
        Accord.Extensions.Imaging.ImageDirectoryReader ir = 
            new ImageDirectoryReader(imageInputfolderName, imgSearchPattern);
        while (ir.Position < ir.Length)
        {
            IImage i = ir.Read();
            w.Write(i);
        }
        w.Close();
    }

It reads all images from a folder and makes a video out of them.

If you want to make it nicer you could probably read the image dimensions instead of hard coding, but you got the point.

Solution 4

The FFMediaToolkit is a good solution in 2020, with .NET Core support.

https://github.com/radek-k/FFMediaToolkit

FFMediaToolkit is a cross-platform .NET Standard library for creating and reading video files. It uses native FFmpeg libraries by the FFmpeg.Autogen bindings.

The README of the library has a nice example for the question asked.

// You can set there codec, bitrate, frame rate and many other options.
var settings = new VideoEncoderSettings(width: 1920, height: 1080, framerate: 30, codec: VideoCodec.H264);
settings.EncoderPreset = EncoderPreset.Fast;
settings.CRF = 17;
var file = MediaBuilder.CreateContainer(@"C:\videos\example.mp4").WithVideo(settings).Create();
while(file.Video.FramesCount < 300)
{
    file.Video.AddFrame(/*Your code*/);
}
file.Dispose(); // MediaOutput ("file" variable) must be disposed when encoding is completed. You can use `using() { }` block instead.

Solution 5

This is a solution for creating a video from an image sequence using Visual Studio using C#.

My starting point was "Hauns TM"'s answer below but my requirements were more basic than theirs so this solution might be more appropriated for less advanced users ( like myself )

Libraries:

using System;
using System.IO;
using System.Drawing;
using Accord.Video.FFMPEG;

You can get the FFMPEG libarary by searching for FFMPEG in "Tools -> NuGet Package Manager -> Manage NuGet Packages for a Solution..."

The variables that I passed into the function are:

• outputFileName = "C://outputFolder//outputMovie.avi"
• inputImageSequence = ["C://inputFolder//image_001.avi", "C://inputFolder//image_002.avi", "C://inputFolder//image_003.avi", "C://inputFolder//image_004.avi"]

Function:

private void videoMaker( string outputFileName , string[] inputImageSequence)
{
  int width = 1920;
  int height = 1080;
  var framRate = 25;

  using (var vFWriter = new VideoFileWriter())
  {
    // create new video file
    vFWriter.Open(outputFileName, width, height, framRate, VideoCodec.Raw);

    foreach (var imageLocation in inputImageSequence)
    {
      Bitmap imageFrame = System.Drawing.Image.FromFile(imageLocation) as Bitmap;
      vFWriter.WriteVideoFrame(imageFrame);
    }
    vFWriter.Close();
  }
}

View more solutions

Comments

[Hauns TM]

Thank you very much! I've seen that one already but there is just one thing that I can't figure out. I have this important prerequisite with duration, durationOfEachImageMs < 300 ms. Where can I set this very short time in accordance to the above example?

[Adam]

the last parameter in AddImage is a double, have you tried .3 instead of 2?

[Hauns TM]

Well, that probably works, but now I run into another issue COMException was caught Details: System.Runtime.InteropServices.COMException was caught Message=Access is denied. Source=DirectShowLib-2005 ErrorCode=-2147024891 StackTrace: at DirectShowLib.DES.DESError.ThrowExceptionForHR(Int32 hr) at Splicer.Renderer.AbstractRenderer.StartRender() at Splicer.Renderer.AbstractRenderer.BeginRender(AsyncCallback callback, Object state) at Splicer.Renderer.AbstractRenderer.Render() Have you any idea where to go from here?

[Daniel Mošmondor]

Are you sure that Splicer framework will allow 30k IClips created?

[Hauns TM]

If anyone is interested, the final resulting video, with "sound effects", can be found on Youtube. Please consider this comment as a demo of the solution only. :o)

[Ivan Kochurkin]

Why did you not answer yesterday? I just needed in it :) But it's great, thank you anyway.

[Ivan Kochurkin]

Is it possible to add audio to created video with aforge library in present time?

[Hauns TM]

To be honest, I haven't tried that. When I created my Youtube clip (se link above) I used a friend's video editing software + my home made avi-file and some different sound-clip files. If you do not have any other video editing software, I think that you can use VideoLAN Movie Creator - VLMC. Please let me know if it works. :-)

[Ivan Kochurkin]

Now I'am using splicer and it's working, but I don't think that it is a best solution, because of this library is not developing, too old and depend on DrirectShow.

[Kiquenet]

any full source code sample ? using Namespaces ? which libraries in References ?

[Kiquenet]

Sample in AForge works for me. I use your code, I use JPG files, and I use this var imageByteArray = imageToByteArray(Image.FromFile(img)); Video generate for me is wrong, not view. Any suggestions ? maybe Image Format ?

[Hauns TM]

Unfortunately, I have no idea. No harm but I believe that it would be easier to answer you if you formulate your own question with your specific problem. Please show your code and output

[Kiquenet]

Windows Media Player is a prerrequisite. It's needed install it before.

[dajuric]

the library is now called DotImaging

[Jack]

Thank you so much for that answer! just a note: If a not found reference run-time error from AForge.Video.FFMPEG.dll dll throw, you need to copy all the dlls on C:\Program Files (x86)\AForge.NET\Framework\Externals\ffmpeg\bin folder to your output directory.

[Admin]

Is there a x64 version of this method? Using Aforge.Video.FFMPEG in a required x64 project, VS says the assembly is mismatched (paraphrased).

[Wu Zhenwei]

I think you probably should use the constructor of Bitmap to create a Bitmap fromStream, and also Dispose the Bitmap objects you created in the for loop manually, instead of waiting for GC to do it.

[Pranay]

@dajuric is there an option to integrate audio (text to speech converted) to the video in parallel using the above library?

[dajuric]

@Pranay As the library depends on the OpenCV video API which does not have such options, the answer is no. But, you can generate a video and then invoke ffmpeg to interleave the video with the audio. Notice that, if you wanted to stream a video while it is being constructed you would need to slice it into chunks and then stream it using DASH.

[malt_man]

I've installed Accord through NuGet but I can't get this code to work. It's not recognizing the VideoWriter class.

[TheLegendaryCopyCoder]

Note that the AForge.Video.FFMPEG.dll binary is licensed under GPL v3.

[choufucai]

Did the AForge support mobile platform such as android and ios, I have imported AForge in Unity project, and build as android, it suffer lots of errors.

[Hauns TM]

I really can't tell. As far as I know, AForge is not .NET core