Video file to image stream in flutter

Question

I have a video file in my assets folder (like mp4). I want to read that video file as an image stream, like an image stream in the camera package https://pub.dev/documentation/camera/latest/camera/CameraController/startImageStream.html

I want to pass the image to Tflite model.

Something like that:

controller = loadVideo('assets/example.mp4');  // how to do that?! 
controller.startImageStream((CameraImage img) {
          Tflite.runPoseNetOnFrame(
              bytesList: img.planes.map((plane) {
                return plane.bytes;
              }).toList(),
              imageHeight: img.height,
              imageWidth: img.width,
              numResults: 1,
              threshold: 0.1,
            ).then((recognitions) {
              // ...
            }
        }

I tried to search in the image_picker and video_player packages, but did not find anything.

Solution

What I did is to split the images and save them as files and then send it to the tf model. It's not fast (because you make file and then read it) but it is working, tested on android and it should work on IOS :)

I made a PR in the package flutter_export_video_frame, now it have a function in the name of exportImagesFromFile. It is Returning a Stream of images from video file as Stream<File>.

To get the video file from the asset folder I use the following function:

import 'dart:async';
import 'dart:io';
import 'package:path_provider/path_provider.dart';
import 'package:flutter/services.dart' show rootBundle;
import 'package:path/path.dart';

Future<File> getImageFileFromAssets(String path) async {
  final byteData = await rootBundle.load('assets/$path');
  final fileName = basename(path);

  final file = new File('${(await getTemporaryDirectory()).path}/$fileName');
  await file.writeAsBytes(byteData.buffer
      .asUint8List(byteData.offsetInBytes, byteData.lengthInBytes));

  return file;
}

And then to move it to the posenet/model, there is a function that gets File as input:

var results = await Tflite.runPoseNetOnImage(
      path: path,
      numResults: 1,
    );

One possible improvement to this method is to make it works only on the RAM (after you read the video, of course)

Comments

[jacob galam]

export_video_frame I tried and it's making a file for each image and it's very slow because of that. video_thumbnail I think it has returned one image and it also save as file if I not mistaken. Can you show me a code that read a full video to frames and pass it a tflite model (posenet)?

[AVEbrahimi]

I would do it if I had enough time, but I am sorry.