Drawing the Difference of two Images to Canvas

Question

I am trying to draw the difference of two Images I have to the Canvas in Flutter and what I really mean is that I only want to draw what is different (you can see what I expect to see below).
It needs to happen synchronously, i.e. in a paint call!

In this answer, the two images are referred to as imageA and imageB.

Code

canvas.drawImage(
  imageA,
  offset,
  Paint(),
);
canvas.drawImage(
  imageB,
  offset,
  Paint()
    ..blendMode = BlendMode.difference,
);

Visualization

^{imageA and imageB drawn separately.}

^{Difference between imageA and imageB. You can see what I expect to see left and what I actually get using the code above right.
The gray/white background color in the left image is supposed to be transparent, i.e. alpha should equal zero where the two images are the same. Additionally, the black shadow around the red square in the expected image is an artifact and not what I want in my app.}

Problem

I am not saying that the expected image is what I expect to get from BlendMode.difference, but I want to know how I can get the expected output, i.e. how I achieve only drawing what is different between two images.
This means that I want to only render pixels from imageB that are different from imageA and otherwise nothing, i.e. alpha value 0.

Explanation

I will try to explain it a bit more clearly again:

• If the color is the same, draw transparent pixel (remove source pixel).

• If the color is different, draw destination pixel.

Solution

Currently flutter does not support what you are trying to do. Based on my research, flutter handles canvas paint on the native level. You can see this in action from this snippet:

  /// Fills the canvas with the given [Paint].
  ///
  /// To fill the canvas with a solid color and blend mode, consider
  /// [drawColor] instead.
  void drawPaint(Paint paint) {
    assert(paint != null);
    _drawPaint(paint._objects, paint._data);
  }
  void _drawPaint(List<dynamic> paintObjects, ByteData paintData) native 'Canvas_drawPaint';

This is from the canvas implementation on the latest version of flutter (1.9.2). Any pixel manipulation that you can do without async code must be done utilizing flutter BlendMode or ImageFilter which lacks personalized masks support.

The only way to do what you intend to do is to actually implement native code, which I don't really think you want to do. I spent the last 6 hours trying to figure out if there wasn't anything out of the official flutter API docs that would do what you want it to do, but the thing is, most of things in flutter are just intended to be async.

If you are going to venture into implementing this natively, why don't you ask to be implemented as a feature in the official flutter github? I mean, so much work, don't let it be wasted.

Just because I didn't wanted to leave empty handed, I have written precisely what you wanted with the image lib but using async:

import 'dart:typed_data';
import 'dart:ui' as ui;
import 'package:image/image.dart' as im;

import 'package:flutter/material.dart';

void main() => runApp(App());

class App extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MyApp();
  }
}

class MyApp extends StatefulWidget {
  @override
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  ByteBuffer imgBytes;
  double canvasHeight = 500;
  double canvasWidth = 300;

  ui.PictureRecorder recorder1;
  Canvas canvas1;

  ui.PictureRecorder recorder2;
  Canvas canvas2;

  ui.Picture picture1;
  ui.Picture picture2;

  @override
  void initState() {
    // Canvas1
    recorder1 = ui.PictureRecorder();
    canvas1 = Canvas(recorder1,
        Rect.fromPoints(Offset(0.0, 0.0), Offset(canvasWidth, canvasHeight)));

    // Draw red background
    var paint = Paint();
    paint.style = PaintingStyle.fill;
    paint.color = Colors.red;

    canvas1.drawRect(Offset(0, 0) & Size(canvasWidth, canvasHeight), paint);

    // Draw black square
    paint = Paint();
    paint.style = PaintingStyle.fill;
    paint.color = Colors.black;

    canvas1.drawRect(
        Offset(canvasWidth * 0.5 / 4, canvasHeight / 2) &
            Size(canvasHeight / 8, canvasHeight / 8),
        paint);

    picture1 = recorder1.endRecording();

    // Canvas2
    recorder2 = ui.PictureRecorder();
    canvas2 = Canvas(recorder2,
        Rect.fromPoints(Offset(0.0, 0.0), Offset(canvasWidth, canvasHeight)));

    // Draw red background
    paint = Paint();
    paint.style = PaintingStyle.fill;
    paint.color = Colors.red;

    canvas2.drawRect(Offset(0, 0) & Size(canvasWidth, canvasHeight), paint);

    // Draw blue square
    paint = Paint();
    paint.style = PaintingStyle.fill;
    paint.color = Colors.blue;

    canvas2.drawRect(
        Offset(canvasWidth * 2.5 / 4, canvasHeight / 2) &
            Size(canvasHeight / 8, canvasHeight / 8),
        paint);

    picture2 = recorder2.endRecording();

    (() async {
      ui.Image img1 =
          await picture1.toImage(canvasWidth.toInt(), canvasHeight.toInt());
      ByteData byteData1 =
          await img1.toByteData(format: ui.ImageByteFormat.png);
      im.Image decodedPng1 = im.decodePng(byteData1.buffer.asUint8List());

      ui.Image img2 =
          await picture2.toImage(canvasWidth.toInt(), canvasHeight.toInt());
      ByteData byteData2 =
          await img2.toByteData(format: ui.ImageByteFormat.png);
      im.Image decodedPng2 = im.decodePng(byteData2.buffer.asUint8List());

      for (int i = 0; i < canvasHeight; i += 1) {
        for (int j = 0; j < canvasWidth; j += 1) {
          int pixel1 = decodedPng1.getPixel(j, i);
          int r1 = pixel1 & 0xff;
          int g1 = (pixel1 >> 8) & 0xff;
          int b1 = (pixel1 >> 16) & 0xff;
          int a1 = (pixel1 >> 24) & 0xff;

          int pixel2 = decodedPng2.getPixel(j, i);
          int r2 = pixel2 & 0xff;
          int g2 = (pixel2 >> 8) & 0xff;
          int b2 = (pixel2 >> 16) & 0xff;
          int a2 = (pixel2 >> 24) & 0xff;

          if (r1 == r2 && g1 == g2 && b1 == b2 && a1 == a2) {
            // Draw black transparent
            decodedPng2.setPixel(j, i, 0);
          } else {
            // Leave as is
          }
        }
      }

      setState(() {
        imgBytes = Uint8List.fromList(im.encodePng(decodedPng2)).buffer;
      });
    })();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: Scaffold(
        appBar: AppBar(
          title: Text('Dummy'),
        ),
        body: imgBytes != null
            ? Center(
                child: Image.memory(
                  Uint8List.view(imgBytes),
                  width: canvasWidth,
                  height: canvasHeight,
                ),
              )
            : Container(),
      ),
    );
  }
}

Comments

[rmtmckenzie]

Does this need to happen in real time? It's possible to do something like what you're asking for, but you're going to need to use the image library which is quite slow. Essentially the issue at hand is that blend modes won't actually get to what you need - if you think about it, it makes sense because how would it decide what rgba(10, 0, 0, 255) - rgba(0, 10, 0, 255) should be without special logic.

[creativecreatorormaybenot]

@rmtmckenzie It needs to happen in real-time, i.e. in the paint method as I mentioned in the question (it does not accept asynchronous code). I can easily do the job manually asynchronously by comparing individual pixels and encoding a new image, but this will cause lag as it does not work in paint and is too slow (btw, I am a contributor to the image library, but I do not think that it offers functions for this).

[creativecreatorormaybenot]

@rmtmckenzie It is not difficult: if the color is the same, remove it (transparent pixels), otherwise draw the destination image. I think there could actually be a blend mode for this, but I could not find one that does it with color, only with shapes.

[szotp]

What if difference is tiny, for example src: #ffffff, dst: #fffffe, should it still draw destination pixel?

[creativecreatorormaybenot]

@szotp Yes! That is the idea. Always draw destination if not the same.

[Kent]

why did left change from black to red but right stayed blue? What is your thinking for the algorithm that would make this result?

[creativecreatorormaybenot]

@kent It is just the destination image (imageB). So when source and destination are different, destination should be drawn. This means that no complicated algorithm is even required. If it was different, then we would need some kind of complicated algorithm. The part that is black in imageA is red in imageB, hence, it is red in the result. A square that is red in imageA is red but blue in imageB, hence the result shows the blue square. The rest of the images is identical, hence, it is transparent in the result.

[Kent]

ok I think I get it. if(A = B) then Transparent else if (A != B) then B. This can be done by loading the raw pixels into ram and then doing a compare of each pixel. It might be expensive depending on the size of the image.

[creativecreatorormaybenot]

@kent I mentioned a few times here that this is what I do at the moment, but I need a synchronous way for performance reasons..

[Kent]

@creativecreatorormaybenot Just double checking you need an async way of doing this? because the way your previous message reads it seems like you are saying you want an sync way. Can you provide the code you have that is working? code shown is the actual (bottom right) correct?

[creativecreatorormaybenot]

@kent Where do I say I need an async way :D? I need a sync, i.e. synchronous, way. You can find example code in one of the answers (I do it like this as well). To the last question: incorrect. The bottom right shows an attempt of making it synchronous using BlendMode, but it obviously does not work. The bottom left shows what I currently have using async functions.

[Kent]

@creativecreatorormaybenot Thats why I was clarifying. Because usually when asking for more performance in code we use async not sync.

[Kent]

Let us continue this discussion in chat.

[creativecreatorormaybenot]

@kent You need sync in Flutter. paint does not allow real-time async code (obviously). It is pointless to discuss this in chat - you can inform yourself a little more here and here.

[creativecreatorormaybenot]

Thank you so much for your answer - the problem is that I have this already. It is hard for me to imagine that this is not possible using regular Canvas operations - I think that I will resort to my previous async implementation. I kinda have to use async code anyway because Flutter does not allow encoding images synchronously. See this issue.

[Nicolas Caous]

About the possibility, it is 100% possible, the thing is that no one bothered to implement this feature on the native canvas implementation of the flutter engine.

[creativecreatorormaybenot]

I mean non-native code. I might look into adjusting the engine after doing some other optimizations using dart:ffi, maybe.

[Nicolas Caous]

But anyway, why do you need to optimize this? Are you working with huge images? Also, there is the [flutter_ffmpeg][pub.dev/packages/flutter_ffmpeg] package that might be helpful.

[creativecreatorormaybenot]

Because I need to paint pixel data - I doubt ffmpeg can help me with that. You might not be aware of just how slow this is in Flutter. It is basically not usable - I cannot go further than 50k pixels if I want to encode pixels in real time. See the issue I linked before or this.

[Nicolas Caous]

I see now what you mean.

[Nicolas Caous]

So, I have been thinking about this problem for quite a while now. I remembered that you can achieve the desired results quite easily using opencv (some bitwise operations and a mask should be a piece of cake). Maybe you could create a opencv binary and statically link it to your dart:ffi plugin. The only down side to this is that a statically linked opencv c++ binary is huge (From past experience doing something similar, it used to be 30Mb). But I don't think that this is your only binary manipulation problem, given the nature of your work. Maybe opencv would be handy to you.

[creativecreatorormaybenot]

That will not work because you cannot decode an image synchronously in Dart. That is what the issue I linked is about.

[Nicolas Caous]

Do you mind sharing what is the functionality that you want to do? Are you getting the camera feed and showing it on the screen with the desired transformation? Or is your stream of images coming from another source? Maybe you could avoid using ui.Image altogether.

[Nicolas Caous]

Also, I want to apologize because, reading back our conversation, it seems that I'm not paying attention to what you are sending me. I have read all the links, it's just that english isn't my first language and sometimes communication gets a little fuzzy. I mean well :)

[creativecreatorormaybenot]

I am happy to share it with you: I have a list of pixels that can be edited in real-time (by the user dragging) and I need to manipulate the bytes, i.e. the pixels individually, so I cannot use lines. Consequently, I need to be able to draw this list of pixels as an image in real-time and the only reasonable fast way to do this that I know of is decoding to an Image an drawing that. I tried different approaches already (see here), but I could not find any more performant way.

[Nicolas Caous]

This list of pixels is a RGBA list encoded using a Uint8List or is it encoded in a file format? What i'm thinking is: since you have the raw pixels, you could write a personalized png encoder (using dart:ffi) that takes in the pixels in whatever format you have and spit out a buffer containing the png image that you can use in Image.memory(buffer). That way, this would all be sync (since ffi calls are sync) and you would only need a setState to update the image that the user is editing.

[creativecreatorormaybenot]

Wowow, I actually did not know about Image.memory - this might be huge. I store it in Uint8List and I planned to manipulate it using dart:ffi anyway as explained previously. Before I discovered decodeImageFromPixels, I actually encoded my own BMP by hand (ouch :D) and then used the async way to generate an Image from that (see an old question, the "Alternative" part). I will make sure to get back to you once I tried Image.memory. I might also try to use my old BMP conversion (which is sync) to test a slower Dart version of it first.

[creativecreatorormaybenot]

Well, I just realized that you are referring to the widget Image and its memory constructor. I was quite sceptical about the idea that I had not been aware of a synchronous way to construct dart:ui Images and I was correct because Image.memory is just a wrapper for asynchronously converting to a ui.Image and displaying that. I need a synchronous way to draw pixel data to a Canvas.

[Nicolas Caous]

O shoot, I did not knew that Image.memory was async. Damm

[Nicolas Caous]

The Image.memory boils down to String _instantiateImageCodec(Uint8List list, _Callback<Codec> callback, _ImageInfo imageInfo, int targetWidth, int targetHeight) native 'instantiateImageCodec'; which is async because it has a callback. :'(