Rendering a Canvas to an Image in a way that doesn't lock up Flutter UI

Question

I need to draw some images in Flutter using geometric primitives, to both show in-app and cache for later use. What I'm doing right now is something similar to this:

import 'dart:ui';

final imageDimension = 600;
final recorder = PictureRecorder();
final canvas = Canvas(
        recorder,
        Rect.fromPoints(const Offset(0.0, 0.0),
            Offset(imageDimension.toDouble(), imageDimension.toDouble())));

/// tens of thousands of canvas operations here:
// canvas.drawCircle(...);
// canvas.drawLine(...);
final picture = recorder.endRecording();

// the following call can take ~10s
final image = await picture.toImage(imageDimension, imageDimension);
final dataBytes = await image.toByteData(format: ImageByteFormat.png);

Here's an example of the outcome:

I know the image operations in this amount are heavy and I don't mind them taking some time. The problem is since they're CPU bound, they lock up the UI (even though they are async and outside of any widget build methods). There doesn't seem to be any way to break the picture.toImage() call into smaller batches to make the UI more responsive.

My question is: Is there a way in Flutter to render a complex image built from geometric primitives in a way that doesn't impact the UI responsiveness?

My first idea was to do the heavy calculation inside a compute() isolate, but that won't work since the calculations use some native code:

E/flutter (20500): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object extends NativeWrapper - Library:'dart:ui' Class: Picture)
E/flutter (20500): #0      spawnFunction (dart:_internal-patch/internal_patch.dart:190:54)
E/flutter (20500): #1      Isolate.spawn (dart:isolate-patch/isolate_patch.dart:362:7)
E/flutter (20500): #2      compute (package:flutter/src/foundation/_isolates_io.dart:22:41)
E/flutter (20500): #3      PatternRenderer.renderImage (package:bastono/services/pattern_renderer.dart:95:32)
E/flutter (20500): #4      TrackRenderService.getTrackRenderImage (package:bastono/services/track_render_service.dart:111:49)
E/flutter (20500): <asynchronous suspension>

I think there is an alternative approach with CustomPaint painting just a couple of elements every frame and after it's all done screenshot it somehow using RenderRepaintBoundary.toImage(), but there's a couple of problems with this approach:

• It's much more complicated and relies on using heuristics to choose the amount of elements that can be rendered to the canvas per frame (I think?)
• I think the Widget would need to be visible in order for it to get rendered so I couldn't really use it to render images in the background (?)
• I'm not exactly sure how I'd get RenderRepaintBoundary for the widget I use for rendering.

Edit: it seems like the screenshot package allows for taking screenshots that are not rendered on the screen. I don't know about its performance characteristics yet, but it seems like it could work together with the CustomPaint class. It still feels like a very convoluted workaround though, I'd be happy to see other options.

Solution 1

Probably the best approach is moving rendering from dart:ui to a library that can be used in the compute isolate.

The image package is written in pure Dart, so it should work well and it does support the primitive operations I need.

I'm currently using the Canvas solution from the other answer, and it works for the most part, but I will switch to this soon.

Solution 2

The solution I'm going with for now is based on pskink's suggestion - periodically rendering the canvas to an image and bootstrapping the new canvas with the rendered image for the next batch of operations.

Since the graphics operations take in the order of ~100ms I decided not to go with scheduleTask, but delaying the processing after every batch to give the other tasks a chance to execute. It is a bit hacky, but should be good enough for now.

Here's a simplified code of the solution:

import 'package:flutter/material.dart';
import 'package:dartx/dartx.dart';
import 'dart:ui' as dart_ui;

  Future<List<Offset>> fetchWaypoints() async {
    // some logic here
  }

  Future<Uint8List> renderPreview(String imageName) async {
    const imageDimension = 1080;
    dart_ui.PictureRecorder recorder = dart_ui.PictureRecorder();
    Canvas canvas = Canvas(recorder);
    final paint = Paint()..color = Colors.black;

    canvas.drawCircle(Offset.zero, 42.0, paint);

    final waypoints = await fetchWaypoints();
    dart_ui.Image intermediateImage =
        await recorder.endRecording().toImage(imageDimension, imageDimension);

    for (List<Offset> chunk in waypoints.chunked(100)) {
      recorder = dart_ui.PictureRecorder();
      canvas = Canvas(recorder);
      canvas.drawImage(intermediateImage, Offset.zero, Paint());

      for (Offset offset in chunk) {
        canvas.drawCircle(offset, 1.0, paint);
      }

      intermediateImage =
          await recorder.endRecording().toImage(imageDimension, imageDimension);

      // give the other tasks a chance to execute
      await Future.delayed(Duration.zero);
    }
    final byteData =
        await intermediateImage.toByteData(format: dart_ui.ImageByteFormat.png);
    return Uint8List.view(byteData!.buffer);
  }

Comments

[pskink]

maybe scheduleTask could be helpful? the docs say: "Tasks will be executed between frames, in priority order, excluding tasks that are skipped by the current schedulingStrategy. Tasks should be short (as in, up to a millisecond), so as to not cause the regular frame callbacks to get delayed." - of course as they say, it is your responsibility to make sure they are short enough to fit between frames

[nietaki]

It definitely seems like sheduleTask could be helpful in similar situations - thanks for that. In this particular case the picture.toImage() is done all in one go and I don't have an idea how to break it down into smaller subtasks...

[pskink]

i meant if you have your 20k drawCircles to be drawn schedule a task first drawing tmp image, followed by 1000 first circles for example, followed by recorder.endRecording().toImage - when toImage completes assign its image to tmp mage and schedule a new task or save the image somewhere if there is no new task to schedule

[nietaki]

Oh, the part I was missing was rendering the intermediate Images and loading them to a new canvas with canvas.drawImage(image!, Offset.zero, Paint());. This coupled with sheduleTask could do the trick - I'll give it a try!

[nietaki]

I played around with the options over the weekend, you can see what I ended up with in my answer

[pskink]

"// the following call can take ~10s" - out of curiosity: does it take now (with one frame delay) also ~10s, or more or less?

[nietaki]

It is slower, by a factor of 2-3, but the app is responsive in the meantime, which is good. Also, check out the other answer, it seems like the best thing to do is ditch dart:ui altogether if you don't need any of its advanced features.

[pskink]

maybe it is because of that 16ms delay - actually you dont need it - you can also do await Future(() {}) or await Future.delayed(Duration.zero) or similar stuff

[nietaki]

Good point with the delay, I'm glad to see these async actions don't compete with the ui rendering too much. Still, even with Duration.zero the rendering of a random example takes up to a minute now, I was underestimating the time increase.

[pskink]

hmmm, i run my initial code (this one with scheduleTask) and 100 small toImage were faster than one big toImage call but maybe it is due some bugs in toImage linux implementation

[nietaki]

Let us continue this discussion in chat.

[nietaki]

I hoped that would work and moved the PictureRecorder and Canvas constructors inside the isolate, but on Android it now causes a different, but related, error: Unhandled Exception: Exception: UI actions are only available on root isolate. - PictureRecorder._constructor (dart:ui/painting.dart:4963:59) It seems like dart:ui being unusable outside of the root isolate is the expected behaviour right now based on this github issue.

[CrimsonFoot]

@nietaki Wow I get it... That's really bad. I think maybe you should draw vectors in primitive types (i.e. String, List<int> ...) I'll be thinking about it. Sorry not to be helpful man.

[nietaki]

No worries and thanks for the suggestion - it really helped me understand what the main problem is :)