Accessing assets from C++ plugin through Flutter

430

1 - what format is your WAV file in? Is it 32 bit floats? Don't forget that WAV files have a header, so you should discard the first few tens of bytes (up to the data segment). Be sure that you start reading the audio data on a float boundary (which may not be a multiple of 4 if the header isn't). If necessary, just use a hex editor to ascertain the offset of the float data and start reading there. Or, truncate the header and rename your asset to song_cut.raw. Audacity should be able to produce a header-less raw audio file.

2 - What sample rate is your audio clip recorded at? Does that match the sample rate of the device? (Note that iOS devices are normally 44.1k, but Android devices are frequently 48k. When using an Android emulator on macOS, who knows what the reported sample rate will be! Expect pitch distortion if your rates don't match - or use a resampler. I think Oboe has one. Alternatively, the sample repo associated with the talk contains one you can use.)

3 - note that the timer interval is finely tuned (for demo purposes) to the approximate time taken to deliver 512 samples at the sound card rate. This might be ok for demos, but isn't for real life. Also, your wav file probably doesn't have exactly 512 samples in it. Either adjust your audio loop to 512 samples, or adjust the 512000 constant to match the number of samples in your loop.

4a - You aren't using the callback method yet, but you probably should as soon as possible. One method I've had success with is to use a lock-free circular buffer. The Oboe callback tries to empty the buffer, while the Dart timer routine tries to fill it. The bigger the buffer the less chance there is of an underflow, but the worse the latency.

4b - The ideal solution would be to have the Oboe callback call up into Dart, but I haven't found a way to do that as C->Dart calls must be on the main Dart thread, but the Oboe callbacks are surely on a high-priority IO thread.

Share:
430
Facundo Farall
Author by

Facundo Farall

Updated on November 29, 2022

Comments

  • Facundo Farall
    Facundo Farall 11 months

    I'm trying to use Google Oboe for a 3D audio processing app due to it's low latency. The app will have a C++ backend, which does the processing, and the frontend is done with Flutter. I'm running a couple of tests to see if it'll work but I'm having issues loading assets from Flutter to Oboe. I checked the example RhythmGame in Oboe's repo, done with Java, but couldn't quiet find a way of doing that straight from Dart to C++. The connection between front and backend is through dart::ffi

    Here's what I've tried so far. Based on the example published by Richard Heap here, I changed the noise variable from just a sine wave to a short fragment of a song in a wav file:

    class _MyAppState extends State<MyApp> {
      final stream = OboeStream();
    
      var noise = Float32List(512);
      Timer t;
    
      @override
      void initState() {
        super.initState();
        // for (var i = 0; i < noise.length; i++) {
        //   noise[i] = sin(8 * pi * i / noise.length);
        // }
    
        _loadSound();
      }
    
      void _loadSound() async {
        final ByteData data = await rootBundle.load('assets/song_cut.wav');
        noise = data.buffer.asFloat32List();
      }
    
    (...)
    

    Then this function in Dart calls the Dart wrapper of the native library:

    void start() {
        stream.start();
        var interval = (512000 / stream.getSampleRate()).floor() + 1;
        t = Timer.periodic(Duration(milliseconds: interval), (_) {
          stream.write(noise);
         });
      }
    

    The wrapper in Dart is:

    void write(Float32List original) {
        var length = original.length;
        var copy = allocate<Float>(count: length)
            ..asTypedList(length).setAll(0, original);
    
        FfiGoogleOboe()._streamWrite(_nativeInstance, copy, length);
        free(copy);
      }
    

    _streamWrite is the native function in C++:

    EXTERNC void stream_write(void* ptr, void* data, int32_t size) {
        auto stream = static_cast<OboeFfiStream*>(ptr);
        auto dataToWrite = static_cast<float*>(data);
        stream->write(dataToWrite, size);
    }
    
    void OboeFfiStream::write(float *data, int32_t size) {
        managedStream->write(data, size, 1000000);
    }
    

    Now I can hear the song but it comes out with too much distortion. When trying with the sine I could hear it too, but it also had some distortion. I'm not yet using the callback mode in Oboe, since I wanted to try if this worked first.

    • Richard Heap
      Richard Heap over 3 years
      youtu.be/X8JD8hHkBMc particularly from 23:00
    • Facundo Farall
      Facundo Farall over 3 years
      Richard, I saw your video a couple of days ago (and by the way, thank you for it, it's the only good example of using ffi I could find, great job), but what I'm trying to do now is replace the sine sound with an mp3 file I'm loading from the assets in pubspec.yaml
    • Richard Heap
      Richard Heap over 3 years
      You can improve your question by adding a lot more details. What you are trying to do, how, what you've tried, what doesn't work. Show your code.
  • Facundo Farall
    Facundo Farall over 3 years
    Can you debug in any way what you do in the C++ package? I'm getting some runtime errors from Oboe that I can't quiet understand, and a step by step debug would be amazing