Made a mobile app anyone can compose and play music by Flutter

I made a iOS and Android mobile app by Flutter. Anyone even doesn't have no knowledge about music can compose and play music by this app (only accompaniment).

音楽の知識がなくてもタップだけで演奏や作曲が出来るアプリを作りました（伴奏のみ）。合うコードだけ表示しているので適当なボタン操作でも曲が作れます。
外でふとコード進行が思い浮かんだ時にでも使ってみてください。 #Flutter
Android https://t.co/NESpvoRl5O
iOS https://t.co/DgvsuStjwC pic.twitter.com/4lOTlAdXP9
— だら🎄5/18～5/24開催Webサービス作るイベント (@dala00) May 13, 2020

I introduce technical things of this.

How to use

The basic usage is just tapping buttons. Music is automatically played with selected code and stroke way.

The most important thing of this app is 'easy'. I wanted to make anyone who don't have no knowledge of music or children can use so easily. I thinks it's succeeded because my 7 years old son is playing music and singing original cute song by this app.

Codes of buttons are minimum codes needed for one song. By this codes, you can play many songs in the world, and create. If you tap main, sub codes randomly, it becomes to a music automatically. You also can change pitched by up and down button at right bottom corner.

Save playing

If you could play good feeling song, you can save it by 'Save last playing' button. And you can play it always.

You don't need tap record button before playing. So you can save always and never lose opportunity.

Change tempo and stroke

You can change tempo by slider always.

And you can select stroke way from some patterns.

I want to make users can create original pattern.

How to make

I describe how to make this app.

Sound

Flutter can't sound. So I wrote native code and connect to Flutter side. There's a document.

Writing custom platform-specific code - Flutter

Many Flutter plugins are using this way. So usually everybody can make app by only Dart code.

Ways to connect natives

Specifically, write Dart code like this. Specify channel id to MethodChannel and send message to native side by invokeMethod.

class Player {
  static const platform =
      const MethodChannel('com.example.anyone-composer/midi');

  Future setTempo(int tempo) async {
    await platform.invokeMethod('setTempo', {'tempo': tempo});
  }
}

The following is ways to receive message on Android and iOS. This is simple too. You can use many data types like written on document.

Android side

Example on Android.

override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
    GeneratedPluginRegistrant.registerWith(flutterEngine);

    MethodChannel(flutterEngine.dartExecutor.binaryMessenger, "com.example.anyone-composer/midi").setMethodCallHandler {
        call, result ->
        if (call.method == "setTempo") {
            val tempo: Int? = call.argument<Int>("tempo")
            // nanikasuru
            result.success(true)
        }
    }
}

iOS side

Example on iOS.

  override func application(
    _ application: UIApplication,
    didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
  ) -> Bool {
    let controller : FlutterViewController = window?.rootViewController as! FlutterViewController
    let midiChannel = FlutterMethodChannel(name: "com.example.anyone-composer/midi",
                                              binaryMessenger: controller.binaryMessenger)
    midiChannel.setMethodCallHandler({
      (call: FlutterMethodCall, result: @escaping FlutterResult) -> Void in
      if call.method == "setTempo" {
        if let map = call.arguments as? Dictionary<String, Int>,
          let tempo = map["tempo"] {
          // nanikasuru
        }
        result(true)
      }
    })

    GeneratedPluginRegistrant.register(with: self)
    return super.application(application, didFinishLaunchingWithOptions: launchOptions)
  }

Playing sound

I used soundfont file (.sf2) and playing Midi. Actually, there is a Flutter package to play Midi.

flutter_midi/FlutterMidiPlugin.java at master · rodydavis/flutter_midi

But some needed function is not present. For example change patch, specify velocity and stop all sounds. So I referenced the plugin and wrote codes on my own. It was so difficult because few document and samples are found...

Notes

On Native side, you have to deal with threads. For example...

Execute heavy processing

Flutterのネイティブで重い処理をさせる方法（Android） - Crieit

Async processing with timer

Swift マルチスレッドでの同期処理（synchronized）

Draw playing animation

When playing, app animate where is playing in stroke. It's just a simple animation that a red line goes to right.

It's drawn by CustomPaint Widget of Flutter. You can draw anything by CustomPaint. CustomPaint uses a class overrides CustomPainter class.

The way to use is so simple.

child: CustomPaint(
  painter: StrokeAndLinePainter(
    context,
    Data.strokeSets[playerStore.strokeIndex],
    playerStore.tempo,
    timeStore.time,
  ),
),

The following is a example of actual painter class. Just using canvas.drawFooBar methods.

  @override
  void paint(Canvas canvas, Size size) {
    strokes.asMap().forEach((index, velocity) {
      final paint = Paint();
      paint.color = Theme.of(context).accentColor;
      final rect = Rect.fromLTWH(
        horizontalMargin +
            (size.width - horizontalMargin * 2) / strokes.length * index,
        margin,
        barWidth,
        size.height - margin * 2,
      );
      canvas.drawRect(rect, paint);
    });
  }

Animation

CustomPaint is custom painting Widget. Not for animation. So you need to
create animation processing by yourself.

I used timer and changing state. First of all, start timer.

      Timer.periodic(Duration(milliseconds: 16), _onTimer);

I'm omitting some codes, this is a updating code. Getting current time from native side and set to Dart side.

  Future _onTimer(Timer timer) async {
    final timeStore = Provider.of<TimeStore>(context, listen: false);
    final time = await _player.getTime();
    timeStore.setTime(time);
  }

This state is updated frequently. So I created dedicated store to redraw only animation part not to redraw all of page.

  Widget _buildStrokePaint() {
    return Consumer2<PlayerStore, TimeStore>(builder: (
      context,
      playerStore,
      timeStore,
      _,
    ) {
      return CustomPaint(
        painter: StrokeAndLinePainter(
          context,
          Data.strokeSets[playerStore.strokeIndex],
          playerStore.tempo,
          timeStore.time,
        ),
      );
    });
  }

I'ts good for battery. I think there are other ways.

Saving playing data

To saving data, I'm not saving actual sound data. Just saving the time when code changed and code. So very small data. Saving it to SQLite.

When playing it, just changing code by saved timing. So you can play sounds with different tempo and strokes.

This time, I wrote a following simple base model. And every models are override this base model.

abstract class Model {
  final String table = '';
  int id;

  Map<String, dynamic> toMap();

  Future insert() async {
    final db = await DbProvider.instance.database;
    id = await db.insert(table, toMap());
  }

  Future update() async {
    final db = await DbProvider.instance.database;
    await db.update(table, toMap(), where: 'id = ?', whereArgs: [id]);
  }

  Future delete() async {
    final db = await DbProvider.instance.database;
    await db.delete(table, where: 'id = ?', whereArgs: [id]);
  }
}

All models are override it and specify table name and write toMap method. Then you can deal data like following.

Record record = Record();
record.name = 'hoge';
await record.insert();

record.name = 'hoge2';
await record.update();

await record.delete();

This way is not good for large application, but I think it's good for small application like this time.

Future dev

I want to create such a futures. It's may not easy...

Record void
Create custom stroke patterns
Edit saved playings

I thought

I could use interesting things like native connection and CustomPaint.

This time, I thought that I should develop all by Native without Flutter. But I got large merit to enable to make UI and dealing with data common by Flutter. I could reduce man-hours on my personally development.

Thanks.

Blog