Every day is an API day

If you practice the piano everyday, you want to record your play and share it with your teacher, but you may not want to store it in your mobile or PC.

That's where API comes in!

This "Piano Lesson" is an application that captures video and send the data to AWS S3 bucket via MuleSoft API with WebSocket connection.

And,

This application completely runs on MuleSoft runtime, yes, self-contained! 1 CloudHub worker works as a web server, api server, and database server. You don't need any other server!!

Resources

Demo@CloudHub

https://mule-worker-skd-piano-lesson.us-e2.cloudhub.io:8082/login
My apologies for the self-signed certificate. To use the demo, you have to ignore the certificate error and trust it temporarily on your browser. I don't have CA signed certificates or Dedicated Load-Balancer for this project...
You can use this credentials for the demo:
- username: saki
- password: saki
Be careful for recordings... since we share the account above, all users can see your recording. If you want to delete your recording, you have to override the recording for the day and piece by re-recording.
Please, please go easy on the S3 data storage... this application limits the size of 1 recording, but... please be kind!

GitHub

https://github.com/ssakoda/piano-lesson
Setup guide and Operation Manual is on the GitHub README.md

Architecture and Design

Architecture

This application uses 2 services - "MuleSoft CloudHub" and "AWS S3". AWS S3 provides only the file storage. All other functions are provided from a MuleSoft CloudHub runtime.

MuleSoft

The MuleSoft runtime application mainly consists of 4 flow configuration files - web, api, database and data. web and api contain "HTTP Listener" to expose functions to user devices, and others are just sets of internal logics.

By providing all endpoints from the same hostname and port, we can safely use "cookies" (with "SameSite=Strict) to implement "session" function (even between "https://" access and "wss://" access!)

web flow provides SPA (Single-Page-Application) resources powered by React framework. This SPA consists of only 2 resources (and favicon.ico) - HTML file and Javascript file. Since it's just 2 files, MuleSoft runtime can responses to the specific GET call by sending out the content of each HTML file and Javascript file by using File component and HTTP Listener.

AWS S3

AWS S3 provides 2 useful functions - "Multipart Upload" and "Object Presigned URL". "Multipart Upload" is very useful for this kind of applications, because the application itself doesn't have to buffer more than 5MB. The application just have to upload minimum 5MB chunks and keep ETags as the return of the upload until the end of the stream. "Object Presigned URL" is to download the stored data file to play the video on the application.

These can be called from client application side, but that causes sever security risks - exposing AWS credentials in the HTMT/js. That's why this client application calls MuleSoft API in the same runtime (hostname/port) with sharing the JWT token in the cookie and the API calls AWS S3 API with AWS credentials.

Considerations

Mobile Browser (iOS Safari)

This application doesn't work on iOS (Safari). Currently, iOS Safari doesn't support "Media Recorder" (By enabling Experimental Features "MediaRecorder", it seems work, but still it just supports "mp4" without buffering). Needs to handle "Media Stream" directly...

Database vs ObjectStore

This application uses ObjectStore as a database, but it just stores objects, has limitations like TTL, not suitable for production use as database against real database. For caching and temporary memory usages, Object Store is good, but not for transactional data storage.

Web Server...

Well... it's nice to have a modern web server (and CA signed certificate)

Design

Flows

web (skd-piano-lesson-web.xml)

This flow provides "SPA resources" and "WebSocket endpoint"

SPA resources GET "/", "/index.html", "/index.js", "/login", "/login/login.html" and "/login/login.js" (and favicon.icon) responses the content of each file in src/main/resources. POST "/login" responses 302 with destination "/" only if username/password are qualified.
WebSocket endpoint "/ws/data" with wss protocol acts as a WebSocket server with WebSocket component. According to https://docs.mulesoft.com/websockets-connector/1.0/websockets-connector-cloudhub, Shared Load Balancer (SLB) doesn't propagate ws/wss trafic (even though the hostname and port are the exact same as https!). Since I don't have Dedicated Load Balancer(DLB), I had to use "mule-worker" hostname and 8082 port for the demo (and that causes certification problem, sorry)

api (skd-piano-lesson.xml)

This flow provides server-side functions like database access, AWS S3 access. This uses API-kit and before the router, session (JWT in cookie) is always validated. It's good to have "web-and-API-server"!

database (skd-piano-lesson-db.xml)

This flow provides database access internally called by web and api flows via Flow Reference component. This function can be exposed to Mongo DB server or something like that, but this application seeks for the limit of "Object Store" capability (which is not recommended for the production use...)

data (skd-piano-lesson-data.xml)

This flow provides WebSocket data handling process called by web flow. The 1st message from WebSocket client is attributes like recording Id, and this process stores the mapping of "websocket key (sec-websocket-key header value)" and "recording id" in Object Store, so that the 2nd message (binary video data) can be linked with recording id via websocket key.

init (skd-piano-lesson-init.xml)

This flow has only "scheduler" to create the initial data (admin credentials). By executing this scheduler from Runtime Manager, initial data load can be achieved.

Implementation

Session and the validation

Since MuleSoft runtime is not a real web server, I had to implement a simple session function by myself, but it's not difficult as sound. The keys are "JWT" and "Set-Cookie header".

JWT (Json Web Token) is a compact URL-safe means of representing claims to be transferred between two parties. So I used this to encrypt user id with a secret string, and set the encrypted string in the client's browser cookie at the end of the successful login process.

MuleSoft provides encryption functions, but to handle the encryption in a clear way for me, I used Java implementation with "com.auth0.jwt.JWT" package (groupId = "com.auth0", artifactId = "java-jwt"). This is just easy like this:

To set the string to client's cookie, it just has to set the following string in "Set-Cookie" header of the response of the login request (after successful login process, it leads the user to "/", so "Location" header and status 302 is set in HTTP Listener response.

To validate the session, it just has to check the "cookie" request header and validate the session id string format, and validate the JWT.

This validation logic can be set not only API process, but also static resource access like "/index.html", and WebSocket server process, because they use exact same hostname and port number!

WebSocket

Originally, I used "socket.io" (https://socket.io/docs/v3/index.html) to implement the WebSocket client with Node.js, but surprisingly and unfortunately, it seems "NOT" compatible with MuleSoft WebSocket component.

What Socket.IO is not

Socket.IO is NOT a WebSocket implementation. Although Socket.IO indeed uses WebSocket as a transport when possible, it adds additional metadata to each packet. That is why a WebSocket client will not be able to successfully connect to a Socket.IO server, and a Socket.IO client will not be able to connect to a plain WebSocket server either.
https://socket.io/docs/v3/index.html

Ok... then, I had to use browser native WebSocket API. "socket.io" is useful, because it can send multiple kinds of data at once, like binary data and attributes as followings:

const socket = io('ws://localhost:3000');

socket.on('connect', () => {
  // either with send()
  socket.send('Hello!');

  // or with emit() and custom event names
  socket.emit('salutations', 'Hello!', { 'mr': 'john' }, Uint8Array.from([1, 2, 3, 4]));
});

// handle the event sent with socket.emit()
socket.on('greetings', (elem1, elem2, elem3) => {
  console.log(elem1, elem2, elem3);
});

https://socket.io/docs/v3/index.html

But, native WebSocket API doesn't allow this kind of "send". I tried to use my own HTTP header, but native WebSocket API or MuleSoft WebSocket component doesn't allow to access to the header. I had to send "recording id" which is previously generated by API call "/api/start", and binary data with linking each other.

Then, I decided to send "recording id" as the 1st websocket message, then send binary data as the following messages. On the server (MuleSoft) side, to save the workload, checked the mediaType and stored mapping. The recording id is sent with mediaType "text/plain", binary data with "application/octet-stream". Once "text/plan" data is received, it stores the mapping of "atteributes.headers.sec-websocket-key" and payload (recording id). If "application/octet-stream", it can assume that the sec-websocket-key mapping is already in Object Store and get the recording id related.
This mapping is also used to buffer the binary data until 5MB.

Binary Data Handling

AWS S3 Multipart Upload is handy, but there is a limitation that "The minimum size of each part must be 5MB (except for the last part)". This "5MB" is sever for a MuleSoft runtime and browser... but Object Store can handle 10MB for a record.

This client application sends a binary data every 10 second, and the server (MuleSoft) side stores it in ObjectStore with sec-websocket-key as the key. At that time, the new binary data and stored binary data are concatenated, then stored again (if < 5MB).

The concatenation is handled by a custom Java class "BinaryUtils", because I couldn't find a way with DataWeave. "Binary" type is converted to "byte[]" in Java arguments/returns, so it just has to be concatenated by ByteArrayOutputStream, easy and simple.

If the mediaType is "application/java", sizeOf(data as Binary) works (because it is actually "byte[]"?), but not with "application/octet-stream" somehow.

RAML for API

#%RAML 1.0
title: skd-piano-lesson

types:
  Error:    
    example:
      {message : "Unknown error"}
    properties:
      message:
        type: string
        description: error message
  User: !include schemas/user-dataType.raml
  Piece:  !include schemas/piece-dataType.raml
  Recording: !include schemas/recording-dataType.raml
  Playlist: !include schemas/playlist-dataType.raml

/me:
  description: Handle login user information
  get:
    displayName: Get login user information
    responses:
      200:
        body:
          application/json:
            type: User
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

/user:
  post:
    displayName: Upsert a User
    description: if matched Id is found, this upserts the given record. if not, inserts. "password" has to be given as "Non-Encrypted" string.
    body:
      application/json:
        type: User
    responses:
      200:
        body:
          application/json:
            type: User
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

  /{user_id}:
    uriParameters:
      user_id:
        displayName: User Id
        type: string
        description: the Id of the User you query
    get:
      displayName: Get a User
      description: Reterns a User if Id matched. "password" is given as "Encrypted" string.
      responses:
        200:
          body:
            application/json:
              type: User
        401:
          description: Unauthorized error
          body:
            application/json:
              type: Error

/pieces:
  get:
    displayName: Get all Pieces of the user
    queryParameters:
      inlist:
        description: if true, this returns all "inlist=true" pieces of the user. if false, "inlist=false"
        type: boolean
        default: true
    responses:
      200:
        body:
          application/json:
            type: Piece[]
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

/piece:
  post:
    displayName: Upsert a Piece
    description: if matched Id is found, this upserts the given record. if not, inserts.
    body:
      application/json:
        type: Piece
    responses:
      200:
        body:
          application/json:
            type: Piece
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error


/playlist:
  get:
    displayName: Get a playlist
    queryParameters:
      datestring:
        description: target datestring (YYYY-MM-DD format
        type: string
        required: true
      piece_id:
        description: if specified, this returns a list with only the piece
        type: string
        required: false
    responses:
      200:
        body:
          application/json:
            type: Playlist
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

/composers:
  get:
    displayName: Get a list of composers the user has
    responses:
      200:
        body:
          application/json:
            type: array
            items:
              properties:
                composer:
                  type: string
                  description: name of the composer
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

/recording:
  /{id}:
    uriParameters:
      id:
        type: string
        displayName: Recording Id
        description: Id of the recording you query
    get:
      displayName: Get a recording and the signed url
      responses:
        200:
          body:
            application/json:
              properties:
                recording:
                  displayName: the recording info
                  type: Recording
                signedUrl:
                  displayName: Signed URL for AWS S3
                  type: string
        401:
          description: Unauthorized error
          body:
            application/json:
              type: Error

/start:
  post:
    displayName: start recording
    description: start recording by creating a recording record and preparing multi-part upload
    body:
      application/json:
        properties:
          piece_id:
            displayName: Piece Id
            description: Id of the piece for the recording
            type: string
            required: true
          datestring:
            displayName: Date String (YYYY-MM-DD)
            description: Lesson date in YYYY-MM-DD format
            type: string
            required: true
    responses:
      200:
        body:
          application/json:
            type: Recording
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error


/stop:
  post:
    displayName: stop recording
    description: stop recording by completing multi-part upload
    body:
      application/json:
        properties:
          recording_id:
            displayName: Recording Id
            description: Id of the recording to stop
            type: string
            required: true
    responses:
      200:
        body:
          application/json:
            type: object
            properties:
              Location:
                description: URL of the recording file
                type: string
              Bucket:
                description: bucket name to be stored 
                type: string
              Key:
                description: Key of the recording file
                type: string
              ETag:
                description: ETag for the upload
                type: string
      401:
        description: Unauthorized error
        body:
          application/json:
            type: Error

Blog

Piano Lesson - MuleSoft Self-contained WebSocket Application

Seitaro