Mobile App backend using Azure Functions and Face API

Humberto Jaimes - humberto@humbertojaimes.net

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-Serverless content collection. New articles are published every day — that's right, every day — from community members and cloud advocates in the month of September.

Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

The Cognitive Locator project, known publicly as 'Busca.me', is a project dedicated to reporting and finding missing persons. The project was founded due to the disasters related to the earthquake of September 19, 2017, which affected multiple states in Mexico. At first, the project was solely focused on finding or reporting people who went missing as a result of the earthquake. However, now that the project has grown, it not only aims to support the people affected at that time, but to anyone who is going through this devastating situation.

The source code of the Cognitive Locator project is under the MIT license in https://github.com/humbertojaimes/cognitive-locator

The project backend was created using Azure Functions. And this is the project architecture

This post will explain how to create two functions. Both functions will have similar functionality as the Cognitive Locator backend core functions.

An HTTP function that register a device in an Azure Notification Hub.
A Blob Storage function that analyze a photo and report the result to a device using a Push Notification.

Note: At this time the post is only intended to explain the Azure Functions part. In the future I can create the Azure Notification Hub configuration and the Xamarin posts.

The original project has the Xamarin Sample

Prerequisites:

An Azure Account
A provisioned a configured Azure Notification Hub instance
A provisioned an Azure Face API service

Device Registration Function

Creating a new project

For this tutorial I am going to use Visual Studio 2019 For Mac.

Create a new Project

1.1 Select "Azure Functions" in the "Cloud" tab

Write "DeviceInstallationRegistration" as the function name and choose the "HttpTrigger"

For this demo, select the "Anonymous" access level.

To finalize the creation of the project use "ServelessSeptember" as the project and solution name.

In the new project create a "Settings" class with the following content.

public class Settings
    {
        public static string AzureWebJobsStorage = Environment.GetEnvironmentVariable("AzureWebJobsStorage");

        public static string FaceAPIKey = Environment.GetEnvironmentVariable("Vision_API_Subscription_Key");
        public static string Zone = Environment.GetEnvironmentVariable("Vision_API_Zone");


        public static string NotificationAccessSignature = Environment.GetEnvironmentVariable("NotificationHub_Access_Signature");
        public static string NotificationHubName = Environment.GetEnvironmentVariable("NotificationHub_Name");


    }

This class is intended to get the keys and connections strings from the Azure Functions portal configuration.

Programming the function

Before writing code, we need to install the "Microsoft.Azure.NotificationHubs" NuGet package.

We need a class that contains the device information.

Create a folder "Domain" and a class "DeviceInformation" class inside it.

public class DeviceInstallation
    {
        public string InstallationId { get; set; }

        public string Platform { get; set; }

        public string PushChannel { get; set; }

    }

In a "Helpers" folder create a class "NotificationsHelper".

This is the content of the class

 public static class NotificationsHelper
    {

        //Notification Hub settings
        public static string ConnectionString = Settings.NotificationAccessSignature;
        public static string NotificationHubPath = Settings.NotificationHubName;

        // Initialize the Notification Hub
        static NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(ConnectionString, NotificationHubPath);

        /// <summary>
        /// Receive the information to register a new mobile device in the Azure Notification Hub
        /// </summary>
        /// <param name="deviceUpdate">The device information</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task RegisterDevice(DeviceInstallation deviceUpdate, ILogger log)
        {
            Installation installation = new Installation();
            installation.InstallationId = deviceUpdate.InstallationId;
            installation.PushChannel = deviceUpdate.PushChannel;
            switch (deviceUpdate.Platform)
            {
                case "apns":
                    installation.Platform = NotificationPlatform.Apns;
                    break;
                case "fcm":
                    installation.Platform = NotificationPlatform.Fcm;
                    break;
                default:
                    throw new Exception("Invalid Channel");
            }
            installation.Tags = new List<string>();
            await hub.CreateOrUpdateInstallationAsync(installation);
            log.LogInformation("Device was registered");
        }


        /// <summary>
        /// Register a device to receive specific notifications related to a report
        /// </summary>
        /// <param name="installationId">Identifier of the device</param>
        /// <param name="requestId">The report identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task AddToRequest(string installationId, string requestId, ILogger log)
        {
            await AddTag(installationId, $"requestId:{requestId}", log);
            log.LogInformation($"Device was registered in the request {requestId}");
        }


        /// <summary>
        /// Remove a device from the notificartions related to a report
        /// </summary>
        /// <param name="installationId">Identifier of the device</param>
        /// <param name="requestId">The report identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task RemoveFromRequest(string installationId, string requestId, ILogger log)
        {
            await RemoveTag(installationId, $"requestId:{requestId}", log);
            log.LogInformation($"Device was removed from the request {requestId}");
        }



        /// <summary>
        /// Remove a device from an specific tag in the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <param name="tag">The Azure Notification Hub Tag</param>
        /// <param name="log"></param>
        /// <returns></returns>
        private static async Task RemoveTag(string installationId, string tag, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);
                if (installation.Tags == null)
                {
                    if (installation.Tags.Contains(tag))
                        installation.Tags.Remove(tag);
                    await hub.CreateOrUpdateInstallationAsync(installation);
                }
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Add a device to an specific tag in the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <param name="tag">The Azure Notification Hub Tag</param>
        /// <param name="log"></param>
        /// <returns></returns>
        private static async Task AddTag(string installationId, string newTag, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);
                if (installation.Tags == null)
                    installation.Tags = new List<string>();
                installation.Tags.Add(newTag);
                await hub.CreateOrUpdateInstallationAsync(installation);
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Remove a device from the Azure Notification Hub
        /// </summary>
        /// <param name="installationId">The device identifier</param>
        /// <returns></returns>
        public static async Task RemoveDevice(string installationId)
        {
            await hub.DeleteInstallationAsync(installationId);
        }


        /// <summary>
        /// Send a push notification about an specific report id. At this moment is only supported 1 device per report
        /// </summary>
        /// <param name="text">The notification message that will be displayed by the mobile device</param>
        /// <param name="requestId">the report identifier</param>
        /// <param name="installationId">The device identifier</param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task SendNotification(string text, string requestId, string installationId, ILogger log)
        {
            try
            {
                Installation installation = await hub.GetInstallationAsync(installationId);

                if (installation.Platform == NotificationPlatform.Fcm)
                {
                    var json = string.Format("{{\"data\":{{\"message\":\"{0}\"}}}}", text);
                    await hub.SendFcmNativeNotificationAsync(json, $"requestid:{requestId}");
                    log.LogInformation($"FCM notification was sent");
                }
                else
                {
                    var json = string.Format("{{\"aps\":{{\"alert\":\" {0}\"}}}}", text);
                    await hub.SendAppleNativeNotificationAsync(json, $"requestid:{requestId}");
                    log.LogInformation($"Apple notification was sent");
                }
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }
        }


        /// <summary>
        /// Send a notification to all Apple and Firebase devices in the notification hub
        /// </summary>
        /// <param name="text">The notification message that will be displayed by the mobile device </param>
        /// <param name="log"></param>
        /// <returns></returns>
        public static async Task SendBroadcastNotification(string text, ILogger log)
        {
            try
            {
                var json = string.Format("{{\"data\":{{\"message\":\"{0}\"}}}}", text);
                await hub.SendFcmNativeNotificationAsync(json);
                log.LogInformation($"FCM notification was sent");

            }
            catch (Exception ex) //If there aren't FCM devices registered in the hub, it throws an error
            {
                log.LogInformation(ex.Message);
            }


            try
            {
                var json = string.Format("{{\"aps\":{{\"alert\":\" {0}\"}}}}", text);
                await hub.SendAppleNativeNotificationAsync(json);
                log.LogInformation($"Apple notification was sent");
            }
            catch (Exception ex) //If there aren't Apple devices registered in the hub, it throws an error
            {
                log.LogInformation(ex.Message);
            }
        }
    }

And finally the code of the function.

[FunctionName("DeviceNotificationsRegistration")]
        public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "put", Route = "devicenotificationsregistrations/")]HttpRequestMessage req, ILogger log)
        {
            try
            {
                log.LogInformation("New device registration incoming");
                var content = await req.Content.ReadAsStringAsync();
                DeviceInstallation deviceUpdate = await req.Content.ReadAsAsync<DeviceInstallation>();
                await NotificationsHelper.RegisterDevice(deviceUpdate, log);
                log.LogInformation("New device registered");
                return req.CreateResponse(HttpStatusCode.OK);
            }
            catch (Exception ex)
            {
                log.LogInformation($"Error during device registration: {ex.Message}");
            }
            return req.CreateErrorResponse(HttpStatusCode.InternalServerError, "Error during device registration");
        }

Person Registration Function

Create a "FaceClientHelper" in the "Helpers" folder.

In the original project, the helper contains several methods that interact with the Face API. For demo purposes, this sample only has one method.

https://github.com/humbertojaimes/cognitive-locator/blob/master/source/CognitiveLocator.Functions/Client/FaceClient.cs

public class FaceClientHelper
    {
        private string FaceAPIKey = Settings.FaceAPIKey;
        private string Zone = Settings.Zone;

        /// <summary>
        /// Analyze a photo using the Face API
        /// </summary>
        /// <param name="url">The image url</param>
        /// <returns></returns>
        public async Task<List<JObject>> DetectFaces(String url)
        {
            using (var client = new HttpClient())
            {
                var service = $"https://{Zone}.api.cognitive.microsoft.com/face/v1.0/detect";
                client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", FaceAPIKey);
                byte[] byteData = Encoding.UTF8.GetBytes("{'url':'" + url + "'}");
                using (var content = new ByteArrayContent(byteData))
                {
                    content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
                    var httpResponse = await client.PostAsync(service, content);

                    if (httpResponse.StatusCode == HttpStatusCode.OK)
                    {
                        List<JObject> result = JsonConvert.DeserializeObject<List<JObject>>(await httpResponse.Content.ReadAsStringAsync());
                        return result;
                    }
                }
            }
            return null;
        }


    }

Create a new function in the project

Choose the blob trigger and write "PersonRegistration" as the function name.

Set the connection string of the blob, for this sample we are use the same storage that comes with the function, so the connection string is "AzureWebJobsStorage"

The code of the function is the following

private static FaceClientHelper client_face = new FaceClientHelper();

        [FunctionName("PersonRegistration")]
        public static async Task Run([BlobTrigger("images/{name}.{extension}")]CloudBlockBlob blobImage, string name, string extension, ILogger log)
        {

            log.LogInformation($"Image: {name}.{extension}");
            string notificationMessage = "Error";
            string json = string.Empty;
            var deviceId = blobImage.Metadata["deviceid"];


            try
            {

                await NotificationsHelper.AddToRequest(deviceId, name, log);
                log.LogInformation($"uri {blobImage.Uri.AbsoluteUri}");
                log.LogInformation($"Zone {Settings.Zone}");

                //determine if image has a face
                List<JObject> list = await client_face.DetectFaces(blobImage.Uri.AbsoluteUri);

                //validate image extension 
                if (blobImage.Properties.ContentType != "image/jpeg")
                {
                    log.LogInformation($"no valid content type for: {name}.{extension}");
                    await blobImage.DeleteAsync();

                    notificationMessage = "Incorrect Image Format";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

                //if image has no faces
                if (list.Count == 0)
                {
                    log.LogInformation($"there are no faces in the image: {name}.{extension}");
                    await blobImage.DeleteAsync();
                    notificationMessage = "The are not faces in the photo";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

                //if image has more than one face
                if (list.Count > 1)
                {
                    log.LogInformation($"multiple faces detected in the image: {name}.{extension}");
                    await blobImage.DeleteAsync();
                    notificationMessage = "Multiple faces detected in the image";
                    await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                    return;
                }

            }
            catch (Exception ex)
            {
                // await blobImage.DeleteAsync();

                log.LogInformation($"Error in file: {name}.{extension} - {ex.Message}");
                notificationMessage = "Error in file registration";
                await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
                return;
            }

            log.LogInformation("person registered successfully");
            notificationMessage = "Person registered successfully";
            await NotificationsHelper.SendNotification(notificationMessage, name, deviceId, log);
        }
    }

Publishing the function

Using the Visual Studio For Mac Wizzard we can publish our function.

Using the Azure Storage Explorer (https://azure.microsoft.com/es-mx/features/storage-explorer/) create a new blob container for the "images".

For demo purposes, we are going to set the access level to public.

Configuring the function settings

Using the azure portal, in the published function "Platform Features" option select "configuration"

Add the following settings in the application settings section.

"Vision_API_Subscription_Key" - The Face API access key
"Vision_API_Zone" - The Face API Azure Region
"NotificationHub_Access_Signature" - The Notification Hub connection string
"NotificationHub_Name" - The Notification Hub name

Testing the Functions

We can register a new device using PostMan (https://www.getpostman.com/).

Get the function URL from the Azure portal

The following json contains the structure expected by the function

{
    "InstallationId": "87653f8dc8e80kiuy",
    "Platform": "fcm",
    "PushChannel": "Cafg:APA91bFPcN01WnfXqRMQhsSySWVY6fgIggW3lpZ_E1tPL94pdChZb0-dIxdhyh9WxNyoxIWggDPctYyOTqRKpssqG5mjr-7nOJxKwBdDPJrmz8b-xt-B5Xna66S4IZRJpAcqNea6biv_"
}

In PostMan send a Put request using the URL and the json body