This tutorial will go over how to use Azure Cognitive Services to automatically get an image dataset and train a Custom Vision machine learning model. The github repo has a simple nodejs console app that uses Bing Image Search to scrape images to a local directory based on search term(s). Then use the images scraped to create a Custom Vision model with Azure Cognitive Services.

Prerequisites

• nodejs
• vs code

First create an Azure account (if you dont have one already)

Click here to create your account

Next we will create a Bing Search Api Resource needed to get our dataset

• Go to portal.azure.com
• Select "Create Resource"
• Select "AI + Machine Learning"
• Select "Bing Search v7"
• Complete the required fields
• Select "Create"
• When the deployment succeeds you will get a notification in the top right corner.
• Select "Go to resource"
• Select "Keys" from the left hand navigation for the resource.
• Copy the Subscription Key and paste it into the script.

Cognitive Service Bing Search Docs

Image Scraper using Bing Search Cognitive Service

We are going to use the image search function of this service however you can use this same endpoint to search web, news, image, video and entity. To run the Image Scraper Script with the Bing Search Cognitive Service resource we created in Azure follow the below steps.

Clone the repo, install packages, and open in VS Code

git clone https://github.com/cassieview/Use-ML-To-Classify-Yoga-Poses.git
cd Use-ML-To-Classify-Yoga-Poses
npm install
code .

Add the key into the app.js script javascript const subscriptionKey = 'addyourkeyhere'
Run the script node app.js

When the script completes you will have a new folder named poses with subfolders for each yoga pose specified in the search term collection. Within each pose folder there will be a test and train folder. The data is split at aproximately 70% to train and 30% to test.

Now that we have a dataset lets create our Custom Vision API Resource to train a model that will recognize yoga poses

• Go to portal.azure.com
• Select "Create Resource"
• Search "Custom Vision"
• Select "Create"
• Complete the required fields
• Select "Create"
• When the deployment succeeds you will get a notification in the top right corner.
• Select "Go to resource"

Create a Custom Vision Project and Train Model

• Select "Quick Start"
• Select "Custom Vision Portal"
• Select 'Sign in"
• Select "New Project"
  • Enter a Name and Resource Group
  • Project Types: Classificaion
  • Classification Types: MultiClass General
  • Domains: General
• Click Create project
• Now we need to add and tag images
  • Select "Add Images" and navigate to the train folder for the scraped images
  • Ctrl + A to select all the images in the folder
  • Select "Open"
  • Add tag for photos to indicate the yoga pose
  • Select "Upload photos"
  • Repeat these steps for each yoga pose since each yoga pose is a class in our classification model.
  • Review each class to see if the images are correct. THere are some images that are returned that are either incorrect or not good examples. If you dont have good data you wont get good results. Review and update the data so that the classification data for each pose is a good, clear dataset.
• Once all the data is uploaded and tagged select "Train"
• They API will provide some stats on how your model is performing. Use this is an indication of how its preforming but not and absolute truth. Test your model with the data from the scarped images in the test folder. Its important to test your model on unseen data.

• Steps to upload a test image and test your model.

  • Select "Quick Test" from the upper right corner
  • Browse to local files and select a image from the test folder
  • The predication will appear on the right side of the window

• Once you have tested your model and feel you have a good dataset. Its time to operationalize the selected model. This is super easy with the Azure endpoints. No need to deploy anything. In the next steps I will show you how to grab the URLs to integrate your model into your project.

Get the Prediction Url

• Select "Performace" from the top navigation in the Custom Vision portal
• Select "Prediction Url"
• Copy the Url to either use an image url or image file
• Additionally by selecting the settings icon in the right corner will bring you to the keys and urls to call the endpoint.

Consume the API in an App

Once the model is trained and at a satisfactory accuracy its ready to be used in your App

• Go to the Resource Group that contains the Custom Vision Resource
• There should be a resource named "YourCustomVisionResourceName_Prediction - Quick start"
• The Select "Api Reference" from the list of helpful links
• This will open a page with docs on how to call the api to get predictions from the model
• If you scoll all the way to the bottom there are code samples in multiple languages to get you started

I love to test APIs with Postman. Here are steps so use postman

• Get the predction url as described above
• Select "Post" from the dropdown and paste in the url
• Go to the settings and get the Prediction Key
• In the headers add the Prediction-key as the key and the actual key as the value
• Auth should be set to no auth
• Select Body > Raw > Json
• Paste in the following Json for the body of the request

{
"Url": "UrlForImageGoesHere"
}

• Send Request

This same logic can be applied to all sorts of image classification problems. If you have quesitons or comments please leave them below. Thanks!

Helpful Doc Links

Quickstart: Search for images with the Bing Image Search SDK for Node.js
Quickstart: Create an image classification project with the Custom Vision Node.js SDK
Cognitive Service Bing Search Docs
Cognitive Service Custom Vision Docs
Create a Node.js app in Azure App Service on Linux