Python, Sentiment Analysis, R and Shiny

Knowing how your customers feel about your product is as important as creating a good product. Wouldn’t it be great if we could determine in a simple way how our customers feel by analyzing the feedback they provide via email messages? With Nylas Neural API’s Sentiment Analysis, we can do that. Sentiment Analysis is a way of computationally analyzing a writer’s work to determine how they feel about a selected topic. With basic sentiment analysis, you can check if the writer feels positive, negative, or neutral about something.

For this post, we’re going to create a simple feedback web page using Python. The sentiment information we gather will be analyzed and stored locally in a SQLite Database to be later picked up and displayed in a Shiny Dashboard using the R Programming Language. (Shiny is an R package that lets you easily display data with a variety of charts and graphs in your own dashboard.) We’re keeping everything local just for the sake of simplicity.

Is your system ready?

If you don’t have the Nylas Python SDK installed and your environment isn’t configured, I would recommend you to read the post How to Send Emails with the Nylas Python SDK where everything is clearly explained.

To complete this project you will need:

Nylas developer account and credentials
Nylas Python SDK installed
Gmail account (we will use Gmail to label and sort emails)
We will install R and Shiny later in this post, so you don’t need to worry about it now

What are we going to do?

Let’s pretend that we are the owners of a small company that produces vegan eggs called “VeggiEggs”. We want to build a Flask application that will collect feedback from our customers and send it to us via email, so that we can store the information and also reply back as a follow up. Then we want a Python script that will read and analyze every email using Sentiment Analysis and finally create an R Shiny Script that will display a dashboard with the information we gathered.

Working on the application

What we need to do first is set up a Label so that the emails are easy to find.

Creating a Label

On Gmail, we can do the following:

On the search bar, press the “Show Search Options” button.

Then, use the “veggieggs” and “feedback” words.

When we create the filter, we can assign the messages to an existing label or create a new one.

We don’t need to nest it, so just enter the label name.

Once we create it, we can apply some extra filters

Now, every email that we receive containing those words “VeggieEggs” and “Feedback” will go into our label and skip the inbox.

Installing the Flask package

As we want to create a Python web application, our best option is to use Flask, one of the most popular Micro Frameworks in the Python world.

To install it, we can type the following command on the terminal:

$ pip3 install flask

Once installed, we’re ready to go.

Creating the Flask application

As we’re going to develop this application locally and not deploy it anywhere else. We’re going to create a file called config.json which will hold our Nylas Tokens information. This comes with Flask, so why not use it?

{
  "NYLAS_OAUTH_CLIENT_ID": "<YOUR_CLIENT_ID>",
  "NYLAS_OAUTH_CLIENT_SECRET": "<YOUR_CLIENT_SECRET>",
  "SECRET_KEY": "<YOUR_SECRET_KEY>"
}

Now, we’re going to create a file called FeedbackForm.py which will handle getting the feedback and sending the email.

# Import your dependencies
from flask import Flask,render_template,json,request,flash,redirect,url_for
from nylas import APIClient

# Create the app and read the configuration file
app = Flask(__name__)
app.config.from_file("config.json", json.load)

# Initialize your Nylas API client
def load_nylas():
    nylas = APIClient(
        app.config["NYLAS_OAUTH_CLIENT_ID"],
    app.config["NYLAS_OAUTH_CLIENT_SECRET"],
    app.config["SECRET_KEY"]
    )
    return nylas

# This the landing page
@app.route("/", methods=['GET','POST'])
def index():
# If we are reading the page, show the feedback form
    if request.method == 'GET':
        return render_template('FeedbackForm.html')
# If we are submitting feedback, read the form
    else:
        name = request.form['name']
        email = request.form['email']
        rating = request.form['rating']
        comments = request.form['comments']

# Check for required fields
        if not name:
            flash('Name is required!')
            return redirect(url_for('index'))
        elif not email:
            flash('Email is required!')
            return redirect(url_for('index'))
        elif not comments:
            flash('Comments are required!')
            return redirect(url_for('index'))           
        else:
# If everything is Ok, send the email with the feedback information
            nylas = load_nylas()
            draft = nylas.drafts.create()
            draft.subject = "VeggiEggs Feedback - {} - {} - {}".format(name, email, rating)
            draft.body = comments
    draft.to = [{"name": "<YOUR_NAME>", "email": "<YOUR_EMAIL>"}]

    draft.send()            
    return render_template('ConfirmationForm.html', 
                   name = name, email = email, 
                   rating = rating, comments=comments),            
                                                  {"Refresh":"5;url=/"}

# Run our application       
if __name__ == "__main__":
    app.run()

For this to work properly, we need to create a file called FeedbackForm.html and we’re going to use TailwindCSS to provide a quick look and feel.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script src="https://cdn.tailwindcss.com"></script>
    <title>Nylas</title>
</head>
<body>
    <div class="bg-green-300 border-green-600 border-b p-4 m-4 rounded">
    {% block content %}
    {% with messages = get_flashed_messages() %}
        {% if messages %}
            {% for message in messages %}
                {{ message }}
            {% endfor %}
        {% endif %}
    {% endwith %}
    <h1 class="font-black">We hope you enjoyed VeggiEggs. Please take some time to leave a review</h1>
    <br>
    <form method="post">
        <label for="name" class="font-bold">* Name</label>
        <input type="text" name="name"
               placeholder="Your name"
               value="{{ request.form['name'] }}"></input>
        <br><br>
        <label for="email" class="font-bold">* Email</label>
        <input type="email" name="email"
               placeholder="Your email"></input>
        <br><br>
        <label for="rating" class="font-bold">Choose a rating:</label>
        <select name="rating">
            <option value="1">Poor</option>
            <option value="2">Fair</option>
            <option value="3">Ok</option>
            <option value="4">Good</option>
            <option value="5" selected>Great</option>
        </select><br><br>
        <p class="font-bold">* Comments</p>
        <textarea name="comments"
               placeholder="Your comments"
               rows=5
               cols=50></textarea>        
        <br><br>
        <button type="submit" class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded-full">Submit</button>
    </form>
    <p class="font-bold">* Required</p>
    {% endblock %}
    </div>
</body>
</html>

And we’re going to need a confirmation page as well, so that people know that their responses were recorded correctly.

We're going to create a page called ConfirmationForm.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <script src="https://cdn.tailwindcss.com"></script>
    <title>Nylas</title>
</head>
<body>
    <div class="bg-green-300 border-green-600 border-b p-4 m-4 rounded">
    {% block content %}
    <h2 class="font-semibold">Thanks {{name}} - {{email}}</h2>
    <h2 class="font-semibold">Your rating was: {{rating}}</h2>
    <h2 class="font-semibold">Your comments were:<br><br> {{comments}}</h2>
    <p class="font-semibold">You will be automatically redirected. 
    If it doesn't work, follow the <a href="{{url_for('index')}}">link</a></p>
    {% endblock %}
    </div>
</body>
</html>

We can execute this script from the terminal by typing:

$ python3 FeedbackForm.py

Now, we need to open localhost:5000 on our favourite browser.

NOTE: If you are using Flask on a Mac that has Monterey or higher, you may have trouble accessing localhost. You can solve this by disabling the Airplay Receiver in your Sharing folder. (Airplay Receiver uses port 5000.) Follow these instructions: How to disable Airplay Receiver.

Once we hit submit, we will get instant feedback.

We use redirection, so it will go back to the main page after 5 seconds. And here we’re displaying the information entered by the user.

If we check our mailbox, we will find this:

We have all the information we need on this email. So we’re ready for the next stage.

Creating a Python script

In this step, we’re going to read all the emails with the label “VeggiEggs”, in order to analyze them with the Sentiment Analysis endpoint as well as to gather important information like the rating and the date the email was sent.

We’re going to create a file called NeuralFeeback.py. It looks like this:

# Load your env variables
from dotenv import load_dotenv
load_dotenv()

# Import your dependencies
import os
from nylas import APIClient
import subprocess

# Initialize your Nylas API client
nylas = APIClient(
    os.environ.get("CLIENT_ID"),
    os.environ.get("CLIENT_SECRET"),
    os.environ.get("ACCESS_TOKEN"),
)

# Gather all messages from the label “VeggiEggs”
messages = nylas.messages.where(in_="VeggiEggs")
# Create variables to hold the information to be passed to R
rating = []
sentiment = []
score = []
date = []
sRating = ""
sSentiment = ""
sScore = ""
sDate = ""

# Read all messages
for message in messages:
# Split the title in order to get the rating value
    line = message.subject.split(" - ")
    rating.append(line[3])
# Send each message to the Sentiment Analysis endpoint
    message_analysis = nylas.neural.sentiment_analysis_message([message.id])
# Extract the Sentiment Analysis result
    sentiment.append(message_analysis[0].sentiment)
    score.append(str(message_analysis[0].sentiment_score))
    date.append(str(message.received_at.date()))    

# Turn the information into a long string separated by commas
sRating = ','.join(rating)
sSentiment = ','.join(sentiment)
sScore = ','.join(score)
sDate = ','.join(date)

# We want to know when R is getting called
print("Calling R")
# Call R as a subprocess and launch Shiny in the browser
subprocess.run(["RScript", "-e", "shiny::runApp('<path_to_your_python_env>', launch.browser = TRUE)", sRating, sSentiment, sScore, sDate])

Setting up R

Installing R

As we’re going to use Shiny to create our dashboard, we need to create a R script. If you don’t have R installed on your computer you can do it like this.

Here’s the latest version of R for the Mac R-4.1.3. Also, while optional, we need to install XQuartz as it is not bundled with Mac by default.

Installing RStudio

While it’s not mandatory, we should install RStudio which is by far the best R IDE.

Installing the necessary packages

We will need to install some packages in order to have our Shiny application running. Open up RStudio and click on “Tools → Install packages”. Type the following packages separated by a comma:

shiny → Makes it incredibly easy to build interactive web applications with R

tidyverse → The tidyverse is an opinionated collection of R packages designed for data science

treemap → Displays hierarchical data as a set of nested rectangles

imager → Fast image processing for images in up to 4 dimensions

plyr → Tools for Splitting, Applying and Combining Data

We need to make sure to leave “Install dependencies” checked and press Install.

Once everything is installed, we can create the script.

Creating the Shiny script

We need to press File → New → R Script and save the file as App.r then type the following

# Load packages
library("shiny")
library("tidyverse")
library("treemap")
library("imager")
library("plyr")

# UI Section of our Shiny app
ui <- fluidPage(
# Dashboard headline
  tags$div(
    HTML("<h1 style='color:blue;text-align:center;'>VeggiEggs Dashboard!</h1>")
  ),  
# Dashboard title
  title = "VeggiEggs Dashboard!",
# Define how our plots are going to be laid off
                 fluidRow(
                   column(width=5, plotOutput(outputId = "distPlot")),
                   column(width=5,offset=2, plotOutput(outputId = "distPlot2"))
                 ),
                 fluidRow(
                   column(width=5, plotOutput(outputId = "distPlot3")),
                   column(width=5,offset=2, plotOutput(outputId = "distPlot4"))
                 )) 

# Server part, here is where the plots are defined
server <- function(input, output) {

# We receive the information coming from Python
  args = commandArgs(trailingOnly=TRUE)
  ratings<-as.integer(strsplit(args[1], ",")[[1]])
  sentiments<-strsplit(args[2], ",")[[1]]
  scores<-as.integer(strsplit(args[3], ",")[[1]])    
  dates<-as.Date(strsplit(args[4], ",")[[1]])

# We create a Data.Frame to better handle the data
  entries = data.frame(ratings, sentiments, scores, dates=as.Date(dates))  

# First plot. Summarize the ratings
  output$distPlot <- renderPlot({
    Ratings <- count(entries, 'ratings')
    names(Ratings)<-c("Ratings","Count")
    Count <- Ratings$Count
    Ratings %>%
      ggplot( aes(x = Ratings, y = Count, fill = Count)) + 
        geom_bar(stat="identity")
  })

# Second plot. How many feedbacks per day
  output$distPlot2 <- renderPlot({
    Freq <- count(entries, 'dates')
    names(Freq)<-c("Dates","Entries")
    Freq %>% 
      ggplot( aes(x=Dates, y=Entries)) +
      geom_line(color="green") +
      geom_point()
  })

# Third plot. Frequency of sentiment analysis
  output$distPlot3 <- renderPlot({
    Sentiment <- count(sentiments)
    group <- paste(Sentiment$x,Sentiment$freq)
    sentiment<-Sentiment$freq
    data <- data.frame(group,sentiment)
    treemap(data,
            index="group",
            vSize="sentiment",
            type="index"
    )
  })

# Fourth Plot. Display an image for each score range  
  output$distPlot4 <- renderPlot({
    Score<-mean(entries$scores)
    if (Score < -0.5){
      image_filename<-"angry-emoji.png"
    }
    if (Score > -0.5 && Score < 0.5){
      image_filename<-"neutral-emoji.png"
    }
    if (Score > 0.5){
      image_filename<-"happy-emoji.jpeg"
    }

    image <- load.image(image_filename)
    plot(image)
  })

}

shinyApp(ui = ui, server = server)

As we are calling this script from Python, there’s nothing more to do here.

Here are the images we’re using, and they need to be in the same folder as our script.

Calling the Dashboard

In order to call the Dashboard, the only thing we need to do is run our Python script by typing this into the terminal window:

$ python3 NeuralFeedback.py

The Dashboard will be opened automatically on the default browser.

That’s it. Hope you like this post! You can get more information on Sentiment Analysis on the Nylas Documentation page.

Join our technical demo to see the Nylas Platform in action

Blog