Plotting Bookmarks with Flask, Matplotlib, and OAuth 2.0

I don’t Bookmark Tweets very often, and I was wondering how constantly I use the Bookmark functionality on Twitter. To figure out exactly how often I bookmark Tweets, I built a demo application in Python using Flask.

This application will authorize your account using the OAuth 2.0 Authorization Code Flow with PKCE and plot up to your last 100 Bookmarked Tweets. The full code sample can be found on our GitHub.

Step 1: Getting set up with the Twitter API

If you don’t already have access to the Twitter API, you can sign up for a developer account. Once you have an approved developer account, you must create a Project in the developer portal. Each Project contains an App with which you can generate the credentials required to use the Twitter API. You will also need to have OAuth 2.0 turned on in your App’s settings. In our platform overview, you can learn more about getting started with the Twitter API.

Step 2: Setting up your environment

After obtaining access to the Twitter API, you are ready to set up your requirements.txt file.

First, you’ll need to create a directory for this project in your terminal and navigate into it. After making a requirements.txt file, you can open it in your chosen code editor. In your terminal, run the following command:

mkdir plot-bookmarks
cd plot-bookmarks
touch requirements.txt

Add these packages to requirements.txt file

certifi==2022.6.15
charset-normalizer==2.0.12
click==8.1.3
cycler==0.11.0
Flask==2.1.2
fonttools==4.33.3
gunicorn==20.1.0
idna==3.3
importlib-metadata==4.11.4
itsdangerous==2.1.2
Jinja2==3.1.2
kiwisolver==1.4.3
MarkupSafe==2.1.1
matplotlib==3.5.2
numpy
oauthlib==3.2.0
packaging==21.3
pandas==1.3.5
Pillow==9.1.1
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.1
requests==2.28.0
requests-oauthlib==1.3.1
six==1.16.0
urllib3==1.26.9
Werkzeug==2.1.2
zipp==3.8.0

To install all the packages in your requirements.txt file, in your terminal, run the following command:

pip install -r requirements.txt

Step 3: Creating your templates

Now that you have your environment set, you are ready to define your templates. Templates in Flask allow you to define the HTML of your pages. For this example, you’ll define two templates, base.html, and index.html.

Inside of the plot-bookmarks directory, you will want to make a new directory in your terminal and two files for templates (base.html and index.html). In your terminal, run the following command:

mkdir templates
cd templates
touch base.html
touch index.html

The base.html file contains the framework for all HTML pages in the application and can be extended to apply to other pages.

Add the following code to base.html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Plot Bookmarks!{% block title %}{% endblock %}</title>
    <link
      rel="stylesheet"
      href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css"
    />
  </head>

  <body>
    <div class="container">
      <h1>Plot Bookmarks by Date</h1>
      {% block containercontent %}{% endblock %}
      <hr />
      <div class="footer">
        <p class="text-muted">
          <small
            >This is a basic demo created by the
            <a href="https://twitter.com/TwitterDev">@TwitterDev</a> team. It is
            written in Python, using
            <a href="http://flask.pocoo.org/">Flask</a>. Basic CSS is provided by
            <a href="http://getbootstrap.com/">Bootstrap</a>. The source code
            for this project can be found at
            <a href="https://github.com/twitterdev/plot-bookmarks"
              >github/twitterdev/plot-bookmarks</a
            >.</small
          >
        </p>
      </div>
    </div>
  </body>
</html>

The index.html template extends from the base.html template and will be used in your code later on to introduce the application to the user. Add this code to index.html.

{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This app lets you plot your Bookmarked Tweets by Date.</p>
<div class="hero-unit">
    <p>To get started, we'll need you to authorize Hello world twitter app to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0</a></p>
    <p>Here's how it works:</p>
    <ol>
        <li>We will redirect you to Twitter.com for you to authorize our app.</li>
        <li>While at Twitter.com, you will authorize bookmarks-to-Pinterest to access your bookmarks. Upon authorization, we will get your recent 100 bookmarks.</li>
        <li>Next we will display your Bookmarked Tweets by date.</li>
    </ol>
</div>
<p>Ready to give this a shot?</p>
<a href="/start" class="btn btn-small btn-success">Let's Begin</a>
{% endblock %}

Step 4: Creating your app.py file

Now that you have your templates set, you will create an app.py file which will be the main file for your application.

To get started, navigate to the plot-bookmarks directory in your terminal and create an app.py file by running the following command:

cd ..
touch app.py

You will also set up environment variables to ensure you aren’t passing in your secrets directory. To achieve this, you will define environment variables. In your terminal, run the following command:

export CLIENT_ID=’xxxxxxxxxxxxxx’
export CLIENT_SECRET=’xxxxxxxxxxx’
export REDIRECT_URI=’http://127.0.0.1:5000/oauth/callback’

You can obtain your own OAuth 2.0 Client ID and Secret inside of your App’s settings in the developer portal. Use your Client ID and Secret instead of the xxxxxxxxxxxxxx.

I am currently using the type of App as a public client, a single-page App. Inside your App’s OAuth settings in the Developer Portal, ensure that you also have the above redirect URI defined for locally testing your application.

You are ready to start editing your app.py file in your chosen code editor. First, you will import the packages you’ll need for your application. You will use base64, hashlib, and re to create your code verifier and code challenge. The os package will be used to set environment variables and randomize values. You will use the requests package to make HTTP requests to the Twitter API and pandas to create a data frame so you can plot values. You will use io and the matplotlib features for creating your bookmarks plot and flask for creating the web framework.

Add this code to your app.py file.

import base64
import hashlib
import os
import re
import requests
import pandas as pd
import io

from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
from requests_oauthlib import OAuth2Session
from flask import Flask, request, redirect, session, render_template, Response

First, you will need to set a variable for your app, which is the start of every Flask app. After, you’ll need to create a secret key for your app to be a random string using the os package.

Add this code to below your import statements.

app = Flask(__name__)

app.secret_key = os.urandom(50)

To bring the environment variables into your code, you can use the os package to get the variables you set in your terminal.

Add this code after the app = Flask(__name__) code block.

client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")

Additionally, you will obtain the environment variable you set for your redirect URI and create a variable for the authorization and token URLs for generating an OAuth 2.0 token.

Add this code below the client_secret variable.

redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"

Scopes are what sets the permissions for an application. With OAuth 2.0, there are more fine-grained permissions, so you only need to ask the user for what you need. For this application, you will need the following scopes:

tweet.read, which gives you access to read Tweets.
users.read allowing you to obtain information about users
bookmark.read giving you access to an authenticated user’s bookmarks

Add this code below the token_url variable.

scopes = ["tweet.read", "users.read", "bookmark.read"]

Since Twitter’s implementation of OAuth 2.0 is PKCE compliant, you will need to set a code verifier, a secure random string. The code verifier is used to create the code challenge. The code challenge is a base64 encoded string of the SHA256 hash of the code verifier.

Add this code below the scopes variable.

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

At this point, you’ll want to set a function for getting an authenticated user’s Bookmarks back from the Twitter API. For debugging, you’ll want to add a print statement so that you know when you are requesting your server. This function uses the requests package to request the Bookmarks lookup endpoint. Since you will want to use the date of the Bookmark for the plot later, make sure that you pass in a tweet.field of created_at as a param.

def get_bookmarks(user_id, token):
    print("Making a request to the bookmarks endpoint")
    params = {"tweet.fields": "created_at"}
    return requests.request(
        "GET",
        "https://api.twitter.com/2/users/{}/bookmarks".format(user_id),
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
        params=params,
    )

Now, you’ll want to set up the landing page of your application. This will render the template from the index.html file you created earlier. The @app.route("/") indicates this is the first page at the beginning of the application a user will be taken to.

@app.route("/")
def hello():
    return render_template("index.html")

After clicking the “Let’s Begin” button on the application you will be directed to a page for the user to log into Twitter and authorize your App to make requests on the behalf of the user.

@app.route("/start")
def demo():
    global twitter
    twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
    authorization_url, state = twitter.authorization_url(
        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
    )
    session["oauth_state"] = state
    return redirect(authorization_url)

After the user logs in, they will be directed to the callback. The callback will generate an object called token that contains a series of tokens that includes an access token. The access token is what is used to make a request to the Bookmarks lookup endpoint. You will print out this token object to debug on the server.

@app.route("/oauth/callback", methods=["GET"])
def callback():
    code = request.args.get("code")
    token = twitter.fetch_token(
        token_url=token_url,
        client_secret=client_secret,
        code_verifier=code_verifier,
        code=code,
    )
    print(token)

After getting a token, you will want to make a request to the authenticated user lookup endpoint. Since the Bookmarks lookup endpoint requires you to have the ID of the user you are obtaining bookmarks on behalf of. You will pass the user_id and token object into the get_bookmarks function you defined earlier in your code. You’ll save your Bookmarks into a JSON object so that you can easily convert into a data frame.

    user_me = requests.request(
        "GET",
        "https://api.twitter.com/2/users/me",
        headers={"Authorization": "Bearer {}".format(token["access_token"])},
    ).json()
    print(user_me)
    user_id = user_me["data"]["id"]
    bookmarks = get_bookmarks(user_id, token).json()

Now you can create a data frame in Pandas for your Bookmarks, and adjust the date so that it is easy to work with. You’ll also plot a count of Bookmarks per day. To do so, you can use the groupby function of Pandas. You will define your labels as the date the Tweet was created and the count as the values.

    df = pd.DataFrame(bookmarks["data"])
    df["created_at"] = pd.to_datetime(df["created_at"])
    df["created_at"] = df["created_at"].dt.date
    counts = pd.DataFrame({"count": 
    df.groupby(["created_at"]).size()}).reset_index()
    labels = counts["created_at"]
    values = counts["count"]

To plot the labels and values you will use Matplotlib. Once you have your graph you can save it and render in a view.

    fig = Figure()
    ax = fig.subplots()
    ax.set_title("Number of Bookmarks by Date")
    ax.bar(labels, values, color="orange", width=26)
    output = io.BytesIO()
    FigureCanvas(fig).print_png(output)
    return Response(output.getvalue(), mimetype="image/png")

if __name__ == "__main__":
    app.run()

To run the file locally, run the following line in your terminal:

python app.py

For further testing or if you run into any errors you may want to consider turning on Flask’s debug mode.

Step 5: Deployment

I currently have this project deployed using Render. I used a similar method to what’s described in their documentation on flask deployment.

To serve the correct URI redirect, I had to deploy the application first to figure out what my URL would be and set it up in my environment variables in Render and the developer portal.

The redirect URI for my application was as follows:

https://plot-bookmarks.onrender.com/oauth/callback

I also set up environment variables for Client ID and Client Secret in my environment variable in my settings in Render.

Next steps

Hopefully, this can be a starting point for you to get started with the Bookmarks endpoint, plotting in Python and OAuth 2.0. This code sample can be easily extended to be part of a complete application that includes more graphs and dynamic content.

Be sure to inform us on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything.

Blog