Discovered new tools.

Oauth2-proxy

It's a reverse proxy and static files server that provides authentication in the middle of a user request.
In telescope, I had to use oauth2-proxy and nginx to secure the route for Supabase studio.
Add studio under our subdomain #3098

Process

Configure our docker-compose for staging/production

# production.yml
  nginx:
    depends_on:
      - oauth2-proxy

  oauth2-proxy:
    image: bitnami/oauth2-proxy:7.2.1
    container_name: 'oauth2-proxy'
    command:
      [
        '--provider=github',
        '--cookie-secure=true',
        '--cookie-secret=${OAUTH2_SUPABASE_COOKIE_SECRET}',
        '--upstream=http://studio:3000',
        '--http-address=0.0.0.0:8080',
        '--reverse-proxy=true',
        '--email-domain=*',
        '--github-org=Seneca-CDOT',
        '--github-team=telescope-admins',
        '--client-id=${OAUTH2_SUPABASE_CLIENT_ID}',
        '--client-secret=${OAUTH2_SUPABASE_CLIENT_SECRET}',
      ]
    depends_on:
      - studio

Set nginx services to depend on oauth2-proxy. So we let oauth2-proxy start first before starting nginx.
Declare our oauth2-proxy services to use image bitnami/oauth2-proxy:7.2.1, we give the following command.
- --provider=github, this tell to use GitHub Oauth services see.
- --cookie-secure=true, Set our cookie to be secure.
- --cookie-secret=${OAUTH2_SUPABASE_COOKIE_SECRET}, set a long seed string for secure cookies.
- --upstream=http://studio:3000, set the upstream to be redirected to after authorized, in our case it would be Supabase studio.
- --http-address=0.0.0.0:8080, listen on for HTTP clients.
- --reverse-proxy=true, set reverse-proxy to be true since we are putting our oauth2-proxy under nginx. It helps to automatically set a redirect route.
- --email-domain=*, this is for restricting certain email domains, but for us, we allow any domain.
- --github-org=Seneca-CDOT, this is for restricting by GitHub organization, for use we allow only member that is in Seneca-CDOT.
- --github-team=telescope-admins, this is for restricting within an organization to specific teams, for use we allow only member that is in Seneca-CDOT organization and telescope-admins teams.
- --client-id=${OAUTH2_SUPABASE_CLIENT_ID}, this is our GitHub Oauth client id see.
- --client-secret=${OAUTH2_SUPABASE_CLIENT_SECRET}, this is our GitHub Oauth client secret see.

Configure our nginx config for staging/production

# nginx.conf.template
  server {
    listen 443 ssl http2;
    server_name ${SUPABASE_HOST};

    location / {
      proxy_set_header Connection "";
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Scheme $scheme;
      proxy_pass http://oauth2-proxy:8080/;
    }
  }

In this PR #2964, we added the DNS and SSL.

In our nginx config, we declare a new server with a server name of our ${SUPABASE_HOST} and listen to port 443.
Set the location to be /, set the proxy to be our oauth2-proxy services.
Set the the proxy header proxy_set_header, proxy_set_header, proxy_set_header, this will give our oauth2-proxy the right information to set up the redirect route.

Hurl

Hurl is an open-source command-line tool that runs HTTP requests defined in a simple plain text format.
The sources code is written in Rust which makes this tool very fast to run.
It's made by a French Telecom company called Orange,

Linux installation

curl -LO https://github.com/Orange-OpenSource/hurl/releases/download/1.6.1/hurl_1.6.1_amd64.deb
sudo dpkg -i hurl_1.6.1_amd64.deb

To check if it's installed

hurl -h

Scenario to test

Note: any POST, PUT, DELETE, and GET request needs user to be authenticated. For that let's assume our API uses basic auth and the username is user1@email.com and the password is password1. For the authentication, we use a base64 which can be generated with the following code below.

console.log(btoa('user1@email.com:password1'))
# Output: dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

POST sending a file, it should return a status code of 201, header Locations of the URL to get the data, and JSON body containing an id based on nanoid, create date in string, update date in string
GET at URL of the location previously returned in POST, and compare if the data returned matches the content posted.
PUT at URL of the location previously returned in POST, and update the content, it should return a status code of 200, and JSON body containing an id based on nanoid, create date in string, newest update date in string (should be greater than create date).
GET at URL of the location previously returned in POST, and compare if the data returned matches the content updated.
DELETE at URL of the location previously returned in POST, it should return a status code of 200

Folder structure

.
├── .github
│   ├── workflows
│   │   └── ci.yml
├── tests
│   ├── fixtures
│   │   └── data.txt
│   ├── integration
│   │   ├── myTest.hurl
├── docker
│   ├── docker-compose.yml
├── Dockerfile

Test case file

# myTest.hurl

# 1. POST request 
# Make a POST and send our txt file located in the fixtures folder
# Set the content type to be a text/plain
POST http://localhost:8080/cat
Content-Type: text/plain
# https://hurl.dev/docs/request.html#file-body
file,./fixtures/data.txt;
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 201
HTTP/1.1 201

# https://hurl.dev/docs/asserting-response.html
[Asserts]
# Assert header Location to contain the URL for getting our data
header "Location" matches "^http:\/\/localhost:8080\/cat\/[A-Za-z0-9_-]+$"

# Assert id to matches id generated with nanoid https://www.npmjs.com/package/nanoid
jsonpath "$.id" matches "^[A-Za-z0-9_-]+$"

# Assert created to return a string
jsonpath "$.created" isString

# Assert created to return a string
jsonpath "$.updated" isString

# https://hurl.dev/docs/capturing-response.html#captures
[Captures]
# Capture the header location value and store to url
# This value can be call later in the code.
url: header "Location"
# Capture the id from the returned json and store cat_id
cat_id: jsonpath "$.id"
# Capture the created date from the returned json and store cat_created
cat_created: jsonpath "$.created"
# Capture the updated date from the returned json and store cat_updated
cat_updated: jsonpath "$.updated"

# 2. GET request 
# GET at the url previously captured
GET {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200
# Check the response content type to be text/plain
Content-Type: text/plain
# Check the response body to match the content of data.txt.
file,./fixtures/data.txt;

# 3. PUT request
# PUT at the url previously captured
PUT {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==
Content-Type: text/plain
# Set the request body with our new content 
# https://hurl.dev/docs/request.html#raw-string-body
```New content```

# Check the response to be 200
HTTP/1.1 200

[Asserts]
# Assert id to be the same as cat_id (id captured in the POST return)
jsonpath "$.id" == {{cat_id}}

# Assert created to be the same as cat_created (created captured in the POST return)
jsonpath "$.created" == {{cat_created}}

# Assert updated to be different as cat_updated (updated captured in the POST return)
jsonpath "$.updated" != {{cat_updated}}

# 4. GET request 
# GET at the url previously captured
GET {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200
# Check the response content type to be text/plain
Content-Type: text/plain
# Check the response body to match the new content updated from the PUT request
```New content```

# 5. DELETE request 
# DELETE at the url previously captured
DELETE {{url}}
# Set the Authorization to be a Basic auth and our base64 credential
Authorization: Basic dXNlcjFAZW1haWwuY29tOnBhc3N3b3JkMQ==

# Check the response to be 200
HTTP/1.1 200

To run the test case. See how to run and command option

Let's assume we are in the root of our folder.
Let's assume we have a docker-compose.yml that is already running our microservices and exposes at port 8080.

hurl --test --file-root "./tests" "./tests/integration/myTest.hurl"

--test, activate the test mode which enables --no-output, --progress, and --summary
--file-root set our file root, we need this because we want our fixture to be accessible.

Samples of response

❯ hurl --test --file-root "./tests" "./tests/integration/myTest.hurl"
./tests/integration/myTest.hurl: RUNNING [1/1]
./tests/integration/myTest.hurl: SUCCESS
--------------------------------------------------------------------------------
Executed:  1
Succeeded: 1 (100.0%)
Failed:    0 (0.0%)
Duration:  10ms

Run on CI pipeline

Let's assume we have a docker-compose.yml that spins up our microservices and exposes at port 8080.

# .github/workflows/ci.yml
name: ci
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

jobs:
  hurl-test:
  runs-on: ubuntu-latest
  steps:
    - name: 🏗 Setup repo
      uses: actions/checkout@v2
    - name: 📦 Install hurl
      run: |
        curl -LO https://github.com/Orange-OpenSource/hurl/releases/download/1.6.1/hurl_1.6.1_amd64.deb
        sudo dpkg -i hurl_1.6.1_amd64.deb
    - name: 🏗 Build Containers
      run: docker-compose -f docker/docker-compose.yml up -d
    - name: ✨ Run Hurl Tests
      run: hurl --test --file-root "./tests" "tests/integration/myTest.hurl"

Opinion about this new testing tool.

In overall, this is a pretty unique testing tool, it's super fast and powerful since the code base is written in rust. The learning curves are pretty simple (I managed to learn the basics in a couple of hours), less setup todo since it's just a plain file, the syntax is English friendly, besides the jsonPath that could take some times to adapt.

There is some limitation, like for the assert, sometimes it's not that easy to compare an object to an object or read from an object since when you read a capture of an object, it tries to construct into a string so you are not able to use jsonPath to get some property.

Thanks to @humphd who showed us this tool. And I'm happy because it's a French company that made this wonderful open-source tool.