Setup Action Mailbox with SendGrid

Rails 6 released with many awesome features and action mailbox was one of them that has come to make the life easier. From Official Action Mailbox Guide:

Action Mailbox routes incoming emails to controller-like mailboxes for processing in Rails. It ships with ingresses for Mailgun, Mandrill, Postmark, and SendGrid. You can also handle inbound mails directly via the built-in Exim, Postfix, and Qmail ingresses.

Basically, Action Mailbox can be used to forward all incoming emails to your Rails app and process it further as required like storing attachments, creating records from the email body in you database and many more.

And today, we will be implementing Action Mailbox with SendGrid.

Requirements

Setup Action Mailbox with SendGrid using the official Rails documentation
Update DNS records to forward emails received in the mailbox towards our Rails app
Test integration in development with built in UI provided by Rails
Test integration in development with NGROK to ensure seamless production release

Tested and working in

Ruby 3.0.0
Rails 7.0.2.4
Action Mailbox 7.0.2.4

You should have

Existing app built with Rails 7 or higher

Let's start integrating Action Mailbox with SendGrid in our Rails app now.

Step 1: Setup action mailbox

We will be following instructions from the Official Rails Guide for Action Mailbox.

Install migrations needed for InboundEmail and ensure Active Storage is set up:

$ rails action_mailbox:install
$ rails db:migrate

Step 2: Add Ingress Configurations

Tell Action Mailbox to accept emails from SendGrid by adding the following to both "development.rb" and "production.rb"

# config/environments/development.rb & config/environments/production.rb
config.action_mailbox.ingress = :sendgrid

Step 3: Generate Password for authenticating requests

First of all, we should generate a strong password that Action Mailbox can use to authenticate requests to the SendGrid ingress.

You can add any strong password or let Rails generate it for you. You can log into Rails console and generate a password for you:

> rails c
irb > SecureRandom.alphanumeric
# => "Kk9YGvzdPN69bfiu"

After that you can use rails credentials:edit in the command line to add the password to your application's encrypted credentials under action_mailbox.ingress_password, where Action Mailbox will automatically find it:

action_mailbox:
  ingress_password: YOUR_STRONG_PASSWORD

If you are using nano editor you can edit credentials with following command:

  $ EDITOR="nano" rails credentials:edit

Alternatively, you can also provide the password in the RAILS_INBOUND_EMAIL_PASSWORD environment variable.

If you are using figaro gem you can add the following to your "config/application.yml":

# config/application.yml

RAILS_INBOUND_EMAIL_PASSWORD: 'YOUR_STRONG_PASSWORD'

Step 4: Setup a Mailbox

Now we should setup a mailbox that will process all incoming emails through our Rails app.

You can generate a new mailbox with:

$ bin/rails generate mailbox forwards

This will create forwards_mailbox inside app/mailboxes

# app/mailboxes/forwards_mailbox.rb
class ForwardsMailbox < ApplicationMailbox
  def process
  end
end

Step 5: Whitelist email domains

We can configure our application_mailbox to accept all incoming emails to our Rails app and forward it to our forwards_mailbox for further processing.

Action Mailbox also accepts regex to whitelist domains or match certain emails. Let's look at how we can configure all these alternatives:

Accept all incoming emails

  # app/mailboxes/application_mailbox.rb
  class ApplicationMailbox < ActionMailbox::Base
    routing :all => :forwards
  end

Accept all emails from single domain

  # app/mailboxes/application_mailbox.rb
  class ApplicationMailbox < ActionMailbox::Base
    routing /.*@email-domain.com/i => :forwards
  end

Accept email from multiple domains

# app/mailboxes/application_mailbox.rb
class ApplicationMailbox < ActionMailbox::Base
  routing /.*@primary-email-domain.com|.*@secondary-email-domain.com/i => :forwards
end

This regex matching is telling application mailbox to forward all emails coming from @email-domain.com to our forwards_mailbox. For e.g. if we configure it to be /.*@gmail.com/i and our Rails app receives email to john-doe@gmail.com then it will be forwarded to our forwards_mailbox where we can further process it since this email matches with the pattern @gmail.com.

NOTE: Your mailbox name should match the name you've given it in the routing params i.e. forwards will route to forwards_mailbox.

Step 6: Test in development

Action Mailbox provides it's own set of UIs to test inbound emails in the development environment. To access this, let's fire up the Rails server first:

$ rails s

Visit Action Mailbox Inbound Emails Localhost URL and click on New inbound email by form. Fill in all required details like From, To, Subject and Body. You can leave other fields blank.

Before clicking on Deliver inbound email, let's add byebug (or any other debugging breakpoint e.g. binding.pry) to our process method so we know action mailbox is actually forwarding our emails to the right place.

# app/mailboxes/forwards_mailbox.rb
class ForwardsMailbox < ApplicationMailbox
  def process
    byebug
  end
end

You should make sure that email in From input box matches the email domain configured. Now when you click Deliver inbound email, the execution of the server process should stop at the process method since we have a breakpoint there. This means action mailbox is correctly forwarding incoming emails and our configurations are correct. You can perform further process as required in your app now.

But wait. Dang, there is an error from Rails while testing inbound emails in development!

Let's dig into what is happening.

Issue with Inbound Action Mailbox Testing in Development

Error reads: "undefined method 'original_filename' for "":String" and "NoMethodError in Rails::Conductor::ActionMailbox::InboundEmailsController#create"

Looking at the code in Action Mailbox of core Rails, I found out that this error is occurring because controller is trying to process the empty attachment further. But finding out why Rails was submitting empty attachment when we haven't chosen any attachment was hard. Note that, this also happens even if you choose one or multiple attachments.

In params, we get this "attachments"=>[""] and controller is trying to process it further with the following code:

private

def new_mail
  Mail.new(mail_params.except(:attachments).to_h).tap do |mail|
    mail[:bcc]&.include_in_headers = true
    mail_params[:attachments].to_a.each do |attachment|
      mail.add_file(filename: attachment.original_filename, content: attachment.read)
    end
  end
end

Here, we are getting error in the line mail.add_file(filename: attachment.original_filename, content: attachment.read) because "attachment" is empty string i.e. "" and not an object which has properties like "original_filename". Hence the error.

After looking into controller, my next stop for debugging the error was to look into the view because it shouldn't have sent the empty attachment in the first place.

View was just using a normal file_upload tag:

<div>
  <%= form.label :attachments, "Attachments" %><br>
  <%= form.file_field :attachments, multiple: true %>
</div>

There couldn't be any issue here, so I looked into the rendered HTML in the webpage and found out that there was a hidden tag for attachment:

<input name="mail[attachments][]" type="hidden" value="" autocomplete="off">
<input multiple="multiple" type="file" name="mail[attachments][]" id="mail_attachments">

Hence, the form is submitting empty attachment to the controller.

This problem could be solved in controller by filtering out attachments that are empty and I was near to submitting a PR to Rails. But then I thought, if I am getting this issue, there are obviously other developers who have been into this since this is an issue in Rails core and not in the code I have written.

Searching further, I found this issue titled Action Mailbox Conductor throws NoMethodError when creating inbound email submitted to Rails Core Github.

And if there is an issue, there must also be a PR. YES, there was one already titled Cannot deliver new inbound email via form but it hadn't been merged yet.

But for this tutorial and until PR is merged, we need this to work in our app. So, I was looking into how I can resolve it the best and searching for the solution that would work for all of us and not just me.

Scrolling further into the issue, I found a monkey patching very suitable for our use case.

Monkey Patching the issue

Add the following to your config/application.rb

# monkey patching to resolve the issue of action mailbox inbound email sending empty attachment
config.to_prepare do
  Rails::Conductor::ActionMailbox::InboundEmailsController.class_eval do
    private

    def new_mail
      Mail.new(mail_params.except(:attachments).to_h).tap do |mail|
        mail[:bcc]&.include_in_headers = true
        mail_params[:attachments].to_a.compact_blank.each do |attachment|
          mail.add_file(filename: attachment.original_filename, content: attachment.read)
        end
      end
    end
  end
end

Don't forget to restart the server and reload the page. After that you can submit the form again.

Voilà!! It works 🥳.

Now, your server should have stuck in the debugging breakpoint.

That's it, we have now successfully setup action mailbox and tested in development.

Now let's test using NGROK so we know that our configuration will work seamlessly (pretty much) in our production environment.

Step 7: Setup NGROK

Let's setup NGROK in our local machine:

Download the application

You can download the application from this download link.

If you are on MacOS, I highly suggest downloading NGROK using homebrew with the command brew install ngrok/ngrok/ngrok. It's easier than manual download and also don't normally give off any issue.
Serve your app using NGROK URL

While keeping the rails server running as it is, open a new tab in your command line.

You can then run the command ngrok http 3000, which will give you an URL connecting your local Rails app running on port 3000 to the internet. You should look at the URL besides the "Forwarding" option, it will be something similar to Forwarding https://e73a-27-34-12-7.in.ngrok.io -> http://localhost:3000

When running the NGROK, you should see a screen similar to the screenshot below:
Access the Rails app with NGROK URL

Open the URL you got before from NGROK e.g. https://e73a-27-34-12-7.in.ngrok.io in your browser and you should be able to see the Rails welcome screen or whatever your default page for the app is.

But, but, there is an error again 😭

Hah, don't worry. I have got you covered.

You should be seeing the Error UI similar to what is in the screenshot below:

This happens because of missing "auth token" which you can get after signing up to NGROK for free.
Sign up to NGROK

You can sign up to NGROK using this signup link.
Add NGROK auth-token to local configuration file

After signing up, you are presented with a dashboard and you can copy the auth-token from setup-and-installation step number 2 called "Connect you account"

Or you can follow this link to your auth token page.

Copy the token given and run the following in your command line:
```
$ ngrok config add-authtoken <your-authtoken>
```
Now restart your NGROK server and go to the new URL provided.

NOTE: URL changes each time you restart the NGROK server unless you use pro version and pay for the static URL.

What, Error? Again!!! 🤕
Resolving blocked host in Rails app

After accessing the NGROK URL, you should see an error page similar to the one below:

This is because Rails blocks https access in development and unauthorized URLs overall.

Let's add the NGROK URL to "config/environments/development.rb"
```
# config/environments/development.rb

config.hosts << "add-your-ngrok-url"
```
Restart the rails server and reload the page in the browser. Now, you should be able to see the default page of your Rails app. This is what I see in mine since it's a new application just for this blog:
Authorize all NGROK URLs in development

Above, we only allowed current URL provided by NGROK and as I have already said this URL changes each time we restart the server.

Changing it every time we restart the server is a hassle so we will add regex which will allow matching NGROK URLs to connect to our Rails app in development.

Let's replace previous configuration with the following:
```
config.hosts << /.+\.ngrok\.io/
```
Now, if you restart the rails server you should still be able to access the default page in your app. Also, try with restarting the NGROK server and accessing the page again which you should still be able to without getting blocked host error.

Step 8: Authenticate domain in SendGrid

You can follow the official SendGrid tutorial to authenticate your domain in SendGrid.

Part of the tutorial also goes through setting up MX records which we will go into detail here.

You can authenticate your domain by following this link to the Domain Authentication page

You will receive list of CNAME and Values similar to what is listed below in the process where prabinpoudel.com.np and em1181 will be different:

em1181.prabinpoudel.com.np
s1._domainkey.prabinpoudel.com.np
s2._domainkey.prabinpoudel.com.np

We will come back to this page after next step again so don't close the page yet.

Let's go to our DNS provider's dashboard and configure these records first.

Setup DNS Records

We need to add DNS records from SendGrid to our DNS provider so our email is actually being processed by SendGrid and routed to our Rails app with Inbound Parse Hook.

I use CloudFlare, so I will be showing you process to setup MX record using the settings from CloudFlare as an example.

Go to DNS tab from the left menu
Click on "Add Record" and choose MX from the dropdown then add the following values to each field

a. Name: "@"
b. Mail Server: "mx.sendgrid.net"
c. TTL: "auto"
d. Priority: "10"

You can also find the instruction for adding MX record in the tutorial to setup Inbound Parse Hook from SendGrid.
Click on "Add Record" again and add all three CNAME records we got previously while authenticating the domain one by one

Copy values from authenticating the domain page add them to CloudFlare:

a. Type: CNAME
b. Name: value from CNAME
c. Target: value from VALUE
d. Proxy Status: Turn the toggle button off (it will be on by default)
e. TTL: Auto
Go back to domain authentication page and click on "I've added these records" and click on "Verify" button

If everything was copied over correctly, you will see a page with the information "It worked! Your authenticated domain for prabinpoudel.com.np was verified."

Else you will get errors and you will have to fix those before moving forward.

Step 9: Configure Inbound Parse in SendGrid

We will be following the official SendGrid doc for configuring inbound parse hook to forward inbound emails to /rails/action_mailbox/sendgrid/inbound_emails with the username "actionmailbox" and the password we generated just before this.

From your SendGrid Dashboard click Settings, and then click Inbound Parse. You are now on the Inbound Parse page. Or you can click on this Inbound Parse Link to go there directly.
Click "Add Host & URL"
You can add/leave the subdomain part as required. I have left it blank because I don't have any subdomain just for receiving emails
Under "Domain", choose your domain name that you just verified
Under the URL we will have to construct one and add it

The format for the URL should be: https://actionmailbox:<your_action_mailbox_ingress_password>@<rails_app_nginx_url>/rails/action_mailbox/sendgrid/inbound_emails

For e.g. it will be https://actionmailbox:my_strong_password@5829-2400-1a00-b050-3fb6-b0ce-5946-b9be.in.ngrok.io/rails/action_mailbox/sendgrid/inbound_emails for my Rails app.

In production it can be a different URL so you should replace rails_app_nginx_url with the URL from where your Rails application is accessible to the internet.
Check "POST the raw, full MIME message" and click on Add

Now we are ready to test our integration with live email using SendGrid and Ngrok.

Step 10: Test if MX records are recognized by the internet

Before we test our integration with live email, we need to make sure that MX records are recognized by the Internet.

It may take some time for DNS records to be recognized throughout the world so email forwarding may yet not work for you. The maximum time period until this happens is 24 hours.

You can test if DNS records for your domain is working correctly and recognized from the website MX Toolbox

Add your domain name e.g. prabinpoudel.com.np
Click on "MX Lookup"

You should see "DNS Record Published" status in the test result table

Step 11: Test incoming email with SendGrid and NGROK

Finally, we are now at the last step. We will now send email to our mail server and we should receive it in our local Rails app and server should stop in our debugging breakpoint.

From your favorite email provider e.g. Gmail, send a test email to your domain e.g. for me I will test it via sendgrid-test@prabinpoudel.com.np.

It takes some time to process the email by SendGrid and receive in our Rails app, maximum ~1 minute.

You can check if the email is being received by SendGrid or not from Parse Webhook Statistics

Tada!! 🎉

You should have received the email and rails server must have stopped in the debugging breakpoint.

Conclusion

Congratulations!!! You have come a long way and gone through a lot of process to integrate Action Mailbox with SendGrid.

You can find a working app for this blog at Action Mailbox with SendGrid. You can view all changes I made for configuring SendGrid with Action Mailbox in the PR: Setting up Action Mailbox with SendGrid

Next, you can deploy the app to staging or production and add new Inbound Parse URL in SendGrid to point to the URL of those applications.

If you have any confusions, suggestions or issues while implementing any steps in this email, please let me know in comment section below and I will do my best to help you.

Thanks for reading. Happy coding and tinkering!

References:

Image Credits: Cover Image by erica steeves from Unsplash

Blog