How to integrate Queue in an Express application

To integrate a queue in an Express application, you can use a queueing library like Bull (which works with Redis) to handle background tasks and job processing. Here’s a step-by-step guide:

1. Install Bull and Redis

First, install Bull and Redis client libraries in your Express project:

npm install bull redis

Make sure Redis is installed and running. If you don’t have Redis installed locally, you can run it using Docker:

docker run -p 6379:6379 -d redis

2. Set Up a Redis Client Configuration

In your project/config folder, create a redisConfig.js file to configure the Redis connection.

const Redis = require('redis');

const redisClient = Redis.createClient({
  host: '127.0.0.1',
  port: 6379,
});

redisClient.on('error', (err) => console.error('Redis Client Error', err));
redisClient.connect();

module.exports = redisClient;

3. Initialize a Queue Using Bull

Create a queue using Bull. In the project/config folder, create a queue.js file.

const Queue = require('bull');

// Initialize a new queue
const exampleQueue = new Queue('example-queue', {
  redis: {
    host: '127.0.0.1',
    port: 6379,
  },
});

module.exports = exampleQueue;

4. Create a Queue Processing Function

In your project/services folder, create a file called queueService.js to handle processing tasks from the queue.

const exampleQueue = require('../config/queue');

// Process jobs in the queue
exampleQueue.process(async (job) => {
  // job.data contains the data passed when adding the job
  console.log(`Processing job with ID: ${job.id} and data:`, job.data);
  // Simulate a task, e.g., sending an email, resizing an image, etc.
  return `Processed data: ${job.data}`;
});

module.exports = exampleQueue;

5. Add a Job to the Queue from a Route

Create an endpoint in project/routes/queueRoutes.js to add a job to the queue:

const express = require('express');
const router = express.Router();
const exampleQueue = require('../config/queue');

// Route to add a job to the queue
router.post('/add-job', async (req, res) => {
  const { data } = req.body;
  try {
    const job = await exampleQueue.add(data);
    res.status(200).send(`Job added with ID: ${job.id}`);
  } catch (error) {
    res.status(500).send(`Failed to add job: ${error.message}`);
  }
});

module.exports = router;

6. Add the Route to the Express App

In project/app.js, import and use the queue routes.

const express = require('express');
const queueRoutes = require('./routes/queueRoutes');

const app = express();
app.use(express.json());

// Other routes...

app.use('/queue', queueRoutes);

module.exports = app;

7. Run and Test Your Application

Start your server:

node app.js

Use Postman or a similar tool to test the queue integration by making a POST request to http://localhost:3000/queue/add-job with JSON data:

{
  "data": {
    "task": "example task",
    "details": "some task details"
  }
}

You should see the job added and processed in the console logs.

8. Monitor the Queue (Optional)

For monitoring, you can use Bull Dashboard to track your jobs:

Install the Bull Board library:

npm install @bull-board/express

Set up the Bull Board in app.js:

const { ExpressAdapter, createBullBoard } = require('@bull-board/express');
const exampleQueue = require('./config/queue');

const serverAdapter = new ExpressAdapter();
createBullBoard({
  queues: [new BullAdapter(exampleQueue)],
  serverAdapter,
});

serverAdapter.setBasePath('/admin/queues');
app.use('/admin/queues', serverAdapter.getRouter());

Now you can navigate to http://localhost:3000/admin/queues to view and manage your jobs in the browser.

This setup integrates a queue into your Express app, allowing you to add and process background tasks easily. Let me know if you need further customization on this!

Blog

How to integrate Queue in an Express application

Le Quan Phat

Join Our Newsletter. No Spam, Only the good stuff.

Related