Build Bot Web.js to Extract Data from Shopping Receipts

In this article, we will discuss how to use WhatsApp Web.js to create a WhatsApp bot capable of extracting information from shopping receipts sent by users. This bot leverages Google Vision to extract text from receipt images and then parses the items purchased and the total amount from the extracted text.

*Step 1: Setting Up the Environment
*
Ensure you have Node.js and npm installed on your computer. Then, create a new directory and initialize a Node.js project.

mkdir whatsapp-receipt-bot
cd whatsapp-receipt-bot
npm init -y

Install the necessary dependencies:

npm install github:pedroslopez/whatsapp-web.js#webpack-exodus qrcode-terminal

*Step 2: Configuring the WhatsApp Client
*
Create a .env file to store environment variables:

Create the index.js file and add the following code to initialize the WhatsApp client:

require('dotenv').config();
const { Client, LocalAuth, MessageMedia } = require('whatsapp-web.js');
const qrcode = require('qrcode-terminal');
const crypto = require("crypto");
const path = require("path");
const fs = require("fs");
const { extractText } = require('./services/visionService');
const { extractReceiptData } = require('./services/receiptService');

const client = new Client({
    authStrategy: new LocalAuth(),
    puppeteer: {
      headless: true,
      args: ["--no-sandbox", "--disable-gpu"],
    },
    webVersionCache: {
      type: "remote",
      remotePath:
        "https://raw.githubusercontent.com/wppconnect-team/wa-version/main/html/2.2412.54.html",
    },
});

client.on('ready', () => {
    console.log('Client is ready!');
});

client.on('qr', qr => {
    qrcode.generate(qr, { small: true });
});

client.on('message_create', message => {
  const chatId = message.from;
  const messageBody = message.body.trim();

  if (message.body === '!ping') {
    client.sendMessage(message.from, 'pong');
  }

  if (message.body === "help") {
    handleHelp(message);
  }

  if (message.hasMedia && !message.fromMe) {
    handleMediaMessage(message, chatId, messageBody);
  }
});

client.initialize();

*Step 3: Handling Messages and Media
*
Add functions to handle incoming messages and media:

async function handleHelp(message) {
  message.reply(`Steps to use our service: 
    1. Send a photo of your shopping receipt.
    2. Wait a few moments while we process your image.
    3. You will receive a list of items and the total amount from the receipt.

    Available commands:
    - To view your scan history: type "history"
    - To view your active session: type "get_session"

    Please send a photo of your receipt now to start!`);

    const tipsPhotoPath = path.join(__dirname, 'public', 'assets', 'tips1.png');
    const media = MessageMedia.fromFilePath(tipsPhotoPath);
    await client.sendMessage(message.from, media, {
      caption: "Tip: Make sure the receipt photo is clear"
    });
}

async function handleMediaMessage(message, chatId, messageBody) {
  message.reply("Processing image and extracting data from the receipt...");
  const media = await message.downloadMedia();
  if (media) {
    const filename = generateRandomFileName(chatId, media.mimetype);
    const filePath = path.join(__dirname, "public/uploads", filename);

    fs.writeFileSync(filePath, media.data, "base64");

    try {
      const text = await extractText(filePath);
      const receiptData = await extractReceiptData(text);

      const responseMessage = `📋 *Receipt Data:*
      ━━━━━━━━━━━━━━━━━━━━
      ${receiptData.items.map(item => `- ${item.name}: ${item.price}`).join('\n')}
      ━━━━━━━━━━━━━━━━━━━━
      *Total:* ${receiptData.total}
      ━━━━━━━━━━━━━━━━━━━━`;

      message.reply(responseMessage);

    } catch (error) {
      message.reply('An error occurred while processing the image. Please try again.');
      console.error(error);
    }
  }
}

generateRandomFileName = (chatId, mimetype) => {
  const extension = mimetype.split("/")[1];
  const randomString = crypto.randomBytes(16).toString("hex");
  const timestamp = Date.now();
  const fileName = `${chatId}-${randomString}-${timestamp}.${extension}`;
  return fileName;
};

Step 1: Setting Up the Environment
We start by creating a new Node.js project and installing the required dependencies:

whatsapp-web.js: A library to automate WhatsApp Web.

qrcode-terminal: A library to display QR codes in the terminal.
dotenv: A library to manage environment variables. 3.googleapis: A library to interact with Google APIs, including Google Vision for text extraction. 4,We initialize a Node.js project and install these dependencies to set up our environment.

Step 2: Configuring the WhatsApp Client
We configure the WhatsApp client using whatsapp-web.js. The client uses LocalAuth for authentication, which saves session data locally to avoid re-scanning the QR code on every restart.

We handle the QR code generation, client readiness, and message creation events. When a QR code is generated, it is displayed in the terminal. When the client is ready, a message is logged. For every new message, we check if it's a command or contains media.

Step 3: Handling Messages and Media
We define functions to handle different types of messages:

handleHelp: Sends a help message with instructions on how to use the bot.
handleMediaMessage: Processes incoming media messages, extracts text from the image, and parses receipt data. 3.In handleMediaMessage, the media is downloaded, saved locally, and then processed. The extractText function extracts text from the image using Google Vision.
The extractReceiptData function parses the text to extract item names and prices, as well as the total amount.

Step 4: Extracting Text and Data from Receipts
We create services for extracting text from images and parsing receipt data:

visionService.js: Uses Google Vision to perform text detection on the uploaded image.
receiptService.js: Parses the extracted text to identify items and their prices, as well as the total amount.

Blog

Build Bot Web.js to Extract Data from Shopping Receipts

Jennifier Syhne

Join Our Newsletter. No Spam, Only the good stuff.

Related