Modr8 — The whitelabel Telegram group management engine

Introduction

This post was originally shared on medium but Dev.to has been my home for tech reading now for quite a while so I wanted to show off the product we've built in the meantime. This posts will offer an insight into how the whitelabel telegram community moderation engine was born. This is the first, in a series of posts which will explain how we work with Telegram communities and also monitor the advancement of spammers/scammers methods to develop our product.

Lets begin…

Inspiration

Whilst being involved in Crypto projects during the 2017 bullrun we noticed the amount of Telegram based communities exploded with often tens of thousands of users all chatting within project communities and groups. It didn’t take long before scammers and spam became a major route to users being exploited and communities turning into cheap forums for bot based advertising.

With a background in Telegram Bot development, even back in the early beta days of the first API release by Telegram.org we set about automating the removal of suspicious media, user bots and also providing moderation commands for admins of chats to manage their community.

In the early days, this started out as small project with an individual bot built to the custom requirements of each community who needed our help. Over time though, we realised that this process would not scale and often the reason for using our services were to have a “personalised” and “on brand” bot that looked as though the community/project had developed the bot itself, without advertising another brands services in their chat. We decided it was time to scale up and offer a platform where users could generate a bot token utilising the official Telegram Bot registration process and then register this within our platform where our engine would handle everything else. This would give the scale and also self service management that many communities need.

From here the brand was born, Modr8 the white-label community bot platform. We ensure you have all the features needed to manage telegram communities, without having to detract from your own brand presence.

What the platform does

Through direct customer feedback and identifying the shortcomings of many free or open source projects we have developed a bot engine which handles everything from simple user management right through to polls, referrals and analytics.

Why personalised bots?

There are tens of bots out there which handle community management, we are well aware. However, these bots all have one thing in common, they all utilise the same Bot token and therefore all call the Telegram API under the registered token and username. Countless times we have seen these bots become unresponsive as they simply can not handle the inbound processing of messages as well as then sending all the outbound responses to the Telegram API. In no time at all you are faced with a flood or API limitation being put against the token and the bot no longer receives messages. If you are in the game of preventing spam and managing your community using your bot, the system has now been rendered useless.

In order to reduce the chance of this happening to our platform, we specifically chose not to go in the direction of simply producing a web platform which allows the management of a single generic bot under a single token that we own. We want users to “bring their own tokens” ensuring they can do 2 things:

1. Personalise their bot profiles using @BotFather to look, feel and be controlled as they wish.
2. Provide unique tokens which means every message they process is only their own, vastly reducing the chance of hitting API limitations.

All sounds great.. but honestly what does it do?

A breakdown of core functionality is listed below:

Fully customisable bot profiles, your token so your bot profile.

User management commands such as bans, kicks, mutes and warnings

Custom command manager allowing you to develop your own commands with your own content. If you want to create a command to show a list of your staff, or a link to your website it can all be done here.

Chat & User Analytics including everything from message sentiment through to most popular commands and active users.

Native Polls allowing you to issue and track the responses from polls created in the platform and issued to your chat by the bot.

Announcements send one off or scheduled and interval based announcements to your chat

Spam prevention turn on and off granular rules to detect and remove spam from your channel.

Referrals & Reputation full modules that allow you to generate referral links for users to invite others to your chat, including full tracking and leaderboards. We also have a reputation system with custom rankings allowing users to say thanks and gain levels in the leaderboards.

API we are the only platform to offer an API which has been utilised by a number of communities already to link to their internal systems, including swag giveaways and gift stores.

Translations responses to messages are now being translated and within the platform settings you can choose from a number of languages really making the bot fit into your community.

… there’s much more too!

How its been built

Modr8 started as a humble single node.js file utilising one of the many Telegram bot API npm packages for message inbound interactions and executing outbound instructions. Each bot had a JSON settings file and it was very much a single project for a single community.

In order to scale, a number of changes had to be made.

Splitting apart the mono project into:

Rules engine: Keeping with Node.js here, but each engine deployed inside docker containers allowing the engines to scale horizontally to keep up with message consumption.

Message producer: The consumer is a simple express App which receives webhook events from the Telegram API, these are then fed directly into the message queue along with parsing the Bot token allowing for the consumers to identify the bot messages they are processing.

Message queue: We spent a lot of time deciding whether to go with Kafka, ActiveMQ, Redis, RabbitMQ or a simple memory based message queue. In the end we fell for RabbitMQ mainly due to us wanting our consumers being “dumb” and therefore not needing too much logic other than a connection to the queue. Kafka is still on our roadmap, but I don’t believe we have hit any limitations in RabbitMQ to need the move yet.

Scheduler: This component similar to the engine is concerned with working with scheduled events. Whilst the rules engine reacts to in the moment inbound events the scheduler is looking for those which require a delay or specific execution time. This is built in node and right now simply polls the database for required events. This is one of the areas we are looking to improve, but right now sufficiently meets requirements for welcome message timeouts, scheduled announcements and clean up of messages marked for future deletion.

Database: At first the project started out with MongoDB purely as it had the easiest migration path from the JSON settings we had in the mono projects before. It also helped us to reduce the workload on data-modelling whilst discovering requirements as the project built out. Mongo is great for rapid prototyping and working with a flexible schema, it definitely hit the sweet spot for us during such heavy changes in the early days. We have however now moved to Postgres, mainly due to the consistency and capacity to handle heavier read/write scenarios. There is some usage of JSON documents within Postgres for a few flexible elements of our model but the vast majority of the platform is pretty well defined now and data structures are known.

Cache: Without a doubt here we have always defaulted to Redis for our cache needs. At one point we were looking at Redis to also handle our pub/sub and queue needs however as Rabbit won that battle we still had Redis supporting caching so it features regardless. The main usage for redis is storing some state for the rules engines, but also caching many of the costly/time consuming API calls that may occur multiple times during the lifecycle of a message being processed. An example of this is the Admin list in the community groups. The admins rarely change, so calling the API every message to see if the user has access to a rule or command is costly. Infact, there are some very strict (but also largely unknown) API limitations which we could easily hit if the chat were particularly busy.

Websites: the main www domain is serving a static site hosted by netlify. We love that service so much it's an amazing product and also powers our contact form without any hassle. The main platform domain is built on the sails.js framework. The simple MVC architecture is all that we needed to get running and sails has a really easy project structure to follow.

Reverse Proxy: We balance workloads amongst the rule engines using rabbit work queues but still we needed a reverse proxy for the platform website and consumer. Since all of the components apart from our www site are in docker containers we found traefik to be a great fit. The auto discovery and easy management of rules has been a perfect fit. Another highly recommended product if you are looking to remove the burden of nginx and haproxy configuration files when working with containers.

Challenges

One of the biggest challenges faced has easily been the switch from single silo projects to the generic platform we have today. It’s one thing personalising a single bot with custom actions, but then moving to a platform model where processes are decoupled, split and also offer the same flexibility for users to make changes and “personalise” the bot experience is a large leap.

With that being said, the pay off is huge and it makes long term plans much easier to attain. As an example we are now translating the bot into various community languages offering a true personalised experience for that locale which has been a major hit with our user base.

Accomplishments

We feel that the biggest accomplishment is offering a service which has truly hit our brief, we wanted to offer something which others in the market were not and that is the elements of “whitelabel” “saas” and “personalised” community management bots. When we receive feedback from our own community it is always humbling to see they notice the effort placed on quick turn around and focus on usability. There’s a few competitor services which are clunky to use and navigate, we have tried to make everything simple for the users to understand and, with that we have seen our bots embedded deeply into the communities and the platform has grown in popularity by recommendation rather than hard marketing.

What we’ve learnt

There have been a number of learning points, some of them have been related to code, but most of them have been around listening to customer feedback. The product we have today has had huge influence from our customers who are open and honest around specific issues they have faced managing communities, especially those who have a large user base and may need cross chat management. This really took the platform from its humble beginnings as a 1 bot 1 chat product, to now being built for enterprise usage and we can happily say we have a number of customers effectively managing chats running into 100k membership across multiple chats.

What’s next for Modr8

We’ve just released chat referrals which allows a user to generate a unique link via the bot and invite others to the community. Surprisingly this has been way more popular than we expected and has already been picked up heavily by the Cryptocurrency projects, especially exchanges who already have many thousand referrals to their chats and run competitions. This took quite a lot of work but it’s been great to see the feedback from customers.

For the future we have a backlog of items suggested by our customers, some of these are minor tweaks but the big hitters are:

Public polls: Right now we offer native telegram polls via the bot but these are anonymous by default. We collect the responses and display a nice historical table of the results but it would be great if we could offer the same functionality as the @PollBot or @Vote. This is a relatively simple change and will be coming in the next release.

Support Features: A long term goal has been to offer a F&Q or support type of feature to the engine. This would allow a community (especially those who have a product behind them) to offer a two way chat support feature in the bot. Much like we see on websites with chat bots, we feel there is a play to be made here in telegram too.

External Feeds: In the early days we did offer extended announcement support which allowed platform users to add their twitter accounts, medium account handles so that any new content would also be posted within the chat. Changes to the twitter API and also the number of feeds to subscribe to pushed this down the list of features to migrate over from the Alpha version. It’s something we want to add in as we see many community admins cross posting content which we can automate for them.

Want to get started?

Its easy to get started with the platform, we offer a free tier of 100 members of less so you can try out all of the features in a small chat before activating a bigger community.

Feel free to check out our website https://www.modr8.net

If you wish you could jump directly to the platform here: https://platform.modr8.net

Our support chat is always open with users who know the system inside and out to offer help with any questions: https://t.me/modr8support