Foreword

I despise trading, whether for stock or crypto. Political manipulation I can see it is morally dubious but I do not care so much about. With that out of the way let"s build Dragonfly! A tool to get real-time sentiment analysis on entities (stocks, coins, political parties) from scraped chunks of text.

Overview of My Submission

The Dragonfly buzzes many riverbeds across the world. The Human will marvel at their flight, their colours, and see the creation of a God that gifts us Beauty in unsuspecting creatures.

But is the Dragonfly a Demon. Mid-flight will capture flies to devour while still alive. Their big glossy eyes have a peripheral view off all, that is under the wings. The Dragonfly Sees, the Dragonfly Hunts. Lean and pretty.

This project tries to give an infrastructure for real-time monitoring of websites on any domain where it is valued to know an opinion regarding an entity. In the basic example submitted to this hackathon, the system is built to monitor the sentiment regarding stock companies on subreddits and generate a timeline of events, those are, positive or negative comments on a certain entity.

The focus of the project is flexibility, wrapping all computationally intensive and domain-dependent elements (scrapers for websites or ML) in grpc microservices, that get called by consumers to populate an event driven architecture (some Kafka topics) that ends up dumping curated events to a MongoDB Time Series. This time series can then be the backbone for dashboards or opinion triggers build on top of it.

The MongoDB Time Series events collection is thought to be a historical archive of heterogenous events (sentiment of twitter/reddit comments on a company/crypto/party and the market fluctuations of the stock/coin or political polls). Both the easy-to-scale nature of the grpc micro-services and the asynchronous capabilities of kafka topics, backed by the consistency of MongoDB, build towards a real-time opinion temperature of an entity in this fast-moving internet, where a subreddit or telegram can x900% a crypto in hours or a twitter influencer can cascade a political opinion into virality.

In order to provide a small frontend I built a streamlit app, where you can call the api to scrape some of the best known stock-market subreddits and to view the opinion events added to the Mongo Time Series. As I have only had the pipeline ready for a few days and the entity-recognition grpc microservice is very simple (keyword matching for the tickets), not much data is still in the system.

I am sure there is some non-neoliberal use for this stack. Actually no, not that sure...

This was fun to learn some basics of event-driven architectures, still some parts of the structure I had no idea how to build and I am pretty sure they are quite cringe. Next steps involve general improvements, deploying it to build a juicy Time Series™, fine-tune some NER and SA models to different domains and to do entity-conditioned SA.

Submission Category: Prime Time

Link to Code

Additional Resources / Info

Epic UI for Kafka Topics: