Making MS Paint IDE

MS Paint IDE is a project that allows you to program with MS Paint. It is an IDE that runs alongside Paint (Or another drawing application) that is able to read created images with a custom OCR, and perform actions like a standard IDE can. This includes syntax highlighting, error checking, searching, replacing, Google Assistant support, and pretty much anything else my limited sanity allowed me to implement.

Why?

This project's idea originated on the popular Minecraft server software forum SpigotMC. People would joke about what IDE they used, saying they would program in MS Paint, because of the clear reason it's not an IDE nor even a text editor. Sometime in 2018, I put "Programming in Paint" in my ideas list, eventually taking a weekend to make a very basic Java Swing app from this. It could kind of read images in 12pt Verdana, highlight them, and execute/build them as Java programs, but that was enough for me to say it worked. Upon posting this to SpigotMC, people seemed to really like it.

So to make a long story short, I took the next logical step and spent the next 2 weeks on a major update to the project. Well, that is what I had planned. That quickly turned into a 10-month long journey of reading research papers developing a robust and precise OCR to read the images, work with new technologies I hadn't even heard of before, and talk to people developing real IDEs such as Eclipse. This small weekend project became my first large-scale application I can share with people to demonstrate my skills and progression as a developer, even in the year timeframe.

How is it built?

The IDE consists of several repos. The following is the repository breakdown:

• MSPaintIDE - The core of the IDE in Java
• PaintInjector - A C# program to add overlayed buttons to MS Paint to create the feeling of native buttons to control the IDE
• NewOCR - A trainable Java OCR to read computer-generated text, without the use of neural networks or libraries (for core functionality). Can be used in any project, and is on Maven Central.
• PaintAssistant - Handles Google authentication and communication for Assistant integration
• AssistantBackend - Code that runs on Firebase Functions to control Google Assistant devices

Each of these core projects involves things I haven't worked with before, from new languages to new technologies I had to learn for the end result.

One of the hardest pieces of the IDE was the OCR. Before the overhaul update, the OCR was integrated into the IDE and contained a picture of each character it could recognize, and just tried every possible combination to find their locations. This was excruciatingly slow and horrible for many reasons, so using something like Tesseract or another OCR library was my next task. These work great undoubtedly, even providing handwriting support, however they weren't perfect. By "perfect" I mean as precise physically possible, also working with non-alphanumeric characters used heavily in programming. If a curly bracket is mistaken for a square bracket, suddenly the IDE is rendered useless. Another key feature that I required was the ability to retain the pixels of each character to recolor them and produce a syntax highlighted copy of the image. Similarly, it also needed to be able to detect the exact font size to move and reproduce the text for refactoring purposes.

This dawned on me shortly after my hunt for the perfect OCR library, so I was driven to make my own. How it works is a much longer process, with a dedicated wiki documenting every exact step along the way, referencing code blocks for context.

What did I learn?

To remain abstract as possible, throughout the well over 1,000 hours I have put into this project the biggest thing I have learned is how to structure a project, and code in general. At its conception, the number of classes could have been counted on a single hand. Looking back I should have remade everything, as the biggest burden in the project is some of the old core code lurking in the depths of the repo, similar to making a new expensive house on a foundation of toothpicks. This realization can be demonstrated in NewOCR, where scalability and modularity were planned from day 1, making it easy to follow and use.

In reference to how the project was received by others, I also learned how different people take jokes differently. I am aware my sarcastic humor isn't for everyone, however, some people get pretty worked up about something like MS Paint IDE. Certain groups of people were genuinely angry that I was trying to use MS Paint to compete with industry-standard tools for programming, and that something like Notepad would be a "more logical choice", and it is "clear that Paint is not an IDE". I found long threads of comments like this pretty funny, and interesting that people seemed so polar opposite from each other, with some loving it like crazy and some absolutely hating even the idea of it.

Plans for the future?

This is a question I have debated since the last release around 8 months ago. There are so many areas of the IDE that could be worked on, so many things that could be fixed, and so many features that have been suggested to me by others. At the end of the last release I was quite relieved to be done, however, I'm eager to see what it could become. My current plan of action is to work on other things, growing my portfolio of weird projects. Even though it's been almost a year, I feel like I still haven't made up my mind of what I want to do with it, if anything at all.