Author: Jason Corso (Professor of Robotics and EECS at the University of Michigan | Co-Founder and Chief Science Officer @ Voxel51)

“Will FiftyOne always be open source?” I must have heard this question two or three times a day over the last few years at Voxel51, where we create the software refinery that empowers machine learning scientists and engineers—whom I call AI/ML specialists in this article—to build highly performant visual AI systems. Our software, called FiftyOne, is open source and has been since its category-defining first release in August 2020. Fast-forward four years and more than two million installs, we’ve evolved into the leading software for dataset and model building, empowering myriad AI/ML specialists across all aspects of Visual AI.

A typical follow-up question is often, “But then how will you make money?” In fact, we do make a powerful Enterprise version of FiftyOne that is used by hundreds of customers. But, this article is not about Enterprise software or making money!

Instead, I wanted to discuss the AI/ML work we do at Voxel51, which is contained in the FiftyOne Brain package, and, perhaps more importantly, why we open source it all. Originally, the FiftyOne Brain was held as closed source, while the main components of FiftyOne were open source. Yet, after careful consideration, we are now fully open sourcing the Fiftyone Brain as well.

This article explains the AI/ML software journey we’ve taken at Voxel51, from these initial steps of building the best-in-class AI/ML developer software to how we are now formally open sourcing all of the AI/ML work we’ve ever done at Voxel51. Our AI is open source AI.

FiftyOne: A Flexible Toolbox for AI/ML Development

Let’s first level-set FiftyOne: “[It] provides the building blocks for optimizing your dataset curation and model building pipelines.” This succinct quote from our docs captures the essence of how we bring value to FiftyOne users.

From our own experiences building high-performing visual AI systems, we knew well that AI/ML specialists struggle with the challenges of curating high-quality datasets and analyzing data-and-model performance. So, from the first line of code we wrote in FiftyOne, we sought to build a flexible software solution that would provide the “picks and shovels” every AI/ML specialist needs to build performant systems. Importantly, we recognize the immense challenge of building performant AI/ML systems. It’s not hard to build an AI/ML system that achieves reasonable performance because today’s models and public datasets can get you most of the way there, but getting to five-nines-level performance takes a huge amount of work and the right tools.

FiftyOne sits at the heart of those tools; since its first release, it has increasingly solidified as a flexible toolbox of inbuilt and extensible capabilities.

Curiously, despite FiftyOne’s role in supporting the AI/ML specialists, there has been surprisingly little actual AI/ML code in its codebase thus far. Guided by our own experiences with AI/ML work and the exceptional feedback we get from our user community, we emphasized the core scalable data model, fast and concise query engine, and flexible front- and back-end user experience.

As a laser-focused startup, this was intentional: our north star was to enrich the AI/ML developer ecosystem with flexible building blocks, such as embeddings exploration and model evaluation capabilities, that everyone can use to build better AI/ML faster, rather than boxing in AI/ML specialists to one way of working.

However, over the last few years, as our user base grew to millions of installs, we had the opportunity to hear from more and more community members about new struggles they faced with their everyday AI/ML work happening outside of FiftyOne. AI/ML work is challenging, sometimes tedious, and often difficult to diagnose the cause of certain system behaviors. As we realized many of these struggles could be addressed by new AI/ML capabilities within FiftyOne, we began to enrich FiftyOne with certain AI/ML components:

• FiftyOne Brain: a component of FiftyOne that implements the AI/ML capabilities enabling users to more effectively build and analyze datasets and models in smart and intuitive ways. It provides the ability to perform deep analysis on datasets to improve model performance with features like visual similarity search, query by text, finding unique and representative samples, finding mistakes, and more.

• Model and Dataset Zoos: components of FiftyOne that collect the most important openly available models and datasets across visual AI/ML, representing them in easy to use formats that are directly pluggable into everyday workflows. These easy-to-use models and datasets are often the starting point for FiftyOne users, as they work through new ideas and new projects.

• Diverse tutorials: a broad collection of AI/ML-oriented workflows within FiftyOne, including, for example, how to evaluate a classifier, how to use image embeddings, how to augment a dataset with albumentations, and more.

• Dozens of AI/ML-focused plugins: FiftyOne is insanely extensible.

Yet, as we planned our future strategic directions over the last year, we have come to realize that although FiftyOne has matured in capabilities and the community has grown significantly, this is but the tip of the iceberg. Even, or perhaps especially, in the face of the rapid advances we’ve seen in AI/ML over the last few years, the challenge of building production-ready AI/ML systems is no simpler.

The real challenge in building high-quality production-ready AI/ML lies in understanding distributions of data in the context of model performance. From the cold-start phase to tracking distribution shift, there are more questions than principled answers. It’s a quagmire of arcane arts and wishful workflows, both at the researcher and the practitioner level.

At Voxel51, we see this as a massive opportunity. We are deeply invested in new AI/ML work to transform this arcane quagmire into easily actuated and flexible workflows, and our open source core is at the bedrock of this innovation.

Open Source AI Unlocks Innovation, Safety, and More

Few disagree that there is a huge potential for AI broadly to impact how to live and work. And this includes broad classes of AI, not just the more recent and popular generative ilks. With tech potential comes opportunity and the omnipresent debate of open vs. closed source approaches.

At Voxel51, we believe the benefits of open source AI—real open source AI, not lip-service—greatly outweigh the risk. Some of these benefits include the following.

• Collaboration and Innovation: Open source AI democratizes access to powerful tools and resources, enabling researchers, developers, and startups to collaborate and build upon each other’s work. This accelerates innovation in AI/ML by lowering entry barriers and making cutting-edge models accessible to everyone. Furthermore, open sourcing AI models, data, and tools reduces redundancy in AI research and development, unlocking the potential for even faster progress.

• Safety through Transparency and Trust: By making the underlying code and data open, AI systems become more transparent, which helps build trust and accountability. Open source models allow for independent scrutiny and auditing of the system’s behavior and ethics. Open source data enables the thorough investigation into privacy and bias concerns behind an AI/ML system. In contrast, proprietary AI/ML systems often obscure how decisions are made and how models operate. Open source AI mitigates the risks of hidden biases, ethical concerns, and opaque performance by making everything publicly available for review.

• Customization and Adaptability: Open access to AI/ML systems enables developers to tailor models for specific needs and niches, fostering more diverse applications across industries and domains that might otherwise not have the resources for proprietary models, without the risk of a foundation being pulled out from underneath their work.

• No Vendor Lock-In: Open source AI helps avoid reliance on proprietary models and platforms that can trap organizations into specific ecosystems. By using open systems, users retain full control over their AI/ML development and deployment. This challenge became more apparent this past summer when OpenAI announced the end of life for GPT 3.5.

Open Source AI Enables Sustainable AI with Ethical Governance

Beyond these immediate technical and business impacts, open source AI is sustainable, ethical AI. Open source AI empowers researchers and students globally, fostering the next generation of AI talent. It’s obvious that wide access can result in faster discovery and more diverse problem-solving approaches, but more importantly, it cultivates an international technical community based on understanding, interoperability, and communication rather than secrecy, marketing, and bias. This openness is especially important given the broad range of potential ethical impacts from AI, including privacy, accountability, job displacement, international relations, and more.

Equally important, as more AI tools and frameworks emerge, open source AI enables the community to work toward standardizing interfaces and improving interoperability, facilitating seamless integration and more cohesive AI ecosystems. Proprietary technology often leads to lack of standards, even at the very simplest levels such as text messaging, ultimately leading to duplicative work and competitive races to release things we do not understand. In contrast, open source AI offers a pathway to more ethical AI development by enabling public oversight, community-based governance, and collaborative standards for safety, privacy, and fairness in AI systems.

Open Source AI at Voxel51

Importantly, as we make these advances they will be open sourced. In fact, as of the release of this article, we are formally open sourcing any and all of the AI/ML work we’ve ever done at Voxel51. This is super exciting for us, as we seriously believe in the value of an open exchange of developments in research, especially AI/ML research considering the wildly unknown performance and safety surface of our rapidly advancing field.

As you may know, I contributed a comprehensive definition of Open Source AI in a February 2024 article. In this definition, I call out seven key pieces necessary to be able to declare something Open Source AI. Let me summarize these here.

1. System Source Code: Implements the key functionality of the AI system, including model architecture, input/output handling, and deployment code, ideally with accompanying documentation.

2. Model Parameters: The trained model weights and necessary configuration parameters to fully deploy the system.

3. Dataset: The raw data and annotations used for training, as well as any validation data for model selection.

4. Hyperparameters: Configuration values or ranges, such as learning rate, needed to train the model.

5. Training Source Code: Transforms the dataset, hyperparameters, and system source code into the trained model parameters.

6. Random Number Generation: The random number generator and seed information required to reproduce certain AI sessions.

7. Software Frameworks: The libraries and frameworks, with exact versioning, necessary to run the system and training source code.

The key point in this comprehensive definition is that AI/ML is not open source until one can reproduce the entire process from soup to nuts, not just run or even fine-tune a downloadable model. To do that, one needs code, data, seeds, hyperparameters, and specs. Anything less is in violation of the core tenets of open source. If you release model weights, then stand up and say so, like Mistral AI does with its “Build with open-weight models” messaging, rather than claiming it is really Open Source AI.

At Voxel51, we believe in Open Source AI. It is critical for the proper study, understanding, and advancement of the field. Our AI is Open Source AI. One key example of how Voxel51 supports Open Source AI is how we integrate FiftyOne with Hugging Face to enable one-line-of-code full dataset sharing, which we’ve used to provide easy access to 50+ datasets.

Closing Remarks

Our AI is open source, and that’s how we believe the future of AI should be built. Open access to code, models, and datasets fosters collaboration, transparency, and rapid innovation, allowing anyone to contribute to and benefit from AI advancements. By fully embracing open source AI, we not only unlock technical breakthroughs but also build a more ethical and inclusive AI future. At Voxel51, this isn’t just an experiment—it’s a commitment. Check out the 1.0 milestone release of open source FiftyOne and contribute to the open sourced FiftyOne Brain today!