This is a submission for the Open Source AI Challenge with pgai and Ollama
What I Built
I developed a fully local, AI-powered Knowledge Base Management System that enables users to upload documents and interact with them through RAG (Retrieval-Augmented Generation). This system harnesses the capabilities of PostgreSQL’s AI extensions alongside Ollama's local models, creating a privacy-centered solution that operates entirely on your own infrastructure.
Key Features:
-
Fully Local Operation
- All AI processing occurs directly on your machine, powered by Llama 3.1
- Ensures that no data leaves your infrastructure, providing complete privacy and control.
-
Intelligent chat interface with RAG capabilities
- Uses Retrieval-Augmented Generation to deliver accurate responses.
- Vector similarity search is driven by pgvector, with efficient querying through pgvectorscale.
-
Function Calling
- Supports extensible operations for expanded interaction capabilities.
-
Modern Web Interface
- Features a clean, intuitive UI designed with shadcn/ui.
- Enables real-time chat interactions with integrated file management.
Demo
In this demo, users can select multiple PDFs and receive answers based on the content. Here, the hackathon announcement is used as a source. Additionally, users have the option to manage files within the app.
User also has option to manage files in the app
Below is the chat interface, where users interact with the documents.
GitHub Repository:
Next.js AI Knowledge-base chatbot
An fully local Open-Source AI knowledge made with Next.js and the AI SDK
Powered by Ollama and Postgres
Features ·
Model Providers ·
Deploy Your Own ·
Running locally
Features
-
Next.js App Router
- Advanced routing for seamless navigation and performance
- React Server Components (RSCs) and Server Actions for server-side rendering and increased performance
-
AI SDK
- Unified API for generating text, structured objects, and tool calls with LLMs
- Hooks for building dynamic chat and generative user interfaces
-
shadcn/ui
- Data Persistence
-
Postgres for saving chat history and user data
-
Min.io for efficient file storage
-
NextAuth.js
- Simple and secure authentication
-
Ollama for AI model management
- Easily switch between different AI models and providers
-
pgvector for vector similarity search
- Efficiently store embeddings for similarity search
-
pgvectorscale for scaling vector similarity search
Model Providers
…
Tools Used
The project leverages several powerful open-source tools:
AI and Database:
- PostgreSQL as the primary database
- pgvector for storing vectors
- pgvectorscale for similarity search
-
Ollama running
-
Llama 3.1 for chat
-
mxbai-embed-large for embeddings
Frontend and Backend:
- Next.js 14 with App Router
- Vercel AI SDK for chat interfaces
- shadcn/ui for component design
- Tailwind CSS for styling
- NextAuth.js for authentication
- Min.io for file storage
- Drizzle as ORM
Final Thoughts
Building this project for the Open Source AI Challenge was a fulfilling journey. The most challenging yet rewarding aspect was enabling it to run entirely locally, offering a significant data privacy advantage.
Using pgvector and pgvectorscale made the RAG implementation seamless.
The pgai Vectorizer currently supports only OpenAI embeddings. While I initially aimed to integrate it with Ollama, the absence of a compatible tokenization endpoint made this integration unfeasible. As an alternative, I implemented the embedding logic directly.
Prize categories:
- Main Category
- Open-source Models from Ollama
Future Improvements
- Fine-tune prompts for better tool interaction
- Add support for more document formats
- Implement multi-model support
- Optimize vector search performance
- Add batch processing capabilities for large document sets
