If you follow the news, you may have heard about this very controversial technology from Microsoft called Recall.
As a developer working primarily within the Mac ecosystem, I usually steer clear of Windows-centric commercial tools. So I decided to see if there was an open source alternative. I did some research and I came across an open source alternative. came across an open source project called OpenRecall - a direct alternative to Microsoft's newly launched Recall feature. Determined to understand its inner workings, I delved into its extensive documentation. This blog post serves as a detailed exploration of OpenRecall, ensuring that no aspect of its architecture and functionality is overlooked.

Disclaimer: This blog post is a detailed exploration based on the OpenRecall documentation provided. It aims to inform and educate without any promotional intent. You can decide for yourself whether this is a viable alternative to commercial products, including Microsoft Recall.

Understanding Microsoft Recall

Before diving into OpenRecall, it's crucial to comprehend the feature it seeks to replicate. Microsoft Recall is designed to enhance user productivity by enabling the search of locally saved and analyzed screen snapshots using natural language. Here are its core components and functionalities:

• Local Processing: All snapshots are stored and processed locally on the device, ensuring data privacy and security.
• Natural Language Search: Users can query their screen history using everyday language, making retrieval intuitive.
• Privacy-Focused Architecture: Incorporates robust security measures, including encryption and user consent for snapshot storage.
• IT Admin Controls: Administrators can manage the availability and behavior of Recall within organizational environments, ensuring compliance with corporate policies.

Recall (preview) is available to Copilot+ PCs through the Windows Insider Program, supporting select languages and adhering to specific system requirements. It emphasizes privacy by ensuring all data remains on the local device, with no cloud dependencies.

Introducing OpenRecall: An Open-Source Memory Keeper

OpenRecall is an intelligent system designed to capture and make searchable your digital activities. Its architecture is meticulously crafted to ensure modularity, maintainability, and efficiency. Below is a comprehensive breakdown of its system architecture and components.

System Architecture at a Glance

The system is divided into four distinct layers:

1. Capture Layer: Manages screenshot capturing and monitors user activity.
2. Processing Layer: Handles text extraction and semantic processing.
3. Storage Layer: Manages data storage, both structured and unstructured.
4. Web Layer: Provides a user interface for interacting with the captured data.

Component Architecture

1. Capture Layer

The Capture Layer is responsible for continuously monitoring screen activity and capturing meaningful snapshots. Its key functionalities include:

• Screen Capture: Utilizes the MSS library for efficient screen capturing.
• Similarity Detection: Implements a similarity checker to avoid redundant snapshots, ensuring only significant changes are recorded.
• Optimized Storage: Saves images in the WebP format to balance quality and storage efficiency.
• Activity-Based Recording: Captures screenshots primarily during active user periods, minimizing unnecessary data capture.

Real-World Example: While coding, OpenRecall intelligently captures snapshots during natural pauses instead of every keystroke. This approach conserves storage and maintains a coherent history of your work sessions.

2. Processing Pipeline

The Processing Layer handles the extraction and semantic processing of text from screenshots:

• Optical Character Recognition (OCR): Utilizes DocTR OCR to extract text from captured images.
• Natural Language Processing (NLP): Processes the extracted text using Sentence Transformers to generate semantic embeddings.
• Semantic Embeddings: Converts text into vector representations, enabling context-aware search functionalities.
• Efficient Storage: Stores these embeddings in a SQLite database as binary blobs for quick retrieval.

Use Case: Imagine researching multiple topics across various browser tabs. Instead of recalling exact phrases, you can search using related terms, and OpenRecall will fetch relevant snapshots based on semantic similarity.

3. Storage Layer

The Storage Layer manages both structured and unstructured data:

• SQLite Database: Stores metadata and semantic embeddings.
• File System: Stores compressed WebP screenshots.
• Timestamp Linking: Associates database entries with their corresponding screenshots using timestamps.

Practical Application: Whether tracking coding sessions or design iterations, OpenRecall organizes data chronologically and semantically, facilitating easy reconstruction of workflows.

4. Web Interface

The Web Layer offers an intuitive interface built with Flask, facilitating user interaction with captured data:

• Timeline View: Allows chronological navigation through screenshots.
• Semantic Search: Enables context-aware search using embeddings.
• Modal Views: Provides detailed examination of individual screenshots.

Example Workflow: During a design review, navigate to specific day's snapshots or search for discussions around a particular design element, streamlining the review process.

Data Flow

The Data Flow within OpenRecall ensures seamless capture, processing, storage, and retrieval of screen snapshots:

1. Input:
2. Screen State: Captured by the Screenshot Capture module.
3. User Activity: Monitored to determine active periods for capturing screenshots.
4. Processing:
5. OCR Processing: Extracts text from new screenshots.
6. NLP Processing: Generates embeddings from extracted text.
7. Storage: Saves embeddings in the database and images in the file system.
8. Output:
9. Web Server: Retrieves data from the database and file system.
10. User Interface: Renders the timeline and search results for user interaction.

Real-World Scenario: While researching a complex topic involving multiple applications, OpenRecall captures screenshots, processes text for semantic search, and organizes data for easy retrieval, enhancing productivity without manual effort.

System Optimizations

OpenRecall incorporates several optimizations to ensure smooth and efficient operation:

• Storage Efficiency:
• WebP Compression: Reduces image file sizes without significant quality loss.
• Binary Blobs: Stores semantic embeddings efficiently.
• Incremental Data: Processes data incrementally to minimize storage overhead.
• Performance Optimization:
• Mobile-Optimized Models: Ensures efficient processing on resource-constrained devices.
• Lightweight Components: Utilizes components that consume minimal system resources.
• Real-time Processing: Enables instantaneous capture and processing of screenshots.
• Memory Management:
• Minimal State: Maintains only essential state information to reduce memory footprint.
• Efficient Fetching: Implements strategies for quick data retrieval.
• Incremental Processing: Processes data in chunks to optimize memory usage.

Performance Example: When operating multiple applications across several monitors, OpenRecall's optimizations ensure that screenshots are compressed efficiently, text extraction occurs in real-time without affecting system performance, and search results are delivered instantly, even with extensive histories.

Privacy Architecture

Privacy is a cornerstone of OpenRecall's design, achieved through:

• Local-First Architecture: All data is stored and processed locally, eliminating cloud dependencies and associated risks.
• User Control: Users have complete authority over data capture, storage, and deletion, ensuring transparency and autonomy.
• Encrypted Storage: Data is stored in encrypted formats to safeguard against unauthorized access.
• Activity-Based Recording: Captures data only during active user periods, respecting user privacy during idle times.

Privacy Use Case: Handling sensitive information, such as financial data or private communications, is secure with OpenRecall. All data remains on the device, encrypted, and accessible only to the user, ensuring confidentiality without compromising functionality.

Common Use Cases

OpenRecall caters to a variety of professional needs, enhancing productivity and organization:

1. Development Work
   1. Track debugging sessions.2. Record documentation references.3. Capture terminal commands and their outputs.4. Review code evolution over time.
2. Research and Writing
   1. Save reference materials.2. Track document revisions.3. Store research notes.4. Find previously read articles.
3. Design Work
   1. Preserve design iterations.2. Record feedback sessions.3. Track inspiration sources.4. Document design decisions.
4. Meeting Follow-up
   1. Review shared screens.2. Find specific discussion points.3. Track action items.4. Reference shared documents.

Getting Started with OpenRecall

Setting up OpenRecall on a Mac involves several steps. Here's a comprehensive guide to help you get started:

1. Clone the Repository:

git clone https://github.com/openrecall/openrecall.git

1. Install Dependencies:

Navigate to the project directory and install the required Python packages.

cd openrecall
pip install -r requirements.txt

1. Run the Application:

Start the OpenRecall application.

python main.py

1. Access the Web Interface:

Open your browser and navigate to http://localhost:5000 to begin interacting with your digital memory.

Conclusion

The trend towards capturing all our activities and being able to recall them at any time could open up new opportunities for many industries. However, with great power comes great responsibility, as the same data can be used for unwanted purposes for which it was not intended, such as misuse by vendors or leakage due to security breaches or intrusion into your PC, and many more.

OpenRecall could be an alternative to Microsoft's Recall feature, offering comprehensive functionality tailored for developers and professionals who prioritise transparency, flexibility and privacy. By meticulously capturing, processing and organising digital activity, it facilitates efficient information retrieval without compromising security.

But remember, open source is not without its risks. It can be hacked like any other system, so it's important to understand what and how we use any piece of software, and to apply basic data security practices such as encryption and continuous updating, especially when as much as our entire digital footprint is at stake.