Introducing Persisted Copilot Chats - Integrated AI Across your Workflow

In our latest Ask Me Anything (AMA) session, we unveiled some exciting enhancements to Pieces Copilot. The spotlight was on the introduction of automatic saving of conversations and the persistence of context across different workflows. This significant update means that whether you're using our Pieces for Developer Desktop Application, any IDE, or the Pieces Web Extension, the context set for each conversation with Pieces Copilot remains consistent and readily accessible.

Our panel, consisting of Tsavo Knott (Co-Founder & CEO), Mark Widman (CTO), Mack Myers (Co-Founder & CPO), and Rutvik Tak (Web Extension Developer) shared their insights on the latest features of Pieces Copilot, and how we built this integrated AI experience with Flutter and Dart.

This included demonstrating how conversations with Pieces Copilot are now automatically saved, and the context set for each conversation persists across different workflows including our Pieces for Developer desktop application, VS Code extension, and the Pieces web extension. The AMA also touched upon the 2024 plans for Pieces and answered some of the questions from the community.

If you missed the live event, don't worry. We've summarized the key takeaways, highlighted some of the most thought-provoking questions and answers, and even shared video snippets of the discussion to get you up to speed!

Topics Covered

So, get comfortable, prepare your favorite beverage, and let's dive into the key moments of our most recent livestream with the developer community.

Evolution of Pieces for Developers

The AMA discussion opened with Tsavo reflecting on the journey of Pieces so far and how the scope has changed from code snippet management to improving end to end developer workflow productivity.

In the early days, our focus was on improving the developer experience for developers through efficient code snippet management. We knew that the copy-paste experience wasn't great for organizing development resources, so we set out to enrich saved snippets with fine-tuned machine learning models to make them more robust. This laid the groundwork for the search and sharing capabilities that we later introduced.

As our users started saving more and more, we wanted to make it easier for them to search for things globally. We added new ways to filter and suggest things, but the real game-changer came at the beginning of this year when we started working on the Pieces Copilot.

The Pieces Copilot uses large language models, either on-device or in the cloud, as well as retrieval-augmented generation, to understand the context of your browser, the tabs you have open, and what you're doing in the IDE. It's a true workflow copilot that understands the full context of your work. We've worked hard to make it performant and private, supporting large language models on MacOS, Linux, and Windows.

Announcing Persistent Copilot Chats

With our latest release, we also introduced the ability to persist conversations with Pieces Copilot. These conversations are deeply contextualized, and the context is persisted across your toolchain. This means that when you say "look at this file" or "look at this website", that's captured and persisted with the full conversation. This feature is available across all our platforms, including the Pieces for Developer Desktop Application, IDEs, and the Pieces Web Extension.

One such example of persistence for power users is the ability to plug in your Enterprise GPT-4 license into Pieces. This means you can now get that Enterprise GPT-4 license everywhere, grounded in your workflow context. This is a big win for those who were previously underutilizing their expensive Enterprise license because of unnecessary context switching, or lack of integrated context of their work-in-progress journey.

Pieces Copilot Chats Experience in Pieces for Developers Desktop Application

Next, Mack took a deep dive into the Pieces Copilot chat functionality in the Pieces desktop application.

Mack started by showcasing his personal Pieces repository. He introduced the various ways to launch the Pieces Copilot, whether from the list view, the gallery view, or the navigation dropdown.

Overview of the Pieces Copilot Chat Experience

Upon entering the Pieces Copilot chat view, Mark highlighted the refinements of the UI, featuring a new aesthetic, layout, and simplified UI elements. The chat view is divided into two main sections: the left side displays the conversations, while the right side shows the content of the selected conversation.

Mark then demonstrated the flexibility of the Pieces Copilot, with the ability to switch between different models, including the GPT 3.5 Turbo, cloud models via OpenAI, Palm 2, and the Llama 2 on-device model for security-conscious users.

The power of the Pieces Copilot was further demonstrated when Mark asked it to generate a Firebase storage rule that only allows authenticated users to access a bucket. The Pieces Copilot quickly provided a simple Firebase storage rule.

Context, Multi-Modality, and Related People

Mark then showcased the various quick-actions that can be taken with the generated code snippet, such as saving it to Pieces, sharing it with a team member, annotating the code, or finding similar snippets. He also demonstrated how the Pieces Copilot can search through the entire Pieces repository to find related snippets, providing a wealth of related data, including related people, anchors (files or folders associated with the snippets), and related links.

Demo for setting Pieces Copilot context with Screenshots

Mack showcased the power of Pieces Copilot in handling code shared in the form of screenshots. This feature is particularly useful when collaborating with team members who often share code snippets as images.

Generating Snippets from Screenshots

Mack demonstrated this by showing a code snippet shared by another team member. Mack simply dragged the image into the Pieces Copilot, which then ran Optical Character Recognition (OCR) on the image to extract all the code.

The OCR process resulted in a high-fidelity extraction of the code from the image, even accurately capturing commented out sections. Once the code was extracted, the Pieces Copilot was able to provide related people based on the context of the code and other saved snippets.

Mark then asked the Pieces Copilot to explain what the code snippet was doing. Without having to read through the complex code, the copilot provided a concise four-line summary of the code's functionality.

Annotating your Code Snippets

Mark then asked the copilot to annotate the code, which resulted in a version of the code with added comments explaining each step. This annotated version of the code can be shared back with the team, providing a clearer understanding of the code's functionality.

Demonstration for Persisted Chats in Different IDEs including VS Code

In the next part of our session, Mark demonstrated the Pieces Copilot conversation and context selection features within VS Code, one of our most popular integrations, to highlight how persistence of context spans across different IDEs and workflows.

We emphasized that Pieces Copilot is not confined to a single platform or ecosystem. Whether you're working within the JetBrains, Visual Studio Code ecosystem, Pieces Copilot is designed to be a workflow copilot across your entire developer day-to-day. It doesn't start and stop in the IDE. What you do in each of these pillars informs the context that should be used to ground the copilot.

Demonstrating Persisted Chat functionality in VS Code extension

Mark started the demonstration by using our open-source TypeScript SDK, which was published just a few weeks ago. He showed how you can ask the Pieces Copilot about the repo, specifically how to create an asset using the Assets API. The copilot was able to point to the relevant files and provide the necessary code to create the asset.

Save, Delete, Edit Conversations in Pieces Copilot

One of the key features highlighted was the ability to save conversations with the copilot. These saved conversations, complete with metadata such as title, description, tags, language, and related person, can be viewed later for future use. This feature is available across all our platforms, including the Pieces for Developers Desktop Application, VS Code extension, and the Pieces Web Extension.

We also touched on the flexibility of the copilot, with the ability to switch between different copilot runtimes, including local LLM, OpenAI models, and Palm 2 models. You can also configure the context to suit your needs.

Pieces Web Extension Copilot

As we moved into the second half of our session, we showcased our Pieces for Developers Chrome Extension. Rutvik, our Lead Web Extensions developer discussed the importance of the browser in a developer's day-to-day workflow, acknowledging that a significant portion of a developer's time is spent across browsers, searching for that elusive snippet hidden within a long conversation on a site or a database issue description.

Ask Copilot and Persistent Copilot Chats in Web Extension

He highlighted the role of Pieces web extension in this context, emphasizing its ability to identify and surface code blocks on websites within long conversations or issues, thereby significantly reducing the time spent scrolling and searching. The ability to ask about any particular snippet found on a webpage directly to the Pieces Copilot was also discussed, eliminating the need to navigate to other sites or resources.

The copilot takes the snippet as context and generates the most helpful answer it can find, based on your workflow between your IDE, desktop app, and the extension itself. This context is also carried over to the desktop app, allowing for a seamless transition between the browser and the desktop app without losing any time.

Importance of Security and Privacy

Tsavo also discussed the importance of privacy and security, emphasizing that Pieces is one of the only companies in the world that allows you to have this entire experience running completely offline with a local language model. This unique feature is particularly beneficial when working with internal or private documents that cannot be sent out to the web or large language models.

Demonstrating the Web Extension in Firefox & Opera

Rutvik also showcased the cross browser support for the Pieces Web Extension. Our extension, which has had solid support within the Chrome and Edge browsers, is also being shipped for Firefox, Opera, and other browsers like Brave. With the latest release, not only you can use Pieces Copilot in all the above mentioned browsers, but also maintain the same context across different tools.

Furthermore, we discussed the use of Flutter in the browser and web apps, emphasizing the versatility it offers. We shared our experience with Flutter on the web, highlighting how it allows us to move fast, ship output that works across different browsers, and manage code easily. We also discussed the benefits of sharing code between our desktop app and extension, which speeds up our work and maintains a consistent experience across our different products.

Choosing Dart and Flutter for Building Pieces

We spent the next few minutes talking specifically about our reason for choosing Dart and Flutter as part of our tech stack across Pieces.

Reasons to use Dart at Pieces

Mark mentioned a lot of engineers on our team are proficient in Dart. Dart has enabled us to write about 90-95% of our code, package up any operating system-specific code into binaries, and interact with Dart FFI. This approach has allowed us to ship our product across different operating systems quickly and efficiently.

We also discussed our belief that AI should be an integral part of the operating system, unifying macOS, Linux, and Windows in a way that is intelligent, proactive, and helpful. Dart has been instrumental in this vision, offering versatility and performance benefits. For instance, Dart transpiled to JavaScript runs 50% faster than handwritten JavaScript, providing the benefit of strong typing and type safety.

Furthermore, we touched on the benefits of using Flutter for our front ends across both our Desktop App and Web Extension, which allows us to build consistent experiences across platforms with a single codebase. This approach has significantly reduced our technical debt and increased our ability to move quickly. We also highlighted the importance of type safety, which Dart provides right out of the box.

Ongoing/Upcoming Enhancements to Dart

Moreover, Tsavo and Rutvik also highlighted some ongoing and upcoming improvements in Dart, such as the isomorphic capability, compiling to WebAssembly, and how it has allowed us to communicate with JavaScript code effectively.

Maintaining Consistency across various Platforms and IDEs with Pieces Copilot

As we approached the end of the AMA we addressed one question from the audience about maintaining context and consistency for Pieces Copilot across all the different platforms.

Abstracting Code: Write Once, Deploy Everywhere

Mack touched upon the centralization of our Copilot functionality to the platform and Pieces OS. This approach allows all our plugins and tools to leverage the same platform for feature development, ensuring a consistent experience across all interfaces. He highlighted the importance of attention to detail in maintaining this consistency, and how our team is working hard to simplify this process by abstracting code to write once and deploy in multiple places.

We look at our products side by side, ensuring that the experience is consistent across all platforms. There's no magic trick here - it's all about reducing code, paying attention to detail, and spending the time to ensure consistency.

Pieces Client-Side SDKs

Tsavo also touched upon how consistency is also crucial at the SDK level and the data layer. A fun fact about our company is that a majority of our client-side SDKs are generated off of protobuf specs or HTTP open API specs. This means that when a new endpoint with a certain capability is released, we can quickly have it available in Kotlin, TypeScript, Dart, Rust, Go, and some other languages. This rapid deployment ensures that our users always have access to the latest features and capabilities, regardless of the platform they are using.

Transforming Code Snippets with Pieces

As we came to the end of the AMA, we received one question from the audience about updating snippets to make them more manageable and readable.

Mack demonstrated Pieces’ smart transforms feature, which allows users to modify code snippets for improved readability, performance, and even translate them into different languages.

Upcoming features and closing remarks

As we transition into December and approach the new year, we shared a few highlights with the audience to hint at our upcoming developments: ghost assets, automatic saving, and collaboration across devices, teams, and enterprises.

Conclusion

We extend our gratitude to everyone who joined us in our latest AMA about the evolution of Pieces Copilot.

Our team at Pieces is dedicated to creating seamless and efficient coding workflow experiences for developers worldwide. The introduction of automatic saving of conversations and the persistence of context for these conversations across different workflows is a testament to this commitment.

We're particularly thrilled about the potential of these new features to enhance the functionality of Pieces Copilot and provide a more unified and efficient coding experience. The ability to maintain context across different platforms is a significant advancement, and we're excited to be at the forefront of this innovation. The ability to tailor the conversations with specific contexts like local files, folders, saved snippets, and even chat messages provides even further flexibility to the Pieces Copilot experience.

We're also eagerly looking forward to the release of more features and the continued growth of our user community. The discussions and plans we've shared today are just the beginning, and we hope you share our excitement. Keep an eye out for announcements about our next AMA as we close out 2023 and get ready for 2024 with some major updates coming to even further unify your experience using Pieces.

Whether you're a current user with feedback, a developer interested in our technology, or someone just curious about our product, we invite you to join our next session and become part of our vibrant community on Discord!

Once again, thank you for your invaluable participation and for being a part of our community. Your feedback and engagement are what drive us to continue innovating. Until the next AMA, happy coding!

Blog