Introducing Swiftide 0.9 with Fluvio as a starting point for indexing streams, lancedb for querying and indexing and RAGAS support

To get started with Swiftide, head over to swiftide.rs, check us out on github, or hit us up on discord.

Fluvio support

Fluvio is a lightweight high-performance distributed data streaming system written in Rust and Web Assembly. In a production environment, data could come and go to many places at the same time. With the Fluvio loader, you can hook into a Fluvio topic and index right away.

The integration is fully configurable with Fluvio's own configuration. Here is an example:

static TOPIC_NAME: &str = "hello-rust";
static PARTITION_NUM: u32 = 0;

let loader = Fluvio::builder()
    .consumer_config_ext(
        ConsumerConfigExt::builder()
            .topic(TOPIC_NAME)
            .partition(PARTITION_NUM)
            .offset_start(fluvio::Offset::from_end(1))
            .build()?,
    )
    .build()?;

indexing::Pipeline::from_loader(loader)
    .then_in_batch(10, Embed::new(FastEmbed::try_default().unwrap()))
    .then_store_with(
        Qdrant::builder()
            .batch_size(50)
            .vector_size(384)
            .collection_name("swiftide-examples")
            .build()?,
    )
    .run()
    .await?;

Lancedb

Lancedb is a popular vector and text database that separates storage from compute. It enables a whole new class of applications where the database is embedded into the application itself. Under the hood it uses Apache Arrow and Tantivy.

Both indexing with different embedded fields and querying is supported. Note that Lancedb does not really need Sparse vectors, as it provides full text search as well. Additionally, Swiftide does not index the data.

An example:

let lancedb = LanceDB::builder()
  .uri(tempdir.child("lancedb").to_str()?)
  .vector_size(1536)
  .with_vector(EmbeddedField::Combined)
  .with_metadata(METADATA_QA_TEXT_NAME)
  .table_name("swiftide_test")
  .build()?;

// and in your indexing pipeline
indexing_pipeline.then_store_with(lancedb.clone())

// and in your query pipeline
query_pipeline.then_retrieve(lancedb.clone())

RAGAS support

Evaluating RAG pipelines is a whole topic by itself. A query pipeline can now take an evaluator, and we have build one for RAGAS to kick it off. The evaluator
can export to json, which can be imported in a python hugging face dataset and used with RAGAS.

This is great, as it will allow you to evaluate the quality of your data, indexing and querying.

A full guide is coming soon!

An example:

let ragas = evaluators::ragas::Ragas::from_prepared_questions(questions);

let pipeline = query::Pipeline::default()
    .evaluate_with(ragas.clone())
    .then_transform_query(GenerateSubquestions::from_client(context.openai.clone()))
    .then_transform_query(query_transformers::Embed::from_client(
        context.openai.clone(),
    ))
    .then_retrieve(context.qdrant.clone())
    .then_answer(Simple::from_client(context.openai.clone()));

pipeline.query_all(ragas.questions().await).await?;

let json = ragas.to_json().await;
std::fs::write("output.json", json)?;

And then in a Python notebook:

from datasets import load_dataset
dataset = load_dataset("json", data_file='output.json')

from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)

import pandas as pd

from ragas import evaluate

result = evaluate(dataset, metrics=[answer_relevancy,faithfulness,context_recall,context_precision]).to_pandas()

# ... And create some amazing plots!

From our own experimentation, Rust feature flags are great to try out many different options fast to get data in.

What's next?

We are working hard on hybrid search in the query pipeline for both Qdrant and Lancedb, improving documentation for the query pipeline and many more improvements. If you have feedback, suggestions or need help, feel free to reach out to us on Discord or via a Github issue.

Call for contributors

There is a large list of desired features, and many more unlisted over at our issues page; ranging from great starter issues, to fun, complex challenges.

You can find the full changelog here.

To get started with Swiftide, head over to swiftide.rs or check us out on github.