Imagine you're managing an e-commerce platform with millions of products. Your search functionality is tightly coupled with your application code, making it challenging to update search logic without redeploying the entire application. This scenario illustrates the critical need for decoupling search logic in modern application development. This blog post explores how OpenSearch templates and aliases can address this challenge, offering practical strategies to enhance search performance and simplify maintenance.

OpenSearch Templates: Streamlining Query Management

Search templates in OpenSearch are parameterized search definitions that separate query logic from application code. They allow for reusable, standardized search patterns across your application.

Key Benefits

• Centralized query management
• Improved code maintainability
• Consistent query execution
• Simplified complex query construction

Practical Example: Setting Up a Products Index and Search Template

To better illustrate how to use search templates in a real-world scenario, let's set up a products index and create a category search template. This example will demonstrate the process from index creation to template usage.

First, we'll create the products index with appropriate mappings:

   PUT /products
   {
     "settings": {
       "number_of_shards": 1,
       "number_of_replicas": 1
     },
     "mappings": {
       "properties": {
         "product_name": { "type": "text" },
         "description": { "type": "text" },
         "category": { "type": "keyword" },
         "price": { "type": "float" },
         "in_stock": { "type": "boolean" },
         "tags": { "type": "keyword" }
       }
     }
   }

Next, we'll add some sample products.

   # Add some sample products
   POST /products/_bulk
   {"index":{}}
   {"product_name":"Laptop Pro X","description":"High-performance laptop for professionals","category":"electronics","price":1299.99,"in_stock":true,"tags":["laptop","professional","high-performance"]}
   {"index":{}}
   {"product_name":"Smartphone Y","description":"Latest smartphone with advanced camera","category":"electronics","price":799.99,"in_stock":true,"tags":["smartphone","camera","latest"]}
   {"index":{}}
   {"product_name":"Wireless Headphones Z","description":"Noise-cancelling wireless headphones","category":"electronics","price":249.99,"in_stock":false,"tags":["headphones","wireless","noise-cancelling"]}
   {"index":{}}
   {"product_name":"Classic T-shirt","description":"Comfortable cotton t-shirt","category":"clothing","price":19.99,"in_stock":true,"tags":["t-shirt","casual","cotton"]}

Creating Search Templates

Use the _scripts API to create new search templates for product and category search:

   # Search Template: products search
   POST _scripts/product_search
   {
     "script": {
       "lang": "mustache",
       "source": """
       {
         "query": {
           "bool": {
             "must": [
               { "match": { "product_name": "{{product_name}}" }},
               { "range": { "price": { "gte": {{min_price}} }}}
             ]
           }
         },
         "size": {{size}}
       }
       """
    }
   }

   # Search Template: products in the category
   POST _scripts/category_search
   {
     "script": {
       "lang": "mustache",
      "source": """
       {
         "query": {
           "bool": {
             "must": [
               { "term": { "category": "{{category}}" }},
               {{#in_stock}}
               { "term": { "in_stock": true }}
               {{/in_stock}}
             ]
           }
         },
         "size": {{size}}
       }
       """
     }
   }

This template uses conditional logic to only include the in_stock filter if the parameter is provided.

Using the Template

Execute the template with specific parameters:

   GET _search/template
   {
     "id": "product_search",
     "params": {
       "product_name": "laptop",
       "min_price": 500,
       "max_price": 2000,
       "size": 20
     }
   }

Updating Templates

Modify existing templates without changing application code:

   PUT _scripts/product_search
   {
     "script": {
       "lang": "mustache",
       "source": {
         "query": {
           "bool": {
             "must": [
               {"match": {"product_name": "{{product_name}}"}},
               {"range": {"price": {"gte": "{{min_price}}{{^min_price}}0{{/min_price}}", "lte": "{{max_price}}{{^max_price}}0{{/max_price}}"}}},
               {"term": {"in_stock": true}}
             ]
           }
         },
         "size": "{{size}}{{^size}}10{{/size}}"
       }
     }
   }

Here we used the syntax for defining the default value for a variable var is as follows:

{{var}}{{^var}}default value{{/var}}

To list all search templates

GET _cluster/state/metadata?pretty&filter_path=**.stored_scripts

Multiple Search Templates

You can bundle multiple search templates and send them to your OpenSearch cluster in a single request using the msearchoperation. This approach saves network round trip time, resulting in faster response times compared to independent requests.

Example of using multiple search templates for product searches:

        GET _msearch/template
        {}
        {"id":"product_search","params":{"product_name":"laptop","min_price":500,"max_price":2000, "size":5}}
        {}
        {"id":"category_search","params":{"category":"electronics","in_stock":true,"size":10}}

In this example, we're executing two different search templates in a single request:

1. The first search uses the "product_search" template to find laptops priced from $500 to $2000, limiting results to 1.
2. The second search uses the "category_search" template to find in-stock electronics products, with up to 1 results.

This feature is particularly useful when you need to execute multiple related searches simultaneously, such as populating different sections of a product catalog page or a dashboard with various product metrics.

Advanced Features

Search templates provide extensive flexibility in their definition. Some advanced features include: conditional logic (as shown in the category_search example), loops and join operations. For more detailed information and examples of these advanced features, refer to the OpenSearch documentation on search templates.

Best Practices

• Use descriptive names for templates and parameters
• Version your templates for tracking changes
• Implement access controls to manage who can create or modify templates
• Regularly review and optimize templates for performance

OpenSearch Aliases: Flexible Index Management

OpenSearch aliases are pointers to one or more indices, providing an abstraction layer for index management. They enable seamless routing of search and indexing requests to different indices.

Key Benefits

• Smooth index transitions without downtime
• Support for versioning of search configurations
• Simplified complex index setups
• Efficient management of multiple data sources

Practical Example: Creating and Updating an Alias

Let's walk through a common scenario of creating and updating an alias.

Creating a New Index Version

   PUT /products-v1
   {
     "settings": {
       "number_of_shards": 1,
       "number_of_replicas": 1
     },
     "mappings": {
       "properties": {
         "product_name": { "type": "text" },
         "description": { "type": "text" },
         "category": { "type": "keyword" },
         "price": { "type": "float" },
         "in_stock": { "type": "boolean" },
         "tags": { "type": "keyword" }
       }
     }
   }

Reindexing Data (if necessary)

   POST _reindex
   {
    "source":
        {"index":"products"},
    "dest":
        {"index":"products-v1"}
   }

After reindexing, you can safely delete the old index:

DELETE products

Creating an Alias

Now, create an alias pointing to the new index:

   POST _aliases
   {
     "actions": [
       {
         "add": {
           "index": "products-v1",
           "alias": "products"
         }
       }
     ]
   }

Test the search

The application should not notice any change as the alias name as the same as of index before:

   GET _search/template
   {
     "id": "product_search",
     "params": {
       "product_name": "laptop",
       "min_price": 500,
       "max_price": 2000,
       "size": 20
     }
   }

Updating an Alias

Now, when you need to switch to a new index version (e.g., products-v2), you can use this command:

   POST _aliases
   {
     "actions": [
       {
         "remove": {
           "index": "products-v1",
           "alias": "products"
         }
       },
       {
         "add": {
           "index": "products-v2",
           "alias": "products"
         }
       }
     ]
   }

Viewing Existing Aliases

List all aliases and their associated indices:

   GET _cat/aliases?v

Best Practices

• Use descriptive naming conventions for aliases
• Implement a versioning strategy for indices
• Regularly review and clean up unused aliases
• Use aliases for read and write operations to facilitate zero-downtime migrations

Use Cases for OpenSearch Templates and Aliases

1. A/B Testing Search Features: Use aliases to switch between search configurations and templates for different search settings.

2. Canary Deployment: Gradually roll out new search features by directing a portion of traffic to new indices or configurations.

3. Blue/Green Deployments: Manage major updates with minimal downtime by switching between two sets of indices using aliases.

4. External Experiments: Allow external teams to conduct experiments on search configurations without altering the application code.

5. Dynamic Search Experience: Adapt to evolving product catalogs in e-commerce applications using templates for standardized search settings and aliases for efficient index management.

6. Multi-Tenant Applications: Implement tenant-specific configurations with search templates and use aliases to segregate and manage client data effectively.

Implementation Guide

When implementing OpenSearch templates and aliases, consider the following steps:

1. Plan Your Index Strategy: Determine how you'll version indices and manage transitions.

2. Design Your Templates: Create templates that cover your common search patterns while remaining flexible.

3. Set Up Aliases: Implement aliases for all indices that your application interacts with directly.

4. Update Application Code: Modify your application to use aliases instead of direct index names, and to leverage search templates.

5. Test Thoroughly: Ensure that your new setup works as expected, including testing index transitions and template updates.

6. Monitor and Optimize: Regularly review the performance of your templates and index structure, optimizing as needed.

Conclusion

Decoupling search logic using OpenSearch templates and aliases enhances scalability, flexibility, and maintainability of search-driven applications. By implementing these strategies, you can achieve more agile and efficient search management, reduce downtime, and streamline updates. As with any architectural decision, consider your specific use case and requirements when applying these techniques.

Feel free to leave your questions in the comments!

References

• tecRacer is Amazon OpenSearch Service Delivery Partner!
• OpenSearch Documentation
• OpenSearch Search Templates
• OpenSearch Index Aliases