Search supported by a database

Old-fashioned search was mostly based on traditional relational databases. Older search engines were based on layered architectures implemented in multi-tiered applications, as shown in figure 1.1.

Figure 1.1 Search based on a traditional database

Queries written in SQL using clauses like where and like provided the foundation for search. These solutions are not necessarily performant and efficient for searching full-text data to provide modern search functionality.
Having said that, some modern databases (Oracle and MySQL, for example) support full-text searching (queries against free text like a blog post, movie review, research article, etc.), but they may struggle to provide efficient searches in near-real time on heavy loads.
The distributed nature of a search engine like Elasticsearch provides instant scalability that most databases are not designed for. A search engine developed with a backing database (with no full-text search capabilities) may not be able to provide relevant search results for queries, let alone cope with volumetric growth and serve results in real time.

Full-text searching with databases

Relational databases like Oracle and MySQL support full-text search functionality, albeit with less functionality than a modern full-text search engine like Elasticsearch. They are fundamentally different when it comes to storing and retrieving data, so you must be sure to prototype your requirements before choosing one of them. Usually, if the schemas are not going to change or data loads are low, and you already have a database engine with full-text search capabilities, beginning with full-text search on a database may make sense.

 Databases vs. search engines

When building a search service on a traditional database, we need to consider and understand whether our requirements can be satisfied by the database efficiently and effectively. Most databases are designed to store large amounts of data but, unfortunately, are not well-suited for use as full-text search engines, for several reasons:

• Indexing and search performance —Full-text search requires efficient indexing and performant search and analytical capabilities, which traditional databases are not optimized for. Databases may struggle with indexing large volumes of data and, as a result, may exhibit poor query performance. Search engines like Elasticsearch and Solr are specifically designed to handle large amounts of text data and provide search results in near-real time. Search engines can handle large-scale data, indexing it and searching it much faster than traditional databases, as they are pretty much designed from the ground up for optimized search operations. Unfortunately, relational databases lack advanced search features such as fuzzy logic, stemming, synonyms, and so on.

• Search —Searching with traditional databases is more or less based on the exact matching of data values. While this is suitable for non-search-related find operations on structured data, it is a definite no-no for natural language queries, which are often complex. User queries are often misspelled, misconstructed grammatically, or incomplete and may contain synonyms and other language structures that databases fail to understand. In natural language queries, users may not use the exact terms they are searching for (spelling mistakes), and unfortunately, traditional databases are not designed to support misspelled user input. This feature is supported by the fuzzy matching search function (words that are similar but not exactly the same) in modern search engines. In traditional databases, data is often normalized, meaning it is spread across multiple tables and columns. This can make it difficult to search for data across multiple fields in a single query. Traditional databases are not designed to handle the types of unstructured and semi-structured data that are common in full-text search scenarios.

  • Text analysis and processing —Search engines must often handle multiple languages and character sets, which traditional databases may not support. Search engines perform text analysis and processing to extract meaning from text, but traditional databases are not designed or optimized for this purpose.
  • Scalability and flexibility—Full-text search engines are designed to handle large amounts of data and high query loads. Traditional databases can have scalability problems when dealing with large amounts of text data.

Search engines are designed from scratch to handle unstructured data, while databases are optimized for handling structured data. These limitations make traditional databases less suitable for use as full-text search engines; specialized search engine technologies such as Elasticsearch, Solr, Lucene, etc., are often used to provide advanced search functionality for text data.

NOTE Many databases have added text search capabilities to their feature sets. However, they still may not be able to deliver performance, scalability, and functionality on par with specialized full-text search engines.

Nothing is stopping us from embracing both worlds: in some use cases, a combination of traditional databases and search engines can be employed. For example, a database can be used for transactional purposes and a search engine for search and analytics. But our focus in this document is search engines—and Elasticsearch in particular. In the next section, we review the era of modern search engines before we introduce Elasticsearch.

 Modern search engines

Modern search engines are trying hard to meet ever-growing business requirements by embracing new and exciting features every day. Cheap hardware combined with the explosion of data is leading to the emergence of these modern search beasts. Let’s consider present-day search engines and the features and functionality they offer. We can summarize what a good modern search engine should provide as follows:

• First-class support for full-text (unstructured) and structured data
• Type-ahead suggestions, auto-correction, and “did-you-mean” recommendations
• Forgiveness for users’ spelling mistakes
• Search capabilities on geolocations
• Easy scalability, either up or down, based on fluctuating demand
• Blazing performance: speedy indexing and search capabilities
• Architecture that provides a high-availability, fault-tolerant distributed system
• Support for machine learning functionalities

In this section, we briefly discuss the high-level features of a modern search engine. Then the following section introduces a couple of search engines available in the market, including Elasticsearch.

 Functionality

Modern search engines were developed to satisfy full-text search requirements while also providing other advanced functions. They are designed to provide fast and relevant search results to users by indexing and searching large volumes of text data (going forward, we will drop the word modern when mentioning search engines).

Search engines can quickly index large amounts of text data and make it searchable. This process typically involves breaking the text data into tokens and building an inverted index, which maps each token to the documents that contain it.

Search engines are also expected to perform advanced text analysis and processing, such as synonyms, stemming, stop words, and other natural language processing techniques, to extract meaning from text and improve search results. They can process user queries and rank search results based on various factors such as relevance and popularity. They can also handle high query loads and large amounts of data and can scale horizontally by adding more nodes to a cluster.

Finally, search engines provide advanced analytics capabilities, looking at the data to provide summaries, conclusions, and intelligence for businesses. They also support rich visualizations, near-real-time search, performance monitoring, and machine learning-based insights.

Popular search engines

While a handful of search engines are available in the market, I’ll mention just few of them, all of which are built on top of Apache Lucene.

Elasticsearch

Shay Banon, founder of Elastic, developed a search product called Compass in early 2000. It was based on an open source search engine library called Apache Lucene. Lucene is Doug Cutting’s full-text search library, written in Java. Because it’s a library, we must import it and integrate it with an application using its APIs. Compass and other search engines use Lucene to provide a generalized search engine service so we don’t have to integrate Lucene from scratch into applications. Shay eventually decided to abandon Compass and focus on Elasticsearch because it had more potential.

 Apache Solr

Apache Solr is an open source search engine that was built on Apache Lucene in 2004. Solr is a strong competitor to Elasticsearch and has a thriving user community, and it is closer to open source than Elasticsearch (Elastic moved from Apache to Elastic License and Server Side Public License ([SSPL] in early 2021). Both Solr and Elasticsearch excel at full-text searching; however, Elasticsearch may have an edge when it comes to analytics.
While both products compete in almost all functionality, Solr is a favorite for large, static datasets working in big data ecosystems. Obviously, we have to run through prototypes and analysis to pick a product; the general trend is for projects that are integrating with a search engine for the first time to consider Elasticsearch due to its top-class documentation, community, and nearly no-hurdle startup. You must make a detailed comparison of your intended use cases for the search engine before adopting and embracing one.

Amazon OpenSearch

Elastic changed its licensing policy in 2021. The licensing, which applies to Elasticsearch release versions 7.11 and above, has been moved from open source to a dual license under an Elastic License and SSPL. This license allows the community to use the product for free, as expected, but managed service providers can no longer provide the products as services. There was a spat between Elastic and Amazon Web Services (AWS) when AWS created a forked version of Elasticsearch—called Open Distro for Elasticsearch—and offered it as a managed service. This spat led to the change in the license, which eventually led to OpenSearch’s birth.

As Elastic moved from the open source licensing model to the SSPL model, a new product called OpenSearch was developed to fill the gaping hole left by the new licensing agreement. The base code for OpenSearch was created from the open source Elasticsearch and Kibana version 7.10.2. The product’s first General Availability version 1.0 was released in July 2021. Watch out for OpenSearch becoming a competitor to Elasticsearch in the search engine space.

Now that we have a fair understanding of what a modern search engine is and the shape of the search landscape, let’s jump into an overview of Elasticsearch.

Elasticsearch overview

Elasticsearch is an open source search and analytics engine. Developed in Java, it is an ultra-fast, highly available search engine built on the popular full-text library Apache Lucene (https://lucene.apache.org). Elasticsearch wraps around the powerful functionality of Lucene by providing a distributed system with RESTful interfaces. Lucene is the powerhouse of Elasticsearch, and Kibana is the administrative UI to manage and work with Elasticsearch.

Full-text searching is where Elasticsearch excels as a modern search engine. It can retrieve relevant documents in response to a user’s search criteria at an awesome speed. We can search for exact terms, too, like keywords, dates, or a range of numbers or dates. Elasticsearch is packed with top-notch features such as relevancy, “did-you-mean” suggestions, auto-completion, fuzzy and geospatial searching, highlighting, and more.

In addition to being a frontrunner in providing near-real-time search capabilities, Elasticsearch stands tall in statistical aggregations on big data. Of course, we must consider the use case before embracing the product, as Elasticsearch may not be the best fit for every use case. Out of the box, Elasticsearch also boasts commendable features such as application performance monitoring, predictive analytics and outlier detection, and security threat monitoring and detection.

Elasticsearch focuses on finding a deeper meaning in the data that’s been collected. It can aggregate data, perform statistical calculations, and find intelligence within the data. We can create rich visualizations and dashboards and share them with others using Kibana tooling. Elasticsearch can find averages, sums, means, and modes as well as undertaking complex analytics such as bucketing data in histograms and other analytical functions.

Furthermore, Elasticsearch runs supervised and unsupervised machine learning algorithms on our data. Models help to detect anomalies, find outliers, and forecast events. In supervised learning mode, we can provide training sets so the model learns and makes predictions.

Elasticsearch also comes with the capability to observe applications and their health by monitoring performance metrics such as the memory and CPU cycles of the web servers in a network. It lets us sift through millions of web server logs to find or debug application issues. Elasticsearch also invests time and resources in building security solutions: for example, alerting us to security threats, IP filtering, endpoint prevention, and more.

Core areas

Elastic, the company behind Elasticsearch, has been positioning itself predominantly in three core areas: search, observability, and security, as shown in figure 1.2. Let’s look at each of these areas in turn.

Figure 1.2 Core application areas of Elastic, the company behind Elasticsearch

 Elastic Enterprise Search

Whether letting users search across varied content providers (like Slack, Confluence, Google Drive, and others) or enabling search capabilities for our applications, apps, and websites, the Elastic Enterprise Search suite helps build models and a customized search engine.

Search can be integrated deep into a multitude of applications in various domains—business, infrastructure, applications, and so on. Users can create a web application backed by Elasticsearch, a mobile app supported by Elasticsearch, or a server-side search “service with Elasticsearch as the spine for search capabilities.

Elastic Observability

Applications running on infrastructure produce a lot of metrics that are usually used for application observability and monitoring. We can use Elasticsearch in the observability space: the state of applications, servers, racks, and modules can all be monitored, logged, tracked, and alerted. We can also use the Elastic tools to perform application management and monitoring on a large scale.

 Elastic Security

Elastic enters the realm of security by enabling threat detection and prevention and providing advanced features such as the capability of removing malware at the source, encryption at rest, and more. As a security information and event management (SIEM) tool, Elastic is positioning itself to protect organizations with its advanced security toolkits.

Elastic Stack

Elasticsearch is the core of the search engine, and a handful of Elastic products complement it. The suite of products is called the Elastic Stack and includes Kibana, Logstash, Beats, and Elasticsearch. (It was formally called ELK Stack but was renamed Elastic Stack after Beats was introduced into the product suite.)

The combination of these four products helps build an enterprise application by integrating, consuming, processing, analyzing, searching, and storing various data sets from disparate sources. As demonstrated in figure 1.3, Beats and Logstash bring the data into Elasticsearch, while Kibana is the visual UI that works on that data.

Figure 1.3 The Elastic Stack: Beats, Logstash, Elasticsearch, and Kibana

Before we move on to look at use cases for Elasticsearch, let’s briefly go over these essential moving parts at a high level.

 BEATS

Beats are single purpose data shippers; they load data from out various external systems and pump it into Elasticsearch. Various types of beats are available out of the box. These include Filebeat, Metricbeat, Heartbeat, etc., and each performs a specific data consumption task. These are single-purpose components: for example, Filebeats are designed for file-based transports and Metricbeats for vital machine and operating system memory and CPU information. The beats’ agents are installed on the servers so they can consume data from their source and send it to their destination.

 LOGSTASH

Logstash is an open source data-processing engine. It extracts data originating from multiple sources, processes it, and sends it to a variety of target destinations. During the processing of the data, Logstash transforms and enriches the data. It supports a myriad of sources and destinations including files, HTTP, JMS, Kafka, Amazon S3, Twitter, and dozens of others. It promotes a pipeline architecture, and every event that goes through the pipeline is parsed as per the preconfigured rules, thus creating a real-time pipeline for data ingestion.

 KIBANA

Kibana is a multipurpose web console that provides a host of options such as executing queries; developing dashboards, graphs, and chart visualizations; and creating dropdowns and aggregations. However, we can use any REST client to talk to Elasticsearch to invoke the APIs, not just Kibana. For example, we can invoke APIs using cURL, Postman, or native language clients.

Elasticsearch use cases

Pinpointing Elasticsearch for a particular use case or domain is difficult. It is omnipresent in many areas from search to analytics to machine learning jobs. It is widely used across a multitude of industries, including finance, defense, transport, government, retail, cloud, entertainment, space, and more. Let’s take a high-level glance at how Elasticsearch can be used in an organization.

Use case: Search engine

Elasticsearch has become the go-to technology for its full-text search capabilities. The product is not limited to full-text searching but can also be used for structured data and geolocation-based searches. Broadly speaking, customers use Elasticsearch in three domains: App Search, Enterprise Search, and Site Search.

In App Search, Elasticsearch serves as a backbone, providing search and analytical capabilities for applications. A search service backed up by Elasticsearch can be designed as a microservice that serves the application’s search requirements, such as searching for customers, orders, invoices, emails, and so on.

In most organizations, data is scattered across many data stores, applications, and databases. For example, organizations are often integrated with Confluence, intranet spaces, Slack, email, databases, cloud drives (iCloud drive, Google Drive, etc.), and others. Collating and searching through vast amounts of data with integrations to varied sources is a challenge for these organizations. This is where Elasticsearch can be employed for Enterprise Search and data organization.

If we have an online business website amassing data, providing search is something of a bare necessity for attracting customers and keeping them happy. Site Search is a software-as-a-service (SaaS) offering from Elastic that, once enabled, crawls through the given site pages, fetching data and building indices backed by Elasticsearch. Once the crawling and indexing are complete, the site can be integrated easily with the search facility. The Site Search module also helps create a search bar and the code snippet related to it. The website administrator can copy the snippet of generated code onto their homepage to enable a search bar instantly, thus making the website fully functional with integrated search.

Use case: Business analytics

Organizations capture tons of data from various sources, and that data often holds the key to survival and success. Elasticsearch can help extract trends, statistics, and metrics from data, giving organizations knowledge about their operations, sales, turnover, profits, and many other features for timely management.

Use case: Security analytics and threat and fraud detection

Data security and potential breaches of it are nightmares for organizations. Elasticsearch’s security analytics help organizations analyze every bit of information—be it from applications, a network, endpoints, or the cloud. This analysis can provide insights into threats and vulnerabilities and let the organization hunt for malware and ransomware, thus alleviating the risk of falling prey to hackers.

 Use case: Logging and application monitoring

Applications spit out a lot of data in the form of application logs and metrics. These logs provide insights into the health of the application. With the advent of the cloud and the microservices world, logs are scattered across services, and meaningful analysis is a cumbersome affair. Elasticsearch is our friend here. One of the popular use cases for Elasticsearch is indexing logs and analyzing them for application errors and debugging purposes.

Elasticsearch is a powerful and flexible search and analytics engine, but it’s not suitable for every use case. Let’s briefly go over the issues we may encounter and use cases for which Elasticsearch is the wrong choice.

Unsuitable Elasticsearch uses

Not every use case can be satisfied by Elasticsearch. It is a powerful and flexible search and analytics engine, but unfortunately, this tool has limitations that we must consider before choosing it for our requirements. Here are a few scenarios where Elasticsearch may be an incorrect or inefficient solution:

• Relational data—Elasticsearch is not the right tool to accommodate search over data that has relationships and needs to perform complex database joins. Elasticsearch is not designed to handle complex relational data structures. If your data is relationship-heavy, a relational database like MySQL or PostgreSQL may be a better fit. Most modern databases (MySQL, PostgreSQL, etc.) also offer full-text search capabilities, although the features are not as advanced as in a modern search engine like Elasticsearch.

• Transactional data—Elasticsearch is an “eventually consistent” search engine, which makes it unsuitable for applications that require immediate consistency, such as financial transactions. For these types of use cases, consider using a traditional relational database or a NoSQL database like MongoDB.

• Geospatial data—While Elasticsearch has built-in support for geospatial data, it may not be the most efficient solution for large-scale geospatial analytics. For these use cases, consider using a dedicated geospatial database like PostGIS or a geospatial analytics platform like ArcGIS.

• High-write workloads—Elasticsearch can handle high-read workloads, but it is not optimized for high-write workloads. If you need to index large amounts of data in real time, consider using a dedicated indexing engine like Apache Flume or Apache Kafka.

• Online analytical processing (OLAP) data—If you need to perform complex multidimensional analysis on large data sets, a traditional OLAP database like Microsoft Analysis Services or IBM Cognos may be a better fit than Elasticsearch.

• Large binary data—While Elasticsearch can handle large amounts of text data, it may not be the best solution for indexing and searching large binary data like videos or images. For these use cases, consider using a dedicated binary data store like Hadoop Distributed File System (HDFS), Amazon S3, or Azure Files.

• Real-time analytics—Elasticsearch is great for performing real-time search and analytics on large data sets, but it may not be the most efficient solution for real-time data processing and analytics. Instead, consider a specialized real-time analytics platform like Apache Spark or Apache Flink.

• Latency-sensitive applications—Although Elasticsearch is designed to handle high-volume search and analytical queries, it can still have latency issues when dealing with large amounts of data. For applications that require sub-millisecond response times, a specialized search engine like Apache Solr or a columnar database like Apache Cassandra may be a better fit.

• Other types—Elasticsearch is not a preferred solution for time-series data, graph data, in-memory data, and various other types of data. If you need to store and analyze time-series data, a specialized time-series database like InfluxDB or TimescaleDB may be a better fit. Similarly, a graph database such as Neo4j may help you tackle graph data.

It’s important to evaluate your specific use case and requirements before choosing Elasticsearch as your technology and tool. In the next section, we discuss common misconceptions about Elasticsearch as a tool, a technology, and a search solution.

Misconceptions

A major misconception about Elasticsearch is mistaking it for a traditional relational database. It’s also a common misunderstanding that setting up Elasticsearch is easy, while in reality, many tweaks are required to set up a decent-sized cluster. In addition, Elasticsearch is often thought of as a technology used for text search, when in fact it can be used for a wide range of search and analytics use cases. The following list summarizes some common misconceptions about Elasticsearch:

• Elasticsearch is easy to set up and manage. While Elasticsearch is relatively straightforward to set up and get started with, it can be challenging to manage and scale as data grows and use cases increase. Although everything works out of the box, making an engineer’s life easy, taking Elasticsearch into the production environment requires effort. We may need to tweak the configuration and fine-tune the memory, manage node failures, or even scale the cluster to handle petabytes of data as our data grows.
• Elasticsearch is a relational database. Elasticsearch is not a relational database and does not support traditional relational database features like transactions, foreign keys, and complex join operations. For example, we can’t enforce referential integrity or perform complex “oin operations. For example, we can’t enforce referential integrity or perform complex join operations in Elasticsearch. If you need these features, a proven relational database like MySQL or PostgreSQL is definitely your solution.
• Elasticsearch can handle all types of data. Elasticsearch is versatile and can handle a wide range of data types, but it is not designed to handle every type of data with equal ease. For example, it may not be the best solution for real-time data processing and analytics or for handling large binary data. If you need to store and process large binary data like videos or images, consider using a dedicated binary data store like HDFS or Amazon S3.
• Elasticsearch is only for text search. While Elasticsearch is great for text search, it can also perform complex analytics on structured and unstructured data. For example, we can use Elasticsearch to perform aggregations, analyze log data, and visualize data using Kibana.
• Elasticsearch can replace all other technologies. Elasticsearch is a powerful and flexible technology, but it is not a one-size-fits-all solution and or the best choice for every use case. It can never replace a traditional relational database, for example.
• Elasticsearch is always faster than other technologies. Elasticsearch is designed for high performance and is expected to perform well under heavy loads. However, there’s only so much Elasticsearch can do, and its performance primarily depends on how well the platform engineers fine-tune it.
• Elasticsearch deals with big data only. Elasticsearch can handle petabytes of data in large data sets, but it is equally performant when dealing with small data sets on the order of a few gigabytes. For example, we can use Elasticsearch to search and analyze data for an organization’s small email database or a startup company without much effort.

These are just a few examples of misconceptions about Elasticsearch. As mentioned earlier, you must carefully evaluate your specific requirements and use case before choosing Elasticsearch or any other technology.

Popular adoption

A long list of organizations use Elasticsearch for everything from searching to business analysis, log analytics, security alert monitoring, and application management, as well as using it as a document store. Let’s consider some of these organizations and how they put Elasticsearch to use in their operations.

Uber powers its rider and event prediction using Elasticsearch. It does so by storing millions of events, searching through them, and analyzing the data at a near-real-time rate. Uber predicts demand based on location, time, day, and other variables, including taking past data into consideration. This helps Uber deliver rides pretty.

Netflix adopted the Elastic Stack to provide customer insights to its internal teams. It also uses Elasticsearch for log event analysis to support debugging, alerting, and managing its internal services. Email campaigns and customer operations are all backed by the Elasticsearch engine. The next time you receive an email from Netflix mentioning a newly added movie or TV series, keep in mind that the campaign analytics behind that simple email were all supported by Elasticsearch.

PayPal embraced Elasticsearch as a search engine to allow customers to store and search through their transactions. The company has implemented transaction search features along with analytics for use by merchants, end customers, and developers.

Similarly, the online e-commerce company eBay adopted Elasticsearch to support full-text searching by end users. As users, we are using Elasticsearch directly when searching through eBay’s inventory. The company also uses the Elastic Stack for analytics, log monitoring, and storing transactional data in a document store.

GitHub, a popular code repository for developers, indexes its 8 million (and counting) code repositories—consisting of over 2 billion documents—with Elasticsearch to enable a powerful search experience for its users. Similarly, Stack Overflow uses Elasticsearch to provide developers with quick and relevant answers, and Medium (a popular blog platform) uses the Elastic Stack to serve reader queries in a near-real-time mode.

Generative AI and modern search

Unless you are living in a cave, you have undoubtedly heard about a recent internet revolution: ChatGPT. ChatGPT is a generative AI tool that was developed and released by the OpenAI team in November 2022. Is isn’t often that a technically superior tool lands in the hands of the general public that can help them in unimaginable ways, such as creating a travel itinerary for a summer trip to Athens, summarizing a legal document in layman’s terms, developing a self-help plan for losing weight, analyzing code for security and performance bugs, designing an application’s data models, comparing and contrasting technologies for a specific use case, writing complaint letters to the Twitter CEO and much more.

ChatGPT (https://chat.openai.com) is a conversational agent (chatbot) built on a GPT (generative pretrained transformer) architecture, and it is capable of generating human-like text based on user prompts. It is an instance of a large language model (LLM) designed for conversation with a specific goal of generating safe, relevant content while engaging in a meaningful dialogue. The model is fed vast amounts of text data and learns to predict the next word in a sentence. It is trained using a diverse range of internet text, but it can also be fine-tuned with specific datasets for various tasks. Through this process, the model learns parts and parcels of human language text: grammar, punctuation, syntax, facts about the world, and some degree of reasoning ability.

NOTE LLM is a broad term that refers to any large-scale model trained to understand or generate human-like text. These models are characterized by their vast number of parameters and their ability to handle a wide range of natural language processing tasks. LLMs can be based on various architectures and training methods.

With the release of ChatGPT to the public, a sudden race in the space of search using AI emerged overnight. ChatGPT has become a disruptor to many industries and is no less a threat to Google search. AI-backed tools similar to ChatGPT will disrupt many industries in the coming years. Under tremendous pressure—and possibly to save its search leader status—Google decided to unleash its version of conversational generative AI: its agent called Bard (https://bard.google.com) was made publicly available in May 2023.

In the meantime, Microsoft committed to investing $10 billion in ChatGPT, over and above its initial investment of $3 billion since 2019. Microsoft’s Edge browser is integrated with ChatGPT via the Bing search engine, which also was made publicly available in May 2023. In addition, Microsoft rolled out AI-powered Microsoft 365 apps, so the AI agent is available in Microsoft Word, Excel, email, and other tools. Meta’s LLaMA is another tool that has begun competing in the generative AI race.

Search engineers are being asked fundamental questions, including how generative AI agents will change the course of search. Let’s answer this question by asking ChatGPT how AI agents can complement or help modern search or change its direction. These are the areas where tools like generative AI will reshape the search space:

• Intuitive Search—Search queries will become more conversational and intuitive. Generative AI models like GPT-4 have an advanced understanding of natural language, enabling them to interpret complex queries more effectively. Users will no longer need to rely on specific keywords or phrases; they can simply ask questions as they would in a conversation with another person. This will allow for more accurate and relevant search results, as AI can better comprehend the context and intent of the query. With the introduction and release of highly capable generative AI agents and models, there is a greater scope to significantly reshape full-text search capabilities provided by modern search engines like Elasticsearch. We can expect to see several key changes that will redefine the search experience for both users and developers as this technology is increasingly integrated into search platforms.
• Personalized search —With the incorporation of generative AI, search results can become more personalized and adaptive. Search engines will be able to learn many valuable data points from users’ preferences, behavior, and search history, which in turn will help engines tailor results to meet individual users’ needs. As the AI gathers more data, it will continually refine its understanding of what users are looking for, leading to an increasingly customized search experience.
• Predictive search —Generative AI has the potential to make search engines more proactive in anticipating user needs. Instead of merely responding to queries, AI-driven search engines may be able to predict what information users are interested in, based on their previous interactions or current context. This will allow search platforms to proactively offer relevant suggestions, increasing the value of the search experience and reducing the need for users to perform additional queries.
• Advanced search —Generative AI will enable search engines to provide more diverse and rich search results. By understanding the context and semantics of a query, AI-driven search engines can generate content summaries and relevant visualizations and even synthesize new information to help answer a user’s question. This will lead to a more comprehensive and informative search experience that goes beyond merely linking to existing content.

Summary

• Search is the new normal and the most sought-after functionality for organizations, enabling competitive advantage.
• Search engines built using relational databases as the backend used to serve our search purposes but can’t fulfill the full-fledged search functionality found in modern search engines.
• Modern search engines provide multifaceted, full-text search capabilities and multifold benefits from basic search to advanced search and analytical functions, all with split-second performance. They are also expected to handle terabytes to petabytes of data and scale if needed.
• Elasticsearch is an open source search and analytics engine built over Apache Lucene. It is a highly available server-side application developed in Java.
• Because Elasticsearch was designed as a programming-language-agnostic product, communication with the server takes place over HTTP using rich RESTful APIs. These APIs receive and send data in JSON format.
• The Elastic Stack is a suite of products composed of Beats, Logstash, Elasticsearch, and Kibana. Beats are single-purpose data shippers, Logstash is a data-processing ETL (extract, transform, load) engine, Kibana is the administrative UI tool, and Elasticsearch is the heart and soul of the stack.
• The Elastic Stack enables an organization to position itself in three core areas: search, observability, and security.
• Elasticsearch has become popular over the last few years due to its structured/unstructured search and analytics capabilities; rich set of RESTful APIs; its schema-free nature; and performance, high-availability, and scalability characteristics.
• AI-powered search is here. With the advent of generative AI and ChatGPT, the search space will be explored further, and search will become more intuitive and predictive.

Excerpt From Elasticsearch in Action, Second Edition Madhusudhan Konda.