| Menu | Next Post: Elastic Anomaly Detection - Categorization |

As the name suggests, the algorithm needs to identify anomalies in the data.

But how does the model identify anomalies?

How do we identify anomalies?

For example, considering the image bellow.

What is abnormal in this image?

What if I add something to this image? Now, considering the updated image below, what is abnormal?

It was probably easier with the second image because the cat is not a dog, making the cat the anomaly in this image, for most people. In this process, you are identifying patterns.

Identifying patterns is an essential part of our learning process, but the answers are not necessarily obvious, because you know what a cat is and what a dog is, not from the pictures I showed you, I never told you this, but because you learned it during your life.

We must always remember that the algorithms will only process the data that you choose to share.

In the case of a child who is still learning the difference between a dog and a cow, for example, we might receive the answer that all animals in the image belong to the same category: animals. This answer is not incorrect; it simply applies different criteria based on similar characteristics observed in the available data.

If we are seeking a more specific answer, considering all possible details, variables, and behavior, we need to ensure that all data that could contribute to the answer is analyzed over time. The more data we have, the better our understanding will be.

In the case of a child, for them to identify the cat as 'abnormal' they would need more examples, more data would need to be “analyzed” over time. The conclusion is the same for the algorithms.

Based on this information, you already know that the question 'What is abnormal?' is answered by taking into account what is considered normal (which can vary), and to determine what is normal, the algorithm identifies patterns over time.

There are multiple types of Anomaly Detection analyses available in Elastic's ML solution, including:

• Single Metric analysis, for jobs that analyze a single time series;

• Multi-Metric analysis, to split a single time series into multiple time series;

• Population analysis, to identify abnormal behaviors in a homogeneous "population" over a period of time;

• Categorization analysis, which is a machine learning process that tokenizes a text field, clusters similar to data together, and classifies it into categories;

The Anomaly Detection feature analyzes the input stream of data, models its behavior using techniques to construct a model that best matches your data, and performs analysis based on the detectors you defined in your job, considering possible rules and dates you want to ignore or disqualify from being modeled.

The blue line in the chart represents the actual data values, while the shaded blue area represents the bounds for the expected values. Initially, the range of expected values is wide due to a limited amount of data in the analyzed time period. Consequently, the model fails to capture the periodicity in the data.

After processing more data, a model is built with coefficients that result in expected values close to the actual values. This leads to the shaded blue area being close to the blue line. By comparing the values to this area, we can determine if they fall outside of it and monitor the anomaly score to indicate the severity of potential anomalies.

Anomaly Score

The anomaly score (severity) is a value from 0 to 100, which indicates the significance of the observed anomaly compared to previously seen anomalies. Highly anomalous values ​​are shown in red.

In order to provide a sensible view of the results, an anomaly score is calculated for each bucket time interval (we use the concept of a bucket to divide up a continuous stream of data into batches, between 10 minutes and 1 hour, for processing).

When you review your machine learning results, there is a multi_bucket_impact property that indicates how strongly the final anomaly score is influenced by multi-bucket analysis; anomalies with medium or high impact on multiple buckets are represented with a cross symbol instead of a circle.

| Menu | Next Post: Elastic Anomaly Detection - Categorization |

This post is part of a series that covers Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution, aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.