Tutorial 6 - Anomaly detection

• 7/23/2022
• 13 minutes to read

In this tutorial we will demonstrate how to use Bayesian networks to perform anomaly detection on un-seen data.

Anomaly detection is the process of detecting data which is considered unusual or represents fault conditions.

We will use a semi-supervised anomaly detection approach. This entails training a model with data that is considered 'normal'. The 'normal' model can then be used to test un-seen data to determine whether it is also considered 'normal', otherwise it is considered anomalous (unusual), and might represent a faulty system. A semi-supervised approach assumes that a model can be built on training data that is considered 'normal', i.e. we may have had to first remove 'abnormal' data from the training data set.

Supervised anomaly detection requires that each case (row) in the data is labeled in some way to identify it as normal or abnormal. There might be a single column with mutually exclusive states, some of which are normal, and some of which are abnormal. Alternatively there may be multiple columns each representing a particular known anomaly (fault), and the data flags each as either false or true. A classification model is then trained to predict abnormal states, and can then be used on un-seed data to predict the probability of each abnormal state. This approach may not be appropriate if there is insufficient data labeled anomalous, or if it is too expensive to label data, or anomalies of the same type are rare.

Unsupervised anomaly detection uses data that contains both 'normal' and 'abnormal' data. The training process will either need to build a model and automatically remove elements of the model which are deemed 'abnormal', or alternatively data considered anomalous can be automatically removed before training begins. In both cases Bayesian networks can be used to train the model, but an additional algorithm is required to determine which parts of the model or data should be discarded.

The following concepts will be covered:

• Parameter learning
• Log likelihood prediction
• Conflict measure
• Batch queries

Companion video (No Audio)

The 'normal' model

For this tutorial we will use a Bayesian network mixture model as our 'normal' model. We could use a variety of different types of model for our 'normal' model, but whichever type we use, as we are building a semi-supervised model, it will not include nodes/variables representing particular anomalies (faults).

Rather than creating and learning a model from scratch, we will re-use the mixture model created in Tutorial 2 - Mixture model.

This model was learned from data, which for the purposes of this tutorial, we will consider normal. Therefore we consider the model a summary of 'normal' behavior.

This model only contains two continuous inputs in order to keep this tutorial simple, however models with far more continuous and/or discrete inputs can be created.

Opening the model

• Launch Bayes Server, and on the Start page click the network entitled 'Tutorial 2 - Mixture model' in the Sample networks pane.

Using the 'normal' model to test un-seen data

We will use our 'model' to test whether un-seen data is anomalous. We know that the un-seen data is considered 'normal' for the first 300 points, after which the system degrades over the next 100 points. Since we do not have outputs in this model representing anomalies (faults) we will use the log-likelihood statistic and conflict measure to detect anomalous behavior.

Adding a data connection

For convenience, we will use Microsoft Excel as the data source, however another database can be substituted.

• Select the data (including the header) in the un-seen data section and copy it to the clipboard (Ctrl+C).
• Open Microsoft Excel and paste the data into a new Microsoft Excel spreadsheet (Ctrl+V).
• Save the new spreadsheet.
• In Bayes Server, click the Data Connections button on the Data tab, Data Sources group. This will launch the Data connection manager.
• Click the New button on the toolbar. This will launch the Data connection editor.
• In the list of data providers, select the appropriate Excel Driver for the version of Microsoft Excel you are using.
• Next to the File Name text box, click the Ellipsis (...) button, and select the Microsoft Excel spreadsheet created in an earlier step.
• Click the Test Connection button, to ensure the new data connection is working.
• Click OK to add the new Data Connection.

Batch query

• Click the Batch query button, on the Data tab, Learning group. This will launch the Data tables window.

• In the Data Connection drop down, select the new Data Connection created in an earlier step, or the Tutorial data connection if you skipped that step. This should enable the Data drop down.

• In the Data drop down, select the worksheet that contains the data. (If the data is on the first worksheet, select Sheet1$). If you are using the pre-installed Tutorial data connection, select Tutorial 6 - Anomaly.

• Click the OK button. This will launch the Data map window.

• In the Data map window, ensure that variable X has automatically been mapped to column X, and variable Y has automatically been mapped to column Y.

  The window should look like this:

  

• Click the OK button. This will launch the Batch query window.

• In the query pane on the left hand side, ensure the following queries/information columns are checked.

  • LogLikelihood
  • Conflict
  • X
  • Y

• Click the Start button on the Batch Query tab, Batch Query group. This outputs the predictions to the window.

  NOTE

  Instead of outputting to the window, you can also output the predictions to a database. This is essential if you are working with large datasets.

  The window should look like this:

  

Analysis

First we will chart the X and Y data to see whether a univariate analysis would have detected anything unusual. Then we will inspect the log-likelihood statistic and conflict measure, to see if our model detects anything unusual.

Charting the batch predictions

• Select the Charts tab in the Batch query window.

• In the X drop down, select the Case output column, and in the Y drop down, select the X output column.

  The window should look like this:

  

• Click the Plot button.

  The window should look like the image below. The plot of X, does not show anything unusual.

  

• Now change the Y drop down to the Y output column.

• Click the Plot button.

  The window should look like the image below. The plot of Y, does not show anything unusual.

  

• Now change the Y drop down to the LogLikelihood output column.

• Click the Plot button.

  The window should look like this:

  

  The plot of LogLikelihood, clearly shows the system degrading after the first 300 points.

  NOTE

  The LogLikelihood statistic tells us how likely it is that this 'normal' model could have generated the data. Therefore, the lower the log-likelihood, the more unusual the data is.

• Now change the Y drop down to the Conflict output column.

• Click the Plot button.

  The window should look like this:

  

  The plot of Conflict, clearly shows the system degrading after the first 300 points.

  NOTE

  The Conflict measure tells us whether our data is in conflict, or contains rare cases. The more positive the value, the higher the conflict.

  Although there are odd points in the first 300 that have positive conflict values, it is only in the last 100 point that the conflict consistently generates increasingly worse positive values.

Unseen data