classification

This article describes how Bayesian networks can be used for classification.

A classification walkthrough using Bayes Server™ is also available.

what is classification?

Classification is the process of using a model to predict unknown values (output variables), using a number of known values (input variables). For example we might want to predict whether a stock market is currently a bull or a bear market, based on a number of market indicators, or we might want to predict whether a patient has a certain disease given a number of symptoms.

In order to perform classification, first we need to model the relationship between the input variables and the output variables we are predicting. This process involves learning a model using data in which both the input variables and the output variables are present. Expert opinion can also be used to build/enhance a model. This model can subsequently be used on unseen data in which only the input data is present, in order to predict the output variables.

Classification is termed a supervised learning approach, because a model is trained specifically for the purpose of predicting the output variable.

Typically, the term classification is concerned with predicting discrete variables. The term regression is used when predicting continuous variables.

classification with Bayesian networks

Bayesian networks are widely used to perform classification tasks, with the following advantages.

• Based on probability theory.
• Not a black box approach.
• Allows rich structure.
• Can mix expert opinion and data to build models
• Backwards reasoning - in addition to predicting outputs given inputs, we can use output values to infer inputs.
• Support for missing data during learning and classification.

Image 1 depicts the possible structure of a Bayesian network used for classification. The dotted lines denote potential links, and the blue box is used to indicate that additional nodes and links can be added to the model, usually between the input and output nodes.

Image 1 - generic structure of a Bayesian network classifier

In order to perform classification with a Bayesian network such as the one depicted in Image 1, first evidence must be set on the input nodes, and then the output nodes can be queryied using standard Bayesian network inference. The result will be a distribution for each output node, so that you can not only determine the most probable state for each output, but also see the probability assigned to each output state.

naive Bayes model

Image 2 shows the structure of a Naive Bayes classifier, which is the simplest form of useful Bayesian network classifier. (We use the term useful, as actually this is not the simplest possible classifier. We could perform classification using a model with no links, however a model of this type would have limited use).

The links in a Naive Bayes model are directed from output to input, which gives the model its simplicity, as there are no interactions between the inputs, except indirectly via the output. Note however that directing links from output to input, is not a requirement for all Bayesian network classifiers.

Image 2 - Naive Bayes model

more complex models

While a Naive Bayes classifier performs well on some tasks, often additional links and nodes provide a richer, more accurate model. Additional links can be added between input variables, or output variables, and between additional structure shown in Image 1. For example, additional nodes that are not input or output nodes, may be used to model more complex distributions such as mixtures of Gaussians, or to encode logical relationships between inputs.

temporal models

Dynamic Bayesian networks are used to model time series/sequential data. In the same way that Bayesian networks can be used for classification, Dynamic Bayesian networks allow the classification of an output in the present, past or future. For example we may wish to predict whether a stock market will be a bull or a bear market over a number of future time steps.

feedback

send feedback