testerConfusion {RNetica}R Documentation

Accessor methods for class "NetworkTester"

Description

The NetworkTester object is created by calls to the testNetwork function, and contains the cached results from the test. These function return the cached results.

Usage

testerNet(tester)
testerTarget(tester)
testerIgnore(tester)
testerErrorRate(tester, node = NULL)
testerLogLoss(tester, node = NULL)
testerQuadraticLoss(tester, node = NULL)
testerConfusion(tester, node = NULL)
testerKappa(tester, node = NULL, weights = c("None", "Linear", "Quadratic"), W = NULL)
testerLambda(tester, node = NULL, weights = c("None", "Linear", "Quadratic"), W = NULL)
## S3 method for class 'NetworkTester'
summary(object, ...)

Arguments

tester

An object of class NetworkTester.

node

One of the target nodes for tester (an element of testerTarget()). If supplied, only results for that node are returned.

weights

Loss weights for Kappa and Lambda. See details.

W

A loss matrix for Kappa and Lambda. If weights are specified, the loss matrix is defined according to the pattern.

object

An object of class NetworkTester to be summarized.

...

Other arguments to summary, ignored.

Details

The testNetwork function takes as input generated data with values for both the testTarget nodes and the observed nodes (the nodes that are not in the testerTarget or testerIgnore lists. It estimates the values for the generated nodes and compares the estimated values with the actual values from the data set. The results are stashed in a NetworkTester object. The functions described here extract information from the NetworkTester object.

The function testerErrorRate gives the fraction of cases where the predicted value for the node was different from the actual value. If node is specified, the value for that node is returned, otherwise a vector of values for all nodes are returned.

The log loss (also called the score) for target node X in case i is defined as the - log (P(X_i = c_i)) ; it ranges from 0 to infinity with lower scores being better. The function testerLogLoss gives the average log loss over all available test cases. If node is specified, the value for that node is returned, otherwise a vector of values for all nodes are returned.

The quadratic loss (also called the Brier score) for target node X in case i is defined as the 1 - 2 P(X_i = c_i)) + ∑_{k=1}^K (P(X_i=k))^2 ; it ranges from 0 to 2 with lower scores being better. The function testerQuadraticLoss gives the average quadratic loss over all available test cases. If node is specified, the value for that node is returned, otherwise a vector of values for all nodes are returned.

The function testerConfusion returns the confusion matrixes associated with each node in the testerTarget node list. This is a square matrix whose rows and columns correspond to the predicted and actual values of the node respectively. The value in cell (i, j) of cases for which the predicted value for the node was in State i and the actual value was in State j. There is one confusion matrix for each target node. If the node argument is specified, then the confusion matrix for that node is returned, otherwise a named list of confusion matrixes is returned.

Cohen's κ and Goodman and Kruskal's λ provide ways of summarizing a confusion matrix (Almond et al., 2015). One obvious way to summarize the confusion matrix is to look at the proportion of results along the main diagonal. This is the error rate described above. However, if one category is very common in the population, the error rate could be very high because of the base rate and not because of the power of the classification system. Kappa and lambda are two ways to normalize the error rate to avoid the base rate issue. Kappa assumes that the two raters (the predicted and the actual) are both rating by chance at their marginal frequencies. It adjusts the base rate for the proportion of time the two agree by chance. Lambda compares the predicted results to a predictor who always predicts the most common category; thus is indicates how much better a differentiated approach is to one that treats all cases equally. Both values are numbers between -1 and 1 (similar to correlations) with 1 indicating perfect agreement, -1 perfect disagreement and 0 the two measures are unrelated.

For both kappa and lambda, the analyst can specify a weight matrix (1 - loss matrix) W which indicates how desirable it is to classify a subject as i, when they really are a j. This is usually scaled so that the main diagnoal is 1 and the smallest value if 0 (or at least all values are positive). If the states are ordered, common choices of weight are linear weights (weights decrease linearly as they move from the diagonal) and quadratic weights (weights decrease quadratically). These can be specified using the keywords for the weights. The default keyword is “None” which produces the identify matrix (the unweighted case).

Value

The functions testerNet returns the NeticaBN on which the test was run.

The functions testerTarget and testerIgnore both return lists of NeticaNode objects. In the first case the target nodes.

The functions testerErrorRate, testerLogLoss, testerQuadraticLoss, testerKappa and testerLambda give a vector of values for the respecrive function. If node is specified, the value for that node is returned, otherwise a named vector of values for all nodes is returned.

The function testerConfusion returns a single confusion matrix if node is specified or a named list of confusion matrixes.

The summary method for NetworkTester objects returns a data frame where the rows represent the target nodes and the columns have the following values:

ErrorRate

testerErrorRate(object)

LogLoss

testerLogLoss(object)

QuadraticLoss

testerQuadraticLoss(object)

kappa

testerKappa(object)

QWK

testerKappa(object,weights="Quadratic")

lambda

testerLambda(object)

LinearLambda

testerLambda(object,weights="Linear")

Author(s)

Russell Almond

References

Almond, R.G., Mislevy, R.J. Steinberg, L.S., Yan, D. and Willamson, D. M. (2015). Bayesian Networks in Educational Assessment. Springer. Chapter 7.

http://norsys.com/onLineAPIManual/index.html: NewNetTester_bn(), DeleteNetTester_bn(), TestWithCaseset_bn(), GetTestConfusion_bn(), GetTestErrorRate_bn(), GetTestLogLoss_bn(), GetTestQuadraticLoss_bn()

See Also

The output of testNetwork is NetworkTester.

Examples

sess <- NeticaSession()
startSession(sess)

irt5 <- ReadNetworks(file.path(library(help="RNetica")$path,
                           "sampleNets","IRT5.dne"), session=sess)

irt5.theta <- NetworkFindNode(irt5,"Theta")
irt5.x <- NetworkFindNode(irt5,paste("Item",1:5,sep="_"))

CompileNetwork(irt5)


## This generates a fixed series of random cases and saves them to a
## file. 
N <- 100L
rnodes <- c(list(irt5.theta),irt5.x)
casefile <- tempfile("irt5testcase",fileext=".cas")
filestream <- CaseFileStream(casefile, session=sess)
rng <- NewNeticaRNG(123456779, session=sess)
WithOpenCaseStream(filestream,
  WithRNG(rng,
    for (n in 1L:N) {
      GenerateRandomCase(rnodes,rng=rng)
      WriteFindings(rnodes,filestream,n)
      lapply(rnodes,RetractNodeFinding) # Only retract findings for
                                        # generated nodes
    }))


irt5.test <- testNetwork(list(irt5.theta),OpenCaseStream(filestream))
summary(irt5.test)


DeleteNetwork(irt5)
stopSession(sess)

[Package RNetica version 0.7-1 Index]