testerConfusion {RNetica} | R Documentation |
"NetworkTester"
The NetworkTester
object is created by calls to
the testNetwork
function, and contains the cached
results from the test. These function return the cached results.
testerNet(tester) testerTarget(tester) testerIgnore(tester) testerErrorRate(tester, node = NULL) testerLogLoss(tester, node = NULL) testerQuadraticLoss(tester, node = NULL) testerConfusion(tester, node = NULL) testerKappa(tester, node = NULL, weights = c("None", "Linear", "Quadratic"), W = NULL) testerLambda(tester, node = NULL, weights = c("None", "Linear", "Quadratic"), W = NULL) ## S3 method for class 'NetworkTester' summary(object, ...)
tester |
An object of class |
node |
One of the target nodes for tester (an element of
|
weights |
Loss weights for Kappa and Lambda. See details. |
W |
A loss matrix for Kappa and Lambda. If |
object |
An object of class |
... |
Other arguments to |
The testNetwork
function takes as input generated data
with values for both the testTarget
nodes and the observed nodes
(the nodes that are not in the testerTarget
or
testerIgnore
lists. It estimates the values for the generated
nodes and compares the estimated values with the actual values from
the data set. The results are stashed in a
NetworkTester
object. The functions described
here extract information from the NetworkTester
object.
The function testerErrorRate
gives the fraction of cases where
the predicted value for the node was different from the actual value.
If node
is specified, the value for that node is returned,
otherwise a vector of values for all nodes are returned.
The log loss (also called the score) for target node
X in case i is defined as the - log (P(X_i = c_i)) ;
it ranges from 0 to infinity with lower scores being better. The
function testerLogLoss
gives the average log loss over all
available test cases. If node
is specified, the value for that
node is returned, otherwise a vector of values for all nodes are
returned.
The quadratic loss (also called the Brier score) for
target node X in case i is defined as the 1 - 2 P(X_i
= c_i)) + ∑_{k=1}^K (P(X_i=k))^2 ; it ranges from 0 to 2 with
lower scores being better. The function testerQuadraticLoss
gives the average quadratic loss over all available test cases. If
node
is specified, the value for that node is returned,
otherwise a vector of values for all nodes are returned.
The function testerConfusion
returns the confusion matrixes
associated with each node in the testerTarget
node list. This
is a square matrix whose rows and columns correspond to the predicted
and actual values of the node respectively. The value in cell (i, j)
of cases for which the predicted value for the node was in State
i and the actual value was in State j. There is one
confusion matrix for each target node. If the node
argument is
specified, then the confusion matrix for that node is returned,
otherwise a named list of confusion matrixes is returned.
Cohen's κ and Goodman and Kruskal's λ provide ways of summarizing a confusion matrix (Almond et al., 2015). One obvious way to summarize the confusion matrix is to look at the proportion of results along the main diagonal. This is the error rate described above. However, if one category is very common in the population, the error rate could be very high because of the base rate and not because of the power of the classification system. Kappa and lambda are two ways to normalize the error rate to avoid the base rate issue. Kappa assumes that the two raters (the predicted and the actual) are both rating by chance at their marginal frequencies. It adjusts the base rate for the proportion of time the two agree by chance. Lambda compares the predicted results to a predictor who always predicts the most common category; thus is indicates how much better a differentiated approach is to one that treats all cases equally. Both values are numbers between -1 and 1 (similar to correlations) with 1 indicating perfect agreement, -1 perfect disagreement and 0 the two measures are unrelated.
For both kappa and lambda, the analyst can specify a weight matrix (1
- loss matrix) W
which indicates how desirable it is to
classify a subject as i, when they really are a j. This
is usually scaled so that the main diagnoal is 1 and the smallest
value if 0 (or at least all values are positive). If the states are
ordered, common choices of weight are linear weights (weights decrease
linearly as they move from the diagonal) and quadratic weights
(weights decrease quadratically). These can be specified using the
keywords for the weights
. The default keyword is “None”
which produces the identify matrix (the unweighted case).
The functions testerNet
returns the
NeticaBN
on which the test was run.
The functions testerTarget
and testerIgnore
both return
lists of NeticaNode
objects. In the first case
the target nodes.
The functions testerErrorRate
, testerLogLoss
,
testerQuadraticLoss
, testerKappa
and testerLambda
give a vector of values for the respecrive function. If node
is specified, the value for that node is returned, otherwise a named
vector of values for all nodes is returned.
The function testerConfusion
returns a single confusion matrix
if node
is specified or a named list of confusion matrixes.
The summary
method for NetworkTester
objects returns a data frame where the rows represent the target nodes
and the columns have the following values:
testerErrorRate(object)
testerLogLoss(object)
testerQuadraticLoss(object)
testerKappa(object)
testerKappa(object,weights="Quadratic")
testerLambda(object)
testerLambda(object,weights="Linear")
Russell Almond
Almond, R.G., Mislevy, R.J. Steinberg, L.S., Yan, D. and Willamson, D. M. (2015). Bayesian Networks in Educational Assessment. Springer. Chapter 7.
http://norsys.com/onLineAPIManual/index.html: NewNetTester_bn(), DeleteNetTester_bn(), TestWithCaseset_bn(), GetTestConfusion_bn(), GetTestErrorRate_bn(), GetTestLogLoss_bn(), GetTestQuadraticLoss_bn()
The output of testNetwork
is NetworkTester
.
sess <- NeticaSession() startSession(sess) irt5 <- ReadNetworks(file.path(library(help="RNetica")$path, "sampleNets","IRT5.dne"), session=sess) irt5.theta <- NetworkFindNode(irt5,"Theta") irt5.x <- NetworkFindNode(irt5,paste("Item",1:5,sep="_")) CompileNetwork(irt5) ## This generates a fixed series of random cases and saves them to a ## file. N <- 100L rnodes <- c(list(irt5.theta),irt5.x) casefile <- tempfile("irt5testcase",fileext=".cas") filestream <- CaseFileStream(casefile, session=sess) rng <- NewNeticaRNG(123456779, session=sess) WithOpenCaseStream(filestream, WithRNG(rng, for (n in 1L:N) { GenerateRandomCase(rnodes,rng=rng) WriteFindings(rnodes,filestream,n) lapply(rnodes,RetractNodeFinding) # Only retract findings for # generated nodes })) irt5.test <- testNetwork(list(irt5.theta),OpenCaseStream(filestream)) summary(irt5.test) DeleteNetwork(irt5) stopSession(sess)