Tutorial: Bayesian Networks in Educational Assessment
Russell Almond,
Florida State University
Roy Levy, Arizona State University [Not available in 2021]
Duanli Yan, ETS
Diego Zapta, ETS
This is a collection of material related to our 2021 NCME Tutorial.
See you online!
Instructions for Attendees. There is now a "live computing"
exercise included in the seminar. To do this we are recommending
everybody who can bring a laptop.
We are also recommending you do thefollowing steps:
 Download student/demonstration version of the software Netica
from Norsys. (Other possible
software packages are listed below, but we
will be preparing the exercises in Netica.) You can try this out
using the student/demonstration version, which is sufficient for the
exercises in the workshop.
 Download and install the appropriate version of R
from CRAN. You do not need
the absolute latest version, if you already have R version 4.X
installed, you should be fine.
 Download and install the the free community edition of R Studio
from RStudio.com.
 Install the package
rjags
from the CRAN library.
You can do this by issuing the
command install.packages("rjags")
in R after starting
it. This also requires you to load install the program JAGS from
http://mcmcjags.sourceforge.net/.
 Launch R Studio, and install the following packages:
arm,R.utils,shinyjs,rmarkdown,futile.logger,devtools
 Download the code for CPTtools, RNetica, Peanut and PNetica
packages. These packages are not yet on CRAN, but can be found on
the RNetica
homepage. The following table has the latest versions. [Note
that compiling RNetica from source (required for Unix versions)
requires downloading the Netica C API from Norsys, see the INSTALL
file in the tarball or the RNetica
homepage for details.]
 Download the example networks to be used (See Under Each session).
Mac and Linux usesrs Netica should run without problems in a variety
of Windows emulators. In particular, it should run
under WINE. I (Russell) have
had success using WINE under both Mac OS X (version 10.6.8 up) and
Ubuntu Linux (version 12.04 up). There are several options:
 Linux users: WINE is available through many major Linux
distributions (including Macports).
 Mac users:
Wineskin
is a Mac app for installing Windows programs inside of a special
wrapper providing the Windows (i.e., WINE) services. The first
time you use Wineskin it downloads extra material (wrappers and
engines) from the internet, so do this in a place where you have a
good connection.
 Mac only: I have created a Wineskin wrapped version of
Netica NeticaMac(older
version MacNetica1). This is
an unlicensed version of Netica, you still need to purchase a
license from Norsys (although
the unlicensed student version is adequate for the tutorial).
 Both Linux and Mac: Codeweavers has a commercially supported
version of Wine
called Crossover Mac
or Linux. They give excellent support.
This tutorial follows the book Bayesian Networks in Educational
Assessment (Almond, Mislevy, Steinberg, Yan and Williamson,
2015). The first part (Sessions I and II) contain an overview of
Bayesian networks (Part I of the book) giving some examples of how
they can be used. The second part (Sessions III and IV) look at
software and techniques for building networks from expert opinion
and data.
Bayesian networks are a technique for managing
multidimensional models. By representing the variables of the model
as nodes in the graph and using edges in the graph to represent
patterns of dependence and independence among the variables, the model
graph serves as a bridge between educational and psychometric experts,
and further helps the computer derive efficient computational
strategies.
This tutorial is based on the
book Bayesian
Networks in Educational Assessment now out from
Springer.
 I. Evidence Centered Design and Bayesian Networks
 Covers basic models of ECD and their application to Bayes
nets.
Slides (pptx),
Slides (pdf),
Session I networks (Netica).
 II. Bayes Net Applations including ACED
 This part looks at a number of simple applications of Bayes
nets to provide more intution about how they work.
Slides (pptx),
Slides (pdf),
Simple Example Networks (Netica),
ACED Subset (Netica),
 III. RNetica and CPTtools
 This looks at Tools for using and building Bayesian networks in
R, particularly, the CPTtools and RNetica packages. It includes
examples in scoring and using the builtin EM algorithm to fit
models to data. The talk is split into two sets of slides.
[I'm still making some tweaks here. Expect an update around noon.]
RNetica Introduction (PDF)
DiBello Models (PDF).
RNetica and DiBello talks (R
markdown) and sample nets.,
Learning CPTs Slides (PDF)
Learning CPTs Handout (PDF).
A simple Learning Example.
 IV. Advanced Topics
 Learning with Markov chain Monte Carlo
(MCMC) and dynamic Bayesian networks (networks which unfold across
time). [Actually, as Roy is unable to make it this year, we won't
cover Dynamic Bayes nets this year.]
MCMC Slides (pptx)
MCMC Slides (pdf)
All Session IV networks (Netica)
 Bibliography
 Bayes net and ECD Bibliography (Note:
this is an out of date version of the book bibliography).
The handout version is also available as one big file containing all
sessions and the bibliography. Honkin'
big handout (PDF) PDF/X version for
printing. [We didn't have a chance to redo the handout this
year, this is still last years version.]
Online Resources
For quick reference, here are the online resources referenced in the
bibliography.
This is a partial list of software packages we have used or think are
worth paying attention to. The list of Bayes net software found at
the bottom of the Bayesian network Wikipedia
entry http://en.wikipedia.org/wiki/Bayesian_network
is a reasonably complete and up to date list of both free and
commercial software.
 Netica (Norsys Software Crop)
 http://www.norsys.com/
Netica is another very complete commercial grade Bayes net
engine, includes some learning tools.
 RNetica (Netica API for R)
 http://pluto.coe.fsu.edu/RNetica
This is a work in progress binding for the Netica API into the R
language. Currently only source verison is available. (Windows
and Mac binaries will be available at the conference).
 Genie/Smile (Decision Systems Lab, Univ. of Pittsburgh)
 http://genie.sis.pitt.edu/
Open source project, free under Gnu Public License. Also
contains a ``translator'' which translates between network
formats.
Useful (Bayesian) Statistical Software
 BUGS (Bayesian inference Using Gibbs Sampling).

http://www.mrcbsu.cam.ac.uk/bugs/welcome.shtml
Downloadable version for Windows. BUGS is no longer actively
maintained. For serious work, I recommend OpenBUGS
http://mathstat.helsinki.fi/openbugs/
 JAGS (Just Another Gibbs Sampler)

https://sourceforge.net/projects/mcmcjags/
A rewrite of Classic BUGS (command line only, no GUI support)
that runs under Linux, MacOS X, and Windows.
 FBM: Flexible Bayesian Modeling

http://www.cs.utoronto.ca/~radford/fbm.software.html
Radford Neal's Flexible Bayesian Modeling and Markov Chain
Sampler.
 R
 http://www.rproject.org/
General purpose statistical computing environment based on S
language.
 Stan
 http://mcstan.org/Stan is a
package for obtaining Bayesian inference using the NoUTurn
sampler, a variant of Hamiltonian Monte Carlo.
 ECD Wiki
 http://ecd.ralmond.net/ecdwiki/
Email Russell to get a password to contribute to the discussion.
 Book page on the Wiki
 http://ecd.ralmond.net/ecdwiki/BN/BN.
We are slowly working at getting sample networks, errata and other
resources for working through the book up at this site.
 ACED Page on ECD Wiki
 http://ecd.ralmond.net/ecdwiki/ACED/ACED
Complete data from ACED field trial and ACED Bayes net are
available at this site. This is a Wiki using the same user
name and password as the ECD wiki.
 Heckerman tutorial on learning (Heckerman, D. [1995])

ftp://ftp.research.microsoft.com/pub/tr/tr9506.pdf
Note: Other Microsoft Research technical reports are available
online from
http://www.research.microsoft.com/
 Association for Uncertainty in Artificial Intelligence home page
 http://www.auai.org/
UAI conference proceedings is the most important publication in this
area.
 CRESST Technical Report Archive

http://www.cse.ucla.edu/products/reports.asp
Early versions of many of the Mislevy references (including in press
references) are available here. (Hint: search for
``Mislevy''). The CRESST web site changes frequently, so
this link may be out of date. If the link is broken, google
"CRESST Reports".
 CiteSeer CrossReference Database
 http://citeseer.ist.psu.edu/cis
Online cross reference database with lots of articles on Bayes nets.
Many of the bibliography entries are available through CiteSeer.
Copyright
All handouts and slides from this tutorial are unpublished work of
their respective authors. Sessions I and II are Copyright 2002–18 by
Educational Testing Service. Session III (RNetica and Learning CPTs)
is Copyright 2017–18 by Russell G. Almond with some material copyright
2002–15 by ETS (Used by permission). Session IV (MCMC and DBN)
is copyright 2017–18 by Roy Levy.
These materials are an unpublished, proprietary work of their
respective authors. Any
limited distribution shall not constitute publication. This work may not be
reproduced or distributed to third parties without the author's prior
written consent. Submit request for the ETS material through
http://www.ets.org/legal/copyright.html. Requestions for material
from Russell Almond
and Roy Levy can be obtained
from the respective authors.
ACED development and data collection was
sponsored by National Science Foundation Grant No. 0313202. Thanks
to Val Shute for permission to use ACED data in this tutorial.
Work on RNetica, CPTtools and Peanut has been sponsored in part by
the Physics Playground and ERebuild projects, on which
many of the tools were tested. Physics Playground development
was supported by the Bill & Melinda Gates Foundation grant
Games as Learning/Assessment: Stealth Assessment (#0PP1035331,
Val Shute, PI) and the National Science foundation grant DIP:
Gamebased Assessment and Support of STEMrelated Competencies
(#1628937, Val Shute, PI). The project Mathematical Learning via
Architectual Design and Modeling Using ERebuild. (#1720533,
Fengfeng Ke, PI) has provided additional support. The tool
development parts of each of these projects was led by Russell Almond .
almond.g.almond (at) gmail.com
ralmond (at) fsu.edu
Last modified: Thursday, June 3, 2021.