Tutorial: Bayesian Networks in Educational Assessment
Florida State University
Roy Levy, Arizona State University
Duanli Yan, ETS
Diego Zapta, ETS
This is a collection of material related to our 2017 NCME Tutorial.
This will also be available via memory stick at the tutorial. See you
in San Antonio!
Instructions for Attendees. There is now a "live computing"
exercise included in the seminar. To do this we are recommending
everybody who can bring a laptop.
If you don't have a laptop, hopefully you will be able to
share with somebody who does. We are also recommending you do the
- Download student/demonstration version of the software Netica
from Norsys. (Other possible
software packages are listed below, but we
will be preparing the exercises in Netica.) You can try this out
using the student/demonstration version. We will have a temporary
license code at the workshop.
- Download and install the appropriate version of R
from CRAN. You do not need
the absolute latest version, if you already have R version 3.X
installed, you should be fine.
- [Optional] Many people prefer to run R from R Studio. You can
download the free community edition of R Studio
- Install the package
rjags from the CRAN library.
You can do this by issuing the
install.packages("rjags") in R after starting
- Download the code for CPTtools, RNetica, Peanut and PNetica
packages. These packages are not yet on CRAN, but can be found on
homepage. The following table has the latest versions. [Note
that compiling RNetica from source (required for Unix versions)
requires downloading the Netica C API from Norsys, see the INSTALL
file in the tarball or the RNetica
homepage for details.]
- Download the example networks to be used (See Under Each session).
Mac and Linux usesrs Netica should run without problems in a variety
of Windows emulators. In particular, it should run
under WINE. I (Russell) have
had success using WINE under both Mac OS X (version 10.6.8 up) and
Ubuntu Linux (version 12.04 up). There are several options:
- Linux users: WINE is available through many major Linux
distributions (including Macports).
- Mac users:
is a Mac app for installing Windows programs inside of a special
wrapper providing the Windows (i.e., WINE) services. The first
time you use Wineskin it downloads extra material (wrappers and
engines) from the internet, so do this in a place where you have a
- Mac only: I have created a Wineskin wrapped version of
Netica MacNetica1. This is
an unlicensed version of Netica, you still need to purchase a
license from Norsys (although
the unlicensed student version is adequate for the tutorial).
- Both Linux and Mac: Codeweavers has a commercially supported
version of Wine
called Crossover Mac
or Linux. They give excellent support.
We will have this material on a CD-ROM and Memory stick at the
tutorial, so don't worry if you only have a slow internet connection.
This tutorial follows the book Bayesian Networks in Educational
Assessment (Almond, Mislevy, Steinberg, Yan and Williamson,
2015). The first part (Sessions I and II) contain an overview of
Bayesian networks (Part I of the book) giving some examples of how
they can be used. The second part (Sessions III and IV) look at
software and techniques for building networks from expert opinion
Bayesian networks are a technique for managing
multidimensional models. By representing the variables of the model
as nodes in the graph and using edges in the graph to represent
patterns of dependence and independence among the variables, the model
graph serves as a bridge between educational and psychometric experts,
and further helps the computer derive efficient computational
This tutorial is based on the
Networks in Educational Assessment now out from
- I. Evidence Centered Design and Bayesian Networks
- Covers basic models of ECD and their application to Bayes
Session I networks (Netica).
- II. Bayes Net Applations including ACED
- This part looks at a number of simple applications of Bayes
nets to provide more intution about how they work.
Simple Example Networks (Netica),
ACED Subset (Netica),
- III. RNetica and CPTtools
- This looks at Tools for using and building Bayesian networks in
R, particularly, the CPTtools and RNetica packages. It includes
examples in scoring and using the built-in EM algorithm to fit
models to data. The talk is split into two sets of slides.
RNetica Slides (PDF)
RNetica Handout (PDF).
Learning CPTs Slides (PDF)
Learning CPTs Handout (PDF).
A simple Learning Example.
- IV. Advanced Topics
- Covers two topics. Learning with Markov chain Monte Carlo
(MCMC) and dynamic Bayesian networks (networks which unfold across
MCMC Slides (PDF)
MCMC Handout (PDF).
DBN Slides (PDF)
DBN Handout (PDF).
All Session IV networks (Netica)
- Bayes net and ECD Bibliography (Note:
this is an out of date version of the book bibliography).
The handout version is also available as one big file containing all
sessions and the bibliography. Honkin'
big handout (PDF).
For quick reference, here are the on-line resources referenced in the
This is a partial list of software packages we have used or think are
worth paying attention to. The list of Bayes net software found at
the bottom of the Bayesian network Wikipedia
is a reasonably complete and up to date list of both free and
- Netica (Norsys Software Crop)
Netica is another very complete commercial grade Bayes net
engine, includes some learning tools.
- RNetica (Netica API for R)
This is a work in progress binding for the Netica API into the R
language. Currently only source verison is available. (Windows
and Mac binaries will be available at the conference).
- Genie/Smile (Decision Systems Lab, Univ. of Pittsburgh)
Open source project, free under Gnu Public License. Also
contains a ``translator'' which translates between network
Useful (Bayesian) Statistical Software
- BUGS (Bayesian inference Using Gibbs Sampling).
Downloadable version for Windows. BUGS is no longer actively
maintained. For serious work, I recommend OpenBUGS
- JAGS (Just Another Gibbs Sampler)
A rewrite of Classic BUGS (command line only, no GUI support)
that runs under Linux, MacOS X, and Windows.
- FBM: Flexible Bayesian Modeling
Radford Neal's Flexible Bayesian Modeling and Markov Chain
General purpose statistical computing environment based on S
- http://mc-stan.org/Stan is a
package for obtaining Bayesian inference using the No-U-Turn
sampler, a variant of Hamiltonian Monte Carlo.
- ECD Wiki
Email Russell to get a password to contribute to the discussion.
- Book page on the Wiki
We are slowly working at getting sample networks, errata and other
resources for working through the book up at this site.
- ACED Page on ECD Wiki
Complete data from ACED field trial and ACED Bayes net are
available at this site. This is a Wiki using the same user
name and password as the ECD wiki.
- Heckerman tutorial on learning (Heckerman, D. )
Note: Other Microsoft Research technical reports are available
- Association for Uncertainty in Artificial Intelligence home page
UAI conference proceedings is the most important publication in this
- CRESST Technical Report Archive
Early versions of many of the Mislevy references (including in press
references) are available here. (Hint: search for
``Mislevy''). The CRESST web site changes frequently, so
this link may be out of date. If the link is broken, google
- CiteSeer Cross-Reference Database
On-line cross reference database with lots of articles on Bayes nets.
Many of the bibliography entries are available through CiteSeer.
All handouts and slides from this tutorial are unpublished work of
their respective authors. Sessions I and II are Copyright 2002–17 by
Educational Testing Service. Session III (RNetica and Learning CPTs)
is Copyright 2017 by Russell G. Almond with some material copyright
2002–15 by ETS (Used by permission). Session IV (MCMC and DBN)
is copyright 2017 by Roy Levy.
These materials are an unpublished, proprietary work of their
respective authors. Any
limited distribution shall not constitute publication. This work may not be
reproduced or distributed to third parties without the author's prior
written consent. Submit request for the ETS material through
http://www.ets.org/legal/copyright.html. Requestions for material
from Russell Almond
and Roy Levy can be obtained
from the respective authors.
ACED development and data collection was
sponsored by National Science Foundation Grant No. 0313202. Thanks
to Val Shute for permission to use ACED data in this tutorial.
almond (at) acm.org
ralmond (at) fsu.edu
Last modified: Friday, Apr 21, 2017