Tutorial: Bayesian Networks in Educational Assessment
Florida State University
Roy Levy, Arizona State University [Not available in 2021]
Duanli Yan, ETS
Diego Zapta, ETS
This is a collection of material related to our 2021 NCME Tutorial.
See you online!
Instructions for Attendees. There is now a "live computing"
exercise included in the seminar. To do this we are recommending
everybody who can bring a laptop.
We are also recommending you do thefollowing steps:
- Download student/demonstration version of the software Netica
from Norsys. (Other possible
software packages are listed below, but we
will be preparing the exercises in Netica.) You can try this out
using the student/demonstration version, which is sufficient for the
exercises in the workshop.
- Download and install the appropriate version of R
from CRAN. You do not need
the absolute latest version, if you already have R version 4.X
installed, you should be fine.
- Download and install the the free community edition of R Studio
- Install the package
rjags from the CRAN library.
You can do this by issuing the
install.packages("rjags") in R after starting
it. This also requires you to load install the program JAGS from
- Launch R Studio, and install the following packages:
- Download the code for CPTtools, RNetica, Peanut and PNetica
packages. These packages are not yet on CRAN, but can be found on
homepage. The following table has the latest versions. [Note
that compiling RNetica from source (required for Unix versions)
requires downloading the Netica C API from Norsys, see the INSTALL
file in the tarball or the RNetica
homepage for details.]
- Download the example networks to be used (See Under Each session).
Mac and Linux usesrs Netica should run without problems in a variety
of Windows emulators. In particular, it should run
under WINE. I (Russell) have
had success using WINE under both Mac OS X (version 10.6.8 up) and
Ubuntu Linux (version 12.04 up). There are several options:
- Linux users: WINE is available through many major Linux
distributions (including Macports).
- Mac users:
is a Mac app for installing Windows programs inside of a special
wrapper providing the Windows (i.e., WINE) services. The first
time you use Wineskin it downloads extra material (wrappers and
engines) from the internet, so do this in a place where you have a
- Mac only: I have created a Wineskin wrapped version of
version MacNetica1). This is
an unlicensed version of Netica, you still need to purchase a
license from Norsys (although
the unlicensed student version is adequate for the tutorial).
- Both Linux and Mac: Codeweavers has a commercially supported
version of Wine
called Crossover Mac
or Linux. They give excellent support.
This tutorial follows the book Bayesian Networks in Educational
Assessment (Almond, Mislevy, Steinberg, Yan and Williamson,
2015). The first part (Sessions I and II) contain an overview of
Bayesian networks (Part I of the book) giving some examples of how
they can be used. The second part (Sessions III and IV) look at
software and techniques for building networks from expert opinion
Bayesian networks are a technique for managing
multidimensional models. By representing the variables of the model
as nodes in the graph and using edges in the graph to represent
patterns of dependence and independence among the variables, the model
graph serves as a bridge between educational and psychometric experts,
and further helps the computer derive efficient computational
This tutorial is based on the
Networks in Educational Assessment now out from
- I. Evidence Centered Design and Bayesian Networks
- Covers basic models of ECD and their application to Bayes
Session I networks (Netica).
- II. Bayes Net Applations including ACED
- This part looks at a number of simple applications of Bayes
nets to provide more intution about how they work.
Simple Example Networks (Netica),
ACED Subset (Netica),
- III. RNetica and CPTtools
- This looks at Tools for using and building Bayesian networks in
R, particularly, the CPTtools and RNetica packages. It includes
examples in scoring and using the built-in EM algorithm to fit
models to data. The talk is split into two sets of slides.
[I'm still making some tweaks here. Expect an update around noon.]
RNetica Introduction (PDF)
DiBello Models (PDF).
RNetica and DiBello talks (R
markdown) and sample nets.,
Learning CPTs Slides (PDF)
Learning CPTs Handout (PDF).
A simple Learning Example.
- IV. Advanced Topics
- Learning with Markov chain Monte Carlo
(MCMC) and dynamic Bayesian networks (networks which unfold across
time). [Actually, as Roy is unable to make it this year, we won't
cover Dynamic Bayes nets this year.]
MCMC Slides (pptx)
MCMC Slides (pdf)
All Session IV networks (Netica)
- Bayes net and ECD Bibliography (Note:
this is an out of date version of the book bibliography).
The handout version is also available as one big file containing all
sessions and the bibliography. Honkin'
big handout (PDF) PDF/X version for
printing. [We didn't have a chance to redo the handout this
year, this is still last years version.]
For quick reference, here are the on-line resources referenced in the
This is a partial list of software packages we have used or think are
worth paying attention to. The list of Bayes net software found at
the bottom of the Bayesian network Wikipedia
is a reasonably complete and up to date list of both free and
- Netica (Norsys Software Crop)
Netica is another very complete commercial grade Bayes net
engine, includes some learning tools.
- RNetica (Netica API for R)
This is a work in progress binding for the Netica API into the R
language. Currently only source verison is available. (Windows
and Mac binaries will be available at the conference).
- Genie/Smile (Decision Systems Lab, Univ. of Pittsburgh)
Open source project, free under Gnu Public License. Also
contains a ``translator'' which translates between network
Useful (Bayesian) Statistical Software
- BUGS (Bayesian inference Using Gibbs Sampling).
Downloadable version for Windows. BUGS is no longer actively
maintained. For serious work, I recommend OpenBUGS
- JAGS (Just Another Gibbs Sampler)
A rewrite of Classic BUGS (command line only, no GUI support)
that runs under Linux, MacOS X, and Windows.
- FBM: Flexible Bayesian Modeling
Radford Neal's Flexible Bayesian Modeling and Markov Chain
General purpose statistical computing environment based on S
- http://mc-stan.org/Stan is a
package for obtaining Bayesian inference using the No-U-Turn
sampler, a variant of Hamiltonian Monte Carlo.
- ECD Wiki
Email Russell to get a password to contribute to the discussion.
- Book page on the Wiki
We are slowly working at getting sample networks, errata and other
resources for working through the book up at this site.
- ACED Page on ECD Wiki
Complete data from ACED field trial and ACED Bayes net are
available at this site. This is a Wiki using the same user
name and password as the ECD wiki.
- Heckerman tutorial on learning (Heckerman, D. )
Note: Other Microsoft Research technical reports are available
- Association for Uncertainty in Artificial Intelligence home page
UAI conference proceedings is the most important publication in this
- CRESST Technical Report Archive
Early versions of many of the Mislevy references (including in press
references) are available here. (Hint: search for
``Mislevy''). The CRESST web site changes frequently, so
this link may be out of date. If the link is broken, google
- CiteSeer Cross-Reference Database
On-line cross reference database with lots of articles on Bayes nets.
Many of the bibliography entries are available through CiteSeer.
All handouts and slides from this tutorial are unpublished work of
their respective authors. Sessions I and II are Copyright 2002–18 by
Educational Testing Service. Session III (RNetica and Learning CPTs)
is Copyright 2017–18 by Russell G. Almond with some material copyright
2002–15 by ETS (Used by permission). Session IV (MCMC and DBN)
is copyright 2017–18 by Roy Levy.
These materials are an unpublished, proprietary work of their
respective authors. Any
limited distribution shall not constitute publication. This work may not be
reproduced or distributed to third parties without the author's prior
written consent. Submit request for the ETS material through
http://www.ets.org/legal/copyright.html. Requestions for material
from Russell Almond
and Roy Levy can be obtained
from the respective authors.
ACED development and data collection was
sponsored by National Science Foundation Grant No. 0313202. Thanks
to Val Shute for permission to use ACED data in this tutorial.
Work on RNetica, CPTtools and Peanut has been sponsored in part by
the Physics Playground and E-Rebuild projects, on which
many of the tools were tested. Physics Playground development
was supported by the Bill & Melinda Gates Foundation grant
Games as Learning/Assessment: Stealth Assessment (#0PP1035331,
Val Shute, PI) and the National Science foundation grant DIP:
Game-based Assessment and Support of STEM-related Competencies
(#1628937, Val Shute, PI). The project Mathematical Learning via
Architectual Design and Modeling Using E-Rebuild. (#1720533,
Fengfeng Ke, PI) has provided additional support. The tool
development parts of each of these projects was led by Russell Almond .
almond.g.almond (at) gmail.com
ralmond (at) fsu.edu
Last modified: Thursday, June 3, 2021.