Tutorial: Bayesian Networks in Educational Assessment

Russell Almond, Florida State University
Roy Levy, Arizona State University [Not available in 2021]
Duanli Yan, ETS
Diego Zapta, ETS

This is a collection of material related to our 2021 NCME Tutorial. See you online!

Instructions for Attendees. There is now a "live computing" exercise included in the seminar. To do this we are recommending everybody who can bring a laptop.

We are also recommending you do thefollowing steps:

  1. Download student/demonstration version of the software Netica from Norsys. (Other possible software packages are listed below, but we will be preparing the exercises in Netica.) You can try this out using the student/demonstration version, which is sufficient for the exercises in the workshop.
  2. Download and install the appropriate version of R from CRAN. You do not need the absolute latest version, if you already have R version 4.X installed, you should be fine.
  3. Download and install the the free community edition of R Studio from RStudio.com.
  4. Install the package rjags from the CRAN library. You can do this by issuing the command install.packages("rjags") in R after starting it. This also requires you to load install the program JAGS from http://mcmc-jags.sourceforge.net/.
  5. Launch R Studio, and install the following packages: arm,R.utils,shinyjs,rmarkdown,futile.logger,devtools
  6. Download the code for CPTtools, RNetica, Peanut and PNetica packages. These packages are not yet on CRAN, but can be found on the RNetica homepage. The following table has the latest versions. [Note that compiling RNetica from source (required for Unix versions) requires downloading the Netica C API from Norsys, see the INSTALL file in the tarball or the RNetica homepage for details.]
    PackageSource (Unix)WindowsMacOSManual
    CPTtoolsCPTtools_0.7-2.tar.gz CPTtools_0.7-2.zip CPTtools_0.7-2.tgz CPTtools-manual_0.7-2.pdf
    RNeticaRNetica_0.8-4.tar.gz RNetica_0.8-4.zip RNetica_0.8-4.tgzRNetica-manual_0.8-4.pdf
    PeanutPeanut_0.8-4.tar.gz Peanut_0.8-4.zip Peanut_0.8-4.tgz Peanut-manual_0.8-4.pdf
    PNeticaPNetica_0.8-5.tar.gz PNetica_0.8-5.zip PNetica_0.8-5.tgz PNetica-manual_0.8-5.pdf
  7. Download the example networks to be used (See Under Each session).

Mac and Linux usesrs Netica should run without problems in a variety of Windows emulators. In particular, it should run under WINE. I (Russell) have had success using WINE under both Mac OS X (version 10.6.8 up) and Ubuntu Linux (version 12.04 up). There are several options:


This tutorial follows the book Bayesian Networks in Educational Assessment (Almond, Mislevy, Steinberg, Yan and Williamson, 2015). The first part (Sessions I and II) contain an overview of Bayesian networks (Part I of the book) giving some examples of how they can be used. The second part (Sessions III and IV) look at software and techniques for building networks from expert opinion and data.

Bayesian networks are a technique for managing multidimensional models. By representing the variables of the model as nodes in the graph and using edges in the graph to represent patterns of dependence and independence among the variables, the model graph serves as a bridge between educational and psychometric experts, and further helps the computer derive efficient computational strategies.

This tutorial is based on the book Bayesian Networks in Educational Assessment now out from Springer.

Slides and Handouts

I. Evidence Centered Design and Bayesian Networks
Covers basic models of ECD and their application to Bayes nets. Slides (pptx), Slides (pdf), Session I networks (Netica).
II. Bayes Net Applations including ACED
This part looks at a number of simple applications of Bayes nets to provide more intution about how they work. Slides (pptx), Slides (pdf), Simple Example Networks (Netica), ACED Subset (Netica),
III. RNetica and CPTtools
This looks at Tools for using and building Bayesian networks in R, particularly, the CPTtools and RNetica packages. It includes examples in scoring and using the built-in EM algorithm to fit models to data. The talk is split into two sets of slides. [I'm still making some tweaks here. Expect an update around noon.] RNetica Introduction (PDF) DiBello Models (PDF). RNetica and DiBello talks (R markdown) and sample nets., Learning CPTs Slides (PDF) Learning CPTs Handout (PDF). A simple Learning Example.
IV. Advanced Topics
Learning with Markov chain Monte Carlo (MCMC) and dynamic Bayesian networks (networks which unfold across time). [Actually, as Roy is unable to make it this year, we won't cover Dynamic Bayes nets this year.] MCMC Slides (pptx) MCMC Slides (pdf) All Session IV networks (Netica)
Bayes net and ECD Bibliography (Note: this is an out of date version of the book bibliography).

The handout version is also available as one big file containing all sessions and the bibliography. Honkin' big handout (PDF) PDF/X version for printing. [We didn't have a chance to redo the handout this year, this is still last years version.]

On-line Resources

For quick reference, here are the on-line resources referenced in the bibliography.

Computer programs and documentation available on the Web:

This is a partial list of software packages we have used or think are worth paying attention to. The list of Bayes net software found at the bottom of the Bayesian network Wikipedia entry http://en.wikipedia.org/wiki/Bayesian_network is a reasonably complete and up to date list of both free and commercial software.

Netica (Norsys Software Crop)
http://www.norsys.com/ Netica is another very complete commercial grade Bayes net engine, includes some learning tools.
RNetica (Netica API for R)
http://pluto.coe.fsu.edu/RNetica This is a work in progress binding for the Netica API into the R language. Currently only source verison is available. (Windows and Mac binaries will be available at the conference).
Genie/Smile (Decision Systems Lab, Univ. of Pittsburgh)
http://genie.sis.pitt.edu/ Open source project, free under Gnu Public License. Also contains a ``translator'' which translates between network formats.

Useful (Bayesian) Statistical Software

BUGS (Bayesian inference Using Gibbs Sampling).
http://www.mrc-bsu.cam.ac.uk/bugs/welcome.shtml Downloadable version for Windows. BUGS is no longer actively maintained. For serious work, I recommend OpenBUGS http://mathstat.helsinki.fi/openbugs/
JAGS (Just Another Gibbs Sampler)
https://sourceforge.net/projects/mcmc-jags/ A rewrite of Classic BUGS (command line only, no GUI support) that runs under Linux, MacOS X, and Windows.
FBM: Flexible Bayesian Modeling
http://www.cs.utoronto.ca/~radford/fbm.software.html Radford Neal's Flexible Bayesian Modeling and Markov Chain Sampler.
http://www.r-project.org/ General purpose statistical computing environment based on S language.
http://mc-stan.org/Stan is a package for obtaining Bayesian inference using the No-U-Turn sampler, a variant of Hamiltonian Monte Carlo.

Other On-Line Resources:

ECD Wiki
http://ecd.ralmond.net/ecdwiki/ Email Russell to get a password to contribute to the discussion.
Book page on the Wiki
http://ecd.ralmond.net/ecdwiki/BN/BN. We are slowly working at getting sample networks, errata and other resources for working through the book up at this site.
ACED Page on ECD Wiki
http://ecd.ralmond.net/ecdwiki/ACED/ACED Complete data from ACED field trial and ACED Bayes net are available at this site. This is a Wiki using the same user name and password as the ECD wiki.
Heckerman tutorial on learning (Heckerman, D. [1995])
ftp://ftp.research.microsoft.com/pub/tr/tr-95-06.pdf Note: Other Microsoft Research technical reports are available on-line from http://www.research.microsoft.com/
Association for Uncertainty in Artificial Intelligence home page
http://www.auai.org/ UAI conference proceedings is the most important publication in this area.
CRESST Technical Report Archive
http://www.cse.ucla.edu/products/reports.asp Early versions of many of the Mislevy references (including in press references) are available here. (Hint: search for ``Mislevy''). The CRESST web site changes frequently, so this link may be out of date. If the link is broken, google "CRESST Reports".
CiteSeer Cross-Reference Database
http://citeseer.ist.psu.edu/cis On-line cross reference database with lots of articles on Bayes nets. Many of the bibliography entries are available through CiteSeer.


All handouts and slides from this tutorial are unpublished work of their respective authors. Sessions I and II are Copyright 2002–18 by Educational Testing Service. Session III (RNetica and Learning CPTs) is Copyright 2017–18 by Russell G. Almond with some material copyright 2002–15 by ETS (Used by permission). Session IV (MCMC and DBN) is copyright 2017–18 by Roy Levy.

These materials are an unpublished, proprietary work of their respective authors. Any limited distribution shall not constitute publication. This work may not be reproduced or distributed to third parties without the author's prior written consent. Submit request for the ETS material through http://www.ets.org/legal/copyright.html. Requestions for material from Russell Almond and Roy Levy can be obtained from the respective authors.

ACED development and data collection was sponsored by National Science Foundation Grant No. 0313202. Thanks to Val Shute for permission to use ACED data in this tutorial.

Work on RNetica, CPTtools and Peanut has been sponsored in part by the Physics Playground and E-Rebuild projects, on which many of the tools were tested. Physics Playground development was supported by the Bill & Melinda Gates Foundation grant Games as Learning/Assessment: Stealth Assessment (#0PP1035331, Val Shute, PI) and the National Science foundation grant DIP: Game-based Assessment and Support of STEM-related Competencies (#1628937, Val Shute, PI). The project Mathematical Learning via Architectual Design and Modeling Using E-Rebuild. (#1720533, Fengfeng Ke, PI) has provided additional support. The tool development parts of each of these projects was led by Russell Almond .

almond.g.almond (at) gmail.com
ralmond (at) fsu.edu
Last modified: Thursday, June 3, 2021.