Received: from mail213-ch1-R.bigfish.com (216.32.181.173) by
FSU-EXCH-NWR01.fsu.edu (10.146.58.165) with Microsoft SMTP Server (TLS) id
14.3.158.1; Tue, 1 Jul 2014 06:42:29 -0400
Received: from mail213-ch1 (localhost [127.0.0.1]) by
mail213-ch1-R.bigfish.com (Postfix) with ESMTP id 5CB663A023C for
; Tue, 1 Jul 2014 10:43:06 +0000 (UTC)
X-Forefront-Antispam-Report: CIP:213.136.76.235;KIP:(null);UIP:(null);IPV:NLI;H:easychair.org;RD:easychair.org;EFVD:NLI
X-SpamScore: 3
X-BigFish: vps3(z59c9hzdb82hc89bh1453I1519M13ddId79ehe003rzz1f42h1ee6h1de0h1fdah2073h2146h1202h1e76h2189h1d1ah1d2ah21bch1fc6h208chzd9hz177df4h17326ah8275dh1de097h186068hz2dh2a8h2b9h839h93fhe5bhfa3h1288h12a5h12a9h12bdh1354h137ah139eh13b6h13eah1441h1504h1537h162dh1631h1758h1898h18e1h1946h19b5h1b0ah2222h224fh1d0ch1d2eh1d3fh1dc1h1dfeh1dffh1e1dh1e23h1fe8h1ff5h2218h2216h226dh22d0h24afh2327h2336h2438h2461h2487h24d7h2516h2545h255eh25f6h2605h268bh26d3h27e2h282bh287eh28a1h28b0h29d0h2a24h)
X-FFO-Routing-Override: fsu.edu%its-exchange2010-633725.customer.frontbridge.com;
Received: from mail213-ch1 (localhost.localdomain [127.0.0.1]) by mail213-ch1
(MessageSwitch) id 1404211383405907_3412; Tue, 1 Jul 2014 10:43:03 +0000
(UTC)
Received: from CH1EHSMHS001.bigfish.com (snatpool2.int.messaging.microsoft.com
[10.43.68.231]) by mail213-ch1.bigfish.com (Postfix) with ESMTP id
5F597180113 for ; Tue, 1 Jul 2014 10:43:03 +0000 (UTC)
Received: from easychair.org (213.136.76.235) by CH1EHSMHS001.bigfish.com
(10.43.70.1) with Microsoft SMTP Server (TLS) id 14.16.227.3; Tue, 1 Jul 2014
10:43:02 +0000
Received: from localhost.localdomain (m2635.contabo.net [213.136.76.235]) by
easychair.org (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id s61Ah0kS014134
for ; Tue, 1 Jul 2014 12:43:00 +0200
Message-ID: <201407011043.s61Ah0kS014134@easychair.org>
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
X-Mailer: MIME::Lite 3.030 (F2.84; A2.12; B3.14; Q3.13)
Date: Tue, 1 Jul 2014 12:43:00 +0200
From: BMAW2014
To: Russell Almond
Subject: BMAW2014 notification for paper 13
Return-Path: bmaw2014@easychair.org
X-MS-Exchange-Organization-AuthSource: FSU-EXCH-NWR01.fsu.edu
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-PRD: easychair.org
X-MS-Exchange-Organization-SenderIdResult: None
Received-SPF: None (FSU-EXCH-NWR01.fsu.edu: bmaw2014@easychair.org does not
designate permitted sender hosts)
X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0
MIME-Version: 1.0
Dear Russell Almond,
We are pleased to inform you that your submission, A Comparison of Two MCMC=
Algorithms for Hierarchical Mixture Models, has been accepted to the Eleve=
nth Bayesian=20
UAI Bayesian Modeling Applications Workshop. Please find below the reviewe=
rs' reports. We will send out a=20
workshop schedule later this week.=20
We received 14 submissions to the workshop. Each paper has been reviewed by
three independent20
reviewers. Eight papers were accepted; five additional papers were accepte
d conditional on addressing20
reviewer concerns; and one paper was rejected.
Please carefully revise your paper according to reviewers' comments for you
r final camera-ready version. In20
particular, please keep in mind both when revising your paper and preparin
g your presentation, that the20
focus of the workshop is on applications. Submissions were subject to a pa
ge limit of 9 pages with one20
additional page for references. It is acceptable to exceed the page limit b
y a few pages if you need extra20
space to respond to reviewer comments.
*** Camera ready versions of the papers are due by 20 July 2014 ***
Please upload your final paper to EasyChair by submitting a revised version
to your original paper. Please20
also prepare the copyright form found at20
http://seor.gmu.edu/~klaskey/BMAW2014/BMAW2014_CopyrightForm_CEUR.txt and s
ubmit as an20
attachment to your paper. We must have a signed copyright form on file in o
rder to publish your paper in20
the proceedings.
The proceedings will be available online during the workshop. Printed abstr
acts will also be provided.
Please note that at least one author of each accepted paper must register f
or the workshop by 20 July for20
the paper to be included in the electronic proceedings.
*** The earlybird registration deadline is July 2. The registration fee in
creases after this date ***
Registration is through the UAI conference website http://auai.org/uai2014/
registration.shtml. Note:20
participants can register for the20
workshop day without registering for the main UAI conference. 20
We look forward to seeing you in Quebec.
Kathy, Jim and Russell
BMAW 2014 co-chairs
----------------------- REVIEW 1 ---------------------
PAPER: 13
TITLE: A Comparison of Two MCMC Algorithms for Hierarchical Mixture Models
AUTHORS: Russell Almond
OVERALL EVALUATION: 0 (borderline paper)
----------- REVIEW -----------
1. Relevance: C2A0The paper positions itself as a tutorial on MCMC for hi
erarchical Bayesian mixture models. This is not the typical paper for whic
h the primary focus is an applied problem. But I think such a tutorial woul
d be of broad interest to the community. It would be more interesting if t
he paper carried through to draw conclusions about the application E280
94 in this case, pause times of students writing essays. There is a brief
description of the problem given at the beginning of the paper, and every n
ow and then the paper mentions that the problem being modeled is pause time
s, but there is actually no data analysis presented E28094 only the mech
anics of running MCMC software. I would have found it much more helpful if
the various modeling decisions being discussed E28094 e.g., prior distr
ibution on number of mixture components, specific decisions on component id
entification, number and length of chains, burn-in time E28094 were dis
cussed more clearly in the context of the ex!
ample, with reasons given for the choices made. I would have liked to see
some actual results. We had some tantalizing hints at what can be learned a
bout cognitive processes used in writing. But there was no data analysis pr
esented whatsoever. If I am trying to learn how to use this kind of model
on an applied problem, I need to actually see the procedures applied to a c
ase study. This many chains were run. Here is the trace plot. It looks li
ke white noise, which suggests no serious pathologies for convergence. Here
are the effective sample sizes. There is some serial correlation, but it
E28099s not too bad, and the sample size is precise enough to give accep
table variances on the MCMC estimates. here is a table of the estimated mea
ns and variances for the mixture components. Here is what the results mea
n for cognitive processes. A paper like that would be extremely helpful.
Also, a good tutorial would allow me to reproduce what was done in the case
study and check again!
st the results presented. I canE28099t do that here. I know code is pr
omised, but I want to see actual plots and tables I can reproduce to make s
ure I have it right.
2. Originality: MCMC on hierarchical mixture distributions is a well-trodd
en area in statistics, if rather unfamiliar to the workshop audience. I ca
nE28099t assess the degree of novelty of the modeling choices made in th
e case study because very little was actually said about what was done in t
he application. Lots of options were given for what one might do at each s
tep, but the actual decisions taken were not described. There are very ma
ny tutorials on MCMC, but IE28099m not aware of another one that walks m
e through an example of hierarchical mixture models. As there are many comp
lexities in MCMC for hierarchical mixture models, I think this would be suf
ficient novelty if the choices were actually described and justified.
3. Significance: A clear, engaging, accessible and well-written tutorial o
n MCMC for hierarchical mixture models would be a significant contribution
to the UAI applications community. This community pioneered graphical proba
bility models, but is not terribly familiar with BUGS, JAGS and other model
ing tools that have become pervasive among statisticians. I would like to
see more of that kind of modeling in this community. Unfortunately, this p
aper is not clear, engaging, accessible and well-written.
4. Presentation: The paper has a sloppy, last-minute feel to it. There are
many grammatical errors. The organization is choppy. It is 3 pages longer
than the page limit, calling to mind the famous Blaise Pascal quote: E2
809CI have made this letter longer than usual, because I lack the time to
make it short.E2809D Some time spent cleaning up, organizing for readab
ility and understandability, and shortening is sorely needed. It canE280
99t make up its mind whether it is a tutorial for people who donE28099t
have much experience with BUGS, JAGS and STAN, or whether it is a list of
tricks written for people who live and breathe this stuff. It desperately
needs some careful editing.
The term "effective sample size" is used for two different concepts: (1) th
e number of IID samples it would take to get the same variance as the MCMC
estimator, and (2) the sum of the parameters of a Dirichlet distribution.
This is confusing. In my experience, "concentration parameter" and "virtua
l sample size" are both used more frequently than "effective sample size" w
hen talking about the sum of the Dirichlet parameters. I recommend using o
ne of these terms instead of effective sample size.20
You never say what a conjugate distribution is -- it might be helpful to re
aders if you gave a brief explanation.
5. Soundness: The paper is technically sound.
p. 6, first sentence in 3.3: with --> which. But this is an odd way of des
cribing HMC. The phrase "uses a different proposal distribution" is rather
puzzling because MH allows you to use ANY proposal distribution -- so it's
different from what? I would say HMC is a variant of MH that uses an a phy
sical analogy to obtain a more efficient proposal distribution. 20
p. 6: "provides BUGS-like support for HMC" is very odd phrasing. What's BUG
S-like support? I would say something like: "Like BUGS and JAGS, STAN pro
vides a higher-level language for specifying HMC models that are compiled a
nd executed by a MCMC engine."
3D3D3D3D3D3D3D3D3D
p. 2: removed from likelihood for --> removed from the likelihood for
p. 3: The a measure --> A measure
p. 3: Figure 1 show --> Figure 1 shows
p. 3: to close to zero --> too close to zero
p. 3: to make MCMC estimate more efficient --> to make MCMC estimation mor
e efficient
p. 4: Two different places where these seem to occur. [This is not a sente
nce]
p. 5: These two packages take a different approach --> Each of these packag
es takes a different approach
p. 5: Neither mix tools nor flex mix provide --> Neither mix tools nor flex
mix provides
p. 6: falls down hills and looses speed --> falls down hills and loses spe
ed
p. 7, 3rd paragraph from the top: The semicolon after Metropolis should be
a colon.
p. 8: A MCMC analysis always follows a number of steps --> A MCMC analysis
always follows a characteristic sequence of steps
p. 8, item 2: In the case of mixture models --> In the case of hierarchical
mixture models
p. 8, first sentence of Section 4.1 is ungrammatical.
p. 10, 3rd full paragraph: these parameter --> these parameters
----------------------- REVIEW 2 ---------------------
PAPER: 13
TITLE: A Comparison of Two MCMC Algorithms for Hierarchical Mixture Models
AUTHORS: Russell Almond
OVERALL EVALUATION: 0 (borderline paper)
----------- REVIEW -----------
This paper describes experiences with
MCMC when working with a hierarchical mixture model. The paper has a
tutorial style, discussing mixture models, hierarchical mixture
models, and estimation in those using MCMC and EM, and may be of great
interest for practitioners. It is not a paper where the application
is central, and I therefore think it is an atypical paper for this
workshop. There is an application motivating the approach (pause times
in student essays), but the modelling of this domain is not given
much attention. I would have liked to see more on that, to be honest,
as the model is central for the presentation later on.
Moving on to the paper at hand, it is fairly well written, and most of
the time clearly presenting the points that are addressed. Some
restructuring is still required, though, making sure that all concepts
are defined before they are used, etc.20
The paper gives some insights into mixture models (e.g., the
non-identifiability of the labels), that have been well studied
already, and also some more detailed tips and tricks like the
augmented parameterization (there is, by the way, a problem with the
parenthesis in Eq 9). The problems that are solved using these tips
and tricks (e.g., slow mixing) are of a general nat ure within MCMC,
but the solutions are obviously tailor-made to the model at
hand. This model is structurally as one would expect, but the choices
of distributional families are maybe a bit unorthodox, which therefore
limits the value of the tailor-made solutions presented. For example,
one would typically consider a Wishart-type distribution for the
variance/precision and not a (truncated) log normal (because the
Wishart is the conjugate prior). In this way, the full model would be
inside the exponential family, giving the opportunity to do, e.g.,
variational inference very efficiently. If sp eed of inferen! ce is
an issue, I believe the Variational Bayes approach would have been
very competitive. Furthermore, one may want to consider some of the
clever "new" models, like the Dirichlet processes, to be able to
handle that the number of mixture components is unknown.20
In conclusion, I found the paper to be a readable "behind the scenes"
description of how inference in a specific model can be performed
using MCMC. The application focus is, however, a bit too limited. A
number of interesting findings/solutions strategies are presented,
and they may be welcomed by practitioners. However, some of the ideas
may not be generalizable (as they are tightly connected to the model
choices being made).
----------------------- REVIEW 3 ---------------------
PAPER: 13
TITLE: A Comparison of Two MCMC Algorithms for Hierarchical Mixture Models
AUTHORS: Russell Almond
OVERALL EVALUATION: 2 (accept)
----------- REVIEW -----------
This paper explains the methods and techniques discovered while building an
d running a hierarchical Bayes mixture model of student-generated events during testing. It takes the form of a tutorial of the lessons learned in the
course of the exercise.20
Relevance: While this paper is unconventional, I believe it is
interesting to this community, since the techniques discussed are
generally applicable, not widely understood, and involve as much art
as they do theory. I think the workshop would be interested in a
detailed look at what is necessary to implement such a model.20
Originality & Significance: The paper's originality is in the
lesser-known details of applying MCMC software methods. To cover the
material in the paper may take more than the allotted 20 minutes
typical of a workshop paper, especially if it generates extensive
discussion. From the presenter's side, it will require extensive
organization of a presentation to avoid getting caught in the
rambling sense of the paper. As for the paper, the author may want to
focus the paper, edit down some of the editorial content and add a
presentation of results, if only on the simulated data mentioned in
the Conclusions.20
Presentation: The paper tends to ramble, not being truly a comparison
of two methods as much as a set of findings about how to get such
models to work. A step by step presentation of the method and the
findings along the way would be appropriate. 20
Soundness: Most interesting to me is the discussion at the end about
model selection with different criteria, since this is "bottom line"
of the exercise. The source of the different approximations and their
different significance would enforce the technical soundness of the
exercise, especially if applied to a worked example.20
Specific comments:
I. Introduction I understand that mixtures of lognormals is a standard
approach to data with long tails, but an alternative is to include
heavy tailed, e.g. Pareto, distributions in the non-transformed space,
so that both sub and super-exponential tailed components can be
extracted.20
2. Mixture Models
"A hierarchical mixture model is by adding.." 3D> "A hierarchical mixture
model is created by adding.."
2.1 "observations", "components": Observations of what? Components of what? The author has an unfortunate habit of introducing terms and notation before explaining what they refer to. It would be helpful to refer to "students" and "pauses" rather than Level 2 and Level 1, since I could never remember if Level 1 was the upper level or vice-versa.20
"get to close to" 3D> "get too close to"
2.3 The reparameterization discussion is interesting. Why does reparameterization not change the model's specification? Is it essentially a variable
transformation of the model variables, or just a change in the MC process?
This is mentioned again in 3.2 as "A simple trick..." Is this related to the choice of step size, on page 6.20
2.4 This discussion is presented as a hypothetical "..would be ..." A summary of how typical and how often such problems arise would be better. I sense that careful consideration of sensible priors probably avoids most of these pitfalls.20
3.1 Some of this section repeats issues already discussed.20
"the components are reorder according" 3D> "the components are reordered a
ccording"
3.2, p6 "conjugate proposal distribution" This is a non-standard term. Please explain.20
"Although BUGS.." Descriptions of the current software are interspersed in the text and would merit their own section.20
20
p.9 Figure 3 needs to be reformatted with indents using a LaTeX algorithm package.20
20
"if they arrive the same" 3D> "if they arrive at the same"20
p.10 "... for all these parameter and exhausting.." 3D> "... for all these parameters and exhausting.."20
5. Model Selection
Since the DIC and WAIS discussions are a significant part of the paper, they should be introduced early on. They both merit better explanations. For
instance, where do you define $D(\overbar\Omega)$? And what is "the complete parameter $\Omega$"? The set of all parameters?20
6. Conclusions
"The procedure described above is realized a code.." 3D> "The procedure de
scribed above is realized as code.."
<>