Bayesian networks remain the cornerstone of modern AI and machine learning. The subclass of Bayesian networks known as “topic models” in particular is generating an increasing amount of interest from our research community. Moreover, “infinite” extensions such as the HDP hold promise to fascilitate the search for model structure and are in particular suitable to learn models that adapt their model complexity in response to a continuous stream of incoming information. Unfortunately, inference in nonparametric Bayesian models can be expensive both in terms of space and time, presenting an important obstacle for their application to large scale machine learning problems.
In this talk I will briefly review a convenient modeling framework called “infinite state Baysian networks” (ISBN) which naturally combines nonparametric Bayesian priors with Bayesian networks. In this context I will describe six ideas for making inference more efficient: a) kd-trees, b) memory bounded inference, c) collapsed variational inference, d) hybrid algorithms combining sampling with variational inference, e) parallel collapsed Gibbs sampling and f) shortcut sampling.
This work was done or is in progress in collaboration with a large crowd of talented collegues and students: Yee Whye Teh, Kenichi Kurihara, Ryan Gomes, Pietro Perona, Ian Porteous, Arthur Ascuncion, Dave Newman, Padhriac Smyth, Nikos Vlassis, Alex Ihler and Bert Kappen.