AI/Vision/Robotics Colloquium
Peter Grunwald
Stanford University
Tuesday, 28 September 1999
306 Soda Hall
4:00 - 5:30
The Maximum Entropy Principle is an often successful yet very controversial method for reasoning under uncertainty. It can be applied in all domains involving probability distributions that are only partially specified. There are several situations in which Maximum Entropy gives counterintuitive and/or undesired results. One of the most serious of these (known alternatively as Pearl's, Dalkey's or Hunter's `bell problem') can occur if Maximum Entropy is applied to Bayesian networks.
In this talk, we give a radically new view of maximum entropy. Through a game-theoretic analysis, we show that in many common situations, picking the distribution P maximizing the entropy is equivalent to an alternative procedure A, which has at least three different interpretations: (1) it picks the distribution that minimizes one's worst-case loss if one uses it in a sequential gambling game; (2) it can be seen as a form of the Minimum Description Length (MDL) Principle; (3) it can be seen as a form of 'robust Bayesian' inference.
While giving the same result as Maximum Entropy in all `sufficiently regular' cases, our procedure continues to give intuitive results in less regular cases which are problematic for Maximum Entropy. We show that, using our procedure, one can properly solve at least three problems where Maximum Entropy fails: (1) Pearl's bell problem; (2) disjunctive constraints and (3), for continuous sample spaces, Bertrand's paradox.