The Generating Function of Ternary Trees and Continued Fractions

Ira Gessel, Guoce Xin

Published 2005-05-11Version 1

Michael Somos conjectured a relation between Hankel determinants whose entries $\frac 1{2n+1}\binom{3n}n$ count ternary trees and the number of certain plane partitions and alternating sign matrices. Tamm evaluated these determinants by showing that the generating function for these entries has a continued fraction that is a special case of Gauss's continued fraction for a quotient of hypergeometric series. We give a systematic application of the continued fraction method to a number of similar Hankel determinants. We also describe a simple method for transforming determinants using the generating function for their entries. In this way we transform Somos's Hankel determinants to known determinants, and we obtain, up to a power of 3, a Hankel determinant for the number of alternating sign matrices. We obtain a combinatorial proof, in terms of nonintersecting paths, of determinant identities involving the number of ternary trees and more general determinant identities involving the number of $r$-ary trees.