Examples of "kullback_leibler_divergence"
is the KullbackLeibler divergence in nats. When the sample space formula_9 is a finite set, the KullbackLeibler divergence is given by
where formula_47 is the Kullback-Leibler divergence.
Consider the KullbackLeibler divergence between the two distributions
which is the Kullback-Leibler divergence or relative entropy
Two simple divergence functions studied by Lee and Seung are the squared error (or Frobenius norm) and an extension of the KullbackLeibler divergence to positive matrices (the original KullbackLeibler divergence is defined on probability distributions).
where formula_2 is the KullbackLeibler divergence from "q" to "p". Viewing the KullbackLeibler divergence as a measure of distance, the I-projection formula_3 is the "closest" distribution to "q" of all the distributions in "P".
For the classical KullbackLeibler divergence, it can be shown that
The relative entropy, or KullbackLeibler divergence, is always non-negative. A few numerical examples follow:
The KullbackLeibler divergence is named after Kullback and Richard Leibler.
The total variation distance is related to the KullbackLeibler divergence by Pinsker's inequality.
This fundamental inequality states that the KullbackLeibler divergence is non-negative.
is the Kullback-Leibler divergence and it is used that formula_84.
The KullbackLeibler divergence from formula_50 to formula_51, for non-singular matrices Σ and Σ, is:
The canonical divergence is given by the Kullback-Leibler divergence formula_328
with equality when "g"("x") = "f"("x") following from the properties of KullbackLeibler divergence.
the KullbackLeibler divergence from "Q" to "P" is defined to be
The directed KullbackLeibler divergence of formula_13 ('approximating' distribution) from formula_14 ('true' distribution) is given by
which is proportional to KullbackLeibler divergence (which is always non-negative), where
the KullbackLeibler divergence of the prior from the posterior distribution.
where formula_8 is the KullbackLeibler divergence between formula_1 and formula_4.