Geometry of statistical manifolds Sample Clauses

Geometry of statistical manifolds. ‌ Consider the parameterized conditional probability distribution πθ(a|s) wholly specified by θ — purposely notated as such since the following analysis will later be applied to stochastic policies in reinforcement learning. Denote S the space of all distributions of this type (e.g. all neural networks of a given structure across the space of θ) and consider θ to be the local coordinate system of S. More accurately, the space under consideration is the quotient space S˜ = S\R where R is defined by the equivalence relation θ ∼ θ′ when πθ(a|s) = πθ′ (a|s) (3.39) i.e. the two parameters specify identical conditional distributions. However, for ease of notation, the space under consideration will simply be denoted S. The divergence between two points θ, θ′ ∈ S is defined by the Kullback-Leibler (KL) divergence between the two distributions they specify as D (π : π′ ) = ∫ π (a|s) log πθ (a|s) KL θ θ θ πθ′ (a|s) Note that the KL-divergence is not symmetric and thus cannot be considered a metric. Choosing DKL as a notion of distance, however, allows for the specification of the Riemannian structure of S. Consider some loss function L(θ). With respect to DKL as the divergence between two coordinates, the steepest descent (natural gradient) direction upon the manifold is F (θ)−1∇L(θ) where F (θ) = ∇θ′ DKL(πθ : πθ′ )|θ=θ′ (3.41) i.e. the Hessian of the KL-divergence at θ. Specifically, the matrix F (θ) is called the F (θ) = Es∼µ,a∼πθ [∇ log πθ(a|s)∇ log πθ(a|s)⊤] (3.42) pre-emptively using the RL notation s ∼ µ to denote the sampling of s from the stationary distribution µ(s). Under certain regularity conditions (for exchanging integration and differntiation), this form of the Fisher matrix can be shown to be equivalent to the Hessian of DKL as follows: F (θ) = ∇θ′ DKL(πθ : πθ′ )|θ=θ′ πθ′ (a|s) θ=θ′ = ∇2 ∫ π (a|s) log πθ (a|s) ds da = − ∫ πθ(a|s) ∇2 log πθ(a|s) ds da (*) = Es∼µ,a∼π [−∇2 log πθ(a|s)] = E − 1 ∇2π (a|s) + E 1 ∇π (a|s)∇π (a|s)⊤ s∼µ,a∼πθ πθ(a|s) s∼µ,a∼πθ πθ(a|s)2