Mathematical Engineering - Bayesian Statistics

Full exam

BAYESIAN STATISTICS A. Guglielmi & M. Gianella 25.06.2024 Properly justify all your answers. Use the indicator function to denote the support of a distr.. Exercise 1We consider the conditional density fX( x|β) =βe− β x 1(0,∞)( x),forβ >0,(1) and let (X 1, . . . , X n) be a random sample from (1). Denote by ( x 1, . . . , x n) the corresponding observed sample. 1.Computeθ=P(X i≤ 1|β) for anyi= 1, . . . , n. Instead of observing the whole sample (x 1, . . . , x n), we only know that observations are less than or equal to 1 (≤1), or larger than 1. 2.Introduce r.v.s (Y 1, . . . , Y n) as a transformation of ( X 1, . . . , X n), representing the only information that we have. 3.Derive the likelihood, i.e. the conditional density of (Y 1, . . . , Y n), given β, under the sole information that (exactly)kobservations in (x 1, . . . , x n) are less than or equal to 1, where kis an integer number between 0 andn. 4.Use reparameterizationθ=g(β), derived at point 1., and find a conjugate priorπ θfor the new parameter θ, under the conditional distribution of (Y 1, . . . , Y n), given θ. Transform backπ θ, and assume this prior πβas the prior for the likelihood derived at point 3. 5.Write the expression of the posterior distribution ofβup to a constant, under the sole information that kobservations in (x 1, . . . , x n) are less than or equal to 1. How would you simulate from this posterior? 6.Set the hyperparameters so that, a priori, the mean and the variance ofθ, introduced at point 4., are 1/2 and 1/12, respectively. Then computeE πβ( β). Solution of Exercise 11.It is straightforward to computeθ=P(X i≤ 1|β) =R 1 0f X( x;β)dx= 1−e− β×1 = 1−e− β . 2.We defineY i= 1 if X i≤ 1 andY i= 0 if X i> 1. This means that, conditionally toβ, (Y 1, . . . , Y n) is a Bernoulli sample with success probability given byθ:=P(X i≤ 1|β). 3.The sole information that we have is that, conditionally toβ, (Y 1, . . . , Y n) is a Bernoulli sample with success probability 1−e− β , and thatP n 1y i= k. Hence the likelihood is l(β;y 1, . . . , y n) =n Y i=1θ y i (1−θ)1 −y i = (1−e− β )k (e− β )n −k = e− (n−k)β (1−e− β )k , β >0. 4.The natural parameterization here isθ= 1−e− β , so thatY i| θiid ∼Be(θ). In this case, the conjugate prior forθis thebeta(a, b) distribution,aandbpositive, with density πθ( θ) =1B (a, b)θ a −1 (1−θ)b −1 1(0,1)( θ). The prior we assume forβis πβ( β) =π θ(1 −e− β )∂∂ β θ (β) =1B (a, b)(1 −e− β )a −1 (e− β )b −1 e− β 1(0,+∞)( β) =1B (a, b)e − bβ (1−e− β )a −1 1(0,+∞)( β). Note thatθ∈(0,1) if and only ifβ >0. 1 5.We have L(β|Y 1, . . . , Y n) ∝e− (n−k)β (1−e− β )k ×e− bβ (1−e− β )a −1 1(0,+∞)( β) ∝e− (n−k+b)β (1−e− β )k +a−1 1(0,+∞)( β). This is the kernel of the density ofβ=−log(1−θ), whereθ∼beta(a+k, b+n−k); see the computations done at point 4. To simulate from this posterior, we need to sample firstθfrom the beta distribution using standard methods, and then computeβ=−log(1−θ). 6.Sinceθ∼beta(a, b) a priori,E π( θ) =a/(a+b) = 1/2, Var(θ) =ab( a+b)2 (a+b+ 1)= 112 , so that a=b= 1, i.e.θ∼ U(0,1). Hence, we need to computeE πβ( β) as follows: Eπβ( β) =E π( −log(1−θ)) =−Z 1 0log(1 −θ)dθ=−Z 1 0log θdθ=−[θlogθ−θ]1 0= 1 .Exercise 2 1.Write the posterior predictive density for observationsX 1, X 2, . . . , X nsuch that Xi| θiid ∼f(·;θ), θ∈Θ, wheref(·;θ) is a density onRfor allθ. Assume thatθ∼πa priori. 2.Outline themethod of composition(also known as theaugmentation trick) in Monte Carlo Simulation. Describe in full details how themethod of compositioncan be applied to simulate a sample from the posterior predictive density derived at point 1. 3.Apply the method derived at point 2. whenf(·;θ) is the gamma density with parameterθ= (α, β)∈ (0,+∞)2 = Θ andθ∼π α× π β, with π αgiven by the distribution U(0.1,10) andπ βby the gamma(1,1) distribution. Solution of Exercise 31.IfX 1, X 2, X 3, . . . |θiid ∼f(·;θ), then the posterior predictive density is mXn+1| X 1,...,X n( x;x 1, . . . , x n) =Z Θf (x;θ)π(dθ|x 1, . . . , x n) 2.Suppose that we want to sample from the marginal distribution of a random variableY 1, but we are not able to sample directly from it. Instead we are able to sample from the conditional law of another random variableY 2, given Y 1, and from the marginal law of Y 2. Hence the marginal law L(Y 1) can be obtained from the joint lawL(Y 1, Y 2) = L(Y 2| Y 1) × L(Y 2). The method of composition works as follows: (i) simulate a drawY 2= y 2from Y 2∼ L (Y 2) (ii) simulate a drawY 1= y 1from the conditional distribution Y 1∼ L (Y 1| Y 2= y 2) ⇒Y=y 1is a draw from the marginal law L(Y 1). If{θ( j) , j= 1, . . . , J}is a Monte Carlo (or MCMC) sample from the posteriorπ(·|x 1, . . . , x n), sampling independently at each step below, the method of composition gives: -x(1) ∼f(·;θ(1) ) -x(2) ∼f(·;θ(2) ) . . . -x( J) ∼f(·;θ( J) ) ⇒(x(1) , x(2) , . . . , x( J) ) is a Monte Carlo (or a MCMC) sample fromm Xn+1| X 1,...,X n( ·;x 1, . . . , x n). 2 3.Because in this case the prior for ( α, β) is not conjugate, we need to have available first a MCMC sample (α( j) , β( j) )J j=1from the posterior. This can be computed via a Metropolis-within-Gibbs sampler, since the full conditional ofβis still a gamma density, while the full conditional ofαhas not a closed form. Then, for anyj, we can samplex( j) ∼f(·;α( j) , β( j) ), that is a gamma distribution (and we know how we can sample from the gamma density). In the end, (x(1) , . . . , x( J) ) is a MCMC sample from the posterior predictive distribution of this model.Exercise 3 Only for the 10CFU-course students Describe in detail the Stick Breaking Construction of the Dirichlet process onRk (k >1). Solution of Exercise 2See the Lecture Notes, Section 8.1.1. 3