Surya Dantuluri

Policy Gradients [Draft]

Explanation, derivation, and implementation of Policy Gradients.

Surya Dantuluri

Material adapted from Sergey Levine’s CS294-112 Fall 2018 and Josh Achiam’s Spinning Up With RL

Currently expanding unfinished article into upcoming research paper. Stay tuned!

What is a Policy Gradient and how it is useful?

Arguably the goal in Reinforcement learning is to find an optimal policy. By nature this optimal policy results in relatively highest rewards(relative to other non-optimal policies). With policy gradients, we technically incentivize the distribution of actions to generate a higher reward and likewise deter distributions of actions that generate sub-optimal rewards. Overtime we generate better trajectories, creating the optimal policy.

Derivation

The goal of RL is to optimize the objective function.

definecolorredRGB255,59,48definecolororangeRGB255,149,0definecoloryellowRGB255,204,0definecolorgreenRGB76,217,100definecolortealblueRGB90,200,250definecolorblueRGB0,122,255definecolorpurpleRGB88,86,214definecolorpinkRGB255,45,85\\definecolor{red}{RGB}{255,59,48}\\definecolor{orange}{RGB}{255,149,0}\\definecolor{yellow}{RGB}{255,204,0}\\definecolor{green}{RGB}{76,217,100}\\definecolor{tealblue}{RGB}{90,200,250}\\definecolor{blue}{RGB}{0,122,255}\\definecolor{purple}{RGB}{88,86,214}\\definecolor{pink}{RGB}{255,45,85}

\\pi\_{\\theta}^\\star = \\text{arg}\\underset{\\pi\_{\\theta}}{\\max}\\color{orange}E\_{\\tau\\sim p\_{\\pi\_{\\theta}}(\\tau)}\[\\sum\_{t} r(s\_t,a\_t)\]

What we’re doing here is taking all the state action pairs along the trajectory, (s_t,a_t)(s\_t, a\_t), summing their rewards up, finding the total reward, and maximizing them with respect to theta\\theta, the parameters of the policy.

Let’s denote the expectation of the sum \\color{orange}E\_{\\tau\\sim p\_{\\pi\_{\\theta}}(\\tau)}\[\\sum\_{t} r(s\_t,a\_t)\] as colorredJ(pi_theta)\\color{red}J(\\pi\_{\\theta}).

Now that we have colorredJ(pi_theta)\\color{red}J(\\pi\_{\\theta}) what do we do? Find the gradient to optimize this expectation via gradient ascent.

Expanding J_thetaJ\_{\\theta} from expectation form (using definition of expectation):

\\color{red}J(\\pi\_{\\theta}) \\color{black}=\\color{orange} E\_{\\tau\\sim p\_{\\pi\_{\\theta}}(\\tau)}\[\\sum\_{t} r(s\_t,a\_t)\] \\color{black}= \\color{green}\\int \\color{blue}P(\\tau|\\pi\_{\\theta})\\color{green}R(\\tau)d\\tau

Finding the gradient:

nabla_thetacolorredJ(pi_theta)colorblack=nabla_thetacolorgreenintcolorblueP(tau∣pi_theta)colorgreenR(tau)dtau\\nabla\_{\\theta}\\color{red}J(\\pi\_{\\theta}) \\color{black}= \\nabla\_{\\theta}\\color{green}\\int \\color{blue}P(\\tau|\\pi\_{\\theta})\\color{green}R(\\tau)d\\tau

=colorgreenintcolorblacknabla_thetacolorblueP(tau∣pi_theta)colorgreenR(tau)dtau= \\color{green}\\int \\color{black}\\nabla\_{\\theta}\\color{blue}P(\\tau|\\pi\_{\\theta})\\color{green}R(\\tau)d\\tau

Using the log derivative trick:

nabla_thetacolorblueP(tau∣pi_theta)colorblack=colorblueP(tau∣pi_theta)fraccolorblacknabla_thetacolorblueP(tau∣pi_theta)P(tau∣pi_theta)=coloryellowP(tau∣pi_theta)colorblacknabla_thetacoloryellowlogP(tau∣pi_theta)\\nabla\_{\\theta}\\color{blue}P(\\tau|\\pi\_{\\theta}) \\color{black}= \\color{blue}P(\\tau|\\pi\_{\\theta})\\frac{\\color{black}\\nabla\_{\\theta}\\color{blue}P(\\tau|\\pi\_{\\theta})}{P(\\tau|\\pi\_{\\theta})} = \\color{yellow}P(\\tau|\\pi\_{\\theta})\\color{black}\\nabla\_{\\theta}\\color{yellow}\\log P(\\tau|\\pi\_{\\theta})

Continuing from our previous step:

=colorgreenintcoloryellowP(tau∣pi_theta)colorblacknabla_thetacolorblacklogP(tau∣pi_theta)colorgreenR(tau)dtau= \\color{green}\\int \\color{yellow}P(\\tau|\\pi\_{\\theta})\\color{black}\\nabla\_{\\theta}\\color{black}\\log P(\\tau|\\pi\_{\\theta})\\color{green}R(\\tau)d\\tau

Going back to expectation form we get:

= \\underset{\\tau\\sim\\pi\_{\\theta}}{E}\[\\nabla\_{\\theta}\\color{yellow}\\log P(\\tau|\\pi\_{\\theta})\\color{green}R(\\tau)\\color{black}\]

But we still don’t have P(tau∣pi_theta)P(\\tau|\\pi\_{\\theta}) which by the way is the probability of the trajectory

P(tau∣pi_theta)=corr(s_0)prod_t=1TP(s_t+1∣s_t,a_t)pi_theta(a_t∣s_t)P(\\tau|\\pi\_{\\theta}) = corr(s\_{0})\\prod\_{t=1}^{T} P(s\_{t+1}|s\_{t},a\_{t})\\pi\_{\\theta}(a\_{t}|s\_{t})

Taking the log of both sides to help us simplify our previous expectation:

logP(tau∣pi_theta)=logcorr(s_o)+sum_t=1T(logP(s_t+1∣s_t,at)+logpi_theta(a_t∣s_t))\\log P(\\tau|\\pi\_{\\theta}) = \\log corr(s\_{o}) + \\sum\_{t=1}^{T} (\\log P(s\_{t+1}|s\_{t},a{t})+ \\log \\pi\_{\\theta}(a\_t|s\_t))

nabla_thetalogP(tau∣pi_theta)=nabla_thetalogcorr(s_o)+sum_t=1T(logP(s_t+1∣s_t,at)+logpi_theta(a_t∣s_t))\\nabla\_{\\theta}\\log P(\\tau|\\pi\_{\\theta}) = \\nabla\_{\\theta} \\log corr(s\_{o}) + \\sum\_{t=1}^{T} (\\log P(s\_{t+1}|s\_{t},a{t})+ \\log \\pi\_{\\theta}(a\_t|s\_t))

And since  logcorr(s_o)\\log corr(s\_{o}) and logpi_theta(a_t∣s_t))\\log \\pi\_{\\theta}(a\_t|s\_t)) don’t depend on theta\\theta, we can simplify the the gradient of the expectation to:

\\nabla\_{\\theta}J(\\pi\_{\\theta}) =  \\underset{\\tau\\sim\\pi\_{\\theta}}{E}\[\\cancel{\\nabla\_{\\theta}\\log corr(s\_{o})} + \\nabla\_{\\theta}\\color{yellow} \\sum\_{t=1}^{T} (\\log P(s\_{t+1}|s\_{t},a{t})+ \\log \\pi\_{\\theta}(a\_t|s\_t))\\color{green}R(\\tau)\\color{black}\]

Unfinished derivation (finished the implementation which I will soon update here). Currently working on a paper and will come back to finish this post.

All articlesBack to top