更精确地估算numeric gradient

Last Updated: 2024-01-04 10:48:15 Thursday

-- TOC --

所谓numeric gradient，就是用具体的值去计算函数某个点的导数，采用centered difference formula：

已知函数\(f\)，取\(h\)为一个固定的很小的数，例如\(h=0.0001\)，则gradient可表示为：

\(g \approx \cfrac{f(x+h)-f(x-h)}{2h}\)

wiki上对这个计算公式，是这样介绍的：

This formula is known as the symmetric difference quotient. In this case the first-order errors cancel, so the slope of these secant lines differ from the slope of the tangent line by an amount that is approximately proportional to \(h^2\). Hence for small values of h this is a more accurate approximation to the tangent line than the one-sided estimation. However, although the slope is being computed at x, the value of the function at x is not involved.

相对于numeric gradient，还有个概念叫analytic gradient，后者就是采用推导出的导数公式，来精确计算导数值。而前者，一般用来检查后者的计算代码是否正确，这叫做gradient check。

本文链接：https://cs.pynote.net/math/202210253/

-- EOF --

-- MORE --