Let’s say we want to approximate a function $f$. In an area around $a$ the value $f(a)$ is probably a decent approximation for $f(x)$.

Using the Fundamental Theorem of Calculus we can calculate the error

$$ f(x) - f(a) = \int_a^x f'(y)dy $$

If $f'$ is bounded by $\|f'\|_\infty$ then our error is smaller than $\|f'\|_\infty (x-a)$. We can now improve this bound by approximating $f'$ itself in the same way:

$$ \begin{aligned} f(x) - f(a) &= \int_a^x \left( f'(a) + \int_a^y f''(s)ds \right) dy\\ &= f'(a) (x-a) + \int_a^x \int_a^y f''(s)ds dy\ \end{aligned} $$

Which results in a more precise approximation of $f$:

$$ f(x) \approx f(a) + f'(a)(x-a) $$

If $f''$ is bounded then our error is now only $\|f''\|_\infty(x-a)^2$ a significant improvement over the previous error close to $a$.

Iterating on this idea yields the taylor approximation. If all the derivatives are bounded by the same constant, the error converges towards zero, the function is a taylor series

$$ \begin{aligned} f(x) &= f(a) + f'(a)(x-a) + f''(a)\frac{(x-a)^2}{2} + \dots\\ &= \sum_{k=0}^\infty f^{(k)}(a)\frac{(x-a)^k}{k!} \end{aligned} $$

Application: Exponential Function

If the change of something is equal to the current size (e.g. number of bacteria as it increases via cell division) then that implies $f=f'$. If we also assume $f(0)=1$ for normalization then we immediately obtain the series representation of the unique function with this property: the exponential function

$$ \exp(x) = \sum_{k=0}^\infty \exp(0) \frac{(x-0)^k}{k!} = \sum_{k=0}^\infty \frac{x^k}{k!} $$

Taylor Approximation in Multiple Dimensions

To obtain the Taylor’s Theorem for functions $f:\mathbb{R}^d\to\mathbb{R}$ from multiple dimensions one can simply reduces them to the one dimensional version of the taylor theorem with the following trick. For $a,x\in\mathbb{R}$ we can define

$$ g :\begin{cases} [0,1] \to \mathbb{R} \\ s\to f(a+s(x-a)) \end{cases} $$

which means we can apply the one dimensional taylor theorem from above to $g$ to get

$$ \begin{aligned} f(x) &= g(1) = g(0) + g'(0)(1-0) + g''(0)\frac{(1-0)^2}{2} + \dots\\ &=f(a) + \nabla f(a)^T(x-a) + \tfrac12 (x-a)^T\nabla^2f(a)(x-a) + \dots \end{aligned} $$