1.1. Approximating derivatives

Note

Important things to retain from this block:

• Derivatives can be approximated in different ways

• Approximations can be derived from Taylor series expansions

• The accuracy of the approximation depends on the order of the Taylor-series expansion it is based on

Differential equations are based on derivatives (e.g. spatial or temporal) of a variable. To solve them numerically, we need to find approximations of these derivatives. In this section, we will take a closer look at how to do that.

The first derivative

In your calculus classes, you might remember working with the analytical definition of a derivative, such as \(\frac{df}{dx}\). This involved finding the limit as the distance between two points becomes infinitesimally small.

Definition:

The derivative of a function \(f(x)\) evaluated at the point \(x_0\) is

\[ \frac{df}{dx}\bigg\rvert_{x_0}=f'(x_0)=\lim_{x \to x_0}\frac{f(x)-f(x_0)}{x-x_0}. \]

Numerically:

\(x - x_0\) cannot tend to 0! Thus:

\[ f'(x_0) \approx \frac{f(x)-f(x_0)}{\Delta x}, \hspace{3mm} \text{where } \Delta x=x-x_0. \]

Fig. 1.2 derivatives of a function \(f(x)\) at a specific points \(x_0\) and \(x_1\)

What does the derivative represent? A rate of change! In this case, how fast \(f(x)\) changes with respect to \(x\) (or in the direction of \(x\)). The derivative can also be with respect to time or another independent variable (e.g., \(y,z,t,etc\)). Look at the figure above. The first derivative evaluated at \(x_0\) is illustrated as a simple slope, as is the first derivative evaluated at \(x_1\) . You can also see that the rate of change at \(x_0\) is larger than at \(x_1\).

Different ways to approximate derivatives

In the Figure above, the derivative approximation was illustrated arbitrarily using two points: the one at which the derivative was evaluated and another point in front of it. However, there are more possibilities. Instead of using absolute points at \(x_{-1,0,1}\) a more general notation is used: \(x_{i-1,i,i+1}\). The simplest ways to approximate the derivative evaluated point \(x_i\) is to use two points:

\[ \text{forward: }\hspace{3mm} \frac{df}{dx}\bigg\rvert_{x_i}\approx\frac{f(x_{i+1})-f(x_{i})}{x_{i+1}-x_i} \hspace{5mm} \text{backward: } \hspace{3mm} \frac{df}{dx}\bigg\rvert_{x_i}\approx\frac{f(x_{i})-f(x_{i-1})}{x_{i}-x_{i-1}} \hspace{5mm} \text{central: } \hspace{3mm} \frac{df}{dx}\bigg\rvert_{x_i}\approx\frac{f(x_{i+1})-f(x_{i-1})}{x_{i+1}-x_{i-1}} \]

Fig. 1.3 graphs illustrating three different ways to approximate derivatives.

Using Taylor-series to analyze difference expressions

Above, we just introduced the difference expressions that approximate the first derivative based on a graphical argument. In this section, we will look closer into how they can be derived mathematically by using Taylor series expansion. This will be useful to understand how accurate the different approximations are.

Taylor series describe a function by an infinite sum of polynomials. By truncating the infinite sum to a finite number of terms, we can approximate the function. The concept of Taylor series might be familiar to you from previous math classes. If you need a refresher, have a look at the page in the appendix.

Forward and backward difference

Let’s first see how we can derive the forward and backward difference scheme. We start with the Taylor series around a point \(x_i\), and evaluate it at a point \(x_{i+1} = x_i + \Delta x\) that is just a little bit to the right of \(x_i\). The expression for the Taylor series is:

\[ f(x_{i+1}) = f(x_i) + \underbrace{(x_{i+1}-x_i)}_{\Delta x}f'(x_i)+\frac{(\Delta x)^2}{2!}f''(x_i) + \frac{\Delta x^3}{3!} f'''(x_i) +... \]

Pause a moment and note that in this expression, we only ever need to evaluate our function \(f\) and its derivatives at the point \(x_i\), not \(x_{i+1}\) or any other point. This is useful because if we know the value of \(f\) (and its derivatives) at that one point \(x_i\), we can compute the value of \(f\) at another point (\(x_{i+1}\)) without needing to evaluate it!

Now, let’s solve the expression for the first derivative:

\[ f'(x_i)=\frac{f(x_i+\Delta x)-f(x_i)}{\Delta x} - \frac{\Delta x}{2!}f''(x_i) - \frac{\Delta x^2}{3!} f'''(x_i) - ... \]

By truncating the derivative to avoid the calculation of the second derivative, we find the forward difference

\[ f'(x_i)=\frac{f(x_i+\Delta x)-f(x_i)}{\Delta x} + \mathcal{O}(\Delta x). \]

This is the same expression you saw for the forward numerical derivative above! Now we have the added knowledge that this comes with an error of the order of the step.

Exercise: Backward difference

In a similar way, we can derive an expression for the backward difference. Use a Taylor series approximation to derive the backward difference for the first derivative \(f'(x_i)\) with a first order error \(\mathcal{O}(\Delta x)\).

Solution

We can get an expression for the backward difference by evaluating the Taylor series expansion at the point \(x_{i-1} = x_i - \Delta x\).

\[ f(x_{i-1}) = f(x_i) + \underbrace{(x_{i-1}-x_i)}_{-\Delta x}f'(x_i)+\frac{(-\Delta x)^2}{2!}f''(x_i) +\frac{(-\Delta x)^3}{3!}f'''(x_i) + ... \]

\[ f'(x_i)=\frac{f(x_i)-f(x_{i-1})}{\Delta x} + \frac{\Delta x}{2!}f''(x_i)- \frac{\Delta x^2}{3!} f'''(x_i) - ... \]

\[ f'(x_i)=\frac{f(x_i)-f(x_i-\Delta x)}{\Delta x}+ \mathcal{O}(\Delta x).\]

Central difference

The central difference can be found by summing the forward and backward difference expressions of the derivative and dividing it by 2:

Forward difference:

\[ f'(x_i)=\frac{f(x_{i+1})-f(x_i)}{\Delta x} - \frac{\Delta x}{2!}f''(x_i) - \frac{\Delta x^2}{3!} f'''(x_i) - ... \]

Backward difference:

\[ f'(x_i)=\frac{f(x_i)-f(x_{i-1})}{\Delta x} + \frac{\Delta x}{2!}f''(x_i)- \frac{\Delta x^2}{3!} f'''(x_i) - ... \]

Summing both terms yields:

\[2 f'(x_i) = \frac{f(x_{i+1}) - f(x_i) + f(x_i) - f(x_{i-1})}{\Delta x} - \frac{\Delta x}{2!}f''(x_i) + \frac{\Delta x}{2!}f''(x_i) + \mathcal{O}(\Delta x^2)\]

After dividing by 2 and simplifying, we get:

\[f'(x_i) = \frac{f(x_{i+1}) - f(x_{i-1})}{2 \Delta x} + \mathcal{O}(\Delta x^2)\]

Note that the second derivative terms cancel each other out, therefore the order error is the step size squared! This means that the central difference is more accurate than the forward or backward difference. You can notice it as well intuitively in the figure of the previous chapter.

TSE to define second derivatives

There are equations that require second derivatives. One example is the diffusion equation. The 1-D diffusion equation reads:

\[ \frac{\partial f}{\partial t}=v\frac{\partial^2 f}{\partial x^2}\,, \]

where \(v\) is the diffusion coefficient. For the moment we will use TSE to find only a numerical expression of the second derivative \(\frac{\partial^2 f}{\partial x^2}\).

The procedure is simple but cumbersome. The general idea is to isolate the second derivative in the TSE without there being a dependency on other derivatives. Below you can find more details about the algebraic manipulation (if you are curious) but you do not need to know it. Here is the result:

\[ f''(x_i)=\frac{f(x_i+2\Delta x)-2f(x_i+\Delta x)+f(x_i)}{\Delta x^2}+ \mathcal{O}(\Delta x). \]

This is the forward difference approximation of the second derivative. Two aspects come to mind:

1. one additional point is needed to compute the second derivative and

2. the error (of the simplest second derivative) is also of the order of the step.

There are also backward and central approximations of the second derivative (not shown here).

Deriving the forward difference approximation for the second derivative

First define a TSE for a point two steps away from \(x_i\):

\[ f(x_i+2\Delta x) = f(x_i) + 2\Delta x f'(x_i)+\frac{(2\Delta x)^2}{2!}f''(x_i)+ \mathcal{O}(\Delta x^3). \]

Now multiply the TSE by two for a point one step away from \(x_i\):

\[ 2f(x_i+\Delta x) = 2f(x_i) + 2\Delta x f'(x_i)+\frac{2\Delta x^2}{2!}f''(x_i) + \mathcal{O}(\Delta x^3). \]

By subtracting the first expression from the second one the first derivative disappears:

\[ f(x_i+2\Delta x) - 2f(x_i+\Delta x) = -f(x_i) + \frac{2\Delta x^2}{2!}f''(x_i)+ \mathcal{O}(\Delta x^3). \]

By solving for \(f''\) we obtain the forward expression:

\[ f''(x_i)=\frac{f(x_i+2\Delta x)-2f(x_i+\Delta x)+f(x_i)}{\Delta x^2}+ \mathcal{O}(\Delta x). \]

Higher-accuracy Finite-Difference Approximations

So far we have found expressions with a relative large magnitude error \(\mathcal{O}(\Delta x)\) with the exception of the central difference approximation. Sometimes a higher accuracy is desired for which better expressions can be found. The procedure is similar to the algebraic manipulation to find the forward approximation of the second derivative: a series of TSE are defined at varying distances from \(x_i\) and after algebraic manipulation a more accurate expression is found. For example, the forward approximation of the first derivative:

\[ f'(x_i)=\frac{-f(x_i+2\Delta x)+4f(x_i+\Delta x)-3f(x_i)}{2\Delta x}+ \mathcal{O}(\Delta x^2). \]

The error magnitude has improved to \(\mathcal{O}(\Delta x^2)\) at the expense of using one more point. The accuracy can be even better by using more points. It is important to note that central differences are more accurate than forward and backward differences when using the same number of points.

The following exercise is about deriving the expression for the forward approximation with higher accuracy.

Exercise

Derive a first derivative \(f'(x)\) with a 2nd-order error \(\mathcal{O}(\Delta x^2)\) finite-difference equation, using the nodes \(x_i\), \(x_i+\Delta x\) and \(x_i+2\Delta x\) or \(x_i\), \(x_{i+1}\) and \(x_{i+2}\).

The finite-difference equations will have the following form:

\[ f'(x_i)= \frac{\alpha f(x_i)+ \beta f(x_i+\Delta x) + \gamma f(x_i+2\Delta x)}{\Delta x} + \mathcal{O}(\Delta x^2) \]

Use the Taylor series of \(f(x_i+\Delta x)\) and \(f(x_i+2\Delta x)\) to find \(\alpha\), \(\beta\) and \(\gamma\).

Tip: You want to find an expression for the first order derivative with a second order error, \(1+2=3\). This means you need to truncate the Taylor series expansion until third order.

Solution

Taylor series for \(f(x_i+\Delta x)\) and \(f(x_i+2\Delta x)\) :

\[ f(x_i+\Delta x) = f(x_i) + \Delta x f'(x_i)+\frac{(\Delta x)^2}{2!}f''(x_i)+ \mathcal{O}(\Delta x^3). \]

\[ f(x_i+2\Delta x) = f(x_i) + 2\Delta x f'(x_i)+\frac{(2\Delta x)^2}{2!}f''(x_i)+ \mathcal{O}(\Delta x^3). \]

Multiply \(f(x_i+\Delta x)\) by 4 and expand the term \(\frac{(2\Delta x)^2}{2!}f''(x_i)\) in TSE \(f(x_i+2\Delta x)\):

\[ 4f(x_i+\Delta x)= 4f(x_i) + 4\Delta x f'(x_i)+2(\Delta x)^2f''(x_i)+ 4\mathcal{O}(\Delta x^3). \]

\[ f(x_i+2\Delta x) = f(x_i) + 2\Delta x f'(x_i)+2(\Delta x)^2f''(x_i)+ \mathcal{O}(\Delta x^3). \]

Now take \(4f(x_i+\Delta x)-f(x_i+2\Delta x)\):

\[ 4f(x_i+\Delta x)-f(x_i+2\Delta x)= 4f(x_i) + 4\Delta x f'(x_i)+2(\Delta x)^2f''(x_i)+ 4\mathcal{O}(\Delta x^3) - f(x_i) + 2\Delta x f'(x_i)+2(\Delta x)^2f''(x_i)+ \mathcal{O}(\Delta x^3). \]

\[ = 3f(x_i)+ 2\Delta x f'(x_i) + 3\mathcal{O}(\Delta x^3) \]

Rearrange for \(f'(x_i)\):

\[ - 2\Delta x f'(x_i) = 3f(x_i)-4f(x_i+\Delta x)+f(x_i+2\Delta x)+ 3\mathcal{O}(\Delta x^3) \]

Divide by \(-2 \Delta x\):

\[ f'(x_i)= \frac{-\frac{3}{2}f(x_i)+2f(x_i+\Delta x)-\frac{1}{2} f(x_i+2\Delta x)}{\Delta x} +\mathcal{O}(\Delta x^2) \]

The solution is: \(\alpha=-\frac{3}{2}\), \(\beta= 2\), \(\gamma= -\frac{1}{2}\)

Attribution

This chapter is written by Jaime Arriaga Garcia, Anna Störiko, Justin Pittman and Robert Lanzafame. Find out more here.