GA 2.6: A stethoscope for beams - neural networks for detecting defects on bridges

CEGM1000 MUDE: Week 2.6. Due: Friday, Dec 20, 2024.

Questions:

This notebook includes boxes with the formatting shown here to list the questions you are expected to answer in your report. You are not expected to write your answers here.

Introduction

In this notebook, you should apply the techniques you learned for regression in a more realistic problem setting. We have a collection of 1000 bridges modeled as 2D beams that all feature a defect (crack) that can be located anywhere along the span of the beam:

Figure 1. A bridge idealized as a 2D beam. Somewhere along the span there is a crack we are trying to detect.

Let us assume that we built these 1000 beams, purposefully inserted a defect in each of them at random locations, and then loaded them to measure how they deform. Our goal is to, given a new bridge for which we do not know where the defect is located, detect where it is without having to demolish the bridge. It is reasonable to assume that the presence of the crack will affect how the bridge deforms under loading, so we can try to use displacement sensors to estimate where the crack is. Since installing and monitoring displacement sensors is an expensive endeavor, we try to use as few sensors as we can.

We will use neural networks to map sensor displacements to crack locations:

Figure 3. Artificial neural network representation, with a zoomed-in view of how a single neuron works.

and train it with the 1000 bridges in our dataset. When it is time to make predictions for a new structure, we can just feed the measured displacements as inputs and get the estimated crack location as output.

The assignment includes the following tasks:

• Pre-process the available data to use it in a neural network
• Train a neural network to learn a mapping from the displacement measurements to the defect location, comment on the choice of hyperparameters (number of hidden layers, nodes per layer, ...)
• Visualise your results and evaluate the accuracy of your network

and is split in two parts: first we use only one sensor at the center of the beam and then look at our results; then we add two more sensors midway between halfspan and the supports and see how that improves predictions.

Preliminaries

Libraries

To run this notebook you need to have installed the following packages:

• Numpy
• Matplotlib
• Pandas
• Scikit-learn

import time
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%config InlineBackend.figure_formats = ['svg']

from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

Load the dataset

Let us take a look at the dataset first. It is a CSV file, and a convenient way to read and manipulate this file type is via the Dataframe of the pandas library.

data = pd.read_csv(r"data/bridges.csv")

#summary of the data
data.describe()

	sample	location	node	x	y	dx	dy
count	3000.000000	3000.000000	3000.000000	3000.000000	3000.0	3000.000000	3000.000000
mean	499.500000	4.836256	28.666667	4.933333	0.0	0.014143	-0.038273
std	288.723115	2.894467	10.210630	2.042126	0.0	0.007008	0.005674
min	0.000000	0.044241	16.000000	2.400000	0.0	0.005423	-0.046335
25%	249.750000	2.220470	16.000000	2.400000	0.0	0.005486	-0.046155
50%	499.500000	4.851140	29.000000	5.000000	0.0	0.014336	-0.035304
75%	749.250000	7.282190	41.000000	7.400000	0.0	0.022605	-0.033339
max	999.000000	9.963194	41.000000	7.400000	0.0	0.022705	-0.033214

We can take a look at how the crack location relates with displacements at three different locations along the beams (25% of the span, midspan, 75% of the span):

from matplotlib import cm

loc1 = data[data['node'] == 16] # point at 25% span
loc2 = data[data['node'] == 29] # point at midspan
loc3 = data[data['node'] == 41] # point at 75% span

fig,axes = plt.subplots(1,3,figsize=(10,3))

axes[0].scatter(loc1['dy'], loc1['location'], c=loc1['sample'], s=0.5)
axes[0].set_xlabel('displacement at 25% span [m]')
axes[0].set_ylabel('crack location [m]')

axes[1].scatter(loc2['dy'], loc1['location'], c=loc2['sample'], s=0.5)
axes[1].set_xlabel('displacement at 50% span [m]')
axes[1].set_ylabel('crack location [m]')

axes[2].scatter(loc3['dy'], loc1['location'], c=loc3['sample'], s=0.5)
axes[2].set_xlabel('displacement at 75% span [m]')
axes[2].set_ylabel('crack location [m]')

plt.tight_layout()
plt.show()

Perhaps you can already see this will be a challenging endeavor, especially using just one of these input features. In the figure above we identify each beam with a unique color, so you can see that to really get it right we might need to gather information from multiple sensors. Look for instance at the middle plot: for a displacement of around $-0.0461$ our defect might either be at $x=0$ or $x=10$, so at the two opposite ends of the beam! Can you reason out why that is the case?

For now let us start with a model with a single feature (displacement at midspan, a.k.a. loc2) and see how well we do.

1. Data Pre-Processing

Splitting the data into training, validation, and testing sets

Task 1.1:

In machine learning, it is common to split the dataset into three parts: a training set, a validation set, and a test set.

Your task is to write a Python code snippet that splits a given dataset into these three parts. The dataset consists of features and targets.

The dataset should be split as follows:

• 80% of the data should go to the training set.
• 10% of the data should go to the validation set.
• 10% of the data should go to the test set.

The splitting should be done in a way that shuffles the data first to ensure that the training, validation, and test sets are representative of the overall distribution of the data. You can set the random state for the shuffling to ensure that the results are reproducible; you can use the values 42 for the first split and 24 for the second split.

The resulting training, validation, and test sets should be stored in the variables X_train, t_train, X_val, t_val, X_test, and t_test.

Hint: You can use the train_test_split function from the sklearn.model_selection module to perform the splitting.

Figure 3. Splitting the dataset into training, validation and test sets.

features = loc2['dy'].to_numpy().reshape(-1,1)
targets = loc2['location'].to_numpy().reshape(-1,1)

X_train, X_val_test, t_train, t_val_test = train_test_split(features, targets, test_size=0.20, random_state=42)
X_val, X_test, t_val, t_test = train_test_split(X_val_test, t_val_test, test_size=0.50, random_state=24)

Normalizing the data

Now, we normalize the data using the MinMaxScaler from scikit-learn. This scaler transforms the data to be between 0 and 1. This is important because the neural net will be trained using gradient descent, which is sensitive to the scale of the data. Notice that we use the training data to fit the scaler. This is important because we assume that the model only sees the training data and we do not use any of the validation or testing data.

Task 1.2:

In machine learning, it is often beneficial to normalize the feature variables to a specific range. This can help the model converge faster during training and can also prevent certain features from dominating others due to their scale.

Your task is to write a Python code snippet that normalizes the feature variables of a training set and a validation set to the range [0, 1]. The feature variables are stored in the variables X_train and X_val.

You should use the MinMaxScaler class from the sklearn.preprocessing module to perform the normalization. This class scales and translates each feature individually such that it is in the given range on the training set. Remember we should fit the normalizer with only the training dataset!

The normalized features should be stored in the variables normalized_X_train and normalized_X_val.

Hint: See how we did this in the workshop assignment!

scaler_x = MinMaxScaler()

scaler_x.fit(X_train)

normalized_X_train = scaler_x.transform(X_train)
normalized_X_val = scaler_x.transform(X_val)

Task 1.3:

We now do the same for the targets. Your task is to write a Python code snippet that normalizes the target variables of a training set and a validation set to the range [0, 1]. The target variables are stored in the variables t_train and t_val.

The normalized targets should be stored in the variables normalized_t_train and normalized_t_val.

Hint: See how we did this in the workshop assignment!

scaler_t = MinMaxScaler()

scaler_t.fit(t_train)

normalized_t_train = scaler_t.transform(t_train)
normalized_t_val = scaler_t.transform(t_val)

Questions:

1.1) What is the purpose of splitting a dataset into training, validation, and test sets in the context of machine learning?

1.2) Why should the MinMaxScaler be fitted on the training data only?

1.3) Why is it crucial that the exact same scaler is used to transform the validation dataset?

2. Defining and training an MLP

Now, we will define and train a Multilayer Perceptron (MLP), which is the name we give to a simple (feedforward) neural network. In Scikit-learn, the MLP is defined in the MLPRegressor class, you can see the documentation here. This class has many hyperparameters that can be tuned to improve the performance of the model. Notice that in Scikit-learn, the model and the optimizer are defined in the same class. This means that we do not need to define an optimizer separately; therefore, some hyperparameters are related to the optimizer. For example, the learning rate. We will indicate the optimization hyperparameters in the next section; for now, we will only define the model hyperparameters.

Training the model

Once we have a model in place we will need to train it. Scikit-learn offers the possibility to directly train a model using the fit method. However, we will define a training loop to have more control over the training process. This will allow us to evaluate the model at each epoch and observe its training.

The first step towards training a model is defining a function that transforms our training dataset into random mini-batches. This is a common practice used for training neural networks due to their computational efficiency and their ability to help the model generalize better. This practice generally leads to better computational efficiency, faster convergence and better generalization performance.

def get_mini_batches(X, t, batch_size):
    """
    This function generates mini-batches from the given input data and labels.

    Parameters:
    X (numpy.ndarray): The features.
    t (numpy.ndarray): The targets corresponding to the input data.
    batchsize (int): The size of each mini-batch.

    Returns:
    list: A list of tuples where each tuple contains a mini-batch of the input data and the corresponding targets.
    """
    # Generate permutations
    perm = np.random.permutation(len(X))
    X_train_perm = X[perm]
    t_train_perm = t[perm]
    
    # Generate mini-batches
    X_batches = []
    t_batches = []
    for i in range(0, len(X_train_perm), batch_size):
        X_batches.append(X_train_perm[i:i+batch_size])
        t_batches.append(t_train_perm[i:i+batch_size])

    return list(zip(X_batches, t_batches))

The following figure illustrates both the way we split the original dataset and how we further split the training dataset into mini-batches. At every epoch the training dataset is shuffled and each mini-batch is considered in isolation by the network. The gradients coming from the single mini-batches are used to update the weights of the network (the randomness involved is why we say we are using Stochastic Gradient Descent).

Figure 4. Dataset splitting, mini-batching and the stochastic nature of MLP training.

Now, we will define some hyperparameters for the training loop. These hyperparameters are related to the optimization process. Define the following hyperparameters:

• learning_rate (float): The learning rate of the optimizer.
• n_epochs (int): The number of epochs to train the model. (For time reasons, we will only train the model for 20 epochs. However, you can increase this number to improve the performance of the model.)
• batch_size (int): The size of each mini-batch.

learning_rate = 0.001
n_epochs = 20
batch_size = 64

Task 2.1:

In this exercise, you are tasked with implementing a function to train a neural network model. The function should also compute and store the loss on the training and validation sets at each epoch. The loss function to be used is the Mean Squared Error (MSE), which is defined as:

$$ MSE = \frac{1}{N} \sum_{n=1}^{N} (t_n - y_n)^2 $$

where $t_n$ is the actual target, $y_n$ is the predicted value, and $N$ is the number of samples.

The function should be named train_model and should take the following parameters:

• model: An instance of a neural network model that we want to train.
• normalized_X_train: The normalized training data.
• normalized_t_train: The normalized training labels.
• normalized_X_val: The normalized validation data.
• normalized_t_val: The normalized validation labels.
• n_epochs: The number of epochs to train the model for.
• batch_size: The size of each mini-batch.
• learning_rate: The learning rate for the model.

The function should perform the following steps:

1. Initialize two empty lists, train_loss_list and val_loss_list, to store the training and validation losses at each epoch.

2. Loop over the specified number of epochs. For each epoch:

   a. Generate mini-batches from the normalized training data and labels using a function get_mini_batches(normalized_X_train, normalized_t_train, batch_size);

   b. For each mini-batch, update the model weights using the partial_fit method of the model. Do not forget to flatten() the targets;

   c. Compute the MSE loss on the training set and append it to train_loss_list;

   d. Compute the MSE loss on the validation set and append it to val_loss_list;

   e. Print the training progress including the current epoch and the training and validation losses;

   f. Return the train_loss_list and val_loss_list lists.

Your task is to write the Python code that implements the train_model function.

def train_model(model, normalized_X_train, normalized_t_train, normalized_X_val, normalized_t_val, n_epochs, batch_size, learning_rate, verbose=True):
    train_loss_list = []
    val_loss_list = []
    model.learning_rate_init = learning_rate

    # Fix random seed for reproducibility
    np.random.seed(42)
    
    for epoch in range(n_epochs):
        
        # Generate mini-batches
        mini_batches = get_mini_batches(normalized_X_train, normalized_t_train, batch_size)
        
        # Train model on mini-batches
        for X_batch, t_batch in mini_batches:
            model.partial_fit(X_batch, t_batch.flatten())
        
        # Compute loss on training and validation sets
        train_loss = mean_squared_error(normalized_t_train, model.predict(normalized_X_train))
        
        # Compute loss on validation set
        val_loss = mean_squared_error(normalized_t_val, model.predict(normalized_X_val))

        # Store loss values
        train_loss_list.append(train_loss)
        val_loss_list.append(val_loss)

        # Print training progress
        if verbose:
            print(f"Epoch {epoch+1}/{n_epochs} - Train Loss: {train_loss_list[-1]:.4f} - Val Loss: {val_loss:.4f}")
        
    return train_loss_list, val_loss_list

Task 2.2: You are tasked with setting up a Multi-Layer Perceptron (MLP). The MLP should have the following characteristics:

• The hidden layer sizes are defined as a tuple. For this task we want to have two hidden layers with 10 and 5 neurons, respectively, so we should write: hidden_layer_sizes=(10,5). Notice that we only specify the hidden layer sizes, the input and output sizes will be automatically inferred when we train the model.
• The activation function can be one of the following: 'identity', 'logistic', 'tanh', 'relu'. But for this task you should set it to 'tanh'.

The configured MLPRegressor should be stored in a variable named model. We then call the training function we defined above and pass this model as argument.

Training neural networks is a stochastic operation: the MLP is given random initial weights and SGD is random by nature. If you want to make sure you always get the same trained network every time you run the notebook (e.g. for comparison reasons), you can pass random_state=0 (or some other integer) to MLPRegressor

model = MLPRegressor(hidden_layer_sizes = (10, 5), 
                    activation = 'tanh',
                    random_state=1)

train_loss_list, val_loss_list = train_model(model, normalized_X_train, normalized_t_train, normalized_X_val, normalized_t_val, n_epochs, batch_size, learning_rate)

Epoch 1/20 - Train Loss: 0.2700 - Val Loss: 0.2962
Epoch 2/20 - Train Loss: 0.1554 - Val Loss: 0.1742
Epoch 3/20 - Train Loss: 0.1036 - Val Loss: 0.1164
Epoch 4/20 - Train Loss: 0.0890 - Val Loss: 0.0979
Epoch 5/20 - Train Loss: 0.0869 - Val Loss: 0.0940
Epoch 6/20 - Train Loss: 0.0868 - Val Loss: 0.0930
Epoch 7/20 - Train Loss: 0.0864 - Val Loss: 0.0928
Epoch 8/20 - Train Loss: 0.0861 - Val Loss: 0.0925
Epoch 9/20 - Train Loss: 0.0859 - Val Loss: 0.0923
Epoch 10/20 - Train Loss: 0.0857 - Val Loss: 0.0922
Epoch 11/20 - Train Loss: 0.0856 - Val Loss: 0.0920
Epoch 12/20 - Train Loss: 0.0854 - Val Loss: 0.0916
Epoch 13/20 - Train Loss: 0.0853 - Val Loss: 0.0914
Epoch 14/20 - Train Loss: 0.0852 - Val Loss: 0.0912
Epoch 15/20 - Train Loss: 0.0851 - Val Loss: 0.0910
Epoch 16/20 - Train Loss: 0.0850 - Val Loss: 0.0908
Epoch 17/20 - Train Loss: 0.0850 - Val Loss: 0.0909
Epoch 18/20 - Train Loss: 0.0849 - Val Loss: 0.0908
Epoch 19/20 - Train Loss: 0.0849 - Val Loss: 0.0905
Epoch 20/20 - Train Loss: 0.0849 - Val Loss: 0.0908

Task 2.3: Plot the validation and training loss curves. Add this plot to your report.

# Create a scatter plot with enhanced styling
plt.figure(figsize=(8, 6))  # Set the figure size

x_axis = list(range(len(train_loss_list)))

# Create a scatter plot
# plt.scatter(x_axis, YOUR_CODE_HERE, label='Validation loss', color='red', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)
# plt.scatter(x_axis, YOUR_CODE_HERE, label='Training loss', color='blue', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)
# Solution:
plt.scatter(x_axis, val_loss_list, label='Validation loss', color='red', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)
plt.scatter(x_axis, train_loss_list, label='Training loss', color='blue', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)

# Add labels and a legend with improved formatting
plt.xlabel('Epochs', fontsize=14, fontweight='bold')
plt.ylabel('Loss', fontsize=14, fontweight='bold')
plt.title('Loss curves', fontsize=16, fontweight='bold')
plt.legend(loc='upper right', fontsize=12)

# Set the y-axis to be logarithmic
plt.yscale('log')

# Customize the grid appearance
plt.grid(True, linestyle='--', alpha=0.5)

# Customize the tick labels
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)

# Add a background color to the plot
plt.gca().set_facecolor('#f2f2f2')
plt.show()

Task 2.4:

Now let us look at how well our model performs. With only one input feature, one easy way to do this is by just plotting $y(x)$ for the whole range of $x$. Since we have normalized our inputs with a MinMaxScaler, it is reasonable to form a simple linspace in $[0,1]$. We can then plot this function $y(x)$ together with our training and validation data to have an idea of how close model predictions are.

In the block below we already prepare a normalized_X_range for you to feed to the model (remember the network learned to use normalized inputs). Your task is to compute model predictions and store them in t_range. Remember to de-normalize what comes out of the network.

Add these plots to your report.

normalized_X_range = np.linspace(0,1,100).reshape(-1,1)
X_range = scaler_x.inverse_transform(normalized_X_range)

normalized_y_range = model.predict(normalized_X_range).reshape(-1,1)
y_range = scaler_t.inverse_transform(normalized_y_range)

fig,axes = plt.subplots(1,2,figsize=(8,3))

axes[0].plot(X_range, y_range, label=r"Network $y(x)$", color='k')
axes[0].scatter(X_train,t_train,s=0.5, label='Training data')
axes[0].set_xlabel('displacement at 50% span [m]')
axes[0].set_ylabel('crack location [m]')

axes[1].plot(X_range, y_range,label=r"Network $y(x)$", color='k')
axes[1].scatter(X_val,t_val,s=0.5, label='Validation data')
axes[1].set_xlabel('displacement at 50% span [m]')
axes[1].set_ylabel('crack location [m]')

axes[0].legend()
axes[1].legend()
plt.tight_layout()

Task 2.5:

Another useful way to visualize our results is to use a so-called parity plot. This consists in plotting targets on the x-axis against network predictions on the y-axis, in other words a $t$ versus $y$ plot. For a model that always makes perfect predictions, $y=t$ for all instances of the target, which would make all points in the plot lie on the diagonal line $y=x$. In practice model predictions are often far from perfect, and this deviation would be seen as the points drifting away from the diagonal.

Your task is to set up two parity plots for this model, one for training data and one for validation data.

Add these plots to your report.

y_train = scaler_t.inverse_transform(model.predict(normalized_X_train).reshape(-1,1))
y_val = scaler_t.inverse_transform(model.predict(normalized_X_val).reshape(-1,1))

fig,axes = plt.subplots(1,2,figsize=(8,3))

axes[0].scatter(t_train,y_train,s=0.5)

axes[0].set_title('Training dataset')
axes[0].set_xlabel('target crack location [m]')
axes[0].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_train), np.min(y_train))
max_val = max(np.max(t_train), np.max(y_train))
axes[0].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[0].legend()

# axes[1].scatter(YOUR_CODE_HERE,YOUR_CODE_HERE,s=0.5)
# Solution
axes[1].scatter(t_val,y_val,s=0.5)

axes[1].set_title('Validation dataset')
axes[1].set_xlabel('target crack location [m]')
axes[1].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_val), np.min(y_val))
max_val = max(np.max(t_val), np.max(y_val))
axes[1].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[1].legend()

plt.tight_layout()

Questions:

2.1) Based on the shape of the loss curves, what can you indicate about the fitting capabilities of the model? (Is it overfitting, underfitting, or neither?)

2.2) Why is the model performing so poorly? Can you give an explanation based on the physics of the problem? Is there a crack location for which this model does make a good prediction? Why is that the case?

2.3) Can you explain why the model performs poorly in light of the assumptions we made for our observation model $p(t\vert x)=\mathcal{N}\left(t\lvert y(x),\beta^{-1}\right)$?

3. Using more input features

Let us now try the model with three sensors as input features and see if that improves our predictions. We follow the exact same procedure as before.

Set up and train the new model

Task 3.1:

Use this block to set up a new neural network that now takes all three input features (displacements at three different sensors). The $[1000\times 3]$ feature matrix features and the $[1000\times 1]$ target vector targets are already done for you.

Now get all the code you need from the code blocks in Part 2 and paste them here. You need to (i) split your dataset again since it has changed; (ii) normalize features and targets; (iii) adjust the number of epochs and model architecture until you are satisfied with the model; (iv) set up an MLPRegressor and call train_model.

To reach a good architecture and number of epochs you will need to also complete Task 3.2 and rerun these two blocks until you are satisfied with the number you picked. Try out different activation functions to see what works best.

features = np.array([loc1['dy'].to_numpy(), loc2['dy'].to_numpy(), loc3['dy'].to_numpy()]).transpose()
targets = loc2['location'].to_numpy().reshape(-1,1)

X_train, X_val_test, t_train, t_val_test = train_test_split(features, targets, test_size=0.20, random_state=42)
X_val, X_test, t_val, t_test = train_test_split(X_val_test, t_val_test, test_size=0.50, random_state=24)

scaler_x = MinMaxScaler()
scaler_x.fit(X_train)

normalized_X_train = scaler_x.transform(X_train)
normalized_X_val = scaler_x.transform(X_val)

scaler_t = MinMaxScaler()
scaler_t.fit(t_train)

normalized_t_train = scaler_t.transform(t_train)
normalized_t_val = scaler_t.transform(t_val)

model = MLPRegressor(hidden_layer_sizes = (10, 5), 
                    activation = 'tanh')

learning_rate = 0.001
n_epochs = 200
batch_size = 64

train_loss_list, val_loss_list = train_model(model, normalized_X_train, normalized_t_train, normalized_X_val, normalized_t_val, n_epochs, batch_size, learning_rate)

Epoch 1/200 - Train Loss: 0.1834 - Val Loss: 0.2043
Epoch 2/200 - Train Loss: 0.0945 - Val Loss: 0.1073
Epoch 3/200 - Train Loss: 0.0606 - Val Loss: 0.0684
Epoch 4/200 - Train Loss: 0.0513 - Val Loss: 0.0566
Epoch 5/200 - Train Loss: 0.0489 - Val Loss: 0.0529
Epoch 6/200 - Train Loss: 0.0472 - Val Loss: 0.0508
Epoch 7/200 - Train Loss: 0.0452 - Val Loss: 0.0488
Epoch 8/200 - Train Loss: 0.0431 - Val Loss: 0.0468
Epoch 9/200 - Train Loss: 0.0411 - Val Loss: 0.0447
Epoch 10/200 - Train Loss: 0.0391 - Val Loss: 0.0425
Epoch 11/200 - Train Loss: 0.0371 - Val Loss: 0.0403
Epoch 12/200 - Train Loss: 0.0351 - Val Loss: 0.0381
Epoch 13/200 - Train Loss: 0.0331 - Val Loss: 0.0359
Epoch 14/200 - Train Loss: 0.0312 - Val Loss: 0.0338
Epoch 15/200 - Train Loss: 0.0293 - Val Loss: 0.0318
Epoch 16/200 - Train Loss: 0.0276 - Val Loss: 0.0299
Epoch 17/200 - Train Loss: 0.0260 - Val Loss: 0.0281
Epoch 18/200 - Train Loss: 0.0245 - Val Loss: 0.0264
Epoch 19/200 - Train Loss: 0.0231 - Val Loss: 0.0247
Epoch 20/200 - Train Loss: 0.0218 - Val Loss: 0.0232
Epoch 21/200 - Train Loss: 0.0207 - Val Loss: 0.0220
Epoch 22/200 - Train Loss: 0.0197 - Val Loss: 0.0207
Epoch 23/200 - Train Loss: 0.0189 - Val Loss: 0.0197
Epoch 24/200 - Train Loss: 0.0181 - Val Loss: 0.0187
Epoch 25/200 - Train Loss: 0.0175 - Val Loss: 0.0179
Epoch 26/200 - Train Loss: 0.0170 - Val Loss: 0.0172
Epoch 27/200 - Train Loss: 0.0165 - Val Loss: 0.0165
Epoch 28/200 - Train Loss: 0.0161 - Val Loss: 0.0160
Epoch 29/200 - Train Loss: 0.0158 - Val Loss: 0.0155
Epoch 30/200 - Train Loss: 0.0155 - Val Loss: 0.0151
Epoch 31/200 - Train Loss: 0.0153 - Val Loss: 0.0147
Epoch 32/200 - Train Loss: 0.0151 - Val Loss: 0.0145
Epoch 33/200 - Train Loss: 0.0149 - Val Loss: 0.0142
Epoch 34/200 - Train Loss: 0.0148 - Val Loss: 0.0140
Epoch 35/200 - Train Loss: 0.0147 - Val Loss: 0.0138
Epoch 36/200 - Train Loss: 0.0146 - Val Loss: 0.0136
Epoch 37/200 - Train Loss: 0.0145 - Val Loss: 0.0135
Epoch 38/200 - Train Loss: 0.0144 - Val Loss: 0.0133
Epoch 39/200 - Train Loss: 0.0143 - Val Loss: 0.0132
Epoch 40/200 - Train Loss: 0.0142 - Val Loss: 0.0130
Epoch 41/200 - Train Loss: 0.0141 - Val Loss: 0.0129
Epoch 42/200 - Train Loss: 0.0140 - Val Loss: 0.0128
Epoch 43/200 - Train Loss: 0.0140 - Val Loss: 0.0127
Epoch 44/200 - Train Loss: 0.0138 - Val Loss: 0.0125
Epoch 45/200 - Train Loss: 0.0137 - Val Loss: 0.0124
Epoch 46/200 - Train Loss: 0.0136 - Val Loss: 0.0123
Epoch 47/200 - Train Loss: 0.0136 - Val Loss: 0.0122
Epoch 48/200 - Train Loss: 0.0135 - Val Loss: 0.0121
Epoch 49/200 - Train Loss: 0.0134 - Val Loss: 0.0120
Epoch 50/200 - Train Loss: 0.0133 - Val Loss: 0.0119
Epoch 51/200 - Train Loss: 0.0132 - Val Loss: 0.0118
Epoch 52/200 - Train Loss: 0.0131 - Val Loss: 0.0117
Epoch 53/200 - Train Loss: 0.0130 - Val Loss: 0.0116
Epoch 54/200 - Train Loss: 0.0129 - Val Loss: 0.0115
Epoch 55/200 - Train Loss: 0.0128 - Val Loss: 0.0114
Epoch 56/200 - Train Loss: 0.0127 - Val Loss: 0.0113
Epoch 57/200 - Train Loss: 0.0126 - Val Loss: 0.0111
Epoch 58/200 - Train Loss: 0.0125 - Val Loss: 0.0111
Epoch 59/200 - Train Loss: 0.0123 - Val Loss: 0.0109
Epoch 60/200 - Train Loss: 0.0122 - Val Loss: 0.0108
Epoch 61/200 - Train Loss: 0.0121 - Val Loss: 0.0106
Epoch 62/200 - Train Loss: 0.0120 - Val Loss: 0.0105
Epoch 63/200 - Train Loss: 0.0119 - Val Loss: 0.0104
Epoch 64/200 - Train Loss: 0.0118 - Val Loss: 0.0103
Epoch 65/200 - Train Loss: 0.0117 - Val Loss: 0.0101
Epoch 66/200 - Train Loss: 0.0115 - Val Loss: 0.0100
Epoch 67/200 - Train Loss: 0.0114 - Val Loss: 0.0099
Epoch 68/200 - Train Loss: 0.0113 - Val Loss: 0.0097
Epoch 69/200 - Train Loss: 0.0112 - Val Loss: 0.0096
Epoch 70/200 - Train Loss: 0.0110 - Val Loss: 0.0094
Epoch 71/200 - Train Loss: 0.0109 - Val Loss: 0.0094
Epoch 72/200 - Train Loss: 0.0108 - Val Loss: 0.0092
Epoch 73/200 - Train Loss: 0.0106 - Val Loss: 0.0090
Epoch 74/200 - Train Loss: 0.0105 - Val Loss: 0.0089
Epoch 75/200 - Train Loss: 0.0104 - Val Loss: 0.0088
Epoch 76/200 - Train Loss: 0.0102 - Val Loss: 0.0087
Epoch 77/200 - Train Loss: 0.0101 - Val Loss: 0.0085
Epoch 78/200 - Train Loss: 0.0100 - Val Loss: 0.0084
Epoch 79/200 - Train Loss: 0.0099 - Val Loss: 0.0083
Epoch 80/200 - Train Loss: 0.0097 - Val Loss: 0.0082
Epoch 81/200 - Train Loss: 0.0096 - Val Loss: 0.0080
Epoch 82/200 - Train Loss: 0.0095 - Val Loss: 0.0079
Epoch 83/200 - Train Loss: 0.0094 - Val Loss: 0.0078
Epoch 84/200 - Train Loss: 0.0093 - Val Loss: 0.0077
Epoch 85/200 - Train Loss: 0.0092 - Val Loss: 0.0075
Epoch 86/200 - Train Loss: 0.0091 - Val Loss: 0.0074
Epoch 87/200 - Train Loss: 0.0089 - Val Loss: 0.0073
Epoch 88/200 - Train Loss: 0.0088 - Val Loss: 0.0072
Epoch 89/200 - Train Loss: 0.0087 - Val Loss: 0.0071
Epoch 90/200 - Train Loss: 0.0086 - Val Loss: 0.0070
Epoch 91/200 - Train Loss: 0.0086 - Val Loss: 0.0069
Epoch 92/200 - Train Loss: 0.0085 - Val Loss: 0.0068
Epoch 93/200 - Train Loss: 0.0084 - Val Loss: 0.0067
Epoch 94/200 - Train Loss: 0.0083 - Val Loss: 0.0066
Epoch 95/200 - Train Loss: 0.0082 - Val Loss: 0.0066
Epoch 96/200 - Train Loss: 0.0081 - Val Loss: 0.0064
Epoch 97/200 - Train Loss: 0.0081 - Val Loss: 0.0063
Epoch 98/200 - Train Loss: 0.0080 - Val Loss: 0.0063
Epoch 99/200 - Train Loss: 0.0079 - Val Loss: 0.0063
Epoch 100/200 - Train Loss: 0.0079 - Val Loss: 0.0061
Epoch 101/200 - Train Loss: 0.0078 - Val Loss: 0.0061
Epoch 102/200 - Train Loss: 0.0077 - Val Loss: 0.0060
Epoch 103/200 - Train Loss: 0.0077 - Val Loss: 0.0059
Epoch 104/200 - Train Loss: 0.0076 - Val Loss: 0.0060
Epoch 105/200 - Train Loss: 0.0076 - Val Loss: 0.0059
Epoch 106/200 - Train Loss: 0.0075 - Val Loss: 0.0058
Epoch 107/200 - Train Loss: 0.0075 - Val Loss: 0.0057
Epoch 108/200 - Train Loss: 0.0075 - Val Loss: 0.0056
Epoch 109/200 - Train Loss: 0.0074 - Val Loss: 0.0058
Epoch 110/200 - Train Loss: 0.0074 - Val Loss: 0.0055
Epoch 111/200 - Train Loss: 0.0073 - Val Loss: 0.0056
Epoch 112/200 - Train Loss: 0.0073 - Val Loss: 0.0055
Epoch 113/200 - Train Loss: 0.0072 - Val Loss: 0.0055
Epoch 114/200 - Train Loss: 0.0072 - Val Loss: 0.0054
Epoch 115/200 - Train Loss: 0.0072 - Val Loss: 0.0053
Epoch 116/200 - Train Loss: 0.0071 - Val Loss: 0.0053
Epoch 117/200 - Train Loss: 0.0071 - Val Loss: 0.0053
Epoch 118/200 - Train Loss: 0.0071 - Val Loss: 0.0053
Epoch 119/200 - Train Loss: 0.0070 - Val Loss: 0.0052
Epoch 120/200 - Train Loss: 0.0070 - Val Loss: 0.0052
Epoch 121/200 - Train Loss: 0.0070 - Val Loss: 0.0052
Epoch 122/200 - Train Loss: 0.0070 - Val Loss: 0.0051
Epoch 123/200 - Train Loss: 0.0069 - Val Loss: 0.0051
Epoch 124/200 - Train Loss: 0.0069 - Val Loss: 0.0051
Epoch 125/200 - Train Loss: 0.0069 - Val Loss: 0.0050
Epoch 126/200 - Train Loss: 0.0068 - Val Loss: 0.0051
Epoch 127/200 - Train Loss: 0.0068 - Val Loss: 0.0050
Epoch 128/200 - Train Loss: 0.0068 - Val Loss: 0.0050
Epoch 129/200 - Train Loss: 0.0068 - Val Loss: 0.0049
Epoch 130/200 - Train Loss: 0.0067 - Val Loss: 0.0050
Epoch 131/200 - Train Loss: 0.0067 - Val Loss: 0.0049
Epoch 132/200 - Train Loss: 0.0067 - Val Loss: 0.0049
Epoch 133/200 - Train Loss: 0.0067 - Val Loss: 0.0049
Epoch 134/200 - Train Loss: 0.0067 - Val Loss: 0.0049
Epoch 135/200 - Train Loss: 0.0066 - Val Loss: 0.0049
Epoch 136/200 - Train Loss: 0.0066 - Val Loss: 0.0048
Epoch 137/200 - Train Loss: 0.0066 - Val Loss: 0.0049
Epoch 138/200 - Train Loss: 0.0066 - Val Loss: 0.0048
Epoch 139/200 - Train Loss: 0.0066 - Val Loss: 0.0047
Epoch 140/200 - Train Loss: 0.0065 - Val Loss: 0.0047
Epoch 141/200 - Train Loss: 0.0065 - Val Loss: 0.0047
Epoch 142/200 - Train Loss: 0.0065 - Val Loss: 0.0047
Epoch 143/200 - Train Loss: 0.0064 - Val Loss: 0.0047
Epoch 144/200 - Train Loss: 0.0064 - Val Loss: 0.0047
Epoch 145/200 - Train Loss: 0.0064 - Val Loss: 0.0046
Epoch 146/200 - Train Loss: 0.0064 - Val Loss: 0.0046
Epoch 147/200 - Train Loss: 0.0064 - Val Loss: 0.0046
Epoch 148/200 - Train Loss: 0.0063 - Val Loss: 0.0046
Epoch 149/200 - Train Loss: 0.0063 - Val Loss: 0.0046
Epoch 150/200 - Train Loss: 0.0063 - Val Loss: 0.0045
Epoch 151/200 - Train Loss: 0.0063 - Val Loss: 0.0045
Epoch 152/200 - Train Loss: 0.0063 - Val Loss: 0.0045
Epoch 153/200 - Train Loss: 0.0062 - Val Loss: 0.0045
Epoch 154/200 - Train Loss: 0.0062 - Val Loss: 0.0045
Epoch 155/200 - Train Loss: 0.0063 - Val Loss: 0.0046
Epoch 156/200 - Train Loss: 0.0063 - Val Loss: 0.0045
Epoch 157/200 - Train Loss: 0.0062 - Val Loss: 0.0045
Epoch 158/200 - Train Loss: 0.0062 - Val Loss: 0.0045
Epoch 159/200 - Train Loss: 0.0063 - Val Loss: 0.0047
Epoch 160/200 - Train Loss: 0.0061 - Val Loss: 0.0044
Epoch 161/200 - Train Loss: 0.0061 - Val Loss: 0.0045
Epoch 162/200 - Train Loss: 0.0061 - Val Loss: 0.0044
Epoch 163/200 - Train Loss: 0.0061 - Val Loss: 0.0044
Epoch 164/200 - Train Loss: 0.0060 - Val Loss: 0.0044
Epoch 165/200 - Train Loss: 0.0060 - Val Loss: 0.0043
Epoch 166/200 - Train Loss: 0.0060 - Val Loss: 0.0043
Epoch 167/200 - Train Loss: 0.0060 - Val Loss: 0.0044
Epoch 168/200 - Train Loss: 0.0060 - Val Loss: 0.0043
Epoch 169/200 - Train Loss: 0.0060 - Val Loss: 0.0044
Epoch 170/200 - Train Loss: 0.0060 - Val Loss: 0.0043
Epoch 171/200 - Train Loss: 0.0059 - Val Loss: 0.0043
Epoch 172/200 - Train Loss: 0.0059 - Val Loss: 0.0043
Epoch 173/200 - Train Loss: 0.0059 - Val Loss: 0.0043
Epoch 174/200 - Train Loss: 0.0059 - Val Loss: 0.0042
Epoch 175/200 - Train Loss: 0.0059 - Val Loss: 0.0042
Epoch 176/200 - Train Loss: 0.0059 - Val Loss: 0.0043
Epoch 177/200 - Train Loss: 0.0058 - Val Loss: 0.0042
Epoch 178/200 - Train Loss: 0.0059 - Val Loss: 0.0042
Epoch 179/200 - Train Loss: 0.0058 - Val Loss: 0.0042
Epoch 180/200 - Train Loss: 0.0058 - Val Loss: 0.0042
Epoch 181/200 - Train Loss: 0.0059 - Val Loss: 0.0044
Epoch 182/200 - Train Loss: 0.0058 - Val Loss: 0.0042
Epoch 183/200 - Train Loss: 0.0058 - Val Loss: 0.0042
Epoch 184/200 - Train Loss: 0.0058 - Val Loss: 0.0043
Epoch 185/200 - Train Loss: 0.0057 - Val Loss: 0.0041
Epoch 186/200 - Train Loss: 0.0057 - Val Loss: 0.0042
Epoch 187/200 - Train Loss: 0.0057 - Val Loss: 0.0041
Epoch 188/200 - Train Loss: 0.0057 - Val Loss: 0.0042
Epoch 189/200 - Train Loss: 0.0057 - Val Loss: 0.0041
Epoch 190/200 - Train Loss: 0.0056 - Val Loss: 0.0041
Epoch 191/200 - Train Loss: 0.0056 - Val Loss: 0.0041
Epoch 192/200 - Train Loss: 0.0056 - Val Loss: 0.0040
Epoch 193/200 - Train Loss: 0.0056 - Val Loss: 0.0040
Epoch 194/200 - Train Loss: 0.0056 - Val Loss: 0.0041
Epoch 195/200 - Train Loss: 0.0056 - Val Loss: 0.0040
Epoch 196/200 - Train Loss: 0.0056 - Val Loss: 0.0041
Epoch 197/200 - Train Loss: 0.0056 - Val Loss: 0.0040
Epoch 198/200 - Train Loss: 0.0055 - Val Loss: 0.0040
Epoch 199/200 - Train Loss: 0.0055 - Val Loss: 0.0040
Epoch 200/200 - Train Loss: 0.0055 - Val Loss: 0.0041

Plot the evolution of training

Task 3.2:

Use this block to plot how the losses evolve with epochs. Once again you can reuse code from before.

Add the plots to your report. Include only results from your final model (the one you are most satisfied with)

# Create a scatter plot with enhanced styling
plt.figure(figsize=(8, 6))  # Set the figure size

x_axis = list(range(len(train_loss_list)))

plt.scatter(x_axis, val_loss_list, label='Validation loss', color='red', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)
plt.scatter(x_axis, train_loss_list, label='Training loss', color='blue', marker='.', s=100, alpha=0.7, edgecolors='black', linewidths=0.5)

# Add labels and a legend with improved formatting
plt.xlabel('Epochs', fontsize=14, fontweight='bold')
plt.ylabel('Loss', fontsize=14, fontweight='bold')
plt.title('Loss curves', fontsize=16, fontweight='bold')
plt.legend(loc='upper right', fontsize=12)

# Set the y-axis to be logarithmic
plt.yscale('log')

# Customize the grid appearance
plt.grid(True, linestyle='--', alpha=0.5)

# Customize the tick labels
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)

# Add a background color to the plot
plt.gca().set_facecolor('#f2f2f2')
plt.show()

Evaluate the new model with parity plots

Task 3.3:

Use this block to make parity plots for your new model for both training and validation datasets. Once again you can reuse code from before.

Add the plots to your report. Include only results from your final model (the one you are most satisfied with)

y_train = scaler_t.inverse_transform(model.predict(normalized_X_train).reshape(-1,1))
y_val = scaler_t.inverse_transform(model.predict(normalized_X_val).reshape(-1,1))

fig,axes = plt.subplots(1,2,figsize=(8,3))

axes[0].scatter(t_train,y_train,s=0.5)

axes[0].set_title('Training dataset')
axes[0].set_xlabel('target crack location [m]')
axes[0].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_train), np.min(y_train))
max_val = max(np.max(t_train), np.max(y_train))
axes[0].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[0].legend()

# axes[1].scatter(YOUR_CODE_HERE,YOUR_CODE_HERE,s=0.5)
# Solution
axes[1].scatter(t_val,y_val,s=0.5)

axes[1].set_title('Validation dataset')
axes[1].set_xlabel('target crack location [m]')
axes[1].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_val), np.min(y_val))
max_val = max(np.max(t_val), np.max(y_val))
axes[1].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[1].legend()

plt.tight_layout()

Questions:

3.1) Based on the shape of the loss curves, what can you indicate about the fitting capabilities of the model? (Is it overfitting, underfitting, or neither?)

3.2) What criterion did you use to measure the quality of your model when trying out different architectures?

3.3) How well does your final model do compared to the one in Part 2? Use the parity plots you obtained to make your argument. Can you give a physical explanation for why this is the case?

3.4) Can you propose explanations for the errors that still remain?

4. Reaching the best possible model through hyperparameter tuning

In the previous part you performed a quick model selection procedure by hand, manually adjusting the layer sizes and activation function until you got to a model that looked good. It is however much more efficient and robust to do this automatically, which is our goal for this part.

Task 4.1:

Create a grid-search strategy to find hyperparameters that give the best prediction on the validation set. Vary the number of layers and number of hidden units per layer. You can assume that all the hidden layers have the same number of hidden units.

Based on your trials in Part 3, set up lists for layer_sizes and layer_numbers to create a grid of models to be tested. You do not need to code anything else. Do not be too ambitious with your setup, the number of models to be trained is len(layer_sizes)*len(layer_numbers), so if you use too many combinations the block will take a while to run!

Note that the activation function is fixed to tanh right now, but also try out relu and logistic and keep the one that gives the best results.

On the second block we plot the final validation losses of all the models on a grid as a kind of heatmap. Add this plot to your report.

layer_sizes = [10, 20] 
layer_numbers = [1, 2, 3, 4, 5, 6]

# Create a grid for the coordinate pairs and store them in an array
val_loss_grid = np.zeros((len(layer_sizes), len(layer_numbers)))

# Loop over all the layer sizes
for i, lsize in enumerate(layer_sizes):
    
    # Loop over all numbers of hidden layers
    for j, lnumber in enumerate(layer_numbers):
    
        # get tuple of hidden layer sizes
        layers = (lsize,) * lnumber
        print("Training NN with hidden layers:  {}".format(layers))
        
        # Create the ANN model with the given hidden layer sizes and activation function
        # Fix random_state to make sure results are reproducible
        model = MLPRegressor(hidden_layer_sizes=layers, activation='relu', random_state=0)
        
        _,  val_loss_list = train_model(model, 
                                        normalized_X_train, 
                                        normalized_t_train,
                                        normalized_X_val, 
                                        normalized_t_val, 
                                        n_epochs=500, 
                                        batch_size=64,
                                        learning_rate=0.001,
                                        verbose=False
                                        )
    
        val_loss_grid[i,j] = val_loss_list[-1]
        
        print("     Final validation loss:    {:.4e}\n".format(val_loss_grid[i,j]))


# Extract the hyperparameters that gave the lowest loss and print
min_size, min_number = np.unravel_index(np.argmin(val_loss_grid), val_loss_grid.shape)
print("\n\nModel with {} layers and {} neurons per layer gave lowest loss of {:.4e}".format(layer_numbers[min_number], layer_sizes[min_size], val_loss_grid[min_size, min_number]))

Training NN with hidden layers:  (10,)
     Final validation loss:    4.7889e-03

Training NN with hidden layers:  (10, 10)
     Final validation loss:    6.9074e-04

Training NN with hidden layers:  (10, 10, 10)
     Final validation loss:    4.8752e-04

Training NN with hidden layers:  (10, 10, 10, 10)
     Final validation loss:    5.6915e-04

Training NN with hidden layers:  (10, 10, 10, 10, 10)
     Final validation loss:    4.7641e-04

Training NN with hidden layers:  (10, 10, 10, 10, 10, 10)
     Final validation loss:    5.0436e-04

Training NN with hidden layers:  (20,)
     Final validation loss:    3.5826e-03

Training NN with hidden layers:  (20, 20)
     Final validation loss:    4.9451e-04

Training NN with hidden layers:  (20, 20, 20)
     Final validation loss:    4.7035e-04

Training NN with hidden layers:  (20, 20, 20, 20)
     Final validation loss:    5.6383e-04

Training NN with hidden layers:  (20, 20, 20, 20, 20)
     Final validation loss:    4.9324e-04

Training NN with hidden layers:  (20, 20, 20, 20, 20, 20)
     Final validation loss:    4.6722e-04



Model with 6 layers and 20 neurons per layer gave lowest loss of 4.6722e-04

# Define the row and column labels
rows = layer_sizes
cols = layer_numbers

plt.figure(figsize=(10, 10))
plt.imshow(val_loss_grid, cmap='jet', interpolation='nearest')

# Add a colorbar
plt.colorbar(label='Validation Loss')

# Add the row and column labels
plt.xticks(range(len(cols)), cols)
plt.yticks(range(len(rows)), rows)

plt.xlabel('Number of Layers')
plt.ylabel('Number of Neurons')

plt.title('Validation Loss Grid')
plt.show()

This code will create a heatmap where the color intensity represents the validation loss. The colorbar on the side provides a reference for the loss values. The row and column labels represent the number of neurons and layers, respectively.

Task 4.2:

It is finally time to plot the performance of our best model in detail! In the next block we set up and train again the best model found by the grid search procedure.

Your task is to use this trained model to produce parity plots for training and validation, like the ones from Parts 2 and 3. Use the code you already developed!

But now we can also finally take a look at the test dataset that we never touched (we did not use it for training nor for selecting the architecture). The error on the test set should serve as the final check for your model, on data that the model has really never seen, directly or indirectly.

Add these plots in your report.

# Normalize the test inputs
normalized_X_test = scaler_x.transform(X_test)

# Set up NN
# Fix random_state=0 to make sure this is consistent with the model in the loop above
layers = (layer_sizes[min_size],) * layer_numbers[min_number]
model = MLPRegressor(hidden_layer_sizes=layers, activation='tanh', random_state=0)

# train NN
_,  val_loss_list = train_model(model, 
                                normalized_X_train, 
                                normalized_t_train,
                                normalized_X_val, 
                                normalized_t_val, 
                                n_epochs=500, 
                                batch_size=64,
                                learning_rate=0.001
                                )

# YOUR_CODE_HERE

#Solution
y_train = scaler_t.inverse_transform(model.predict(normalized_X_train).reshape(-1,1))
y_val = scaler_t.inverse_transform(model.predict(normalized_X_val).reshape(-1,1))
y_test = scaler_t.inverse_transform(model.predict(normalized_X_test).reshape(-1,1))

fig,axes = plt.subplots(1,3,figsize=(10,3))

axes[0].scatter(t_train,y_train,s=0.5)

axes[0].set_title('Training dataset')
axes[0].set_xlabel('target crack location [m]')
axes[0].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_train), np.min(y_train))
max_val = max(np.max(t_train), np.max(y_train))
axes[0].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[0].legend()

axes[1].scatter(t_val,y_val,s=0.5)

axes[1].set_title('Validation dataset')
axes[1].set_xlabel('target crack location [m]')
axes[1].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_val), np.min(y_val))
max_val = max(np.max(t_val), np.max(y_val))
axes[1].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[1].legend()

axes[2].scatter(t_test,y_test,s=0.5)

axes[2].set_title('Test dataset')
axes[2].set_xlabel('target crack location [m]')
axes[2].set_ylabel('predicted crack location [m]')

min_val = min(np.min(t_test), np.min(y_test))
max_val = max(np.max(t_test), np.max(y_test))
axes[2].plot([min_val, max_val], [min_val, max_val], 'k--', label='Ideal Fit', alpha=0.7)
axes[2].legend()

plt.tight_layout()

Epoch 1/500 - Train Loss: 0.1664 - Val Loss: 0.1845
Epoch 2/500 - Train Loss: 0.1592 - Val Loss: 0.1527
Epoch 3/500 - Train Loss: 0.0719 - Val Loss: 0.0745
Epoch 4/500 - Train Loss: 0.0708 - Val Loss: 0.0775
Epoch 5/500 - Train Loss: 0.0538 - Val Loss: 0.0570
Epoch 6/500 - Train Loss: 0.0384 - Val Loss: 0.0404
Epoch 7/500 - Train Loss: 0.0244 - Val Loss: 0.0249
Epoch 8/500 - Train Loss: 0.0164 - Val Loss: 0.0152
Epoch 9/500 - Train Loss: 0.0152 - Val Loss: 0.0139
Epoch 10/500 - Train Loss: 0.0147 - Val Loss: 0.0132
Epoch 11/500 - Train Loss: 0.0143 - Val Loss: 0.0131
Epoch 12/500 - Train Loss: 0.0140 - Val Loss: 0.0128
Epoch 13/500 - Train Loss: 0.0138 - Val Loss: 0.0126
Epoch 14/500 - Train Loss: 0.0134 - Val Loss: 0.0120
Epoch 15/500 - Train Loss: 0.0131 - Val Loss: 0.0117
Epoch 16/500 - Train Loss: 0.0129 - Val Loss: 0.0116
Epoch 17/500 - Train Loss: 0.0127 - Val Loss: 0.0112
Epoch 18/500 - Train Loss: 0.0121 - Val Loss: 0.0107
Epoch 19/500 - Train Loss: 0.0121 - Val Loss: 0.0106
Epoch 20/500 - Train Loss: 0.0115 - Val Loss: 0.0101
Epoch 21/500 - Train Loss: 0.0113 - Val Loss: 0.0100
Epoch 22/500 - Train Loss: 0.0109 - Val Loss: 0.0096
Epoch 23/500 - Train Loss: 0.0102 - Val Loss: 0.0088
Epoch 24/500 - Train Loss: 0.0107 - Val Loss: 0.0094
Epoch 25/500 - Train Loss: 0.0095 - Val Loss: 0.0082
Epoch 26/500 - Train Loss: 0.0091 - Val Loss: 0.0078
Epoch 27/500 - Train Loss: 0.0092 - Val Loss: 0.0080
Epoch 28/500 - Train Loss: 0.0085 - Val Loss: 0.0073
Epoch 29/500 - Train Loss: 0.0082 - Val Loss: 0.0071
Epoch 30/500 - Train Loss: 0.0081 - Val Loss: 0.0069
Epoch 31/500 - Train Loss: 0.0076 - Val Loss: 0.0065
Epoch 32/500 - Train Loss: 0.0073 - Val Loss: 0.0063
Epoch 33/500 - Train Loss: 0.0071 - Val Loss: 0.0061
Epoch 34/500 - Train Loss: 0.0069 - Val Loss: 0.0059
Epoch 35/500 - Train Loss: 0.0073 - Val Loss: 0.0064
Epoch 36/500 - Train Loss: 0.0071 - Val Loss: 0.0061
Epoch 37/500 - Train Loss: 0.0064 - Val Loss: 0.0055
Epoch 38/500 - Train Loss: 0.0063 - Val Loss: 0.0054
Epoch 39/500 - Train Loss: 0.0062 - Val Loss: 0.0053
Epoch 40/500 - Train Loss: 0.0064 - Val Loss: 0.0054
Epoch 41/500 - Train Loss: 0.0064 - Val Loss: 0.0056
Epoch 42/500 - Train Loss: 0.0060 - Val Loss: 0.0051
Epoch 43/500 - Train Loss: 0.0059 - Val Loss: 0.0051
Epoch 44/500 - Train Loss: 0.0059 - Val Loss: 0.0050
Epoch 45/500 - Train Loss: 0.0058 - Val Loss: 0.0050
Epoch 46/500 - Train Loss: 0.0058 - Val Loss: 0.0050
Epoch 47/500 - Train Loss: 0.0057 - Val Loss: 0.0049
Epoch 48/500 - Train Loss: 0.0057 - Val Loss: 0.0048
Epoch 49/500 - Train Loss: 0.0056 - Val Loss: 0.0047
Epoch 50/500 - Train Loss: 0.0056 - Val Loss: 0.0049
Epoch 51/500 - Train Loss: 0.0057 - Val Loss: 0.0049
Epoch 52/500 - Train Loss: 0.0055 - Val Loss: 0.0046
Epoch 53/500 - Train Loss: 0.0055 - Val Loss: 0.0047
Epoch 54/500 - Train Loss: 0.0057 - Val Loss: 0.0047
Epoch 55/500 - Train Loss: 0.0054 - Val Loss: 0.0046
Epoch 56/500 - Train Loss: 0.0054 - Val Loss: 0.0046
Epoch 57/500 - Train Loss: 0.0059 - Val Loss: 0.0052
Epoch 58/500 - Train Loss: 0.0051 - Val Loss: 0.0043
Epoch 59/500 - Train Loss: 0.0051 - Val Loss: 0.0043
Epoch 60/500 - Train Loss: 0.0051 - Val Loss: 0.0043
Epoch 61/500 - Train Loss: 0.0049 - Val Loss: 0.0041
Epoch 62/500 - Train Loss: 0.0051 - Val Loss: 0.0042
Epoch 63/500 - Train Loss: 0.0048 - Val Loss: 0.0039
Epoch 64/500 - Train Loss: 0.0047 - Val Loss: 0.0039
Epoch 65/500 - Train Loss: 0.0048 - Val Loss: 0.0041
Epoch 66/500 - Train Loss: 0.0054 - Val Loss: 0.0047
Epoch 67/500 - Train Loss: 0.0047 - Val Loss: 0.0040
Epoch 68/500 - Train Loss: 0.0044 - Val Loss: 0.0036
Epoch 69/500 - Train Loss: 0.0043 - Val Loss: 0.0034
Epoch 70/500 - Train Loss: 0.0046 - Val Loss: 0.0037
Epoch 71/500 - Train Loss: 0.0041 - Val Loss: 0.0034
Epoch 72/500 - Train Loss: 0.0042 - Val Loss: 0.0033
Epoch 73/500 - Train Loss: 0.0038 - Val Loss: 0.0031
Epoch 74/500 - Train Loss: 0.0038 - Val Loss: 0.0030
Epoch 75/500 - Train Loss: 0.0037 - Val Loss: 0.0030
Epoch 76/500 - Train Loss: 0.0034 - Val Loss: 0.0028
Epoch 77/500 - Train Loss: 0.0033 - Val Loss: 0.0026
Epoch 78/500 - Train Loss: 0.0032 - Val Loss: 0.0025
Epoch 79/500 - Train Loss: 0.0030 - Val Loss: 0.0023
Epoch 80/500 - Train Loss: 0.0029 - Val Loss: 0.0022
Epoch 81/500 - Train Loss: 0.0027 - Val Loss: 0.0021
Epoch 82/500 - Train Loss: 0.0026 - Val Loss: 0.0019
Epoch 83/500 - Train Loss: 0.0025 - Val Loss: 0.0019
Epoch 84/500 - Train Loss: 0.0024 - Val Loss: 0.0019
Epoch 85/500 - Train Loss: 0.0022 - Val Loss: 0.0016
Epoch 86/500 - Train Loss: 0.0022 - Val Loss: 0.0016
Epoch 87/500 - Train Loss: 0.0020 - Val Loss: 0.0013
Epoch 88/500 - Train Loss: 0.0018 - Val Loss: 0.0013
Epoch 89/500 - Train Loss: 0.0017 - Val Loss: 0.0013
Epoch 90/500 - Train Loss: 0.0016 - Val Loss: 0.0011
Epoch 91/500 - Train Loss: 0.0015 - Val Loss: 0.0010
Epoch 92/500 - Train Loss: 0.0014 - Val Loss: 0.0010
Epoch 93/500 - Train Loss: 0.0013 - Val Loss: 0.0010
Epoch 94/500 - Train Loss: 0.0013 - Val Loss: 0.0009
Epoch 95/500 - Train Loss: 0.0012 - Val Loss: 0.0009
Epoch 96/500 - Train Loss: 0.0012 - Val Loss: 0.0009
Epoch 97/500 - Train Loss: 0.0011 - Val Loss: 0.0009
Epoch 98/500 - Train Loss: 0.0011 - Val Loss: 0.0008
Epoch 99/500 - Train Loss: 0.0011 - Val Loss: 0.0008
Epoch 100/500 - Train Loss: 0.0011 - Val Loss: 0.0008
Epoch 101/500 - Train Loss: 0.0011 - Val Loss: 0.0009
Epoch 102/500 - Train Loss: 0.0010 - Val Loss: 0.0008
Epoch 103/500 - Train Loss: 0.0010 - Val Loss: 0.0008
Epoch 104/500 - Train Loss: 0.0010 - Val Loss: 0.0009
Epoch 105/500 - Train Loss: 0.0010 - Val Loss: 0.0008
Epoch 106/500 - Train Loss: 0.0010 - Val Loss: 0.0008
Epoch 107/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 108/500 - Train Loss: 0.0010 - Val Loss: 0.0008
Epoch 109/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 110/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 111/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 112/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 113/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 114/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 115/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 116/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 117/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 118/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 119/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 120/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 121/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 122/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 123/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 124/500 - Train Loss: 0.0009 - Val Loss: 0.0007
Epoch 125/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 126/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 127/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 128/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 129/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 130/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 131/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 132/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 133/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 134/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 135/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 136/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 137/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 138/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 139/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 140/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 141/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 142/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 143/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 144/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 145/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 146/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 147/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 148/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 149/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 150/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 151/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 152/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 153/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 154/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 155/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 156/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 157/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 158/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 159/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 160/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 161/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 162/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 163/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 164/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 165/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 166/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 167/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 168/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 169/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 170/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 171/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 172/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 173/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 174/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 175/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 176/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 177/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 178/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 179/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 180/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 181/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 182/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 183/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 184/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 185/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 186/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 187/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 188/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 189/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 190/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 191/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 192/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 193/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 194/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 195/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 196/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 197/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 198/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 199/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 200/500 - Train Loss: 0.0009 - Val Loss: 0.0009
Epoch 201/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 202/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 203/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 204/500 - Train Loss: 0.0008 - Val Loss: 0.0006
Epoch 205/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 206/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 207/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 208/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 209/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 210/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 211/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 212/500 - Train Loss: 0.0007 - Val Loss: 0.0005
Epoch 213/500 - Train Loss: 0.0007 - Val Loss: 0.0005
Epoch 214/500 - Train Loss: 0.0007 - Val Loss: 0.0005
Epoch 215/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 216/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 217/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 218/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 219/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 220/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 221/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 222/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 223/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 224/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 225/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 226/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 227/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 228/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 229/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 230/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 231/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 232/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 233/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 234/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 235/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 236/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 237/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 238/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 239/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 240/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 241/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 242/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 243/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 244/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 245/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 246/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 247/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 248/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 249/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 250/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 251/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 252/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 253/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 254/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 255/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 256/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 257/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 258/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 259/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 260/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 261/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 262/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 263/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 264/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 265/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 266/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 267/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 268/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 269/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 270/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 271/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 272/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 273/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 274/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 275/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 276/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 277/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 278/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 279/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 280/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 281/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 282/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 283/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 284/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 285/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 286/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 287/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 288/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 289/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 290/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 291/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 292/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 293/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 294/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 295/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 296/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 297/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 298/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 299/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 300/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 301/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 302/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 303/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 304/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 305/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 306/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 307/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 308/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 309/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 310/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 311/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 312/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 313/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 314/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 315/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 316/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 317/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 318/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 319/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 320/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 321/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 322/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 323/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 324/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 325/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 326/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 327/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 328/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 329/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 330/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 331/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 332/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 333/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 334/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 335/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 336/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 337/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 338/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 339/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 340/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 341/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 342/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 343/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 344/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 345/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 346/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 347/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 348/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 349/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 350/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 351/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 352/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 353/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 354/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 355/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 356/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 357/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 358/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 359/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 360/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 361/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 362/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 363/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 364/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 365/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 366/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 367/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 368/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 369/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 370/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 371/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 372/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 373/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 374/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 375/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 376/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 377/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 378/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 379/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 380/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 381/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 382/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 383/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 384/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 385/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 386/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 387/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 388/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 389/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 390/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 391/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 392/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 393/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 394/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 395/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 396/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 397/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 398/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 399/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 400/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 401/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 402/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 403/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 404/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 405/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 406/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 407/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 408/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 409/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 410/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 411/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 412/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 413/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 414/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 415/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 416/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 417/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 418/500 - Train Loss: 0.0007 - Val Loss: 0.0008
Epoch 419/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 420/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 421/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 422/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 423/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 424/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 425/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 426/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 427/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 428/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 429/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 430/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 431/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 432/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 433/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 434/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 435/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 436/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 437/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 438/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 439/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 440/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 441/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 442/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 443/500 - Train Loss: 0.0009 - Val Loss: 0.0008
Epoch 444/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 445/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 446/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 447/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 448/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 449/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 450/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 451/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 452/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 453/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 454/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 455/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 456/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 457/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 458/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 459/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 460/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 461/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 462/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 463/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 464/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 465/500 - Train Loss: 0.0008 - Val Loss: 0.0007
Epoch 466/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 467/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 468/500 - Train Loss: 0.0008 - Val Loss: 0.0008
Epoch 469/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 470/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 471/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 472/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 473/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 474/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 475/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 476/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 477/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 478/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 479/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 480/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 481/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 482/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 483/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 484/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 485/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 486/500 - Train Loss: 0.0010 - Val Loss: 0.0009
Epoch 487/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 488/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 489/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 490/500 - Train Loss: 0.0006 - Val Loss: 0.0006
Epoch 491/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 492/500 - Train Loss: 0.0007 - Val Loss: 0.0007
Epoch 493/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 494/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 495/500 - Train Loss: 0.0007 - Val Loss: 0.0006
Epoch 496/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 497/500 - Train Loss: 0.0005 - Val Loss: 0.0005
Epoch 498/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 499/500 - Train Loss: 0.0006 - Val Loss: 0.0005
Epoch 500/500 - Train Loss: 0.0006 - Val Loss: 0.0006

Questions:

4.1) How does hyperparameter tuning in machine learning relate to the concept of model complexity?

4.2) Given a comprehensive list of layer sizes and numbers, and given a relatively small training dataset, we expect the top left corner of the heatmap to have high validation errors. Why is that?

4.3) Following up on the previous question, we also expect the bottom right corner of the heatmap to have high validation errors. Why is that?

4.4) How does the performance of your final model for this part compare with the one you tweaked manually?

End of notebook.

© Copyright 2023 MUDE Teaching Team TU Delft. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.