Visualizing Convolutional Nets for NLU Tasks

With deep learning becoming the foundation for Talentpair’s matching algorithm, we wanted to get a better understanding how our convolutional neural networks make the decisions they do. Google’s TensorBoard has been a tremendous help to achieve our goal (read more about Talentpair’s deep learning stack here). Unfortunately, due to our model setup, we couldn’t take advantage of the TensorBoard output visualizations (we developed a deep-siamese network to model document similarities).

Ever since Talentpair’s machine learning team attended NIPS in 2017, we wanted to dig deeper into our networks internals and learn about its decision making process. NIPS 2017 focused on the interpretability of machine learning, and hosted workshops for its participants. We also felt that understanding and visualizing the internals of neural networks might be the only way to make networks compliant with the upcoming General Data Protection Regulation (GDPR) by the European Union.

Research and examples in the deep learning community have largely focused on visualizing internals of networks for computer vision problems, e.g. http://www.heatmapping.org/. But we haven‘t found similar projects for NLU tasks.

We looked at the implementation of layer visualizations for computer vision problems and extended the solutions for the NLU space. In this blog article, we’ll guide you through the steps to visualize your convolutions nets for your NLU tasks.

The heatmaps for the network layers will visualize the layer activations and how they contribute to the overall network prediction. Therefore, we’ll calculate the gradient between the layer we want to investigate and the overall output of the network (e.g. the class prediction). Most heatmap examples from the computer vision space use the class activation as a basis for a heatmap (e.g. Francois Chollet’s example in Deep Learning with Python), but we have tested the heatmap examples below on networks predicting something other than class probabilities: vector representations.

To show how you can go from a trained model to producing a heatmap for your NLU tasks, we have broken down the process into 7 steps and provided you a complete Jupyter notebook to follow the steps. First, we’ll introduce a demo Keras model for our document classifications. The model was trained to label documents into 20 newsgroup categories. Then, we’ll show you how you can access the individual layers of the trained Keras model. We’ll extend Francois Chollet’s example of creating an activation heatmap and applying it to our NLU problem. And last, we’ll guide your through the steps of turning the activation heatmap into a human interpretable visualization.

Starting point

To demonstrate the visualizations, we have developed a generic convolutional network to classify documents and trained it with the newsgroup-20 data set. You can follow the code in our demonstration notebook.

The demo model is pretty straight forward: We implemented two convolutional layers and classified the newsgroup category via dense layer.

sequence_input = Input(shape=(maxlen_text,))

x = Embedding(name='embedding_layer',
              input_dim=max_words_to_keep,
              output_dim=token_vec_size,
              input_length=maxlen_text)(sequence_input)
x = Dropout(.20)(x)

x = Conv1D(64, 5, activation='relu', name='1-conv1d', padding='same')(x)
x = MaxPooling1D(pool_size=4)(x)
x = Dropout(.20)(x)

x = Conv1D(64, 5, activation='relu', name='2-conv1d', padding='same')(x)
x = MaxPooling1D(pool_size=2)(x)
x = Dropout(.20)(x)

x = Flatten()(x)

output = Dense(units=output_dim,activation='softmax')(x)

Exporting the layer you want to investigate and its output

Before we can create our activation heatmap, we need to access the trained network layer and “extract” it from the neural network. To do that, we created this little helper function which accepts any Keras model and layer name as input arguments and returns the requested layer as well as its output vector dimensions. Since we are dealing with one dimensional data (list of document tokens), we are only interested in the vector length of the layer output. In contrast, heatmaps for image convolutions will have an x and y component.

def get_conv_layer(model, layer_name):
    conv_layer = model.get_layer(layer_name)
    output_dim = conv_layer.output_shape[1]
    return conv_layer, output_dim

Creating an Activation Heatmap

The core concept of the heatmap creation is the gradient calculation between the layer, we want to investigate, and the final output of the neural network. We are calculating which of the tokens presented to the network pushed the output of the entire network into one or the direction. In other words, we want to determine how much each token contributed the most our final prediction.

Before we dive into the function to generate the heatmap, let’s take a brief look at our tokens. Our initial network input is a document based on a maximum of 300 tokens. Longer documents get truncated, shorter documents get zero-padded to fit the expected 300 token length. Depending on which layer we’ll want to visualize, the input data has been manipulated by various convolution and pooling layers. Therefore, the layer we want to investigate may not receive 300 tokens for each of our training examples, but e.g. 75 tokens if we want to visualize the 2rd convolutional layer. This is also the reason, why the heatmap is a bit fuzzy. But more about this later.

The function get_heatmap is the core of the visualization. On the first look, the function looks confusing since it references various Keras backend functions (functions which are executed on the Tensorflow, Theano or MXNet level). But the function does basically the following steps:

1. Obtain the probability of labels with the highest certainty from the final output of the network
2. Obtain the output of the convolutional layer we want to visualize
3. Calculate the gradients between the network output and the layer
4. Average the gradients across all of the convolution filters (as well as the visualization samples, in our case: one sample)
5. Using one document evaluate the network graph up to the the gradient calculation and the overall network output, separately
6. Loop over every dimension of the layer output vector and multiply the layer output with the gradients element-wise

def get_heatmap(model, layer_name, matrix, y_labels):

    # obtain probability of the label with the highest certainty
    network_output = model.get_output_at(0)[:, np.argmax(y_labels)]
    # obtain the output vector and its dimension of the convolutional layer we want to visualize
    conv_layer, layer_output_dim = get_conv_layer(model, layer_name)

    # Setting up the calculation of the gradients between the output and the conv layer. Will be executed in the iteration step
    grads = K.gradients(network_output, conv_layer.output)[0]
    # average the gradients across our samples (one sample) and all filters
    pooled_grads = K.mean(grads, axis=(0, 2))

    # set up the computation graph
    iterate = K.function([model.get_input_at(0)], [pooled_grads, conv_layer.output[0]])
    # execute the computation graph with our converted document (matrix) as an input
    pooled_grad_value, conv_layer_output_value = iterate([matrix])

    # loop over every layer output vector element and multiply it by the gradient of the element
    for i in range(layer_output_dim):
        conv_layer_output_value[i] *= pooled_grad_value[i]

    # calculating the average output value for each output dimension across all filters
    heatmap = np.mean(conv_layer_output_value, axis=-1)
    return norm_heatmap(heatmap)

For easier plotting of the heatmap, let’s normalize the map. This step is optional since matplotlib would do it for us as well, but the normalization will become handy when color-code the tokens of interest.

def norm_heatmap(heatmap):
    # element-wise maximum calculation, basically setting all negative values to zero
    heatmap = np.maximum(heatmap, 0)
    # normalizing the heatmap to values between 0 and 1
    norm_heatmap = heatmap / np.max(heatmap)
    return norm_heatmap

The previous steps allow us to create two visualization:

• A heatmap for the layer output vector, which will show you areas of interest for the final network output
• Overlay the generated heatmap with the original network input to highlight which words in the documents have been of interest

Generate the visual heatmap

While the first visualization is straightforward, we’ll need to do some more data wrangling to overlay the heatmap with the original input. More on that later. To generate a visual heatmap, we can plot the vector with matplotlib. A vector that is only 1 dimension tall won’t be very visible so we’ll stack the same vector a few times on top of each other. The variable repeat_vector_n_times is just a little helper to set the high of the heatmap dynamically. The length of the inspected vectors changes with every layer. We’ll calculate the ratio between height and width (in our case the dimension of the output of the layer) dynamically.

def plot_heatmap(heatmap, height_ratio=0.05):
        # calculating how often the vector should be repeated to display a height relative to the vector length
        repeat_vector_n_times = int(heatmap.shape[0] * height_ratio)
        plt.matshow([heatmap] * repeat_vector_n_times)

The function above will generate heatmaps like below.

Overlay the heatmap with the original document

Before we overlay the heatmap with the original text, let’s define two helper functions. These functions will handle the coloring of the tokens. They also assume this example is run in a Jupyter notebook. In this case, iPython’s display module contains a handy HTML object, which let’s you set a custom <text> tag.

def cstr(s, color='black'):
    return "<text style=\"color:{}\">{}</text>".format(color, s)

The function color below takes a list of color names as an input (in reversed order, starting with the color for highest value) and returns the color name for the given heatmap value called hvalue. This provides a way to drop out the noise and pick out and segment the most valuable info. The hvalue needs to be above the given threshold to get color coded.

def color(hvalue, threshold, max=1, cdefault='black', colors=['red', 'yellow', 'green', 'cyan', 'blue']):
    num_colors = len(colors)
    if hvalue < threshold:
        return cdefault
    for i, color in enumerate(colors):
        if hvalue > (max - (max - threshold) / num_colors * (i + 1)):
            return color

Determine tokens of interest

Once we calculated the heatmap of the conv layer output, we can “stretch it” and overlay it with the original text aka the input tokens. We need to stretch the heatmap, because the conv layer has a length of 75, but we have 300 input tokens.

The function get_token_indices calculates the original indices of the input tokens. The function re-maps the indices of the heatmap back to the indices of the input tokens and saves the heatmap value to the index of the input token. We have added some additional functionality: saving always the highest heatmap value, and dynamically changing how far we stretch the indices depending on the ratio between conv layer output dimensions and length of the input vector.

def get_token_indices(model, layer_name, threshold, matrix, y_labels):
    heatmap = get_heatmap(model=model, layer_name=layer_name, matrix=matrix, y_labels=y_labels)
    _, output_dim = get_conv_layer(model, layer_name)

    # depending on the ration between the input and layer output shape, we need to calculate
    # how many original tokens have contributed to the layer output
    dim_ratio = matrix.shape[1] / output_dim
    if dim_ratio < 1.5:
        window_size = 1
    else:
        window_size = 2

    indices = {}
    indices_above_threshold = np.where(heatmap > threshold)[0].tolist()
    for i in indices_above_threshold:
        scaled_index = i * int(dim_ratio)
        for ind in range(scaled_index - window_size, scaled_index + window_size + 1):
            if ind not in indices or indices[ind] < heatmap[i]:
                indices.update({ind: heatmap[i]})
    return indices

Highlighting the tokens of interest

Once we determined which input tokens are of interest, we then highlight them using our previously defined helper function color. The function then will print all colored tokens via the html_print function. Remember this function will only work as is in a Jupyter notebook.

def get_highlighted_tokens(tokens, matrix, model, layer_name, threshold, y_labels):
    indices = get_token_indices(model, layer_name, threshold, matrix, y_labels)

    ctokens = []
    for i, t in enumerate(tokens):
        if i in indices.keys():
            _color = color(indices[i], threshold=threshold)
            ctokens.append(cstr(t, color=_color))
        else:
            ctokens.append(t)
    return html_print(cstr(' '.join(ctokens), color='black') )

Put everything together

After all the setup, let’s try out the visualization! In the demo notebook, we visualized the output of the second convolutional layer. The sample text is classified to be in the science/space category and the highlighted tokens support the classification. The area of the highest interest includes words like national, air, and space.

One comment about displaying the tokens. Since we tokenized the text, we lost any formatting. During the preprocessing steps, we also lost the any unknown tokens. That’s why the text is sometimes tricky to read.

We hope that this visualization demo is helpful understanding the internals of your neural networks for NLU tasks. If you have questions about the steps or suggestions how to simplify the steps, please leave us a comment.