Visualize activations functions using keras

December 22, 2017

This post was kindly contributed by SAS Programming for Data Mining - go there to comment and to read the full post.

In keras, we can visualize activation functions‘ geometric properties using backend functions over layers of a model.

We all know the exact function of popular activation functions such as ‘sigmoid‘, ‘tanh‘, ‘relu‘, etc, and we can feed data to these functions to directly obtain their output. But how to do that via keras without explicitly specifying their functional forms?

This can be done following the four steps below:

1. define a simple MLP model with a one dimension input data, a one neuron dense network as the hidden layer, and the output layer will have a ‘linear‘ activation function for one neuron.
2. Extract layers’ output of the model (fitted or not) via iterating through model.layers
3. Using backend function K.function() to obtain calculated output for a given input data
4. Feed desired data to the above functions to obtain the output from appropriate activation function.

The code below is a demo:

from keras.layers import Dense, Activation
from keras.models import Sequential
import keras.backend as K
import numpy as np
import matplotlib.pyplot as plt

def NNmodel(activationFunc='linear'):
Define a neural network which can be arbitrary but we keep it simple here
model = Sequential()
model.add(Dense(1, input_shape=(1,), activation=activationFunc))

model.add(Dense(1, activation='linear'))
model.compile(loss='mse', optimizer='adagrad', metrics=['mse'])
return model

def VisualActivation(activationFunc='relu', plot=True):
x = (np.arange(100)-50)/10
y = np.log(x+51/10)

model = NNmodel(activationFunc = activationFunc)
# optional to fit the model. If not fitted, initialized parameters will be used, x, epochs=1, batch_size=128, verbose=0)

# define the computing process
inX = model.input
outputs = [layer.output for layer in model.layers]
functions = [K.function([inX], [out]) for out in outputs]

# compute based on original inputs
for i in range(100):
test = x[i].reshape(-1, 1)
layer_outs = [func([test]) for func in functions]
activationLayer[i] = layer_outs[0][0][0][0]

# process results
activationDf = pd.DataFrame.from_dict(activationLayer, orient='index')
result=pd.concat([pd.DataFrame(x), activationDf], axis=1)
result.columns=['X', 'Activated']
result.set_index('X', inplace=True)
if plot:

return result

# Now we can visualize them (assuming default settings) :
actFuncs = ['sigmoid', 'tanh', 'hard_sigmoid', 'softplus', 'selu', 'elu']

figure = plt.figure()
for i, f in enumerate(actFuncs):
figure.add_subplot(3, 2, i+1)
out=VisualActivation(activationFunc=f, plot=False)
plt.plot(out.index, out.Activated)

This figure is the output from above code. As we can see, the geometric property of each activation function is well captured.

This post was kindly contributed by SAS Programming for Data Mining - go there to comment and to read the full post.

Tags: , ,

Welcome! offers news and tutorials about the various SAS® software packages, contributed by bloggers. You are welcome to subscribe to e-mail updates, or add your SAS-blog to the site.


Dear readers, proc-x is looking for sponsors who would be willing to support the site in exchange for banner ads in the right sidebar of the site. If you are interested, please e-mail me at:
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.