How to exactly add L1 regularisation to tensorflow error function
24,532
You can use TensorFlow's apply_regularization and l1_regularizer methods. Note: this is for Tensorflow 1, and the API changed in Tensorflow 2, see edit below.
An example based on your question:
import tensorflow as tf
total_loss = meansq #or other loss calcuation
l1_regularizer = tf.contrib.layers.l1_regularizer(
scale=0.005, scope=None
)
weights = tf.trainable_variables() # all vars of your graph
regularization_penalty = tf.contrib.layers.apply_regularization(l1_regularizer, weights)
regularized_loss = total_loss + regularization_penalty # this loss needs to be minimized
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(regularized_loss)
Note: weights
is a list
where each entry is a tf.Variable
.
Edited: As Paddy correctly noted, in Tensorflow 2 they changed the API for regularizers. In Tensorflow 2, L1 regularization is described here.
Comments
-
Abhishek almost 2 years
Hey I am new to tensorflow and even after a lot of efforts could not add L1 regularisation term to the error term
x = tf.placeholder("float", [None, n_input]) # Weights and biases to hidden layer ae_Wh1 = tf.Variable(tf.random_uniform((n_input, n_hidden1), -1.0 / math.sqrt(n_input), 1.0 / math.sqrt(n_input))) ae_bh1 = tf.Variable(tf.zeros([n_hidden1])) ae_h1 = tf.nn.tanh(tf.matmul(x,ae_Wh1) + ae_bh1) ae_Wh2 = tf.Variable(tf.random_uniform((n_hidden1, n_hidden2), -1.0 / math.sqrt(n_hidden1), 1.0 / math.sqrt(n_hidden1))) ae_bh2 = tf.Variable(tf.zeros([n_hidden2])) ae_h2 = tf.nn.tanh(tf.matmul(ae_h1,ae_Wh2) + ae_bh2) ae_Wh3 = tf.transpose(ae_Wh2) ae_bh3 = tf.Variable(tf.zeros([n_hidden1])) ae_h1_O = tf.nn.tanh(tf.matmul(ae_h2,ae_Wh3) + ae_bh3) ae_Wh4 = tf.transpose(ae_Wh1) ae_bh4 = tf.Variable(tf.zeros([n_input])) ae_y_pred = tf.nn.tanh(tf.matmul(ae_h1_O,ae_Wh4) + ae_bh4) ae_y_actual = tf.placeholder("float", [None,n_input]) meansq = tf.reduce_mean(tf.square(ae_y_actual - ae_y_pred)) train_step = tf.train.GradientDescentOptimizer(0.05).minimize(meansq)
after this I run the above graph using
init = tf.initialize_all_variables() sess = tf.Session() sess.run(init) n_rounds = 100 batch_size = min(500, n_samp) for i in range(100): sample = np.random.randint(n_samp, size=batch_size) batch_xs = input_data[sample][:] batch_ys = output_data_ae[sample][:] sess.run(train_step, feed_dict={x: batch_xs, ae_y_actual:batch_ys})
Above is the code for a 4 layer autoencoder, "meansq" is my squared loss function. How can I add L1 reguarisation for the weight matrix (tensors) in the network?
-
Florian Koch about 7 yearsyour answer could be more helpful if you include a small code sample
-
ArtificiallyIntelligence over 6 yearsis tf.trainable_variables() also including biases??
-
Stefan over 6 yearsit should. tf.trainable_variables() returns a list of variables, so you can iterate over them to see whether the variable is actually in there. (see tensorflow.org/programmers_guide/variables)
-
ArtificiallyIntelligence over 6 yearsThe reason that I ask is that, usually, people don't regularize, as you see in many papers, simply weight is what is regularized.
-
Stefan over 6 yearsThat is a good remark, thanks. Biases are commonly not regularized. Also, commonly you don't apply L1 regularization to all your weights of the graph - the above code snippet should merely demonstrate the principle of how to use a regularize.
-
Vadim about 4 yearstf.contrib.layers.l1_regularizer does not available anymore