How to output a prediction in Tensorflow?

Question

I am trying to use a Tensorflow DNN for a Kaggle Competion. The data is about 100 columns of categorical data, 29 columns of numerical data, and 1 column for the output. What I did was I split it into training and testing with X and y using Scikit's train test split function, where X is a list of each rows without the "id" or the value that needs to be predicted, and y is the value that is needed to be predicted. I then built the model, shown below:

import tensorflow as tf
import numpy as np
import time
import pickle
with open('pickle.pickle', 'rb') as f:
    trainX, trainy, testX, testy = pickle.load(f)
trainX = np.array(trainX)
trainy = np.array(trainy)
trainy = trainy.reshape(trainy.shape[0], 1)
testX = np.array(testX)
testy = np.array(testy)
print (trainX.shape)
print (trainy.shape)
testX = testX.reshape(testX.shape[0], 130)
testy = testy.reshape(testy.shape[0], 1)
print (testX.shape)
print (testy.shape)
n_nodes_hl1 = 256
n_nodes_hl2 = 256
n_nodes_hl3 = 256

n_classes = 1

batch_size = 100


# Matrix = h X w
X = tf.placeholder('float', [None, len(trainX[0])])
y = tf.placeholder('float')

def model(data):

    hidden_1_layer = {'weights':tf.Variable(tf.random_normal([trainX.shape[1], n_nodes_hl1])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl1]))}

    hidden_2_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl2]))}

    hidden_3_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3])),
                      'biases':tf.Variable(tf.random_normal([n_nodes_hl3]))}

    output_layer = {'weights':tf.Variable(tf.random_normal([n_nodes_hl3, n_classes])),
                      'biases':tf.Variable(tf.random_normal([n_classes]))}

    # (input_data * weights) + biases

    l1 = tf.add(tf.matmul(data, hidden_1_layer['weights']), hidden_1_layer['biases'])
    l1 = tf.nn.sigmoid(l1)

    l2 = tf.add(tf.matmul(l1, hidden_2_layer['weights']), hidden_2_layer['biases'])
    l2 = tf.nn.sigmoid(l2)

    l3 = tf.add(tf.matmul(l2, hidden_3_layer['weights']), hidden_3_layer['biases'])
    l3 = tf.nn.sigmoid(l3)

    output = tf.matmul(l3, output_layer['weights']) + output_layer['biases']

    return output


def train(x):

    pred = model(x)
    #loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(pred, y))
    loss = tf.reduce_mean(tf.square(pred - y))
    optimizer = tf.train.AdamOptimizer(0.01).minimize(loss)

    epochs = 1

    with tf.Session() as sess:
        sess.run(tf.initialize_all_variables())
        print ('Beginning Training \n')
        for e in range(epochs):
            timeS = time.time()
            epoch_loss = 0

            i = 0
            while i < len(trainX):

                start = i
                end = i + batch_size
                batch_x = np.array(trainX[start:end])
                batch_y = np.array(trainy[start:end])

                _, c = sess.run([optimizer, loss], feed_dict = {x: batch_x, y: batch_y})
                epoch_loss += c
                i += batch_size
            done = time.time() - timeS
            print ('Epoch', e + 1, 'completed out of', epochs, 'loss:', epoch_loss, "\nTime:", done, 'seconds\n')
        correct = tf.equal(tf.arg_max(pred, 1), tf.arg_max(y, 1))
        acc = tf.reduce_mean(tf.cast(correct, 'float'))
        print("Accuracy:", acc.eval({x:testX, y:testy}))


train(X)

Output for 1 epoch:

Epoch 1 completed out of 1 loss: 1498498282.5 
Time: 1.3765859603881836 seconds

Accuracy: 1.0

I do realize that the loss is very high, and I am using 1 epoch just for testing purposes, and yes, I know my code is quite messy. But all I want to do is print out a prediction. How would I do that? I know that I need to feed a list of features for X, but I just don't understand how to do it. I also don't quite understand why my accuracy is at 1.0, so if you have any suggestions for that, or any ways to change my code, I would be more that happy to listen to any ideas. Thanks in advance

Solution 1

To get a prediction you just have to evaluate pred, which is the operation that defines the output of the model.

How to do it? With pred.eval(). But you need an input to evalaute its prediction, so you have to provide a feed_dict dictionary to eval() with the sample (or samples) you want to process.

The resulting code looks like:

predictions = pred.eval(feed_dict = {x:testX})

Notice how this is very similar to acc.eval({x:testX, y:testy}), because the idea is the same. You have an operation (acc in this case) which needs some input to be evaluated, and you can evaluate it either by calling acc.eval() or sess.run(acc) with the corresponding feed_dict with the necessary inputs.

Solution 2

The simplest way would be to use the existing session while training (between iterations):

print (sess.run(model, {x:X_example}))

where X_example is some numpy example tensor.

Solution 3

The below line will give you probability scores for every class for example is you 3 classes then the below line will give you a array of shape of 1x3 Considering you want prediction of a single data point X_test you can do the following:

output = sess.run(pred, {x:X_test})

the maximum number in the above variable output will be you prediction so for that we will modify the above statement :

output = sess.run(tf.argmax(pred, 1), {x:X_test})
print("your prediction for X_test is :", output[0])

Other thing you can do is :

output = sess.run(pred, {x:X_test})
output = np.argmax(output)
print("your prediction for X_test is :", output)