Tensorflow Fundamentals - K-means Cluster Part 2

From the previous post, I have shown how to calculate k-mean cluster using Tensorflow. In this post, I will add a bit more advanced implementations. In particular, I will show you how to implement conditional statement in Tensorflow.


The difference is at guessing the initial set of centroids. In the previous implementation, I simply chose k random points as initial centroids. Here, instead, I am selecting the first centroid to be the point furthest away from the origin. Next ith initial centroid for i = {1,2,...,k} is chosen such that the sum of the distances from previous i-1 centroids is the largest. This way, we can significantly reduce iteration number required to achieve the final state.

Tensorflow Fundamentals - K-means Cluster Part 1

Now that we are familiar with Tensorflow, let us actually write code. For this series of posts, we are going to implement K-means clustering algorithm with Tensorflow.

K-means clustering algorithm is to divide a set of points into k-clusters. The simplest algorithm is
1. choose k random points
2. cluster all points into corresponding k groups, where each point in the group is closest to the centroid
3. update the centroids by finding geometric centroids of the clusters
4. repeat steps 2 & 3 until satisfied

Below is my bare-minimum implementation in Tensorflow.

Tensorflow Fundamentals - Interactive Session

As I have discussed in my previous posts, Tensorflow's computation graphs will not evaluate the expression unless one explicitly asks it to do so. One may find this quite annoying during debugging, so Tensorflow provides what is called Interactive Session, which let's you evaluate the expressions as you go, with minimal code.

This is really simple; just call
sess = tf.InteractiveSession()

in the beginning, and this will act like the block
with tf.Session() as sess:

See the code below.

Tensorflow Fundamentals - Computation Graph Part 3

Next up, we want to now evaluate an expression from given input values. Let us construct a function (graph)

f(x,y) = x + y

where x,y are input values to fed into the graph f. To do this, we need to use tf.placeholder methods to define input nodes, and supply feed_dict parameter to run method as shown below:

The code is easy enough to be self-explanatory. The output of the code shall look similar to

$ python tf_computation_graph_p3.py 2>/dev/null
8 + -6 = 2
-6 + 4 = -2
-10 + -1 = -11
-6 + 0 = -6
5 + 9 = 14
7 + 6 = 13
3 + 8 = 11
3 + 6 = 9
5 + -4 = 1
0 + -3 = -3

Tensorflow Fundamentals - Computation Graph Part 2

Let's continue our journey on Tensorflow's computation graph.

We will now make use of Tensorflow's variables. They are different from constant in that
1. they are mutable during the execution, i.e., they can change their values
2. they will store their state (value), which shall equal to that from the lastest execution

For example, you can define a counter variable that will increment its value on each execution:
counter = counter + 1

Below is the simple demo code for doing this in Tensorflow:

The only catch here is that
1. tf.assign is another operation that will assign a new value to the variable on each execution (run)
2. one must initialize all variables

Running it will output
0
1
2
3
4
5

So far so good!

Tensorflow Fundamentals - Computation Graph Part 1

This post is intended for anyone, including myself, who is having difficulty grasping the very basic concepts of Tensorflow: Computation Graph. This post is heavily based on Tensorflow's official documentation.

The way Tensorflow and Theano operates is just different from all those other ones in that there are two distinct phases. In the first phase one builds the computation graph that defines the computations to be performed. It is like a function where given whatever inputs, it will spit out outputs. For example, let us say you define
f(x,y) = x + y

Then f is the computation graph in Tensroflow. This phase is called construction phase.

In the second phase, one executes the graph by feeding in the inputs. For example,
f(1,2) = 3
f(3,2) = 5

and so on for any x,y pairs you feed in. The graph will output the numerical values given numerical inputs. This phase is called execution phase.

One difference, however, is that not only the mathematical operations, such as additions, subtractions, multiplications, and divisions but also each variables are considered as operation nodes in Tensorflow's computation graphs. Thus, with the example above, we now have three ops:
x
y
+

Here, x y are constant ops, meaning that their values will be some constant directly fed in during the execution phase. The output of the graph f can be fed into a more complex graph.

Let's do a very simple example in code. We will define the computation graph for
f = pi + 1

and compute f for constant pi = 3.14159...


The output of the script shall yield
4.14159

So far so easy. We will progressively construct and execute more complex and useful graphs, so stay with me.

Output Size Calculation for Convolution and Pooling Layers

I keep forgetting the exact equation for calculating the output size of convolution and pooling layers. Below are equations directly from tensorflow's official documentation:

For 'SAME' padding, we have
out_height = ceil(float(in_height) / float(strides[1]))
out_width  = ceil(float(in_width) / float(strides[2]))

For 'VALID' padding, we have
out_height = ceil(float(in_height - filter_height + 1) / float(strides[1]))
out_width  = ceil(float(in_width - filter_width + 1) / float(strides[2]))

Note that in tensorflow, a typical input is 4D tensor with shape [batch_size,  height, width, channels]. Thus, strides[1] and strides[2] corresponds to stride_height and stride_width, respectively.

Hopefully this is useful!

Handwriting Recognition with Deep Learning

In this post, we will go over an example to apply deep learning, in particular convolutional neural network, to recognize handwritten numbers from 0 to 9 using TensorFlow backend Keras. If you don't have deep learning environment setup yet, checkout this post.

Thankfully, there is publicly available handwritten digits and its labels on the web that we can use. For more info, check out MNIST website. Even better, Keras has MNIST data module, so we will use  it.

Use your favorite editor and create mnist.py file with the following:

Upon running the file with
$ python mnist.py

you will see convolutional network being trained with data and tested with unseen data with accuracy of about 99%!

Setup Deep Learning Development Environment in Ubuntu

In this tutorial, I will go through step by step instructions to setup deep learning development environment for Ubuntu. We will install the following python packages on fresh Ubuntu 16.04:
opencv
tensorflow
keras
matplotlib
numpy
scipy
sklearn
tk

Let's dig it! First, I would install virtualenv, in case you need multiple python environments.
$ sudo apt-get install virtualenv -y

To create an environment, simply run
$ virtualenv ENV
where replace ENV with the deep learning environment name you would like.

To activate the new environment,
$ source ~/ENV/bin/activate
where again replace ENV with the name chosen above.

Next, we need to install pip, which helps us install these python packages with ease.
$ sudo apt-get install python-pip -y

You may want to upgrade pip to the latest version:
$ pip install --upgrade pip

Next, let's install python packages within the environment.
$ pip install tensorflow keras numpy scipy matplotlib sklearn

For OpenCV and TK, we need to install it from apt-get:
$ sudo apt-get install libopencv-dev python-opencv python-tk -y

That's it! Now you are ready to develop your neural network with tensorflow backend keras! If you want to test out if your environment is successfully setup, check out this post.

How to Install NVIDIA Drivers on Ubuntu for CUDA / cuDNN

OK, so I have purchased my desktop system with dedicated GPU! The last time I purchased a system with dGPU was back in 1990s, so it's been about 20 years. I bought one because I wanted to study tensorflow / keras, and I simply couldn't do it with just CPUs, so I had to get NVIDIA GPU.

Anyways, I noticed that Ubuntu doesn't install its driver automatically, so here is how to do so manually. First, download CUDA from NVIDIA here. As of now, the latest version is CUDA 8.0. If you are going to use tensorflow, make sure it supports the version of CUDA you are going to download from the official documentation page.

Follow the instruction on NVIDIA download page to install. I am going to download local runfile for Ubuntu 16.04. For this option, I need to run
$ sudo sh cuda_8.0.61_375.26_linux.run

***
Note that the runfile will probably not work unless you make sure 2 things:

1. Install gcc on your system
$ sudo apt-get install build-essential

2. Run it in console mode. Follow the answer by Rey on this post for details.
***

Download cuDNN from here. I am going to download cuDNN v5.1 Library for Linux for CUDA 8.0. Decompress the file into /usr/local/cuda/cudnn folder:
$ tar xfz cudnn-8.0-linux-x64-v5.1.tgz -d cudnn
$ sudo mv cudnn /usr/local/cuda/cudnn

Next, add library paths by creating /etc/ld.so.conf.d/cuda.conf file with the following lines:
/usr/local/cuda/lib64/
/usr/local/cuda/cudnn/lib64/

Refresh ld cache by
$ sudo ldconfig

Finally, install tensorflow-gpu. I am going to use pip:
$ sudo apt-get install python-pip
$ pip install tensorflow-gpu

Now, when you run the following command, you should see all CUDA library opened successfully:
$ python -c 'import tensorflow'

Happy hacking!