OK, so I have purchased my desktop system with dedicated GPU! The last time I purchased a system with dGPU was back in 1990s, so it's been about 20 years. I bought one because I wanted to study tensorflow / keras, and I simply couldn't do it with just CPUs, so I had to get NVIDIA GPU.
Anyways, I noticed that Ubuntu doesn't install its driver automatically, so here is how to do so manually. First, download CUDA from NVIDIA here. As of now, the latest version is CUDA 8.0. If you are going to use tensorflow, make sure it supports the version of CUDA you are going to download from the official documentation page.
Follow the instruction on NVIDIA download page to install. I am going to download local runfile for Ubuntu 16.04. For this option, I need to run
$ sudo sh cuda_8.0.61_375.26_linux.run
Note that the runfile will probably not work unless you make sure 2 things:
1. Install gcc on your system
$ sudo apt-get install build-essential
2. Run it in console mode. Follow the answer by Rey on this post for details.
Download cuDNN from here. I am going to download cuDNN v5.1 Library for Linux for CUDA 8.0. Decompress the file into /usr/local/cuda/cudnn folder:
$ tar xfz cudnn-8.0-linux-x64-v5.1.tgz -d cudnn
$ sudo mv cudnn /usr/local/cuda/cudnn
Next, add library paths by creating /etc/ld.so.conf.d/cuda.conf file with the following lines:
Refresh ld cache by
$ sudo ldconfig
Finally, install tensorflow-gpu. I am going to use pip:
$ sudo apt-get install python-pip
$ pip install tensorflow-gpu
Now, when you run the following command, you should see all CUDA library opened successfully:
$ python -c 'import tensorflow'