Debugging Python Module Built with Pybind11

Pybind11 is a wonderful library to build native c++ functions callable within Python. Let's see how we can debug a module built with Pybind11.

For this project, we will use the very simple Pybind11 example module. Let's get started.

Clone the repository
$ git clone https://github.com/pybind/python_example.git
$ cd python_example

Install the package
$ python setup.py install

Run gdb on Python
$ gdb -q python

Break into the desired function, such as
(gdb) b add
(gdb) b src/main.cpp:4

Run the command
(gdb) r tests/test.py

That is it!

Network Visualization with Intermediate Layer Shapes

Network visualization is very important. Looking at the network graph is much easier than reading through the network description. In this post, I will discuss how to create visual graph of a (Pytorch) network using Netron and include intermediate layer shapes as well.

Netron is an excellent tool for network visualization. At the moment, however, Netron does not support Pytorch natively (experimental feature but not stable). The best thing is to convert Pytorch model to ONNX and then use Netron to graph it.

The following is an example code that graphs ResNet50.
import torch
from torchvision import models
import onnx
from onnx import shape_inference

DEVICE = 'cuda:1'
PATH = 'resnet.onnx'
model = models.resnet50().to(DEVICE)
model.eval()
dummy_input = torch.randn(1,3,224,224).to(DEVICE)
torch.onnx.export(model, dummy_input, PATH, verbose=False)
onnx.save(shape_inference.infer_shapes(onnx.load(PATH)), PATH) # this is required for displaying intermediate shapes



To graph it, simply download Netron and run
$ netron resnet.onnx

That's it!

Building Caffe2 and Pytorch for Ubuntu 18.04 LTS

I am trying to export my pytorch model to Android devices. It seems that using onnx and caffe2 is the easiest way to do so. Here, I will describe the steps to build and install pytorch & caffe2 for Ubuntu 18.04 LTS.

Basically, one has to follow the instructions here, but I had some problems building. The easiest way to build is to disable all unnecessary features. For example, since I am only going to port it to caffe2 and do not intend to use GPU for running pytorch model (at least for this python environment), I can set some flags:
$ export NO_CUDA=1
$ export NO_DISTRIBUTED=1

Also, I will build pytorch and caffe2 together. Hence, I set the flag
$ export FULL_CAFFE2=1

With these environment variables set, I successfully built and installed caffe2 and pytorch from the source, following the official instruction.

Happy hacking!

Pytorch Implementation of BatchNorm

Batch Normalization is a really cool trick to speed up training of very deep and complex neural network. Although Pytorch has its own implementation of this in the backend, I wanted to implement it manually just to make sure that I understand this correctly. Below is my implementation on top of Pytorch's dcgan example (BN class starts at line 103)


Although this implementation is very crude, it seems to work well when tested with this example. To run this, type in
$ python main.py --cuda --dataset cifar10 --dataroot .

Dimensionality Reduction and Scattered Data Visualization with MNIST

We live in 3D world, and we can only view scattered data in 1D, 2D, or 3D. Yet, we deal with data that have very large dimension. Consider MNIST dataset, which is considered to be a toy example in deep learning field, consists of 28 X 28 gray images; that is 784 dimensions.

How would this MNIST data look like in 2D or 3D after dimensionality reduction? Let's figure it out! I am going to write the code in Pytorch. I have to say, Pytorch is so much better than other deep learning libraries, such as Theano or Tensorflow. Of course, it is just my personal opinion, so let's not get into this argument here.

What I want to do is to take Pytorch's MNIST example found here, and make some modifications to reduce the data dimension to 2D and plot scattered data. This will be a very good example that shows how to do all the following in Pytorch:
1. Create a custom network
2. Create a custom layer
3. Transfer learning from an existing model
4. Save and load a model

Here is the plot I get from running the code below.

This code is tested on Pytorch 0.3.1.