Examining the Bottleneck between CPU and NVIDIA GPU

I was investigating which part of my computer is the culprit for slowing down neural net training. I first thought it was CPU doing the image preprocessing, as my CPU is Intel's low-end series G4560, which only costs about $90, whereas my GPU is NVIDIA's high-end series GTX 1070 that costs more than whopping $400, thanks to cryptocurrency booming.

To my surprise, it was actually the GPU that was lagging behind this time, at least for the current network that I am training. I would like to share how I found out whether GPU or CPU was lagging. Below is the code, most of which is taken from Patrick Rodriguez's repository keras-multiprocess-image-data-generator.

To run the script, you first need to install necessary modules. Save the following as requirement.txt

cycler

functools32

matplotlib

numpy

nvidia-ml-py

pkg-resources

psutil

pyparsing

python-dateutil

pyt

six

subprocess32

Next, run the command below to automate installing all the necessary modules:

$ pip install -r requirement.txt

Lastly, you also need python-tk module, so install it via
$ sudo apt-get install python-tk

Now, you can run the script
$ python sysmonitor.py

Note that you must have NVIDIA GPU in order for the script to work.

Setup SSH Login Email Alert on Cent OS 7

This is similar to this post but not exactly.

First, you need to install mailx
# yum install -y mailx

Second, create /etc/ssh/ssh_alert.sh with the following:
#!/bin/sh

# Change these two lines:
sender="sender@some_email.com"
recepient="recepient@some_email.com"

if [ "$PAM_TYPE" != "close_session" ]; then
    host="`hostname`"
    subject="SSH Login: $PAM_USER from $PAM_RHOST on $host"
    # Message to send, e.g. the current environment variables.
    message="`env`"
    echo "$message" | mailx -r "$sender" -s "$subject" "$recepient"
fi

Lastly, add the following line to /etc/pam.d/sshd
session required pam_exec.so seteuid /bin/sh /etc/ssh/ssh_alert.sh

That's it!

Enable Network on Cent OS 7 Minimal

I was going to be testing out a very minimal server, so I chose to install Cent OS 7 minimal version. To my surprise, after installation I noticed that network was not ON by default!

After some searching, I realized that I need to edit /etc/sysconfig/network-scripts/ifcfg-eth0 to read:
ONBOOT="yes"

Note that you will need to replace eth0 with your network adapter name.

In addition, there is no ifconfig or wget on Cent OS minimal, which I understand. To install these, simply run
# yum install -y net-tools wget

Install VirtualBox Guest Addition for Cent OS

Here is how to install VirtualBox Guest Additions on Cent OS 7. For Ubuntu or Debian, refer to here.

If you just insert the Guest Addition image and try to run the Linux installation script, you will probably end up with
# ./VBoxLinuxAdditions.run
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.22 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Removing installed version 5.1.22 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Starting the VirtualBox Guest Additions.
Failed to set up service vboxadd, please check the log file
/var/log/VBoxGuestAdditions.log for details.

The log file indicates
# cat /var/log/VBoxGuestAdditions.log

vboxadd.sh: failed: Look at /var/log/vboxadd-install.log to find out what went wrong.
vboxadd.sh: failed: modprobe vboxguest failed.

Finally, the next log file reads
# cat /var/log/vboxadd-install.log
/tmp/vbox.0/Makefile.include.header:112: *** Error: unable to find the sources of your current Linux kernel. Specify KERN_DIR=<directory> and run Make again.  Stop.
Creating user for the Guest Additions.
Creating udev rule for the Guest Additions kernel module.

Basically, you need to specify the kernel source directory. So, here is what you need to do.

First, you need to install dkms, but if you do, you are like to receive
# yum install dkms
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.oasis.onnetcorp.com
 * extras: data.nicehosting.co.kr
 * updates: data.nicehosting.co.kr
No package dkms available.
Error: Nothing to do

To install this, the easiest way is to install EPEL repo.
# yum install epel-release -y

After installing EPEL, try installing dkms again:
# yum install dkms -y

Next, install headers and sources
# yum install kernel-devel -y

You can now locate your kernel sources. In my case, I get
# ls /usr/src/kernels
3.10.0-514.26.2.el7.x86_64  3.10.0-514.26.2.el7.x86_64.debug

Export the environment variable with the first directory. Make sure to replace it the correct version for your system as it may differ from mine.
# export KERN_DIR=/usr/src/kernels/3.10.0-514.26.2.el7.x86_64

Finally, you are ready to install VirtualBox Guest Additions.
# ./VBoxLinuxAdditions.run
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.22 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Removing installed version 5.1.22 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Starting the VirtualBox Guest Additions.

By the way, if you are running commands with sudo, you need to make sure to run with -E option:

$ sudo -E ./VBoxLinuxAdditions.sh 

Otherwise, your sudo command will shadow KERN_DIR environment variable that was exported from the user.

Adding User to Sudoer in Cent OS 7

I am just so used to sudo command with Ubuntu that I am going to need to do the same with Cent OS. Here is how to add a user to sudoer in Cent OS. This is a bit different from how you would do with Debian systems discussed here.

To get root access in Cent OS, run
$ su

As a root, to give a user permission to use sudo command, run
# usermod -aG wheel USER_NAME

That's it. The user must log back in order for the change to take effect.

Setup SSH Remote Login for Cent OS

Maybe it's time. After using Ubuntu and Debian for years, I now officially want to switch to Cent OS.  More on this topic later. For today's post, I will just go over how to install ssh server and enable it at boot time.

To install ssh server on Cent OS, simply run
# yum install openssh-server

Unlike Ubuntu, where ssh server will be automatically started at boot time, Cent OS seems that one must manually do so. For one time start, run
# service sshd start

However, if you want your ssh server to keep running even after reboot or restart, run
# chkconfig sshd on
Created symlink from /etc/systemd/system/multi-user.target.wants/sshd.service to /usr/lib/systemd/system/sshd.service.

That's all. If you want to turn it off, then
# chkconfig sshd off
Note: Forwarding request to 'systemctl disable sshd.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/sshd.service.

Enjoy Cent OS!