To my surprise, it was actually the GPU that was lagging behind this time, at least for the current network that I am training. I would like to share how I found out whether GPU or CPU was lagging. Below is the code, most of which is taken from Patrick Rodriguez's repository keras-multiprocess-image-data-generator.
To run the script, you first need to install necessary modules. Save the following as requirement.txt
Next, run the command below to automate installing all the necessary modules:
$ pip install -r requirement.txt
Lastly, you also need python-tk module, so install it via
$ sudo apt-get install python-tk
Now, you can run the script
$ python sysmonitor.py
Note that you must have NVIDIA GPU in order for the script to work.