Kaldi series 1 - setup/debug with CLion

In the series of posts, I will describe how to run automatic speech recognition (ASR) system with Kaldi. For the best debugging experience, I will describe step by step run/debug instructions with CLion, the best C++ IDE on non-Windows systems. FYI, I am running these commands on macOS 10.15 (Catalina), but should be similar on Linux systems.

In the very first series, we will simply setup Kaldi project on CLion for running and debugging.
$ git clone https://github.com/kaldi-asr/kaldi.git && cd kaldi

Kaldi recently added CMake support (Thank you so much!), and it will be so much easier for CLion to load the project now. Run CLion and open up Kaldi directory. Run Build --> Build All in Debug. This process will take quite some time, so please be patient.

Unfortunately, there are other things to take care of. The following commands will take some time to run, so be patient.
$ cd tools && make -j4
$ extras/install_irstlm.sh && cd ..

Once you are done, let's run a pre-trained model to see if it works fine.
$ cd egs/apiai_decode/s5
$ ./download-model.sh

We also need to let CMake-built binary files to be used. Edit path.sh as below:
export KALDI_ROOT=`pwd`/../../..
export KALDI_CMAKE_ROOT=`pwd`/../../../cmake-build-debug
[ -f $KALDI_ROOT/tools/env.sh ] && . $KALDI_ROOT/tools/env.sh
export PATH=$PWD/utils/:$KALDI_ROOT/tools/openfst/bin:$PWD:$PATH
[ ! -f $KALDI_ROOT/tools/config/common_path.sh ] && echo >&2 "The standard file $KALDI_ROOT/src/path.sh is not present -> Exit!" && exit 1
. $KALDI_ROOT/tools/config/common_path.sh
export LC_ALL=C

Lastly, edit tools/config/common_path.sh by replacing KALDI_ROOT to KALDI_CMAKE_ROOT as follows:
# we assume KALDI_CMAKE_ROOT is already defined
[ -z "$KALDI_CMAKE_ROOT" ] && echo >&2 "The variable KALDI_CMAKE_ROOT must be already defined" && exit 1
# The formatting of the path export command is intentionally weird, because
# this allows for easy diff'ing

export PATH=\
${KALDI_CMAKE_ROOT}/src/bin:\
${KALDI_CMAKE_ROOT}/src/chainbin:\
${KALDI_CMAKE_ROOT}/src/featbin:\
${KALDI_CMAKE_ROOT}/src/fgmmbin:\
${KALDI_CMAKE_ROOT}/src/fstbin:\
${KALDI_CMAKE_ROOT}/src/gmmbin:\
${KALDI_CMAKE_ROOT}/src/ivectorbin:\
${KALDI_CMAKE_ROOT}/src/kwsbin:\
${KALDI_CMAKE_ROOT}/src/latbin:\
${KALDI_CMAKE_ROOT}/src/lmbin:\
${KALDI_CMAKE_ROOT}/src/nnet2bin:\
${KALDI_CMAKE_ROOT}/src/nnet3bin:\
${KALDI_CMAKE_ROOT}/src/nnetbin:\
${KALDI_CMAKE_ROOT}/src/online2bin:\
${KALDI_CMAKE_ROOT}/src/onlinebin:\
${KALDI_CMAKE_ROOT}/src/rnnlmbin:\
${KALDI_CMAKE_ROOT}/src/sgmm2bin:\
${KALDI_CMAKE_ROOT}/src/sgmmbin:\
${KALDI_CMAKE_ROOT}/src/tfrnnlmbin:\
${KALDI_CMAKE_ROOT}/src/cudadecoderbin:\
$PATH

Tedious Kaldi setup is all done finally. Now, you need some audio file for testing, so simply create a wav file with your voice, saying whatever you want to be transcribed (in English). Make sure to use 16KHz sampling rate w/ 16-bit encoding. Save this file as test.wav. Let's run it!

$ ./recognize-wav.sh /PATH/TO/YOUR/WAV/test.wav

You should see its transcript in the log. Now let's debug, decoding for example, with CLion. As you can see from the log, the main decoding execution command is as follows:
nnet3-latgen-faster --frame-subsampling-factor=3 --frames-per-chunk=50 --extra-left-context=0 --extra-right-context=0 --extra-left-context-initial=-1 --extra-right-context-final=-1 --minimize=false --max-active=7000 --min-active=200 --beam=15.0 --lattice-beam=8.0 --acoustic-scale=1.0 --allow-partial=true --word-symbol-table=exp/api.ai-model/words.txt exp/api.ai-model/final.mdl exp/api.ai-model//HCLG.fst 'ark,s,cs:apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:data/test-corpus/utt2spk scp:data/test-corpus/cmvn.scp scp:data/test-corpus/feats.scp ark:- |' 'ark:|lattice-scale --acoustic-scale=10.0 ark:- ark:-  >exp/lat.1'

This big command consists of multiple execution piped in a convoluted way, so let's do one by one. The main execution binary nnet3-latgen-faster takes 4 arguments, as you can see from
$ nnet3-latgen-faster

By the way, it is likely that you will get command not found error, so let's do this first
$ export KALDI_CMAKE_ROOT=$(pwd)/../../../cmake-build-debug
$ source ../../../tools/config/common_path.sh

Now, try again
$ nnet3-latgen-faster

The first two arguments are provided from the files, i.e. exp/api.ai-model/final.mdl and exp/api.ai-model/HCLG.fst.

The third argument is features, which is read from stdin from running the command
apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:data/test-corpus/utt2spk scp:data/test-corpus/cmvn.scp scp:data/test-corpus/feats.scp ark:-

We will create this features file separately, by running
$ apply-cmvn --norm-means=false --norm-vars=false --utt2spk=ark:data/test-corpus/utt2spk scp:data/test-corpus/cmvn.scp scp:data/test-corpus/feats.scp ark:features.feat

You should see features.feat file created. We can now run the decoding with this file as an input
$ nnet3-latgen-faster --frame-subsampling-factor=3 --frames-per-chunk=50 --extra-left-context=0 --extra-right-context=0 --extra-left-context-initial=-1 --extra-right-context-final=-1 --minimize=false --max-active=7000 --min-active=200 --beam=15.0 --lattice-beam=8.0 --acoustic-scale=1.0 --allow-partial=true --word-symbol-table=exp/api.ai-model/words.txt exp/api.ai-model/final.mdl exp/api.ai-model/HCLG.fst ark:features.feat ark:lat.1

Here, I simply replaced the third argument as the feature file and the fourth argument as lat.1 output file, without piping to lattice-scale.

Finally, it is time to run this on CLion debug mode. From CLion's edit configuration, select nnet3-latgen-faster. Enter the program arguments, copied from the above and make sure to set the working directory as the current directory, i.e., egs/apiai_decode/s5. You can set the breakpoint in main function, say line 38 and start debugging with CLion. It should all work well!

WTF? Fix to "string.h not found" in macOS

OK, I love macOS but I sometimes hate the hassle when it comes to Xcode and its toolchains. Randomly I get errors like "string.h" not found... WTF?

Here is the fix. You probably have the Xcode command line tools installed; Run
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.1X.pkg

where you want to put the right version yourself. (X = 4 for Mojave, 5 for Catalina, etc)

If you don't have the package, try deleting and re-installing the tools and re-try

sudo rm -rf /Library/Developer/CommandLineTools

xcode-select --install

If that does not work, try
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

Hope this fixes!

*** EDIT ***

Sometimes with CLion, you may get similar error. For that, the fix is rather simple. Go to Tools --> CMake --> Reset Cache and Reload Project. That's it!

Compile GNU Coreutils from Scratch on macOS Mojave

Here, I will discuss how to compile GNU coreutils from scratch. You have two options. I recommend Option 2 below.

Option 1:
First, download the source code from its repo. I will use v8.31
$ git clone https://github.com/coreutils/coreutils.git -b v8.31
$ cd coreutils

Next, clone git submodule
$ git submodule update --init

Next, bootstrap
$ ./bootstrap
./bootstrap: line 470: autopoint: command not found
./bootstrap: Error: 'autopoint' not found
./bootstrap: line 470: gettext: command not found
./bootstrap: Error: 'gettext' not found
./bootstrap: Error: 'makeinfo' version == 4.8 is too old
./bootstrap:        'makeinfo' version >= 6.1 is required

./bootstrap: See README-prereq for how to get the prerequisite programs

Well, I need to get prerequisite programs first. Install gettext using brew
$ brew install gettext && brew link gettext --force

Let's try again
$ ./bootstrap 
./bootstrap: Error: 'makeinfo' version == 4.8 is too old
./bootstrap:        'makeinfo' version >= 6.1 is required

./bootstrap: See README-prereq for how to get the prerequisite programs

To check the version, run
$ makeinfo --version
makeinfo (GNU texinfo) 4.8

Copyright (C) 2004 Free Software Foundation, Inc.
There is NO warranty.  You may redistribute this software
under the terms of the GNU General Public License.
For more information about these matters, see the files named COPYING.

So, I do need to update makeinfo. Note that this is the same as texi2any from GNU texinfo package.
$ pushd && cd ~/Downloads
$ wget http://ftp.gnu.org/gnu/texinfo/texinfo-6.6.tar.gz
$ cd texinfo-6.6
$ ./configure
$ make -j4
$ sudo make install
$ makeinfo --version
texi2any (GNU texinfo) 6.6

Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Now, we are ready to bootstrap again
$ popd && ./bootstrap

Option 2:
Download the distribution source code
$ wget https://ftp.gnu.org/gnu/coreutils/coreutils-8.31.tar.xz
$ tar xfj coreutils-8.31.tar.xz && cd coreutils-8.31

------------------------------------------------------------------
The easy parts are left.
$ ./configure

Finally, we should be able to build it
$ make -j4
$ sudo make install

Happy hacking!

Import CMAKE Project on NetBeans

In the previous post, I discussed how to import a CMAKE project for Eclipse, and it wasn't that easy. Today, I will discuss how to import a CMAKE project on NetBeans. Again, I will use cmake-exmaple as an example project.

Clone and download the CMAKE project on your computer.
$ git clone https://github.com/bast/cmake-example.git ~/cmake-example

First, open up NetBeans C++ IDE. If you are running it on Mac OS X, I recommend running it from Terminal
$ /Applications/NetBeans/NetBeans\ 8.2.app/Contents/MacOS/netbeans 

The reason is that if you simply run NetBeans from GUI, your environment variables might not sync.

Next, select File --> New Project --> C/C++ Project with Existing Sources --> Next. Choose Browse the folder which contains your CMAKE project, in this example ~/cmake-example folder. Select Custom under Select Configuration Mode and select Next. Check-box Pre-Build Step is Required. Specify Run in Folder as the project root directory, i.e., ~/cmake-example. Enter the following for Command
cmake -H. -Bbuild

Continue with remaining options and adjust what is necessary. Usually, the default setting should work.

Once the project has been imported, you will need to verify the settings. Right-click the project name cmake-example on the project pane on the left, and select Properties. Make sure in the Build --> Pre-Build option, you have the Command Line is set as cmake -H. -Bbuild and Pre-Build First box is checked.

In the Build --> Make option, the Working Directory should be set as build, since this is where the build will take place. Select Apply and Close. 

Press <F11> key to build the project. It should build successfully. Before we debug this, we have to make sure to set debug flag in the CMakeLists.txt file by setting CMAKE_CXX_FLAGS.

Open up CMakeLists.txt file in the project root directory, that is ~/cmake-example/CMakeLists.txt and not the file in the src directory. Modify line 35 to look as below:

# project version
set(VERSION_MAJOR 1)
set(VERSION_MINOR 0)
set(VERSION_PATCH 0)
set(CMAKE_CXX_FLAGS '-g')

Now, we are ready to run or debug. Re-build the project, and set breakpoint on src/main.cpp file. You should be able to debug it successfully.

Happy coding!

Import CMAKE Project to Eclipse CDT

In this post, I will discuss how to import a CMAKE project to Eclipse CDT. Upon Google search, I ran into this solution, but this did not work for me, so here is what I did instead. I am going to make use of cmake-example git repo project to demonstrate how, but you can easily do this for your own.

Open up Eclipse CDT and select File --> New --> C/C++ Project --> C++ Managed Build --> Next. Enter the project name, say cmake-example, and make sure the project type is Empty Project. Also, select the appropriate Toolchains; this will be Linux GCC or MacOSX GCC. Select Finish.

Go to the project root folder, and we will clone the git repository.
$ cd ~/Eclipse/workspace/cmake-example
$ git init
$ git remote add origin https://github.com/bast/cmake-example.git
$ git fetch
$ git checkout -t origin/master

Let's verify that the project compiles.
$ cmake -H. -BDebug
$ cd Debug
$ make -j4

Make sure that the project builds successfully. Also, note down your $PATH environment variable (highlighted in blue below) to be used later. Your variable may differ from mine.
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin

Now, on Eclipse right-click this project in the Project Explorer pane on the left and select Properties. In the C/C++ Build tab, uncheck both Use default build command and Generate Makefile automatically. Make sure the Build Directory as ${workspace_loc:/cmake-example}/Debug. This is the folder we created with CMAKE in the previous step and built the project. This folder contains CMAKE-generated Makefile, and we are simply asking Eclipse to execute make command in this particular folder.

This is the first step, where the Eclipse simply runs the make command of the Makefile generated from CMAKE. This setup is good if we are not going to edit CMAKE configs anymore. In reality, we probably will need to edit CMAKE configs.

Every time you modify CMakeLists.txt, you will need to re-create Makefile by running the CMAKE command again and again
$ cmake ..

Let's simply automate this build command with Eclipse. This is the second step of this post. Create build.sh in the project folder,
$ vim ~/Eclipse/workspace/cmake-example/build.sh

Simply write down the build commands that you would run from the Debug directory as follows:
cmake ..
make -j4

Now, in the Eclipse open up Project Properties window again. Expand C/C++ Build entry on the left and select Environment. Select Add button, and enter PATH for the Name field and your $PATH environment variable (noted in the previous step) for the Variable field. This is to make sure the Eclipse shell will be able to perform exactly what you can do with your own shell.

In the C/C++ Build tab, enter the following build command in place of make
sh ../build.sh

Select Apply and Close to close the properties window. The project should now successfully build, even if you have modified CMakeLists.txt files.

Lastly, you can modify the run command by selecting Run --> Run Configurations... and browse the executable for C/C++ Application:
~/Eclipse/workspace/cmake-example/Debug/bin/unit_tests

Happy hacking!

Debugging Bash Scripts

Bash scripts are extremely handy when mostly dealing with Linux commands, such as find, grep, sed, and so on. One can write functions, just like in any other languages. However, I have to admit that I haven't really dealt with Bash scripting much so far. One of the reasons, I suppose, is that it is difficult to debug Bash scripts. I didn't think there was any debugging tool for Bash. Well, I just realized that I was wrong! There is in fact a very nice debugger for Bash: bashdb

To install, you can run the following on macOS
$ brew install bashdb

or the following for Ubuntu
$ sudo apt-get install bashdb

To start debugging, run
$ bashdb bash_script.sh argument1 argument2 ...

Many of the commands are similar to that of gdb, but for more info, type in help. Also, this documentation can be of great reference.

Happy bashing!

Replace Lines with Given Strings in Shell

Let's learn how to replace given lines by given string. Consider, for example,

$ cat some_file

abcdefg

hijklmn

opqrstu

vwxyz

12345

67890

Let's replace line 2 with string THIS_IS_NEW_LINE2

$ sed '2s/.*/THIS_IS_NEW_LINE2/g' some_file

abcdefg

THIS_IS_NEW_LINE2

opqrstu

vwxyz

12345

67890

You can replace multiple lines with multiple -e options

$ sed -e '2s/.*/THIS_IS_NEW_LINE2/g' -e '3s/.*/THIS_IS_NEW_LINE3/g' some_file

abcdefg

THIS_IS_NEW_LINE2

THIS_IS_NEW_LINE3

vwxyz

12345

67890

Of course, you can always use -i option to modify file on the fly

$ sed -i '2s/.*/THIS_IS_NEW_LINE2/g' some_file

$ cat some_file

abcdefg

THIS_IS_NEW_LINE2

opqrstu

vwxyz

12345

67890

Print Selected Lines in Shell

Let's assume you want to print certain lines in given text. For example, consider

$ cat some_file

1

2

3

4

5

6

7

8

9

Let's say you want to print lines 3-5. This is very easy using sed:

$ sed -n '3,5p' some_file

3

4

5

Let's say you want to print all lines, except 3-5. This can be done with d option:

$ sed '3,5d' some_file

1

2

6

7

8

9

Note that by default sed prints all lines, so -n option asks sed to print only 3 to 5 lines and suppresses the default behavior. On the other hand, the d letter in the single quotes asks sed to delete the lines 3-5 from printing by default, so it prints only the rest of the lines instead. 

Find Line Number Matching Given Expression in Shell

Say you want to find the line number that matches a given search string. For instance,
$ cat some_file.txt
this is some file line 1
it was written in Vim line 2
let's learn unix commands line 3

To find an expression, say unix, we can use grep
$ grep unix some_file
let's learn unix commands line 3

To show the line number of the expression, use -n option
$ grep -n unix some_file
3:let's learn unix commands line 3

To output the line number only, pipe with cut command
$ grep -n unix some_file | cut -d : -f 1
3

Here, -d option specifies delimiter, and -f option specifies the position separated by the delimiter.

If there are multiple lines matching the given expression, you can always use -m option of grep to print just first N of them
$ grep -n -m 2 unix some_file | cut -d : -f 1
1
2

Happy hacking!

Build HTK on macOS with Minimum Effort

In the last post, I presented instructions to build HTK on Ubuntu. In this post, I will show you how to compile HTK on macOS.

As before, download HTK latest stable version source code, 3.4.1 at the time of writing, from here. Extract the files as usual.

$ tar xfz HTK-3.4.1.tar.gz

$ cd htk

Next, you need some required software tools from Apple.

$ xcode-select --install

Now, let's configure and make

$ ./configure --prefix=$(pwd)

$ make

You will probably encounter an error complaining

(cd HTKLib && /Applications/Xcode.app/Contents/Developer/usr/bin/make HTKLib.a) \

 || case "" in *k*) fail=yes;; *) exit 1;; esac;

gcc  -ansi -g -O2 -DNO_AUDIO -D'ARCH="darwin"' -I/usr/include/malloc -Wall -Wno-switch -g -O2 -I. -DPHNALG   -c -o HGraf.o HGraf.c

HGraf.c:73:10: fatal error: 'X11/Xlib.h' file not found

There are two options here. The first is to simply skip HSLab package that requires X11, thereby building without X11.

$ ./configure --prefix=$(pwd) --disable-hslab

$ make

$ make install

You will see all binary files in the bin folder.

The other option is to install X11 library. Download XQuartz from here and install it. This will create /opt/X11/ folder with necessary library and header files. We can now manually compile HTKLib package by specifying the header files
$ cd HTKLib
$ gcc  -ansi -g -O2 -DNO_AUDIO -D'ARCH="darwin"' -I/usr/include/malloc -Wall -Wno-switch -g -O2 -I. -DPHNALG   -c -o HGraf.o HGraf.c -I /opt/X11/include
$ cd ..

We also need to set library path:
$ export LIBRARY_PATH=/opt/X11/lib

Now, we are ready to resume the make process

$ make

$ make install

Viola! You should see bin directory with all HTK binary files!

Running Keras Deep Neural Network Inference Web Server using WebDNN

Web app is a great tool for simple demo across any platforms. I find it very useful because I can easily show my trained model's output to anyone like a breeze. The catch, however, would be setting up the server, but once the server is up and running, it cannot be any better to boast in your resume and show off to your boss with your trained model.

Here is a tutorial for setting up a simple neural network inference server using WebDNN library. Although its Github page and official documentations are very clear, I still had to spend some time to get it to work. In addition, the official documentation instructs users to compile and install emscripten from source, which will consume quite some time; I will show you how to get around this with install pre-compiled version.

First, one needs to clone WebDNN repository from Github:
$ git clone https://github.com/mil-tokyo/webdnn.git && cd webdnn

Note that WebDNN only supports Python3.6+, so you need to install this unless you already have it on the system. To check the your Python3 version, run
$ python3 --version

Make sure that it is 3.6+. There are plenty of resources that you can search on Google on how to install Python 3.6+. For instance, on Mac OS X, using homebrew is probably the easiest way:
$ brew install python3

Once you have Python 3.6+, it is a good idea to create virtual environment for what you will need to do.
$ virtualenv -p `which python3` python3

Now, activate the environment and install necessary packages
$ source python3/bin/activate
$ pip install tensorflow-gpu keras h5py

The Python environment is now complete. You now need to setup emscripten environment.
$ git clone https://github.com/juj/emsdk.git && cd emsdk
$ ./emsdk install latest
$ ./emsdk activate latest
$ source ./emsdk_env.sh

Now, we need to install eigen library.
$ wget http://bitbucket.org/eigen/eigen/get/3.3.3.tar.bz2
$ tar jxf 3.3.3.tar.bz2
$ export CPLUS_INCLUDE_PATH=$PWD/eigen-eigen-67e894c6cd8f
$ cd ..

Finally, we are ready. Let's first create pre-trained ResNet Keras model that we will use. Run Python and run the following lines
$ python
>>> from keras.applications import resnet50
>>> model = resnet50.ResNet50(include_top=True, weights='imagenet')
>>> model.save("resnet50.h5")

Exit Python and run the following
$ python ./bin/convert_keras.py resnet50.h5 --input_shape '(1,224,224,3)' --out output

After some time, it will generates files in output directory. To run the server, we first need to modify the example/resnet/script.js file. We must make sure to point to the correct directory by changing the weight path. For the version I have, I modified line 39 to point to output directory.
let runner = await WebDNN.load(`/output`, {backendOrder: backend_name});

Finally, we are ready to run the server. To start the server, run the following in webdnn directory
$ python -m http.server

On your web browser, go to address localhost:8000/example/resnet/index.html

You should be able to test ResNet50 model!

Enable X11 Forwarding from macOS X

In order to enable X11 forwarding from macOS client, you must first download XQuartz from https://www.xquartz.org/ and install it. Then, you must log out and log back in.

Next, you can remote into your server via ssh with -X or -Y option:

$ ssh username@server_ip -X

To verify that you have X11 forwarding, simply examine $DISPLAY variable while you are remotely logged in:

$ echo $DISPLAY

localhost:10.0

By the way, some Linux servers may not have X11 forwarding feature disabled by default. In this case, you must enable it by editing /etc/ssh/sshd_config file:

X11Forwarding yes

Enjoy Mac Life!

Alternative to scp

For transferring files from remote computer, I use scp command. In this post. I will cover an alternative command: rsync.

The syntax is very similar to scp. For example,
$ rsync some/local/file user@remote:/location/in/remote
$ rsync user@remote:/location/in/remote some/local/file

However, the options are quite different. I would say just use -aP
$ rsync -aP some/local/file user@remote:/location/in/remote
$ rsync -aP user@remote:/location/in/remote some/local/file

Here, -a stands for archive, which includes recursive -r option, and -P stands for --partial and --progress. With these options, you can resume downloading the files when interrupted.

For more info, look up man page:
$ man rsync

Ah, by the way, if you are using port other than 22, make sure to use -e option as below:
$ rsync -aP -e 'ssh -P 1234' some/local/file user@remote:/location/in/remote
$ rsync -aP -e 'ssh -P 1234' user@remote:/location/in/remote some/local/file
where 1234 is the port you want to use.

Resume Downloading with wget

It is quite annoying when you are downloading a large file with wget or curl, and somehow your download is interrupted, because you have to download from scratch again. Well, here is what you can do to resume downloading where you left off.

With wget, simply add -c option:
$ wget -c http://url/to/download/file

That's it!

How to Use Vim on Macbook Pro Touch Bar

I really love my new Macbook Pro 15" model, with just one exception: its touch bar. I have no idea why the heck Apple wants to force everyone to use the touch bar in this model. You see, in the 13" Macbook Pro, you actually have an option to choose either the standard function keys or the touch bar, but with the 15" model, there is no such option.

My big complaints about touch bar should probably go into a separate thread, so I will start right into with my solutions to cope with the touch bar. I map F2 and F3 to work as :tabp and :tabn in Vim, but it just won't work with the touch bar, as there no long is standard function keys. What is even worse is that there is no physical ESC key, which is the most essential key in using Vim.

I actually remap CAPS LOCK to work as ESC key. This is supported by Mac OS X by default. To set this, go to Preferences --> Keyboard --> Modifier Keys --> Caps Lock --> Escape. For standard function key shortcuts, I simply remap ;n to Fn where n could be from 1 to 12. That is, for example, I have the following liens in ~/.vimrc:

nnoremap ;2 :tabp<CR>
nnoremap ;3 :tabn<CR>

for switching between tabs with ;2 and ;3 keystrokes. It takes some time to get used to, but at least this remapping makes touch bar usable.

GDB on Mac OS X (Safe Method)

The officially supported debugging package for Mac OS X is lldb, which is a fine debugger. However, many people including me still prefer to use gdb, the GNU debugger. In fact, I use cgdb, which provides nice color interface to gdb, and that is why I must install gdb on my Mac. Unfortunately, it is not so easy to install and enable gdb to debug on Mac. In the previous post, I presented a method to do this, but I figure now that it was quite unsafe way to do so. For more details, please take a look at the official documentation.

Here is a better way of enabling gdb on your Mac. First, if you haven't installed gdb, do so.
$ brew install gdb

If you are using Sierra or above, run the following as well:
$ echo "set startup-with-shell off" >> ~/.gdbinit

Next, this is the part where we give gdb debug other processes. Open up Keychain Access application and on the menu select Keychain Access -> Certificate Assistant -> Create a certificate.

Enter gdb-cert for the name, select Self Signed Root for Identity Type, select Code Signing for Certificate Type, and check the box Let me override defaults.

Click Continue several times until you see Specify a Location For the Certificate window. Select System for the Keychain and proceed.

Once the certificate is created, double click on it, and in the Trust section locate Code Signing item. Select Always Trust and exit. You will be prompted with your admin password to make the change.

Finally, you can close Keychain Access app and type in the following in terminal:
$ sudo killall taskgated
$ codesign -fs gdb-cert $(which gdb)

That's it! You will be able to use gdb on your Mac!

Displaying Exact Values for Digital Color Meter on Mac OS X

Mac OS X's built-in app Digital Color Meter is an extremely useful tool for me, and I love it. However, the only problem is that it is not exact.

I manually created an image with RGB values of arbitrary numbers, say (38, 205, 86) using OpenCV. I then opened up the image with Preview, and measured the color with Digital Color Meter. Unfortunately, its output was close but not exact.

I also noticed that there is an drop down menu, such as
Display in native values
Display in P3
...

After playing around with all of these options, I am very happy to report that the option Display in sRGB produces exact match with the pixel!

Java Compilation with Package and Jar Dependencies

In this tutorial, I will go over Java package and dependencies basics.

Let's go over the simplest java code with package declaration. Let's assume we are working on ~/java_package directory. Create Hello.java file below in the current directory:


To compile and execute this, we will first create a directory whose name is equal to the package name. In this case, the directory name should be pkgtest. So, create this directory and move Hello.java file into this. Run all of the following from ~/java_package directory:
$ mkdir pkgtest
$ mv Hello.java pkgtest/

Next, we need to compile. This is very simple.
$ javac pkgtest/Hello.java

Finally, we need to run it.
$ java pkgtest/Hello
hello

Note that you must run this from ~/java_package directory; otherwise java will complain with the error below:
$ cd pkgtest
$ java Hello
Error: Could not find or load main class Hello


Next, we will see how to import the package. Modify pkgtest/Hello.java and create ./JarTest.java as below:


To compile and run, run the following from ~/java_package directory:
$ javac JarTest.java
$ java JarTest
hello
hi


Next, we will create a jar library file and use this to compile and run. First, let's create the jar library. Run the following in ~/java_package directory
$ javac pkgtest/Hello.java
$ jar cvf pkgtest.jar pkgtest/Hello.class
added manifest
adding: pkgtest/Hello.class(in = 648) (out= 372)(deflated 42%)

This creates pkgtest.jar file, which contains Hello class.

Next, we will rename the pkgtest directory so that we don't compile from the source.
$ mv pkgtest pkgtest_bk

Let's see what happens if don't specify the jar file as we compile.
$ javac JarTest.java
JarTest.java:1: error: package pkgtest does not exist
import pkgtest.Hello;
              ^
JarTest.java:5: error: cannot find symbol
        Hello.print("hello");
        ^
  symbol:   variable Hello
  location: class JarTest
JarTest.java:6: error: cannot find symbol
        Hello h = new Hello();
        ^
  symbol:   class Hello
  location: class JarTest
JarTest.java:6: error: cannot find symbol
        Hello h = new Hello();
                      ^
  symbol:   class Hello
  location: class JarTest
4 errors

As you can see above, java compiler complains that it cannot find some symbols, since we renamed the package directory. To resolve this, let's link the jar file:
$ javac -cp ".:pkgtest.jar" JarTest.java
$ java -cp ".:pkgtest.jar" JarTest
hello
hi

Viola! Let's call it a day.

Useful Environment Variables for Mac OS X

Here is a list of some useful environment variables for Mac OS X:

C_INCLUDE_PATH: header search path for gcc

CPLUS_INCLUDE_PATH: header search path for g++

How to Deal with Argument List Too Long Error Message

In Mac OS X terminal,when you provide too many arguments to commands, you may get the error message below:

$ rm *.jpg
-bash: /bin/rm: Argument list too long

This is because there are too many jpg files in the directory, all of which are fed into rm arguments. Here are two ways to achieve it.

$ find . -name '*.jpg' -exec rm {} \;

$ find . -name '*.jpg' | xargs rm

Note that they will be slow, since we are basically calling the rm command as many times as the number of jpg files in the folder, but this works though!