Wow, I can't believe I wasn't aware of these super useful commands till now!
The following command will change back to your previous directory
$ cd -
The following command will push the current directory to stack and move to the designated directory
$ pushd /new/directory/to/go/to
The following directory will pop from the stack and move to that directory