I tend to install command line software a lot, particularly when starting at a new client. For instance I install new versions of Ruby or other development infrastructure every few months. The kind of software that mostly works without me having to think about it, but when it does break, it’s obscure.
It’s not possible (or desirable) to memorise every possible error in these situations, so I’ve needed to develop some skill to address problems when they do come up.
One pattern I find myself following frequently: when I hit a roadblock, I try to get more information via verbose output, and then search the web using the verbose output (check this handy blog post about using error messages to search). While the additional information is often hard to understand, there’s someone on the web who explains enough of it for me to take action.
At some point I learned that just about every piece of software does output, including small command line utilities.
Debugging SSH
For example: recently I learned about ssh -v while investigating why a private key that I copied to a remote server was not working. I had no problems with the same key locally. The -v turns on verbose output.
With the verbose output turned on I was able to see that a simple ssh git@github.com whose output normally looks like this:
⚡ ssh git@github.com
PTY allocation request failed on channel 0
Hi iftheshoefritz! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.
… involves a whole lot more behind the scenes:
⚡ ssh -vT git@github.com
OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data ~/.ssh/config
debug1: ~/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
... and 50+ more lines after that
In this particular case, the connection to github worked perfectly from my local machine, but the same key did not allow me to connect to github from my remote server.
Careful reading the verbose SSH output lead to spotting this: debug1: key_parse_private2: no end marker. I didn’t understand, but the Internet knew. I found an issue on an open source project where someone was fighting the same battle, and I realised that a script I used had damaged the key file subtly (such that I didn’t spot the difference side by side).
Extending the pattern to other commands
Back to verbosity in general: many other commands that have an option to give more verbose output, for instance
bundle install --verbose
gem install --verbose
yard --verbose
pod install --verbose
patch --verbose
asdf install ruby 2.6.5 --verbose
You may be noticing a pattern here. Command line authors often follow conventions for their bare-bones UX, and one of those is --verbose (--verbose is the GNU way. -v is the Unix equivalent; you can up the verbosity in some commands in increments by adding more vs, as in ssh -vv).
I don’t memorise these, it’s usually enough to know that they exist. I found my option yesterday by typing man ssh or by googling: “verbose output SSH” (I honestly don’t remember which).
Just using the extra information that command line utilities hide by default has made a huge difference to my life as a consultant. It’s surprising the amount of variety that comes up even between people working with the same stack, so it really helps to be able to dig down to a deeper level to understand the differences.
It’s a massive win to be able to learn more about what might be going wrong without needing to wait for a client to be available to help. Sometimes I end up solving problems that no-one at the client had even realised existed. The next time a new developer joins their team, I’m there with mysterious knowledge, ready to help.
This b(log) post also appeared on my personal blog at iftheshoefritz.com
