Tab completion in GNU Readline: Ruby edition

Back in 2016, I wrote about the ways we’re using GNU Readline for tab completion in gitsh. The examples in that post, like the gitsh tab completion code, were written in C. I’d have preferred to use Ruby, but the complex tab completion in gitsh needed Readline features beyond those provided by the Ruby Readline module.

Having built on Ruby’s Readline module to add complex tab completion to gitsh, and then contributed those changes back to Ruby, I can finally write the post I wanted to write two years ago. What follows is an edited version of my original post, replacing all of that complex C with lovely Ruby.

GNU Readline is a powerful line editor with support for fancy editing commands, history, and tab completion. Even if you’re not familiar with the name Readline you might still be using it: it’s integrated into all kinds of tools including GNU Bash, various language REPLs, and our own gitsh project.

This post will talk you through the more advanced Readline tab completion features gitsh uses and show you how to use them in your own programs.

To avoid getting lost in the details of the gitsh code, we’ll use a simplified example application for this post.

Basic tab completion

To get us started, here’s the simplest Readline program I can think of. It uses Readline to get input from the user, echoes that input back, and then exits.

require "readline"

input = Readline.readline("> ", false)
if input
  puts "You entered: #{input}"
end

The Readline.readline method prompts the user for input, with all of Readline’s power behind it. This includes tab completion for file system paths. If you don’t want to complete anything more than filenames you don’t need to go any further than this.

It also includes history support: if you pass true as the second argument user’s will be able to quickly repeat their previous inputs.

Custom completion options

In gitsh—and many other programs that use Readline—it’s useful to be able to complete things other than paths. In gitsh, we’re interested in completing things like Git commands, branch names, and remotes. For the purpose of this example, let’s say we’re only interested in completing values from a fixed list of the names of some characters from The Hitchiker’s Guide to the Galaxy.

Here’s our expanded program with custom tab completion:

require "readline"

CHARACTER_NAMES = [
  "Arthur Dent",
  "Ford Prefect",
  "Tricia McMillan",
  "Zaphod Beeblebrox",
].freeze

Readline.completion_proc = proc do |input|
  CHARACTER_NAMES.select { |name| name.start_with?(input) }
end

puts "Who's your favourite Hitchiker's Guide character?"
input = Readline.readline("> ", false)
if input
  puts "You entered: #{input}"
end

We’re making use of a new Readline feature here: we’ve set a completion_proc to provide custom completion options.

When the user hits their tab key Readline will invoke the proc we’ve assigned to completion_proc. The partial argument we’re completing will be passed as an argument to the proc.

If we modify our completion_proc to print its argument, we’d see something like this:

Who's your favourite Hitchiker's Guide character?
> I like Arth⇥
input="Arth"

Note that we’re only passed "Arth", and not the whole input. Given this information, we need to return an Array of possible completions.

Quoting and escaping

Our current implementation works well enough when the user is entering the name of a single character. But what would happen if they needed to enter a list of characters, separated by spaces? How would we know if we were seeing a space between a character’s first name and last name, or a space between two different characters?

Shells like bash, zsh, and gitsh get around this with quoting and escaping.

We could quote each character’s name:

"Arthur Dent" "Ford Prefect"

Or we could escape the spaces that don’t indicate the start of a new character’s name:

Arthur\ Dent Ford\ Prefect

Quoting and escaping are important for tab completion. As we’ve seen, Readline passes only the last argument of the user’s input to our completion proc. If we want to support quoting and escaping we need some way of telling Readline if the space separating two words counts as the start of a new argument. We also need to make sure that when we complete an argument containing a space that it is appropriately escaped.

The cases we need to cover are:

Adding quoting support

Quoting is easier than escaping, so let’s tackle that first.

All we need to do is tell Readline which characters our program uses as delimiters for quoted strings, by setting Readline.completer_quote_characters:

Readline.completer_quote_characters = "\"'"

Now, when we press tab within a single- or double-quoted string, Readline will pass everything after the opening quote to our completion proc.

It’ll even close the quotes for us if there’s only one possible completion, or leave them open if there are several to choose from.

Adding escaping support

The first thing we need to do to support escaping is to make sure that the completion options we return are properly escaped.

We’d expect unquoted input to produce escaped output, and quoted input to produce unescaped but quoted output:

Conveniently, we’ve already set Readline.completer_quote_characters, so Readline is aware of whether or not we are completing a quoted string.

We can modify our completion_proc to call Readline.completion_quote_character which will tell us what the argument we’re completing was quoted with. If the completion_quote_character returns nil, we’re completing an unquoted argument, so we need to produce escaped character names:

Readline.completion_proc = proc do |input|
  options = CHARACTER_NAMES
  if Readline.completion_quote_character.nil?
    options = options.map { |o| o.gsub(" ", "\\ ") }
  end

  options.select { |name| name.start_with?(input) }
end

If Readline has seen an un-closed quote Readline.completion_quote_character will will return the appropriate quote character (in our case ' or ", since those are the characters we listed in completer_quote_characters). If completion_quote_character is nil, we know we’re not completing a quoted argument.

Detecting escaped word breaks

This is getting pretty good, but we’re still left with one case we can’t handle. If the user input contains a space that’s escaped:

Readline will still see the space as an argument boundary. Our completion function will be passed "D", when we want it to be passed "Arthur\ D".

To handle this, we need to give Readline a proc that it can call to discover if the space between words is escaped, which we can do with the Readline.quoting_detection_proc setting.

Our quoting_detection_proc takes the whole line of input and the index of the space that might indicate a break between arguments, or a quote character that might indicate the start of a quoted string. It should return true if the character is quoted, and false if it is quoted:

Readline.quoting_detection_proc = proc do |input, index|
  index > 0 && input[index - 1] === "\\" &&
    !Readline.quoting_detection_proc.call(input, index-1)
end

It’s worth noting that this implementation is recursive. In many shells, it’s possible to escape the \ character with another \ character. The sequence \\ represents a literal \ and doesn’t escape the character that follows it. The recursion makes sure we handle any number of \ characters before a space, and always do the right thing.

When is quoting_detection_proc called?

Readline won’t make use of quoting_detection_proc unless it believes some kind of quoting or escaping is being used in the user’s input.

There are several practical implications of this:

• quoting_detection_proc is only ever called if Readline.completer_quote_characters is set.

• quoting_detection_proc is only ever called if the input contains an unclosed quoted string, or a literal \ character. This limits the kind of escaping schemes we can implement to those that use a \ in some way.

Which characters separate arguments?

Readline will only invoke quoting_detection_proc with characters that would, if unescaped, indicate a break between arguments.

For our quoting_detection_proc implementation to work, we need to customise the list of word break characters:

Readline.completer_word_break_characters = " "

Notice that we’ve been happily completing space-separated arguments from the very first example, so why do we need to explicitly specify this now?

The default value of completer_word_break_characters includes the \ character, which we use for escaping. If encountering a \ indicated a word break, we wouldn’t get very far with escaped spaces; Readline would include the space in the value passed to our completion proc, but stop at the \.

An alternative solution to this problem would be to decrement Readline.point in our quoting_detection_proc, but since we don’t need \ characters to act as word breaks, we can happily remove them from completer_word_break_characters.

Running the example code

The code in this blog post won’t run correctly everywhere. The version of Ruby is important, and so is the line editing library that Ruby was compiled against.

GNU Readline vs. libedit

Ruby can use two different line editors: GNU Readline, which has more features but a more restrictive license; and libedit, which has fewer features and more bugs but a more permissive license. Everything in this post beyond basic tab completion requires GNU Readline.

You can determine which line editor your version of Ruby is compiled against using the following command:

ruby -r readline -e Readline.vi_editing_mode?

If the command exits silently, then your Ruby is built with GNU Readline. If it fails with a NotImplementedError, then it is built with libedit.

Ruby version

Most of these features have been available for a long time, but there are two more recent additions, which were extracted from gitsh and contributed upstream.

• Readline.quoting_detection_proc was added in Ruby 2.4 (see Ruby patch #12659).
• Readline.completion_quote_character was merged into Ruby trunk in September 2018, and should be in a production version of Ruby soon (see Ruby patch #13050).