--- title: 'Tab completion in GNU Readline: Ruby edition' teaser: 'Learn about the new Readline features coming to Ruby soon. ' tags: gitsh,readline,ruby author: George Brocklehurst published_on: 2018-10-19 --- Back in 2016, I wrote about the ways we're using [GNU Readline for tab completion][c-edition] in [gitsh]. The examples in that post, like the gitsh tab completion code, were written in C. I'd have preferred to use Ruby, but the complex tab completion in gitsh needed Readline features beyond those provided by the [Ruby `Readline` module][ruby-readline]. Having built on Ruby's `Readline` module to add complex tab completion to gitsh, and then contributed those changes back to Ruby, I can finally write the post I wanted to write two years ago. What follows is an edited version of my original post, replacing all of that complex C with lovely Ruby. [c-edition]: https://thoughtbot.com/blog/tab-completion-in-gnu-readline [ruby-readline]: http://ruby-doc.org/stdlib-2.3.3/libdoc/readline/rdoc/Readline.html --- [GNU Readline][readline] is a powerful [line editor][line-editor] with support for [fancy editing commands][cheatsheet], history, and tab completion. Even if you're not familiar with the name Readline you might still be using it: it's integrated into all kinds of tools including [GNU Bash][bash], various language REPLs, and our own [gitsh] project. This post will talk you through the more advanced Readline tab completion features gitsh uses and show you how to use them in your own programs. To avoid getting lost in the details of the gitsh code, we'll use a simplified [example application][example-repo] for this post. [bash]: https://www.gnu.org/software/bash/ [cheatsheet]: http://readline.kablamo.org/emacs.html [example-repo]: https://github.com/georgebrock/readline-example [gitsh]: https://github.com/thoughtbot/gitsh [line-editor]: https://en.wikipedia.org/wiki/GNU_Readline [readline]: https://cnswww.cns.cwru.edu/php/chet/readline/rltop.html ## Basic tab completion To get us started, here's the simplest Readline program I can think of. It uses Readline to get input from the user, echoes that input back, and then exits.

require "readline"

input = Readline.readline("> ", false)
if input
  puts "You entered: #{input}"
end

The [`Readline.readline`][docs-readline] method prompts the user for input, with all of Readline's power behind it. This includes tab completion for file system paths. If you don't want to complete anything more than filenames you don't need to go any further than this. It also includes history support: if you pass `true` as the second argument user's will be able to quickly repeat their previous inputs. ### Custom completion options In gitsh—and many other programs that use Readline—it's useful to be able to complete things other than paths. In gitsh, we're interested in completing things like Git commands, branch names, and remotes. For the purpose of this example, let's say we're only interested in completing values from a fixed list of the names of some characters from The Hitchiker's Guide to the Galaxy. Here's our expanded program with custom tab completion:

require "readline"

CHARACTER_NAMES = [
  "Arthur Dent",
  "Ford Prefect",
  "Tricia McMillan",
  "Zaphod Beeblebrox",
].freeze

Readline.completion_proc = proc do |input|
  CHARACTER_NAMES.select { |name| name.start_with?(input) }
end

puts "Who's your favourite Hitchiker's Guide character?"
input = Readline.readline("> ", false)
if input
  puts "You entered: #{input}"
end

main.rb at revision 6435ff4

We're making use of a new Readline feature here: we've set a [`completion_proc`][docs-completion_proc] to provide custom completion options. When the user hits their tab key Readline will invoke the proc we've assigned to `completion_proc`. The partial argument we're completing will be passed as an argument to the proc. If we modify our `completion_proc` to print its argument, we'd see something like this:

Who's your favourite Hitchiker's Guide character?
> I like Arth⇥
input="Arth"

Output from completion_proc modified to print arguments

Note that we're only passed `"Arth"`, and not the whole input. Given this information, we need to return an `Array` of possible completions. ## Quoting and escaping Our current implementation works well enough when the user is entering the name of a single character. But what would happen if they needed to enter a list of characters, separated by spaces? How would we know if we were seeing a space between a character's first name and last name, or a space between two different characters? Shells like bash, zsh, and gitsh get around this with quoting and escaping. We could quote each character's name:

"Arthur Dent" "Ford Prefect"

Or we could escape the spaces that don't indicate the start of a new character's name:

Arthur\ Dent Ford\ Prefect

Quoting and escaping are important for tab completion. As we've seen, Readline passes only the last argument of the user's input to our completion proc. If we want to support quoting and escaping we need some way of telling Readline if the space separating two words counts as the start of a new argument. We also need to make sure that when we complete an argument containing a space that it is appropriately escaped. The cases we need to cover are:

Input	Expected output
`"Arthu⇥`	`"Arthur Dent"`
`"Arthur D⇥`	`"Arthur Dent"`
`Arthu⇥`	`Arthur\ Dent`
`Arthur\ D⇥`	`Arthur\ Dent`

### Adding quoting support Quoting is easier than escaping, so let's tackle that first. All we need to do is tell Readline which characters our program uses as delimiters for quoted strings, by setting [`Readline.completer_quote_characters`][docs-completer_quote_characters]:

Readline.completer_quote_characters = "\"'"

Changes introduced by revision b9d4d6c

Now, when we press tab within a single- or double-quoted string, Readline will pass everything after the opening quote to our completion proc. It'll even close the quotes for us if there's only one possible completion, or leave them open if there are several to choose from. ### Adding escaping support The first thing we need to do to support escaping is to make sure that the completion options we return are properly escaped. We'd expect unquoted input to produce escaped output, and quoted input to produce unescaped but quoted output:

Input	Expected output
`Arthu⇥`	`Arthur\ Dent`
`"Arthu⇥`	`"Arthur Dent"`

Conveniently, we've already set `Readline.completer_quote_characters`, so Readline is aware of whether or not we are completing a quoted string. We can modify our `completion_proc` to call [`Readline.completion_quote_character`][docs-completion_quote_character] which will tell us what the argument we're completing was quoted with. If the `completion_quote_character` returns `nil`, we're completing an unquoted argument, so we need to produce escaped character names:

Readline.completion_proc = proc do |input|
  options = CHARACTER_NAMES
  if Readline.completion_quote_character.nil?
    options = options.map { |o| o.gsub(" ", "\\ ") }
  end

  options.select { |name| name.start_with?(input) }
end

Changes introduced by revision 00358bd

If Readline has seen an un-closed quote `Readline.completion_quote_character` will will return the appropriate quote character (in our case `'` or `"`, since those are the characters we listed in `completer_quote_characters`). If `completion_quote_character` is `nil`, we know we're not completing a quoted argument. ### Detecting escaped word breaks This is getting pretty good, but we're still left with one case we can't handle. If the user input contains a space that's escaped:

Input	Expected output
`Arthur\ D⇥`	`Arthur\ Dent`

Readline will still see the space as an argument boundary. Our completion function will be passed `"D"`, when we want it to be passed `"Arthur\ D"`. To handle this, we need to give Readline a proc that it can call to discover if the space between words is escaped, which we can do with the [`Readline.quoting_detection_proc`][docs-quoting_detection_proc] setting. Our `quoting_detection_proc` takes the whole line of input and the index of the space that might indicate a break between arguments, or a quote character that might indicate the start of a quoted string. It should return `true` if the character is quoted, and `false` if it is quoted:

Readline.quoting_detection_proc = proc do |input, index|
  index > 0 && input[index - 1] === "\\" &&
    !Readline.quoting_detection_proc.call(input, index-1)
end

From main.rb at revision 4117ec0

It's worth noting that this implementation is recursive. In many shells, it's possible to escape the `\` character with another `\` character. The sequence `\\` represents a literal `\` and doesn't escape the character that follows it. The recursion makes sure we handle any number of `\` characters before a space, and always do the right thing. #### When is `quoting_detection_proc` called? Readline won't make use of `quoting_detection_proc` unless it believes some kind of quoting or escaping is being used in the user's input. There are several practical implications of this: * `quoting_detection_proc` is only ever called if `Readline.completer_quote_characters` is set. * `quoting_detection_proc` is only ever called if the input contains an unclosed quoted string, or a literal `\` character. This limits the kind of escaping schemes we can implement to those that use a `\` in some way. #### Which characters separate arguments? Readline will only invoke `quoting_detection_proc` with characters that would, if unescaped, indicate a break between arguments. For our `quoting_detection_proc` implementation to work, we need to customise the list of word break characters:

Readline.completer_word_break_characters = " "

From main.rb at revision 4117ec0

Notice that we've been happily completing space-separated arguments from the very first example, so why do we need to explicitly specify this now? The default value of [`completer_word_break_characters`][docs-completer_word_break_characters] includes the `\` character, which we use for escaping. If encountering a `\` indicated a word break, we wouldn't get very far with escaped spaces; Readline would include the space in the value passed to our completion proc, but stop at the `\`. An alternative solution to this problem would be to decrement [`Readline.point`][docs-point] in our `quoting_detection_proc`, but since we don't need `\` characters to act as word breaks, we can happily remove them from `completer_word_break_characters`. ## Running the example code The code in this blog post won't run correctly everywhere. The version of Ruby is important, and so is the line editing library that Ruby was compiled against. ### GNU Readline vs. libedit Ruby can use two different line editors: GNU Readline, which has more features but a more restrictive license; and [libedit][libedit], which has fewer features and more bugs but a more permissive license. Everything in this post beyond basic tab completion requires GNU Readline. You can determine which line editor your version of Ruby is compiled against using the following command: ```shell ruby -r readline -e Readline.vi_editing_mode? ``` If the command exits silently, then your Ruby is built with GNU Readline. If it fails with a `NotImplementedError`, then it is built with libedit. [libedit]: http://thrysoee.dk/editline/ ### Ruby version Most of these features have been available for a long time, but there are two more recent additions, which were extracted from gitsh and contributed upstream. * `Readline.quoting_detection_proc` was added in Ruby 2.4 (see [Ruby patch #12659][patch-12659]). * `Readline.completion_quote_character` was merged into Ruby trunk in September 2018, and should be in a production version of Ruby soon (see [Ruby patch #13050][patch-13050]). [patch-12659]: https://bugs.ruby-lang.org/issues/12659 [patch-13050]: https://bugs.ruby-lang.org/issues/13050 [docs-readline]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-readline [docs-completion_proc]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-completion_proc-3D [docs-completer_quote_characters]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-completer_quote_characters-3D [docs-completion_quote_character]: https://github.com/ruby/ruby/blob/trunk/ext/readline/readline.c#L1312-L1324 [docs-quoting_detection_proc]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-quoting_detection_proc-3D [docs-completer_word_break_characters]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-completer_word_break_characters-3D [docs-point]: https://ruby-doc.org/stdlib-2.4.0_rc1/libdoc/readline/rdoc/Readline.html#method-c-point