APIs, HTTP, & JSON

In this week's episode, Chris is joined by Gabe Berke-Williams to discuss all things API, HTTP, and JSON. Tune in to learn the tools, workflows, and approach that Chris, Gabe, and thoughtbot at large use to interact with APIs.

Note - This week's topic was requested by an Upcase subscriber and we'd love to do more like it. If you have any ideas for things you'd like to see then please drop us a line at help@upcase.com or by posting in the Upcase category on the forum.

JSONView Chrome Extension

JSONView is a Chrome extension that automatically formats JSON documents viewed in the browser, handles pretty-printing, expanding, and colorizing the document.

Great with larger JSON documents, e.g. Github API responses

cURL

The workhorse for making any HTTP request, including interacting with JSON APIs, is cURL. cURL is an executable that allows you to make any HTTP request from the command line. It is available on just about every platform, typically by default, and covers every single option, protocol, and flag out there.

Viewing the Headers

Amongst the many options, the ones we most often use are -i and -I to view the headers (-i includes the headers with the response, -I only shows the response headers).

Also View The Request Headers

By default cURL will only output content from the request, but if needed, you can also get cURL to display data about the request it is making using the -v flag, e.g. $ curl -v https://api.github.com/users/thoughtbot/repos.

Copy as cURL in Chrome Dev Tools

In some cases you may want to save off a request as made by your browser to allow you to replay it as a cURL request. Luckily, Chrome makes this easy with the Copy as cURL command in the Network tab of the dev tools.

Note, you'll need to check the Preserve Log option to save any request that cases a page refresh, such as submitting a form.

HTTPie

[httpie][] is a friendly cURL with syntax highlighting and a more intuitive syntax. It is a python library, but is packaged for use by non-python devs and in my case I was able to install it with

$ brew install httpie

The main nicety of HTTPie is that it will pretty-print, colorize, and automatically include the response headers. This makes it much more pleasant and user-friendly to play around with a new API that you're exploring.

It also has the added benefit that it is aware of piping and will disable the fancy printing, header display, etc if you pipe the stdout of HTTPie to another command or redirect to a file. Best of both worlds!

[httpie]: https://github.com/jakubroztocil/httpie

When to Use cURL Over HTTPie

In some cases HTTPie or other more user-friendly utilities may fall short, lack an option or configuration, and in those cases cURL remains the go-to utility.

cURL truly has every HTTP related option built in and can do anything you need, if perhaps require a bit of digging to figure it out. Luckily, the cURL man page documents all of the options and flags in great and readable detail. Check out $ man curl for the full man page.

Interacting with APIs In Ruby

Using cURL From Ruby

One of the most straightforward ways to make HTTP requests from Ruby is to use backticks to call out to cURL from Ruby, then handle the response as needed:

require "json"

THOUGHTBOT_REPOS_ENDPOINT = "https://api.github.com/users/thoughtbot/repos"

def with_curl
  response = `curl #{THOUGHTBOT_REPOS_ENDPOINT}`
  JSON.parse(response)
end

While it's nice to have this option available to leverage the full power of cURL, often you'll be better served by using a Ruby-specific HTTP library.

rest-client

[rest-client][] is a Ruby gem designed to make HTTP requests more friendly. It can be used for all variety of HTTP requests, from simple GET requests:

require "rest-client"
require "json"

THOUGHTBOT_REPOS_ENDPOINT = "https://api.github.com/users/thoughtbot/repos"

def with_rest_client
  raw_response = RestClient.get(THOUGHTBOT_REPOS_ENDPOINT)
  repos = JSON.parse(raw_response)
end

To significantly more complicated POST requests with headers and body:

def create_event(calendar)
  endpoint = "#{GOOGLE_CALENDAR_API_BASE}/#{calendar}/events"

  request_body = {
    start: { "date": start_date.strftime("%Y-%m-%d") },
    end: { "date": end_date.strftime("%Y-%m-%d") },
    summary: contact_name,
    description: details
  }

  headers = {
    "Authorization" => "Bearer #{auth_token}",
    "Content-Type" => "application/json"
  }

  RestClient.post(endpoint, request_body.to_json, headers)
end

[rest-client]: https://github.com/rest-client/rest-client

HTTParty

[HTTParty][] is another Ruby HTTP library that aims to simplify HTTP interactions. HTTParty provides a similar set of methods to RestClient, but has the added functionality that you can include HTTParty in your class and your class will now have HTTP methods available:

class StackExchange
  include HTTParty
  base_uri 'api.stackexchange.com'

  def initialize(service, page)
    @options = { query: {site: service, page: page} }
  end

  def questions
    self.class.get("/2.2/questions", @options)
  end
end

stack_exchange = StackExchange.new("stackoverflow", 1)
puts stack_exchange.questions

[HTTParty]: https://github.com/jnunemaker/httparty

Faraday

[Faraday][] is a more full featured HTTP library for Ruby that has a number of advanced features including:

• Wrapping any of multiple HTTP client adapters
• Block syntax for additional control over defining requests
• Ability to add both request and response middleware to wrap and process your requests and response.

[Faraday]: https://github.com/lostisland/faraday

JQ

[jq][] is sed for JSON. It allows you to slice and dice JSON documents to focus on the parts you need.

Sample:

[
  {
      "full_name": "upcase",
      "html_url": "https://github.com/thoughtbot/upcase"
  },
  {
      "full_name": "payload",
      "html_url": "https://github.com/thoughtbot/payload"
  },
  {
      "full_name": "upcase-exercises",
      "html_url": "https://github.com/thoughtbot/upcase-exercises"
  }
]

$ jq '.[] | select(.full_name | contains("upcase")) | .html_url'

The above jq command applied to the JSON array of repos will output:

https://github.com/thoughtbot/upcase
https://github.com/thoughtbot/upcase-exercises

Another example, this time piping from a cURL request into a jq select:

curl https://api.github.com/users/thoughtbot/repos | \
  jq '.[] | select(.full_name | contains("capybara")) | { full_name, html_url }'

JQ Resources

jq has great [documentation][] including a solid [tutorial][]. In addition, they even have a live playground app for testing in the browser: [jqplay][].

Along with the core jq docs, Gabe has written a great introductory post on the thoughtbot blog, [jq is sed for JSON][].

[jq]: http://stedolan.github.io/jq/ [jqplay]: https://jqplay.org/ [sed for JSON]: https://robots.thoughtbot.com/jq-is-sed-for-json [documentation]: http://stedolan.github.io/jq/manual/ [tutorial]: http://stedolan.github.io/jq/tutorial/ [jq is sed for JSON]: https://robots.thoughtbot.com/jq-is-sed-for-json

HTTP Test Applications

Paw

[Paw][] is a solid Mac app for defining and testing API requests. It allows you define and save off requests for replay and testing later.

[Paw]: https://luckymarmot.com/paw

Postman

[Postman][] is similar to Paw in that it is an application that allows you to define and save complex HTTP requests, but Postman runs as a Chrome application and is free.

It has the added benefit that you can save off and share request collections which is a great way to document and share knowledge within a team or about a service.

[Postman]: https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm

Interacting With JSON in Vim

From time to time I've found the need to edit JSON documents directly in Vim, for instance to build a fake API server, and in those cases the following tips have made that editing much more efficient.

Opening A URL In Vim

Vim can directly open the HTML document returned by GETing a URL, using cURL in the background to request the document, then opening it.

:e https://api.github.com/users/thoughtbot/repos

Vim Pretty Printed JSON

Combine this [JSON Syntax Plugin][] with the following snippet to pretty print JSON.

" Requires 'jq' (brew install jq)
function! s:PrettyJSON()
  %!jq .
  set filetype=json
endfunction
command! PrettyJSON :call <sid>PrettyJSON()

This can now be invoked as :PrettyJson<cr> on the Vim command line and will set the fileytpe to JSON and pretty print the document by piping it out to jq.

[JSON Syntax Plugin]: https://github.com/elzr/vim-json