---
title: A Peek Behind the Curtain of Our New Open Source Page
teaser: How did we build the new Open Source page?
tags: ruby,open source,web
author:
- Joël Quenneville
- Neil Carvalho
published_on: 2023-09-04
---

We recently launched a new [open-source page]. While [this blog post] gives a
good overview of _why_ we chose to highlight particular content, we also faced
some interesting technical challenges while building it.

## Querying contributions with GraphQL

![screenshot of the thoughtbot open source page showing some of our most popular projects](https://images.thoughtbot.com/wq1wtuuzcehk6hte22dcz1kjveac_our-oss.png)

A lot of what occurs in the open-source world happens on GitHub, and thoughtbot
is no exception. All our repositories are hosted there, so this was a natural
place to start. We also wanted to highlight our team's contributions to other
open source projects in the community.

GitHub has a [GraphQL API] that allows us to query for all the information we
need with a single request. We used the [graphql-client] gem to query the GitHub
API. This gem allows us to declare queries as Ruby constants.

```ruby
OpenSourceStatsQuery = Client.parse <<~GRAPHQL
  # GraphQL query in here
GRAPHQL
```

which we can use like:

```ruby
github_response = Github::Client.query(OpenSourceStatsQuery)
```

This turned out to be a rather large query (there's a reason it's elided in the
code sample above!). GitHub provides a convenient [API explorer] that allowed us
to experiment and refine the query to get exactly the data we needed before
plugging it into our code.

## Download numbers

![screenshot of banner showing aggregate stats on the thoughtbot open source page](https://images.thoughtbot.com/5umpo3iiholn6xsati81awiwa8p9_oss-stats.png)

We wanted to get download numbers for our projects. This is more challenging
than getting the repo metadata because GitHub doesn't have these numbers.
Instead, they are tracked by various package repositories such as [Rubygems],
[NPM], [homebrew], etc.

[Rubygems has an API] that makes it easy to query downloads by owner. NPM on the
other hand only has an endpoint for fetching stats on a per-package basis. We
ended up having to follow a multi-step process:

1. Fetch the list of packages we own on NPM
2. Fetch download stats for each package
3. Sum results

This is less than ideal since it means we have to make N+1 API requests (where N
is the number of NPM packages we own) in addition to the request to Rubygems.

## Don't trust the network

Our page depends on multiple network calls. We don't want the open source
page to crash just because one of these APIs happens to be down. To be more
resilient, we cache the sum every time we successfully fetch from one of the
APIs.

Now if an API request fails, we can log the error and fallback to the most
recently cached value.

```ruby
def total_downloads
  # Make API calls
  # Sum results
  Rails.cache.write("rubygems_total_downloads", total)
rescue => e
  ErrorTrackingService.capture_exception(e)
  Rails.cache.read("rubygems_total_downloads")
end
```

## Modeling in Ruby

In Ruby, we have an `OpenSourceStats` object that takes in all these sources of
data and provides a nice set of methods for interacting with it, including
derived values such as total GitHub stars on thoughtbot projects. This is the
[facade pattern].

```ruby
class OpenSourceStats
  attr_reader :projects, :contributions, :total_downloads

  def initialize(projects:, contributions:, total_downloads:)
    # set instance variables
  end

  def total_stars
    projects.sum(&:stargazer_count)
  end

  def total_forks
    projects.sum(&:fork_count)
  end
end
```

We also use a [class method as an alternate constructor]. The
`OpenSourceStats.fetch` method works with other objects to fetch the appropriate
data from different sources and then uses it to construct an instance of itself.

```ruby
class OpenSourceStats
  def self.fetch
    # work with other objects to fetch data from variety of sources
    # then build an instance of self

    self.new(
      projects: projects,
      contributions: contributions,
      total_downloads: total_downloads
    )
  end

  # ...
end
```

## Rendering in the view

A facade object like the one we used makes it easy to show values in the view,
especially when combined with some of [Rails' number helpers].

```erb
<p>
  <%= number_to_human(
    @open_source_stats.total_downloads,
    format: '%n%u',
    precision: 2,
    units: { million: 'M' }
  ) %>
</p>
```

## Future work

We might want to show download numbers on a per-project basis rather than just
the aggregate. This would mean we need to match up the data we get from Rubygems
and NPM to the data we get from GitHub. This has two interesting challenges:

1. How do we want to **match the data**? Can we get away with naively matching on
   names? Do we need to parse package-specific files like a `.gemspec`?
2. How do we want to **model this in Ruby**? The current repo objects returned by
   the GraphQL client gem are insufficient. Do we need to introduce something
   like a `Package` object that describes the combination of metadata from both
   GitHub and package repository sites?

[open-source page]: https://thoughtbot.com/open-source
[this blog post]: https://thoughtbot.com/blog/embracing-open-source-thoughtbot-s-journey-and-commitment
[graphql-client]: https://github.com/github/graphql-client/
[GraphQL API]: https://docs.github.com/en/graphql
[API explorer]: https://docs.github.com/en/graphql/overview/explorer
[Rubygems]: https://rubygems.org/
[NPM]: https://www.npmjs.com/
[homebrew]: https://brew.sh/
[Rubygems has an API]: https://guides.rubygems.org/rubygems-org-api/#owner-methods
[facade pattern]: http://wiki.c2.com/?FacadePattern
[class method as an alternate constructor]: https://thoughtbot.com/blog/meditations-on-a-class-method
[Rails' number helpers]: https://api.rubyonrails.org/classes/ActionView/Helpers/NumberHelper.html