In this episode of our AI in Focus series, Chad was joined by former Rails core contributor Kasper Timm Hansen as they paired on a real Rails app. They explored the possibility of replacing a dedicated CRM data enrichment service with structured data calls to an LLM. Along the way, they discussed Kasper’s ActiveJob::Performs gem, his innovative Riffing approach to software development as well as shared some hard won Rails wisdom. Read on for the highlights and watch the full replay on YouTube.
Why replace a dedicated service with ChatGPT?
At thoughtbot, we have an internal Rails app called Hub which we use to augment our CRM, amongst other things. We have a service that automatically enriches CRM opportunities with metadata about the company we’re interested in: their industry, founding date, funding level, etc. Up until now, we’ve used a dedicated serviced like Apollo for this.
But now that LLMs like ChatGPT are getting better at returning structured data, it raised an interesting question: could we get the same data or better using ChatGPT’s function calling interface? And would it be reliable enough?
How we structured the Rails integration
Our existing Rails structure had a clear entry point:
OpportunityEnrichmentService.new(opportunity).enrich
This relied on a ApolloAiService, which returned a hash of fields like industry, total_funding, and annual_revenue from Apollo’s API. The idea for this live stream was to replace that class with one that would call ChatGPT using the Ruby OpenAI gem, but keeping the data structure intact.
We set about TTD-ing a new class called LlmEnrichment, which would be responsible for fetching the same information using OpenAI’s nascent function-calling mechanism.
ChatGPT’s function calling mode
LLMs are evolving rapidly beyond responding simply with freeform text. With new function calling modes, we can define a schema that tells the model exactly what structured fields to return. Here’s an example of such a structured data call to ChatGPT using the OpenAI gem:
class LlmEnrichment
def initialize(organization_name)
@organization_name = organization_name
end
def enrich
response = client.chat(
parameters: {
model: "gpt-4",
messages: [
{
role: "user",
content: "Populate the organization data for #{@organization_name}."
}
],
tools: [
{
type: "function",
function: {
name: "enrich_organization",
description: "Populates the data of an organization",
parameters: {
type: "object",
properties: {
founded_on: {
type: "string",
description: "ISO8601 date company was founded"
},
funding_level: {
type: "string",
description: "The total amount of funding raised",
enum: ["bootstrapped", "seed", "series_a", "unknown"]
},
},
required: [
"founded_on",
"funding_level"
]
}
}
}
],
tool_choice: {
type: "function",
function: {
name: "enrich_organization"
}
}
}
)
JSON.parse(response.dig("choices", 0, "message", "tool_calls", 0, "function", "arguments"), symbolize_names: true)
end
private
def client
@client ||= OpenAI::Client.new
end
end
This lets the LLM act like it’s “calling” a known function with specific arguments. We don’t even need to run that function in our app. We just read the arguments ChatGPT prepared and use them as our enrichment data.
Testing with TDD
Test-driven development is at the heart of our development approach at thoughtbot, and so we started the session by writing a test for LlmEnrichment. The specs stubbed the OpenAI response and focused on verifying the returned data hash structure.
We quickly ran into some quirks as dates came back with slight variations on every request, not all fields were returned each time and we didn’t have any idea of the LLM’s confidence in it’s answers.
Required to the rescue
The fix for the missing data fields was to specify an array of keys in the required parameter:
required: [
"founded_on",
"funding_level",
"industry"
]
Additionally, OpenAI offers a strict mode intended to “ensure function calls reliably adhere to the function schema, instead of being best effort.”
With the required keys firmly in force, the responses were now at least consistent in their formatting, if not yet in their content. With thoughtbot itself as the test company, the model managed to get our founding year correct each time (2003!), but there was some inconsistency in the funding level. It couldn’t make up its mind if we were bootstrapped or privately funded or unknown!
Riffing to explore the design
To break through the initial uncertainty of where to put this LLM call, we explored a technique called riffing. Invented and taught by Kasper, riffing involves opening a scratch file and free-forming your thoughts in code. It’s like wire-framing for backend logic: fast, low-fidelity, and highly collaborative.
Instead of debating structure too early, riffing let us quickly try out class and method names, explore possible API shapes and spot test edge cases.
Kasper teaches this as a process to help teams ship better code faster, and this session was a great live demo of it in action.
Bonus: ActiveJob::Performs
As a bonus, we also explored Kasper’s gem ActiveJob::Performs, which simplifies job structure in Rails by tying a job method directly to an instance method:
class OpportunityEnricher < ApplicationRecord
performs def enrich
# your logic here
end
end
It could be a good way to cut out ActiveJob boilerplate, reinforce naming conventions, and add structure to your background jobs.
The payoff: structured data, flexible control
By the end of the session, we had achieved our aim of integrating an LLM into thoughtbot’s Hub app that could fetch structured company metadata. The data may not have been deterministic like that from other dedicated CRM enrichment services, and therefore it would need a little more work to ensure its reliability.
However we’re excited by the opportunity for richer prompting and more detailed responses that LLM integrations offer. One proven approach to improving reliability is be requiring the LLM to provide confidence values for each response, as well as allowing “unknown”.
Looking to integrate AI into your Rails app?
Whether you’re building internal tools or customer-facing features, we’re excited by the possibilities that LLM-powered structured data retrieval is opening up. Get in touch with thoughtbot and begin imagining the future of your AI-enhanced app today.