When AI doesn't help!

Recently, Our team addressed a pending technical debt of refactoring our legacy codebase from the unmaintained Moment.js to a more modern Day.js. Despite the Moment implementation working satisfactorily, a strategic decision was made to upgrade due to our desire to reuse large sections of this legacy codebase for a new project and we did not want Moment to be a part of the new project.

We recognize that many existing projects may continue to use Moment, but we would like to discourage Moment from being used in new projects going forward.

We followed the guidance of Moment docs to use alternatives. The team took a cautious approach addressed a slew of test failures, manually tested the upgrade and eventually shipped it. Two weeks after shipping, we hit a production issue that scarily traced back to this library refactor.

Timezone bug

As mentioned, we had this Day.js based solution to retrieve UTC offsets for both the local app and the server (hypothetically in Singapore) as per the code below.

const now = dayjs()
const localUtcOffset = now.utcOffset()
const serverUtcOffset = now.tz('Asia/Singapore').utcOffset()

The local UTC offset (localUtcOffset) returned correctly as 60, but the server offset (serverUtcOffset) unexpectedly returned 0 instead of 480, which is the correct UTC offset for Singapore (UTC+8). Due to this bug, time representation in some areas of our mobile app was off by 8 hours. Naturally, I turned to ChatGPT for some guidance and insights.

ChatGPT Solution 1

Day.js requires explicit UTC mode before applying .tz(). Your current approach doesn’t properly load the timezone offset when calling .utcOffset(). Instead, you should ensure that the time instance is properly initialized in the correct timezone.

const now = dayjs().tz('Asia/Singapore') // Correctly set timezone
const localUtcOffset = dayjs().utcOffset()
const serverUtcOffset = now.utcOffset() // Now correctly returns 480

The above code didn’t work and serverUtcOffset continued to return 0.

ChatGPT Solution 2

You must explicitly create the dayjs object in UTC mode before applying .tz()

const now = dayjs().utc() // Ensure base time is in UTC
const localUtcOffset = now.utcOffset()
const serverUtcOffset = now.tz('Asia/Singapore').utcOffset()

There was lengthy explanation with nice headings such as Fix, Why does this work?, etc,. but serverUtcOffset continued to return 0 instead of 480.

Probing further

As I continued to have conversations, the solutions started becoming diverse such as:

• Pass true (non existent) parameter to .tz.
• Use @digitalbazaar/timezone (hallucinated non existent library) to obtain timezone information.
• Use different third party libraries such as Luxon instead of Day.js.
• Use other Internationalisation libraries to obtain the timezone offsets.

I tried a few other Large Language Models (LLM) such as DeepSeek and all of them kept repeating like ChatGPT after a few conversations. I was confused by the eventual responses from multiple LLMs, all returning the same code however I tried to steer them away. To test the code differently, removing the context of our mobile app, I opened a Day.js playground on my browser and executed the code suggested by ChatGPT. Surprisingly, it returned 480, not 0 like it did in our app. After some research on StackOverflow (which feels like a relic in the age of LLMs) and reading some GitHub issues, I discovered the reason for this divergent behaviour.

Modern React Native mobile apps don’t use the standard JavaScript engines such as JavaScriptCore or V8. Instead, it uses the Hermes engine, which is optimised for mobile performance but lacks certain internationalization features. This is why the DayJS code returned 0 instead of 480 in our app.

After evaluation some of the solutions posted by other developers, we added moment-timezone back in areas of the code that needed timezones.

What can I do?

Here are the things that worked for me trying to debug the root cause of this pesky timezone problem.

Fail fast, move on

When working with modern frameworks, understanding the limitations of information that can be packed in an LLM is important. Some of the limitations can be:

• JavaScript Engines (Hermes vs V8)
• New architecture based differences
• Platform specific integrations (Push notifications, Deep links)
• Native Modules Support

With better prompting, we may be able to extract a bit more information from the LLM, but for certain problems, either the LLM has the information to solve our problems or not. When we start reaching that point of diminishing returns like in my case, it is better to stop and move on.

It’s time to pick that old debugging hat from the cupboard, dust it and wear it.

Sharpen debugging skills

Validate your boundaries

The timezone bug we encountered is a perfect example of what can go wrong at system boundaries. System boundaries are where different components, libraries, or platforms meet, and they’re often fertile ground for bugs. Understanding these boundaries and their types is practically useful for preventing and solving tricky issues.

Take a break

Sometimes, the most powerful debugging technique is stepping away from the problem. I’ve experienced this many times where problems that were unsolvable despite hours of debugging, become clear and solutions found within minutes of returning with a well rested and fresh mind the next day. Remember to document your current progress (A simple progress.md in your codebase) somewhere. This post contains more on taking breaks and other ways you can disconnect from the problem to gain a fresh perspective later.

If you’re curious about more strategies like this, we highly recommend our debugging series. These posts are packed with strategies, examples, and practical techniques that will transform how you approach tough problems. Whether you’re debugging issues unsolvable by LLMs or traditional software bugs, these principles remain invaluable.

Document the struggle

A few recent real world examples that stumped our team were:

• Upgrading MacOS and Xcode versions broke our App builds with a “This bundle is invalid” error.
• Upgrading the third party Location library stopped requesting GPS access on Android.
• The DayJS upgrade problem in this blog post

These issues often originate due to changes at the system boundaries, but the path to resolution isn’t straightforward and there’s a struggle. When developers invest significant time investigating and solving such problems, documenting the journey is useful. This documentation serves as a knowledge base for team members when they encounter similar challenges in the future.

In addition to the above, developers may also consider documenting project specific quirks and limitations that AI struggles with. Create a dedicated “Troubleshooting” section in your project README, covering complex version upgrades, platform specific issues, and native module integration challenges. Regular updates to this documentation as the ecosystem evolves will save development time and prevent recurring issues.

Conduct an incident review

If problems impact real users on production, it’s worth conducting an incident review process that involves systematically documenting and analysing the disruption in order to learn from it. The process includes capturing the timeline of events, technical root cause analysis, and identifying both immediate fixes and long-term improvements. It’s important to focus on the overall improvements of how a team works, rather than individual blame, helping teams grow stronger from each incident.

Conclusion

While LLMs are trained on public codebases and documentation, they can often overlook quirks like the example above. Despite feeding the context of a React Native app, they could not help me. While LLMs process terabytes of data, their knowledge remains confined to patterns in their training sets. Real-world coding problems involve niche errors that can escape AI’s grasp. This incident also reminds me how AI should be treated as a collaborator, not an oracle of information. I still love AI, leverage its speed and use it as a tool to aid me during development. But reading documentation, experimenting with code snippets, and engaging in developer forums are useful skills for developers to cultivate. Technology can augment but cannot replace the adaptive problem-solving nature of humans.