Watching for hallucinations when using language models to generate Python code

I’m working on a pretty kick-butt project at work right now and I’m leveraging AI to help me build it out. I’m running into issues where I need someone with more knowledge, and this is where I’m leveraging language models.

One of the language modesl I use frequently hallucinated and told me straight up incorrect information!

The problem I’m trying to solve

As I’m developing this application, I’m learning a lot about threading and file locking. Admittedly, I’m not too familiar with either of these from an implementation standpoint.

My problem is that I have multiple threads and different processes in a microservice architecture accessing a shared file. The question I’m posing is how can I manage file locking across multiple processes and threads?

While the threading package handles this, it’s specific to threads, not other processes. Now it’s a question of what packages are thread safe and can be used across multiple processes?

This isn’t an easy question to answer, so I asked multiple language models such as ChatGPT and Claude 2. ChatGPT gave some solid responses, but Claude 2 gave even a better response, which prompted this week’s topic.

Claude 2 Response

Take a second to see what’s wrong with the response below (not the prompt).

If you made it all the way to the bottom you’ll see an additional parameter in the code it provided: lock_file = True

Holy smokes, after scratching my head for hours, I found a viable solution! I verified that this was actually a parameter by reading the Pandas documentation to double check and alas, it doesn’t exist.

Being slightly distraught, I (hesitantly) triple checked by asking ChatGPT:

Language model hallucinations

A while back, I wrote a newsletter called Bytes and Brew. The idea behind it was “learn how to use AI while drinking your cup of coffee” and was mostly surrounding how to leverage ChatGPT for every day use.

One of the editions was about LLM hallucinations: what they are, why it’s bad, and how you can limit them.

I’ll let the article explain details, but language models tend to hallucinate at times, or provide false information at times. This is a phenomenon that occurs when the model’s training data does not:

1. Sufficiently cover the topic in question, or

2. When the complexity of the question exceeds the model’s capabilities to interpret and respond based on the data it has been trained on.

Essentially, the model tries to “guess” an appropriate response, which can lead to inaccuracies.

What I’m building up to here is that this is where your knowledge about software engineering comes into play.

Realizing that Claude was producing incorrect information highlights a crucial idea of using language models in software development: use them as assistants rather than sole developers.

While language models are powerful, keep in mind they will never replace a developer, but rather add to their productivity. Language models excel at offering insights, generating code snippets, and explaining complex concepts.

The drawback with using language models is that they lack understanding and judgement that human developers are able to bring. It’s essential to use these tools in tandem with human expertise and not as a replacement.

Another issue I detected while prompting

I prompted ChatGPT before this hallucination on how I can use portalocker with pandas, and this is the code it gave me:

with open(file_path, 'a') as f:
    # Ensure file is exclusively locked
    portalocker.lock(f, portalocker.LOCK_EX)
    # Append without writing headers
    df.to_csv(f, header=False, index=False)
    # Unlock the file after writing
    portalocker.unlock(f)

In this code, we:

• Open the file,

• Lock it using an exclusive lock,

• Write to the file,

• Unlock it.

While in a normal context without pandas, this would be a perfectly acceptable solution. However, there’s a few things that doesn’t make sense:

• to_csv() handles the opening and closing of the file,

• We’re not using a context manager for locking, so we have to manually handle locking and unlocking the file.

After reviewing this code, I challenged ChatGPT and told it to explain itself and told it that the to_csv method handles the opening of the file. After some back and forth, I was able to settle on the following solution:

import portalocker

# Create the lock object
lock = portalocker.Lock(
   file, 
   mode = 'a', 
   flags = portalocker.LOCK_EX
)

# Lock and write to the csv file
with lock as file:
    df.to_csv(
        file, 
        mode = 'a', 
        header = False,
        index = False
    )

Maybe there’s a better way to handle this, as the file gets opened twice: once with pandas, another with portalocker. Who knows?