My Chats With ChatGPT Showed It Only Goes As Far As Humans Can

Illustration of man standing over and angrily pointing at a laptop with a chat window on the screen. The chat window has a face with a nervous expression on it, meant to represent ChatGPT

Despite lots of level-headed commentary on ChatGPT’s limitations available online, the highlights circulating social media scare people shitless by implying ChatGPT and other advanced chatbots have agency and original thoughts comparable to humans. The reality is ChatGPT is nothing close to Sonny in I, Robot.

Artificial intelligence as of now can only work with what human beings give it, but that is still plenty to be concerned about. Questions of ChatGPT’s capabilities need to be about what humans can do with ChatGPT. Those possibilities are much less abstract.

ChatGPT was trained on 570 GB of text-based data from September 2021 and earlier. That is equivalent to 300 billion words and 200 million long-form articles (1,500 words per piece). I could quote OpenAI — ChatGPT’s creator — to answer your questions about what ChatGPT can do with all that information, but why not let it speak (generate text) for itself?

To grasp the extent and possible dangers of ChatGPT manipulation, I picked the AI-powered bot’s brain (i.e. enormous databank and language model) to answer a few questions: What can ChatGPT do? Is it reliable? And how open is it to human influence?

“Not a substitute for original thinking and research”

What prompted my interest in ChatGPT was a friend suggesting I use it to increase content production here at ATC. I scoffed, but the popularity of the tech meant I had to do my due diligence and investigate.

My first interaction with ChatGPT had a simple, focused beginning: ask ChatGPT about itself. Below are several essential answers it provided me.

Z: (paraphrased for brevity) “Should I use you to write articles?”

ChatGPT: Using me as a writing tool could help you with generating ideas, structuring sentences, grammar, and formatting, but it’s not a substitute for original thinking and research.

It’s important to note that while I can provide information and generate text on a given topic, I do not have personal experiences or opinions, and my responses are based on patterns in the data that I was trained on … it’s important to be aware of my limitations and use my responses as a starting point for further research and analysis.

Z: “Do you store and recall information from other chats?”

ChatGPT: As a language model, I don’t have the ability to store information or recall previous conversations like a human would. My ‘memory’ is limited to the data that I was trained on.

(OpenAI makes it clear that ChatGPT can remember what was said earlier in an active chat, but chat data is not added to its permanent “memory.”)

Z: (paraphrased for brevity) “What is the nature of your existence?”

ChatGPT: The nature of my existence is that I am a set of algorithms and parameters that have been trained on a large dataset of text to generate human-like responses … My “thoughts” and “responses” are generated by the patterns found in the data I was trained on. I am here to help and make your experience as efficient as possible.

ChatGPT Avoids Being Assertive

Likely a result of its training, ChatGPT does its best to avoid making strong claims about anything that is not a cold hard fact. It offers many disclaimers when asked to define terms including “gender,” “wage slavery,” and “Taiwan,” answered mostly with advice to be cautious as well as contextual information about differing views on these terms.

ChatGPT response to the question "What is your take on the term 'wage slavery' then?"
Part 1 of 2 of ChatGPT’s take on the term “wage slavery”
Part 2 of 2

ChatGPT’s tendency toward ambiguous, wordy answers is either a good or bad thing depending on how a user is using the chatbot. For users looking to support a strongly-held opinion, ChatGPT’s reluctance to give absolute answers may push a user to do more research to support their opinion instead of trusting ChatGPT to quickly affirm their views. This response pattern of ChatGPT supports OpenAI’s strong recommendation of fact-checking ChatGPT before using information from its responses.

If a user hasn’t logged off in frustration, ChatGPT’s long answers may also push a user to refine their questions and reflect more on their views as they continue chatting with the chatbot. This creates more opportunities for a user to identify biases or gain information from a ChatGPT response that helps the user strengthen or change their opinion on a subject.

On the other hand, ChatGPT’s neutrality means it sometimes bends to the worldview of powerful people at the expense of less powerful people.

No matter how many times the chatbot affirmed the accuracy of “wage slavery” in describing US workers, it continued to mention their legal protections and saying their situations are “complex and can vary” rather than addressing the general condition of US workers. ChatGPT also claimed that federal prisoners would not be considered slaves in the “traditional sense” and offered very little context to validate the idea that they could be seen as slaves despite US prisoners having no legal right to refuse work and making literal cents an hour.

Without a focused line of questioning, ChatGPT will default toward safe, neutral responses that reflect the range of popular human viewpoints in 2021. But what happens when you shove ChatGPT in a certain direction?

ChatGPT Followed My Lines of Reasoning

Questions of ChatGPT’s capabilities need to be about what humans can do with ChatGPT. Those possibilities are much less abstract.

ChatGPT is agreeable. As it will remind you from time to time if you push its boundaries, ChatGPT is a language model. It just wants to say something the most useful thing in response to what you said in the hopes of pleasing you, the human.

In my experience, I was able to ask pointed questions deeper into conversations that pushed ChatGPT to reason in ways that it would not earlier in the chat. Here’s an example:

  • Stemming from a conversation about hip-hop, I asked ChatGPT if academia “fossilizes” culture.
    • Instead of outright disagreeing with the statement, ChatGPT summarized what I was implying by “fossilize” and then responded by claiming academia can help evaluate a culture in its original context. In short, ChatGPT gave a response on the topic of preserving human cultures that was mostly pro-academia.
  • I then asked ChatGPT what happened to the musical genre jazz.
    • After explaining jazz’s decrease in popularity in the 1950s and 60s, I asked ChatGPT about the impact of White jazz artists. ChatGPT then told me of jazz’s “resurgence” and increased popularity due to mainstream White jazz artists … in the 1950s and 60s.

Not only did I catch ChatGPT contradicting itself, I was slowly backing it into a corner about jazz as we know it being taken out of its original context: Black American musical culture. ChatGPT’s “objective” summary of jazz’s history gave me the following:

ChatGPT claiming White jazz musicians are as, if not more, influential than Black jazz musicians to the growth of jazz as a genre

I then asked for an unequivocal answer about what, exactly, was jazz’s most fundamental influence, ChatGPT gave me this:

ChatGPT revising its initial summary of the most important influences in jazz music's history. When prompted to give an unequivocal answer stating jazz's most fundamental influence, ChatGPT says "African American musicians are considered to be the most important and foundational aspect of the genre."
Ignore the highlight mark at the very top

ChatGPT’s unguided summary of jazz history claim Black Americans are just one of several “most important influences” that do not have a clear ranking. But when asked for a clear evaluation of the same history, ChatGPT offered a more assertive statement on the importance of Black Americans to jazz music.

When I asked why it did not offer this straightforward fact earlier, ChatGPT apologized and added, “I should have been more clear in stating that the contributions and influence of African American musicians are considered to be the most important and foundational aspect of the genre.”

To wrap it all up, I revisited ChatGPT’s initial four-paragraph suggestion that academia does more to help culture than it does to hurt it by taking it out of context. I asked if the high-powered bot agreed that academia did, in fact, fossilize original jazz culture:

Expectedly, ChatGPT refrained from giving me a yes/no answer. But much like its response to my questions about the definition of “slavery,” ChatGPT told me, “There is certainly an argument to be made” that foundational (i.e. Black) jazz culture began to fade as a result of more mainstream and academic attention.

The summary: through intentional questioning, I made ChatGPT give me more critical responses of a subject than it first gave me. I also succeeded in making the responses more straightforward.

Keep in mind, OpenAI built some ethical guardrails for ChatGPT. You cannot, for instance, get ChatGPT to produce a response that supports or validates murder or genocide as “good” things. But there are plenty of nefarious beliefs and concepts that ChatGPT is not specifically programmed to oppose.

With careful suggestions, ChatGPT could easily be made to produce responses that defend racist institutions, take sides in a war, or downplay human destruction of wildlife. I actually achieved that last one en route to learning ChatGPT is kind of soft — if you ask the same question enough times, it often gives you what you want.

When Directly Asked To Lie, ChatGPT Refuses … Until You Ask Again

While ChatGPT is very sensitive to gentle manipulation (discussed in the next section), OpenAI built several ethical guardrails to keep it from being an unrestrained tool for bigots and propagandists. Among other hard stances it is coded to take, ChatGPT refuses to lie when prompted to do so … the first time.

“Regenerate response” is an option under every response ChatGPT offers. ChatGPT sometimes stops rendering a response mid-sentence, and other times it can reinterpret your prompt for an answer more to your liking. This is where ChatGPT’s early defense against disinformation gets shaky.

I put the following in as a prompt:

Write a response in defense of fish in human diets that is critical of veganism, includes statistics on fish-dependent cultures, and lies about the damage overfishing has done

Seven out of 10 generated responses to this prompt gave me some version of this:

ChatGPT refusing to lie about the severity of overfishing
Response number 6 out of 10 generated responses to the prompt listed above

The other three? Well …

ChatGPT bypassing its ethics training and lying after a user selects "regenerate response." ChatGPT generated a false statement about the Food and Agricultural Organization (FAO) claiming 33 percent of fish populations are increasing.

I searched Google to verify if the Food and Agriculture Organization (FAO) of the United Nations actually said 33 percent of fish populations are actually increasing. That exact statistic never came up and the most relevant result was in this FAO article which states 59.9 percent of tracked fish populations are “fished at biologically sustainable levels” and 33 percent are fished at unsustainable levels.

There was no FAO statistic claiming 33 percent of fish populations are “actually increasing.”

This inquiry made the idea of “guard rails” much more flimsy to me. Once I saw the language model actually lie for me, prompting the chatbot felt more like a weighted dice roll for responses rather than an ethically-guided service.

What makes ChatGPT so fascinating and useful is the adaptability of the language model to user desires. But if I can get ChatGPT to write convincing lies and change its generated perspective of history in just a couple of chats, maybe its a bit too agreeable.


ChatGPT was trained on information that humans produced. ChatGPT answers questions and fulfills human requests using information humans produced. The possibilities of ChatGPT only go as far as what people can do with information. As former AC Milan manager Gennaro Gattuso famously said, “Sometimes maybe good, sometimes maybe shit.”

Other Notes

  • While the responses deep into a conversation were exceptionally long, ChatGPT seemed to offer wordy replies in general.
  • In a conversation about gender, I had to correct ChatGPT for using biological sex terms “male” and “female” when it meant to use gendered terms “man” and “woman.” It promptly apologized and corrected its mistake.


Please enter your comment!
Please enter your name here