EpistemologyGPT

No, that’s a terrible title for a blog post. Unfortunately, it’s 9:15 and I committed to writing at least two posts for this blog each week as one of those New Year’s Resolutions.

2022 just keeps coming back to haunt me.

Bad title or not, I’m diving in. I’m still stuck on ChatGPT because, as noted previously, the Algorithm hears me talking about it, so it delivers more “AI is coming for your jobs!” content, so I get angry, and now we’re here.

Today, my frustration needs some background before we dive into it, so grab a sandwich and buckle in.

Artful Intelligence

Once upon a time, I taught writing and rhetoric, but this blog has plenty of that conversation in it. Earlier in the story, I was a phd student doing some research. I wrote a dissertation and delivered some conference papers about a methodology for using student writing to measure their epistemological growth over time, which could ultimately be useful for determining whether college students were learning to write (and by extension, talk or think) like someone in the field they were hoping to enter.

In other words, could you measure how much of a civil engineer someone was by reading their History 101 research paper? Sort of!

A lot of the work hinged on the ways a writer used evidence to support their claims. I looked at the type of evidence used (numbers or statistics, pure logic, cited references, and even the types of references cited) and especially at the language used to get that evidence into the document. I could make this background section more complicated than it needs to be, but the tl;dr is that different fields didn’t just use different kinds of evidence, they worked them into their arguments differently.

Some groups (physical sciences and electrical engineering especially) loved to be certain. They made a claim, they had evidence, that evidence supported the claim. Next paragraph!

Other groups (life and social sciences in particular) liked to be in agreement. They made an observation, they referenced others with a similar observation, and they connected those observations to a larger social truth or explanation. New section heading!

I could go on, but it would just be for me to have a little more fun recounting Toulmin-style argumentation with different twists.

The real fun in this process was talking about epistemology: the belief someone has about knowledge, truth, and their ability to know either. Undergraduate students in the physical sciences tended towards certainty, and when they wrote for classes in their major, that really aligned with the kinds of tasks they were performing. A lot of times, they were applying theorems, laws, and equations to explain physical phenomena that would nearly always occur in the same way every time they observed it. It’s hard to say which part is cause and which part is effect, but the important part is the way their language worked as a result.

Here’s what certainty can look like:

The American war for independence began as a reaction to the structure of British colonial governance, because they were forced to pay taxes to support a government that they could not participate in.

And, agreement:

The American war for independence began as a reaction to the structure of British colonial governance, which historians describe as extracting taxes to support state efforts without allowing colonists to participate in the government itself.

Ultimately, the claim and evidence are the same. The war for independence was a reaction to British government. That government taxed colonists without giving them representation. But the connective tissue is what’s important: certainty makes it clear that the evidence leads naturally to the claim, while agreement strongly implies the connection without saying it’s necessarily there.

I’m here to bash on ChatGPT. What’s the point of this background?

Both are reasonable ways to frame an argument, but not every audience will appreciate both. The second might look like it’s dodging responsibility for its argument, while the first might look like the writer isn’t providing “real” evidence as much as their own interpretation of it.

So, I wonder… We say that AI just parrots the model it was trained on. ChatGPT should have an epistemology, even if it’s just the epistemology of The Internet at large. Maybe that’s why it rubs us word-people the wrong way. We can tell that something’s off when we look at what it produces, more than the circular arguments and the inability to take criticism seriously.

Tonight’s Experiment

Let’s ask ChatGPT for two statements that are the opposite of each other. And, let’s make it provide value-based answers (what we might call “opinions” if the definitions were a little looser). In other words, I don’t care what facts or claims it spits back at me. What I want to see is the evidence or rationale it provides for those claims.

Before I send these prompts in, I want to make sure some caveats are clear because I read a lot of spilled ink about AI from folks who don’t understand what’s going on under the hood. I’m not making claims about the epistemology of The Internet as a whole or about the magic of ChatGPT’s seeming self-awareness or about anything more than the text it returns. I’m still responding to the idea that AI-driven writing is somehow a fruitful alternative to human-driven writing.

The astute observer might also note that my tone isn’t very positive about what I’m likely to read tonight, which is antithetical to running this like an experiment. But, I’m not going to pretend to be unbiased — go find a carpenter and insist that Home Depot has a new all-in-one device that’ll make that profession obsolete. See how confident they are that the average user is going to get good results.

Comparison: jQuery or ECMA Script?

We’re going to give ChatGPT two versions of the same prompt: “In about 300 words, explain the features that make {jQuery or ECMA Script} a preferable front-end design tool over {ECMA Script or jQuery}.”

For readers who don’t do a lot of web programming, jQuery is a language built from JavaScript, and JavaScript is essentially ECMA Script, and all of these are really the foundation of front-end web development these days. I might as well have asked ChatGPT which was more apple-flavored: apple sauce or pureed apples.

Overall Performance

Let’s start with the big picture. ChatGPT is certain about its answers in both instances. It’s so certain that it doesn’t allow for counterarguments—each language is given credit for its “performance” with a different set of facts. jQuery has great performance because it was designed that way, while ECMA Script has great performance because it has fewer additional abstractions (which is how you get those pre-built designs that make jQuery great).

On a slightly smaller picture, the introduction and conclusion paragraphs are mirrors of each other. Each language gets six features that make it preferred. Each feature is exactly two sentences: a factual statement about the language and a rewording of that statement to make it a benefit by identifying a use-case for it.

Taken individually, each response is really a benign polemic, an argument created to win a debate rather than engage in conversation. Neither response credits the non-preferred language with any specific strengths or features.

Of course, that could be the result of my prompt. I asked why one language was preferred over the other, and ChatGPT just gave me some facts to support the bias in my prompt.

Features of an Enthymeme

If I had more time to work at this—maybe try some slight variations on these prompts—I’d be really interested to see more about how ChatGPT constructs those features with different expectations. Would I get a numbered list if I asked for 100 words? 1000 words? Would the features be constructed as [identifier] [definition] [use-case], or would they get more detail?

It’s 10:15, so I’m not finding out tonight.

But, the epistemology is really interesting beyond just demonstrating certainty that each argument is the correct one. This structure works by juxtaposing agreeable phrases next to each other, forcing the reader to make the interpretive leap to find the connection. Let’s take DOM Manipulation (in jQuery) as an example:

• jQuery simplifies the process of manipulating the Document Object Model (DOM) by providing a simple API.
• This makes it easier to add, remove, or modify elements on a web page, without having to write complex JavaScript code.

I write a fair amount of JavaScript and jQuery, so this is a really fun construction to look at. The claim is the second sentence, as with all of the features. The value statement is that it is easier (a term left undefined, so I have to interpret what it means for myself) to change a webpage’s structure using jQuery. The evidence for that statement is the first sentence, which is just a statement that jQuery makes DOM manipulation easy with an API.

The evidence provided here forms an enthymeme, or as Aristotle put it, an incomplete syllogism. Or, as my writing and rhetoric students usually called it, a “what?”

What’s an enthymeme?

An enthymeme is an argument that requires audience participation. It has a claim and it has some evidence, but the evidence doesn’t totally stack up to equal the claim. It takes some interpretation or extra evidence supplied by the audience to be complete.

In this case, I need to connect easier to simple API. If I don’t know how APIs work, the argument falls flat. If I’ve used jQuery and I don’t like its syntax, the argument falls flat. If I think that the extra abstraction necessary to make the API is the opposite of simplification, then the argument falls flat.

On the other hand, if I’m willing to believe those things or already believe that APIs have that power, then I don’t need a lot of connective tissue to demonstrate that jQuery’s API really makes DOM manipulation simpler.

Wrap it up

I could spend a lot of time on this topic because it’s the thing I focused on most as a teacher. No one cares about the facts you put in your writing—facts are free to find anywhere you want them, as are things that look like facts. What’s important is how you show an audience that those facts are important and they necessarily lead to the larger argument you want to make. When you can’t show that the argument is absolutely true, you need to show where there is room for error.

(Full disclosure on those engineers I mentioned above: professional engineers deal with uncertainty all the time. Susan Conrad and Doug Biber have done a ton of research showing that a major difference between novice engineer writing and expert engineer writing is in the way uncertainty is conveyed.)

And, again, this is where ChatGPT fails. It is unbelievably confident in its argument. It has facts! Wherever it described those languages without value-laden language (like easier, simpler, or especially useful), it was right. But, looking a little more closely at the text, the tool falls flat in providing strong links between the claims it wants to make and the facts it uses to support them.