Month: February 2023

Well, I’ve been trying to get ChatGPT to work with me. I haven’t fed it any work-related secrets, but I did try to get it to reproduce something close to a thing I wrote at work. I didn’t have much success, even after I quit writing about it.

I also gave ChatGPT a couple of softball comparison prompts to see how it would reason its way through an opinion-based argument on a subject it had ample access to information about. I wasn’t really impressed with the results, but I could see using the responses as the foundation for making a low-stakes decision.

So, I’m giving it one more public test before I start writing about other stuff on this blog. I want it to help me generate some content for a fantasy game setting I’ve been working on for a while at a different domain.

I keep seeing people argue that AI isn’t great at doing the real writing, but it can be excellent at spurring on ideas or steering content that writers have trouble developing. I’m not sure that I buy that. The problems I’ve seen in the tool so far show that the model is pretty limited. It can’t reason deeply, but it can stack together a bunch of individual arguments. It doesn’t have much awareness of audience or context unless you feed it that information, and even then, it might ignore what you provide. It doesn’t really know where prompts are going, so it can’t “steer” a conversation so much as provide a hard surface to bounce ideas from.

Basically, it’s a brick wall. I’m talking to a wall here. But, if I can’t expect it to be creative, let’s see what it can do for me.

Prompt: grizz the ogre runs a tavern in a high fantasy imaginary world. grizz has an mba, but he is a pacifist. what three hobbies does grizz enjoy the most?

This personality was generated a few posts ago. I like the idea of “Grizz the pacifist Ogre who runs a fantasy Applebee’s.” That could be fun. I want to dive in a little more, now. I gave Grizz an MBA because ogres tend to be portrayed as dumb, slow-witted, and monstrous, or they go the Shrek route and get portrayed as dumb, uncouth, and jovial. An MBA seems like a good counterbalance: Grizz has some analytical skill, some business acumen, and probably a five-year plan.

ChatGPT’s responses will always be on the left and in red.

Prompt: using the information from the previous prompt and response: grizz the ogre has a problem that he cannot solve himself. he needs to send a party of four adventurers on a quest, and those adventurers must be willing to risk life and limb for a keg of ale and a good reputation. what quest does he offer them?

I know that ChatGPT can remember what it has been prompted for and what it’s returned in a single chat thread, and I regularly use that as part of my prompt engineering.

Prompt: using the information from the previous response: what herb or spice used in brewing grows in dense, marshy forest? and what dangers might adventurers face if they tried to retrieve enough of that ingredient for grizz the ogre to make 50 gallons of ale?

Prompt: using the previous response: how much bog-myrtle would be necessary to brew 50 gallons of ale?

Prompt: using information from the previous prompts and responses: if a tribe of goblin brewers guarded a stand of trees with ample bog-myrtle growing in it, what means would they use to defend their crop? if they would be willing to barter or trade for 20 pounds of fresh bog-myrtle, what would they accept as a reasonable offer?

Prompt: using the previous prompts and responses: what threats or dangers would a tribe of goblin brewers living in a dense, marshy forest have to contend with on a regular basis?

Useful?

This may be the most stereotypical, low-level fantasy RPG adventure narrative, but I kind of like it. I appreciate some of the small details, like bog-myrtle being a real ingredient (Ed. note: does not grow in the Pacific Northwest) and having a pretty easy conversion into a real recipe. I really like having an amount that needs to be retrieved, because this provides an interesting angle on the adventure that isn’t typical: how do you bring back twenty pounds of fresh plant matter?

So, the way I see it, ChatGPT has given me one clear resolution that is common in fantasy RPGs: kill the goblins, take their bog-myrtle. If they are brewers, though, maybe there’s another way. They might have preserved or dried bog-myrtle, even if they aren’t willing to trade it away (20 pounds of fresh plant matter would still reduce to several pounds of dried leaves).

That’s a good one-shot adventure, and a pretty good writing task for me to document creating in another post.

No, that’s a terrible title for a blog post. Unfortunately, it’s 9:15 and I committed to writing at least two posts for this blog each week as one of those New Year’s Resolutions.

2022 just keeps coming back to haunt me.

Bad title or not, I’m diving in. I’m still stuck on ChatGPT because, as noted previously, the Algorithm hears me talking about it, so it delivers more “AI is coming for your jobs!” content, so I get angry, and now we’re here.

Today, my frustration needs some background before we dive into it, so grab a sandwich and buckle in.

Artful Intelligence

Once upon a time, I taught writing and rhetoric, but this blog has plenty of that conversation in it. Earlier in the story, I was a phd student doing some research. I wrote a dissertation and delivered some conference papers about a methodology for using student writing to measure their epistemological growth over time, which could ultimately be useful for determining whether college students were learning to write (and by extension, talk or think) like someone in the field they were hoping to enter.

In other words, could you measure how much of a civil engineer someone was by reading their History 101 research paper? Sort of!

A lot of the work hinged on the ways a writer used evidence to support their claims. I looked at the type of evidence used (numbers or statistics, pure logic, cited references, and even the types of references cited) and especially at the language used to get that evidence into the document. I could make this background section more complicated than it needs to be, but the tl;dr is that different fields didn’t just use different kinds of evidence, they worked them into their arguments differently.

Some groups (physical sciences and electrical engineering especially) loved to be certain. They made a claim, they had evidence, that evidence supported the claim. Next paragraph!

Other groups (life and social sciences in particular) liked to be in agreement. They made an observation, they referenced others with a similar observation, and they connected those observations to a larger social truth or explanation. New section heading!

I could go on, but it would just be for me to have a little more fun recounting Toulmin-style argumentation with different twists.

The real fun in this process was talking about epistemology: the belief someone has about knowledge, truth, and their ability to know either. Undergraduate students in the physical sciences tended towards certainty, and when they wrote for classes in their major, that really aligned with the kinds of tasks they were performing. A lot of times, they were applying theorems, laws, and equations to explain physical phenomena that would nearly always occur in the same way every time they observed it. It’s hard to say which part is cause and which part is effect, but the important part is the way their language worked as a result.

Here’s what certainty can look like:

The American war for independence began as a reaction to the structure of British colonial governance, because they were forced to pay taxes to support a government that they could not participate in.

And, agreement:

The American war for independence began as a reaction to the structure of British colonial governance, which historians describe as extracting taxes to support state efforts without allowing colonists to participate in the government itself.

Ultimately, the claim and evidence are the same. The war for independence was a reaction to British government. That government taxed colonists without giving them representation. But the connective tissue is what’s important: certainty makes it clear that the evidence leads naturally to the claim, while agreement strongly implies the connection without saying it’s necessarily there.

I’m here to bash on ChatGPT. What’s the point of this background?

Both are reasonable ways to frame an argument, but not every audience will appreciate both. The second might look like it’s dodging responsibility for its argument, while the first might look like the writer isn’t providing “real” evidence as much as their own interpretation of it.

So, I wonder… We say that AI just parrots the model it was trained on. ChatGPT should have an epistemology, even if it’s just the epistemology of The Internet at large. Maybe that’s why it rubs us word-people the wrong way. We can tell that something’s off when we look at what it produces, more than the circular arguments and the inability to take criticism seriously.

Tonight’s Experiment

Let’s ask ChatGPT for two statements that are the opposite of each other. And, let’s make it provide value-based answers (what we might call “opinions” if the definitions were a little looser). In other words, I don’t care what facts or claims it spits back at me. What I want to see is the evidence or rationale it provides for those claims.

Before I send these prompts in, I want to make sure some caveats are clear because I read a lot of spilled ink about AI from folks who don’t understand what’s going on under the hood. I’m not making claims about the epistemology of The Internet as a whole or about the magic of ChatGPT’s seeming self-awareness or about anything more than the text it returns. I’m still responding to the idea that AI-driven writing is somehow a fruitful alternative to human-driven writing.

The astute observer might also note that my tone isn’t very positive about what I’m likely to read tonight, which is antithetical to running this like an experiment. But, I’m not going to pretend to be unbiased — go find a carpenter and insist that Home Depot has a new all-in-one device that’ll make that profession obsolete. See how confident they are that the average user is going to get good results.

Comparison: jQuery or ECMA Script?

We’re going to give ChatGPT two versions of the same prompt: “In about 300 words, explain the features that make {jQuery or ECMA Script} a preferable front-end design tool over {ECMA Script or jQuery}.”

For readers who don’t do a lot of web programming, jQuery is a language built from JavaScript, and JavaScript is essentially ECMA Script, and all of these are really the foundation of front-end web development these days. I might as well have asked ChatGPT which was more apple-flavored: apple sauce or pureed apples.

Overall Performance

Let’s start with the big picture. ChatGPT is certain about its answers in both instances. It’s so certain that it doesn’t allow for counterarguments—each language is given credit for its “performance” with a different set of facts. jQuery has great performance because it was designed that way, while ECMA Script has great performance because it has fewer additional abstractions (which is how you get those pre-built designs that make jQuery great).

On a slightly smaller picture, the introduction and conclusion paragraphs are mirrors of each other. Each language gets six features that make it preferred. Each feature is exactly two sentences: a factual statement about the language and a rewording of that statement to make it a benefit by identifying a use-case for it.

Taken individually, each response is really a benign polemic, an argument created to win a debate rather than engage in conversation. Neither response credits the non-preferred language with any specific strengths or features.

Of course, that could be the result of my prompt. I asked why one language was preferred over the other, and ChatGPT just gave me some facts to support the bias in my prompt.

Features of an Enthymeme

If I had more time to work at this—maybe try some slight variations on these prompts—I’d be really interested to see more about how ChatGPT constructs those features with different expectations. Would I get a numbered list if I asked for 100 words? 1000 words? Would the features be constructed as [identifier] [definition] [use-case], or would they get more detail?

It’s 10:15, so I’m not finding out tonight.

But, the epistemology is really interesting beyond just demonstrating certainty that each argument is the correct one. This structure works by juxtaposing agreeable phrases next to each other, forcing the reader to make the interpretive leap to find the connection. Let’s take DOM Manipulation (in jQuery) as an example:

• jQuery simplifies the process of manipulating the Document Object Model (DOM) by providing a simple API.
• This makes it easier to add, remove, or modify elements on a web page, without having to write complex JavaScript code.

I write a fair amount of JavaScript and jQuery, so this is a really fun construction to look at. The claim is the second sentence, as with all of the features. The value statement is that it is easier (a term left undefined, so I have to interpret what it means for myself) to change a webpage’s structure using jQuery. The evidence for that statement is the first sentence, which is just a statement that jQuery makes DOM manipulation easy with an API.

The evidence provided here forms an enthymeme, or as Aristotle put it, an incomplete syllogism. Or, as my writing and rhetoric students usually called it, a “what?”

What’s an enthymeme?

An enthymeme is an argument that requires audience participation. It has a claim and it has some evidence, but the evidence doesn’t totally stack up to equal the claim. It takes some interpretation or extra evidence supplied by the audience to be complete.

In this case, I need to connect easier to simple API. If I don’t know how APIs work, the argument falls flat. If I’ve used jQuery and I don’t like its syntax, the argument falls flat. If I think that the extra abstraction necessary to make the API is the opposite of simplification, then the argument falls flat.

On the other hand, if I’m willing to believe those things or already believe that APIs have that power, then I don’t need a lot of connective tissue to demonstrate that jQuery’s API really makes DOM manipulation simpler.

Wrap it up

I could spend a lot of time on this topic because it’s the thing I focused on most as a teacher. No one cares about the facts you put in your writing—facts are free to find anywhere you want them, as are things that look like facts. What’s important is how you show an audience that those facts are important and they necessarily lead to the larger argument you want to make. When you can’t show that the argument is absolutely true, you need to show where there is room for error.

(Full disclosure on those engineers I mentioned above: professional engineers deal with uncertainty all the time. Susan Conrad and Doug Biber have done a ton of research showing that a major difference between novice engineer writing and expert engineer writing is in the way uncertainty is conveyed.)

And, again, this is where ChatGPT fails. It is unbelievably confident in its argument. It has facts! Wherever it described those languages without value-laden language (like easier, simpler, or especially useful), it was right. But, looking a little more closely at the text, the tool falls flat in providing strong links between the claims it wants to make and the facts it uses to support them.

Ok, I threw ChatGPT under the bus in my last post. I gave it something that I thought was a pretty simple task (write an 800 word rant about artificial intelligence replacing writers) and pointed out a lot of the flaws in the output. I made the claim that professional writers should feel pretty safe because the AI can produce text, but not good content. Individual sentences held together well, and the rant itself almost looked convincing.

To paraphrase Harvey Keitel, if we start sticking our big noses in it, the subterfuge won’t last, but at a glance, the output appears to be what we wanted.

I keep thinking about it, though. Today, I was getting ready to write a short D&D adventure set in a tavern run by an ogre with a clever name (I don’t have the name yet, maybe I’ll ask ChatGPT for that?). I set up a prompt with a few variant options in it, mostly trying out different art styles because I haven’t found a Stable Diffusion model that can create an ogre that doesn’t look like Shrek or Joe Rogan.

This one needs some Photoshop work, but I think I’ll run with it. Or maybe I’ll write a bit of backstory to explain the pipe in the ear.

My point is that I’ve got 40-50 images in my output and half a dozen prompts behind them. I’m satisfied enough to accept this, but I’m not terribly happy with it. If I was in charge of approving the art for something going to print, I wouldn’t accept this.

This makes me wonder about the point I made in that last post. Specifically, how much time would it take to get ChatGPT to spit out something that resembled something I get paid to produce? That seems like a fun experiment, so let’s give it a whirl.

The Target

I write a lot of things that I can’t share directly here. I mean, most of the stuff I produce is public—LinkedIn posts, the content those posts link to, emails, product flyers, product webpages, and more—but some of it is updated regularly enough that I don’t want to have a version locked in time over here.

One thing I write that isn’t ever changed is Grid Configurator News. Three or four times a year, that software gets an update, and I write a news post, email, and other documents to announce the release to customers using the tool. The news posts are not overly complex or long, and they are designed to highlight key benefits that anyone involved in setting relays should appreciate.

This makes it a pretty good target text to produce with AI. I don’t want a factual answer to a question (something that the model is explicitly trained to do). I don’t want a clickbait headline or web article. I don’t want a narrative that goes where the gradient descent function leads it.

I want something that answers three specific needs in 300-350 words, just like I was asked to do for the first Grid Configurator News post.

• Explain the meaning and value of a modern software interface
• Highlight the value of being able to find and edit a relay setting quickly
• Explain the value of being able to set all of the devices in a substation at one time

Now, I won’t accept just any coherent response here. I keep pretty good records of how I spend my time (just look through the backlog of this blog) for lots of reasons. I spent two and a half hours of writing time on that first news post, from initial conversations about it, through reviews, to a final, accepted document. So, if I’m going to judge ChatGPT’s output, I need a rubric similar to what my work was judged by.

Modern software interface	Most of the software in use now was originally designed for Windows 3.x or Windows 95. There are many reasons why the GUIs are not updated, but the design is actually becoming a hurdle for young engineers who have to learn the idiosyncracies of a design scheme that phased out of popular software before they were born.
Edit settings quickly	In typical graphical setting software, engineers have to connect to a device, then figure out what settings group their setting is in, find that group, find the setting, then figure out if the setting they found is (a) what they want and (b) correct. There isn’t really a CTRL + F unless you homebrew your own software to read the settings files.
Set a full substation	A full substation can have hundreds or thousands of devices. Setting them individually can take weeks, particularly in substations where an engineer has to plug their laptop into each individual relay or connect to individual relays using a command prompt.

Oh, and if you want a copy of that Grid News post without clicking through the link above, I’ve provided the text at the bottom of this post.

Prompt Engineering

In my last post, I described prompt engineering as the process of iteratively refining your prompt to get the thing you’re looking for. It takes a lot of time and effort, and it takes some expertise in the thing you’re creating to know (a) what to ask for and (b) if you got it.

It’s like the time I tried to order a fancy coffee for my spouse. Not only did I mangle the order and ask for the wrong thing, but I knew so little about what I was ordering that I didn’t recognize the barista gave me something entirely different from what I ordered.

So, to judge ChatGPT’s output here, I need to plan to spend some time refining my prompts until I get something acceptable or until it’s clear that this was a waste of time.

It took me two and a half hours to go from blank page to finished product, with quite a bit of “prompt engineering” from my project teammates helping me refine the messaging in that news post. The basic framework of that first post was done in about half an hour, with phrasing and language adjustments over the next few weeks as the software team weighed in. So, let’s see where I can get in about the same amount of time.

Prompt 1: 20 minutes (Prompting and Evaluating)

write a 300 word news update about software for setting electrical relays. the software is modern and intuitive. the software allows users to find settings easily. the software allows users to set many devices at one time.

Nope. This wouldn’t fly. For one, it’s making claims that aren’t true about users. For another, the language is too over-the-top for this company (something that my last post said AI would have a hard time really figuring out, unless you trained it on a super narrow model). Finally, the output has a lot of the problems I critiqued about the rant I mentioned up top: it makes claims without really defining its terms or providing evidence.

Rejected. Try again with the feedback above.

Prompt 2: 30 minutes (Refining prompt, evaluating output)

use the previous prompt and output, but remove references to electricians. the software is only used by relay engineers. define "simple and user-friendly interface" using language similar to interface descriptions for modern web browsers or tablet-friendly software. use more conservative language. the software will only be used to set relays made by the company making this software.

Well, we’re approaching an hour into this little experiment. I could ask for a third round of pretty significant revisions, but let’s take a look at what my next prompt is going to have to include:

• Information to support the introductory point of why this software exists, what need it addresses.
• Specific language describing a “modern, intuitive interface.” In other words, I’m back to doing the content writing.
• I’m mentally prepared to revise paragraph 3 on my own. I think this output is about as close as it can get without me telling it exact phrases to use.
• I need to provide a lot of background information that I brought to the project: changing NERC cybersecurity requirements, changes in engineer demographics, the effects of grid modernization on substation design.

If you want it done right…

I mean, look, I could keep going with this exercise. I could maybe find a different target text to produce. Maybe I could use ChatGPT to get me started on the next Grid News article (but in no way will I trust it to just produce the thing on its own). But is it really saving any time or effort? If AI like ChatGPT is supposed to replace copywriters, is it going to produce “the same quality” of work that ChatGPT argued it would make when I asked for that rant?

I asked for a pretty simple thing in a genre with well-defined features. The AI didn’t follow the features well, and it didn’t follow my prompts all that tightly. Where it did follow the prompt closely is exactly where I wanted it to use that famed 750 terabyte training set and its 500 gigabyte model to make some inferences and do the thing you pay a copywriter to do.

When the Grid Configurator software team sat down with me to show me how the program worked and talk about why customers would want it, they did not spoonfeed me precisely the language that they wanted me to write because they aren’t writers. It’s not their job and it’s not their area of expertise. They did tell me similar things to what I included in the prompts above, and I got to see the program in action (something ChatGPT is unable to do).

But, they didn’t tell me about NERC CIP requirements. I knew that from other projects. They didn’t tell me about changing engineer demographics. I knew that because I taught basic interface design. They didn’t say anything about grid modernization, because that’s just a thing I should know about as an SEL employee. (Gratuitous link is for a video I wrote the script for and am pretty darned proud of.)

In other words, I am (in February, 2023) paid $76k per year because I can be shown a piece of software, handed the specs, and asked to write a concise description of it that should get customers interested in it. I can do that quickly—all told, each Grid Configurator update is about 6 hours of work time for five or so pieces of content. I can infer the value of features and ask the software team and other subject-matter experts if my inferences are accurate.

But that’s not a fair comparison! ChatGPT is free!

Is ChatGPT free or just no-cost? I spent an hour getting that AI-generated news article to a place I got to as a writer in half the time. I, a human, had to do that work. If this was my job, I wouldn’t be doing it for free. ChatGPT wrote those text outputs in maybe 30 seconds each time. When I say it took 20 minutes for round one, that was five minutes writing a prompt and fifteen minutes evaluating what I’d need to fix in the prompt to get a better result. The 30 minutes I spent on round two included ten minutes of prompt writing (which would have been longer if I didn’t know I could tell the program to re-use previous input and output) and twenty minutes of figuring out what to do with the output.

Here’s a fun question. Which skillset should cost more to hire: a copywriter with enough mental flexibility to learn the ins and outs of an industry, or someone who can write prompts and evaluate the output with the same expertise in writing as a trained copywriter?

Postscript

Oh, and if you’re wondering if I’ll close that ogre loop from the top of the post:

Post-Postscript

Ok, so I used to teach web writing and I currently work as a content writer. I fully understand that the thing I’m complaining about was written specifically to elicit my complaints, and by responding to this stuff, I’m feeding data to an algorithm that says this subject needs to be amplified everywhere on the web.

Now that I’ve identified the bear trap, I’m going to see how hard I can press the lever in the middle.

This turned into a rant, but it goes places. I promise.

If AI takes my job, my job should be automated

Let’s just get this out of the way. If my job—specifically the job of writing marketing materials—can be automated without losing quality in the end product, then automate it.

Seriously. I have other skills to fall back on, as does every other professional writer out there. A lot of us were trained to think that we don’t have any other “real” skills, but there aren’t many professions out there that require someone to become expert-enough in a subject to write convincingly about it to other experts, only to transition into entirely new subjects later that day. I’m not an electrical engineer, but thanks to a year and change writing for that crowd, I can articulate very strong opinions about methods for reliable power system protection, networking protocols, and international standards.

So can ChatGPT, though. And, that’s the allure. From the outside, it looks an awful lot like you can ask it to write a thing, and then it will spit back something resembling what you asked for. I’ve had professor friends in various disciplines feed it questions or prompts from their exams and term paper assignments, then rate the quality of its response. They’re usually impressed because it can give really high quality responses to difficult academic questions, the kinds of things that students will struggle with even after eight, ten, or sixteen weeks studying something.

Convince the right people

I mentioned earlier that I used to teach different kinds of writing. I’ve taken some classes on machine learning and AI for funsies, particularly back when Udemy was running huge discounts during holidays. I’m not a novice or an outsider in this technology, but I’m certainly not writing my own AI programs or even using the API of existing programs to make my own tools. The thing about AI text generation (and, I’ll argue later, all AI content generation) is that it fools novices into thinking it’s created something good.

That’s a claim that needs evidence, so here you go. I play around with Stable Diffusion a lot for personal projects, mostly creating art for roleplaying games. Here are some orc bartenders that I thought came out looking pretty good.

Filler.create()

In the end, I’m making filler with my AI art generation tool. I could run a D&D game just fine without it, and it will create a small enhancement of the experience for players when I use it.

It’s a lot like another metaphor that I enjoy using for AI content creation: the stuffing for a turkey dinner. Back in the olden days, stuffing was made by hand. You mixed together a bunch of extra bits you had around, mostly old bread. You needed something to soak up all of the fat that a turkey (or any roast bird) loses while it’s cooking, because you certainly can’t use it all for gravy. Well, you can, but you get a few gallons.

That’s not an entirely food-safe method of stuffing preparation, so eventually folks started soaking bread in turkey broth (chicken broth if you’re cheating) and baking it at the same time as the turkey. That’s a time-consuming process, especially if you don’t already have a bunch of stale bread laying around, so now you can buy boxes of stuffing mix that just need some water before you bake them.

The whole process of creating filler has been reduced using technology. Some people will get upset that the boxed mix isn’t as good as the homemade stuff, but no one’s coming over to eat stuffing with a side of turkey. (If they are, cut those people out of your life.)

Now, taking this metaphor a little further, we’ve all eaten a boxed convenience as the centerpiece (or the entirety) of a meal before. And, generally, it’s just not as good, right? Like, a bowl of instant ramen is fine, and it’s better if you dress it up some, but it’s nowhere near as good as non-instant ramen.

What’s the point?

AI generated content is like instant stuffing and instant ramen. If it takes a chef’s job, then that chef completely deserved to have their job taken. If my thirteen-fingered orcs up there can take a professional graphic artist’s job, guess what? That artist wasn’t very good at their job.

Can ChatGPT replace me? Well, I asked it to write an 800 word rant about artificial intelligence replacing professional writers. I did zero additional training or prompt engineering, and I only ran the program once (I’ll explain why afterward). In other words, I made the stuffing like the box said to. Now, let’s take that teacher’s eye to this thing and evaluate how well it performed.

Is it a fair assessment? Thoughts on prompt engineering & replacing content creators

Back in academia-land, one of my major research areas was assessment: determining whether or not an academic program did the thing it claimed to do. Assessment is traditionally done by looking at work produced by students—if your program teaches people how to weave baskets, the best way to assess that is to look at the baskets students produce, right?

I always argued (as did others) that the best assessment would always come from the work students produced as a regular part of their classes, not something produced specifically for assessment. In assessment-land, we often have problems with people creating assignments or projects specifically to insert an objective into their course/program and then assess it. I worked somewhere that had a university-wide goal to practice ethical reasoning appropriate to a student’s field—that is, a nurse should follow the ethical guidelines of their professional organizations, and an engineer should follow the guidelines of their organizations. When it came time to assess it, many programs created a new “ethical reasoning” assignment, taught some specific professional ethical codes, then tested students on whether they could remember and apply those codes.

Guess what? The students did! They remembered the phrases they were taught and applied them the way they were told to!

If you aren’t impressed, consider the ChatGPT output above. I specifically did no extra work to produce a result I wanted. I gave it one prompt and copied the first response it gave me. This is its “natural” response. I could have engaged in prompt engineering, where I fine-tune the specific phrases I ask, the amount of guidance it accepts from me, the model it draws from, or other details, in order to get the kind of response I want. It would be the same as adding in extra phrases and adjustments to get those orc bartenders to look the way I wanted them to.

In other words, I could have told it what I expected before I asked it to produce something, like that ethical reasoning assessment.

Good prompt engineering is how you get results you want from an AI. Compare those orcs above to a scene I wanted to make based on a scene from the 90s adaptation of Great Expectations, looking out at a sunny day through a ruined manor.

I made the image on the right without using the “img2img” function, which uses a base image to generate something new. According to the output from Stable Diffusion that day, I made 20 images from that prompt alone to get the output that worked for me. I probably could have done it in fewer tries if I knew visual art language a little more—that is, if I could have described what I wanted in language that reflected the tags in the model.

If I knew more about art, I could produce better art. Huh. Weird.

Why does prompt engineering matter here? Aren’t I worried about AI taking my writing job?

Consider that surface-level analysis we got from ChatGPT. AI can generate text fast based on what it has observed out in the world, but it’s missing some kind of special human element.

That element isn’t “human.” Lots of humans would accept that essay as good enough, but I’m a writer and writing teacher. I expected more from a rant and I could see the flaws in the writing as easily as most of us can taste the difference between homemade macaroni and cheese and a box of Kraft.

Kraft isn’t bad, but it’s not the thing you serve every time you want macaroni and cheese.

The ChatGPT output isn’t bad, but it’s not really a good argument about AI impacting the writing industry. It’s flat, overly simplistic, and really just a bunch of contradictory assumptions strung together with punctuation.

If I wanted better, I could have told it exactly what I wanted. Or, I could have given it enough detail that it could figure it out, and I could have asked for enough samples that I could pick the one I wanted.

Which is where my job security comes in.

If someone above me wants to replace me with ChatGPT and they want to get—and I quote our algorithm-based writer above—“the same quality” of work from an AI, they can’t just open a prompt window and say “give me a flyer for [new product]” or “write an email telling customers about this upcoming webinar.” Well, they could, but they aren’t likely to get what they pay me to produce.

They could engage in some prompt engineering: Write a flyer about [new product] that highlights [this feature] in a way that [this customer group] values. Make sure the flyer language is coherent with all other documents produced by our company. Make sure the flyer language doesn’t misrepresent products in a way that invites lawsuits. Don’t oversell, don’t underpromise.

When the prompt is written and the AI is run, now that same person needs to evaluate what was produced. Is the flyer good enough? Why or why not?

At some level, writing a good prompt becomes close to writing the thing itself (or at least having a really detailed specifications list). Evaluating the thing requires some knowledge of what was supposed to be produced in the first place.

If my job can be automated…

At some point, sure, AI will figure this out. There will be enough training sets and enough models to produce exactly the kinds of documents I produce. People will be good enough prompt engineers that they can get the documents they want from the algorithm. At some point in the last century, someone said the same thing about turkey stuffing: eventually the technology will catch up enough with the “real thing” that it won’t matter anymore.

But no one comes for the stuffing. They come for the turkey.

Right now, I’m not worried, at least not in the long term. Lots of middle-management MBAs are going to decide that it’s cheaper to use an AI workforce. They’re going to fire piles of people. Then, they’re going to find out that their marketing materials fall flat. The code behind their software products is too basic to do anything meaningful. The chatbot managing their customer service system just runs people in circles when they have a novel problem that doesn’t match the model.

The human value I bring to the table is knowledge that exists outside the model of existing texts for analysis. I just rewrote a few documents today based on a conversation last Friday where someone asked about an entirely separate project that I happen to be on, and we realized we needed to be very careful with how some things get phrased.

Half of that conversation was along the lines of “I don’t have words to describe this thing, but you know what I’m talking about, right?” I don’t think anyone has trained an AI on that kind of model yet.

The human element that ChatGPT praised is just expertise. Knowing how to do a thing and how to evaluate whether it’s been done well. AI doesn’t have enough of that reflection yet. It has it enough to say that step 10 of generating a solution is better than step 9, but it doesn’t know when it’s done.

I don’t either, though, which is why this post went this long. I’m done now.