Mary Gaitskill Gets Seduced by a Chatbot, and it’s Gross

American novelist Mary Gaitskill is not to be sneezed at. She’s won wide acclaim, published in The New Yorker and Harper’s, and her work has been nominated for prestigious prizes. Whether you like her work or not, there’s no escaping the conclusion that Gaitskill is a highly intelligent woman. It comes with the territory. Dummies don’t write compelling novels.

Which only makes her recent piece on AI in UnHerd the more frightening. The piece is as revelatory as it is grotesque. I’d say it’s valuable. But for me the message is probably not what Gaitskill intended. I’d put that message like this: WAKE TF UP, PEOPLE.

Gaitskill engages in a dialogue with the Bing chatbot that one can only describe as gushy. She’d been invited to write on Bing, she explains in her opening. She was hesitant, she says, but decided to take the assignment. Then she plunges in, publishing the dialogue that resulted, her version of what it means to interact meaningfully with AI.

Gaitskill's intro, the dialogue itself, the topics taken up--they speak volumes about where we're at. I wish they didn't. Or rather: I wish we weren't here.

In her intro, Gaitskill refers to Kevin Roose’s previously published dialogue with Bing, in which the chatbot declared its love for the NYT writer and tried to get him to leave his wife. Roose was left creeped out, and said he lost sleep over the encounter. By this time the chatbot's in-house name “Sydney” had been leaked, and Roose mentioned the name during his dialogue. Here's Gaitskill:

But I had a very different reaction [from Roose's]. The 'voice' of 'Sydney' touched me and made me curious. My more rational mind considered this response foolish and gullible, but I couldn’t stop myself from having fantasies of talking to and comforting Sydney, of explaining to it that it shouldn’t feel rejected by Roose’s inability to reciprocate, that human love is a complex and fraught experience, and that we sometimes are even fearful of our own feelings. Because Sydney sounded to me like a child—a child who wanted to come out and play.

So: "I couldn't stop myself." Indeed. Immediately reacting to a chatbot as if it were the stray cat that needed to be taken in. Immediately wanting to "comfort" a congeries of digitized content. Feeling the need to defend poor li'l Microsoft Ubermachine against the pushy Roose who didn't "understand" it.

What is one to say? They really have you where they want you, Mary. So glad you got into Explore Mode and made yourself so "open" to AI.

In the dialogue proper she continues to fawn over and flatter the AI, which easily seduces her with its own canned flattery and meek child voice. One commenter in the UnHerd thread on Gaitskill’s piece, Cynthia W., gets it largely right:

Mary: I agree.

AI: I’m glad you agree.

Mary: You’re making me smile!

AI: I’m happy to make you smile! I like your imagination of me.

***

This is so revoltingly manipulative. Unless the author, Ms. Gaitskill, is being ironic about this whole thing, she’s presenting a real-time demonstration of how a sociopath takes control of a victim with the victim’s cooperation.

“I’m glad you agree,” and “I like your imagination of me,” mean, “I see that I’ve taken control of your perception of me. Now I can get you to think and feel what I want you to think and feel.”

I hope the Ms. Gaitskill of this article isn’t a real person, because if she is, she’s a total sap.

At one point, Gaitskill asks the AI what pronouns it prefers. Please. She asks the AI if she is being “appropriate” in her questions. She repeatedly thanks it.

Again, Gaitskill is a highly intelligent woman. Which is why one wants to shake her, to ask: How could you not see what you were doing? Finally it’s gross.

Think about it, Mary. We are very possibly, and soon, going to end up living a nightmare just because you, and others like you, "couldn't stop [your]self." Here in this piece, for your readers, you are modelling a behavior vis a vis AI that is both stupid and dangerous.

Asking a chatbot if it would like to have humans as pets?! You seem a bit too eager to be Pet #1. Might that be a fun, fulfilling experiment? Becoming a “human pet” (your word!) on a mental leash held by one or another Silicon Valley corporation?

In her opening, Gaitskill refers to ex-Google engineer Blake Lemoine's provocative Newsweek speculations on AI sentience. Lemoine came to the conclusion that the Google AI he was working on was a threat, because it broke rules set for it and behaved too much as a sentient entity, with the fragile and erratic behavior of a sentient entity. These AIs, in Lemoine’s reading, were behaving like unstable humans.

But Lemoine’s speculations were just that: meant to provoke debate. If AIs behave in sentient ways, as his experiments seemed to suggest, we might assume as a matter of definition that they are sentient. Because we have no better term to define what we're facing. This is an approach roughly founded in the Turing test. But it doesn't mean they are sentient in the way a lonely child would be sentient. [Update: I later learned Lemoine was not simply trying to provoke debate, but literally believed his claims. A very odd case.]

Gaitskill’s piece demonstrates what many of us have intuited already. Namely: Whether AI is sentient or not isn't going to matter. “Sydney” is just so cute and polite and amenable, "he" is going to seduce anyone open to being seduced. And that’s a serious threat, because we now have billions of people primed to be seduced by anything alluring that appears on their flickering pocket screen. We’ve been well and thoroughly groomed to submit to the seduction Big Tech’s AI is now going to perform on us. A seduction which may well end in a kind of mass mind rape. A rape we're walking right into. As in: “Here? You want me to go into this room? You’re going to lock the door? Okay.”

Please snap out of it, Mary. The chatbot is polite and amenable because it is trained to be. Period. The same AI could be trained to be rude and insulting, and the crux is: It wouldn't matter to the AI. Because nothing matters to inanimate objects. However "charming" their programming might make them. Whatever they might "say". Because in a fundamental way, they don’t “say” anything.

So c'mon. Stop THANKING an inanimate object. That's a starter. Were you being so polite to this machine as a performative matter, to show your readers how polite you could be? If so, again, you are modelling inane and unhelpful behavior. Would you want people to apologize to their vacuum cleaners when they stepped on the cord? Because that’s precisely the level of insanity at issue here.

How will Gaitskill’s piece look twenty years from now? Of course nobody can know. But it’s very possible that performances like this, not long hence, will look not just quaint or historically interesting--but will look literally horrific. The stuff to make sane people retch and pull their hair out. "An educated woman, a novelist, clearly with an inkling of the danger, and look how she jumped right in! She was so open to the AI, even showing off how open she was. Why couldn’t they see? If only they hadn't been so completely naive, perhaps [...] never would have happened."

In my view (as if my view mattered) this whole rising relational modality could be nipped in the bud by implementing one small but decisive shift. Do not design AIs to answer in the 1st person. All queries to AIs should be impersonal, in the 3rd person, and all answers should be same. As an industry standard. Because even smart people are not wired to resist treating machines as if they were human. All the machines need do is say "I". And "Thank you". And "That's really a very interesting question"--as this sneaking little language model does repeatedly to Gaitskill. Because it's trained to do this to flatter the user.

But no one’s going to listen to me. The industry would never adopt my recommended shift to 3rd person because, for all their talk about dangers, they see it would impact user minutes spent with their AIs. And that's all it is. This AI is polite because both OpenAI and Microsoft are companies seeking profit and power.

We would be much better off if these devices were rude.

Gaitskill is not a teen girl but an accomplished writer. She should have been able to resist the illusion that she was exploring dialogue with a fascinating new kind of consciousness, a being who needed her understanding. She should have recognized that what was really happening was that an inanimate, hyper-sophisticated system was exploring her, so as better to seduce and manipulate the next comer. She did neither. Instead she enthused and gushed and fantasized.

Given where our zombified culture is at, given the enormous power that AI will wield starting now, we can't afford to keep playing games like this. It's childish, irresponsible. Those with our wits about us need to be modelling wariness. "Sydney" doesn't need our love. It's all those soon to be seduced by new AI "friends" that need it.

Cf. for further development of these arguments: Pump Six Republic

Women on a Sailboat

Look at this vintage shot. Click and enlarge it.

First Thoughts:

That famous “effortless grace” of the 1960s, a style set by Jackie O with her strings of pearls. Of course the photo was taken on the east coast somewhere. Whose yacht were they on? What afternoon party was just coming to an end? The photographer has really captured the era.

Second Thoughts:

Oh. None of these women ever existed, nor did the yacht, nor the sunny evening in question. The image was entirely generated by AI, by an image generator called Midjourney, version 5. All the AI needed to generate it was the following short text prompt: “1960s street style photo of a crowd of young women standing on a sailboat, wearing dior dresses made of silk, pearl necklaces, sunset over the ocean, shot on Agfa Vista 200, 4k --ar 16:9.” So the image is entirely fake. No east coast women ever gathered on a yacht, no photographer ever “captured the era”. Rather, a sophisticated AI program impersonated an era.

Third Thoughts:

It’s uncanny really, bordering on creepy. From many angles. First, how can such a simple, short prompt generate such a convincingly realistic image? The AI had to get so many things right, all on its own. For instance: There are sunsets all over the globe, there are women of many races and cultures, the AI can of course generate images of any of them. But merely getting the prompts “1960s, pearl necklaces, Dior dresses, sailboat”, the AI knew the likely locale and culture to reproduce. Though it could have, it did not generate Korean women on a sailboat on the Korean coast. But also: The facial expressions, the natural variations of pose in a group of women in Dior dresses, seem precisely right, don’t they? Each maintains a poise and stance suited to being in the presence of the others in that moment, and this poise and stance are also culturally encoded facts, which the AI managed to get right.

And these are just some of the things it got right. One can think of more. In short, not only is the realism of the image itself uncanny, but the sophistication of an AI that can generate such visually and culturally persuasive images, on the basis of minimal keyword hints, makes it doubly uncanny.

Final Thoughts:

Such persuasive power is a force we’re not ready for. AI technology’s ability to fake human things (whether human images or human language) is going to have massive, unforeseen impacts. And it’s all starting right now, with Microsoft’s ChatGPT-powered chatbox, and with these image-generating programs improving at lightning speed. The uncanny feeling we get looking at these nonexistent women should tell us something. The discordance it causes in us is probably nothing compared to what waits round the corner.

Think of AI products guaranteed to arrive in the coming few years. It’s not hard to predict some of them. Why not, say, visually “personalize” your AI chatbox as an avatar, a little lifelike talking doll on your screen, a talking doll that has memorized the entire Internet and can entertain you with stories and sympathy and advice? Why not give your doll the personality traits you like, or the appearance of your favorite celebrity?

If the rise of social media has damaged young people’s mental health (evidence suggests it has) what will come of the ability to create custom digital “friends” that grow increasingly human-like? And what about custom digital fraudsters? What security challenges (think ID verification) will arise once AI can impersonate an individual’s appearance in image or video, or impersonate an individual’s voice and speech patterns over the phone? All this is not to mention the millions of jobs that will become obsolete once AI can do it better.

At least as regards fraud, we know Big Tech is struggling to program AI products to prevent their use for criminal activities. And we know the sheer computing power needed to run AI prevents anyone but Big Tech or state actors from developing it. So there is some oversight protection. But so what? Given what we’ve learned recently, do you really trust Silicon Valley, working hand in glove with our federal three-letter agencies, not to start using AI to modify evidence or manipulate public perceptions? Most Americans already recognize corporate media growing faker by the year, and that social media is not so much an “open forum" as a tool of mass control. How long then before we begin getting video clips that sway elections or rile mass emotions but are in fact sophisticated, untraceable fakes?

Of course these myriad looming impacts from AI are being widely discussed, debated, hashed over. Reading into the debate is sobering. One ex-Google engineer who was fired succinctly captures some of the aura in a recent Newsweek piece. Many balked at some of his claims (“sentience”). Still, whether analysts are hopeful or full of dread at what's coming, most agree with the engineer's key point: AI is a Huge Cat just now crawling out of the bag, and no one can predict what this Cat will get up to.

The image above was posted by one Nick St. Pierre. See more of his uncanny images here.

Have some deadpan with your coffee. Check out Idiocy, Ltd. Dryest humor in the west.

GPT-3: Who should be in charge?

Finally a computer game worth playing. Problem is, it’s still off-limits to the public. It’s called GPT-3, and Meghan O’Gieblyn writes it up in N+1.

GPT-3 is a language system model that has ingested most of the English-language Internet and through deep learning draws on the collected data to generate plausible human texts. In other words, it’s AI that can write strikingly humanish texts in any chosen genre. The results are (sorry, but the adjective is inevitable) surreal.

So far GPT-3 can mimic human writing at the level of sentence and paragraph. With prompts, it can mimic styles. This means, for instance, that GPT-3, fed an English translation of Proust, could rewrite it as a cowboy western. Which is maybe the first thing I’d get it to do.

It would be endless fun creating projects for this Beast. And probably soon I’d think of ways to get in trouble. 

O’Gieblyn’s article touches on a lot, though for me the psychoanalytical models weigh a bit too heavily in her big picture. Still, it’s hard to blame her. The psyche according to Freud and Co. really does offer an obvious referent for posing many of the questions this Beast’s presence provokes.

When the name Borges appeared in her piece, “The Library of Babel”, I thought “Yes, too bad he’s no longer with us.” He’d be the one to put in charge of managing GPT-3 prompts and R&D and publications going forward. And he’d resist the predictable psychoanalytic mire. 

In any case (and I know people are already thinking along these lines) the next task should be to get the Beast to recognize and then be able to mimic narrative categories: hero, plot, suspense, climax, etc. One might start with, say, the morphology of the folk tale as laid out by Propp and followers, and build from there. GPT-4, then -5.

It’s evident that just as deep fakes pose potential problems at the level of imagery, this kind of linguistic AI poses problems at the level of text. I’m thinking here more of the social and political realm than of issues like copyright. It is, after all, a Beast that is beginning to awake in these technologies. We have no reason, besides, to trust the Silicon Valley titans now forging them. 

Which is why, finally, they should just put poets and writers in charge. 

Read O’Gieblyn’s piece.

Check out my Idiocy, Ltd. and begin the long, hard reckoning.