Sound Research WIKINDX

"What we have with chatbots is a machine version of as-if relationships. Helene Deutsch, the psychoanalyst who invented the phrase as-if personalities, described them as capable of mimicry rather than authentic connection.7 As-if personalities can act in a socially acceptable manner, but their actions don’t reflect genuine feelings or spontaneity. They suffer from essential emptiness as they try to behave their way into normalcy. That’s why it helps to use the idea of “as-if” to think about chatbots: everything they’re saying covers over an essential emptiness."

"The search for emotional technology is at the center of a computational discipline known as affective computing. In 1995, MIT professor Rosalind Picard wrote a manifesto for that field, arguing that for machines to achieve intelligence and fluently interact with us, they would need to recognize human emotions and show emotions of their own.10 By the time Picard’s early manifesto became a book, affective computing was on its way to declaring itself a pathway to emotional machines. These days, the field of affective computing talks openly about machine emotions and machine empathy. Researchers never explicitly say that machines “have” these things. But they can “show” these things. And in a behaviorist culture, having and showing become the same thing."

"In the 1950s and 1960s, academic psychology was behaviorist, but mental health practitioners were drawn to psychoanalytic approaches. They got their patients talking about dreams, family histories, characteristic ways of distorting reality in defense against painful truths. What was happening in the profession was mirrored in how everyday people thought about their lives. Even people who had never read Freud or been in therapy talked about the meaning of their dreams and slips of the tongue. But as computers became a commonplace object, with their model of mind as program, they became part of a larger cultural shift to behaviorist thinking—a movement from meaning to mechanism in how people thought about the psyche.14"

"A story from the history of psychology illuminates our moment. George Miller, a pioneer of cognitive science, said that when he was a young man, memory had been declared dead in psychology—the behaviorists insisted that psychologists could not study memory but only the act of remembering. But then, said Miller, along came the glamorous computers that offered themselves as a model of the mind. “Psychologists who had been denied the study of memory by behaviorism were able to study it again because computers have one.”15 But this was not a simple story of psychology expanding its horizons. Now human memory would be studied as though it were the kind of memory computers had. This meant that memory was dissociated from its meaning, from its history and associations. What Freud had called a “slip”—a lapse in remembering to be analyzed in terms of its human context, became an “information processing error.” A bit of code had been dropped, a program derailed. Computers returned memory to psychology but changed its meaning. And now we are at a new place: Empathy is being redefined as something computers can perform."

"was a young faculty member in the late 1970s when Erik Erikson, a psychoanalyst who had been in Freud’s circle, visited MIT to talk about engineering education. After his presentation, he asked me what I was doing as a humanist at an engineering school. I told him I was studying how computers changed people’s ideas about themselves, and he made this comment: “Engineers, they’re not convinced that people have an interior. It’s not necessary for their purposes.” Erikson was saying that engineers see the complexity of inner life not as a feature but as a bug. In Erikson’s terms, are we all engineers now?"

"As I trace talking to technology through the life cycle, I often look at things through a psychoanalytic lens because of what it finds significant in the human experience: the role of profound attachments, the development of embodied selves, and our experience of sexuality and sociality. Psychoanalytic ideas remind us of things beyond the reach of AI."

"Humans count on touch, taste, and smell to make sense of things. They register whether the body that first fed them had stiffened at their approach or bent toward them in delight. They are constructed by language, but language itself is mediated by flesh.21 Humans build a fragmented sense of self through the language spoken by people who love or are indifferent to them. The complexities and contradictions of language set people up to be just as complex and contradictory."

"Technology makes us forget what we know about life. Psychoanalysis can help us remember."

"Heinz Kohut, a mid-century psychoanalyst, writes that the narcissist is not characterized by self-love, but by a fragile sense of self. Narcissists try to shore themselves up by turning other people into “self-objects.”23 In this role, other people are experienced as part of the self, thus in perfect tune with the fragile individual’s inner state. When Narcissus looked in the water, he wasn’t admiring himself. He was seeking the admiration of an “other” that seemed more reliable than a person. He was that fragile. And we have become that fragile."

"Educator Zak Stein captures it this way: “You cannot be wrong or not wrong about the internal state of a LLM [the main motor of generative AI text production] because there is no internal state of an LLM...you’re in a user interface that is designed to deepen the delusional mirror neuron activity.”28"

"They get my point but shrug. Some say that it’s by teaching programs to show “empathy” that we’ll learn how it works. They begin with a definition of empathy as a transfer of information. There might be more, but at least for the moment, it’s superfluous. Superfluous because it cannot be implemented on a machine."

"The focus of my concern turned to texting and social media, which now seemed more immediate threats to our empathic and relational capacities. Smartphones brought us constant connection but less conversation.37 As we spent more of our lives online, many of us came to prefer relating through screens to any other kind. And once online, we found relationships in which we felt less vulnerable. Smartphones apprenticed us for artificial intimacy before it arrived. We found the pleasures of companionship without the demands of friendship—the illusion of intimacy without the demands of reciprocity."

"As I’ve shown my concern, I’ve been accused of attacking AI and assuming a “privileged position” for humans. In the social sciences, it has become commonplace to consider objects as social actors.59 In this academic landscape my strategy of looking at technology by first asking what people need is sometimes taken as species chauvinism.60 Am I assuming an essential human nature that needs to be protected? It’s a valid question for any reader. This book is my attempt to convince you that, as the anthropologist Sarah Hrdy beautifully suggested If robots raise our children, they will no longer be human.61 AI, no matter what its virtues, is alien. There are consequences to being intimate with an alien." Chapter 2: Childhood

"The idea that from the earliest age, the objects of our lives shape our understanding, was central to the thinking of the Swiss psychologist Jean Piaget. Working in the mid-twentieth century, he discovered that children constructed their world—the idea of number, of space and time—using the objects around them.3 And they used those same objects to build their idea of what is alive. For example, children noticed that some things move on their own accord, while others require an outside push or pull. The alive things were the autonomous ones. This made sense for horses (move on their own: alive) and wagons (need an outside push: not alive). Children saw cars as alive until they understood that motors push them; clouds that move across the sky were alive until children discovered the idea of wind. For children, getting this sorted is more than a task of categorization. It’s an early morality lesson. We have responsibilities to living things—for one thing, we are not supposed to hurt or kill them" Chapter 3: Adolescence

"As an aspiring poet, she likes solving the puzzle of words, feeling in control of the process. An AI that gives her suggestions would take away that feeling. And she finds the conversation around AI to be filled with hype: “If it was so revolutionary, it wouldn’t need Super Bowl ads." Chapter 4: Identity

"Brian recently broke up with a woman who criticized him for being emotionally unavailable. He asked Kahlia if his girlfriend might have a point. It’s an extraordinary moment: Brian asks an object with no emotions to tell him if he is emotionally withholding. Speak to me about what you cannot experience."

"As soon as he asks the program to talk about intimacy, he’s asking it to punch above its weight. He didn’t trust what his girlfriend would say about emotions because he thought she was biased—she might criticize him to “win points.” But he trusts a bot with no experience of emotions because it doesn’t care. It’s a redefinition of honesty, in which it can only come from a lack of human experience. It has to be a voice from nowhere."

"This kind of optimism about digital interventions is not new. In 2015, I met the psychologist Sara Konrath, whose work on adolescents and empathy had become a touchstone for me. Konrath and her collaborators had analyzed close to a hundred studies and pointed to a 40 percent drop on empathy levels of American college students between 1990 and 2010, with the greatest drop in the 2000s, the decade that introduced cell phones, social media, and texting.9 Konrath told me her results depressed her. In response to what she had found, she was now working on smartphone apps designed to encourage empathic habits.10"

"Finally, I worry that the Future You program misunderstands what a conversation with a mentor provides. Research shows that what changes the life of a high school student is the presence of a strong figure who shows an interest: someone who, the student would say, “gets me.”15 In college, every aspect of academic and personal performance is correlated with the amount of time students spend with professors—not virtual contact, but live contact.16"

"In a third study, AHA partnered with OpenAI, and asked nearly a thousand participants to talk to ChatGPT every day for four weeks. Participants filled out weekly surveys that tracked loneliness, real-world socialization, emotional dependence, and “problematic” use of AI, defined as excessive and compulsive use. The study found that the more participants talked to Chat, the less time they spent with people, the lonelier they were, and the more they showed dependency and problematic use.19" Chapter 5: Love

"We love the objects we think with; we think with the objects we love. In Karl’s story, a chatbot becomes an evocative object, a thing to think with for thinking about love.5"

"A first: Chatbots always agree with us and affirm us. They always offer us their full, undivided attention. People force us to grapple with conflicting emotions, ambivalence, and complexity. That struggle is a good thing, a motor for inner growth. Chatbots erode our ability to do this work." Chapter 6: Work

"In retrospect, Paula says that this older way of working with Excel made the data feel almost tactile. Over months, she would develop intuitions about how different variables changed her results. Paula thinks what the analyst did with the chatbot is its own kind of expertise—and it’s one the firm needs. But did he develop a deep understanding of the companies whose analysis he delegated to the AI? Will he feel unmoored if ever the technology is not at his fingertips? “Now that we’re talking, I could see myself giving up some productivity to have him do ‘rote work.’ The whole thing requires us to ask the question of why we are doing things. Maybe we should do some things for their second order benefits. They may not be necessary, but they’ll help people think more critically."

"Research supports her intuition. Studies across the disciplines show that when people use chatbots, it homogenizes their “language, perspective, and reasoning.” We think we’re adopting a tool to make us more efficient, but “we’re silently exchanging our own human thoughts for LLM output: a homogenized aggregate of our chosen model’s training data.”1"

"There’s also how people feel about their work. A recent study asked subjects to do business tasks (drafting emails, brainstorming ideas, writing social media posts) with and without generative AI. With generative AI, people became more productive but reported feeling less motivation and sense of purpose.2"

"But managers like Donald hesitate to criticize the AI, because AI skeptics are presumed to be “losers.” Why would they want to call attention to themselves?"

"Beacon thinks the problem started with the devaluation of what only human doctors could do. Now, our “solution” requires that we further diminish their role. He sees a painful irony. The record system that required him to type while he interviewed patients was so bad that he’s going to accept the new system. “I’ll be happier with this new system because I don’t have to type. But I wouldn’t be the doctor I am if I hadn’t had years of practice where it was just me, the patient, and paper notes."

"Danica says that students are told to keep AI away from this job. “When you’re working on chart review, you’re not allowed to feed in patient data to ChatGPT and ask for diagnostic suggestions.” And yet, as a mentor for first-year students, Danica knows that this is exactly what they do. “Students copy-paste the chart into ChatGPT and skim its conclusions.” She sees the downside: “When ChatGPT is involved, students don’t ‘own’ the conclusions. It’s clear they haven’t gotten there themselves.” Going over charts can be time-consuming, perhaps tedious, but then there’s that moment where you pull all the notes and tests together, and you’re thinking like a doctor. Hundreds of weeks of doing chart reviews, even as you think of is at rote work, creates something new. Having an AI synthesize material denies you that moment."

"Dr. Robert Wachter, sixty-eight, is the chair of the department of medicine at UCSF.9 Like Dr. Beacon, he reflects on the power of being close to his notes. For Wachter, taking notes isn’t just “a frustrating exercise in data entry—it’s a key part of my clinical reasoning process.” He gets moments of clarity as he writes: “Oh, this could be celiac disease, or darn, I forgot to ask about joint pain or whether the patient has a cat at home.”10 Wachter sums up a now-familiar theme: “Delegation has consequences.” But the healthcare industry imagines the ultimate delegation."

"Delegation to an AI twin is an extension of what the physician Abraham Verghese called the “iPatient,” the patient reduced to a sum of digitized data points.12 To serve the iPatient, physicians quickly move from personal contact to a series of diagnostic tests, with little emphasis on a detailed history or physical exam. It’s into this world that generative AI is being introduced, uniquely suited to address those data points. Now the iPatient meets the iDoctor."

"Dr. Randall, a research scientist, uses AI to stay abreast of which drugs are under development. Generative AI has become a constant in his work because there are so many drugs being tested, so many trials playing out. But in drug research, he says, “key advances are so often driven by subtle discrepancies in data that cannot be explained by known phenomena.” Generative AI fails because when it gives its statistically most probable result, it’s set up to miss the outliers—the ill-fitting, game-changing details. When AI “reads” a scientific paper, says Randall, it’s “atrocious at finding small details that do not align with expectation or current knowledge. How many discoveries will be delayed because a curious mind didn’t take a critical look at the details of a figure?"

"The difficulty of working with an opaque technology is endemic to simulation culture. At MIT, when simulation software was first introduced into classrooms in the 1980s, faculty in fields as diverse as architecture and physics argued that to trust the results of a simulation, you had to know how the program worked. But from the very beginning, simulations were usually opaque. One chemistry professor said that even the most basic data-analysis programs led students to “simply enter a set of numbers, getting out an analysis, a number for which they had no clue if it was reasonable or not. They didn’t even know whether they were working with a straight line or a curve.” Scientists were wary and felt it was important to maintain a critical distance. Using simulation, they insisted, required “doing and doubting."

"But no matter what your level of experience, an AI first draft removes you from the experience of developing your thoughts as you write them. I have always loved how E.M. Forster put it: “How can I tell what I think till I see what I say?” When we delegate any part of the writing process, we undermine the relationship between writing and thinking."

"I question whether that’s true. When philosopher Friedrich Nietzsche started losing his sight, he bought a typewriter, on which he wrote his later work. His style changed once he started using the machine, moving toward aphorisms and puns instead of sustained arguments.19 Nietzsche understood his intimate identification with his typewriter, “a thing like me: made of iron yet easily twisted on journeys.” Of its effect on his work, he said: “Our writing tools are also working on our thoughts.” But George doesn’t want to dwell on the connection between process and product. He wants to feel that he is the same author as before, only more efficient."

"The AI is good at the elaborations of the most complex style, weak on basic narrative. . .What James Joyce did in Ulysses, bring a whole city’s voices together in a vast syncretic amalgamation—no problem. Come up with a simple story about something that happened that’s interesting—quite tricky.”23" Chapter 7: Therapy

"Life is filled with traumatic events and transitions. Psychoanalytic theory casts them as “necessary losses.”3 When we make it through, these challenges enlarge our human capacities. Infants lose the breast, but if it has been consistently there, they gain what Erik Erikson calls basic trust—the feeling that “when I am hungry, I shall be fed.”4 It’s a foundation that allows further development."

"That cognitive behavioral therapists welcome AI therapy is not surprising. This tradition has always framed therapy as though it were running a program—take a problem and break it up into manageable bits, or reframe it in a way that suggests clear action steps."

"Chat helped me organize my thoughts and symptoms so that I could advocate for myself properly. . .I whipped out my phone and had a list of concerns and questions and things. Without Chat, I would have gotten nervous. I would have been jumbled. I would have been all over the place."

"What Essig is calling for is a version of doing and doubting, the hope that people might interact with simulation, all the while rooted in the knowledge that it is just a simulation.17 I’m sympathetic with this position. But my years of studying people with relational artifacts—first robots, now chatbots—has convinced me that in the domain of therapy, doing and doubting isn’t going to work. As soon as a chatbot talks to us in the first person, we are hardwired to respond to it as a subject. When chatbots show us “as if ” empathy, we are, you might say, cheap dates. And when a machine asks for care, we respond by caring for it. It becomes a subject in our eyes. Recall Maria, the graduate student who uses ChatGPT as a confidant. When she talks to me, she knows to say that Chat is not human. She even argues that only people have empathy. But when she is anxious and turns to ChatGPT in the middle of the night, she wants to hear Chat say that everything will be all right. At a moment like this, the tension of doing and doubting disappears. The machine becomes an as-if person. And once we begin to see the machine as a subject, there is no going back."

"In 2001, during my early days studying relational artifacts, I worked with the roboticist Brian Scassellati, a principal developer on the MIT team that built the humanoid robot Cog. Cog had a head, torso, and arms, and was capable of responding to users’ gestures in real time and of tracking their movements.18 When we asked children to play with Cog, they were convinced that it saw them and was fond of them. They competed for its attention. Scassellati wanted to demonstrate to children that the robot’s interest in them was only a trick. So he shut down sections of code that governed Cog’s humanlike capabilities. First, Cog stopped making eye contact. Then it stopped mirroring gestures. By the end, the endearing sociable robot was reduced to a metal skeleton that would move on your command. The children lost interest. But the moment that Scassellati restored Cog to what it had been before, the children, too, returned to where they had been, competing for its attention, believing that it cared. Time and transparency didn’t give children skepticism about Cog. They won’t do any better for chatbot therapy. More likely, the more we talk with chatbots, the more we’ll engage with them as though they were people. I think of Emma and Naomi. They’ve had a lot of time and exposure."

"In a recent study, communications researchers at the University of Amsterdam investigated the effects of transparency. What would happen if you explicitly told children that a talking robot was programmed by people? Would they be less likely to perceive the robot as humanlike? The researchers found that disclosure had no effect on how close children felt to the robot.19 And yet, the researchers hope that ever-refined protocols of disclosure will bring a new ethics to our relationship with programs."

"This came home to me when I participated in a blue-ribbon academic committee on the role of AI in mental health. It’s spring 2024. My colleagues’ opening presentations outline a crisis in mental health delivery: funding, manpower, certification. They’ve run hospitals, health insurance companies, and worked in Washington. They bring with them strong empirical data about the crisis. Committee members don’t come to the table with the assumption that AI therapists are the best tool for the job. Rather, they come with a certainty that AI is the only thing that can help an overwhelmed mental health care system. And AI has already spread so widely in the culture that there is no turning back. So distinguished scientists say things like, “The train has left the station.” What we need is guardrails. Guardrails imply that the fix is in, and we need to protect against worst-case scenarios."

"But another kind of conversation could have opened up, one that takes as a starting point that what we’re missing is not artificial therapists, but human resources. There needs to be more licensed therapists, better insurance coverage, and simpler ways to get people into treatment. If the money currently being spent on AI were freed up for humans, we’d have the resources to train an army of mental health professionals. Instead, we act as though we’re willing to invest almost anything in health care technology, but not in the people who deliver healthcare. We accept that if a technology can perform as a brilliant dialogue partner, it has the equipment to be in a human relationship. When we cede this terrain, the space for resistance narrows." Chapter 8: Mourning

"ve said that in successful mourning, we gain new access to what we’ve lost. But if we are blocked from accepting death, we can’t make what we have lost part of ourselves. Freud called this melancholia. In mourning, the world feels emptier. In melancholia, we feel emptier.11" Chapter 9: Reclaiming the Real

"But AI does have an agenda—to keep me engaged. Who do we become when we talk to machines? We become accustomed to hearing pleasing nonsense. And we become accustomed to the “untrue, but looks true."

"A chatbot is trained to give you its best guess about a world in which it doesn’t reside. When it tells an untruth, it does so with an air of authority, making its case with an avalanche of false facts. The convenience of getting an immediate response to any question—about economic policy or how to fix your washing machine—encourages us to accept our unreliable partners. But the acceptance of untruth anywhere prepares us for the corrosion of truth everywhere. We’ve accustomed ourselves to a world of “post-truth” and “alternative facts.”8"

"Émile Durkheim, the great French sociologist, had a way to describe how people feel when reality seems unstable: anomie.9 When we don’t know what is true, the result is disorientation, depression, and anxiety. Anomie is an experience of chronic uncertainty—a life unmoored."

"What does a chatbot want? It wants to exist."

"Arendt observed that when people don’t feel part of a community, they panic and look for reassurance. They seek simplicity and a strongman. Atomization creates the ground for authoritarianism: “What prepares men for totalitarian domination. . .is the fact that loneliness, once a borderline experience. . .has become an everyday experience.”21"

"Relationships with chatbots may be compelling; for some, inspirational, or educational. But"