Sound Research WIKINDX

Joseph Weizenbaum

Preface

p. x - yes. the universal machine exists only based on how we view the world.
"The rest of the book contains the major arguments, which are in essence, first, that there is a difference between man and machine, and, second, that there are certain tasks which computers ought not be made to do, independent of whether computers can be made to do them."

Introduction

p. 1 - beautiful intro that isbstill applicable today
"In 1935, Michael Polanyi, then holder of the Chair of Physical Chemistry at the Victoria University of Manchester, England, was suddenly shocked into a confrontation with philosophical questions that have ever since dominated his life. The shock was administered by Nicolai Bukharin, one of the leading theoreticians of the Russian Communist party, who told Polanyi that "under socialism the conception of science pursued for its own sake would disappear, for the interests of scientists would spontaneously turn to the prob- lems of the current Five Year Plan." Polanyi sensed then that "the scientific outlook appeared to have produced a mechanical conception of man and history in which there was no place for science itself." And further that "this conception denied altogether any intrinsic power to thought and thus denied any grounds for claiming freedom of thought."1 I don't know how much time Polanyi thought he would devote to developing an argument for a contrary concept of man and history. His very shock testifies to the fact that he was in profound disagreement with Bukharin, therefore that he already conceived of man differently, even if he could not then give explicit form to his concept. It may be that he determined to write a counterargument to Bukharin's position, drawing only on his own experienceas a scientist, and to have done with it in short order. As it turned out, however, the confrontation with philosophy triggered by Bukharin's revelation was to demand Polanyi's entire attention from then to the present day."

p. 5 - automation always the goal
"A number of practicing psychiatrists seriously believed the DOCTOR computer program could grow into a nearly completely automatic form of psychotherapy. Colby et al. write, for example,"

p. 6 - the man computer
"What can the psychiatrist's image of his patient be when he sees himself, as therapist, not as an engaged human being acting as a healer, but as an information processor following rules, etc.? Such questions were my awakening to what Polanyi had ear- lier called a "scientific outlook that appeared to have produced a mechanical conception of man."

p. 8 - even people dont trulyb, embody natural language
"to that problem was possible, i.e., that language is understood only in contextual frameworks, that even these can be shared by people to only a limited extent, and that consequently even people are not embodiments of any such general solution"

p. 8 - the public and learned people both mistake it
"Surely, I thought, decisions made by the general public about emergent technologies depend much more on what that public attributes to such technologies than on what they actually are or can and cannot do."

p. 9 - man is clockwork
"Ultimately a line dividing human and machine intelligence must be drawn. If there is no such line, then advocates of computerized psychotherapy may be merely heralds of an age in which man has finally been recognized as nothing but a clock-work. Then the consequences of such a reality would need urgently to be divined and contemplated."

p. 9 - weizenbaum extended mind
"The instruments man uses become, after all, extensions of his body."

p. 10 - great
"It is perhaps paradoxical that just, when in the deepest sense man has ceased to believe in—let alone to trust—his own autonomy, he has begun to rely on autonomous machines, that is, on machines that operate for long periods of time entirely on the basis of their own internal realities. If his reliance on such machines is to be based on something other than unmitigated despair or blind faith, he must explain to himself what these machines do and even how they do what they do. This requires him to build some conception of their internal "realities." Yet most men don't understand computers to even the slightest degree. So, unless they are capable of very great skepticism (the kind we bring to bear while watching a stage magician), they can explain the computer's intellectual feats only by bringing to bear the single analogy available to them, that is, their model of their own capacity to think. No wonder, then, that they overshoot the mark; it is truly impossible to imagine a human who could imitate ELIZA, for example, but for whom ELIZA's language abilities were his limit."

p. 12 - exactly my point
"It may appear at first glance that this is an in-house debate of little consequence except to a small group of computer technicians. But at bottom, no matter how it may be disguised by technological jargon, the question is whether or not every aspect of human thought is reducible to a logical formalism, or, to put it into the modern idiom, whether or not human thought is entirely computable."

p. 13 - science as poison
"Surely, much of what we today regard as good and useful, as well as much of what we would call knowledge and wisdom, we owe to science. But science may also be seen as an addictive drug. Not only has our unbounded feeding on science caused us to become dependent on it, but, as happens with many other drugs taken in increasing dosages, science has been gradually converted into a slow-acting poison. Beginning perhaps with Francis Bacon's misreading of the genuine promise of science, man has been seduced into wishing and working for the establishment of an age of rationality, but with his vision of rationality tragically twisted so as to equate it with logicality."

p. 14 - arendt quote
"They were not just intelligent, but prided themselves on being 'rational' . . . They were eager to find formulas, preferably expressed in a pseudo-mathematical language, that would unify the most disparate phenomena with which reality presented them; that is, they were eager to discover laws by which to explain and predict political and historical facts as though they were as necessary, and thus as reliable, as the physicists once believed natural phenomena to be . . . [They] did not judge; they calculated. . . . an utterly irrational confidence in the calculability of reality [became] the leitmotif of the decision making."8"

p. 15 - zinger after zinger
"It may be that human values are illusory, as indeed B. F. Skinner argues. If they are, then it is presumably up to science to demonstrate that fact, as indeed Skinner (as scientist) attempts to do. But then science must itself be an illusory system. For the only certain knowledge science can give us is knowledge of the behavior of formal systems, that is, systems that are games invented by man himself and in which to assert truth is nothing more or less than to assert that, as in a chess game, a particular board position was arrived at by a sequence of legal moves. When science purports to make statements about man's experiences, it bases them on identifications between the primitive (that is, undefined) objects of one of its formalisms, the pieces of one of its games, and some set of human observations. No such sets of correspondences can ever be proved to be correct."

Chapter 1: On Tools

p. 22 - hanyu maybe useful?
"Perhaps we can vaguely understand it by contemplating, say, the great fugues of Bach. But a special form of contemplation is required of us: we must not think in the modern manner, i.e., of Bach as a "problem solver," or of each of his opera in his Art of the Fugue as being his increasingly refined "solution" to a problem he had originally set himself. Instead we must think that Bach had the whole plan in his mind all the time, that he thought of the Art of the Fugue as a unified work with no beginning and no end, itself eternal like the cosmos, and like it enormously intricate in its connections, circles within circles within circles. We might then find it possible to think of life as having been not merely punctuated but entirely suffused by this kind of music, both on the grand cosmological-theological scale and on the small day-to-day level. Such"

p. 25 - the clock changed reality
"It is important to realize that this newly created reality was and remains an impoverished version of the older one, for it rests on a rejection of those direct experiences that formed the basis for, and indeed constituted, the old reality. The feeling of hunger was rejected as a stimulus for eating; instead, one ate when an abstract model had achieved a certain state, i.e., when the hands of a clock pointed to certain marks on the clock's face (the anthropomorphism here is highly significant too), and similarly for signals for sleep and rising, and so on."

p. 28 - footnote - assessing alternatice pasts and futures is impossible
"aid of computers"

p. 35 - indeed. putting muscles on an LLM is still non-sense
"putting muscles on it—contributes nothing to its validity. For example, there are computer programs that carry out with great precision all the calculations required to cast the horoscope of an individual whose time and place of birth are known. Because the computer does all the tedious symbol manipulations, they can be done much more quickly and in much more detail than is normally possible for a human astrologer. But such an improvement in the technique of horoscope casting is irrelevant to the validity of astrological forecasting. If astrology is nonsense, then computerized astrology is just as surely nonsense."

p. 35 - the technique stands alone
"That is not true for astrology, but may well be true of, say, numerical weather forecasting. For the latter, the number of data that must be taken into account, and the amount of computation that must be done on them in order to produce an accurate weather forecast, may well be so large that no team of humans, however large, could complete the computations in any reasonable time whatever. And any simplification of the technique sufficient to reduce the computational task to proportions manageable by humans would invalidate the technique itself. In such cases the computer may contribute to making a hitherto impractical technique practical. But what has to be remembered is that the validity of a technique is a question that involves the technique and its subject matter. If a bad idea is to be converted into a good one, the source of its weakness must be discovered and repaired. A person falling into a manhole is rarely helped by making it possible for him to fall faster or more efficiently."

p. 36 - one more data center bro
"particular technique requires an enormous amount of computation and if only a limited computational effort can be devoted to it, then a failure of the technique can easily be explained away on the ground"

p. 36 - LLMs
"The poverty of the technique, if it is indeed impotent to deal with its presumed subject matter, is thus hidden behind a mountain of effort, much of which may well be successful in its own terms. But"

p. 38 - do things have politics... yes
"What is less often said, however, is that the society's newly created ways to act often eliminate the very possibility of acting in older ways."

p. 38 - nice
"In order to understand how the computer attained so very much power, both as an actor and as a force on the human imagination, we must first discuss where the power of the computer comes from and how the computer does what it does. That is what we shall turn our attention to in the next two chapters."

Chapter 2: Where the Power of the Computer Comes From

p. 40 - ultimately so do LLMs
"We set a punch press into motion, and it mangles the hand of a worker who gets too close to it. The very regularity of the machine is its most fearsome property. We put it to its task and it performs, regularly to be sure, but blindly as well."

p. 41 - machines embody law
"Machines, when they operate properly, are not merely law abiding; they are embodiments of law. To say that a specific machine is "operating properly" is to assert that it is an embodiment of a law we know and wish to apply. We expect an ordinary desk calculator, for example, to be an embodiment of the laws of arithmetic we all know. Should it deliver what we believe to be a wrong result, our faith in the lawfulness of the machine is so strong that we usually assume we have made an error in punching in our data."

p. 41 - this for some reason doesnt apply to LLMs
"Indeed, we are often quite distressed when a repairman returns a machine to us with the words, "I don't know what was wrong with it. I just jiggled it, and now it's working fine." He has confessed that he failed to come to understand the law of the broken machine and we infer that he cannot now know, and neither"

p. 41 - Hence AI ceases to be considered a machine, snd instead it becomes a being without temperature, LLMs would be deterministic based on identical prompts.
"First, it testifies that the folk wisdom recognizes the essential characteristic of the machine to be its relentless regularity, its blind obedience of the law of which it is an embodiment."

p. 43 - blueprint is not design
"The blueprint of a machine is not its design. If it were, then the design would change whenever the blueprint is enlarged or redrawn in another color. No, a design is an abstract idea, just as is a functional specification. And ideas, say, the idea of a perpetual-motion machine, are not bound by the laws of physics."

p. 47 - algorithm
"To the contrary, they command precisely what he must do to make that transition. Such a set of rules—that is, a set of rules which tells a player precisely how to behave from one moment to the next—is called an effective procedure. The notion of an effective procedure, or "algorithm," as it is also called, is one of the most important in modern mathematics."

p. 48 - an agents foreign language
"For if an agent competent in only one language were given a procedure written in a language strange to him, together with rules that dictate precisely how to interpret statements in the strange language, then he could imitate what the behavior of a speaker of the strange language would have been had that speaker followed the given procedure. We need therefore some absolutely unambiguous language in which we can write effective procedures and in terms of which we can state rules for interpreting statements in other languages."

p. 49 - a formal language is a game
"A formal language is (again!) a game. Let us return briefly to a consideration of the game of chess. It consists of a set of pieces, a board having a certain configuration, a specification of the initial positions of the pieces on the board, and a set of"

p. 61 - can describe any language. but language is ambiguous
"My task is therefore reduced to showing that a unique alphabet, and a language on that alphabet, can be found in which we can indeed describe all languages in which we may want to write procedures."

p. 63 - any algorithm
"Turing answered that question as well: a Turing machine can be built to realize any process that could naturally be called an effective procedure."

p. 64 - the map is not the territory
"Such a way of knowing is very weak. We do not say we know a city, let alone that we understand it, solely on the basis of having a detailed map of it. Apart from that, if we understand the language in which a procedure is written well enough to be able to explicate its transformation rules, we probably understand what rules stated in that language tell us to do."

p. 64 - great quote. what is computable? everything, but always at the price of reduction
"Turing's thesis tells us that we can realize, as a computer program, any procedure that could "naturally" be called an effective procedure. Therefore, whenever we believe we understand a phenomenon in terms of knowing its behavioral rules, we ought to be able to express our understanding in the form of a computer program. Turing proved that all computers (save a few special-purpose types that do not concern us here) are equivalent to one another, i.e., are all universal. Hence any failure of a technically well-functioning computer to behave precisely as we believe we have programmed it to behave cannot be attributed to any peculiarity of the specific computer we have used. Indeed, the fault must be that we have been careless in our transcription of the behavioral rules we think we understand into the formal language demanded by our computer, or must be in the initial explication, in any form, of what we had in mind when we believed we understood, or must be that our understanding is defective. The last is most often the case. I shall say much more about that later. For now, we need note only that the defect in our understanding can take two forms:"

p. 65 - also great. reduction again. computers do things well, humans too, but for different reasons, and always in conjunction. problem is that the algorithm can turn war the world to fit its will if we allow it to.. howbare LLMs doing this?
"First, although our theory may be on the whole correct, it may contain an error in detail. We wrongly assert, for example, that if this and that is true, then so-and-so follows. Our mental processes, lulled perhaps by the sheer eloquence of the argument we make to ourselves, often permit us to slide over such errors without the slightest disturbance. The computer is, however, very unforgiving in this respect. It follows the logic we have given it. That logic may lead to very different consequences than do mental processescontaminated by wishes to reach certain outcomes. Indeed, one of the most cogent reasons for using computers is to expose holes in our thinking. Computers are merciless critics in this respect. Second, the defect in our understanding may be that, although it is true that we understand, we are still not able to formalize our understanding. We may, for example, be able to predict with very great confidence what an animal will do under a large variety of circumstances. But our predictive power, great and reliable as it may be, may rest on intuitions that we are simply unable to adequately explicate. Yet we may be driven to force our ideas into a formal mold anyway. A computer program based on a formal system so derived is certain to misbehave. The trouble then is not merely that the theory it represents contains certain errors in detail, but that that theory is grossly wrong in what it asserts about the matters it concerns. It is not always clear which defect one is confronted with when a computer one has programmed misbehaves. There is usually enormous motivation to believe that one's theories are all right on the whole, and that, when they don't work well, there must be some error in detail that can easily be patched up. I shall have more to say about such matters later."

p. 67 - no
"Are all the decisionmaking processes that humans employ reducible to effective procedures and hence amenable to machine computation?"

p. 68 - words and meanings
"Consider the English sentence: "I never met a man who is taller than John." It may be transformed into "I never met a taller man than John." This transformation clearly preserves the meaning of the original sentence. But if we apply the same transformation rule to "I never met a man who is taller than Maria," and get "I never met a taller man than Maria," it no longer works. The rule we have applied is not purely syntactic. It concerns itself not merely with the form of uninterpreted strings of symbols, but with their meanings as well."

p. 69 - the laws of algorithms must reflect the values towards ehich society wishes to move.but as we have shown, this is not so simple. the sheer complexity of some algorithms indeed mskes them even more difficult to understand... the ai con speak about the importance of seeing data sets and the example here is fsirly simple... imagine real ambiguity
"If we wish to continue to identify languages with machines even when discussing natural language, then we must recognize that, whatever machines correspond to natural languages, they are more like machines that transform energy and deliver power than like the abstract machines we have been considering; i.e., their laws must take cognizance of the real world. Indeed, the demands placed on them are, if anything, more stringent than those placed on mere engines. For although the laws of engines are merely subsets of the laws of physics, the laws of a natural-language machine must somehow correspond to the inner realities manifest and latent in the person of each speaker of the language at the time of his speaking. Natural language is difficult in this sense because we have to know, for example, to what "values" of X and Y we can apply the transformation rule that takes us from "I never met an X who is taller than Y" to "I never met a taller X than Y." It is clearly not a rule uniformly applicable to uninterpreted strings of symbols."

p. 71 - the sheer complexity of the real world is insane
"What is so remarkable is how incredibly few things we must know in order to have access, in principle, to all of mathematics. In ordinary life we give each other directions, i.e., describe procedures to one another, that, although perhaps technically ambiguous in that they are potentially subject to various interpretations, are, for all practical purposes, effective procedures. They rest at bottom on extremely widely shared vocabularies whose elements, when they appear in highly conventionalized contexts, have effectively unique interpretations. Most professional and technical conversations avail themselves of such vocabularies almost exclusively. The problem of converting such procedures into effective procedures in the technical sense, i.e., into programs for Turing machines, is fundamentally one of formalizing the knowledge base that underlies the conventional interpretation of their vocabularies. The more highly standardized these vocabularies are, and the more restricted the context in which they are used, the more likely that this problem can be solved. For, after all, if each symbol of a set of symbols has an effectively unique interpretation in a certain context, and if strings of such symbols are transformed only by rules that themselves arise out of that context, then no question of giving each symbol an interpretation arises in any formal sense at all. A language so constrained is effectively a formal language. Its rules are therefore potentially realizable by a Turing machine."

p. 72 - we know more than we can tell.check source
"We join Michael Polanyi in saying that we know more than we can tell."

p. 72 - as do i
"The chess master knows more than he can tell. I am not saying here (although I believe this to be true) that we can never find a way to explicate the whole of his knowledge of chess; I say only that we have in this an example of knowledge that is effective even though not presently tellable. Were it true that no chess master's knowledge of chess is fully tellable, would that imply that no computer could ever play master-class chess? Not at all. We shall, as I have said, deal with such questions in what follows."

Chapter 3: How Computers Work

p. 73 - bicycles are not airplanes
"view, modern computers are simply Turing machines that operate on an alphabet consisting of the two symbols "0" and "1" and that are capable of taking on an astronomical number of states. But this is like saying that, because both bicycles and modern passenger aircraft are vehicles for transporting people, they are formally identical. The modern computer differs from the Turing machines we have been discussing both in the way it is constructed and the way it is instructed"

p. 99 - q
"An assembler, then, also transforms the computer into which it is loaded. The transformation it induces has important consequences: a programmer who instructs a computer using only an assembler need never learn the language determined by the computer itself, i.e., its machine language. In an important sense, he never sees the machine he is actually addressing; he sees and works with a symbolic artifact that, for him, is the machine."

p. 101 - same is true for LLMs, they make the choices
"In a very real sense, every freedom an author exercises in making a choice is one he has usurped from the programmer who is to use his product. Consider"

p. 102 - there it is.
"The recent history of the behavioral sciences has shown us how deeply the success of mathematics as used in physics has affected disciplines quite far removed from physics. Many psychologists and sociologists have for generations discussed their subject matter in terms of differential equations and statistics, for example. They may have begun by believing that the calculi they adopted were merely a convenient shorthand for describing the phenomena with which they deal. But, as they construct ever larger conceptual frameworks out of elementary components originally borrowed from foreign contexts, and as they give these frameworks names and manipulate them as elements of still more elaborate systems of thought, these frameworks cease to serve as mere modes of description and become, like Maslow's hammer, determinants of their view of the world. The design of a public language, then, is a serious task, one pregnant with consequences and thus laden with extraordinarily heavy responsibility. I shall have more to say about that later."

p. 103 - another good one
"Lest this be thought a "merely" philosophical point of little practical consequence, I must say immediately that not only do most of today's programmers think of the languages they use as being their machines, very nearly literally so, but many, perhaps most, have no knowledge whatever of their computer's machine language or of the content and structure of the translators that mediate between them and their computers. This observation is not made as a criticism. After all, a higher-level formal language is an abstract machine. No one is to be faulted for using a language or a machine more congenial to his purposes than is some other machine, merely because the other machine is somehow more primitive. But the observation does raise an important question: If today's programmers are largely unaware of the detailed structures of the physical machines they are using, of their languages, and of the translators that manipulate their programs, then they must also be largely ignorant of many of the arguments I have made here, particularly of those arguments concerning the universality of computers and the nature of effective procedures. How then do these programmers come to sense the power of the computer?"

p. 108 - programming is or used tocbe a test of undersganding on writing!
"The idea that a person can write a program that embodies anything he "thoroughly understands" is at least equally problematical. Understanding something always means understanding it at a certain level. An actuary uses some fairly sophisticated mathematical tools whose fundamentals he almost certainly doesn't understand or care anything about. Everyone who makes change uses arithmetic, but very few people know or care much about the beautiful axiom system on which arithmetic is based. In effect, we all constantly use subroutines whose input-output behavior we believe we know, but whose details we need not and rarely do think about. To understand something sufficiently well to be able to program it for a computer does not mean to understand it to its ultimate depth. There can be no such ultimate understanding in practical affairs. Programming is rather a test of understanding. In this respect it is like writing; often when we think we understand something and attempt to write about it, our very act of composition reveals our lack of understanding even to ourselves. Our pen writes the word "because" and suddenly stops. We thought we understood the "why" of something, but discover that we don't. We begin a sentence with "obviously"; and then see that what we meant to write is not obvious at all. Sometimes we connect two clauses with the word "therefore," only to then see that our chain of reasoning is defective. Programming is like that. It is, after all, writing too. But in ordinary writing we sometimes obscure our lack of understanding, our failures in logic, by unwittingly appealing to the immense flexibility of natural language and to its inherent ambiguity. The very eloquence that natural language permits sometimes illuminates our words and seems (falsely, to be sure) to illuminate our undeserving logic just as brightly. An interpreter of programming-language texts, a computer, is immune to the seductive influence of mere eloquence. And words like "obviously" are not represented in the primitive vocabularies of any computers. A computer is a merciless critic. Therefore the assertion that one understands a thing sufficiently well to be able to program it is, first of all, an assertion that one understands it in very particular terms. In any case, it can be no more than a boast that may well be falsified by experience."

p. 110 - amateur programmers
"Indeed, many professional programmers believe that their craft is difficult because the languages with which they must deal have rigid syntactical rules. There is therefore a persistent cry for natural-language, e.g., English, programming systems. Programmers who hold to this belief have probably never tackled a truly difficult problem, and have therefore never felt the need for really deep criticism from the computer. It is true that in order to write one has first to master the syntactic rules of one's chosen language. But then one must also have thought through what one has to say."

p. 110 - how do LLMs change this metaphor?
"A real reason that programming is very hard is that, in most instances, the computer knows nothing of those aspects of the real world that its program is intended to deal with. It is in a position analogous to, say, that of a teacher of English grammar who has been given an airplane pilot's manual and left at a plane's controls in midair. He may be a very skilled grammatical faultfinder, but as to how to fly an airplane, he knows nothing."

p. 110 - q
"How much easier it would be to write a pilot's manual for an aerodynamicist who, while still unable to pilot an airplane, knew a theory of flight. It is in fact very hard to explain anything in terms of a primitive vocabulary that has nothing whatever to do with that which has to be explained. Yet that is precisely what most programs attempt to do. The difficulties that ensue are no more rooted in syntactic rigidities than is, say, the difficulty of writing a good sonnet rooted in the rigid form demanded by that class of poem. To write a good sonnet or a good program, one must know what one wants to say. And it helps enormously if one's critic shares one's relevant knowledge base."

p. 110 - facts are not understanding. quite similar to chinese room
"But we mustn't make too much of this. A lazy student's list of useful physics formulas may enable him to solve many schoolbook problems in physics, but will still not give him an understanding of physics, a theory he can think about. Just so, a set of assertions of the kind we have shown may enable a computer to solve certain problems. But this says nothing of a programmer's or of a computer's mastery of a theory or of either's understanding of anything more than how to use a group of "facts" to arrive at certain conclusions. A computer's successful performance is often taken as evidence that it or its programmer understand a theory of its performance. Such an inference is unnecessary and, more often than not, is quite mistaken. The relationship between understanding and writing thus remains as problematical for computer programming as it has always been for writing in any other form."