"I said earlier that ‘new’ has two meanings, and that ‘surprising’ has three. I didn’t say how many meanings ‘valuable’ has – and nobody could. Our aesthetic values are difficult to recognize, more difficult to put into words, and even more difficult to state really clearly. (For a computer model, of course, they have to be stated really, really clearly.)

Moreover, they change: who will proudly admit, today, to having worn a beehive hairdo or flared trousers in the 1960s? They vary across cultures. And even within a given ‘culture’, they are often disputed: different subcultures or peer groups value different types of dress, jewellery or music. And where transformational creativity is concerned, the shock of the new may be so great that even fellow artists find it difficult to see value in the novel idea.

Even in science, values are often elusive and sometimes changeable. Just what ‘simplicity’ or ‘elegance’ means, as applied to scientific theories, is something that philosophers of science have long tried – and failed – to pin down precisely. And whether a scientific finding or hypothesis is ‘interesting’ depends on the other theories current at the time, and on social questions too (might it have some medical value, for instance?).

Because creativity by definition involves not only novelty but value, and because values are highly variable, it follows that many arguments about creativity are rooted in disagreements about value. This applies to human activities no less than to computer performance. So even if we could identify and program our aesthetic values so as to enable the computer to inform and monitor its own activities accordingly there would still be disagreement about whether the computer even appeared to be creative."

Parallel to intelligece. Intelligence is also tied to several value parameters, and as Boden argues, creativity is one of them. Those who accumulate great wealth and thereby power are often viewed as intelligent, but the two need not have any correlation, for example. What constitutes value is often what is viewed as succesful, but surely that is a reduction of such a rich concept.

• combinational: Novel combinations of familiar things/ideas leading to an "intelligible conceptual pathway between them for the combination to ‘‘make sense’’" (p.32) rather than just a random combination.
• exploratory: Conceptual spaces are pre-existing "structured styles of thought" (p.32). Ways of writing, sculptural forms, navigating geography and physical space, etc. Not all ways of navigating those spaces have been explored, although they do exist; new possibilities might be found and this might lead further to questioning the bounds of that space and what further potential possibilities might exist . . .
• transformational: If exploratory creativity leads to questioning the limits of the conceptual space, transformational creativity takes it the obvious step further. Some conceptual spaces (e.g., a cartographic map) can't be changed; mental maps can be changed. "[T]hinking something which, with respect to the conceptual spaces in their minds, they couldn’t have thought before. The supposedly impossible idea can come about only if the creator changes the pre-existing style in some way" (p.34).

"Moreover, AI has focused on intellectual rationality while ignoring social/emotional intelligence—never mind wisdom. An AGI that could interact fully with our world would need those capacities, too. Add the prodigious richness of human minds, and the need for good psychological/computational theories about how they work, and the prospects for human-level AGI look dim"

"Like the fisherman in the fairy story, whose wish for his soldier-son to come home was granted by his being brought back in a coffin, we could be nastily surprised by powerful AI systems lacking our understanding of relevance."

Virtual machines are required for every little thing

"Even the mundane activity of eating with a knife and fork requires that many virtual machines be integrated—some dealing with physical objects (muscles, fingers, utensils, various kinds of sensors), others with intentions, plans, expectations, desires, social conventions, and preferences. To understand how that activity is possible, we need not only neuroscientific data about the brain, but also detailed computational theories about the psychological processes involved."

Assuming that all these things add up to an understanding of the task is still banking entirely on emergence of a mind through these little bits and pieces. There is still no intentionality (Merleau-Pontian), let alone Searly intentionality.

"it is the virtual machine that’s crucial here (see Chapters 1 and 6). Some virtual machines can be implemented only in hugely powerful hardware. So mega-transistored computer chips may well be necessary. But just what computations will they be performing? In other words, just what virtual machines will be implemented in them? To match human (or even mouse) intelligence, these will need to be informationally powerful, in ways that computational psychologists don’t yet fully understand.

Let’s suppose—what is, I think, unlikely—that every neuron in a human brain will, eventually, be mapped. In itself, this won’t tell us what they are doing. (The tiny nematode worm C. elegans has only 302 neurons, whose connections are known precisely. But we can’t even identify the synapses as excitatory/inhibitory.)"

The nematode worm is fully mapped, but we don't understand the synapses fully. Even if we did, would it be a nematode worm? (as it relates to the physicalist view)

Boden is a skeptic as well. 

"In my judgement, the S-skeptics are right. The discussion of mind-as-virtual-machine in Chapter 6 implied that there is no obstacle in principle to human-level AI intelligence (possibly excepting phenomenal consciousness). The question here is whether this is likely in practice.

Besides the intuitive implausibility of many post-Singularity predictions, and the near-absurdity (imho) of transhumanist philosophy, the S-skeptics have other arguments on their side."

Boden is positive about AI, but also realistic.

"Skepticism Defended

In my judgement, the S-skeptics are right. The discussion of mind-as-virtual-machine in Chapter 6 implied that there is no obstacle in principle to human-level AI intelligence (possibly excepting phenomenal consciousness). The question here is whether this is likely in practice.

Besides the intuitive implausibility of many post-Singularity predictions, and the near-absurdity (imho) of transhumanist philosophy, the S-skeptics have other arguments on their side.

AI is less promising than many people assume. Chapters 2 through 5 mentioned countless things that current AI cannot do. Many require a human sense of relevance (and tacitly assume completion of the semantic web: see Chapter 2). Moreover, AI has focused on intellectual rationality while ignoring social/emotional intelligence—never mind wisdom. An AGI that could interact fully with our world would need those capacities, too. Add the prodigious richness of human minds, and the need for good psychological/computational theories about how they work, and the prospects for human-level AGI look dim.

Even if it were practically feasible, it’s doubtful whether the necessary funding would materialize. Governments are currently putting enormous resources into brain emulation (see the following section), but the money required for building artificial human minds would be much greater still.

Thanks to Moore’s Law, further AI advances can certainly be expected. But increases in computer power, and in data availability (given cloud storage, and 24/7 sensors throughout the Internet of Things), won’t guarantee human-like AI. That’s bad news for S-believers, because ASI needs AGI first.

S-believers ignore the limitations of current AI. They simply don’t care, because they have a trump card: the notion that exponential technological advance is rewriting all the rule books. This licenses them to make predictions at will. They occasionally allow that “by-end-of-the-century” predictions may be unrealistic. However, they insist that “never” is a long time.

Never is indeed a long time. So the S-skeptics, myself included, may be wrong. They have no knock-down argument— especially if they allow AGI’s possibility in principle (as I do). They may even be persuaded that the Singularity, albeit hugely delayed, will happen eventually.

Nevertheless, careful consideration of state-of-the-art AI gives good reason to back the skeptics’ hypothesis (or their bet, if you prefer), rather than the wild speculations of the S-believers."

"Female crickets can do this on hearing the song of a conspecific male. However, the cricket can recognize only one song, sung at only one speed and frequency. Speed and frequency vary for different species of cricket. But the female doesn’t choose between different songs, for she doesn’t possess feature detectors coding a range of sounds. She uses a mechanism that’s sensitive only to one frequency. This isn’t a neural mechanism, like the auditory detectors in human brains. It’s a fixed-length tube in her thorax, connected to the ears on her front legs and to her spiracles. The length of the tube is an exact proportion of the wavelength of the male’s song. The physics ensures that phase cancellations (between the air in the tube and the air outside) occur only for songs with the right frequency, and that the intensity difference depends wholly on the direction of the sound source. The female insect is neurally hardwired to move in that direction: he sings, she goes. Situated behavior, indeed."

In short, the virtual machines implemented in our brains are both sequential and parallel. Human intelligence requires subtle cooperation between them. And human-level AGI, if it’s ever achieved, will do so too."

Hybridism is an absolute necessity, and so the idea of any definition of AGI that assumes the mind as separate from the body is of no use. (Arxiv 2025 paper re. AGI definition assumes exactly this...) This further stresses the importance of embodiment or embodied intelligence, which is also what Sutton and Silver, and Lecun are gravitating towards (though without the proper reasoning to boot).

If we are ever to achieve AGI, emotions such as anxiety will have to be included—and used"

While I agree, I think Boden's view here is too strongly suggesting that 'using' emotions as computational mechanisms can really be considered anything like real emotions, and would be severely limited in comparison. Ultimately all this is doing is mimicking the behaviour by introducing something more analogous to a decision tree than actual emotion, making it rather ambiguous. Shouldn't the goal of AGI be a little more lofty than this? 

Let's even assume we actually program an AGI capable of these computational modes of emotion - how are they regulated and balanced? Human emotions can be both productive and destructive, and that is often a claim to human inferiority as opposed to machines (although hypothetical ones), but humans would still be in charge of determining how they shold be distributed in various situations. 

Here the AI scientist will point to AGI's potential super ability to determine what is most suitable given literally any situation (which would suggest ASI, not just AGI), but that takes actual emotion to decide, and it isn't necessarilly a process of active reflection, but rather intuition that takes over, so we're back where we started. 

This is a false equivalence, in my opinion. Before the paint brush came similar tools, which ultimately led to somebody imagining it in the first place. The very first imagining of what kind of a painting the paint brush would produce is probably very far from the refined concept of painting that we have today, but it could still all be traced further back. Perhaps to chalk on a cavewall, or to cutting in a tree - nevertheless these things paved the way, and at some point the very idea of imagining something as simple as a painting would've been equally unimaginable. 

What makes the claim to the 'uninmaginativeness' of computer-generated art stronger in this case is that the digital domain does not abide by physical rules, and as such our ability to imagine in the days of less advanced computers was significantly more limited. I can now readily imagine countless different types of CG visual art because we now have cognitive models that allow me to imagine it. 

Even then, this imaginativeness must have been gradual, but yet with someone having a rough imagination of what drawing on the computer might look like. Once realising a simple version of this, their ability to imagine these artworks gradually expand, so ultimately I see no difference between this example and of other tools. It is a tool all the same, and the argument of "more-than" in this case is not strong.

"In particular, there are very few combinational systems. One might think it’s easy to model combinational creativity. After all, nothing could be simpler than making a computer produce unfamiliar associations of already stored ideas. The results will often be historically novel, and (statistically) surprising. But if they’re also to be valuable, they must be mutually relevant. That’s not straightforward, as we’ve seen. The jokegenerating programs mentioned in Chapter 2 use joke templates to help provide relevance. Similarly, symbolic AI’s case-based reasoning constructs analogies thanks to precoded structural similarities. So, their “combinational” creativity has a strong admixture of exploratory creativity as well."

I still maintain that AI's inablity to judge its own output makes the claims of creativity dubious. Humans will often judge their own outputs inaccurately, but they judge them none-the-less, and I think that the framing problem is equally relevant here.

"However (as remarked at the outset of this chapter), full AGI would do very much more. Difficult though it is to build a high-performing AI specialist, building an AI generalist is orders of magnitude harder. (Deep learning isn’t the answer: its aficionados admit that “new paradigms are needed” to combine it with complex reasoning—scholarly code for “we haven”t got a clue”.)"

While some still believe LLMs to be the path to AGI, several researchers are leaving this path - LeCun, Sutton and Silver. . .

While I understand the connection, it falls flat for me in the sense that there is no such thing as a sociocultural existence for AI agents, so there can be no distinction between them really... It seems to me to be a way of thinking about the problem rather than being an actual part of the problem, precisely because a hypothetical AGI would not have the limitations of humans, so once the system is in place, it would theoretically operate as one and would not actually need distributed cognition.

AI does not have a human's sense of relevance, look at Boden's funny example:

"My own favorite example is: If a man of twenty can pick ten pounds of blackberries in an hour, and a woman of eighteen can pick eight, how many will they gather if they go blackberrying together? For sure, “eighteen” isn’t a plausible answer. It could be much more (because they’re both showing off) or, more probably, much less. This example was even more telling fifty years ago, when I first encountered it. But why is that? Just what kinds of knowledge are involved here? And could an AGI overcome what appear to be the plain arithmetical facts?"

A funny contrast to the promises of LLMs scaling laws that have gradually faded away. . .