• combinational: Novel combinations of familiar things/ideas leading to an "intelligible conceptual pathway between them for the combination to ‘‘make sense’’" (p.32) rather than just a random combination.
• exploratory: Conceptual spaces are pre-existing "structured styles of thought" (p.32). Ways of writing, sculptural forms, navigating geography and physical space, etc. Not all ways of navigating those spaces have been explored, although they do exist; new possibilities might be found and this might lead further to questioning the bounds of that space and what further potential possibilities might exist . . .
• transformational: If exploratory creativity leads to questioning the limits of the conceptual space, transformational creativity takes it the obvious step further. Some conceptual spaces (e.g., a cartographic map) can't be changed; mental maps can be changed. "[T]hinking something which, with respect to the conceptual spaces in their minds, they couldn’t have thought before. The supposedly impossible idea can come about only if the creator changes the pre-existing style in some way" (p.34).

A funny contrast to the promises of LLMs scaling laws that have gradually faded away. . .

AI does not have a human's sense of relevance, look at Boden's funny example:

"My own favorite example is: If a man of twenty can pick ten pounds of blackberries in an hour, and a woman of eighteen can pick eight, how many will they gather if they go blackberrying together? For sure, “eighteen” isn’t a plausible answer. It could be much more (because they’re both showing off) or, more probably, much less. This example was even more telling fifty years ago, when I first encountered it. But why is that? Just what kinds of knowledge are involved here? And could an AGI overcome what appear to be the plain arithmetical facts?"

While I understand the connection, it falls flat for me in the sense that there is no such thing as a sociocultural existence for AI agents, so there can be no distinction between them really... It seems to me to be a way of thinking about the problem rather than being an actual part of the problem, precisely because a hypothetical AGI would not have the limitations of humans, so once the system is in place, it would theoretically operate as one and would not actually need distributed cognition.

"However (as remarked at the outset of this chapter), full AGI would do very much more. Difficult though it is to build a high-performing AI specialist, building an AI generalist is orders of magnitude harder. (Deep learning isn’t the answer: its aficionados admit that “new paradigms are needed” to combine it with complex reasoning—scholarly code for “we haven”t got a clue”.)"

While some still believe LLMs to be the path to AGI, several researchers are leaving this path - LeCun, Sutton and Silver. . .

"In particular, there are very few combinational systems. One might think it’s easy to model combinational creativity. After all, nothing could be simpler than making a computer produce unfamiliar associations of already stored ideas. The results will often be historically novel, and (statistically) surprising. But if they’re also to be valuable, they must be mutually relevant. That’s not straightforward, as we’ve seen. The jokegenerating programs mentioned in Chapter 2 use joke templates to help provide relevance. Similarly, symbolic AI’s case-based reasoning constructs analogies thanks to precoded structural similarities. So, their “combinational” creativity has a strong admixture of exploratory creativity as well."

I still maintain that AI's inablity to judge its own output makes the claims of creativity dubious. Humans will often judge their own outputs inaccurately, but they judge them none-the-less, and I think that the framing problem is equally relevant here.

This is a false equivalence, in my opinion. Before the paint brush came similar tools, which ultimately led to somebody imagining it in the first place. The very first imagining of what kind of a painting the paint brush would produce is probably very far from the refined concept of painting that we have today, but it could still all be traced further back. Perhaps to chalk on a cavewall, or to cutting in a tree - nevertheless these things paved the way, and at some point the very idea of imagining something as simple as a painting would've been equally unimaginable. 

What makes the claim to the 'uninmaginativeness' of computer-generated art stronger in this case is that the digital domain does not abide by physical rules, and as such our ability to imagine in the days of less advanced computers was significantly more limited. I can now readily imagine countless different types of CG visual art because we now have cognitive models that allow me to imagine it. 

Even then, this imaginativeness must have been gradual, but yet with someone having a rough imagination of what drawing on the computer might look like. Once realising a simple version of this, their ability to imagine these artworks gradually expand, so ultimately I see no difference between this example and of other tools. It is a tool all the same, and the argument of "more-than" in this case is not strong.

If we are ever to achieve AGI, emotions such as anxiety will have to be included—and used"

While I agree, I think Boden's view here is too strongly suggesting that 'using' emotions as computational mechanisms can really be considered anything like real emotions, and would be severely limited in comparison. Ultimately all this is doing is mimicking the behaviour by introducing something more analogous to a decision tree than actual emotion, making it rather ambiguous. Shouldn't the goal of AGI be a little more lofty than this? 

Let's even assume we actually program an AGI capable of these computational modes of emotion - how are they regulated and balanced? Human emotions can be both productive and destructive, and that is often a claim to human inferiority as opposed to machines (although hypothetical ones), but humans would still be in charge of determining how they shold be distributed in various situations. 

Here the AI scientist will point to AGI's potential super ability to determine what is most suitable given literally any situation (which would suggest ASI, not just AGI), but that takes actual emotion to decide, and it isn't necessarilly a process of active reflection, but rather intuition that takes over, so we're back where we started. 

In short, the virtual machines implemented in our brains are both sequential and parallel. Human intelligence requires subtle cooperation between them. And human-level AGI, if it’s ever achieved, will do so too."

Hybridism is an absolute necessity, and so the idea of any definition of AGI that assumes the mind as separate from the body is of no use. (Arxiv 2025 paper re. AGI definition assumes exactly this...) This further stresses the importance of embodiment or embodied intelligence, which is also what Sutton and Silver, and Lecun are gravitating towards (though without the proper reasoning to boot).

"Female crickets can do this on hearing the song of a conspecific male. However, the cricket can recognize only one song, sung at only one speed and frequency. Speed and frequency vary for different species of cricket. But the female doesn’t choose between different songs, for she doesn’t possess feature detectors coding a range of sounds. She uses a mechanism that’s sensitive only to one frequency. This isn’t a neural mechanism, like the auditory detectors in human brains. It’s a fixed-length tube in her thorax, connected to the ears on her front legs and to her spiracles. The length of the tube is an exact proportion of the wavelength of the male’s song. The physics ensures that phase cancellations (between the air in the tube and the air outside) occur only for songs with the right frequency, and that the intensity difference depends wholly on the direction of the sound source. The female insect is neurally hardwired to move in that direction: he sings, she goes. Situated behavior, indeed."

Boden is positive about AI, but also realistic.

"Skepticism Defended

In my judgement, the S-skeptics are right. The discussion of mind-as-virtual-machine in Chapter 6 implied that there is no obstacle in principle to human-level AI intelligence (possibly excepting phenomenal consciousness). The question here is whether this is likely in practice.

Besides the intuitive implausibility of many post-Singularity predictions, and the near-absurdity (imho) of transhumanist philosophy, the S-skeptics have other arguments on their side.

AI is less promising than many people assume. Chapters 2 through 5 mentioned countless things that current AI cannot do. Many require a human sense of relevance (and tacitly assume completion of the semantic web: see Chapter 2). Moreover, AI has focused on intellectual rationality while ignoring social/emotional intelligence—never mind wisdom. An AGI that could interact fully with our world would need those capacities, too. Add the prodigious richness of human minds, and the need for good psychological/computational theories about how they work, and the prospects for human-level AGI look dim.

Even if it were practically feasible, it’s doubtful whether the necessary funding would materialize. Governments are currently putting enormous resources into brain emulation (see the following section), but the money required for building artificial human minds would be much greater still.

Thanks to Moore’s Law, further AI advances can certainly be expected. But increases in computer power, and in data availability (given cloud storage, and 24/7 sensors throughout the Internet of Things), won’t guarantee human-like AI. That’s bad news for S-believers, because ASI needs AGI first.

S-believers ignore the limitations of current AI. They simply don’t care, because they have a trump card: the notion that exponential technological advance is rewriting all the rule books. This licenses them to make predictions at will. They occasionally allow that “by-end-of-the-century” predictions may be unrealistic. However, they insist that “never” is a long time.

Never is indeed a long time. So the S-skeptics, myself included, may be wrong. They have no knock-down argument— especially if they allow AGI’s possibility in principle (as I do). They may even be persuaded that the Singularity, albeit hugely delayed, will happen eventually.

Nevertheless, careful consideration of state-of-the-art AI gives good reason to back the skeptics’ hypothesis (or their bet, if you prefer), rather than the wild speculations of the S-believers."

"Moreover, AI has focused on intellectual rationality while ignoring social/emotional intelligence—never mind wisdom. An AGI that could interact fully with our world would need those capacities, too. Add the prodigious richness of human minds, and the need for good psychological/computational theories about how they work, and the prospects for human-level AGI look dim"

Boden is a skeptic as well. 

"In my judgement, the S-skeptics are right. The discussion of mind-as-virtual-machine in Chapter 6 implied that there is no obstacle in principle to human-level AI intelligence (possibly excepting phenomenal consciousness). The question here is whether this is likely in practice.

Besides the intuitive implausibility of many post-Singularity predictions, and the near-absurdity (imho) of transhumanist philosophy, the S-skeptics have other arguments on their side."

"it is the virtual machine that’s crucial here (see Chapters 1 and 6). Some virtual machines can be implemented only in hugely powerful hardware. So mega-transistored computer chips may well be necessary. But just what computations will they be performing? In other words, just what virtual machines will be implemented in them? To match human (or even mouse) intelligence, these will need to be informationally powerful, in ways that computational psychologists don’t yet fully understand.

Let’s suppose—what is, I think, unlikely—that every neuron in a human brain will, eventually, be mapped. In itself, this won’t tell us what they are doing. (The tiny nematode worm C. elegans has only 302 neurons, whose connections are known precisely. But we can’t even identify the synapses as excitatory/inhibitory.)"

The nematode worm is fully mapped, but we don't understand the synapses fully. Even if we did, would it be a nematode worm? (as it relates to the physicalist view)

Virtual machines are required for every little thing

"Even the mundane activity of eating with a knife and fork requires that many virtual machines be integrated—some dealing with physical objects (muscles, fingers, utensils, various kinds of sensors), others with intentions, plans, expectations, desires, social conventions, and preferences. To understand how that activity is possible, we need not only neuroscientific data about the brain, but also detailed computational theories about the psychological processes involved."

Assuming that all these things add up to an understanding of the task is still banking entirely on emergence of a mind through these little bits and pieces. There is still no intentionality (Merleau-Pontian), let alone Searly intentionality.

"Like the fisherman in the fairy story, whose wish for his soldier-son to come home was granted by his being brought back in a coffin, we could be nastily surprised by powerful AI systems lacking our understanding of relevance."

O'Callaghan presents 3 theories of sound:

1. Sounds are properties of bodies. A view held from Locke onwards -- bodies and objects possess sounds when they vibrate at particular frequencies and amplitudes.

2. Classic acoustic theory. Sounds are waves produced by vibrating bodies -- thus, we do not immediately hear the property a body 'possesses' but only hear the sound as a compression wave through a medium.

3. Sounds are events. O'Callaghan arrives at the 'event view' of sound from Aristotle's writings: sound is a movement (as per Aristotle) but O'Callaghan interprets this as -- the movement need not be the medium but can be the event that causes the medium's disturbance/movement. Sound is 'the act of one thing moving another' (p.27). cf Aristotle (Arisotle 350 BCE).

Discussing Libet's (1985) experiment. While we prefer to believe that we have willed our hand to rise, our intention, there is a precursor in the subconscious, the readiness potential, that can be measured using EEG and that precedes the hand movement and the reported time at which subjects noted their intention to lift the hand. The conclusion is that intention (and thus free will?) is manufactured after the event. See also (Rosenberg 2018, p.99).

Detailing a set of experiments by Libet helping to explain how our brain builds a consciousness of the present and the time delay involved.

• Stimulating the brain directly, an electrical impulse must be applied for at least 500 milliseconds to produce a perception (see, for example Libet 1985). Shorter impulses had no effect neither did increasing the intensity (with shorter times). Sensation can be detected by the brain within 500msecs but the subject is not consciously aware of it (e.g. we can react 'instinctively').
• Stimulate the skin then, 200 milliseconds after, stimulate the brain – the skin stimulation is not perceived but is masked by the brain stimulation. The brain edits past events to give an impression of the 'present'.
• Stimulate the brain then, 200 milliseconds later, stimulate the skin. Brain stimulus is perceived after about 500msecs but the skin stimulus is perceived as being before the brain stimulus. A temporal reordering: "There is no guarantee that the order in which we perceive events actually corresponds to the order of their occurrence" (Westerhoff 2011, p.100).