A Multi-Level view of LLM Intentionality

Prompted by Keith Frankish's recent streamed discussion of LLM intentionality on YouTube, there's a particular idea I wanted to share which I'm not sure is widely enough appreciated but which I think gives a valuable perspective from which to think about LLMs and what kinds of intentions they may have. This is not an original idea of my own -- at least some of my thinking on this was sparked by reading about the Waluigi problem on LessWrong.

In this post, I'm going to be making the case that LLMs might have intentionality much like ours, but please understand that I'm making a point in principle and not so much arguing for the capabilities of current LLMs, which are probably not there yet. I'm going to be talking about what scope there is to give the benefit of the doubt to arbitrarily competent future LLMs, albeit ones that follow more or less the same paradigms as those of today. I'm going to try to undermine some proposed reasons for skepticism about the intentions or understanding of LLMs, not because I think the conclusions are wrong but because I think the arguments are too weak to support them.

I should also note that I will follow Keith (and Daniel Dennett, and others) in assuming an interpretivist account of intentions. That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it *really* has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case).

The basic idea I want to introduce is that LLMs might have intentions and agency on multiple levels, so we may be missing something if we restrict our analysis to one level alone.

In Defence of Strong AI: Meta-meaning

Yesterday's post was quite long and wide-ranging, with a lot of examples etc. It focused specifically on how it might be possible to get semantics from syntax, particularly with reference to The Chinese Room thought experiment.

However I feel that I could sum up my position more succinctly by looking at the meaning of the word "meaning" itself.

In Defence of Strong AI: Semantics from Syntax

Is it possible to get semantics from syntax?

If it were impossible to get semantics (meaning) from syntax (symbols and formal rules for manipulating them) then we probably wouldn't have bothered broadcasting the Arecibo image to the universe in the hope of communicating with aliens.

Without any way to explain the meaning of this image, it seems to consist of syntax alone. Whether it could be deciphered by aliens in practice or not, it at least seems possible in principle, which seems to suggest that it may be possible in some cases to derive semantics from syntax.

The assertion that you can't, that there is no way to get from syntax to semantics, is perhaps the strongest response from John Searle to the systems/virtual mind refutation of his Chinese Room thought experiment (as outlined in my post on the topic).

Searle illustrates his point by saying that if he is in The Chinese Room and has a formal system of rules (syntax) for manipulating Chinese words, he has no way to get from this syntax to the semantics of what is being said. If he has no way to do this, then how can the system do it?

In Searle's view, the systems/virtual mind reply to the Chinese Room is just hand-waving until this is explained.

Let's see if I can attempt to do just that.

In Defence of Strong AI: The Chinese Room Refuted

It has become obvious to me that in order to fully explain my world view, I must first defend strong AI.

Strong AI is the position that it is possible in principle to have an artificial intelligence, implemented as a computing machine much like the machines we have today, which would be capable of consciousness, feeling and sensation. This is in contrast to the position of weak AI, which only claims that we could in principle make a computing machine which would merely behave as if it were conscious.

This is important not just for the ethics of how we might treat sentient computers but because it cuts to the heart of what it is that our minds actually are. If our minds are dependent on something other than simple computation, this puts limits on certain questions we might ask about our universe. For example, if computation cannot produce consciousness, then we immediately know that we cannot all be living in a computer simulation.

Firstly, I would like to point out the problems I see with some of the most popular criticisms of strong AI, starting with perhaps the most famous: John Searle's "The Chinese Room". Later I will attempt to build a positive case of my own.