Theories of Mind as Languages of Thought for Thought about Thought
Theories of Mind as Languages of Thought for Thought about Thought: Blending the Advantages of Bayesian Network/Causal Grammar Models and Programming Pattern Models
Further Reading: True AGI's Recursive Rational Probabilistic Programming Language: memo's Implementation, Embodied Games
Abstract
What is a "theory of mind" like? As cognitive scientists, how do we formalize the various forms of theories of mind we study? In this paper, we argue that it is valuable to view theories of mind as a programming language: a language specifically designed for setting up and reasoning about problems involving other minds. Drawing on ideas from the theory and history of programming languages, we show how this perspective can help us formalize concepts in theories of mind, precisely articulate the differences between multiple theories of mind, and reason about how we develop our own theories of mind over time.
Introduction
How should we view theories of mind? Under the influential "theory-theory" view (Carey, 1985; Gopnik & Meltzoff, 1997), we should indeed see a theory of mind as a theory: akin to a scientific theory (Gopnik & Wellman, 1992). But here, what exactly does "theory" mean? How can we precisely and computationally represent a given theory of mind θ? And what exactly is θ like—how can we formalize the space T of possible theories of mind to which θ belongs?
A precise mathematical language capable of representing theories of mind has three key values.
First, such a language allows us to state ideas and hypotheses about theories of mind in a clear and unambiguous way.
Second, such a language enables us to systematically instantiate these ideas and hypotheses into executable computational models, allowing for valuable simulations and analyses—for example, Horschler et al. (2023) recently demonstrated how a series of executable models can be used to characterize theories of mind in non-human primates.
Finally, a systematic computational foundation for theories of mind can be used to design real-world AI systems with human-like social intelligence in a principled way.
Therefore, the goal of this paper is to provide a method for formalizing θ and T. We propose that a theory of mind θ should be viewed as a programming language (PL) in which minds can set up and solve problems involving psychological reasoning about others. Thus, the space T of theories of mind is defined by the space of potential programming languages that can be used for psychological reasoning.
This paper unfolds this view in four stages.
We first discuss two previous attempts to formalize theories of mind and explain why a new interpretation is valuable.
Next, we introduce our view through a concrete example of a programming language that embodies a specific theory of mind. We use this example to show how the design of a programming language concretizes concepts within a theory of mind.
Then, moving from the specific to the general, we discuss how our view helps us formalize the differences between various possible theories of mind based on changes in programming language syntax and semantics.
Finally, we discuss how our view, by drawing on the history of programming language design, provides insights into our understanding of the process of theoretical change.
Three Attempts at Formalizing Theories of Mind
What do we hope to gain from the formalization of theories of mind? In our view, a formal description of theories of mind should at least explain two key features: (1) how theories of mind enable their possessors to efficiently represent and reason about situations involving others' mental states; and (2) how possessors of these theories grow, revise, or entirely replace their theories in the face of new evidence.
Bayesian Network Models One early attempt at formalization (Gopnik et al., 2004; Gopnik & Wellman, 2012) responded to these needs by formalizing theories of mind as causal Bayesian networks (Pearl, 2000). In this model, latent concepts such as an agent's beliefs and desires can be represented as nodes in a Bayesian network, with directed edges pointing to the agent's actions (Figure 1). Tasks such as prediction and planning can be supported by efficient probabilistic inference algorithms (Baker et al., 2008). Ultimately, the theories themselves can also change in the face of new evidence, through hierarchical inference over the space of possible Bayesian networks (Goodman et al., 2006). For example, a child might initially be highly confident that the mind can be well-modeled by the network shown in Figure 1(a). Over time, she might come to prefer the less parsimonious network in Figure 1(b), having found that it more accurately models the mind under perception-limited situations.
This model faces two challenges. First, writing a Bayesian network for every possible social situation is practically infeasible. While an agent's beliefs and desires at a given moment might be representable by a fixed Bayesian network, human reasoning about minds is infinite, generative, and recursive: we can just as easily reason about an agent's beliefs about another's beliefs, and these change dynamically with behavior. This suggests that fixed Bayesian networks are insufficient for representing theories of mind, challenging the first requirement (1). Second, this model does not fully explain how an individual develops a theory. Goodman et al.'s (2006) model considers inference between two pre-set Bayesian networks but does not explain the origin of these hypotheses themselves, challenging the second requirement (2).
To address these challenges, Tenenbaum et al. (2007) proposed formalizing intuitive theories as causal grammars rather than fixed Bayesian networks, where the former can generate a series of situation-specific Bayesian networks. Causal grammars support reasoning about an infinite number of situations within a domain and enable learning by inferring over the hypotheses generated by the grammar. However, the formal system of "grammars" is still insufficient to express the full richness of theories of mind—for example, it cannot express recursive beliefs about beliefs (Griffiths & Tenenbaum, 2007).
Programming Pattern Models Another approach stems from the development of general-purpose probabilistic programming languages (PPLs) such as Church (Goodman et al., 2012) and WebPPL (Goodman & Stuhlmüller, 2014). In PPLs, complex probabilistic models can be generated through simple programs: for example, repetitive nodes in a Bayesian network can be automatically produced by loops.
Building on this idea, Goodman et al. (2015) viewed theories of mind as a "toolbox of programming patterns" within a general-purpose PPL. For instance, in this framework, representing the behavior of a rational, goal-driven agent corresponds to the "planning-as-inference" programming pattern (Botvinick & Toussaint, 2012), where the agent infers "which actions might have led to the achievement of that goal" to achieve a certain objective. Specifically, in WebPPL, we might write code like this:
Like causal grammars, these programming patterns allow programmers to flexibly generate situation-specific probabilistic models. However, these patterns are much more powerful than causal grammars. For example, representing "thought about thought" or recursive social inference corresponds to the nested inference pattern: running an inference query on a probabilistic model that is itself recursively executing inference queries (Stuhlmüller & Goodman, 2014; Zhang & Amin, 2022).
Specifically, if we want to infer an agent's goal based on observed behavior observed_action, which is to perform Inverse Planning (Baker, Saxe, & Tenenbaum, 2009), we might write it like this:
Like causal grammars, these programming patterns allow programmers to flexibly generate situation-specific probabilistic models. However, these patterns are much more powerful than causal grammars. For example, representing "thought about thought" or recursive social inference corresponds to the nested inference pattern: running an inference query on a probabilistic model that is itself recursively executing inference queries (Stuhlmüller & Goodman, 2014; Zhang et al., unpublished). In practice, this toolbox has achieved significant success and has been used to quantitatively model a range of complex psychological phenomena in social cognition (Evans et al., 2017).
So have we finally addressed requirement (1)? Certainly, we have made progress in restoring the infiniteness, generativity, and recursiveness of human thought. But programming patterns are still not a satisfactory formalization of theories of mind, for two reasons:
First, as we will show later in this paper, it is easy to write models that apply these programming patterns but whose behavior violates our intuitions—thus, these patterns do not fully capture key aspects of typical theories of mind. Second, and more importantly, we now lose the ability for (2) theoretical change: there is no formal, "enumerable" "programming pattern" structure for us to reason about when thinking about the mind.
Programming Language Models Therefore, in this paper, we propose a new perspective that blends the advantages of Bayesian network/causal grammar models and programming pattern models. In our view, a theory of mind is not like a set of programming patterns, but more like a full programming language. Like Church and WebPPL, it is a probabilistic programming language. But unlike Church or WebPPL, it is not a general-purpose probabilistic programming language, but a domain-specific language (DSL) specifically designed for the domain of reasoning about minds.
In the discipline of programming language theory, there has long been a tradition of designing DSLs to express theories in various domains (Bentley, 1986; Bernstein & Kjolstad, 2016; Iverson, 2007), including scientific fields such as classical mechanics (Sussman & Wisdom, 2015), ray optics (Hanrahan & Lawson, 1990), and color perception (Chen, Chang, & Zhu, 2024).
Unlike relying on programmers to use certain programming patterns, a DSL's syntax directly provides the conceptual vocabulary for setting up problems in the target domain, while the language's semantics determine how these problems are solved. Thus, the "theory" is encoded in the design of the language.
Importantly, the syntax and semantics of any programming language can be formalized mathematically and even mechanized as members of a collection of possible syntaxes and semantics (schemas) (Felleisen, Findler, & Flatt, 2009; Pierce, 2002). Therefore, theoretical change can be formally modeled as a learning process over the space of possible programming languages.
What kind of DSL are we talking about here?
So far, our discussion of DSLs has been quite abstract. Before continuing, let's briefly introduce a concrete example of a domain-specific language (DSL) specifically for "reasoning about reasoning": a real-world language called "memo" (short for mental models) (Chandra, Chen, Tenenbaum, & Ragan-Kelley, 2025). memo was developed as a practical tool for building efficient computational models of social cognition.
In our discussion, we will use memo (and some hypothetical variants later) to illustrate the problem, just as Goodman et al. (2015) used the real-world language Church as an example when discussing the "probabilistic language of thought hypothesis."
Let's quickly look at memo. In memo, the core of the model is the process by which an agent makes choices. For example, suppose Ali chooses a bar b to drink at after work one day. Since he is a rational agent, he is more likely to choose a bar closer to his workplace. In memo, we can use the chooses expression to model this choice behavior, expressing that Ali chooses a bar b from the set of all bars, Bars, with probability proportional to (wpp, i.e., "with probability proportional to") some decreasing function of the distance from his workplace to b:
Why Use DSLs? How Can "Less" Be "More"?
Statements in memo, such as chooses, observes, and thinks, roughly correspond to sampling, conditioning, and recursive querying in general-purpose probabilistic programming languages (PPLs). So, what exactly do we gain from writing these models in memo? In fact, aren't we sacrificing expressiveness by using a specialized language? A language that might not be suitable for expressing other types of programs?
An important lesson from decades of research in the programming language field is: generality comes at a cost—the burden of power. Making a language more expressive can lead programmers to unintentionally express things they did not intend, which increases the burden on programmers to ensure the correctness of their programs. Therefore, programming language pioneer Hudak (1996) once wrote: the ideal abstraction for problem-solving is a programming language that exactly matches the domain of study—"no more, no less."
Church and WebPPL themselves embody this principle. Theoretically, according to the logic of Turing completeness, any probabilistic computation in a WebPPL program can be manually implemented using a regular (non-probabilistic) programming language—in fact, WebPPL programs are executed by automatically translating them into JavaScript (Goodman & Stuhlmüller, 2014). The added value of WebPPL compared to JavaScript is that it guarantees probabilistic computations are always correct—for example, by ensuring distributions are always properly normalized. WebPPL empowers users to build complex probabilistic models by restricting them to perform only correct probabilistic inference.
Similarly, the value of memo compared to WebPPL lies in its restrictiveness: memo empowers users to build complex models of theories of mind by restricting them to perform only correct reasoning about agents.
Let's illustrate this with an example. Consider how we might implement the Ali-Zoe scenario in WebPPL. Ali choosing a nearby bar (Example 1) can be modeled by applying the "planning-as-inference" pattern (here we use factor, a variant of condition, to model gradient probabilistic choices):
When we run this program, we find that the model incorrectly predicts a 0% probability of them meeting. Oh, of course! Since we are performing inference under the condition zoe_b != ali_b, it's no surprise that the inference results never satisfy the condition of "them meeting."
What went wrong here? The problem is that WebPPL, at the language level, does not support the concept of "agents"—it only supports the concept of "random choices." Zoe's choice of zoe_b is conceptually indistinguishable from Ali's choice of ali_b. Therefore, when we apply the "planning-as-inference" pattern, WebPPL produces an inference as if Zoe could choose not only which bar she goes to but also which bar Ali goes to—that is, as if Zoe's "wishful thinking" granted her "mind control" over Ali. Chandra, Chen et al. (2025) refer to this type of bug as "perpetration confusion": the confusion of which agents have control over which random choices.
Even experienced programmers can easily encounter this bug in practice (Levine, 2018).
This example illustrates that while using programming patterns in general-purpose probabilistic programming languages provides us with "building blocks" to express many computations consistent with theories of mind, these same "building blocks" can also be combined to express computations that violate our theories of mind—for example, computations representing one agent having "mind control" over another. A DSL like memo, on the other hand, solves this problem by providing structures that are exactly and only what is needed for psychological reasoning: for example, by providing syntax and semantics for reasoning about agents at the language level, preventing issues like "mind control" from the outset.
In the next section, we will examine how a DSL like memo embodies a theory of mind through its design.
How Does a DSL Embody a Theory of Mind?
A programming language is ultimately defined by two formal structures: its syntax and its semantics. In this section, we will discuss, through two case studies, how memo's syntax and semantics specifically embody principles within theories of mind.
In doing so, we will demonstrate how variations in memo's syntax and semantics can generate different versions of theories of mind that we might wish to formalize. Research on theories of mind aims not only to characterize the theories held by neurotypical adults but also those held by children, elderly individuals, and non-human primates. Furthermore, it seeks to characterize how these minds theorize about "atypical" minds, such as omniscient beings (e.g., deities) and machines (Brink & Wellman, 2020); simultaneously, it also seeks to understand how individuals decide which theory to attribute to an agent they observe in the world (Burger & Jara-Ettinger, 2020; Gray, Young, & Waytz, 2012). Various variants of DSLs like memo can serve as formal expressive tools for these theories.
A Case Study on Syntax: False Beliefs
In memo, the thinks statement is used to represent an agent's beliefs. We have seen how thinks can be used to express an agent's true beliefs and uncertainties. But this structure can also be used to express an agent's false beliefs. For example, to model "Zoe thinks Ali will choose a cheap bar," when in reality Ali chose a bar near his workplace, we could write:
Of course, the full generality of this expression is often unnecessary in practice. In most cases, memo programmers only need to express that an agent correctly knows another agent's decision. Suppose Zoe knew Ali's whereabouts in advance, perhaps because he posted an update on social media. In principle, we could model this as: Zoe thinks Ali chose a bar (following an arbitrary distribution), and then she correctly observed where he actually went:
Of course, the full generality of this expression is often unnecessary in practice. In most cases, memo programmers only need to express that an agent correctly knows another agent's decision. Suppose Zoe knew Ali's whereabouts in advance, perhaps because he posted an update on social media. In principle, we could model this as: Zoe thinks Ali chose a bar (following an arbitrary distribution), and then she correctly observed where he actually went:
However, such a writing style is not satisfactory: it is verbose and requires choosing a "dummy" prior distribution in thinks, which is never actually used for computation.
Because such situations are very common in daily modeling, memo provides an efficient, lightweight shorthand to solve these problems: the knows statement.
The knows statement in memo is consistent with the concept of "knowledge before belief" proposed by Phillips et al. (2021): in many ways, knows is more fundamental than thinks / observes. Its syntax is simpler than the compound, recursive structure of thinks (see Figure 2), is used more frequently than thinks or observes in practice, and requires fewer lines of code to implement in the memo compiler.
Suppose we create a new "simplified" version of memo, removing the complex thinks and observes statements and retaining only the simpler knows statement. What would we lose? This programming language, which we might call "memo-junior," could still be used to model various scenarios, such as Examples 1 and 2 in the "Quick Start" section above. However, "memo-junior" would not be able to express situations where an agent has their own beliefs that differ from the true state of the world. In this way, we can begin to formalize a candidate hypothesis about the theory of mind in three-year-old children using memo-junior.
A Case Study on Semantics: Omniscience
Consider the following attempt to implement Example 3:
This model would raise an error in memo—the problem is that Zoe, in her mental model, does not know Ali's choice b. For Zoe to choose a bar based on some function of Ali's b makes no sense; in fact, she should not even be able to compute such a function! If Zoe could choose a bar in this way, it would mean she has some "mind-reading" ability over Ali. Therefore, memo prohibits such statements.
But here, what does "prohibits" formally mean? This restrictive rule can be found in memo's formal semantics (Chandra, Amin, & Zhang, 2025), serving as a precondition for deriving whether a chooses statement is well-typed (see Figure 3): if an agent a uses an expression e to define the probability distribution for their decision, then a must know the value of e.
In practice, this precondition is checked in the memo compiler, and a check is performed every time a chooses statement is encountered. If the check fails, the system prompts the programmer to correct it as we did in Example 3: explicitly write out Zoe's mental model of Ali's bar choice, using knows or thinks statements as needed.
In this way, the preconditions and postconditions in memo's semantics capture fundamental principles about "agency," such as the "no mind-reading" constraint. The designers of memo explicitly enumerated the principles they wanted the semantics to embody and explained in detail how memo's semantics concretize these principles (Chandra, Chen et al., 2025, see Section 3.1). But these principles were chosen by memo's designers based on their intuitions—they can also be modified to accommodate other different intuitions.
For example, if we relax the precondition discussed above, we can begin to formalize a theory of mind that includes the possibility of "omniscient agents"—a characteristic feature of psychological theories often exhibited by children when describing their parents (Barrett, Richert, & Driesenga, 2001).
How Do Theories—or DSLs—Change Over Time?
Gopnik and Meltzoff (1997) analogized the dynamics of theoretical change to the change of scientific theories (Kuhn, 1962; Lakatos, 1970). But how do we formally model this process? The perspective of viewing theories of mind as DSLs suggests that we can see theory building as programming language design. In this section, we will discuss how this metaphor supports three perspectives on theoretical change.
Rational Perspective
Returning to Goodman et al.'s (2006) view that theoretical change is hierarchical inference over the space of Bayesian networks, we can now say that theoretical change is driven by hierarchical inference over the syntax and semantics of possible programming languages. We develop our theories in the direction of "usefulness"—that is, stronger expressive efficiency and predictive power. After all, real-world programming languages often develop in this way: not from "ivory tower" innovations, but from practical utility demonstrated in solving real problems (Meijer, 2007; Meyerovich & Rabkin, 2012; Stroustrup, 2020).
How can we formalize this idea? Ellis et al. (2021) described a system that "grows" DSLs by identifying useful patterns and formalizing them into reusable abstractions. If we view theories of mind as DSLs, we can imagine using such a system to formalize the growth process of theories of mind in the same way.
Cultural Perspective
A large body of evidence suggests that learning natural language for theories of mind helps children acquire the ability to express false beliefs (De Villiers & de Villiers, 2014; De Villiers & Pyers, 2002; Morgan & Kegl, 2006; Pyers, 2020; Pyers & Senghas, 2009; Schick, De Villiers, De Villiers, & Hoffmeister, 2007). For example, Pyers and Senghas (2009) wrote: "The sentence 'Mary thought she saw a ghost' has a main clause that is true ('Mary thought'), and a subordinate clause that is false ('she saw a ghost'). Such syntax may provide learners with the logical tools necessary to understand others' false beliefs."
The DSL perspective provides a way to formalize this hypothesis. As we discussed earlier, we can model the theory of mind in three-year-old children as a "memo-junior" lacking complex thinks expressions. When children are exposed to the word "thinks" in the context of others' beliefs, this may provide them with the recursive syntactic structure unique to thinks, thereby helping them eventually invent the semantics of thinks and thus acquire the ability to express false beliefs.
Pragmatic Perspective
Theoretical change is a messy process. Gopnik and Meltzoff (1997) described a transitional period where a new theory is carefully considered, but the old one has not yet been fully abandoned. Indeed, Amsterlaw and Wellman (2006) found that a child might show signs of using two to three different theories of mind across different tasks on a single day.
Interestingly, programming language design follows the same pattern: the development of programming languages itself is a messy process (Steele Jr, 1998). Take the Python programming language as an example: in the early 2000s, Python users increasingly realized that the abstractions for handling text ("strings") had fundamental flaws—the language failed to distinguish between sequences of characters and sequences of bytes representing those characters. This made processing text input with non-Latin alphabets difficult, a serious problem for an increasingly globalized community of programmers.
Therefore, the language designers completely redesigned these abstractions in the next major version, Python 3, which was released in 2008 (van Rossum, 2008). We can view this new version as a "paradigm shift."
However—just like paradigm shifts in science—this transition was not immediate (Coghlan, 2012). The old version, Python 2, remained widely used for the next twelve years and was only officially deprecated in 2020. During this period, many programmers were reluctant to switch to Python 3 for various reasons: some did not want to rewrite existing Python 2 code into Python 3, while others relied on libraries that only existed in Python 2.
Thus, for many years, the choice between Python 2 and Python 3 became a practical issue: programmers starting new projects or those who truly needed Python 3 features typically chose the new version, while programmers maintaining large existing codebases continued to use Python 2 as much as possible. Many programmers even carefully wrote code compatible with both versions (Malloy & Power, 2019).
We can expect that the process by which children adopt new theories of mind will exhibit similar dynamics. Although a new theory might be more compelling, retaining an old, tested theory might have practical value—perhaps to continue utilizing models and computational methods that are only compatible with the old theory. Therefore, a child might exhibit behavioral patterns that, to an observer, do not seem entirely consistent with any single specific theory of mind.
Discussion and Future Work
In this paper, we propose that it is valuable to view theories of mind as a domain-specific programming language: a language that can be used to set up and reason about problems involving other minds. Taking the memo probabilistic programming language as an example, we discussed how a programming language encodes a theory of mind, how different variants of a programming language encode different theories of mind, and how the "programming language" metaphor helps us understand the development process of theories of mind.
Finally, we reflect on some further questions that PL (programming language) theory might provide us with tools to answer:
Combining Intuitive Theories
To solve real-world problems, our theories of mind must be able to interact with intuitive theories from other domains, such as our intuitive theories of the physical world (Liu, Outa, & Akbiyik, 2024). How should we combine these intuitive theories to perform computations that span multiple knowledge systems?
If we view intuitive theories as DSLs (domain-specific languages), then we can begin to think about this problem from the perspective of "DSL composition." Indeed, the PL community has long studied how to design DSLs so that they can interact modularly with each other (see Felleisen et al., 2018).
DSLs with well-designed "application programming interfaces" (APIs) allow programmers to write multilingual programs, combining the strengths of multiple DSLs. For example, to create a website, a programmer might use HTML to define page structure, CSS to define layout styles, and JavaScript to define animations. HTML, CSS, and JavaScript are three very different languages, each specialized for a different domain of web design (structure, style, and scripting). However, because all three DSLs are equipped with well-designed APIs, programmers can easily write interactive programs between them.
We can imagine that DSLs for psychological reasoning and DSLs for physical reasoning could also be equipped with appropriate APIs, allowing them to be combined for reasoning about everyday socio-physical situations (see example illustration).
Comparing Intuitive Theories
Intuitively, adult humans possess a "richer" theory of mind than three-year-old children or non-human primates. But how do we formalize statements about how "rich" or "expressive" a theory of mind is? Can we quantify and measure the richness or expressiveness of a theory?
If we view theories of mind as DSLs, then we can consider applying formal definitions of "expressivity" from the programming language domain, such as Felleisen's (1991) definition, to formalize the expressive power of the theory of mind represented by that DSL.
By using this definition as a standard for measuring the expressive power of a theory of mind, we can test a hypothesis: that theories of mind tend towards greater expressivity during their evolution and development.
Theorems of Mind
People are not only capable of reasoning about specific situations but also about general propositions concerning others. For example, we might agree that in any situation, if an agent forgets the value of variable x, then they must have previously had a belief about x.
Where do these general intuitions come from? If we view theories of mind as DSLs, then we can apply general theorem statement and proof techniques from programming languages to formalize and study these "theorems of mind."
For example, Chandra, Amin, and Zhang (2025) demonstrated how to formalize a property: in any memo model, if an agent knows the value of variable x, then they must be certain about x (i.e., from their perspective, H(x) = 0). However, the converse is not true, as an agent might be certain about a false belief.
Thus, we can derive a "theorem of mind" from first principles: knowledge implies certainty, but certainty does not imply knowledge.
memo: True AGI's Recursive Rational Probabilistic Programming Language: memo's Implementation, Embodied Games
The Frame Problem
Our complete theory of other minds is very extensive; it includes not only beliefs and desires but also emotions, memory, attention, perception, and many other aspects. When faced with a specific situation, how do we solve the "frame problem" (McCarthy & Hayes, 1981)—that is, how to efficiently invoke only the relevant parts of the theory?
In this regard, the concept of "program slicing" in PL (program slicing, Weiser, 1984) might be valuable: a "slice" of a program (or programming language) refers to the relevant subset retained after removing irrelevant parts.
We can imagine that a rich and general theory of mind could be "sliced" into a customized, situation-specific theory of mind, removing unnecessary syntax and semantics to improve efficiency.
Original Link: https://osf.io/qhc6y/download