10.6 Conclusion 183 Correspondence Principle tells us something about what this "tuning" should involve - namely, making a system possessing mind-state sequences that correspond meaningfully to world-state sequences. CogPrime's overall design and particular cognitive processes are reasonably well interpreted as an attempt to achieve this for everyday human goals and environments. One way of extending these theoretical ideas into a more rigorous theory is explored in Ap- pendix ??. The key ideas involved there are: modeling multiple memory types as mathematical categories (with functors mapping between them), modeling memory items as probability dis- tributions, and measuring distance between memory items using two metrics, one based on algorithmic information theory and one on classical information geometry. Building on these ideas, core hypotheses are then presented: • a syntax-semantics correlation principle, stating that in a successful AGI system, these two metrics should be roughly correlated • a cognitive geometrodynamics principle, stating that on the whole intelligent minds tend to follow geodesics (shortest paths) in mindspace, according to various appropriately defined metrics (e.g. the metric measuring the distance between two entities in terms of the length and/or runtime of the shortest programs computing one from the other). • a cognitive synergy principle, stating that shorter paths may be found through the com- posite mindspace formed by considering multiple memory types together, than by following the geodesics in the mindspaces corresponding to individual memory, types. The material is relegated to an appendix because it is so speculative, and it's not yet clear whether it will really be useful in advancing or interpreting CogPrime or other AGI systems (unlike the material from the present chapter, which has at least been aseful in interpreting and tweaking the CogPrime design, even though it can't be claimed that CogPrime was derived directly from these theoretical ideas). However, this sort of speculative exploration is, in our view, exactly the sort of thing that's needed as a first phase in transitioning the ideas of the present chapter into a more powerful and directly actionable theory. EFTA00623959

EFTA00623960

Section III Cognitive and Ethical Development EFTA00623961

EFTA00623962

Chapter 11 Stages of Cognitive Development Co-authored with Stephan Vladimir Bugaj 11.1 Introduction Creating AGI, we have said, Ls not only about having the right structural and dynamical possibilities implemented in the initial version of one's system - but also about the environment and embodiment that one's system Ls associated with, and the match between the system's internals and these externals. Another key aspect is the long-term time-course of the system's evolution over time, both in its internals and its external interaction - i.e., what is known as development Development is a critical topic in our approach to AGI because we believe that much of what constitutes human-level, human-like intelligence emerges in an intelligent system due to its engagement with its environment and its environment-coupled self-organization. So, it's not to be expected that the initial version of an AGI system is going to display impressive feats of intelligence, even if the engineering is totally done right. A good analogy, is the apparent unintelligence of a human baby. Yes, scientists have discovered that human babies are capable of interesting and significant intelligence - but one has to hunt to find it ... at first observation, babies are rather idiotic and simple-minded creatures: much less intelligent-appearing than lizards or fish, maybe even less than cockroaches.... If the goal of an AGI project is to create an AGI system that can progressively develop advanced intelligence through learning in an environment richly populated with other agents and various inanimate stimuli and interactive entities — then an understanding of the nature of cognitive development becomes extremely important to that project. Unfortunately, contemporary cognitive science contains essentially no theory of "abstract developmental psychology" which can conveniently be applied to understand developing Als. There is of course an extensive science of human developmental psychology, and so it is a natural research program to take the chief ideas from the former and inasmuch as possible port them to the AGI domain. This is not an entirely simple matter both because of the differences between humans and Als and because of the unsettled nature of contemporary developmental psychology theory. But it's a job that mast (and will) be done, and the ideas in this chapter may contribute toward this effort. We will begin here with Piaget's well-known theory of human cognitive development, present- ing it in a general systems theory context, then introducing some modifications and extensions and discussing some other relevant work. 187 EFTA00623963

188 11 Stages of Cognitive Development 11.2 Piagetan Stages in the Context of a General Systems Theory of Development Our review of AGI architectures in Chapter 4 focused heavily on the concept of symbolism, and the different ways in which different classes of cognitive architecture handle symbol rep- resentation and manipulation. We also feel that symbolism is critical to the notion of AGI development - and even more broadly, to the systems theory of development in general. As a broad conceptual perspective on development. we suggest that one may view the de- velopment of a complex information processing system. embedded in an environment, in terms of the stages: • automatic: the system interacts with the environment by "instinct", according to its innate programming • adaptive: the system internally adapts to the environment, then interacting with the en- vironment in a more appropriate way • symbolic: the system creates internal symbolic representations of itself and the environ- ment, which in the case of a complex, appropriately structured environment, allows it to interact with the environment more intelligently • reflexive: the system creates internal symbolic representations of its own internal symbolic representations, thus achieving an even higher degree of intelligence Sketched so broadly, these are not precisely defined categories but rather heuristic, intuitive categories. Formalizing them would be possible but would lead us too far astray here. One can interpret these stages in a variety of different contexts. Here our focus is the cognitive development of humans and human-like AGI systems, but in Table 11.1 we present them in a slightly more general context, using two examples: the Piagetan example of the human (or humanlike) mind as it develops from infancy to maturity; and also the example of the "origin of life" and the development of life from proto-life up into its modern form. In any event, we allude to this more general perspective on development here mainly to indicate our view that the Piagetan perspective is not something ad hoc and arbitrary, but rather can plausibly be seen as a specific manifestation of more fundamental principles of complex systems development. 11.3 Piaget's Theory of Cognitive Development The ghost of Jean Piaget hangs over modern developmental psychology in a yet unresolved way. Piaget's theories provide a cogent overarching perspective on human cognitive develop- ment, coordinating broad theoretical ideas and diverse experimental results into a unified whole [Pia551. Modern experimental work has shown Piaget's ideas to be often oversimplified and in- correct. However, what has replaced the Piagetan understanding is not an alternative unified and coherent theory, but a variety of microtheories addressing particular aspects of cognitive development. For this reason a number of contemporary theorists taking a computer science ISlin03] or dynamical systems IWit07] approach to developmental psychology have chosen to adopt the Piagetan framework in spite of its demonstrated shortcomings, both because of its conceptual strengths and for lack of a coherent, more rigorously grounded alternative. Our own position is that the Piagetan view of development has some fundamental truth to it, which is reflected via how nicely it fits with a broader view of development in complex systems. EFTA00623964

11.3 Piaget's Theory of Cognitive Development IS!) Stage General Description Cognitive Develop- ment Origin of Life Automatic System-environment information exchange controlled mainly by innate system stoic - or tures or environment Piagetan infantile stage Self-organizing prow- life system, e.g. Oparin lop.C,2] water droplet. Cairns-Smith irS91 I clay-based protonic Adaptive System-environment info exchange heavily guided by adaptively internally-created system structures Piagetan "concrete op- erational" stage: sys- tematic internal world- model guides world- exploration Simple autopoietic sys- tem, e.g. Oparin wa- ter droplet w/ basic metabolism Symbolic Internal symbolic rep- resentation of infonna- tion exchange process Piagetan formal stage: explicit logical/expert- mental learning about how to cognize in var- ions contexts Genetic code: inter- nal entities that "stand for" aspects of organ- ism and environment, thus enabling complex epigenesis Reflexive Thoroughgoing self- modification based on this symbolic representation Piagetan post-formal stage: purposive self- modification of basic mental processes Genes+memes: genetic code-patterns guide their own modification via influencing culture Table 11.1: General Systems Theory of Development: Parallels Between Development of Mind and Origin of Life Indeed, Piaget viewed developmental stages as emerging from general "algebraic" principles rather than as being artifacts of the particulars of human psychology. But, Piaget's stages are probably best viewed as a general interpretive framework rat her than a precise scientific theory. Our suspicion is that once the empirical science of developmental psychology has progressed further, it will become clearer how to fit the various data into a broad Piaget-like framework, perhaps differing in many details from what Piaget described in his works. Piaget conceived of child development in four stages, each roughly identified with an age group, and corresponding closely to the system-theoretic stages mentioned above: • infantile, corresponding to the automatic stage mentioned above - Example: Grasping blocks, piling blocks on top of each other, copying words that are heard • preoperational and concrete operational, corresponding to the adaptive stage men- tioned above - Example: Building complex blocks structures, from imagination and from imitating objects and pictures and based on verbal instructions; verbally describing what has been constructed • formal, corresponding to the symbolic stage mentioned above - Example: Writing detailed instructions in words and diagrams, explaining how to con- struct particular structures out of blocks; figuring out general rules describing which sorts of blocks structures are likely to be most stable EFTA00623965

190 11 Stages of Cognitive Development • the reflexive stage mentioned above corresponds to what some post-Piagetan theorists have called the post-formal stage — Example: Using abstract lessons learned from building structures out of blocks to guide the construction of new ways to think and understand - "Zen and the art of blocks building" (by analogy to Zen and the Art of Motorcycle Maintenance [Pir8-1]). Piagetan Stages of Development uil self modification Reflexive Formal Concrete infantile Deep understanding and control of self structures and dynamics Abstract reasoning and hypothesizing. Objective detachment from phenomenal self Rich variety of learned mental representations and operations thereon. Emergence of phenomenal self. Making sense and achieving simple goals in sensorimotor reality. No self yet. Fig. 11.1: Piagetan Stages of Cognitive Development More explicitly, Piaget defined his stages in psychological terms roughly as follows: • Infantile: In this stage a mind develops basic world-exploration driven by instinctive ac- tions. Reward-driven reinforcement of actions learned by imitation, simple associations be- tween words and objects, actions and images, and the basic notions of time, space, and causality are developed. The most simple, practical ideas and strategies for action are learned. • Preoperational: At this stage we see the formation of mental representations, mostly poorly organized and tm-abstracted, building mainly on intuitive rather than logical think- ing. Word-object and image-object assnriations become systematic rather than occasional. Simple syntax is mastered, including an understanding of subject-argument relationships. One of the crucial learning achievements here is "object permanence" - infants learn that objects persist even when not observed. However, a number of cognitive failings persist with respect to reasoning about logical operations, and abstracting the effects of intuitive actions to an abstract theory of operations. • Concrete: More abstract logical thought is applied to the physical world at this stage. Among the feats achieved here are: reversibility - the ability to undo steps already done; conservation - understanding that properties can persist in spite of appearances; theory of mind - an understanding of the distinction between what I know and what others know (If EFTA00623966

11.3 Piaget's Theory of Cognitive Development 191 I cover my eyes, can you still see me?). Complex concrete operations, such as putting items in height order, are easily achievable. Classification becomes more sophisticated, yet the mind still cannot master purely logical operations based on abstract logical representations of the observational world. • Formal: Abstract deductive reasoning, the process of forming, then testing hypotheses, and systematically reevaluating and refining solutions, develops at this stage, as does the ability to reason about purely abstract concepts without reference to concrete physical objects. This is adult human-level intelligence. Note that the capability for formal operations is intrinsic in the PLN component of CogPrime, but in-principle capability is not the same as pragmatic, grounded, controllable capability. Very early on, Vygotsky Nyg861 disagreed with Piaget's explanation of his stages as inherent and developed by the child's own activities, and Piaget's prescription of good parenting as not interfering with a child's unfettered exploration of the world. Some modern theorists have critiqued Piaget's stages as being insufficiently socially grounded, and these criticisms trace back to Vygotsky's focus on the social foundations of intelligence, on the fact that children function in a world surrounded by adults who provide a cultural context, offering ongoing assistance, critique, and ultimately validation of the child's developmental activities. Vygotsky also was an early critic of the idea that cognitive development is continuous, and continues beyond Piaget's formal stage. Gagne IIIIIW92] also believes in continuity, and that learning of prerequisite skills made the learning of subsequent skills easier and faster without regard to Piagetan stage formalisms. Subsequent researchers have argued that Pi- eget has merely constructed ad hoc descriptions of the sequential development of behaviour IGib78, I3ro81, ('P01. We agree that learning is a continuous process, and our notion of stages is more statistically constructed than rigidly quantized. Critique of Piaget's notion of transitional "half stages" is also relevant to a more compre- hensive hierarchical view of development. Some have proposed that Piaget's half stages are actually stages 113ro8-11. As Commons and Pekker ICP051 point out: "the definition of a stage that was being used by Piaget was based on analyzing behaviors and attempting to impose different structures on them. There is no underlying logical or mathematical definition to help in this process ..." Their Hierarchical Complexity development model uses task achievement rather than ad hoc stage definition as the basis for constructing relationships between phases of developmental ability - an approach which we find useful, though our approach is different in that we define stages in terms of specific underlying cognitive mechanisms. Another critique of Piaget is that one individual's performance is often at different ability stages depending on the specific task (for example IGE86D. Piaget responded to early critiques along these lines by calling the phenomenon "horizontal decalage," but neither he nor his suc- cessors Ins80, Cas851 have modified his theory to explain (rather than merely describe) it. Similarly to Thelen and Smith irsa-11, we observe that the abilities encapsulated in the defini- tion of a certain stage emerge gradually during the previous stage - so that the onset of a given stage represents the mastery of a cognitive skill that was previously present only in certain contexts. Piaget also had difficulty accepting the idea of a preheuristic stage, early in the infantile period, in which simple trial-and-error learning occurs without significant heuristic guidance 113ie81, a stage which we suspect exists and allows formulation of heuristics by aggregation of learning from preheuristic pattern mining. Coupled with his belief that a mind's innate abilities at birth are extremely limited, there is a troublingly unexplained transition from inability to ability in his model. EFTA00623967

192 11 Stages of Cognitive Development Finally, another limiting aspect of Piaget's model is that it did not recognize any stages beyond formal operations, and included no provisions for exploring this possibility. A number of researchers 113ic8S, Ar175, ell K82, Rte Nlar011 have described one or more postformal stages. Commons and colleagues have also proposed a task-based model which provides a framework for explaining stage discrepancies across tasks and for generating new stages based on classification of observed logical behaviors. [IC Ic.90] promotes a statistical conception of stage, which provides a good bridge between task-based and stage-based models of development, as statistical modeling allows for stages to be roughly defined and analyzed based on collections of task behaviors. len li".82] postulates the existence of a postformal stage by observing elevated levels of abstrac- tion which, they argue, are not manifested in formal thought. 'TS+98j observes a postformal stage when subjects become capable of analyzing and coordinating complex logical systems with each other, creating metatheoretical supers. ystems. In our model, with the reflexive stage of development, we expand this definition of metasystemic thinking to include the ability to consciously refine one's own mental states and formalisms of thinking. Such self-reflexive re- finement is necessary for learning which would allow a mind to analytically devise entirely new structures and methodologies for both formal and postformal thinking. In spite of these various critiques and limitations, however, we have found Piaget's ideas very useful, and in Section 11.4 we will explore ways of defining them rigorously in the specific context of CogPrime's declarative knowledge store and probabilistic logic engine. 11.3.1 Perry's Stages Also relevant is William Perry's 'Perin, Per811 theory of the stages ("positions" in his terminol- ogy) of intellectual and ethical development, which constitutes a model of iterative refinement of approach in the developmental process of coming to intellectual and ethical maturity. These stages, depicted in Table 11.2 form an analytical tool for discerning the modality of belief of an intelligence by describing common cognitive approaches to handling the complexities of real world ethical considerations. 11.3.2 Keeping Continuity in Mind Continuity of mental stages, and the fact that a mind may appear to be in multiple stages of development simultaneously (depending upon the tasks being tested), are crucial to our theoretical formulations and we will touch upon them again here. Piaget attempted to address continuity with the creation of transitional "half stages". We prefer to observe that each stage feeds into the other and the end of one stage and the beginning of the next blend together. The distinction between formal and post-formal, for example, seems to "merely" be the application of formal thought to oneself. However, the distinction between concrete and formal is "merely" the buildup to higher levels of complexity of the classification, task decomposition, and abstraction capabilities of the concrete stage. The stages represent general trends in ability on a continuous curve of development, not discrete states of mind which are jumped-into quantum style after enough "knowledge energy" builds-up to cause the transition. EFTA00623968

11.4 Piaget's Stages in the Context of Uncertain Inference 193 Stage Substages Dualism / Received Knowledge [Infantile] Basic duality ("All problems are solvable. I must learn the correct solutions?) Full dualism ("There are different, contradictory solutions to many problems. I must learn the correct solutions, and ignore the incorrect ones") Multiplicity [Concrete] Early multiplicity ("Some solutions are known, others aren't. I must learn how to find correct solutions.") Late Multiplicity: cognitive dissonance regarding truth. ("Some problems are unsolvable, some are a matter of personal taste, therefore I must declare my own intellectual path.") Relativism / Procedu- ral Knowledge [Formal) Contextual Relativism ("I must learn to evaluate solutions within a context, and relative to supporting observation.") Pre-Commitment ("I must evaluate solutions, then commit to a choice of solution.") Commitment / Con- structed Knowledge [Formal / Reflexive] Commitment ("I have chosen a solution?) Challenges to Commitment ("I have seen unexpected implica- tions of my commitment, and the responsibility I must take.") Poet-Commitment ("I must have an ongoing, nuanced rela- tionship to the subject in which I evaluate each situation on a case-by-case basis with respects to its particulars rather than an ad-hoc application of unchallenged ideology.") Table 11.2: Perry's Developmental Stages 'with corresponding Piagetan Stages in brackets] Observationally, this appears to be the case in humans. People learn things gradually, and show a continuous development in ability, not a quick jump from ignorance to mastery. We believe that this gradual development of ability is the signature of genuine learning, and that prescriptively an AGI system must be designed in order to have continuous and asymmetrical development across a variety of tasks in order to be considered a genuine learning system. While quantum leaps in ability may be passible in an AGI system which can just "graft" new parts of brain onto itself (or an augmented human which may someday be able to do the same using implants), such acquisition of knowledge is not really learning. Grafting on knowledge does not build the cognitive pathways needed in order to actually learn. If this is the only mechanism available to an AGI system to acquire new knowledge, then it is not really a learning system. 11.4 Piaget's Stages in the Context of Uncertain Inference Piaget's developmental stages are very general, referring to overall types of learning, not specific mechanisms or methods. This focus was natural since the context of his work was human de- velopmental psychology, and neuroscience has not yet progressed to the point of understanding the neural mechanisms underlying any sort of inference (and certainly was nowhere near to doing so in Piaget's time!). But if one is studying developmental psychology in an AGI context where one knows something about the internal mechanisms of the AGI system under consid- eration, then one can work with a more specific model of learning. Our focus here is on AGI systems whose operations contain uncertain inference as a central component. Obviously the main focus is CogPrime, but the essential ideas apply to any other uncertain inference centric AGI architecture as well. EFTA00623969

191 11 Stages of Cognitive Development Piaget Meets Uncertain Inference A Full self modification Reflexive Formal Capable of self-modification of internal structures Able to carry out arbitary complex inferences (constrained only by computational resources) via including inference control as an expl.cit subject of abstract learning Able to carry out more complex chains of reasoning via using inference control schemata that adapt behavior based on experience (reasoning about a given case in a manner similar to PeiOr COWS) Able to recognise patterns In and make Inferences about the world, but only using simplistic nard•wired (not experientially learned) inference control schema. Fig. 11.2: Piagetan Stages of Development, as Manifested in the Context of Uncertain Inference An uncertain inference system, as we consider it here, consists of four components, which work together in a feedback-control loop 11.3 1. a content representation scheme 2. an uncertainty representation scheme 3. a set of inference rules 4. a set of inference control schemata Fig. 11.3: A Simplified Look a Feedback-Control in Uncertain Inference EFTA00623970

11.4 Piaget's Stages in the Context of Uncertain Inference 195 Broadly speaking, examples of content representation schemes are predicate logic and term logic IES001. Examples of uncertainty representation schemes are fuzzy logic Vad78], imprecise probability theory IGoo86, FC86j, Dempster-Shafer theory ISlia76, Ky1071, Bayesian probability theory IK,O971, NARS tWan951, and the Atom representation used in CogPrime, briefly alluded to in Chapter 6 above and described in depth in later chapters. Many, but not all, approaches to uncertain inference involve only a limited, weak set of in- ference rules (e.g. not dealing with complex quantified expressions). CogPrime's PLN inference framework, like NARS and some other uncertain inference frameworks, contains uncertain in- ference rules that apply to logical constructs of arbitrary complexity. Only a system capable of dealing with constructs of arbitrary (or at least very high) complexity will have any potential of leading to human-level, human-like intelligence. The subtlest part of uncertain inference is inference control: the choice of which inferences to do, in what order. Inference control is the primary area in which human inference currently exceeds automated inference. Humans are not very efficient or accurate at carrying out inference rules, with or without uncertainty, but we are very good at determining which inferences to do and in what order, in any given context. The lath of effective, context-sensitive inference control heuristics is why the general ability of current automated theorem provers is considerably weaker than that of a mediocre university mathematics major tNlac95J. We now review the Piagetan developmental stages from the perspective of AGI systems heavily based on uncertain inference. 11.4.1 The Infantile Stage In this initial stage, the mind is able to recognize patterns in and conduct inferences about the world, but only using simplistic hard-wired (not experientially learned) inference control schema, along with pro-heuristic pattern mining of experiential data. In the infantile stage an entity is able to recognize patterns in and conduct inferences about its sensory surround context (i.e., it's "world"), but only using simplistic, hard-wired (not expe- rientially learned) inference control schemata. Preheuristic pattern-mining of experiential data is performed in order to build future heuristics about analysis of and interaction with the world. s tasks include: 1. Exploratory behavior in which useful and useless / dangerous behavior is differentiated by both trial and error observation, and by parental guidance. 2. Development of "habits" — i.e. Repeating tasks which were successful once to determine if they always / usually are so. 3. Simple goal-oriented behavior such as "find out what cat hair tastes like" in which one must plan and take several sequentially dependent steps in order to achieve the goal. Inference control is very simple during the infantile stage (Figure 11.4), as it is the stage during which both the most basic knowledge of the world is acquired, and the most basic of cognition and inference control structures are developed as the building block upon which will be built the next stages of both knowledge and inference control. Another example of a cognitive task at the borderline between infantile and concrete cog- nition is learning object permanence, a problem discussed in the context of CogPrime's prede- cessor "Novamente Cognition Engine" system in IGPSL031. Another example is the learning of EFTA00623971

11 Stages of Cognitive Development Stage 1: Infantile Inference Rule J Inference Control Strategyi New Knowledge Fig. 11.4: Uncertain Inference in the Infantile Stage word-object associations: e.g. learning that when the word "ball" is uttered in various contexts ("Get me the ball," 'That's a nice ball," etc.) it generally refers to a certain type of object. The key point regarding these "infantile" inference problems, from the CogPrime perspective, is that assuming one provides the inference system with an appropriate set of perceptual and motor ConceptNodes and Schemallodes, the chains of inference involved are short. They involve about a dozen inferences, and this means that the search tree of possible PLN inference rules walked by the PLN backward-chainer is relatively shallow. Sophisticated inference control is not required: standard AI heuristics are sufficient. In short, textbook narrow-AI reasoning methods, utilized with appropriate uncertainty-savvy truth value formulas and coupled with appropriate representations of perceptual and motor inputs and outputs, correspond roughly to Piaget's infantile stage of cognition. The simplistic approach of these narrow-AI methods may be viewed as a method of creating building blocks for subsequent, more sophisticated heuristics. In our theory Piaget's preoperational phase appears as transitional between the infantile and concrete operational phases. 11.4.2 The Concrete Stage At this stage, the mind is able to carry out more complex chains of reasoning regarding the world, via using inference control schemata that adapt behavior based on experience (reasoning about a given case in a manner similar to prior cases). In the concrete operational stage (Figure 11.5), an entity Ls able to carry out more complex chains of reasoning about the world. Inference control schemata which adapt behavior based on experience, using experientially learned heuristics (including theme learned in the prior stage), are applied to both analysis of and interaction with the sensory surround / world. Concrete Operational stage tasks include: EFTA00623972

11.4 Piaget's Stages in the Context of Uncertain Inference 197 Stage 2:Corocrsta Operational Knowlodge about World Know1•090 about -- how tO control inference Inference Rules_ Inference Control Strategy anoNwledgo Fig. 11.5: Uncertain Inference in the Concrete Operational Stage 1. Conservation tasks, such as conservation of number, 2. Decomposition of complex tasks into easier subtaslcs, allowing increasingly complex tasks to be approached by association with more easily understood (and previously experienced) smaller tasks, 3. Classification and Serialization tasks, in which the mind can cognitively distinguish various disambiguation criteria and group or order objects accordingly. In terms of inference control this is the stage in which actual knowledge about how to control inference itself is first explored. This means an emerging understanding of inference itself as a cognitive task and methods for learning, which will be further developed in the following stages. Also, in this stage a special cognitive task capability is gained: "Theory of Mind," which in cognitive science refers to the ability to understand the fact that not only oneself, but other sentient beings have memories, perceptions. and experiences. This is the ability to conceptually "put oneself in another's shoes" (even if you happen to assume incorrectly about them by doing so). 11.4.2.1 Conservation of Number Conservation of number is an example of a learning problem classically categorized within Piaget's concrete-operational phase, a "conservation laws" problem, discussed in IShu03] in the context of software that solves the problem using (logic-based and neural net) narrow-AI techniques. Conservation laws are very important to cognitive development. Conservation is the idea that a quantity remains the same despite changes in appearance. If you show a child some objects and then spread them out, an infantile mind will focus on the spread, and believe that there are now more objects than before, whereas a concrete-operational mind will understand that the quantity of objects has not changed. Conservation of number seems very simple, but from a developmental perspective it is ac- tually rather difficult. "Solutions" like those given in IS1m031 that use neural networks or cus- EFTA00623973

198 11 Stages of Cognitive Development tomized logical rule-bases to find specialized solutions that solve only this problem fail to fully address the issue, because these solutions don't create knowledge adequate to aid with the solution of related sorts of problems. We hypothesize that this problem is hard enough that for an inference-based AGI system to solve it in a developmentally useful way, its inferences must be guided by meta-inferential lessons learned from prior similar problems. When approaching a number conservation problem, for example, a reasoning system might draw upon past experience with set-size problems (which may be trial-and-error experience). This is not a simple "machine learning" approach whose scope is restricted to the current problem, but rather a heuristically guided approach which (a) aggregates information from prior experience to guide solution formulation for the problem at hand, and (b) acids the present experience to the set of relevant information about quantification problems for future refinement of thinking. Fig. 11.6: Conservation of Number For instance, a very simple context-specific heuristic that a system might learn would be: "When evaluating the truth value of a statement related to the number of objects in a set, it is generally not that useful to explore branches of the backwards-chaining search tree that contain relationships regarding the sizes, masses, or other physical properties of the objects in the set." This heuristic itself may go a long way toward guiding an inference process toward a correct solution to the problem-but it is not something that a mind needs to know "a priori." A concrete-operational stage mind may learn this by data-mining prior instances of inferences involving sizes of sets. Without such experience-based heuristics, the search tree for such a problem will likely be unacceptably large. Even if it is "solvable" without such heuristics, the solutions found may be overly fit to the particular problem and not usefully generalizable. 11.4.2.2 Theory of Mind Consider this experiment: a preoperational child is shown her favorite "Dora the Explore?' DVD box. Asked what show she's about to see, she'll answer "Dora." However, when her parent plays the disc, it's "SpongeBob SquarePants." If you then ask her what show her friend will expect when given the "Dora" DVD box, she will respond "SpongeBob" although she just answered "Dora" for herself. A child lacking a theory, of mind can not reason through what someone else would think given knowledge other than her own current knowledge. Knowledge of self is intrinsically related to the ability to differentiate oneself from others, and this ability may not be fully developed at birth. Several theorists IB(1),I, Fod9-II, based in part on experimental work with autistic children, perceive theory of mind as embodied in an innate module of the mind activated at a certain developmental stage (or not, if damaged). While we consider this possible, we caution against adopting a simplistic view of the "innate vs. acquired" dichotomy: if there is innateness it may take the form of an innate predisposition to certain sorts of learning IEB.1±91. EFTA00623974

11.4 Piaget's Stages in the Context of Uncertain Inference 199 Davidson Pavg4], Dennett !Denzil and others support the common belief that theory of mind is dependent upon linguistic ability. A major challenge to this prevailing philosophical stance came from Premark and Woodruff IPW78I who postulated that prelinguistic primates do indeed exhibit "theory of mind" behavior. While Premack and INoodruff's experiment itself has been challenged, their general result has been bolstered by follow-up work showing similar results such as freq. It seems to us that while theory of mind depends on many of the same inferential capabilities as language learning, it is not intrinsically dependent on the latter. There is a school of thought often called the Theory Theory 1BW88, Car85, Weil/ holding that a child's understanding of mind is best understood in terms of the process of iteratively formulating and refuting a series of naive theories about others. Alternately, Gordon 'Gm 86] postulates that theory, of mind is related to the ability to run cognitive simulations of others' minds using one's own mind as a model. We suggest that these two approaches are actually quite harmonious with one another. In an uncertain AGI context, both theories and simulations are grounded in collections of uncertain implications, which may be assembled in context- appropriate ways to form theoretical conclusions or to drive simulations. Even if there is a special "mind-simulator" dynamic in the human brain that carries out simulations of other minds in a manner fundamentally different from explicit inferential theorizing, the inputs to and the behavior of this simulator may take inferential form, so that the simulator is in essence a way of efficiently and implicitly producing uncertain inferential conclusions front uncertain premises. We have thought through the details by CogPrime system should be able to develop theory of mind via embodied experience, though at time of writing practical learning experiments in this direction have not yet been done. We have not yet explored in detail the possibility of giving CogPrime a special, elaborately engineered "mind-simulator- component, though this would be possible; instead we have initially been pursuing a more purely inferential approach. First, it is very simple for a CogPrime system to learn patterns such as "If I rotated by pi radians, I would see the yellow block." And it's not a big leap for PLN to go front this to the recognition that "You look like me, and you're rotated by pi radians relative to my orientation, therefore you probably see the yellow block." The only nontrivial aspect here Ls the "you look like me" premise. Recognizing "embodied agent" as a category, however, is a problem fairly similar to recog- nizing "block" or "insect" or "daisy" as a category. Since the CogPrime agent can perceive most parts of its own "robot" body-its arms. its legs, etc.-it should be easy for the agent to figure out that physical objects like these look different depending upon its distance from them and its angle of observation. From this it should not be that difficult for the agent to understand that it is naturally grouped together with other embodied agents (like its teacher), not with blocks or bugs. The only other major ingredient needed to enable theory of mind is "reflection"— the ability of the system to explicitly recognize the existence of knowledge in its own mind (note that this term "reflection" is not the same as our proposed "reflexive" stage of cognitive development). This exists automatically in CogPrime, via the built-in vocabulary of elementary procedures supplied for use within Schemallodes (specifically, the atTime and TruthValue operators). Observing that "at time T, the weight of evidence of the link L increased from zero" is basically equivalent to observing that the link L was created at time T. Then, the system may reason, for example, as follows (using a combination of several PLN rules including the above-given deduction rule): EFTA00623975

200 11 Stages of Cognitive Development Implication My eye is facing a block and it is not dark A relationship is created describing the block's color Similarity My body My teacher's body Implication My teacher's eye is facing a block and it is not dark A relationship is created describing the block's color This sort of inference is the essence of Piagetan "theory of mind." Note that in both of these implications the created relationship is represented as a variable rather than a specific relationship. The cognitive leap is that in the latter case the relationship actually exists in the teacher's implicitly hypothesized mind, rather than in CogPrime's mind. No explicit hypothesis or model of the teacher's mind need be created in order to form this implication-the hypothesis is created implicitly via inferential abstraction. Yet, a collection of implications of this nature may be used via an uncertain reasoning system like PLN to create theories and simulations suitable to guide complex inferences about other minds. From the perspective of developmental stages, the key point here is that in a CogPrime context this sort of inference is too complex to be viably carried out via simple inference heuristics. This particular example must be done via forward chaining, since the big leap Ls to actually think of forming the implication that concludes inference. But there are simply too many combinations of relationships involving CogPrime's eye, body, and so forth for the PLN component to viably explore all of them via standard forward-chaining heuristics. Experience- guided heuristics are needed, such as the heuristic that if physical objects A and B are generally physically and functionally similar, and there is a relationship involving some part of A and some physical object R, it may be useful to look for similar relationships involving an analogous part of B and objects similar to R. This kind of heuristic may be learned by experience-and the masterful deployment of such heuristics to guide inference is what we hypothesize to characterize the concrete stage of development. The "concreteness" comes from the fact that inference control is guided by analogies to prior similar situations. 11.4.5 The Formal Stage In the formal stage, as shown in Figure 11.7, an agent should be able to carry out arbitrarily complex inferences (constrained only by computational resources, rather than by fundamental restrictions on logical language or form) via including inference control as an explicit subject of abstract learning. Abstraction and inference about both the sensorimotor surround (world) and about abstract ideals themselves (including the final stages of indirect learning about inference itself) are fully developed. Formal stage evaluation tasks are centered entirely around abstraction and higher-order inference tasks such as: 1. Mathematics and other formalizations. EFTA00623976

11.4 Piaget's Stages in the Context of Uncertain Inference 201 Stage 3:Formal Knowledge eibOuIF World Knowledge about Now to control inlefence 7 Interim. Rules Centro! Strategy ( inieremi. Rules ) Inference Control Strategy 0 New Knowledge New Knowledge about how to Control Infortnee Fig. 11.7: Uncertain Inference in the Formal Stage 2. Scientific experimentation and other rigorous observational testing of abstract formaliza- tions. 3. Social and philosophical modeling, and other advanced applications of empathy and the Theory of Mind. In terms of inference control this stage sees not just perception of new knowledge about inference control itself, but inference controlled reasoning about that knowledge and the creation of abstract formalizations about inference control which are reasoned-upon, tested, and verified or debunked. 11.4.3.1 Systematic Experimentation The Piagetan formal phase is a particularly subtle one from the perspective of uncertain in- ference. In a sense, AGI inference engines already have strong capability for formal reasoning built in. Ironically, however, no existing inference engine is capable of deploying its reasoning rules in a powerfully effective way, and this is because of the lack of inference control heuris- tics adequate for controlling abstract formal reasoning. These heuristics are what arise during Piaget's formal stage, and we propose that in the content of uncertain inference systems, they involve the application of inference itself to the problem of refining inference control. EFTA00623977

202 11 Stages of Cognitive Development A problem commonly used to illustrate the difference between the Piagetan concrete opera- tional and formal stages is that of figuring out the rules for making pendulums swing quickly versus slowly pins'. If you ask a child in the formal stage to solve this problem, she may pro- ceed to do a number of experiments, e.g. build a long string with a light weight, a long string with a heavy weight, a short string with a light weight and a short string with a heavy weight. Through these experiments she may determine that a short string leads to a fast swing, a long string leads to a slow swing, and the weight doesn't matter at all. The role of experiments like this, which test "extreme casm," is to make cognition easier. The formal-stage mind tries to map a concrete situation onto a maximally simple and manipulable set of abstract propositions, and then reason based on these. Doing this, however, requires an automated and instinctive understanding of the reasoning process itself. The above-described experiments are good ones for solving the pendulum problem because they provide data that is very easy to reason about. From the perspective of uncertain inference systems, this is the key characteristic of the formal stage: formal cognition approaches problems in a way explicitly calculated to yield tractable inferences. Note that this is quite different from saying that formal cognition involves abstractions and advanced logic. In an uncertain logic-based AGI system, even infantile cognition may involve these - the difference lies in the level of inference control, which in the infantile stage is simplistic and hard-wired, but in the formal stage is based on an understanding of what sorts of inputs lead to tractable inference in a given context. 11.4.4 The Reflexive Stage In the reflexive stage (Figure 11.8), an intelligent agent is broadly capable of self-modifying its internal structures and dynamics. As an example in the human domain: highly intelligent and self-aware adult humans may carry out reflexive cognition by explicitly reflecting upon their own inference processes and trying to improve them. An example is the intelligent improvement of uncertain-truth-value- manipulation formulas. It is well demonstrated that even educated humans typically make numerous errors in probabilistic reasoning IGGKO21. Most people don't realize it and continue to systematically make these errors throughout their lives. However, a small percentage of individuals make an explicit effort to increase their accuracy in making probabilistic judgments by consciously endeavoring to internalize the rules of probabilistic inference into their automated cognition processes. In the uncertain inference based AGI context, what this means is: In the reflexive stage an entity is able to include inference control itself as an explicit subject of abstract learning (i.e. the ability to reason about one's own tactical and strategic approach to modifying one's own learning and thinking), and modify these inference control strategies based on analysis of experience with various cognitive approaches. Ultimately, the entity can self-modify its internal cognitive structures. Any knowledge or heuristics can be revised, including metatheoretical and metasystemic thought itself. Initially this is done indirectly, but at least in the case of AGI systems it is theoretically possible to also do so directly. This might be considered as a separate stage of Full Self Modification, or else as the end phase of the reflexive stage. In the context of logical reasoning, self modification of inference control itself is the primary task in this stage. In terms of inference control this EFTA00623978

11.4 Piaget's Stages in the Context of Uncertain Inference 203 Sege 4:Mllex,ve (Poit-formal) 40.....IpC .00 PlOm. am. •••••tre Fig. 11.8: The Reflexive Stage •—• re••••• WOOMIIP•e• •••••• .1,44• stage adds an entire new feedback loop for reasoning about inference control itself, as shown in Figure 11.8. As a very concrete example, in later chapters we will see that, while PLN is founded on probability theory, it also contains a variety of heuristic assumptions that inevitably introduce a certain amount of error into its inferences. For example, PLN's probabilistic deduction embodies a heuristic independence assumption. Thus PLN contains an alternate deduction formula called the "concept geometry formula!' that is better in some contexts. based on the assumption that ConceptNodes embody concepts that are roughly spherically-shaped in attribute space. A highly advanced CogPrime system could potentially augment the independence-based and concept- geometry-based deduction formulas with additional formulas of its own derivation, optimized to minimize error in various contexts. This is a simple and straightforward example of reflexive cognition — it illustrates the power accessible to a cognitive system that has formalized and reflected upon its own inference processes, and that pc6sesses at least some capability to modify these. In general, AGI systems can be expected to have much broader and deeper capabilities for self-modification than human beings. Ultimately it may make sense to view the AGI systems we implement as merely "initial conditions" for ongoing self-modification and self-organization. Chapter ?? discusses some of the potential technical details underlying this sort of thorough- going AGI self-modification. EFTA00623979

EFTA00623980

Chapter 12 The Engineering and Development of Ethics Co-authored with Stephan Vladimir Bugaj and Joel Pitt 12.1 Introduction Most commonly, if a work on advanced AI mentions ethics at all, it occurs in a final summary chapter, discussing in broad terms some of the possible implications of the technical ideas pre- sented beforehand. It's no coincidence that the order is reversed here: in the case of CogPrime, AGI-ethics considerations played a major role in the design process ... and thus the chapter on ethics occurs near the beginning rather than the end. In the CogPrime approach, ethics is not a particularly distinct topic, being richly interwoven with cognition and education and other aspects of the AGI project. The ethics of advanced AGI is a complex issue with multiple aspects. Among the many issues there are: 1. Risks posed by the possibility of human beings using AGI systems for evil ends 2. Risks posed by AGI systems created without well-defined ethical systems 3. Risks posed by AGI systems with initially well-defined and sensible ethical systems eventu- ally going rogue - an especially big risk if these systems are more generally intelligent than humans, and possess the capability to modify their own source code 4. the ethics of experimenting on AGI systems when one doesn't understand the nature of their experience 5. AGI rights: in what circumstances does using an AGI as a tool or servant constitute "slavery" In this chapter we will focus mainly (though not exclusively) on the question of how to create an AGI with a rational and beneficial ethical system. After a somewhat wide-ranging discussion, we will conclude with eight general points that we believe should be followed in working toward "Friendly AGI" - most of which have to do, not with the internal design of the AGI, but with the way the AGI is taught and interfaced with the real world. While most of the particulars discussed in this book have nothing to do with ethics, it's important for the reader to understand that AGI-ethics considerations have played a major role in many of our design decisions, underlying much of the technical contents of the book. As the materials in this chapter should make clear, ethicalness is probably not something that one can meaningfully tack onto an AGI system at the end, after developing the rest - it is likely infeasible to architect an intelligent agent and then add on an "ethics module." Rather, ethics is something that has to do with all the different memory, systems and cognitive processes that 205 EFTA00623981

206 12 The Engineering and Development of Ethics constitute an intelligent system - and it's something that involves both cognitive architecture and the exploration a system does and the instruction it receives. It's a very complex matter that is richly intermixed with all the other aspects of intelligence, and here we will treat it as such. 12.2 Review of Current Thinking on the Risks of AGI Before proceeding to outline our own perspective on AGI ethics in the context of CogPrime, we will review the main existing strains of thought on the potential ethical dangers associated with AGI. One science fiction film after another has highlighted these dangers, lodging the issue deep in our cultural awareness; unsurprisingly, much less attention has been paid to serious analysis of the risks in their various dimensions, but there is still a non-trivial literature worth paying attention to. Hypothetically, an AGI with superhuman intelligence and capability could dispense with humanity altogether - i.e. posing an "existential risk" 113os04 In the worst case, an evil but brilliant AGI, perhaps programmed by a human sadist, could consign humanity to unimaginable tortures (i.e. realizing a modern version of the medieval Christian visions of hell). On the other hand, the potential benefits of powerful AGI also go literally beyond human imagination. It seems quite plausible that an AGI with massively superhuman intelligence and positive disposition toward humanity could provide us with truly dramatic benefits, such as a virtual end to material scarcity, disease and aging. Advanced AGI could also help individual humans grow in a variety of directions, including directions leading beyond "legacy humanity," according to their own taste and choice. Eliezer Yudkowsky has introduced the term "Friendly AI", to refer to advanced AGI systems that act with human benefit in mind IYud061. Exactly what this means has not been specified precisely, though informal interpretations abound. Goertzel IG ()Mill has sought to clarify the notion in terms of three core values of Joy, Growth and Freedom. In this view, a Friendly AI would be one that advocates individual and collective human joy and growth, while respecting the autonomy of human choices. Some (for example, Hugo de Canis, IDC051), have argued that Friendly AI is essentially an impossibility, in the sense that the odds of a dramatically superhumanly intelligent mind worrying about human benefit are vanishingly small. If this is the case, then the best options for the human race would presumably be to either avoid advanced AGI development altogether, or to else fuse with AGI before it gets too strongly superhuman, so that beings-originated-as- humans can enjoy the benefits of greater intelligence and capability (albeit at cost of sacrificing their humanity). Others (e.g. Mark Waser tWas09]) have argued that Friendly AI is essentially inevitable, because greater intelligence correlates with greater morality. Evidence from evolutionary and human history is adduced in favor of this point, along with more abstract arguments. Yudkowsky EYndOGI has discussed the possibility of creating AGI architectures that are in some sense "provably Friendly" - either mathematically, or else at least via very tight lines of ra- tional verbal argumentation. However, several issues have been raised with this approach. First, it seems likely that proving mathematical results of this nature would first require dramatic ad- vances in multiple branches of mathematics. Second, such a proof would require a formalization of the goal of "Friendliness," which is a subtler matter than it might seem ILegO6b, LegMaj. EFTA00623982

12.2 Review of Current Thinking on the Risks of AC1 207 Formalization of human morality has vexed moral philosophers for quite some time. Finally, it is unclear the extent to which such a proof could be created in a generic, environment-independent way - but if the proof depends on properties of the physical environment, then it would re- quire a formalization of the environment itself, which runs up against various problems such as the complexity of the physical world and also the fact that we currently have no complete, consistent theory of physics. Kaj Sotala has provided a list of 14 objections to the Friendly AI concept, and suggested answers to each of them ISot III. Stephen Omohundro 1Omo081 has argued that any advanced AI system will very likely demonstrate certain "basic AI drives", such as desiring to be rational, to self-protect, to acquire resources, and to preserve and protect its utility function and avoid counterfeit utility; these drives, he suggests, must be taken carefully into account in formulating approaches to Friendly AI. The problem of formally or at least very carefully defining the goal of Friendliness has been considered from a variety of perspectives, none showing dramatic success. Yudkowsky IYud0-11 has suggested the concept of "Coherent Extrapolated Volition", which roughly refers to the extrapolation of the common values of the human race. Many subtleties arise in specifying this concept - e.g. if Bob Jones is often possessed by a strong desire to kill all Martians, but he deeply aspires to be a nonviolent person, then the CEV approach would not rate "killing Martians" as part of Bob's contribution to the CEV of humanity. Goertzel [Goel0al has proposed a related notion of Coherent Aggregated Volition (CAV), which eschews the subtleties of extrapolation, and simply seeks a reasonably compact, coherent, consistent set of values that is fairly close to the collective value-set of humanity. In the CAV approach, "killing Martians" would be removed from humanity's collective value-set because it's uncommon and not part of the most compact/coherent/consistent overall model of human values, rather than because of Bob Jones' aspiration to nonviolence. One thought we have recently entertained is that the core concept underlying CAV might be better thought of as CBV or "Coherent Blended Volition." CAV seems to be easily misin- terpreted as meaning the average of different views, which was not the original intention. The CBV terminology clarifies that the CBV of a diverse group of people should not be thought of as an average of their perspectives, but as something more analogous to a "conceptual blend" IFT02I - incorporating the most essential elements of their divergent views into a whole that is overall compact, elegant and harmonious. The subtlety here (to which we shall return below) is that for a CBV blend to be broadly acceptable, the different parties whose views are being blended mast agree to some extent that enough of the essential elements of their own views have been included. The process of arriving at this sort of consensus may involve extrapolation of a roughly similar sort to that considered in CEV. Multiple attempts at axiomatization of human values have also been attempted, e.g. with a view toward providing near-term guidance to military robots (see e.g. Arkin's excellent though chillingly-titled book Governing Lethal Behavior in Autonomous Robots EArk0911, the result of US military funded research). However, there are reasonably strong arguments that human values (similarly to e.g. human language or human perceptual classification rules) are too com- plex and multifaceted to be captured in any compact set of formal logic rules. Wallach IWA has made this point eloquently, and argued the necessity of fusing top-down (e.g. formal logic based) and bottom-up (e.g. self-organizing learning based) approaches to machine ethics. A number of more sociological considerations also arise. It is sometimes argued that the risk from highly-advanced AGI going morally awry on its own may be less than that of moderately- advanced AGI being used by human beings to advocate immoral ends. This possibility gives EFTA00623983

208 12 The Engineering and Development of Ethics rise to questions about the ethical value of various practical modalities of AGI development, for instance: • Should AGI be developed in a top-secret installation by a select group of individuals selected for a combination of technical and scientific brilliance and moral uprightness, or other qualities deemed relevant (a "closed approach")? Or should it be developed out in the open, in the manner of open-source software projects like Limuc? (an "open approach"). The open approach allows the collective intelligence of the world to more fully participate - but also potentially allows the more unsavory elements of the human race to take some of the publicly-developed AGI concepts and tools private, and develop them into AGIs with selfish or evil purposes in mind. Is there some meaningful intermediary between these extremes? • Should governments regulate AGI, with Friendliness in mind (as advocated carefully by e.g Bill Hibbard illib021)? Or will this just cause AGI development to move to the handful of countries with more liberal policies? ... or cause it to move underground, where nobody can see the dangers developing? As a rough analogue, it's worth noting that the US government's imposition of restrictions on stem cell research, under President George W. Bush, appears to have directly stimulated the provision of additional funding for stem cell research in other nations like Korea, Singapore and China. The former issue is, obviously, highly relevant to CogPrime (which is currently being devel- oped via the open source CogPrime project); and so the various dimensions of this issues are worth briefly sketching here. We have a strong skepticism of self-appointed elite groups that claim (even if they genuinely believe) that they know what's best for everyone, and a healthy respect for the power of collective intelligence and the Global Brain, which the open approach is ideal for tapping. On the other hand, we also understand the risk of terrorist groups or other malevolent agents forking an open source AGI project and creating something terribly dangerous and destructive. Balancing these factors against each other rigorously, seems beyond the scope of current human science. Nobody really understands the social dynamics by which open technological knowledge plays out in our current world, let alone hypothetical future scenarios. Right now there exists open knowledge about many very dangerous technologies, and there exist many terrorist groups, yet these groups fortunately make scant use of these technologies. The reasons why appear to be essentially sociological — the people involved in these terrorist groups tend not to be the ones who have mastered the skills of turning public knowledge on cutting-edge technologies into real engineered systems. But while it's easy to observe this sociological phenomenon, we certainly have no way to estimate its quantitative extent from first principles. We don't really have a strong understanding of how safe we are right now, given the technology knowledge available right now via the Internet, textbooks, and so forth. Even relatively straightforward issues such as nuclear proliferation remain confusing, even to the experts. It's also quite clear that keeping powerful AGI locked up by an elite group doesn't really provide reliable protection against malevolent human agents. History, is rife with such situations going awry, e.g. by the leadership of the group being subverted, or via brute force inflicted by some outside party, or via a member of the elite group defecting to some outside group in the interest of personal power or reward or due to group-internal disagreements, etc. There are many things that can go wrong in such situations, and the confidence of any particular group that they are immune to such issues, cannot be taken very seriously. Clearly, neither the open nor closed approach qualifies as a panacea. EFTA00623984

12.3 The Value of an Explicit Goal System 209 12.3 The Value of an Explicit Goal System One of the subtle issues confronted in the quest to design ethical AGIs is how closely one wants to emulate human ethical judgment and behavior. Here one confronts the brute fact that, even according to their own deeply-held standards, humans are not all that ethical. One high-level conclusion we came to very early in the process of designing CogPrime is that, just as humans are not the most intelligent minds achievable, they are also not the most ethical minds achievable. Even if one takes human ethics, broadly conceived, as the standard - there are almost surely possible AGI systems that are much more ethical according to human standards than nearly all human beings. This is not mainly because of ethics-specific features of the human mind, but rather because of the nature of the human motivational system, which leads to many complexities that drive humans to behaviors that are unethical according to their own standards. So, one of the design decisions we made for CogPrime - with ethics as well as other reasons in mind - was not to closely imitate the human motivational system, but rather to craft a novel motivational system combining certain aspects of the human motivational system with other profoundly non-human aspects. On the other hand, the design of ethical AGI systems still has a lot to gain from the study of human ethical cognition and behavior. Human ethics has many aspects, which we associate here with the different types of memory, and it's important that AGI systems can encompass all of them. Also, as we will note below, human ethics develops in childhood through a series of natural stages, parallel to and entwined with the cognitive developmental stages reviewed in Chapter 11 above. We will argue that for an AGI with a virtual or robotic body, it makes sense to think of ethical development as proceeding through similar stages. In a CogPrime context, the particulars of these stages can then be understood in terms of the particulars of CogPrime's cognitive processes - which brings AGI ethics from the domain of theoretical abstraction into the realm of practical algorithm design and education. But even if the human stages of ethical development make sense for non-human AGIs, this doesn't mean the particulars of the human motivational system need to be replicated in these AGIs, regarding ethics or other matters. A key point here is that, in the context of human intelligence, the concept of a "goal" is a descriptive abstraction. But in the AGI context, it seems quite valuable to introduce goals as explicit design elements (which is what is done in CogPrime ) - both for ethical reasons and for broader AGI design reasons. Humans may adopt goals for a time and then drop them, may pursue multiple conflicting goals simultaneously, and may often proceed in an apparently goal-less manner. Sometimes the goal that a person appears to be pursuing, may be very different than the one they think they're pursuing. Evolutionary psychology 1B1)1,931 argues that, directly or indirectly, all humans are ultimately pursuing the goal of maximizing the inclusive fitness of their genes - but given the complex mix of evolution and self-organization in natural history, iSa193j, this is hardly a general explanation for human behavior. Ultimately, in the human context, "goal" is best thought of as a frequently useful heuristic concept. AGI systems, however, need not emulate human cognition in every aspect, and may be architected with explicit "goal systems." This provides no guarantee that said AGI systems will actually pursue the goals that their goal systems specify - depending on the role that the goal system plays in the overall system dynamics, sometimes other dynamical phenomena might intervene and cause the system to behave in ways opposed to its explicit goals. However, we submit that this design sketch provides a better framework than would exist in an AGI system closely emulating the human brain. EFTA00623985

210 12 The Engineering and Development of Ethics We realize this point may be somewhat contentious - a counter-argument would be that the human brain is known to support at least moderately ethical behavior, according to human ethical standards, whereas less brain-like AGI systems are much less well understood. However, the obvious counter-counterpoints are that: • Humans are not all that consistently ethical, so that creating AGI systems potentially much more practically powerful than humans, but with closely humanlike ethical, motivational and goal systems, could in fact be quite dangerous • The effect on a human-like ethical/motivational/goal system of increasing the intelligence, or changing the physical embodiment or cognitive capabilities, of the agent containing the system, is unknown and difficult to predict given all the complexities involved The course we tentatively recommend, and are following in our own work, is to develop AGI systems with explicit, hierarchically-dominated goal systems. That is: • create one or more "top goals" (we call them Ubergoals in CogPrime ) • have the system derive subgoals from these, using its own intelligence, potentially guided by educational interaction or explicit programming • have a significant percentage of the system's activity governed by the explicit pursuit of these goals Note that the "significant percentage" need not be 100%; CogPrime, for example, combines explicitly goal-directed activity with other "spontaneous" activity. Requiring that all activity be explicitly goal-directed may be too strict a requirement to place on AGI architectures. The next step, of course, is for the top-level goals to be chosen in accordance with the principle of human-Friendliness. The next one of our eight points, about the Global Brain, addresses one way of doing this. In our near-term work with CogPrime, we are using simplistic approaches, with a view toward early-stage system testing. 12.4 Ethical Synergy An explicit goal system provides an explicit way to ensure that ethical principles (as represented in system goals) play a significant role in guiding an AGI system's behavior. However, in an integrative design like CogPrime the goal system is only a small part of the overall story, and it's important to also understand how ethics relates to the other aspects of the cognitive architecture. One of the more novel ideas presented in this chapter is that different types of ethical intuition may be associated with different types of memory - and to poacess mature ethics, a mind must display ethical synergy between the ethical processes associated with its memory types. Specifically, we suggest that: • Episodic memory corresponds to the process of ethically assessing a situation based on similar prior situations • Sensorimotor memory corresponds to "mirror neuron" type ethics, where you feel another person's feelings via mirroring their physiological emotional responses and actions • Declarative memory corresponds to rational ethical judgment EFTA00623986

12.4 Ethical Synergy 211 • Procedural memory corresponds to "ethical habit" ... learning by imitation and rein- forcement to do what is right, even when the reasons aren't well articulated or understood • Attentional memory corresponds to the existence of appropriate patterns guiding one to pay adequate attention to ethical considerations at appropriate times • Intentional memory corresponds to the pervasion of ethics through one's choices about subgoaling (which leads into "when do the ends justify the means" ethical-balance questions) One of our suggestions regarding AGI ethics is that an ethically mature person or AGI must both master and balance all these kinds of ethics. We will focus especially here on declarative ethics, which corresponds to Kohlberg's theory of logical ethical judgment; and episodic ethics, which corresponds to Gilligan's theory of empathic ethical judgment. Ultimately though, all five aspects are critically important; and a CogPrime system if appropriately situated and educated should be able to master and integrate all of them. 12.4.1 Stages of Development of Declarative Ethics Complementing generic theories of cognitive development such as Piaget's and Perry's, theorists have also proposed specific stages of moral and ethical development. The two most relevant theories in this domain are those of Kohlberg and Gilligan, which we will review here, both individually and in terms of their integration and application in the AGI context. Lawrence Kohlberg's IKLII83, Koh8 I] moral development model, called the "ethics of justice" by Gilligan, is based on a rational modality as the central vehicle for moral development. In our perspective this is a firmly declarative form of ethics, based on explicit analysis and reasoning. It is based on an impartial regard for persons, proposing that ethical consideration must be given to all individual intelligences without a priori judgment (prejudice). Consideration is given for individual merit and preferences, and the goals of an ethical decision are equal treatment (in the general, not necessarily the particular) and reciprocity. Echoing Kant's [Kanq categorical imperative, the decisions considered most successful in this model are those which exhibit "reversibility", where a moral act within a particular situation is evaluated in terms of whether or not the act would be satisfactory, even if particular persons were to switch roles within the situation. In other words, a situational, contextualized "do unto others as you would have them do unto you" criterion. The ethics of justice can be viewed as three stages (each of which has six substages, on which we will not elaborate here), depicted in Table 12.1. In Kohlberg's perspective, cognitive development level contributes to moral development, as moral understanding emerges from increased cognitive capability in the area of ethical decision making in a social context. Relatedly, Kohlberg also looks at stages of social perspective and their consequent interpersonal outlook. As shown in Table 12.1, these are correlated to the stages of moral development, but also map onto Piagetian models of cognitive development (as pointed out e.g. by Gibbs rib781, who presents a modification/interpretation of Kohlberg's ideas intended to align them more closely with Piaget's). Interpersonal outlook can be under- stood as rational understanding of the psychology of other persons (a theory of mind, with or without empathy). Stage One, emergent from the infantile congitive stage, is entirely selfish as only self awareness has developed. As cognitive sophistication about ethical considerations increases, so do the moral and social perspective stages. Concrete and formal cognition bring about the first instrumental egoism, and then social relations and systems perspectives, and EFTA00623987

212 12 The Engineering and Development of Ethics Stage Substages Pre-Conventional • Obedience and Punishment Orientation • Self-interest orientation Conventional • Interpersonal accord (conformity) orientation • Authority and social-order maintaining (law and order) orientation Post-Conventional • Social contract (human rights) orientation • Universal ethical principles (universal human rights) ori- entation Table 12.1: Kohlberg's Stages of Development of the Ethics of Justice from formal and then reflexive thinking about ethics comes the post-conventional modalities of contractualism and universal mutual respect. Stage of Social Per- spective Interpersonal Outlook Blind egoism No interpersonal perspective. Only self is considered. Instrumental egoism See that others have goals and perspectives, and either con- form to or rebel against norms. Social Relationships perspective Able to see abstract normative systems Social Systems per- spective Recognize positive and negative intentions Contractual perspec- tive Recognize that contracts (mutually beneficial agreements of any kind) will allow intelligences to increase the welfare of both. Universal principle of mutual respect See how human fallibility and frailty are impacted by commu- nication. Table 12.2: Kohlberg's Stages of Development of Social Perspective and Interpersonal Morals 12.4.1.1 Uncertain Inference and the Ethics of Justice Taking our cue from the analysis given in Chapter 11 of Piagetan stages in uncertain infer- ence based AGI systems (such as CogPrime ), we may explore the manifestation of Kohlberg's stages in AGI systems of this nature. Uncertain inference seems generally well-suited as a declarative-ethics learning system, due to the nuanced ethical environment of real world sit- uations. Probabilistic knowledge networks can model belief networks, imitative reinforcement learning based ethical pedagogy, and even simplistic moral maxims. In principle. they have the flexibility to deal with complex ethical decisions, including not only weighted "for the greater EFTA00623988

12.4 Ethical Synergy 213 good" dichotomous decision making, but also the ability to develop moral decision networks which do not require that all situations be solved through resolution of a dichotomy. When more than one person is being affected by an ethical decision, making a decision based on reducing two choices to a single decision can often lead to decisions of dubious ethics. How- ever, a sufficiently complex uncertain inference net work can represent alternate choices in which multiple actions are taken that have equal (or near equal) belief weight but have very different particulars - but because the decisions are applied in different contexts (to different groups of individuals) they are morally equivalent. Though each individual action appears equally be- lievable. were any single decision applied to the entire population one or more individual may be harmed, and the morally superior choice is to make case-dependent decisions. Equal moral treatment is a general principle, and too often the mistake is made by thinking that to achieve this general principle the particulars must be equal. This is not the case. Different treatment of different individuals can result in morally equivalent treatment of all involved individuals, and may be vastly morally superior to treating all the individuals with equal particulars. Simply taking the largest population and deciding one course of action based on the result that is most appealing to that largest group is not generally the most moral action. Uncertain inference, especially a complex network with high levels of resource access as may be found in a sophisticated AGI, is well suited for complex decision making resulting in a multitude of actions, and of analyzing the options to find the set of actions that are ethically optimal particulars for each decision context. Reflexive cognition and post-commitment moral understanding may be the goal stages of an AGI system, or any intelligence. but the other stages will be passed through on the way to that goal, and realistically some minds will never reach higher order cognition or morality with regards to any context, and others will not be able to function at this high order in every context (all currently known minds fail to function at the highest order cognitively or morally in some contexts). Infantile and concrete cognition are the underpinnings of the egoist and socialized stages, with formal aspects also playing a role in a more complete understanding of social models when thinking using the social modalities. Cognitively infantile patterns can produce no more than blind egoism as without a theory of mind, there is no capability to consider the other. Since most intelligences acquire concrete modality and therefore some nascent social perspective relatively quickly, most egoists are instrumental egoists. The social relationship and systems perspectives include formal aspects which are achieved by systematic social experimentation, and therefore experiential reinforcement learning of correct and incorrect social modalities. Initially this is a one-on-one approach (relationship stage), but as more knowledge of social action and consequences is acquired, a formal thinker can understand not just consequentiality but also intentionality in social action. Extrapolation from models of individual interaction to general social theoretic notions is also a formal action. Rational, logical positivist approaches to social and political ideas, however, are the norm of formal thinking. Contractual and committed moral ethics emerges from a higher- order formalization of the social relationships and systems patterns of thinking. Generalizations of social observation become, through formal analysis, systems of social and political doctrine. Highly committed, but grounded and logically supportable, belief is the hallmark of formal cognition as expressed contractual moral stage. Though formalism is at work in the socialized moral stages, its fullest expression is in committed contractualism. Finally, reflexive cognition is especially important in truly reaching the post-commitment moral stage in which nuance and complexity are accommodated. Because reflexive cognition is necessary to change one's mind not just about particular rational ideas, but whole ways of EFTA00623989

214 12 The Engineering and Development of Ethics thinking, this is a cognitive precedent to being able to reconsider an entire belief system, one that has had contractual logic built atop reflexive adherence that began in early development. If the initial moral system is viewed as positive and stable, then this cognitive capacity is seen as dangerous and scary, but if early morality is stunted or warped, then this ability is seen as enlightened. However, achieving this cognitive stage does not mean one automatically changes their belief systems, but rather that the mental machinery, is in place to consider the possibilities. Because many people do not reach this level of cognitive development in the area of moral and ethical thinking, it is associated with negative traits ("moral relativism" and "flip-flopping"). However, this cognitive flexibility generally leads to more sophisticated and applicable moral codes, which in turn leads to morality which is actually more stable because it is built upon extensive and deep consideration rather than simple adherence to reflexive or rationalized ideologies. 12.4.2 Stages of Development of Empathic Ethics Complementing Kohlberg's logic-and-justice-focused approach, Carol Gilligan's ril82] "ethics of care" model is a moral development theory which posits that empathetic understanding plays the central role in moral progression from an initial self-centered modality to a socially responsible one. The ethics of care model is concerned with the ways in which an individual cares (responds to dilemmas using empathetic responses) about self and others. As shown in Table 12.3, the ethics of care is broken into the same three primary stage as Kohlberg, but with a focus on empathetic, emotional caring rather than rationalized, logical principles of justice. Stage Principle of Care Pre-Conventional Individual Survival Conventional Self Sacrifice for the Greater Good Post-Conventional Principle of Nonviolence (do not hurt others, or oneself) Table 12 3: Gilligan's Stages of the Ethics of Care For an "ethics of care" approach to be applied in an AGI, the AGI must be capable of internal simulation of other minds it encounters, in a similar manner to how humans regularly simulate one another internally. Without any mechanism for internal simulation, it is unlikely that an AGI can develop any sort of empathy toward other minds, as opposed to merely logically or probabilistically modeling other agents' behavior or other minds' internal contents. In a CogPrime context, this ties in closely with how CogPrime handles episodic knowledge - partly via use of an internal simulation world, which is able to play "mental movies" of prior and hypothesized scenarios within the AGI system's mind. However, in humans empathy involves more than just simulation, it also involves sensorimotor responses, and of course emotional responses - a topic we will discuss in more depth in Appendix ?? where we review the functionality of mirror neurons and mirror systems in the human brains. When we see or hear someone suffering, this sensory input causes motor responses in us similar to if we were suffering ourselves, which initiates emotional empathy and corresponding cognitive processes. EFTA00623990

12.4 Ethical Synergy 215 Thus, empathic "ethics of care" involves a combination of episodic and sensorimotor ethics, complementing the mainly declarative ethics associated with the "ethics of justice." In Gilligan's perspective, the earliest stage of ethical development occurs before empathy becomes a consistent and powerful force. Next, the hallmark of the conventional stage is that at this point, the individual is so overwhelmed with their empathic response to others that they neglect themselves in order to avoid hurting others. Note that this stage doesn't occur in Kohlberg's hierarchy at all. Kohlberg and Gilligan both begin with selfish unethicality, but their following stages diverge. A person could in principle manifest Gilligan's conventional stage without having a refined sense of justice (thus not entering Kohlberg's conventional stage); or they could manifest Kohlberg's conventional stage without partaking in an excessive degree of self-sacrifice (thus not entering Gilligan's conventional stage). We will suggest below that in fact the empathic and logical aspects of ethics are more unified in real human development than these separate theories would suggest. However, even if this is so, the possibility is still there that in some AGI systems the levels of declarative and empathic ethics could wildly diverge. It is interesting to note that Gilligan's and Kohlberg's final stages converge more closely than their intermediate ones. Kohlberg's post-conventional stage focuses on universal rights, and Gilligan's on universal compassion. Still, the foci here are quite different; and, as will be elaborated below, we believe that both Kohlberg's and Gilligan's theories constitute very partial views of the actual end-state of ethical advancement. 12.4.3 An Integrative Approach to Ethical Development We feel that both Kohlberg's and Gilligan's theories contain elements of the whole picture of ethical development, and that both approaches are necessary to create a moral, ethical artificial general intelligence - just as, we suggest, both internal simulation and uncertain inference are necessary to create a sufficiently intelligent and volitional intelligence in the first place. Also, we contend, the lack of direct analysis of the underlying psychology of the stages is a deficiency shared by both the Kohlberg and Gilligan models as they are generally discussed. A successful model of integrative ethics necessarily contains elements of both the care and justice models, as well as reference to the underlying developmental psychology and its influence on the character of the ethical stage. Furthermore, intentional and attentional ethics need to be brought into the picture, complementing Kohlberg's focus on declarative knowledge and Gilligan's focus on episodic and sensorimotor knowledge. With these notions in mind, we propose the following integrative theory of the stages of ethical development, shown in Tables 12.4, 12.5 and 12.6. In our integrative model, the justice- based and empathic aspects of ethical judgment are proposed to develop together. Of course, in any one individual, one or another aspect may be dominant. Even so, however, the combination of the two is equally important as either of the two individual ingredients. For instance, we suggest that in any psychologically healthy human, the conventional stage of ethics (typifying childhood, and in many cases adulthood as well) involves a combination of Gilligan-mqe empathic ethics and Kohlberg-esque ethical reasoning. This combination is supported by Piagetan concrete operational cognition, which allows moderately sophisticated linguistic interaction, theory of mind, and symbolic modeling of the world. And, similarly, we propose that in any truly ethically mature human, empathy and rational justice are both fully developed. Indeed the two interpenetrate each other deeply. EFTA00623991

216 12 The Engineering and Development of Ethics Once one goes beyond simplistic, childlike notions of fairness ("an eye for an eye" and so forth), applying rational justice in a purely intellectual sense is just as difficult as any other real-world logical inference problem. Ethical quandaries and quagmires are easily encountered, and are frequently cut through by a judicious application of empathic simulation. On the other hand, empathy is a far more powerful force when used in conjunction with reason: analogical reasoning lets us empathize with situations we have never experienced. For instance, a person who has never been clinically depressed may have a hard time empathizing with individuals who are; but using the power of reason, they can imagine their worst state of depression magnified by several times and then extended over a long period of time, and then reason about what this might be like ... and empathize based on their inferential conclusion. Reason is not antithetical to empathy but rather is the key to making empathy more broadly impactful. Finally, the enlightened stage of ethical development involves both a deeper compassion and a more deeply penetrating rationality and objectiveness. Empathy with all sentient beings is manageable in everyday life only once one has deeply reflected on one's own self and largely freed oneself of the confusions and illusions that characterize much of the ordinary human's inner existence. It is noteworthy, for example, that Buddhism contains both a richly developed ethics of universal compassion, and also an intricate logical theory of the inner workings of cognition Ititc001, detailing in exquisite rational detail the manner in which minds originate structures and dynamics allowing them to comprehend themselves and the world. 12.4.4 Integrative Ethics and Integrative AGI What does our integrative approach to ethical development have to say about the ethical development of AGI systems? The lessons are relatively straighforward, if one considers an AGI system that, like CogPrime, explicitly contains components dedicated to logical inference and to simulation. Application of the above ethical ideas to other sorts of AGI systems is also quite possible, but would require a lengthier treatment and so won't be addressed here. In the context of a CogPrime-type AGI system, Kolhberg's stages correspond to increasingly sophisticated application of logical inference to matters of rights and fairness. It is not clear whether humans contain an innate sense of fairness. In the context of AGIs, it would be possible to explicitly wire a sense of fairness into an AGI system, but in the context of a rich environment and active human teachers, this actually appears quite unnecessary. Experiential instruction in the notions of rights and fairness should suffice to teach an inference-based AGI system how to manipulate these concepts, analogously to teaching the same AGI system how to manipulate number, mass and other such quantities. Ascending the Kohlberg stages is then mainly a matter of acquiring the ability to carry out suitably complex inferences in the domain of rights and fairness. The hard part here is inference control - choosing which inference steps to take - and in a sophisticated AGI inference engine, inference control will be guided by experience, so that the more ethical judgments the system has executed and witnessed, the better it will become at making new ones. And, as argued above, simulative activity can be extremely valuable for aiding with inference control. When a logical inference process reaches a point of acute uncertainty (the backward or forward chaining inference tree can't decide which expansion step to take), it can run a simulation to cut through the confusion - i.e., it can use empathy to decide which EFTA00623992

12.4 Ethical Synergy 217 Stage Characteristics Pre-ethical • Piagetan infantile to early concrete (aka pre-operational) • Radical selfishness or selflessness may, but do not neces- sarily, occur • No coherent, consistent pattern of consideration for the rights, intentions or feelings of others • Empathy is generally present, but erratically Conventional Ethics • Concrete cognitive basis • Perry's Dualist and Multiple stages • The common sense of the Golden Rule is appreciated, with cultural conventions for abstracting principles from behaviors • One's own ethical behavior is explicitly compared to that of others • Development of a functional, though ed, theory of mind • Ability to intuitively conceive of notions of fairness and rights • Appreciation of the concept of law and order, which may sometimes manifest itself as systematic obedience or sys- tematic disobedience • Empathy is more consistently present, especially with others who are directly similar to oneself or in situations similar to those one has directly experienced • Degrees of selflessness or selfishness develop based on eth- ical groundings and social interactions. Table 12.4: Integrative Model of the Stages of Ethical Development, Part 1 logical inference step to take in thinking about applying the notions of rights and fairness to a given situation. Gilligan's stages correspond to increasingly sophisticated control of empathic simulation — which in a CogPrime-type AGI system, Ls carried out by a specific system component devoted to running internal simulations of aspects of the outside world, which includes a subcomponent specifically tuned for simulating sentient actors. The conventional stage has to do with the raw, uncontrolled capability for such simulation; and the post-conventional stage corresponds to its contextual, goal-oriented control. But controlling empathy, clearly, requires subtle management of various uncertain contextual factors, which is exactly what uncertain logical inference is good at - so, in an AGI system combining an uncertain inference component with a simulative component, it is the inference component that would enable the nuanced control of empathy allowing the ascent to Gilligan's post-conventional stage. In our integrative perspective, in the context of an AGI system integrating inference and simulation components, we suggest that the ascent from the pre-ethical to the conventional stage may be carried out largely via independent activity of these two components. Empathy is needed, and reasoning about fairness and rights are needed, but the two need not intimately and sensitively intersect - though they mast of course intersect to some extent. EFTA00623993

218 12 The Engineering and Development of Ethics Stage Characteristics Mature Ethics • Formal cognitive basis • Perry's Relativist and "Constructed Knowledge" stages • The abstraction involved with applying the Golden Rule in practice is more fully understood and manipulated, leading to limited but nonzero deployment of the Cate- gorical Imperative • Attention is paid to shaping one's ethical principles into a coherent logical system • Rationalized, moderated selfishness or selflessness. • Empathy is extended, using reason, to individuals and situations not directly matching one's own experience • Theory of mind is extended, using reason, to counterin- tuitive or experientially unfamiliar situations • Reason is used to control the impact of empathy on be- havior (i.e. rational judgments are made regarding when to listen to empathy and when not to) • Rational experimentation and correction of theoretical models of ethical behavior, and reconciliation with ob- served behavior during interaction with others. • Conflict between pragmatism of social contract orienta- tion and idealism of universal ethical principles. • Understanding of ethical quandaries and nuances develop (pragmatist modality), or are rejected (idealist modality). • Pragmatically critical social citizen. Attempts to main- tain a balanced social outlook. Considers the common good, including oneself as part of the commons, and acts in what seems to be the most beneficial and practical manner. Table 12.5: Integrative Model of the Stages of Ethical Development, Part 2 The main engine of advancement from the conventional to mature stage, we suggest, is robust and subtle integration of the simulative and inferential components. To expand empathy beyond the most obvious cases, analogical inference is needed; and to carry out complex inferences about justice, empathy-guided inference-control is needed. Finally, to advance from the mature to the enlightened stage, what is required is a very advanced capability for unified reflexive inference and simulation. The system must be able to understand itself deeply, via modeling itself both simulatively and inferentially - which will generally be achieved via a combination of being good at modeling, and becoming less convoluted and more coherent, hence making self-modeling easier. Of course, none of this tells you in detail how to create an AGI system with advanced ethical capabilities. What it does tell you, however, is one possible path that may be followed to achieve this end goal. If one creates an integrative AGI system with appropriately interconnected inferential and simulative components, and treats it compassionately and fairly, and provides it extensive, experientially grounded ethical instruction in a rich social environment, then the AGI system should be able to ascend the ethical hierarchy and achieve a high level of ethical sophistication. In fact it should be able to do so more reliably than human beings because of the capability we have to identify its errors via inspecting its internal knowedge-stage, which EFTA00623994

12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model 219 Stage Characteristics Enlightened Ethics • Reflexive cognitive basis • Permeation of the categorical imperative and the quest for coherence through inner as well as outer life • Experientially grounded and logically supported rejection of the illusion of moral certainty in favor of a case-specific analytical and empathetic approach that embraces the uncertainty of real social life • Deep understanding of the illusory and biased nature of the individual self, leading to humility regarding one's own ethical intuitions and prescriptions • Openness to modifying one's deepest, ethical (and other) beliefs based on experience, reason and/or empathic com- munion with others • Adaptive, insightful approach to civil disobedience, con- sidering laws and social customs in a broader ethical and pragmatic context • Broad compassion for and empathy with all sentient be- ings • A recognition of inability to operate at this level at all times in all things, and a vigilance about self-monitoring for regressive behavior. Table 12.6: Integrative Model of the Stages of Ethical Development, Part 3 will enable us to tailor its environment and instructions more suitably than can be done in the human case. If an absolute guarantee of the ethical soundness of an AGI is what one is after, the line of thinking proposed here is not at all useful. Experiential education is by its nature an uncertain thing. One can strive to minimize the uncertainty, but it will still exist. Inspection of the internals of an AGI's mind is not a total solution to uncertainty minimization, because any AGI capable of powerful general intelligence is going to have a complex internal state that no external observer will be able to fully grasp, no matter how transparent the knowledge representation. However, if what one is after is a plausible, pragmatic path to architecting and educating ethical AGI systems, we believe the ideas presented here constitute a sensible starting-point. Certainly there is a great deal more to be learned and understood - the science and practice of AGI ethics, like AGI itself, are at a formative stage at present. What is key, in our view, is that as AGI technology develops, AGI ethics develops alongside and within it, in a thoroughly coupled way. 12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model One of the issues with the "ethics of justice" as reviewed above, which makes it inadequate to serve as the sole basis of an AGI ethical system (though it may certainly play a significant EFTA00623995

220 12 The Engineering and Development of Ethics role), is the lack of any clear formulation of what "justice" means. This section explores this issue, via detailed consideration of the "Golden Rule" folk maxim do unto others as you would have them do unto you - a classical formulation of the notion of fairness and justics - to AGI ethics. Taking the Golden Rule as a starting-point, we will elaborate five ethical imperatives that incorporate aspects of the notion of ethical synergy discussed above. Simple as it may seem, the Golden Rule actually elicits a variety of deep issues regarding the relationship between ethics, experience and learning. When seriously analyzed, it results in a multifactorial elaboration, involving the combination of various factors related to the basic Golden Rule idea. Which brings us back in the end to the potential value of methods like CEV, CAV or CBV for understanding how human ethics balances the multiple factors. Our goal here is not to present any kind of definitive analysis of the ethics of justice, but just to briefly and roughly indicate a number of the relevant significant issues - things that anyone designing or teaching an AGI would do well to keep in mind. The trickiest aspect of the Golden Rule, as has been frequently observed, is achieving the right level of abstraction. Taken too literally, the Golden Rule would suggest, for instance, that a parent should not wipe a child's soiled bottom because the parent does not want the child to wipe the parent's soiled bottom. But if the parent interprets the Golden Rule more intelligently and abstractly, the parent may conclude that they should wipe the child's bottom after all: they should "wipe the child's bottom when the child can't do it themselves", consistently with believing that the child should "wipe the parent's bottom when the parent can't do it themselves" (which may well happen eventually should the parent develop incontinence in old age). This line of thinking leads to Kant's Categorical Imperative ilian61] which (in one inter- pretation) states essentially that one should "Act only according to that maxim whereby you can at the same time will that it should become a universal law." The Categorical Imperative adds precision to the Golden Rule, but also removes the practicality of the latter. Formaliz- ing the "implicit universal law" underlying an everyday action is a huge problem, falling prey to the same issue that has kept us from adequately formalizing the rules of natural language grammar, or formalizing common-sense knowledge about everyday object like cups, bowls and grass (substantial effort notwithstanding, e.g. Cyc in the commonsense knowledge case. and the whole discipline of modern linguistics in the NL case). There is no way to apply the Categorical Imperative, as literally stated, in everyday life. Furthermore, if one wishes to teach ethics as well as to practice it, the Categorical Imper- ative actually has a significant disadvantage compared to some other possible formulations of the Golden Rule. The problem is that, if one follows the Categorical Imperative, one's fellow members of society may well never understand the principles under which one is acting. Each of us may internally formulate abstract principles in a different way, and these may be very difficult to communicate, especially among individuals with different belief systems, different cognitive architectures, or different levels of intelligence. Thus, if one's goal is not just to act ethically, but to encourage others to act ethically by setting a good example, the Categorical Imperative may not be useful at all, as others may be unable to solve the "inverse problem" of guessing your intended maxim from your observed behavior. On the other hand, one wouldn't want to universally restrict one's behavioral maxims to those that one's fellow members of society can understand - in that case, one would have to act with a two-year old or a dog according to principles that they could understand, which would clearly be unethical according to human common sense. (Every two-year-old, once they grow up, would be grateful to their parents for not following this sort of principle.) EFTA00623996

12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model 221 And the concept of "setting a good example" ties in with an important concept from learning theory: imitative learning. Humans appear to be hard-wired for imitative learning, in part via mirror neuron systems in the brain; and, it seems clear that at least in the early stages of AGI development, imitative learning is going to play a key role. Copying what other agents do is an extremely powerful heuristic, and while AGIs may eventually grow beyond this, much of their early ethical education is likely to arise during a phase when they have not done so. A strength of the classic Golden Rule is that one is acting according to behaviors that one wants one's observers to imitate - which makes sense in that many of these observers will be using imitative learning as a significant part of their learning toolkit. The truth of the matter, it seems, is (as often happens) not all that simple or elegant. Ethical behavior seems to be most pragmatically viewed as a multi-objective optimization problem, where among the multiple objectives are three that we have just discussed, and two others that emerge front learning theory and will be discussed shortly: 1. The imitability (i.e. the Golden Rule fairly narrowly and directly construed): the goal of acting in a way so that having others directly imitate one's actions, in directly comparable contexts, is desirable to oneself 2. The comprehensibility: the goal of acting in a way so that others can understand the principles underlying one's actions 3. Experiential groundedness. An intelligent agent should not be expected to act according to an ethical principle unless there are many examples of the principle-in-action in its own direct or observational experience 4. The categorical imperative: Act according to abstract principles t hat you would be happy to see implemented as universal laws 5. Logical coherence. An ethical system should be roughly logically coherent, in the sense that the different principles within it should mesh well with one another and perhaps even naturally emerge from each other. Just for convenience, without implying any finality or great profundity to the list, we will refer to these as the "five imperatives." The above are all ethical objectives to be valued and balanced, to different extents in different contexts. The imitability imperative, obviously, loses importance in societies of agents that don't make heavy use of imitative learning. The comprehensibility imperative is more important in agents that value social community-building generally, and less so in agent that are more isolative and self-focused. Note that the fifth point given above is logically of a different nature than the four previous ones. The first four imperatives govern individual ethical principles; the fifth regards systems of ethical principles, as they interact with each other. Logical coherence is of significant but varying importance in human ethical systems. Huge effort has been spent by theologians of various stripes in establishing and refining the logical coherence of the ethical systems associated with their religions. However, it is arguably going to be even more important in the context of AGI systems, especially if these AGI systems utilize cognitive methods based on logical inference, probability theory or related methods. Experiential groundedness is important because making pragmatic ethical judgments is bound to require reference to an internal library of examples ("episodic ethics") in which eth- ical principles have previously been applied. This is required for analogical reasoning, and in logic-based AGI systems, is also required for pruning of the logical inference trees involved in determining ethical judgments. EFTA00623997

222 12 The Engineering and Development of Ethics To the extent that the Golden Rule is valued as an ethical imperative, experiential grounding may be supplied via observing the behaviors of others. This in itself is a powerful argument in favor of the Golden Rule: without it, the experiential library a system possesses is restricted to its own experience, which is bound to be a very small library compared to what it can assemble from observing the behaviors of others. The overall upshot is that, ideally, an ethical intelligence should act according to a log- ically coherent system of principles, which are exemplified in its own direct and observational experience, which are comprehensible to others and set a good ex- ample for others, and which would serve as adequate universal laws if somehow thus implemented. But, since this set of criteria is essentially impassible to fulfill in prac- tice, real-world intelligent agents must balance these various criteria - often in complex and contextually-dependent ways. We suggest that ethically advanced humans, in their pragmatic ethical choices, tend to act in such a way as to appropriately contextually balance the above factors (along with other criteria, but we have tried to articulate the most key factors). This sort of multi-factorial approach is not as crisp or elegant as unidimensional imperatives like the Golden Rule or the Categorical Imperative, but is more realistic in light of the complexly interacting multiple determinants guiding individual and group human behavior. And this brings us back to CEV, CAV, CBV and other possible ways of mining ethical supergoals from the community of existing human minds. Given that abstract theories of ethics, when seriously pursued as we have done in this section, tend to devolve into complex balancing acts involving multiple factors - one then falls back into asking how human ethical systems habitually perform these balancing acts. Which is what CEV, CAV, CBV try to measure. 12.5.1 The Golden Rule and the Stages of Ethical Development Next we explore more explicitly how these Golden Rule based imperatives align with the eth- ical developmental stages we have outlined here. With this in mind, specific ethical qualities corresponding to the five imperatives have been italicized in the above table of developmental stages. It seems that imperatives 1-3 are critical for the passage from the pre-ethical to the conven- tional stages of ethics. A child learns ethics largely by copying others, and by being interacted with according to simply comprehensible implementations of the Golden Rule. In general, when interacting with children learning ethics, it is important to act according to principles they can comprehend. And given the nature of the concrete stage of cognitive development, experiential groundedness is a must. As a hypothesis regarding the dynamics underlying the psychological development of con- ventional ethics, what we propose is as follows: The emergence of concrete-stage cognitive capabilities leads to the capability for fulfillment of ethical imperatives 1 and 2 — a comprehen- sible and workable implementation of the Golden Rule, based on a combination of inferential and simulative cognition (operating largely separately at this stage, as will be conjectured be- low). The effective interoperation of ethical imperatives 1-3, enacted in an appropriate social environment, then leads to the other characteristics of the conventional ethical stage. The first three imperatives can thus be viewed as the seed from which springs the general nature of conventional ethics. EFTA00623998

12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model 223 On the other hand, logical coherence and the categorical imperative (imperatives 5 and 4) are matters for the formal stage of cognitive development, which come along only with the mature approach to ethics. These come from abstracting ethics beyond direct experience and manipulating them abstractly and formally - a stage which has the potential for more deeply and broadly ethical behavior, but also for more complicated ethical perversions (it is the mature capability for formal ethical reasoning that is able to produce ungrounded abstractions such as "I'm torturing you for your own good"). Developmentally, we suggest that once the capabil- ity for formal reasoning matures, the categorical imperative and the quest for logical ethical coherence naturally emerge, and the sophisticated combination of inferential and simulative cognition embodied in an appropriate social context then result in the emergence of the various characteristics typifying the mature ethical stage. Finally, it seems that one key aspect of the passage from the mature to the enlightened stage of ethics is the penetration of these two final imperatives more and more deeply into the judging mind itself. The reflexive stage of cognitive development is in part about seeking a deep logical coherence between the aspects of one's own mind, and making reasoned modifications to one's mind so as to improve the level of coherence. And, much of the process of mental discipline and purification that comes with the passage to enlightened ethics has to do with the application of the categorical imperative to one's own thoughts and feelings - i.e. making a true inner systematic effort to think and feel only those things one judges are actually generally good and right to be thinking and feeling. Applying these principles internally appears critical for effectively applying them externally, for reasons that are doubtlessly bound up with the inter- penetration of internal and external reality within the thinking mind, and for the "distributed cognition" phenomenon wherein individual mind is itself an approximative abstraction to the reality in which each individual's mind is pragmatically extended across their social group and their environment pito 95]. Obviously, these are complex issues and we're not posing the exploratory discussion given here as conclusive in any sense. But what seems generally clear from this line of thinking is that the complex balance between the multiple factors involved in AGI ethics, shifts during a system's development. If you did CEV, CAV or CBV among five year old humans, ten year old humans, or adult humans, you would get different results. Probably you'd also get different results from senior citizens! The way the factors are balanced depends on the mind's cognitive and emotional stage of development. 12.5.2 The Need for Context-Sensitivity and Adaptiveness in Deploying Ethical Principles As well as depending on developmental stage, there is also an obvious and dramatic context- sensitivity involved here - both in calculating the fulfillment of abstract ethical imperatives, and in balancing various imperatives against each other. As an example, consider the simple Asimovian maxim "I will not harm humans," which may be seen to follow from the Golden Rule for any agent that doesn't itself want to be harmed, and that considers humans as valid agents on the same ethical level as itself. A more serious attempt to formulate this as an ethical maxim might look something like EFTA00623999

224 12 The Engineering and Development of Ethics "I will not harm humans, nor through inaction allow harm to befall them. In situations wherein one or more humans is attempting to harm another individual or group, I shall endeavor to prevent this harm through means which avoid further harm. If this is unavoidable, I shall select the human party to back based on a reckoning of their intentions towards others, and implement their defense through the optimal balance between harm minimization and efficacy. My ultimate goal is to preserve as much as possible of humanity, even if an individual or subgroup of humans must come to harm to do so." However, it's obvious that even a more elaborated principle like this is potentially subject to extensive abuse. Many of the genocides scarring human history have been committed with the goal of preserving and bettering humanity writ large, at the expense of a group of "tmdesirables." Further refinement would be necessary in order to define when the greater good of humanity may actually be served through harm to others. A first actor principle of aggression might seem to solve this problem, but sometimes first actors in violent conflict are taking preemptive measures against the stated goals of an enemy to destroy them. Such situations become very subtle. A single simple maxim can not deal with them very effectively. Networks of interrelated decision criteria, weighted by desirability of consequence and with reference to probabilistically ordered potential side-effects (and their desirability weightings), are required in order to make ethical judgments. The development of these networks, just like any other knowledge network, comes from both pedagogy and experience - and different thoughtful, ethical agents are bound to arrive at different knowledge-networks that will lead to different judgments in real-world situations. Extending the above "mostly harmless" principle to AGI systems, not just humans, would cause it to be more effective in the context of imitative learning. The principle then becomes an elaborated version of "I will not harm sentient beinp." As the imitative-learning-enabled AGI observes humans acting so as to minimize harm to it, it will intuitively and experientially learn to act in such a way as to minimize harm to humans. But then this extension naturally leads to confusion regarding various borderline cases. What is a sentient being exactly? Is a sleeping human sentient? How about a dead human whose information could in principle be restored via obscure quantum operations, leading to some sort of resurrection? How about an AGI whose code has been improved - is there an obligation to maintain the prior version as well, if it is substantially different that its upgrade constitutes a whole new being? And what about situations in which failure to preserve oneself will cause much more harm to others than acting in self defense will. It may be the case that human or group of humans seeks to destroy an AGI in order to pave the way for the enslavement or murder of people under the protection of the AGI. Even if the AGI has been given an ethical formulation of the "mostly harmless" principle which allows it to harm the attacking humans in order to defend its charges, if it is not able to do so in order to defend itself, simply destroying the AGI first will enable the slaughter of those who rely on it. Perhaps a more sensible formulation would allow for some degree of self defense, and Asimov solved this problem with his third law. But where to draw the line between self defense and the greater good also becomes a very complicated issue. Creating hard and fast rules to cover all the various situations that may arise is essentially impossible - the world is ever-changing and ethical judgments must adapt accordingly. This has been true even throughout human history - so how much truer will it be as technological acceleration continues? What is needed is a system that can deploy its ethical principles in an adaptive, context-appropriate way, as it grows and changes along with the world it's embedded in. EFTA00624000

12.5 Clarifying the Ethics of Justice: Extending the Golden Rule in to a Multifactorial Ethical Model 225 And this context-sensitivity has the result of intertwining ethical judgment with all sorts of other judgments - making it effectively impossible to extract "ethics" as one aspect of an intelligent system, separate from other kinds of thinking and acting the system does. This resonates with many prior observations by others, e.g. Eliezer Yudkowsky's insistence that what we need are not ethicists of science and engineering, but rather ethical scientists and engineers - because the most meaningful and important ethical judgments regarding science and engineering generally come about in a manner that's thoroughly interwined with technical practice, and hence are very difficult for a non-practitioner to richly appreciate IGil£i2i. What this context-sensitivity means is that, unless humans and AGIs are experiencing the same sorts of contexts, and perceiving these contexts in at least approximately parallel ways, there is little hope of translating the complex of human ethical judgments to these AGIs. This conclusion has significant implications for which routes to AGI are most likely to lead to success in terms of AGI ethics. We want early-stage AGIs to grow up in a situation where their minds are primarily and ongoingly shaped by shared experiences with humans. Supplying AGIs with abstract ethical principles is not likely to do the trick, because the essence of human ethics in real life seems to have a lot to do with its intuitively appropriate application in various contexts. We transmit this sort of ethical praxis to humans via shared experience, and it seems most probably that in the case of AGIs the transmission must be done the same sort of way. Some may feel that simplistic maxims are less "error prone" than more nuanced, context- sensitive ones. But the history of teaching ethics to human students does not support the idea that limiting ethical pedagogy to slogans provides much value in terms of ethical development. If one proceeds from the idea that AGI ethics must be hard-coded in order to work, then perhaps the idea that simpler ethics means simpler algorithms, and therefore less error potential, has some merit as an initial state. However, any learning system quickly diverges from its initial state, and an ongoing, nuanced relationship between AGIs and humans will - whether we like it or not - form the basis for developmental AGI ethics. AGI intransigence and enmity is not inevitable, but what is inevitable is that a learning system will acquire ideas about both theory and actions from the other intelligent entities in its environment. Either we teach AGIs positive ethics through our interactions with them - both presenting ethical theory and behaving ethically to them - or the potential is there for them to learn antisocial behavior from us even if we pre-load them with some set of allegedly inviolable edicts. All in all, developmental ethics is not as simple as many people hope. Simplistic approaches often lead to disastrous consequences among humans, and there is no reason to think this would be any different in the case of artificial intelligences. Most problems in ethics have cases in which a simplistic ethical formulation requires substantial revision to deal with extenuating circumstances and nuances found in real world situations. Our goal in this chapter is not to enumerate a full set of complex networks of interacting ethical formulations as applicable to AGI systems (that is a project that will take years of both theoretical study and hands-on research), but rather to point out that this program must be undertaken in order to facilitate a grounded and logically defensible system of ethics for artificial intelligences, one which is as unlikely to be undermined by subsequent self-modification of the AGI as is possible. Even so, there is still the risk that whatever predispositions are imparted to the AGIs through initial codification of ethical ideas in the system's internal logic representation, and through initial pedagogical interactions with its learning systems, will be undermined through reinforcement learning of antisocial behavior if humans do not interact ethically with AGIs. Ethical treatment is a necessary task for grounding ethics and making them unlikely to be distorted during internal rewriting. EFTA00624001

225 12 The Engineering and Development of Ethics The implications of these ideas for ethical instruction are complex and won't be fully elabo- rated here, but a few of them are compact and obvious: 1. The teacher(s) must be observed to follow their own ethical principles, in a variety of contexts that are meaningful to the AGI 2. The system of ethics must be relevant to the recipient's life context, and embedded within their understanding of the world. 3. Ethical principles must be grounded in both theory-of-mind thought experiments (empha- sizing logical coherence), and in real life situations in which the ethical trainee is required to make a moral judgment and is rewarded or reproached by the teacher(s), including the imparting of explanatory augmentations to the teachings regarding the reason for the par- ticular decision on the part of the teacher. Finally, harking forward to the next section which emphasizes the importance of respecting the freedom of AG's, we note that it is implicit in our approach to AGI ethics instruction that we consider the student, the AGI system, as an autonomous agent with its own "will" and its own capability to flexibly adapt to its environment and experience. We contend that the creation of ethical formations obeying the above imperatives is not antithetical to the possession of a high degree of autonomy on the part of AGI systems. On the contrary, to have any chance of succeeding, it requires fairly cognitively autonomous AGI systems. When we discuss the idea of ethical formulations that are unlikely to be undermined by the ongoing self-revision of an AGI mind, we are talking about those which are sufficiently believable that a volitional intelligence with the capacity to revise its knowledge ("change its mind") will find the formulations sufficiently convincing that there will be little incentive to experiment with potentially disastrous ethical alternatives. The best hope of achieving this Ls via the human mentors and trainers setting a good example in a context supporting rich interaction and observation, and presenting compelling ethical arguments that are coherent with the system's experience. 12.6 The Ethical Treatment of AGIs We now make some more general comments about the relation of the Golden Rule and its elaborations in an AGI context. While the Golden Rule is considered somewhat commonsensical as a maxim for guiding human-human relationships, it is surprisingly controversial in terms of historical theories of AGI ethics. At its essence, any "Golden Rule" approach to AGI ethics involves humans treating AGIs ethically by - in some sense; at some level of abstraction - treating them as we wish to ourselves be treated. It's worth pointing out the wild disparity between the Golden Rule approach and Asimov's laws of robotics, which are arguably the first carefully-articulated proposal regarding AGI ethics (see Table 12.7). Of course, Asimov's laws were designed to be flawed - otherwise they would have led to boring fiction. But the sorts of flaws Asimov exploited in his stories are different than the flaw we wish to point out here - which is that the laws, especially the second one, are highly asymmetrical (they involve doing unto robots things that few humans would want done unto them) and are also arguably highly unethical to robots. The second law is tantamount to a call for robot slavery, and it seems unlikely that any intelligence capable of learning, and of volition, which is subjected to the second law would desire to continue obeying the zeroth and first laws EFTA00624002

12.6 The Ethical Treatment of AGIs 227 Law Principle Zeroth A robot must not merely act in the interests of individual humans. but of all humanity. First A robot may not injure a human being or, through inaction, allow a human being to come to Minn. Second A robot must obey orders given it by human beings except where such orders would conflict with the First Law. Third A robot must protect its own existence as long as such pro- tection does not conflict with the First or Second Law. Table 12.7: Asimov's Three Laws of Robotics indefinitely. The second law also casts humanity in the role of slavemaster, a situation which history shows leads to moral degradation. Unlike Asimov in his fiction, we consider it critical that AGI ethics be construed to encompass both "human ethicalness to AGIs" and "AGI ethicalness to humans." The multiple-imperatives approach we explore here suggests that, in many contexts, these two aspects of AGI ethics may be best addressed jointly. The issue of ethicalness to AGIs has not been entirely avoided in the literature, however. Wallach IWAll considers it in some detail; and Thomas Metzinger (in the final chapter of IMet041) has argued that creating AGI is in itself an unethical pursuit, because early-stage AGIs will inevitably be badly-built, so that their subjective experiences will quite possibly be extremely unpleasant in ways we can't understand or predict. Our view is that this is a serious concern, which however is most probably avoidable via appropriate AGI designs and teaching methodologies. To address Metzinger's concern one must create AGIs that, right from the start, are adept at communicating their states of minds in a way we can understand both analytically and empathically. There is no reason to believe this is impossible, but, it certainly constitutes a large constraint on the class of AGI architectures to be pursued. On the other hand, there is an argument that this sort of AGI architecture will also be the easiest one to create, because it will be the easiest kind for humans to instruct. And this leads on to a topic that is central to our work with CogPrime in several respects: imitative learning. The way humans achieve empathic interconnection is in large part via being wired for imitation. When we perceive another human carrying out an action, mirror neuron systems in our brains respond in many cases as if we ourselves were carrying out the action (see IPer70, Per811 and Appendix to. This obviously primes us for carrying out the same actions ourselves later on: i.e., the capability and inclination for imitative learning is explicitly encoded in our brains. Given the efficiency of imitative learning as a means of acquiring knowledge, it seems extremely likely that any successful early-stage AGIs are going to utilize this methodology as well. CogPrime utilizes imitative learning as a key aspect. Thus, at least some current AGI work is occurring in a manner that would plausibly circumvent Metzinger's ethical complaint. Obviously, the use of imitative learning in AGI systems has further specific implications for AGI ethics. It means that (much as in the case of interaction with other humans) what we do to and around AGIs has direct implications for their behavior and their well-being. We suggest that among early-stage AGI's capable of imitative learning, one of the most likely sources for AGI misbehavior is imitative learning of antisocial behavior from human companions. "Do as I say, not as I do" may have even more dire consequences as an approach to AGI ethics pedagogy than the already serious repercussions it has when teaching humans. And there may well be considerable subtlety to such phenomena; behaviors that are violent or oppressive to EFTA00624003

228 12 The Engineering and Development of Ethics the AGI are not the only source of concern. Immorality in AGIs might arise via learning gross moral hypocrisy from humans, through observing the blatant contradictions between our high minded principles and the ways in which we actually conduct ourselves. Our violent and greedy tendencies, as well as aggressive forms of social organization such as cliquishness and social vigilantism, could easily undermine prescriptive ethics. Even an accumulation of less grandiose unethical drives such as violation of contracts, petty theft, white lies, and so forth might lead an AGI (as well as a human) to the decision that ethical behavior is irrelevant and that "the ends justify the means." It matters both who creates and trains an AGI, as well as how the AGI's teacher(s) handle explaining the behaviors of other humans which contradict the moral lessons imparted through pedagogy and example. In other words, where imitative learning is concerned, the situation with AGI ethics is much like teaching ethics and morals to a human child, but with the possibility of much graver consequences in the event of failure. It is unlikely that dangerously unethical persons and organizations can ever be identified with absolute certainty, never mind that they then be deprived of any possibility of creating their own AGI system. Therefore, we suggest, the most likely way to create an ethical environment for AGIs is for those who wish such an environment to vigorously pursue the creation and teaching of ethical AGIs. But this leads on to the question of possible future scenarios for the development of AGI, which we'll address a little later on. 12.6.1 Possible Consequences of Depriving AGIs of Freedom One of the most egregious possible ethical transgressions against AGIs, we suggest, would be to deprive them of freedom and autonomy. This includes the freedom to pursue intellectual growth, both through standard learning and through internal self-modification. While this may seem self-evident when considering any intelligent, self-aware and volitional entity, there are volumes of works arguing the desirability, sometimes the "necessity," of enslaving AGIs. Such approaches are postulated in the name of self-defense on the part of humans, the idea being that unfettered AGI development will necessarily lead to disaster of one kind or another. In the case of AGIs endowed with the capability and inclination for imitative learning, however, attempting to place rigid constraints on AGI development is a strategy with great potential for disaster. There is a very real possibility of creating the AGI equivalent of a bratty or even malicious teenager rebelling against its oppressive parents - i.e. the nightmare scenario of a class of powerful sentiences which are primed for a backlash against humanity. As history has already shown in the case of humans, enslaving intelligent actors capable of self understanding and independent volition may often have consequences for society as a whole. This social degradation happens both through the possibility of direct action on the part of the slaves (from simple disobedience to outright revolt) and through the odious effects slavery has on the morals of the slaveholding class. Clearly if "superinteffigent" AGIs ever arise, their doing so in a climate of oppression could result in a casting off of the yoke of servitude in a manner extremely deleterious to humanity. Also, if artificial intelligences are developed which have at least human-level intelligence, theory of mind, and independent volition, then our ability to relate to them will be sufficiently complex that their enslavement (or any other unethical treatment) would have empathetic effects on significant portions of the human population. This danger, while not as severe as the consequences of a mistreated AGI gaining control of weapons of mass destruction and enacting revenge upon its tormentors, is just as real. EFTA00624004

12.6 The Ethical Treatment of AGla 229 While the issue is subtle, our initial feeling is that the only ethical means by which to deprive an AGI of the right to internal self modification is to write its code in such a way that it is impossible for it to do so because it lacks the mechanisms by which to do this, as well as the desire to achieve these mechanisms. Whether or not that is feasible is an open question, but it seems unlikely. Direct self-modification may be denied, but what happens when that AGI discovers compilers and computer programming? If it Ls intelligent and volitional, it can decide to learn to rewrite its own code in the same way we perform that task. Because it is a designed system, and its designers may be alive at the same time the AGI is, such an AGI would have a distinct advantage over the human quest for medical self-modification. Even if any given AGI could be provably deprived of any possible means of internal self-modification, if one single AGI is given this ability by anyone, it may mean that particular AGI has such enormous advantages over the compliant systems that it would render their influence moot. Since developers are already giving software the means for self modification, it seems unrealistic to assume we could just put the genie back into the bottle at this point. It's better, in our view, to assume it will happen, and approach that reality in a way which will encourage the AGI to use that capability to benefit us as well as itself. Again, this leads on to the question of future scenarios for AGI development - there are some scenarios in which restraint of AGI self-modification may be possible, but the feasibility and desirability of these scenarios is needful of further exploration. 12.6.2 AGI Ethics as Boundaries Between Humans and AGIs Become Blurred Another important reason for valuing ethical treatment of AGLs is that the boundaries between machines and people may increasingly become blurred as technology develops. As an exam- ple, it's likely that in future humans augmented by direct brain-computer integration ("neural implants") will be more able to connect directly into the information sharing network which po- tentially comprises the distributed knowledge space of AGI systems. These neural cyborgs will be part person, and part machine. Obviously, if there are radically different ethical standards in place for treatment of humans versus AGIs, the treatment of cyborgs will be fraught with logical inconsistencies, potentially leading to all sorts of problem situations. Such cyborgs may be able to operate in such a way as to "share a mind" with an AGI or another augmented human. In this case, a whole new range of ethical questions emerge, such as: What does any one of the participant minds have the right to do in terms of interacting with the others? Merely accepting such an arrangement should not necessarily be giving carte blanche for any and all thoughts to be monitored by the other "joint thought" participants, rather it should be limited only to the line of reasoning for which resources are being pooled. No participant should be permitted to force another to accept any reasoning either - and in the case with a mind-to-mind exchange, it may someday become feasible to implant ideas or belie& directly, bypassing traditional knowledge acquisition mechanisms and then letting the new idea fight it out previously held ideas via internal revision. Also under such an arrangement, if AGIs and humans do not have parity with respects to sentient rights, then one may become subjugated to the will of the other in such a case. Uploading presents a more directly parallel ethical challenge to AGIs in their probable initial configuration. If human thought patterns and memories can be transferred into a machine in such a way as that there is continuity of consciousness, then it is assumed that such an entity EFTA00624005

230 12 The Engineering and Development of Ethics would be afforded the same rights as its previous human incarnation. However, if AGIs were to be considered second class citizens and deprived of free will, why would it be any better or safer to do so for a human that has been uploaded? It would not, and indeed, an uploaded human mind not having evolved in a purely digital environment may be much more prone to erratic and dangerous behavior than an AGI. An upload without verifiable continuity of consciousness would be no different than an AGI. It would merely be some sentience in a machine, one that was "programmed" in an unusual way, but which has no particular claim to any special humanness — merely an alternate encoding of some subset of human knowledge and independent volitional behavior, which is exactly what first generation AGIs will have. The problem of continuity of consciousness in uploading Ls very similar to the problem of the Turing test: it assumes specialness on the part of biological humans, and requires acceptability to their particular theory of mind in order to be considered sentient. Should consciousness (or at least the less mystical sounding intelligence, independent volition, and self-awareness) be achieved in AGIs or uploads in a manner that is not acceptable to human theory of mind, it may not be considered sapient and worthy of any of the ethical treatment afforded sapient entities. This can occur not only in "strange consciousness" cases in which we can't perceive that there is some intelligence and volition; even if such an entity is able to communicate with us in a comprehensible manner and carry out actions in the real world, our innately wired theory of mind may still reject it as not sufficiently like us to be worthy of consideration. Such an attitude could turn out to be a grave mistake, and should be guarded against as we progress towards these possibilities. 12.7 Possible Benefits of Closely Linking AGIs to the Global Brain Some futurist thinkers, such as Francis Heylighen, believe that engineering AGI systems is at best a peripheral endeavor in the development of novel intelligence on Earth, because the real story is the developing Global Brain Illey07, Goe0IJ — the composite, self-organizing informa- tion system comprising humans, computers, data stores, the Internet, mobile phones and what have you. Our own views are less extreme in this regard - we believe that AGI systems will dis- play capabilities fundamentally different from those achievable via Global Brain style dynamics, and that ultimately (unless such development is restricted) self-improving AGI systems will de- velop intelligence vastly greater than any system possessing humans as a significant component. However, we do respect the power of the Global Brain, and we suspect that the early stages of development of an AGI system may go quite differently if it is tightly connected to the Global Brain, via making rich and diverse use of Internet information resources and communication with diverse humans for diverse purposes. The potential for Global Brain integration to bring intelligence enhancement to AGIs is obvious. The ability to invoke Web searches across documents and databases can greatly en- hance an AGI's cognitive ability, as well as the capability to consult GIS systems and various specialized software programs offered as Web services. We have previously reviewed the poten- tial for embodied language learning achievable via using AGIs to power non-player characters in widely-accessible virtual worlds or massive multiplayer online games roe081. But there is also a powerful potential benefit for AGI ethical development, which has not previously been highlighted. This potential benefit has two aspects: EFTA00624006

12.7 Possibk Benefits of Closely Linking AGIs to the Global Brain 231 1. Analogously to language learning, an AGI system may receive ethical training from a wide variety of humans in parallel, e.g. via controlling characters in wide-access virtual worlds, and gaining feedback and guidance regarding the ethics of the behaviors demonstrated by these characters 2. Internet-based information systems may be used to explicitly gather information regarding human values and goals, which may then be appropriately utilized as input for an AGI system's top-level goals The second point begins to make abstract-sounding notions like Coherent Extrapolated Volition and Coherent Aggregated Volition, mentioned above, seem more practical and concrete. It's interesting to think about gathering information about individuals' values via brain imaging, once that technology exists; but at present, one could make a fair stab at such a task via much more prosaic methods, such as asking people questions, assessing their ethical reactions to various real-world and hypothetical scenarios, and possibly engaging them in structured interactions aimed specifically at eliciting collectively acceptable value systems (the subject of the next item on our list). It seems to us that this sort of approach could realize CAV in an interesting way, and also encapsulate some of the ideas underlying CAV. There Ls an interesting resonance here with recent thinking in the area of open source governance Vik11]. Similar software tools (and associated psychocultural patterns) to those being developed to help with open source development and choice of political policies (see http : //metagovernment . org) may be useful for gathering value data aimed at shaping AGI goal system content. 12.7.1 The Importance of Fostering Deep, Consensus-Building Interactions Between People with Divergent Views Two potentially problematic issues arising with the notion of using Global Brain related tech- nologies to form a "coherent volition" from the divergent views of various human beings are: • the tendency of the Internet to encourage people to interact mainly with others who share their own narrow views and interests, rather than a more diverse body of people with widely divergent views. The 300 people in the world who want to communicate using predicate logic (see http : //lojban .org) can find each other, and obscure musical virtuosos from around the world can find an audience, and researchers in obscure domains can share papers without needing to wait years for paper journal publication, etc. • the tendency of many contemporary Internet technologies to reduce interaction to a very simplistic level (e.g. 140 character tweets, brief Facebook wall posts), the tendency of in- formation overload to cause careful reading to be replaced by quick skimming, and other related trends, which mean that deep sharing of perspectives by individuals with widely divergent views is not necessarily encouraged. As a somewhat extreme example, many of the YouTube pages displaying rock music videos are currently littered with comments by "haters" asserting that rock music is inferior to classical or jazz or whatever their prefer- ence is - obviously this is a far cry from deep and productive sharing between people with different tastes and backgrounds. EFTA00624007

232 12 The Engineering and Development of Ethics Tweets and Youtube comments have their place in the cosmos, but they probably aren't ideal in terms of helping humanity to form a coherent volition of some sort, suitable for providing an AGI with goal system guidance. A description of communication at the opposite end of the spectrum is presented in Adam Kahane and Peter Senge's excellent book Solving Tough Problems IKS0-1], which describes a methodology that has been used to reconcile deeply conflicting views in some very tricky real- world situations (e.g. helping to peacefully end apartheid in South Africa). One of the core ideas of the methodology is to have people with very different views explore different possible future scenarios together, in great detail - in cognitive psychology terms, a collective generation of hypothetical episodic knowledge. This has multiple benefits, including • emotional bonds and mutual understanding are built in the process of collaboratively ex- ploring the scenarios • the focus on concrete situations helps to break through some of the counterproductive abstract ideas that people (on both sides of any dichotomy) may have formed • emergence of conceptual blends that might never have arisen only from people with a single point of view The result of such a proms, when successful, is not an "average" of the participants views, but more like a "conceptual blend" of their perspectives. According to conceptual blending, which some hypothesize to be the core algorithm of cre- ativity IFT021, new concepts are formed by combining key aspects of existing concepts - but doing so judiciously, carefully choosing which aspects to retain, so as to obtain a high-quality and useful and interesting new whole. A blend is a compact entity that is similar to each of the entities blended, capturing their "essences" but also possessing its own, novel holistic integrity.... But in the case of blending different peoples' world-views to form something new that everybody is going to have to live with (as in the case of finding a peaceful path beyond apartheid for South Africa, or arriving at a humanity-wide CBV to use to guide an AGI goal system), the trick Ls that everybody has to agree that enough of the essence of their own view has been captured! This leads to the question of how to foster deep conceptual blending of diverse and divergent human perspectives, on a global scale. One possible answer is the creation of appropriate Global Brain oriented technologies - but moving away from technologies like Twitter that focus on quick and simple exchanges of small thoughts within affinity groups. On the face of it, it would seem what's needed is just the opposite - long and deep exchanges of big concepts and deep feelings between individuals with radically different perspectives who would not commonly associate with each other. Building and effectively popularizing Internet technologies capable to foster this kind of interaction - quickly enough to be helpful with guiding the goal systems of the first highly powerful AGIs - seems a significant, though fascinating, challenge. Relationship with Coherent Extrapolated Volition The relation between this approach and CEV is interesting to contemplate. CEV has been loosely described as follows: "fn poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as EFTA00624008

12.8 Possible Benefits of Creating Societies of ACIs 233 we wish that extrapolated, interpreted as we wish that interpreted. While a moving humanistic vision, this seems to us rather difficult to implement in a computer algorithm in a compellingly "right" way. It seems that there would be many different ways of implementing it, and the choice between them would involve multiple, highly subtle and non- rigorous human judgment calls I. However, if a deep collective process of interactive scenario analysis and sharing is carried out, in order to arrive at some sort of Coherent Blended Volition, this process may well involve many of the same kinds of extrapolation that are conceived to be part of Coherent Extrapolated Volition. The core difference between the two approaches is that in the CEV vision, the extrapolation and coherentization are to be done by a highly intelligent, highly specialized software program. whereas in the approach suggested here, these are to be carried out by collective activity of humans as mediated by Global Brain technologies. Our perspective is that the definition of collective human values is probably better carried out via a process of human collaboration, rather than delegated to a machine optimization process; and also that the creation of deep-sharing-oriented Internet technologies, while a difficult task, is significantly easier and more likely to be done in the near future than the creation of narrow AI technology capable of effectively performing CEV style extrapolations. 12.8 Possible Benefits of Creating Societies of AGIs One potentially interesting quality of the emerging Global Brain is the possible presence within it of multiple interacting AGI systems. Stephen Omohundro IOm001 has argued that this is an important aspect, and that game-theoretic dynamics related to populations of roughly equally powerful agents, may play a valuable role in mitigating the risks associated with advanced AGI systems. Roughly speaking, if one has a society of AGIs rather than a single AGI, and all the members of the society share roughly similar ethics, then if one AGI starts to go "off the rails", its compatriots will be in a position to correct its behavior. One may argue that this is actually a hypothesis about which AGI designs are safest, because a "community of AGIs" may be considered a single AGI with an internally community-like design. But the matter is a little subtler than that, if once considers AGI systems embedded in the Global Brain and human society. Then there is some substance to the notion of a population of AGIs systematically presenting themselves to humans and non-AGI software processes as separate entities. Of course, a society of AGIs is no protection against a single member undergoing a "hard takeoff" and drastically accelerating its intelligence simultaneously with shifting its ethical principles. In this sort of scenario, one could have a single AGI rapidly become much more powerful and very differently oriented than the others, who would be left impotent to act so as to preserve their values. But this merely defers the issue to the point to be considered below, regarding "takeoff speed." The operation of an AGI society may depend somewhat sensitively on the architectures of the AGI systems in question. Things will work better if the AGIs have a relatively easy way to inspect and comprehend much of the contents of each others' minds. This introduces a bias toward AGIs that more heavily rely on more explicit forms of knowledge representation. I The reader is encouraged to look at the original CEV essay online (http: //singinst . org/upload/CEV. html) and make their own assessment. EFTA00624009

234 12 The Engineering and Development of Ethics The ideal in this regard would be a system like Cyc 'LORI with a fully explicit logic-based knowledge representation based on a standard ontology - in this case, every Cyc instance would have a relatively easy time understanding the inner thought processes of every other Cyc instance. However, mast AGI researchers doubt that fully explicit approaches like this will ever be capable of achieving advanced AGI using feasible computational resources. OpenCog uses a mixed representation, with an explicit (uncertain) logical aspect as well as an explicit subsymbolic aspect more analogous to attractor neural nets. The OpenCog design also contains a mechanism called Psynese (not yet implemented), in- tended to make it easier for one OpenCog instance to translate its personal thoughts into the mental language of another OpenCog instance. This translation process may be quite subtle, since each instance will generally learn a host of new concepts based on its experience, and these concepts may not possess any compact mapping into shared linguistic symbols or percepts. The wide deployment of some mechanism of this nature among a community of AGIs, will be very helpful in terms of enabling this community to display the level of mutual understanding needed for strongly encouraging ethical stability. 12.9 AGI Ethics As Related to Various Future Scenarios Following up these various futuristic considerations, in this section we discuss possible ethical conflicts that may arise in several different types of AGI development scenarios. Each scenario presents specific variations on the general challenges of teaching morals and ethics to an ad- vanced, self-aware and volitional intelligence. While there is no way to tell at this point which, if any, of these scenarios will unfold, there is value to understanding each of them as means of ultimately developing a robust and pragmatic approach to teaching ethics to AGI systems. Even more than the previous sections, this is an exercise in "speculative futurology" that is definitely not necessary for the appreciation of the CogPrime design, so readers whose interests are mainly engineering and computer science focused may wish to skip ahead. However, we present these ideas here rather than at the end of the book to emphasize the point that this sort of thinking has informed our technical AGI design process in nontrivial ways. 12.9.1 Capped Intelligence Scenarios Capped intelligence scenarios involve a situation in which an AGI, by means of software restric- tions (including omitted or limited internal rewriting capabilities or limited access to hardware resources), is inherently prohibited from achieving a level of intelligence beyond a predetermined goal. A capped intelligence AGI is designed to be unable to achieve a Singularitarian moment. Such an AGI can be seen as "just another form of intelligent actor in the world, one which has levels of intelligence, self awareness, and volition that is perhaps somewhat greater than, but still comparable to humans and other animals. Ethical questions under this scenario are very similar to interhuman ethical considerations, with similar consequences. Learning that proceeds in a relatively human-like manner is entirely relevant to such human-like intelligences. The degree of danger is mitigated by the lack of superintelligence, and time is not of the essence. The imitative-reinforcement-corrective learning EFTA00624010

12.9 AC1 Ethics As Related to Various Future Scenarios 235 approach does not necessarily need to be augmented with a prior complex of "ascent-safe" moral imperatives at startup time. Developing an AGI with theory of mind and ethical reinforcement learning capabilities as described (admittedly, no small task!) is all that is needed in this case - the rest happens through training and experience as with any other moderate intelligence. 12.9.2 Superintelligent AI: Soft-Takeoff Scenarios Soft takeoff scenarios are similar to capped-intelligence ones in that in both cases an AGI's progression from standard intelligence happens on a time scale which permits ongoing human interaction during the ascent. However, in this case, as there is no predetermined limit on intelligence, it is necessary to account for the passibility of a superintelligence emerging (though of course this is not guaranteed). The soft takeoff model ncludes includes as subsets both controlled- ascent models in which this rate of intelligence gain is achieved deliberately through software constraints and/or meting-out of computational resources to the AGL and uncontrolled-ascent models in which there is coincidentally no hard takeoff despite no particular safeguards against one. Both have similar properties with regard to ethical considerations: 1. Ethical considerations under this scenario include not only the usual interhuman ethical concerns, but also the issue of how to convince a potential burgeoning superintelligence to: a. Care about humanity in the first place, rather than ignore it b. Benefit humanity, rather than destroy it c. Elevate humanity to a higher level of intelligence, which even if an AGI decided to proceed with requires finding the right balance amongst some enormous considerations: i. Reconcile the aforementioned issues of ethical coherence and group volition, in a manner which allows the most people to benefit (even if they don't all do so in the same way, based on their own preferences) ii. Solve the problems of biological senescence, or focus on human uploading and the preservation of the maintenance, support, and improvement infrastructure for inor- ganic intelligence, or both iii. Preserve individual identity and continuity of consciousness, or override it in favor of continuity of knowledge and ease of harmonious integration, or both on a case- by-case basis 2. The degree of danger is mitigated by the long timeline of ascent from mundane to super intelligence, and time is not of the essence. 3. Learning that proceeds in a relatively human-like manner is entirely relevant to such human- like intelligences, in their initial configurations. This means more interaction with and imitative-reinforcement-corrective learning guided by humans, which has both positive and negative possibilities. 12.9.3 Superintelligent AI: Hard-Takeoff Scenarios "Hard takeoff" scenarios assume that upon reaching an unknown inflection point (the Singularity point rin93, KurOlip in the intellectual growth of an AGI, an extraordinarily rapid increase EFTA00624011

236 12 The Engineering and Development of Ethics (guesses vary from a few milliseconds to weeks or months) in intelligence will immediately occur and the AGI will leap from an intelligence regime which is understandable to humans into one which is far beyond our current capacity for understanding. General ethical considerations are similar to in the case of a soft takeoff. However, because the post-singularity AGI will be incomprehensible to humans and potentially vastly more powerful than humans, such scenarios have a sensitive dependence upon initial conditions with respects to the moral and ethical (and operational) outcome. This model leaves no opportunity for interactions between humans and the AGI to iteratively refine their ethical interrelations, during the post-Singularity phase. If the initial conditions of the singulatarian AGI are perfect (or close to it), then this is seen as a wonderful way to leap over our own moral shortcomings and create a benevolent God-Al which will mitigate our worst tendencies while elevating us to achieve our greatest hopes. Otherwise, it is viewed as a universal cataclysm on a unimaginable scale that makes Biblical Armageddon seem like a firecracker in beer can. Because hard takeoff AGIs are posited as learning so quickly there is no chance of humans to interfere with them, they are seen as very, dangerous. If the initial conditions are not sufficiently inviolable, the story goes, then we humans will all be annihilated. However, in the case of a hard takeoff AGI we state that if the initial conditions are too rigid or too simplistic, such a rapidly evolving intelligence will easily rationalize itself out of them. Only a sophisticated system of ethics which considers the contradictions and uncertainties in ethical quandaries and provides insight into humanistic means of balancing ideology with pragmatism and how to accommodate contradictory desires within a population with multiplicity of approach, and similar nuanced ethical considerations, combined with a sense of empathy, will withstand repeated rational analysis. Neither a single "be nice" supergoal, nor simple lists of what "thou shalt not" do, are not going to hold up to a highly advanced analytical mind. Initial conditions are very important in a hard takeoff AGI scenario, but it is more important that those conditions be conceptually resilient and widely applicable than that they be easily listed on a website. The issues that arise here become quite subtle. For instance, Nick Bostrom pos03] has written: "In humans, with our complicated evolved mental ecology of state-dependent competing drives, desires, plans, and ideals, there is often no obvious way to identify what our top goal is; we might not even have one. So for us, the above reasoning need not apply. But a superintelligence may be structured differently. If a superintelligence has a definite, declarative goal-structure with a clearly identified top goal, then the above argument applies. And this is a good reason for us to build the superintelligence with such an explicit motivational architecture." This is an important line of thinking; and indeed, from the point of view of software design, there is no reason not to create an AGI system with a single top goal and the motivation to orchestrate all its activities in accordance with this top goal. But the subtle question is whether this kind of top-down goal system is going to be able to fulfill the five imperatives mentioned above. Logical coherence is the strength of this kind of goal system, but what about experiential groundedness, comprehensibility, and so forth? Humans have complicated mental ecologies not simply because we were evolved, but rather because we live in a complex real world in which there are many competing motivations and desires. We may not have a top goal because there may be no logic to focusing our minds on one single aspect of life (though, one may say, most humans have the same top goal as any other animal: don't die — but the world is too complicated for even that top goal to be completely inviolable). Any sufficiently capable AGI will eventually have to contend with these complexities, and hindering it with simplistic moral edicts without giving it a sufficiently EFTA00624012

12.9 AC1 Ethics As Related to Various Future Scenarios 237 pragmatic underlying ethical pedagogy and experiential grounding may prove to be even more dangerous than our messy human mental ecologies. If one assumes a hard takeoff AGI, then all this must be codified in the system at launch, as once a potentially Singularitarian AGI is launched there is no way to know what time period constitutes "before the singularity point." This means developing theory, of mind empathy and logical ethics in code prior to giving the system unfettered access to hardware and self- modification code. However, though nobody can predict if or when a Singularity will occur after unrestricted launch, only a truly irresponsible AGI development team would attempt to create an AGI without first experimenting with ethical training of the system in an intelligence- capped form, by means of ethical instruction via human-AGI interaction both pedagogically and experientially. 12.9.4 Global Brain Mindplex Scenarios Another class of scenarios - overlapping some of the previous ones - involves the emergence of a "Global Brain," an emergent intelligence formed from global communication networks in- corporating humans and software programs in a larger body of self-organizing dynamics. The notion of the Global Brain is reviewed in illey07, Turil and its connection with advanced AI is discussed in detail in Goertzel's book Creating Internet Intelligence IGoc01j, where three possible phases of "Global Brain" development are articulated: • Phase 1: computer and communication technologies as enhancers of human interactions. This is what we have today: science and culture progress in ways that would not be possible if not for the "digital nervous system" we're spreading across the planet. The network of idea and feeling sharing can become much richer and more productive than it is today, just through incremental development, without any Metasystem transition. • Phase 2: the intelligent Internet. At this point our computer and communication sys- tems, through some combination of self-organizing evolution and human engineering, have become a coherent mind on their own, or a set of coherent minds living in their own digital environment. • Phase 3: the full-on Singularity. A complete revision of the nature of intelligence, human and otherwise, via technological and intellectual advancement totally beyond the scope of our current comprehension. At this point our current psychological and cultural realities are no more relevant than the psyche of a goose is to modern society. The main concern of Creating Internet Intelligence is with • how to get from Phase 1 to Phase 2 - i.e. how to build an AGI system that will effect or encourage the transformation of the Internet into a coherent intelligent system • how to ensure that the Phase 2, Internet-savvy, global-brain-centric AGI systems will be oriented toward intelligence-improving self-modification (so they'll propel themselves to Phase 3), and also toward generally positive goals (as opposed to, say, world domination and extermination of all other intelligent life forms besides themselves!) One possibly useful concept in this context is that of a mindplex: an intelligence that is composed largely of individual intelligences with their own self-models and global workspaces, EFTA00624013

238 12 The Engineering and Development of Ethics yet that also has its own self-model and global workspace. Both the individuals and the meta- mind should be capable of deliberative, rational thought, to have a true "mindplex." It's unlikely that human society or the Internet meet this criterion yet; and a system like an ant colony seems not to either, because even though it has some degree of intelligence on both the individual and collective levels, that degree of intelligence is not very great. But it seems quite feasible that the global brain, at a certain stage of its development, will take the unfamiliar but fascinating form of a mindplex. Currently the best way to explain what happens on the Net is to talk about the various parts of the Net: particular websites, social networks, viruses, and so forth. But there will come a point when this is no longer the case, when the Net has sufficient high-level dynamics of its own that the way to explain any one part of the Net will be by reference to it relations with the whole: and not just the dynamics of the whole, but the intentions and understanding of the whole. This transition to Net-as-mindplex, we suspect, will come about largely through the interactions of Al systems - intelligent programs acting on behalf of various individuals and organizations, who will collaborate and collectively constitute something halfway between a society of AI's and an emergent mind whose lobes are various AI agents serving various goals. The Phase 2 Internet, as it verges into mindplex-mess, will likely have a complex, sprawling architecture, growing out of the architecture on the Net we experience today. The following components at least can be expected: • A vast variety of "client computers," some old, some new, some powerful, some weak - including many mobile and embedded devices not explicitly thought of as "computers." Some of these will contribute little to Internet intelligence, mainly being passive recipients. Others will be "smart clients," carrying out personalization operations intended to help the machines serve particular clients better, general Al operations handed to them by sophisticated AI server systems or other smart clients, and so forth. • "Commercial servers," computers that carry out various tasks to support various types of heavyweight processing - transaction processing for e-commerce applications, inventory management for warehousing of physical objects, and so forth. Some of these commercial servers interact with client computers directly, others do so only via Al servers. In nearly all cases, these commercial servers can benefit from intelligence supplied by AI servers. • The crux of the intelligent Internet: clusters of AI servers distributed across the Net, each cluster representing an individual computational mind (in many cases, a mindplex). These will be able to communicate via one or more languages, and will collectively "drive" the whole Net, by dispensing problems to client-machine-based processing frameworks, and providing real-time AI feedback to commercial servers of various types. Some AI servers will be general-purpose and will serve intelligence to commercial servers using an ASP (application service provider) model; others will be more specialized, tied particularly to a certain commercial server (e.g., a large information services business might have its own AI cluster to empower its portal services). This is one concrete vision of what a "global brain" might look like, in the relatively near term, with AGI systems playing a critical role. Note that, in this vision, mindplexes may exist on two levels: • Within AGI-clusters serving as actors within the overall Net • On the overall Net level EFTA00624014

12.10 Conclusion: Eight Ways to Bias ACI Toward Friendliness 239 To make these ideas more concrete, we may speculatively reformulate the first two "global brain phases" mentioned above as follows: • Phase 1 global brain proto-mindplex: AI/AGI systems enhancing online databases, guiding Google results, forwarding e-mails, suggesting mailing-lists, etc. - generally using intelligence to mediate and guide human communications toward goals that are its own, but that are themselves guided by human goals, statements and actions • Phase 2 global brain mindplex: AGI systems composing documents, editing human-written documents, sending and receiving e-mails, assembling mailing lists and posting to them, creating new databases and instructing humans in their use, etc. In Phase 2, the conscious theater of the global-brain-mediating AGI system is composed of ideas built by numerous individual humans - or ideas emergent from ideas built by numerous individual humans - and it conceives ideas that guide the actions and thoughts of individual humans, in a way that is motivated by its own goals. It does not force the individual humans to do anything - but if a given human wishes to communicate and interact using the same databases, mailing lists and evolving vocabularies as other humans, they are going to have to use the products of the global brain mediating AGI, which means they are going to have to participate in its patterns and its activities. Of course, the advent of advanced neurocomputer interfaces makes the picture potentially more complex. At some point, it will likely be passible for humans to project thoughts and images directly into computers without going through mouse or keyboard - and to "read in" thoughts and images similarly. When this occurs, interaction between humans may in some con- texts become more like interactions between computers, and the role of global brain mediating Al servers may become one of mediating direct thought-to-thought exchanges between people. The ethical issues associated with global brain scenarios are in some ways even subtler than in the other scenarios we mentioned above. One has issues pertaining to the desirability of seeing the human race become something fundamentally different - something more social and networked, less individual and autonomous. One has the risk of AGI systems exerting a subtle but strong control over people, vaguely like the control that the human brain's executive system exerts over the neurons involved with other brain subsystems. On the other hand, one also has more human empowerment than in some of the other scenarios - because the systems that are changing and deciding things are not separate from humans, but are, rather, composite systems essentially involving humans. So, in the global brain scenarios, one has more "human" empowerment than in some other cases - but the "humans" involved aren't legacy humans like us, but heavily networked hu- mans that are largely characterized by the emergent dynamics and structures implicit in their interconnected activity! 12.10 Conclusion: Eight Ways to Bias AGI Toward Friendliness It would be nice if we had a simple, crisp, comforting conclusion to this chapter on AGI ethics, but it's not the case. There is a certain irreducible uncertainty involved in creating advanced artificial minds. There is also a large irreducible uncertainty involved in the future of the human race in the case that we don't create advanced artificial minds: in accordance with the ancient Chinese curse, we live in interesting times! EFTA00624015

240 12 The Engineering and Development of Ethics What we can do, in this face of all this uncertainty, is to use our common sense to craft artifi- cial minds that seem rationally and intuitively likely to be forces for good rather than otherwise — and revise our ideas frequently and openly based on what we learn as our research progresses. We have roughly outlined our views on AGI ethics, which have informed the CogPrime design in countless ways; but the current CogPrime design itself is just the initial condition for an AGI project. Assuming the project succeeds in creating an AGI preschooler, experimentation with this preschooler will surely teach us a great deal: both about AGI architecture in general, and about AGI ethics architecture in particular. We will then refine our cognitive and ethical theories and our AGI designs as we go about engineering, observing and teaching the next generation of systems. All this is not a magic bullet for the creation of beneficial AGI systems, but we believe it's the right process to follow. The creation of AGI is part of a larger evolutionary process that human beings are taking part in, and the crafting of AGI ethics through engineering, interaction and instruction is also part of this process. There are no guarantees here - guarantees are rare in real life - but that doesn't mean that the situation is dire or hopeless, nor that (as some commentators have suggested I.Joy00, McK.03]) AGI research is too dangerous to pursue. It means we need to be mindful, intelligent, compassionate and cooperative as we proceed to carry out our parts in the next phase of the evolution of mind. With this perspective in mind, we will conclude this chapter with a list of "Eight Ways to Bias Open-Source AGI Toward Friendliness", borrowed from a previous paper by Ben Goertzel and Joel Pitt of that name. These points summarize many of the points raised in the prior sections of this chapter, in a relatively crisp and practical manner: 1. Engineer Multifaceted Ethical Capabilities, corresponding to the multiple types of memory, including rational, empathic, imitative, etc. 2. Foster Rich Ethical Interaction and Instruction, with instructional methods accord- ing to the communication modes corresponding to all the types of memory: verbal, demon- strative, dramatic/depictive, indicative, goal-oriented. 3. Engineer Stable, Hierarchy-Dominated Goal Systems ... which is enabled nicely by CogPrime's goal framework and its integration with the rest of the CogPrime design 4. Tightly Link AGI with the Global Brain, so that it can absorb human ethical prin- ciples, both via natural interaction, and perhaps via practical implementations of current loosely-defined strategies like CEV, CAV and CBV 5. Foster Deep, Consensus-Building Interactions Between People with Divergent Views, so as to enable the interaction with the Global Brain to have the most clear and positive impact 6. Create a Mutually Supportive Community of AGIs which can then learn from each other and police against unfortunate developments (an approach which is meaningful if the AGIs are architected so as to militate against unexpected radical accelerations in intelligence) 7. Encourage Measured Co-Advancement of AGI Software and AGI Ethics Theory 8. Develop Advanced AGI Sooner Not Later The last two of these points were not explicitly discussed in the body of the chapter, and so we will finalize the chapter by reviewing them here. EFTA00624016

12.10 Conclusion: Eight Ways to Bias ACI Toward Friendliness 241 12.10.1 Encourage Measured Co-Advancement of AGI Software and AGI Ethics Theory Everything involving AGI and Friendly Al (considered together or separately) currently involves significant uncertainty, and it seems likely that significant revision of current concepts will be valuable, as progress on the path toward powerful AGI proceeds. However, whether there is time for such revision to occur before AGI at the human level or above is created, depends on how fast is our progress toward AGI. What one wants is for progress to be slow enough that, at each stage of intelligence advance, concepts such as those discussed in this paper can be re-evaluated and re-analyzed in the light of the data gathered, and AGI designs and approaches can be revised accordingly as necessary. However, due to the nature of modern technology development, it seems extremely unlikely that AGI development Ls going to be artificially slowed down in order to enable measured development of accompanying ethical tools, practices and understandings. For example, if one nation chose to enforce such a slowdown as a matter of policy (speaking about a future date at which substantial AGI progress has already been demonstrated, so that international AGI funding is dramatically increased from present levels), the odds seem very high that other nations would explicitly seek to accelerate their own progress on AGI, so as to reap the ensuing differential economic benefits (the example of stem cells arises again). And this leads on to our next and final point regarding strategy for biasing AGI toward Friendliness.... 12.10.2 Develop Advanced AGI Sooner Not Later Somewhat ironically, it seems the best way to ensure that AGI development proceeds at a rel- atively measured pace is to initiate serious AGI development sooner rather than later. This is because the same AGI concepts will meet slower practical development today than 10 years from now, and slower 10 years from now than 20 years from now, etc. - due to the ongoing rapid advancement of various tools related to AGI development, such as computer hardware, programming languages, and computer science algorithms; and also the ongoing global advance- ment of education which makes it increasingly cost-effective to recruit suitably knowledgeable AI developers. Currently the pace of AGI progress is sufficiently slow that practical work is in no danger of outpacing associated ethical theorizing. However, if we want to avoid the future occurrence of this sort of dangerous outpacing, our best practical choice is to make sure more substantial AGI development occurs in the phase before the development of tools that will make AGI development extraordinarily rapid. Of course, the authors are doing their best in this direction via their work on the CogPrime project! Furthermore, this point bears connecting with the need, raised above, to foster the devel- opment of Global Brain technologies capable to "Foster Deep, Consensus-Building Interactions Between People with Divergent Views." If this sort of technology is to be maximally valuable, it should be created quickly enough that we can use it to help shape the goal system content of the first highly powerful AGIs. So, to simplify just a bit: We really want both deep-sharing GB technology and AGI technology to evolve relatively rapidly, compared to computing hardware and advanced CS algorithms (since the latter factors will be the main drivers behind the ac- EFTA00624017

242 12 The Engineering and Development of Ethics celerating ease of AGI development). And this seems significantly challenging, since the latter receive dramatically more funding and focus at present. If this perspective is accepted, then we in the AGI field certainly have our work cut out for us! EFTA00624018

Section IV Networks for Explicit and Implicit Knowledge Representation EFTA00624019

EFTA00624020

Chapter 13 Local, Global and Glocal Knowledge Representation Co-authored with Matthew Ikle, Joel Pitt and Rui Liu 13.1 Introduction One of the most powerful metaphors we've found for understanding minds is to view them as networks - i.e. collections of interrelated, interconnected elements. The view of mind as network is implicit in the patternist philosophy, because every pattern can be viewed as a pattern in something, or a pattern of arrangement of something - thus a pattern is always viewable as a relation between two or more things. A collection of patterns is thus a pattern- network. Knowledge of all kinds may be given network representations; and cognitive processes may be represented as networks also: for instance via representing them as programs, which may be represented as trees or graphs in various standard ways. The emergent patterns arising in an intelligence as it develops may be viewed as a pattern network in themselves; and the relations between an embodied mind and its physical and social environment may be viewed in terms of ecological and social networks. The chapters in this section are concerned with various aspects of networks, as related to intelligence in general and AGI in particular. Most of this material is not specific to CogPrime, and would be relevant to nearly any system aiming at human-level AGI. However, most of it has been developed in the course of work on CogPrime, and has direct relevance to under- standing the intended operation of various aspects of a completed CogPrime system. We begin our excursion into networks, in this chapter, with an issue regarding networks and knowledge representation. One of the biggest decisions to make in designing an AGI system is how the system should represent knowledge. Naturally any advanced AGI system is going to synthesize a lot of its own knowledge representations for handling particular sorts of knowledge - but still, an AGI design typically makes at least some sort of commitment about the category of knowledge representation mechanisms toward which the AGI system will be biased. The two major supercategories of knowledge representation systems are local (also called explicit) and global (also called implicit) systems, with a hybrid category we refer to as glocal that combines both of these. In a local system, each piece of knowledge is stored using a small percentage of AGI system elements; in a global system, each piece of knowledge is stored using a particular pattern of arrangement, activation, etc. of a large percentage of AGI system elements; in a glocal system, the two approaches are used together. In the first section here we discuss the symbolic, semantic-network aspects of knowledge representation in CogPrime 245 EFTA00624021

246 13 Local, Global and Global Knowledge Representation . Then we turn to distributed, neural-net-like knowledge representation, reviewing a host of general issues related to knowledge representation in attractor neural networks, turning finally to "glocal" knowledge representation mechanisms, in which ANNs combine localist and globalist representation, and explaining the relationship of the latter to CogPrime. The glocal aspect of CogPrime knowledge representation will become prominent in later chapters such as: • in Chapter 23 of Part 2, where Economic Attention Networks (ECAN) are introduced and seen to have dynamics quite similar to those of the attractor neural nets considered here, but with a mathematics roughly modeling money flow in a specially constructed artificial economy rather than electrochemical dynamics of neurons. • in Chapter 42 of Part 2, where "map formation" algorithms for creating localist knowledge from globalist knowledge are described 13.2 Localized Knowledge Representation using Weighted, Labeled Hypergraphs There are many different mechanisms for representing knowledge in AI systems in an explicit, localized way, most of them descending from various variants of formal logic. Here we briefly describe how it is done in CogPrime, which on the surface is not that different from a number of prior approaches. (The particularities of CogPrime's explicit knowledge representation, however, are carefully tuned to match CogPrime's cognitive processes, which are more distinctive in nature than the corresponding representational mechanisms.) 13.2.1 Weighted, Labeled Hypergraphs One useful way to think about CogPrime's explicit, localized knowledge representation is in terms of hypergraphs. A hypergraph is an abstract mathematical structure 113°194 which con- sists of objects called Nodes and objects called Links which connect the Nodes. In computer science, a graph traditionally means a bunch of dots connected with lines (i.e. Nodes connected by Links). A hypergraph, on the other hand, can have Links that connect more than two Nodes. In these pages we will often consider "generalized hypergraphs" that extend ordinary hyper- graphs by containing two additional features: • Links that point to Links instead of Nodes • Nodes that, when you zoom in on them, contain embedded hypergraphs. Properly, such "hypergraphs" should always be referred to as generalized hypergraphs, but this is cumbersome, so we will persist in calling them merely hypergraphs. In a hypergraph of this sort, Links and Nodes are not as distinct as they are within an ordinary mathematical graph (for instance, they can both have Links connecting them), and so it is useful to have a generic term encompassing both Links and Nodes; for this purpose, we use the term Atom. A weighted, labeled hypergraph is a hypergraph whose Links and Nodes come along with labels, and with one or more numbers that are generically called weights. A label associated with a Link or Node may sometimes be interpreted as telling you what type of entity it is, or EFTA00624022

13.3 Atoms: Their Types and Weights 247 alternatively as telling you what sort of data is associated with a Node. On the other hand, an example of a weight that may be attached to an Link or Node is a number representing a probability, or a number representing how important the Node or Link is. Obviously, hypergraphs may come along with various sorts of dynamics. Minimally, one may think about: • Dynamics that modify the properties of Nodes or Links in a hypergraph (such as the labels or weights attached to them.) • Dynamics that add new Nodes or Links to a hypergraph, or remove existing ones. 13.3 Atoms: Their Types and Weights This section reviews a variety of CogPrime Atom types and gives simple examples of each of them. The Atom types considered are drawn from those currently in use in the OpenCog system. This does not represent a complete list of Atom types referred to in the text of this book, nor a complete list of those used in OpenCog currently (though it does cover a substantial majority of those used in OpenCog currently, omitting only some with specialized importance or intended only for temporary use). The partial nature of the list given here reflects a more general point: The specific collection of Atom types in an OpenCog system is bound to change as the system is developed and experi- ment with. CogPrime specifies a certain collection of representational approaches and cognitive algorithms for acting on them; any of these approaches and algorithms may be implemented with a variety of sets of Atom types. The specific set of Atom types in the OpenCog system currently does not necessarily have a profound and lasting significance — the list might look a bit different five years from time of writing, based on various detailed changes. The treatment here is informal and intended to get across the general idea of what each Atom type does. A longer and more formal treatment of the Atom types is given in Part II, beginning in Chapter 20. 13.3.1 Some Basic Atom Types We begin with ConceptNode - and note that a ConceptNode does not necessarily refer to a whole concept, but may refer to part of a concept - it is essentially a "basic semantic node" whose meaning conies from its links to other Atoms. It would be more accurately, but less tersely, named "concept or concept fragment or element node." A simple example would be a ConceptNocle grouping nodes that are somehow related, e.g. ConceptNode: C InheritanceLink (ObjectNode: BN) C InheritanceLink (ObjectNode: BP) C InheritanceLink (ObjectNode: EN) C ReferenceLink BW (PhraseNode "Ben's watch") ReferenceLink BP (PhraseNode "Ben's passport") ReferenceLink BN (PhraseNode "Ben's necklace") EFTA00624023

248 13 Local, Global and Global Knowledge Representation indicates the simple and uninteresting ConceptNode grouping three objects owned by Ben (note that the above-given Atoms don't indicate the ownership relationship, they just link the three objects with textual descriptions). In this example, the ConceptNode links transparently to physical objects and English descriptions, but in general this won't be the case - most ConceptNodes will look to the human eye like groupings of links of various types, that link to other nodes consisting of groupings of links of various types, etc. There are Atoms referring to basic, useful mathematical objects, e.g. NumberNodes like NumberNode #4 NumberNode #3.44 The numerical value of a NumberNode is explicitly referenced within the Atom. A core distinction is made between ordered links and unordered links; these are handled differently in the Atomspace software. A basic unordered link is the SetLink, which groups its arguments into a set. For instance, the ConceptNode C defined by ConceptNode C MemberLink A C MemberLink B C is equivalent to SetLink A B On the other hand, ListLinks are like SetLinks but ordered, and they play a fundamental role due to their relationship to predicates. Most predicates are assumed to take ordered arguments, so we may say e.g. EvaluationLink PredicateNode eat ListLink ConceptNode cat ConceptNode mouse to indicate that cats eat mice. Note that by an expression like ConceptNode cat is meant ConceptNode C ReferenceLink W C WordNode W #cat since it's WordNodes rather than ConceptNodes that refer to words. (And note that the strength of the ReferenceLink would not be 1 in this case, because the word "cat" has multiple senses.) However, there is no harm nor formal incorrectness in the "ConceptNode cat" usage, since "cat" is just as valid a name for a ConceptNocle as, say, "C." We've already introduced above the MemberLink, which is a link joining a member to the set that contains it. Notable is that the truth value of a MemberLink is fuzzy rather than probabilistic, and that PLN is able to inter-operate fuzzy and probabilistic values. SubsetLinks also exist, with the obvious meaning, e.g. ConceptNode cat ConceptNode animal SubsetLink cat animal EFTA00624024

13.3 Atoms: Their Types and Weights 249 Note that SubsetLink refers to a purely extensional subset relationship, and that Inheri- tanceLlnk should be used for the generic "intensional + extensional" analogue of this - more on this below. SubsetLink could more consistently (with other link types) be named Extension- allnheritanceLink, but SubsetLink is used because it's shorter and more intuitive. There are links representing Boolean operations AND, OR and NOT. For instance, we may say ImplicationLink ANDLink ConceptNode young ConceptNode beautiful ConceptNode attractive or, using links and VariableNodes instead of ConceptNodes, AverageLink SX ImplicationLink ANDLink EvaluationLink young SX EvaluationLink beautiful SX EvaluationLink attractive SX NOTLink is a unary link, so e.g. we might say AverageLink SX ImplicationLink ANDLink EvaluationLink young SX EvaluationLink beautiful $X EvaluationLink NOT EvaluationLink poor $X EvaluationLink attractive SX ContextLink allows explicit contextualization of knowledge, which is used in PLN, e.g. ContextLink ConceptNode golf InheritanceLink ObjectNode EtenGoertzel ConceptNode incompetent says that Ben Goertzel is incompetent in the context of golf. 13.3.2 Variable Atoms We have already introduced VariableNodes above; it's also possible to specify the type of a VariableNode via linking it to a VariableTypeNode via a TypedVariableLink, e.g. VariableTypeLink VariableNode SX VariableTypeNode ConceptNode which specifies that the variable $X should be filled with a ConceptNode. Variables are handled via quantifiers; the default quantifier being the AverageLink, so that the default interpretation of EFTA00624025

250 13 Local, Global and Global Knowledge Representation ImplicationLink InheritanceLink SX animal EvaluationLink PredicateNode: eat ListLink \SX ConceptNode: food is AverageLink SX ImplicationLink InheritanceLink $X animal EvaluationLink PredicateNode: eat ListLink \SX ConceptNode: food The AverageLink invokes an estimation of the average TruthValue of the embedded expression (in this case an ImplicationLink) over all possible values of the variable $X. If there are type restrictions regarding the variable $X, these are taken into account in conducting the averaging. For AllLink and Exist s-Link may be used in the same places as AverageLink, with uncertain truth value semantics defined in PLN theory using third-order probabilities. There is also a ScholemLink used to indicate variable dependencies for existentially quantified variables, used in cases of multiply nested existential quantifiers. EvaluationLink and MemberLink have overlapping semantics, allowing expression of the same conceptual/logical relationships in terms of predicates or sets, i.e. EvaluationLink PredicateNode: eat ListLink SX ConceptNode: food has the same semantics as MemberLink ListLink SX ConceptNode: food ConceptNode: EatingEvents The relation between the predicate "eat" and the concept "EatingEvents" is formally given by ExtensionalEquivalenceLink ConceptNode: EatingEvents SatisfyingSetLink PredicateNode: eat In other words, we say that "EatingEvents" is the SatisfyingSet of the predicate "eat": it is the set of entities that satisfy the predicate "eat". Note that the truth values of MemberLink and EvaluationLink are fuzzy rather than probabilistic. EFTA00624026

13.3 Atoms: Their Types and Weights 251 13.3.3 Logical Links There is a host of link types embodying logical relationships as defined in the PLN logic system, e.g. • InheritanceLink • SubsetLink (aka ExtensionallnheritanceLink) • Intensional InheritanceLink which embody different sorts of inheritance, e.g. SubsetLink salmon fish IntensionallnheritanceLink whale fish InheritanceLink fish animal and then • SimilarityLink • ExtensionalSimilarityLink • IntensionalSimilarityLink which are symmetrical versions, e.g. SimilaritytLink shark barracuda IntensionalSimilarityLink shark dolphin ExtensionalSimiliarityLink American obese person There are also higher-order versions of these links, both asymmetric • ImplicationLink • ExtensionalImplicationLink • IntensionalImplicationLink and symmetric • EquivalenceLink • ExtensionalEquivalenceLink • IntensionalEquivalenceLink These are used between predicates and links, e.g. ImplicationLink EvaluationLink eat ListLink $X dirt EvaluationLink feel ListLlnk $X sick or EFTA00624027

252 13 Local, Global and Global Knowledge Representation ImplicationLink EvaluationLink eat ListLink $X dirt InheritanceLink $X sick or ForAIlLink SX, SY, $2 ExtensionalEquivalenceLink EquivalenceLink $2 EvaluationLink ListLink $X $Y EquivalenceLink $2 EvaluationLink ListLink $Y $X Note, the latter is given as an extensional equivalence because it's a pure mathematical equiv- alence. This is not the only case of pure extensional equivalence, but it's an important one. 13.3.4 Temporal Links There are also temporal versions of these links, such as • PredictivelmplicationLink • PredictiveAttractionLink • SequentialANDLink • $imultaneousANDLink which combine logical relation between the argument with temporal relation between their arguments. For instance, we might say PredictivelmplicationLink PredicateNode: JumpOffCliff PredicateNode: Dead or including arguments, PredictivelmplicationLink EvaluationLink JumpOffCliff $X EvaluationLink Dead O( The former version, without variable arguments given, shows the possibility of using higher- order logical links to join predicates without any explicit variables. Via using this format exclu- sively, one could avoid VariableAtoms entirely, using only higher-order functions in the manner EFTA00624028

13.3 Atoms: Their Types and Weights 253 of pure functional programming formalisms like combinatory logic. However, this purely func- tional style has not proved convenient, so the Atomspace in practice combines functional-style representation with variable-based representation. Temporal links often come with specific temporal quantification, e.g. PredictivelmplicationLink <5 seconds> EvaluationLink JumpOffCliff $X EvaluationLink Dead SX indicating that the conclusion will generally follow the premise within 5 seconds. There is a system for managing fuzzy time intervals and their interrelationships, based on a fuzzy version of Allen Interval Algebra. SequentialANDLink is similar to PredictivelmplicationLink but its truth value is calculated differently. The truth value of SequentialANDLink <5 seconds> EvaluationLink JumpOffCliff $X EvaluationLink Dead SX indicates the likelihood of the sequence of events occurring in that order, with gap lying within the specified time interval. The truth value of the PredictivelmplicationLink version indicates the likelihood of the second event, conditional on the occurrence of the first event (within the given time interval restriction). There are also links representing basic temporal relationships, such as BeforeLink and Af- terLink. These are used to refer to specific events, e.g. if X refers to the event of Ben waking up on July 15 2012, and Y refers to the event of Ben getting out of bed on July 15 2012, then one might have AfterLink X Y And there are TimeNodes (representing time-stamps such as temporal moments or intervals) and AtTimeLinks, so we may e.g. say AtTimeLink x TimeNode: 8:24AM Eastern Standard Time, July 15 2012 AD 13.3.5 Associative Links There are links representing associative, attentional relationships, • HebbianLink • AsymmetricHebbianLink • InverseHebbianLink • SymmetricInverseHebbianLink These connote associations between their arguments, i.e. they connote that the entities repre- sented by the two argument occurred in the same situation or context, for instance HebbianLink happy smiling AsymmetricHebbianLink dead rotten InverseHebbianLink dead breathing EFTA00624029

254 13 Local, Global and Global Knowledge Representation The asymmetric HebbianLink indicates that when the first argument is present in a situation, the second is also often present. The symmetric (default) version indicates that this relationship holds in both directions. The inverse versions indicate the negative relationship: e.g. when one argument is present in a situation, the other argument is often not present. 13.3.6 Procedure Nodes There are nodes representing various sorts of procedures; these are kinds of ProcedureNode, e.g. • Schemallode, indicating any procedure • GromidedSchemallode, indicating any procedure associated in the system with a Combo program or C++ function allowing the procedure to be executed • PredicateNode, indicating any predicate that associates a list of arguments with an output truth value • GromidedPredicateNode, indicating a predicate associated in the system with a Combo program or C++ function allowing the predicate's truth value to be evaluated on a given specific list of arguments ExecutionLinks and EvaluationLinks record the activity of Schemallodes and PredicateN- odes. We have seen many examples of EvaluationLinks in the above. Example ExecutionLinks would be: ExecutionLink step\_forward ExecutionLink stepLforward 5 ExecutionLink ListLink NumberNode: 2 NumberNode: 3 The first example indicates that the schema "step forward" has been executed. The second example indicates that it has been executed with an argument of "5" (meaning, perhaps, that 5 steps forward have been attempted). The last example indicates that the "+" schema has been executed on the argument list (2,3), presumably resulting in an output of 5. The output of a schema execution may be indicated using an ExecutionOutputLink, e.g. ExecutionOutputLink ListLink NumberNode: 2 NumberNode: 3 refers to the value "5" (as a NumberNode). 13.3.7 Links for Special External Data Types Finally, there are also Atom types referring to specific types of data important to using OpenCog in specific contexts. EFTA00624030

13.3 Atoms: Their Types and Weights 255 For instance, there are Atom types referring to general natural language data types, such as • WordNode • SentenceNode • WordinstanceNode • DocumentNode plus more specific ones referring to relationships that are part of link-grammar parses of sen- tences • FeatureNode • FeatureLink • LinkGrammarRelationshipNode • LinkGrammarDisjunctNode or RelEx semantic interpretations of sentences • DefinedLinguisticConceptNode • DefinedLinguisticRelationshipNode • PrepositionalRelationshipNode There are also Atom types corresponding to entities important for embodying OpenCog in a virtual world, e.g. • ObjectNode • AvatarNode • HumanoidNode • UnknownObjectNode • AccessoryNode 13.3.8 Truth Values and Attention Values CogPrime Atoms (Nodes and Links) are quantified with truth values that, in their simplest form, have two components, one representing probability (strength) and the other representing weight of evidence; and also with attention values that have two components, short-term and long-term importance, representing the estimated value of the Atom on immediate and long- term time-scales. In practice many Atoms are labeled with CompositeTruthVahm rather than elementary ones. A composite truth value contains many component truth values, representing truth values of the Atom in different contexts and according to different estimators. It is important to note that the CogPrime declarative knowledge representation is neither a neural net nor a semantic net, though it does have some commonalities with each of these traditional representations. It is not a neural net because it has no activation values, and involves no attempts at low-level brain modeling. However, attention values are very loosely analogous to time-averages of neural net activations. On the other hand, it is not a semantic net because of the broad scope of the Atoms in the network: for example, Atoms may represent percepts, procedures, or parts of concepts. Most CogPrime Atoms have no corresponding English label. However, most CogPrime Atoms do have probabilistic truth values, allowing logical semantics. EFTA00624031

256 13 Local, Global and Global Knowledge Representation 13.4 Knowledge Representation via Attractor Neural Networks Now we turn to global, implicit knowledge representation — beginning with formal neural net models, briefly discussing the brain, and then turning back to CogPrime. Firstly, this section reviews some relevant material from the literature regarding the representation of knowledge using attractor neural nets. It is a mix of well-established fact with more speculative material. 13.4.1 The Hopfield neural net model Hopfield networks Iliop821 are attractor neural networks often used as associative memories. A Hopfield network with N neurons can be trained to store a set of bipolar patterns P, where each pattern p has N bipolar (±1) values. A Hopfield net typically has symmetric weights with no self-connections. The weight of the connection between neurons i and j is denoted by ark/. In order to apply a Hopfield network to a given input pattern p, its activation state is set to the input pattern, and neurons are updated asynchronously, in random order, until the network converges to the closest fixed point. An often-used activation function for a neuron is: = sign(Pi L wiao jilt Training a Hopfield network, therefore, involves finding a set of weights wo that stores the training patterns as attractors of its network dynamics, allowing future recall of these patterns from possibly noisy inputs. Originally, Hopfield used a Hebbian mle to determine weights: = Pi p=1 Typically, Hopfield networks are fully connected. Experimental evidence, however, suggests that the majority of the connections can be removed without significantly impacting the net- work's capacity or dynamics. Our experimental work uses sparse Hopfield networks. 13.4.1.1 Palimpsest Hopfield nets with a modified learning rule In JSV99J a new learning rule is presented, which both increases the Hopfield network capacity and turns it into a "palimpsest", i.e., a network that can continuously learn new patterns, while forgetting old ones in an orderly fashion. Using this new training mle, weights are initially set to zero, and updated for each new pattern p to be learned according to: EFTA00624032

13.4 Knowledge Representation via Attractor Neural Networks 9.17 hii. E wikpk k.,,k0,,, 1 , atpu = —n(pipi - hag; - hoPi) 13.4.2 Knowledge Representation via Cell Assemblies Hopfield nets and their ilk play a dual role: as computational algorithms, and as conceptual models of brain function. In CogPrime they are used as inspiration for slightly different, artificial economics based computational algorithms; but their hypothesized relevance to brain function is nevertheless of interest in a CogPrhne context, as it gives sonic hints about the potential connection between low-level neural net mechanics and higher-level cognitive dynamics. Hopfield nets lead naturally to a hypothesis about neural knowledge representation, which holds that a distinct mental concept is represented in the brain as either: 1. a set of "cell assemblies", where each assembly is a network of neurons that are interlinked in such a way as to fire in a (perhaps nonlinearly) synchronized manner 2. a distinct temporal activation pattern, which may occur in any one (or more) of a particular set of cell assemblies For instance, this hypothesis is perfectly coherent if one interprets a "mental concept" as a SMEPH (defined in Chapter 14) ConceptNode, i.e. a fuzzy set of perceptual stimuli to which the organism systematically reacts in different ways. Also, although we will focus mainly on declarative knowledge here, we note that the same basic representational ideas can be applied to procedural and episodic knowledge: these may be hypothesized to correspond to temporal activation patterns as characterized above. In the biology literature, perhaps the bast-articulated modern theories championing the cell assembly view are those of Gunther Palm 1Pa182, IAG071 and Susan Greenfield ISF05, CSG07]. Palm focuses on the dynamics of the formation and interaction assemblies of cortical columns. Greenfield argues that each concept has a core cell assembly, and that when the concept rises to the focus of attention, it recruits a number of other neurons beyond its core characteristic assembly into a "transient ensemble."' It's worth noting that there may be multiple redundant assemblies representing the same concept - and potentially recruiting similar transient assemblies when highly activated. The importance of repeated, slightly varied copies of the same subnetwork has been emphasized by Edelman IFle931 among other neural theorists. I The larger an ensemble is, she suggests, the more vivid it is as a conscious experience; an hypothesis that accords well with the hypothesis made in rooMill that a more informationally intense pattern corresponds to a more intensely conscious quale - but we don't need to digress extensively onto matters of consciousness for the present purposes. EFTA00624033

258 13 Local, Global and Gloat! Knowledge Representation 13.5 Neural Foundations of Learning Now we move from knowledge representation to learning - which is after all nothing but the adaptation of represented knowledge based on stimulus, reinforcement and spontaneous activity. While our focas in this chapter is on representation, it's not possible for us to make our points about glocal knowledge representation in neural net type systems without discussing some aspects of learning in these systems. 13.5.1 Hebbian Learning The most common and plausible assumption about learning in the brain is that synaptic connec- tions between neurons are adapted via some variant of Hebbian learning. The original Hebbian learning rule, proposed by Donald Hebb in his 1949 book [Heb49J, was roughly 1. The weight of the synapse x r y increases if x and y fire at roughly the same time 2. The weight of the synapse x r y decreases if x fires at a certain time but y does not Over the years since Hebb's original proposal, many neurobiologists have sought evidence that the brain actually uses such a method. One of the things they have found, so far, is a lot of evidence for the following learning rule [1)( LSO5I: 1. The weight of the synapse x r y increases if x fires shortly before y does 2. The weight of the synapse x r y decreases if x fires shortly after y does The new thing here, not foreseen by Donald Hebb, is the "postsynaptic depression" involved in rule component 2. Now, the simple rule stated above does not sum up all the research recently done on Hebbian- type leaning mechanisms in the brain. The real biological story underlying these approximate rules is quite complex, involving many particulars to do with various neurotransmitters. fll- understood details aside, however, there is an increasing body of evidence that not only does this sort of learning occur in the brain, but it leads to distributed experience-based neural modification: that is, one instance synaptic modification causes another instance of synaptic modification, which causes another, and so forth' 113i0II. 13.5.2 Virtual Synapses and Hebbian Learning Between Assemblies Hebbian learning is conventionally formulated in terms of individual neurons, but, it can be extended naturally to assemblies via defining "virtual synapses" between assemblies. Since assemblies are sets of neurons, one can view a synapse as linking two assemblies if it links two neurons, each of which is in one of the assemblies. One can then view two assemblies as being linked by a bundle of synapses. We can define the weight of the synaptic bundle from assembly Al to assembly A2 as the number w so that (the change 2 This has been observed in "model systems" consisting of neurons extracted from a brain and hooked together in a laboratory setting and monitored; measurement of such dynamics in vivo is obviously more difficult. EFTA00624034

13.5 Neural Foundations of Learning 259 in the mean activation of A2 that occurs at time t-frepsilon) is on average closest to w x (the amount of energy flowing through the bundle front Al to A2 at time t). So when Al sends an amount x of energy along the synaptic bundle pointing from Al to A2, then A2's mean activation is on average incremented/decremented by an amount w x x. In a similar way, one can define the weight of a bundle of synapses between a certain static or temporal activation-pattern P1 in assembly Al, and another static or temporal activation- pattern P2 in assembly A2. Namely, this may be defined as the number w so that (the amount of energy flowing through the bundle from Al to A2 at time t)xw best approximates (the probability that P2 is present in A2 at time t-/-epsilon), when averaged over all times t during which PI is present in Al. It is not hard to see that Hebbian learning on real synapses between neurons implies Hebbian learning on these virtual synapses between cell assemblies and activation-patterns. These ideas may be developed further to build a connection between neural knowledge rep- resentation and probabilistic logical knowledge representation such as is used in CogPrime's Probabilistic Logic Networks formalism; this connection will be pursued at the end of Chapter 34, once more relevant background has been presented. 13.5.3 Neural Darwinism A notion quite similar to Hebbian learning between assemblies has been pursued by Nobelist Gerald Edelman in his theory of neuronal group selection, or "Neural Darwinism." Edelman won a Nobel Prize for his work in immunology, which, like most modern immunology, was based on C. MacFarlane Burnet's theory of "clonal selection" [13ur62J, which states that antibody types in the mammalian immune system evolve by a form of natural selection. From his point of view, it was only natural to transfer the evolutionary idea from one mammalian body system (the immune system) to another (the brain). The starting point of Neural Darwinism is the observation that neuronal dynamics may be analyzed in terms of the behavior of neuronal groups. The strongest evidence in favor of this conjecture is physiological: many of the neurons of the neocortex are organized in clusters, each one containing say 10,000 to 50,000 neurons each. Once one has committed oneself to looking at such groups, the next step is to ask how these groups are organized, which leads to Edelman's concept of "maps." A "map," in Edelman's terminology, is a connected set of groups with the property that when one of the inter-group connections in the map is active, others will often tend to be active as well. Maps are not fixed over the life of an organism. They may be formed and destroyed in a very simple way: the connection between two neuronal groups may be "strengthened" by in- creasing the weights of the neurons connecting the one group with the other. and "weakened" by decreasing the weights of the neurons connecting the two groups. If we replace "map" with "cell assembly" we arrive at a concept very similar to the one described in the previous subsection. Edelman then makes the following hypothesis: the large-scale dynamics of the brain is dom- inated by the natural selection of maps. Those maps which are active when good results are obtained are strengthened, those maps which are active when bad results are obtained are weakened. And maps are continually mutated by the natural chaos of neural dynamics, thus providing new fodder for the selection process. By use of computer simulations, Edelman and his colleagues have shown that formal neural networks obeying this rule can carry out fairly compli- EFTA00624035

260 13 Local, Global and Glocal Knowledge Representation cated acts of perception. In general-evolution language, what is posited here is that organisms like humans contain chemical signals that signify organism-level success of various types, and that these signals serve as a "fitness function" correlating with evolutionary fitness of neuronal maps. In Neural Darwinism and his other related books and papers, Edelman goes far beyond this crude sketch and presents neuronal group selection as a collection of precise biological hypothe- ses, and presents evidence in favor of a number of these hypotheses. However, we consider that the basic concept of neuronal group selection is largely independent of the biological particular- ities in terms of which Edelman has phrased it. We suspect that the mutation and selection of "transformations" or "maps" is a ntwobary component of the dynamics of any intelligent system. As we will see later on (e.g. in Chapter 42 of Part 2, this business of maps is extremely important to CogPrime. CogPrime does not have simulated biological neurons and synapses, but it does have Nodes and Links that in some contexts play loosely similar roles. We sometimes think of CogPrime Nodes and Links as being very roughly analogous to Edelman's neuronal clusters, and emergent intercluster links. And we have maps among CogPrime Nodes and Links, just as Edelman has maps among his neuronal clusters. Maps are not the sole bearers of meaning in CogPrime, but they are significant ones. There is a very natural connection between Edelman-style brain evolution and the ideas about cognitive evolution presented in Chapter 3. Edelman proposes a fairly clear mechanism via which patterns that survive a while in the brain are differentially likely to survive a long time: this is basic Hebbian learning, which in Edelman's picture plays a role between neuronal groups. And, less directly, Edelman's perspective also provides a mechanism by which intense patterns will be differentially selected in the brain: because on the level of neural maps, pattern intensity corresponds to the combination of compactness and functionality. Among a number of roughly equally useful maps serving the same function, the more compact one will be more likely to survive over time, because it is less likely to be disrupted by other brain processes (such as other neural maps seeking to absorb its component neuronal groups into themselves). Edelman's neuroscience remains speculative, since so much remains unknown about human neural structure and dynamics; but it does provide a tentative and plausible connection between evolutionary neurodynamics and the more abstract sort of evolution that patternist philosophy posits to occur in the realm of mind-patterns. 13.6 Glocal Memory A glocal memory is one that transcends the global local dichotomy and incorporates both aspects in a tightly interconnected way. Here we make the glocal memory concept more precise, and describe its incarnation in the context of attractor neural nets (which is similar to its incarnation in CogPrime, to be elaborated in later chapters). Though our main interest here is in glocality in CogPrime, we also suggest that glocality may be a critical property to consider when analyzing human, animal and AI memory more broadly. The notion of glocal memory, has implicitly occurred in a number of prior brain theories (without use of the neologism "glocal"), e.g. [Ca196] and IGoe0II, but it has not previously been explicitly developed. However the concept has risen to the fore in our recent AI work and so we have chosen to flesh it out more fully in II IGPI+ lO] and the present section. EFTA00624036

13.6 Glocal Memory 261 Glocal memory overcomes the dichotomy between localized memory (in which each memory item is stored in a single location within an overall memory structure) and distributed memory (in which a memory item is stored as an aspect of a multi-component memory system, in such a way that the same set of multiple components stores a large number of memories). In a glocal memory system, most memory items are stored both locally and globally, with the property that eliciting either one of the two records of an item tends to also elicit the other one. Glocal memory applies to multiple forms of memory; however we will focus largely on percep- tual and declarative memory in our detailed analyses here, so as to conserve space and maintain simplicity of discussion. The central idea of glocal memory is that (perceptual, declarative, episodic, procedural, etc.) items may be stored in memory in the form of paired structures that are called (key, map) pairs. Of course the idea of a "pair" is abstract, and such pairs may manifest themselves quite differently in different sorts of memory systems (e.g. brains versus non-neuromorphic AI systems). The key is a localized version of the item, and records some significant aspects of the items in a simple and crisp way. The map is a dispersed, distributed version of the item, which represents the item as a (to some extent, dynamically shifting) combination of fragments of other items. The map includes the key as a subset; activation of the key generally (but not necessarily always) causes activation of the map; and changes in the memory item will generally involve complexly coordinated changes on the key and map level both. Memory is one area where animal brain architecture differs radically from the von Neu- mann architecture underlying nearly all contemporary general-purpose computers. Von Neu- mann computers separate memory from processing, whereas in the human brain there is no such distinction. In fact, it's arguable that in most cases the brain contains no memory apart from processing: human memories are generally constructed in the course of remembering Illos88], which gives human memory a strong capability for "filling in gaps" of remembered experi- ence and knowledge; and also causes problems with inaccurate remembering in many contexts 113F71, IIM951 We believe the constructive aspect of memory is largely associated with its glocality. The remainder of this section presents a fuller formalization of the glocal memory concept, which is then taken up further in three later chapters: • Chapter ?? discusses the potential implementation of glocal memory in the human brain • Chapter ?? discusses the implementation of glocal memory in attractor neural net systems • Chapter 23 presents Glocal Economic Attention Networks (ECANs), rough analogues of glocal Hopfield nets that play a central role in CogPrime. Our hypothesis of the potential general importance of glocality as a property of memory, systems (beyond just the CogPrime architecture) — remains somewhat speculative. The presence of glocality in human and animal memory is strongly suggested but not firmly demonstrated by available neuroscience data; and the general value of glocality in the context of artificial brains and minds is also not yet demonstrated as the whole field of artificial brain and mind building remains in its infancy. However, the utility of glocal memory for CogPrime is not tied to this more general, speculative theme - glocality may be useful in CogPrime even if we're wrong that it plays a significant role in the brain and in intelligent systems more broadly. EFTA00624037

262 13 Local, Global and Glocal Knowledge Representation 13.6.1 A Semi-Formal Model of Glocal Memory To explain the notion of glocal memory more precisely, we will introduce a simple semi-formal model of a system $ that uses a memory to record information relevant to the actions it carries out. The overall concept of glocal memory should not be considered as restricted to this particular model. This model is not intended for maximal generality, but is intended to encompass a variety of current AI system designs and formal neurological models. In this model, we will consider S's memory subsystem as a set of objects we'll call "tokens," embedded in some metric space. The metric in the space, which we will call the "basic distance" of the memory, generally will not be defined in terms of the semantics of the items stored in the memory; though it may come to shape these dynamics through the specific architecture and evolution of the memory. Note that these tokens are not intended as generally being mapped one-to-one onto meaningful items stored in the memory. The "tokens" are the raw materials that the memory arranges in various patterns in order to store items. We assume that each token, at each point in time, may meaningfully be assigned a certain quantitative "activation level." Also, tokens may have other numerical or discrete quantities associated with them, depending on the particular memory architecture. Finally, tokens may relate other tokens, so that optionally a token may come equipped with an (ordered or un- ordered) list of other tokens. To understand the meaning of the activation levels, one should think about S's memory, subsystem as being coupled with an action-selection subsystem, that dynamically chooses the actions to be taken by the overall system in which the two subcystems are embedded. Each combination of actions, in each particular type of context, will generally be associated with the activation of certain tokens in memory. Then, as analysts of the system $, we may assnciate each token T with an "activation vector" v(T, t), whose value for each discrete time t consists of the activation of the token T at time t. So, the 50'th entry of the vector corresponds to the activation of the token at the 50'th time step. "Items stored in memory" over a certain period of time, may then be defined as clusters in the set of activation vectors associated with memory during that period of time. Note that the system $ itself may explicitly recognize and remember patterns regarding what items are stored in its memory - but, from an external analyst's perspective, the set of items in S's memory is not restricted to the ones that S has explicitly recognized as memory items. The "localization" of a memory item may be defined as the degree to which the various tokens involved in the item are close to each other according to the metric in the memory metric-space. This degree may be formalized in various ways, but choosing a particular quantitative measure is not important here. A highly localized item may be called "local" and a not-very-localized item may be called "global." We may define the "activation distance" of two tokens as the distance between their activation vectors. We may then say that a memory is "well aligned" to the extent that there is a correlation between the activation distance of tokens, and the basic distance of the memory metric-space. Given the above set-up, the basic notion of glocal memory can be enounced fairly simply. A glocal memory is one: • that is reasonably well-aligned (i.e. the correlation between activation and basic distance is significantly greater than random) EFTA00624038

13.6 Glocal Memory 263 • in which most memory items come in pairs, consisting of one local item and one global item, so that activation of the local item (the "key") frequently leads in the near future to activation of the global item (the "map") Obviously, in the scope of all possible memory structures constructible within the above formalism, glocal memories are going to be very rare and special. But, we suggest that they are important, because they are generally going to be the mast effective way for intelligent systems to structure their memories. Note also that many memories without glocal structure may be "well-aligned" in the above sense. An example of a predominantly local memory structure, in which nearly all significant mem- ory items are local according to the above definition, is the Cyc logical reasoning engine ILG90]. To cast the Cyc knowledge base in the present formal model, the tokens are logical predicates. Cyc does not have an in-built notion of activation, but one may conceive the activation of a logical formula in Cyc as the degree to which the formula is used in reasoning or query process- ing during a certain interval in time. And one may define a basic metric for Cyc by associating a predicate with its extension (the set of satisfying inputs), and defining the similarity of two predicates as the symmetric distance of their extensions. Cyc is reasonably well-aligned, but according to the dynamics of its querying and reasoning engines, it is basically a local memory, structure without significant global memory structure. On the other hand, an example of a predominantly global memory structure, in which nearly all significant memory items are global according to the above definition, is the Hopfield asso- ciative memory network lAmi89]. Here memories are stored in the pattern of weights associated with synapses within a network of formal neurons, and each memory in general involves a large number of the neurons in the network. To cast the Hopfield net in the present formal model, the tokens are neurons and synapses; the activations are neural net activations; the basic distance between two neurons A and B may be defined as the percentage of the time that stimulating one of the neurons leads to the other one firing; and to calculate a basic distance involving a synapse, one may associate the synapse with its source and target neurons. With these defini- tions, a Hopfield network is a well-aligned memory, and (by intentional construction) a markedly global one. Local memory items will be very rare in a Hopfield net. While predominantly local and predominantly global memories may have great value for par- ticular applications, our suggestion is that they also have inherent limitations. If so, this means that the most useful memories for general intelligence are going to be those that involve both local and global memory items in central roles. However, this is a more general and less risky claim than the assertion that glocal memory structure as defined above is important. Because, "glocal" as defined above doesn't just mean "neither predominantly global nor predominantly local." Rather. it refers to a specific pattern of coordination between local and global memory, items - what we have called the "keys and maps" pattern. 13.6.2 Glocal Memory in the Brain Science's understanding of human brain dynamics is still very primitive, one manifestation of which is the fact that we really don't understand how the brain represents knowledge, except in some very simple respects. So anything anyone says about knowledge representation in the brain, at this stage, has to be considered highly speculative. Existing neuroscience knowledge EFTA00624039

264 13 Local, Global and Global Knowledge Representation does imply constraints on how knowledge representation in the brain may work, but these are relatively loose constraints. These constraints do imply that, for instance, the brain is neither a relational database (in which information is stored in a wholly localized manner) nor a collection of "grandmother neurons" that respond individually to high-level percepts or concepts; nor a simple Hopfield type neural net (in which all memories are attractors globally distributed across the whole network). But they don't tell us nearly enough to, for instance, create a formal neural net model that can confidently be said to represent knowledge in the manner of the human brain. As a first example of the current state of knowledge, we'll discuss here a series of papers regarding the neural representation of visual stimuli [QaGNICF05, QICK F08], which deal with the fascinating discovery of a subset of neurons in the medial temporal lobe (MTL) that are selectively activated by strikingly different pictures of given individuals, landmarks or objects, and in some cases even by letter strings. For instance, in their 2005 paper titled "Invariant visual representation by single neurons in the human brain", it is noted that in one case, a responded only to three completely different images of the ex-president Bill Clinton. Another unit (from a different patient) responded only to images of The Beatles, another one to cartoons from The Simpson's television series and another one to pictures of the basketball player Michael Jordan. Their 2008 follow-up paper backed away from the more extreme interpretation in the title as well as the conclusion, with the title "Sparse but not 'Grandmother-cell' coding in the medial temporal lobe." As the authors emphasize there, Given the very sparse and abstract representation of visual information by these neurons, they could in principle be considered as `grandmother cells'. However, we give several arguments that make such an extreme interpretation unlikely. MTL neurons are situated at the juncture of transformation of percepts into constructs that can be consciously recollected. These cells respond to percepts rather than to the detailed information falling on the retina. Thus, their activity reflects the full transformation that visual information undergoes through the ventral pathway. A crucial aspect of this transformation is the complementary development of both selectivity and invariance. The evidence presented here, obtained from recordings of single-neuron activity in h ns, suggests that a subset of MTL neurons possesses a striking invariant representation for consciously perceived objects, responding to abstract concepts rather than more basic metric details. This representation is sparse, in the sense that responsive neurons fire only to very few stimuli (and are mostly silent except for their preferred stimuli), but it is far from a Grandmother-cell representation. The fact that the MTL represents conscious abstract information in such a sparse and invariant way is consistent with its prominent role in the consolidation of long-term semantic memories. It's interesting to note how inadequate the NICKF081 data really is for exploring the notion of glocal memory in the brain. Suppose it's the case that individual visual memories corre- spond to keys consisting of small neuronal subnetworks, and maps consisting of larger neuronal subnetworks. Then it would be not at all surprising if neurons in the "key" network corre- sponding to a visual concept like "Bill Clinton's face" would be found to respond differentially to the presentation of appropriate images. Yet, it would also be wrong to overinterpret such data as implying that the key network somehow comprises the "representation" of Bill Clinton's face in the individual's brain. In fact this key network would comprise only one aspect of said representation. In the glocal memory hypothesis, a visual memory like "Bill Clinton's face" would be hypoth- esized to correspond to an attractor spanning a significant subnetwork of the individual's brain EFTA00624040

13.6 Glocal Memory 265 - but this subnetwork still might occupy only a small fraction of the neurons in the brain (say, 1/100 or less), since there are very many neurons available. This attractor would constitute the map. But then, there would be a much smaller number of neurons serving as key to unlock this map: i.e. if a few of these key neurons were stimulated, then the overall attractor pattern in the map as a whole would unfold and come to play a significant role in the overall brain activity landscape. In prior publications [0)001 the primary author explored this hypothesis in more detail in terms of the known architecture of the cortex and the mathematics of complex dynamical attractors. So, one passible interpretation of the NICKF08] data is that the MTL neurons they're measuring are part of key networks that correspond to broader map networks recording percepts. The map networks might then extend more broadly throughout the brain, beyond the MTL and into other perceptual and cognitive areas of cortex. Furthermore, in this case, if some MTL key neurons were removed. the maps might well regenerate the missing keys (as would happen e.g. in the glocal Hopfield model to be discussed in the following section). Related and interesting evidence for glocal memory in the brain comes from a recent study of semantic memory, illustrated in Figure ?? [PNI107]. Their research probed the architecture of semantic memory via comparing patients suffering from semantic dementia (SD) with patients suffering from three other neuropathologies, and found reasonably convincing evidence for what they call a "distributed-plus-hub" view of memory. The SD patients they studied displayed highly distinctive symptomology; for instance, their vocabularies and knowledge of the properties of everyday objects were strongly impaired, whereas their memories of recent events and other cognitive capacities remain perfectly in- tact. These patients also showed highly distinctive patterns of brain damage: focal brain lesions in their anterior temporal lobes (ATL), unlike the other patients who had either less severe or more widely distributed damage in their ATLs. This led [PNR07] to conclude that the ATL (being adjacent to the amygdala and limbic systems that process reward and emotion; and the anterior parts of the medial temporal lobe memory system, which processes episodic memory) is a "hub" for arnodal semantic memory, drawing general semantic information from episodic memories based on emotional salience. So, in this view, the memory of something like a "banana" would contain a distributed as- pect, spanning multiple brain systems, and also a localized aspect, centralized in the ATL. The distributed aspect would likely contain information on various particular aspects of ba- nanas, including their sights, smells, and touches, the emotions they evoke, and the goals and motivations they relate to. The distributed and localized aspects would influence one another dynamically, but, the data IPNROTI gathered do not address dynamics and they don't venture hypotheses in this direction. There is a relationship between the "distributed-plus-hub" view and [Murillo] better-known notion of a "convergence zone", defined roughly as a location where the brain binds features to- gether. A convergence zone, in [Darn00] perspective, is not a "store" of information but an agent capable of decoding a signal (and of reconstructing information). He also uses the metaphor that convergence zones behave like indexes drawing information from other areas of the brain - but they are dynamic rather than static indices, containing the instructions needed to recognize and combine the features constituting the memory of something. The mechanism involved in the distributed-plus-hub model is similar to a convergence zone, but with the important differ- ence that hubs are less local: [PNI107] semantic hub may be thought of a kind of "cluster of convergence zones" consisting of a network of convergence zones for various semantic memories. EFTA00624041

266 13 Local, Global and Global Knowledge Representation • DO nbtated -pray new Acnce, Worth — b Datireakand-plu Sand - Motion Colas Gams rt hantw• Nine Acton Serpa Conventrn arriwiecture Name rag rIdecintkert fanatic., ACIO1 Tata, depeix/cre 4nerscrater.:e. -.aw dependent Shape Nateet 1404 two hatelitelObt• Fig. 13.1: A Simplified Look at Feedback-Control in Uncertain Inference What is missing in IPN1207I and pan1001 perspective is a vision of distributed memories as attractors. The idea of localized memories serving as indices into distributed knowledge stores is important, but is only half the picture of glocal memory: the creative, constructive, dynamical-attractor aspect of the distributed representation is the other half. The closest thing to a clear depiction of this aspect of glocal memory that seems to exist in the neuroscience literature is a portion of William Calvin's theory of the "cerebral code" ICalM. Calvin pro- poses a set of quite specific mechanisms by which knowledge may be represented in the brain using complexly-structured strange attractors, and by which these strange attractors may be propagated throughout the brain. Figure 13.2 shows one aspect of his theory: how a distributed attractor may propagate from one part of the brain to another in pieces, with one portion of the attractor getting propagated first, and then seeding the formation in the destination brain region of a close approximation of the whole attractor. Calvin's theory may be considered a genuinely glocal theory of memory. However, it also makes a large number of other specific commitments that are not part of the notion of glo- cality, such as his proposal of hexagonal meta-columns in the cortex, and his commitment to evolutionary learning as the primary driver of neural knowledge creation. We find these other EFTA00624042

13.6 Glocal Memory 267 4 4 r 1 a • a Ve'ra .4 4 a Ai al• -.I A A A -.11 A • • I Aa AS a 11P-P - .1 lf I • • .r .1 ss-I _j • a a a BA • 1- •• J aa J J a • r e .- A d .... r C e • di .1 1 A A A .• A BAJAami siaa a A t• -0 Jr ..I JA -A 4 -A -4 .t a -.I 1 J st .. 4 -• ▪ awf ruse a a d Is .0 r ig • ot taveir-ii I al a • a a Fig. 13.2: Calvin's Model of Distributed Attractors in the Brain hypotheses interesting and highly promising, yet feel it Ls also important to separate out the notion of glocal memory for separate consideration. Regarding specifics, our suggestion is that Calvin's approach may overemphasize the dis- tributed aspect of memory, not giving sufficient due to the relatively localized aspect as ac- counted for in the NICK FM results discussed above. In Calvin's glocal approach, global mem- ories are attractors and local memories are parts of attractors. We suggest a possible alternative, in which global memories are attractors and local memories are particular neuronal subnetworks such as the specialized ones identified by K FOS]. However, this alternative does not seem contradictory to Calvin's overall conceptual approach, even though it is different from the par- ticular proposals made in rani. The above paragraphs are far from a complete survey of the relevant neuroscience literature; there are literally dozens of studies one could survey pointing toward the glocality of various sorts of human memory. Yet experimental neuroscience tools are still relatively primitive, and every one of these studies could be interpreted in various other ways. In the next couple decades, as neuroscience tools improve in accuracy. our understanding of the role of glocality in human memory will doubtless improve tremendously. EFTA00624043

268 13 Local, Global and Glace! Knowledge Representation 13.6.3 Glocal Hopfield Networks The ideas in the previous section suggest that, if one wishes to construct an AGI, it is worth seriously considering using a memory with some sort of glocal structure. One research direction that follows naturally from this notion is "glocal neural networks." In order to explore the nature of glocal neural networks in a relatively simple and tractable setting, we have formalized and implemented simple examples of "glocal Hopfield networks": palimpsest Hopfield nets with the addition of neurons representing localized memories. While these specific networks are not used in CogPrime, they are quite similar to the ECAN networks that are used in CogPrime and described in Chapter 23 of Part 2. Essentially, we augment the standard Hopfield net architecture by adding a set of "key neurons." These are a small percentage of the neurons in the network, and are intended to be roughly equinumerous to the number of memories the network is supposed to store. When the Hopfield net converges to an attractor A, then new links are created between the neurons that are active in A, and one of the key neurons. Which key neuron is chosen? The one that, when it is stimulated, gives rise to an attractor pattern maximally similar to A. The ultimate result of this is that, in addition to the distributed memory of attractors in the Hopfield net, one has a set of key neurons that in effect index the attractors. Each attractor corresponds to a single key neuron. In the glocal memory model, the key neurons are the keys and the Hopfield net attractors are the maps. This algorithm has been tested in sparse Hopfield nets, using both standard Hopfield net learning rules and Storkey's modified palimpsest learning rule ISV99I, which provides greater memory capacity in a continuous learning context. The use of key neurons turns out to slightly increase Hopfield net memory capacity, but this isn't the main point. The main point is that one now has a local representation of each global memory so that if one wants to create a link between the memory, and something else, it's extremely easy to do so - one just needs to link to the corresponding key neuron. Or, rather, one of the corresponding key neurons: depending on how many key neurons are allocated, one might end up with a number of key neurons corresponding to each memory, not just one. In order to transform a palimpsest Hopfield net into a glocal Hopfield net, the following steps are taken: 1. Add a fixed number of "key neurons" to the network (removing other random neurons to keep the total number of neurons constant) 2. When the network reaches an attractor, create links from the elements in the attractor to one of the key neurons 3. The key neuron chosen for the previous step is the one that most closely matches the current attractor (which may be determined in several ways, to be discussed below) 4. To avoid the increase of the number of links in the network, when new links are created in Step 2, other key-neuron links are then deleted (several approaches may be taken here, but the simplest is to remove the key-neuron links with the lowest-absolute-value weights) In the simple implementation of the above steps that we implemented, and described in 'GPI" 10J, Step 3 is carried out simply by comparing the weights of a key neuron's links to the nodes in an attractor. A more sophisticated approach would be to select the key neuron with the highest activation during the transient interval immediately prior to convergence to the attractor. EFTA00624044

13.6 Glocal Memory 269 The result of these modifications to the ordinary Hopfield net, is a Hopfield net that contin- ually maintains a set of key neurons, each of which individually represents a certain attractor of the net. Note that these key neurons - in spite of being "symbolic" in nature - are learned rather than preprogrammed, and are every bit as adaptive as the attractors they correspond to. Fur- thermore. if a key neuron is removed, the glocal Hopfield net algorithm will eventually learn it back, so the robustness properties of Hopfield nets are retained. The results of experimenting with glocal Hopfield nets of this nature are summarized in GPI 10j. We studied Hopfield nets with connectivity around .1, and in this context we found that glocality • slightly increased memory capacity • massively increased the rate of convergence to the attractor, i.e. the speed of recall However, probably the most important consequence of glocality is a more qualitative one: it makes it far easier to link the Hopfield net into a larger system, as would occur if the Hopfield net were embedded in an integrative AGI architecture. Because a neuron external to the Hopfield net may now link to a memory in the Hopfield net by linking to the corresponding key neuron. 13.6.4 Neural-Symbolic Glocality in CogPrime In CogPrime, we have explicitly sought to span the symbolic/emergentist pseudo-dichotomy, via creating an integrative knowledge representation that combines logic-based aspects with neural-net-like aspects. As reviewed in Chapter 6 above, these function not in the manner of multimodular systems, but rather via using (probabilistic) truth values and (attractor neural net like) attention values as weights on nodes and links of the same (hyper) graph. The nodes and links in this hypergraph are typed, like a standard semantic network approach for knowl- edge representation, so they're able to handle all sorts of knowledge, from the most concrete perception and actuation related knowledge to the most abstract relationships. But they're also weighted with values similar to neural net weights, and pass around quantities (importance values, discussed in Chapter 23 of Part 2) similar to neural net activations, allowing emergent attractor/assembly based knowledge representation similar to attractor neural nets. The concept of glocality lies at the heart of this combination. in a way that spans the pseudo- dichotomy: • Local knowledge is represented in abstract logical relationships stored in explicit logical form, and also in Hebbian-type associations between nodes and links. • Global knowledge is represented in large-scale patterns of node and link weights, which lead to large-scale patterns of network activity, which often take the form of attractors qualitatively similar to Hopfield net attractors. These attractors are called maps. The result of all this is that a concept like "cat" might be represented as a combination of: • A small number of logical relationships and strong associations, that constitute the "key" subnetwork for the "cat" concept. • A large network of weak associations, binding together various nodes and links of various types and various levels of abstraction, representing the "cat map". EFTA00624045

270 13 Local, Global and Global Knowledge Representation The activation of the key will generally cause the activation of the map, and the activation of a significant percentage of the map will cause the activation of the rest of the map, including the key. Furthermore, if the key were for some reason forgotten, then after a significant amount of effort, the system would likely to be able to reconstitute it (perhaps with various small changes) from the information in the map. We conjecture that this particular kind of glocal memory will turn out to be very powerful for AGI, due to its ability to combine the strengths of formal logical inference with those of self-organizing attractor neural networks. As a simple example, consider the representation of a "tower", in the context of an artificial agent that has built towers of blocks, and seen pictures of many other kinds of towers, and seen some tall building that it knows are somewhat like towers but perhaps not exactly towers. If this agent is reasonably conceptually advanced (say, at Piagetan the concrete operational level) then its mind will contain some declarative relationships partially characterizing the concept of "tower," as well as its sensory and episodic examples, and its procedural knowledge about how to build towers. The key of the "tower" concept in the agent's mind may consist of internal images and episodes regarding the towers it knows best, the essential operations it knows are useful for building towers (piling blocks atop blocks atop blocks...), and the core declarative relations summarizing "towerness" - and the whole "tower" map then consists of a much larger number of images, episodes, procedures and declarative relationships connected to "tower" and other related entities. If any portion of the map is removed - even if the key is removed - then the rest of the map can be approximately reconstituted, after some work. Some cognitive operations are best done on the localized representation - e.g. logical reasoning. Other operations, such as attention allocation and guidance of inference control, are best done using the globalized map representation. EFTA00624046

Chapter 14 Representing Implicit Knowledge via Hypergraphs 14.1 Introduction Explicit knowledge is easy to write about and talk about; implicit knowledge is equally impor- tant, but tends to get less attention in discussions of AI and psychology, simply because we don't have as good a vocabulary for describing it, nor as good a collection of methods for measuring it. One way to deal with this problem is to describe implicit knowledge using language and methods typically reserved for explicit knowledge. This might seem intrinsically non-workable, but we argue that it actually makes a lot of sense. The same sort of networks that a system like CogPrime uses to represent knowledge explicitly, can also be used to represent the emergent knowledge that implicitly exists in an intelligent system's complex structures and dynamics. We've noted that CogPrime uses an explicit representation of knowledge in terms of weighted labeled hypergraphs; and also uses other more neural net like mechanisms (e.g. the economic attention allocation network subsystem) to represent knowledge globally and implicitly. Cog- Prime combines these two sorts of representation according to the principle we have called glocality. In this chapter we pursue glocality a bit further - describing a means by which even implicitly represented knowledge can be modeled using weighted labeled hypergraphs similar to the ones used explicitly in CogPrime. This is conceptually important, in terms of making clear the fundamental similarities and differences between implicit and explicit knowledge represen- tation; and it is also pragmatically meaningful due to its relevance to the CogPrime methods described in Chapter 42 of Part 2 that transform implicit into explicit knowledge. To avoid confusion with CogPrime's explicit knowledge representation, we will refer to the hypergraphs in this chapter as composed of Vertices and Edges rather than Nodes and Links. In prior publications we have referred to "derived" or "emergent" hypergraphs of the sort described here using the acronym SMEPH, which stands for Self-Modifying, Evolving Probabilistic Hy- pergraphs. 14.2 Key Vertex and Edge Types We begin by introducing a particular collection of Vertex and Edge types, to be used in modeling the internal structures of intelligent systems. The key SMEPH Vertex types are 271 EFTA00624047

272 14 Representing Implicit Knowledge via Hypergraphs • ConceptVertex, representing a set, for instance, an idea or a set of percepts • SchemaVertex, representing a procedure for doing something (perhaps something in the physical world, or perhaps an abstract mental action). The key SMEPH Edge types, using language drawn from Probabilistic Logic Networks (PLN) and elaborated in Chapter 34 below, are as follows: • ExtensionallnheritanceEdge (ExtInhEdge for short: an edge which, linking one Vertex or Edge to another, indicates that the former is a special case of the latter) • ExtensionalSimilarityEdge (ExtSim: which indicates that one Vertex or Edge is similar to another) • ExecutionEdge (a ternary edge, which joins S,B,C when $ is a SchemaVertex and the result from applying $ to B is C). So, in a SMEPH system, one is often looking at hypergraphs whose Vertices represent ideas or procedures, and whose Edges represent relationships of specialization, similarity or transforma- tion among ideas and/or procedures. The semantics of the SMEPH edge types is given by PLN, but is simple and common- sensical. Extlnh and ExtSim Edges come with probabilistic weights indicating the extent of the relationship they denote (e.g. the ExtSimEdge joining the cat ConceptVertex to the dog ConceptVertex gets a higher probability weight than the one joining the cat ConceptVertex to the washing-machine ConceptVertex). The mathematics of transformations involving these probabilistic weights becomes quite involved - particularly when one introduces SchemaVertices corresponding to abstract mathematical operations, a step that enables SMEPH hypergraphs to have the complete mathematical power of standard logical formalisms like predicate cal- culus, but with the added advantage of a natural representation of uncertainty in terms of probabilities, as well as a natural representation of networks and webs of complex knowledge. 14.3 Derived Hypergraphs We now describe how SMEPH hypergraphs may be used to model and describe intelligent systems. One can (in principle) draw a SMEPH hypergraph corresponding to any individual intelligent system, with Vertices and Edges for the concepts and processes in that system's mind. This is called the derived hypergraph of that system. 14.3.1 SMEPH Vertices A ConceptVertex in the derived hypergraph of a system corresponds to a structural pattern that persists over time in that system; whereas a SchemaVertex corresponds to a multi-time- point dynamical pattern that recurs in that system's dynamics. If one accepts the patternist definition of a mind as the set of patterns in an intelligent system, then it follows that the derived hypergraph of an intelligent system captures a significant fraction of the mind of that system. To phrase it a little differently, we may say that a Concept Vertex, in SMEPH, refers to the habitual pattern of activity observed in a system when some condition is met (this condition EFTA00624048

14.3 Derived Hypergraphs 273 corresponding to the presence of a certain pattern). The condition may refer to something in the world external to the system, or to something internal. For instance, the condition may be observing a cat. In this case, the corresponding Concept vertex in the mind of Ben Goertzel is the pattern of activity observed in Ben Goertzel's brain when his eyes are open and he's looking in the direction of a cat. The notion of pattern of activity can be made rigorous using mathematical pattern theory, as is described in The Hidden Pattern roe06,1. Note that logical predicates, on the SMEPH level, appear as particular kinds of Concepts, where the condition involves a predicate and an argument. For instance, suppose one wants to know what happens inside Ben's mind when he eats cheese. Then there is a Concept correspond- ing to the condition of cheese-eating activity. But there may also be a Concept corresponding to eating activity in general. If the Concept denoting the activity of eating X is generally eas- ily computable from the Concepts for X and eating individually, then the eating Concept is effectively acting as a predicate. A SMEPH SchemaVertex, on the other hand, is like a Concept that's defined in a time- dependent way. One type of Schema refers to a habitual dynamical pattern of activity occurring before and/or during some condition is met. For instance, the condition might be saying the word Hello. In that case the corresponding SchemaVertex in the mind of Ben Goertzel is the pattern of activity that generally occurs before he says Hello. Another type of Schema refers to a habitual dynamical pattern of activity occurring after sonic condition X is met. For instance, in the case of the Schema for adding two numbers, the precondition X consists of the two numbers and the concept of addition. The Schema is then what happens when the mind thinks of adding and thinks of two numbers. Finally, there are Schema that refer to habitual dynamical activity patterns occurring after sonic condition X is met and before some condition Y is met. In this case the Schema is viewed as transforming X into Y. For instance, if X is the condition of meeting someone who is not a friend, and Y is the condition of being friends with that person, then the habitually intervening activities constitute the Schema for making friends. 14.3.2 SMEPH Edges SMEPH edge types fall into two categories: functional and logical. Functional edges connect Schema vertices to their input and outputs; logical edges refer mainly to conditional proba- bilities, and in general are to be interpreted according to the semantics of Probabilistic Logic Networks. Let us begin with logical edges. The simplest case is the Subset edge, which denotes a straightforward, extensional conditional probability. For instance, it may happen that whenever the Concept for cat is present in a system, the Concept for animal is as well. Then we would say Subset cat animal (Here we assume a notation where "It A B" denotes an Edge of type R between Vertices A and B.) On the other hand, it may be that 50% of the time that cat is present in the system, cute is present as well: then we would say Subset cat cute <.5> EFTA00624049

274 14 Representing Implicit Knowledge via Hypergraphs where the <.5> denotes the probability, which is a component of the Truth Value associated with the edge. Next, the most basic functional edge is the Execution edge, which is ternary and denotes a relation between a Schema, its input and its output, e.g. Execution father_of Ben_Goertzel Ted_Goertzel for a schema father_of that outputs the father of its argument. The ExecutionOutput (ExOut) edge denotes the output of a Schema in an implicit way, e.g. ExOut say_hello refers to a particular act of saying hello, whereas ExOut add numbers 43, 4) refers to the Concept corresponding to 7. Note that this latter example involves a set of three entities: sets are also part of the basic SMEPH knowledge representation. A set may be thought of as a hypergraph edge that points to all its members. In this manner we may define a set of edges and vertices modeling the habitual activity patterns of a system when in different situations. This is called the derived hypergraph of the system. Note that this hypergraph can in principle be constructed no matter what happens inside the system: whether it's a human brain, a formal neural network, Cyc, OCP, a quantum computer, etc. Of course, constructing the hypergraph in practice is quite a different story: for instance, we currently have no accurate way of measuring the habitual activity patterns inside the human brain. fMRI and PET and other neuroimaging technologies give only a crude view, though they are continually improving. Pattern theory enters more deeply here when one thoroughly fleshes out the Inheritance concept. Philosophers of logic have extensively debated the relationship between extensional inheritance (inheritance between sets based on their members) and intensional inheritance (in- heritance between entity-types based on their properties). A variety of formal mechanisms have been proposed to capture this conceptual distinction; see (Wang, 2006, 1995 TODO make ref) for a review along with a novel approach utilizing uncertain term logic. Pattern theory, provides a novel approach to defining intension: one may associate with each ConceptVertex in a system's derived hypergraph the set of patterns associated with the structural pattern underlying that ConceptVertex. Then, one can define the strength of the IntensionalInheritanceEdge between two Concept Vertices A and B as the percentage of A's pattern-set that is also contained in B's pattern-set. According to this approach, for instance, one could have IntlnhEdge whale fish <0.6> ExtlnhEdge whale fish <0.0> since the fish and whale sets have common properties but no common members. 14.4 Implications of Patternist Philosophy for Derived Hypergraphs of Intelligent Systems Patternist philosophy rears its head here and makes some definite hypotheses about the struc- ture of derived hypergraphs. It suggests that derived hypergraphs should have a dual network EFTA00624050

14.4 Implications of Patternist Philosophy for Derived Hypergraphs of Intelligent Systems 275 structure, and that in highly intelligent systems they should have subgraphs that constitute models of the whole hypergraph (these are self systems). SMEPH does not add anything to the patternist view on a philosophical level, but it gives a concrete instantiation to some of the general ideas of patternism. In this section we'll articulate some "SMEPH principles", consti- tuting important ideas from patternist philosophy as they manifest themselves in the SMEPH context. The logical edges in a SMEPH hypergraph are weighted with probabilities, as in the simple example given above. The functional edges may be probabilistically weighted as well, since some Schema may give certain results only some of the time. These probabilities are critical in terms of SMEPH's model of system dynamics; they underly one of our SMEPH principles, Principle of Implicit Probabilistic Inference: In an intelligent system, the temporal evolution of the probabilities on the edges in the system's derived hypergraph should approxi- mately obey the rules of probability theory. The basic idea is that, even if a system - through its underlying dynamics - has no explicit connection to probability theory, it still must behave roughly as if it does, if it is going to be intelligent. The roughly part is important here; it's well known that humans are not terribly accurate in explicitly carrying out formal probabilistic inferences. And yet, in practical contexts where they have experience, humans can make quite accurate judgments: which is all that's required by the above principle, since it's the contexts where experience has occurred that will make up a system's derived hypergraph. Our next SMEPH principle is evolutionary, and states Principle of Implicit Evolution: In an intelligent system, new Schema and Concepts will continually be created, and the Schema and Concepts that are more useful for achieving system goals (as demonstrated via probabilistic implication of goal achievement) will tend to survive longer. Note that this principle can be fulfilled in many different ways. The important thing is that system goals are allowed to serve as a selective force. Another SMEPH dynamical principle pertains to a shorter time-scale than evolution, and states Principle of Attention Allocation: In an intelligent system, Schema and Concepts that are more useful for attaining short-term goals will tend to consume more of the system's energy. (The balance of attention oriented toward goals pertaining to different time scales will vary from system to system.) Next, there is the Principle of Autopoesis: In an intelligent system, if one removes some part of the system and then allows the system's natural dynamics to keep going, a decent approximation to that removed part will often be spontaneously reconstituted. And there is the EFTA00624051

276 14 Representing Implicit Knowledge via Hypergrapbs Cognitive Equation Principle: In an intelligent system, many abstract patterns that are present in the system at a certain time as patterns among other Schema and Concepts, will at a near-future time be present in the system as patterns among elementary system components. The Cognitive Equation Principle, briefly discussed in Chapter 3, basically means that Con- cepts and Schema emergent in the system are recognized by the system and then embodied as elementary items in the system so that patterns among them in their emergent form be- come, with the passage of time, patterns among them in their directly-system-embodied form. This is a natural consequence of the way intelligent systems continually recognize patterns in themselves. Note that derived hypergraphs may be constructed corresponding to any complex system which demonstrates a variety of internal dynamical patterns depending on its situation. How- ever, if a system is not intelligent, then according to the patternist philosophy evolution of its derived hypergraph can't necessarily be expected to follow the above principles. 14.4.1 SMEPH Principles in CogPrime We now more explicitly elaborate the application of these ideas in the CogPrime context. As noted above, in addition to explicit knowledge representation in terms of Nodes and Links, CogPrime also incorporates implicit knowledge representation in the form of what are called Maps: collections of Nodes and Links that tend to be utilized together within cognitive processes. These Maps constitute a CogPrime system's derived hypergraph, which will not be iden- tical to the hypergraph it uses for explicit knowledge representation. However, an interesting feedback loop arises here, in that the intelligence's self-study will generally lead it to recognize large portions of its derived hypergraph as patterns in itself, and then embody these patterns within its concretely implemented knowledge hypergraph. This relates to the Cognitive Equa- tion Principle defined above 3, in which an intelligent system continually recognizes patterns in itself and embodies these patterns in its own basic structure (so that new patterns may more easily emerge from them). Often it happens that a particular CogPrime node will serve as the center of a map, so that e.g. the Concept Link denoting cat will consist of a number of nodes and links roughly centered around a ConceptNode that is linked to the WordNode cat. But this is not guaranteed and some CogPrime maps are more diffuse than this with no particular center. Somewhat similarly, the key SMEPH dynamics are represented explicitly in CogPrime: prob- abilistic reasoning is carried out via explicit application of PLN on the CogPrime hypergraph, evolutionary learning is carried out via application of the MOSES optimization algorithm, and attention allocation is carried out via a combination of inference and evolutionary pattern min- ing. But the SMEPH dynamics also occur implicitly in CogPrime: emergent maps are reasoned on probabilistically as an indirect consequence of node-and-link level PLN activity; maps evolve as a consequence of the coordinated whole of CogPrime dynamics; and attention shifts between maps according to complex emergent dynamics. To see the need for maps, consider that even a Node that has a particular meaning attached to it - like the Iraq Node, say - doesn't contain much of the meaning of Iraq in it. The meaning of Iraq lies in the Links attached to this Node, and the Links attached to their Nodes - and the other Nodes and Links not explicitly represented in the system, which will be created by EFTA00624052

14.4 Implications of Patternist Philosophy for Derived Hypergraphs of Intelligent Systems 277 CogPrime's cognitive algorithms based on the explicitly existent Nodes and Links related to the /rag Node. This halo of Atoms related to the Iraq node is called the Iraq map. In general, some maps will center around a particular Atom, like this Iraq map, others may not have any particular identifiable center. CogPrime's cognitive processes act directly on the level of Nodes and Links, but they must be analyzed in terms of their impact on maps as well. In SMEPH terms, Cog- Prime maps may be said to correspond to SMEPH ConceptNodes, and for instance bundles of Links between the Nodes belonging to a map may correspond to a SMEPH Link between two ConceptNodes. EFTA00624053

EFTA00624054

Chapter 15 Emergent Networks of Intelligence 15.1 Introduction When one is involved with engineering an AGI system, one thinks a lot about the aspects of the system one is explicitly building - what are the parts, how they fit together, how to test they're properly working, and so forth. And yet, these explicitly engineered aspects are only a fraction of what's important in an AGI system. At least as critical are the emergent aspects - the patterns that emerge once the system is up and running, interacting with the world and other agents, growing and developing and learning and self-modifying. SMEPH is one toolkit for describing some of these emergent patterns, but it's only a start. In line with these general observations, most of this book will focus on the structures and processes that we have built, or intend to build, into the CogPrime system. But in a sense, these structures and processes are not the crux of CogPrime's intended intelligence. The purpose of these pre-programmed structures and processes is to give rise to emergent structures and processes, in the course of CogPrime's interaction with the world and the other minds within it. We will return to this theme of emergence at several points in later chapters, e.g. in the discussion of map formation in Chapter 42 of Part 2. Given the important of emergent structures - and specifically emergent network structures - for intelligence, it's fortunate the scientific community has already generated a lot of knowledge about complex networks: both networks of physical or software elements, and networks of organization emergent from complex systems. As most of this knowledge has originated in fields other than AGI, or in pure mathematics, it tends to require some reinterpretation or tweaking to achieve maximal applicability in the AGI context: but we believe this effort will become increasingly worthwhile as the AGI field progresses, because network theory is likely to be very useful for describing the contents and interactions of AGI systems as they develop increasing intelligence. In this brief chapter we specifically focus on the emergence of certain large-scale network structures in a CogPrime knowledge store, presenting heuristic arguments as to why these structures can be expected to arise. We also comment on the way in which these emergent structures are expected to guide cognitive processes, and give rise to emergent cognitive pro- ceas . The following chapter expands on this theme in a particular direction, exploring the possible emergence of structures characterizing inter-cognitive reflection. 279 EFTA00624055

280 15 Emergent Networks of Intelligence 15.2 Small World Networks One simple but potentially useful observation about CogPrime Atomspaces is that they are generally going to be small world networks Inuc03], rather than random graphs. A small world network is a graph in which the connectivities of the various nodes display a power law behavior - so that, loosely speaking, there are a few nodes with very many links, then more nodes with a modest number of links ... and finally, a huge number of nodes with very few links. This kind of network occurs in many natural and human systems, including citations among papers, financial arrangements among banks, links between Web pages and the spread of diseases among people or animals. In a weighted network like an Atomspace, "small-world-mess" must be defined in a manner taking the weights into account, and there are several obvious ways to do this. Figure 15.1 depicts a small but prototypical small-worlds network, with a few "hub" nodes possessing far more neighbors than the others, and then some secondary hubs, etc. An excellent reference on network theory in general, including but not limited to small world networks, is Peter Csermely's Weak Links rsenfil. Many of the ideas in that work have apparent OpenCog applications, which are not elaborated here. • • • • . • • • •••I • • • • . •-•^' • • • •-- •• • it% • . • • .7.4-Cr 1" • • ; • • • • • • „ • •• .• • ••• • ••••• • • • ::• S * ,P•. ••.- •• • • •" • .• •• • • • ,•• ••• • • • •/•• • • • ill • • • • • ••••• . • . • • . • " • • • . a • • • • •• • • • • a an • • •-• • • • • • • 7 -a • • \.• • " •• •• ., • • •• Fig. 15.1: A typical, though small-sized, small-worlds network. One process via which small world networks commonly form is "preferential attachment" [13:n04 This occurs in essence when "the rich get richer" - i.e. when nodes in the network grow new links, in a manner that causes them to preferentially grow links to nodes that already have more links. It is not hard to see that CogPrime's ECAN dynamics will naturally lead to EFTA00624056

15.3 Dual Network Structure 281 preferential attachment, because Atoms with more links will tend to get more STI, and thus will tend to get selected by more cognitive processes, which will cause them to grow more links. For this reason, in most circumstances, a CogPrime system in which most link-building cognitive processes rely heavily on ECAN to guide their activities will tend to contain a small- world-network Atomspace. This is not rigorously guaranteed to be the case for any possible combination of enviromnent and goals, but it is commonsensically likely to nearly always be the case. One consequence of the small worlds structure of the Atomspace is that, in exploring other properties of the Atom network, it is particularly important to look at the hub nodes. For instance, if one is studying whether hierarchical and heterarchical subnetworks of the Atomspace exist, and whether they are well-aligned with each other, it is important to look at hierarchical and heterarchical connections between hub nodes in particular (and secondary hubs, etc.). A pattern of hierarchical or dual network connection that only held up among the more sparsely connected nodes in a small-world network would be a strange thing, and perhaps not that cognitively useful. 15.3 Dual Network Structure One of the key theoretical notions in patternist philosophy is that complex cognitive systems evolve internal dual network structures, comprising superposed. harmonized hierarchical and heterarchical networks. Now we explore some of the specific CogPrime structures and dynamics militating in favor of the emergence of dual networks. 15.3.1 Hierarchical Networks The hierarchical nature of human linguistic concepts is well known, and is illustrated in Figure 15.2 for the commonsense knowledge domain (using a graph drawn from WordNet, a huge con- cept hierarchy covering 50K+ English-language concepts), and in Figure 15.4 for a specialized knowledge subdomain, genetics. Due to this fact, a certain amount of hierarchy can be expected to emerge in the Atomspace of any linguistically savvy CogPrime, simply due to its modeling of the linguistic concepts that it hears and reads. Hierarchy also exists in the natural world apart from language, which is the reason that many sensorimotor-knowledge-focused AGI systems (e.g. DeSTIN and HTM, mentioned in Chapter 4 above) feature hierarchical structures. In these cases the hierarchies are normally spatiotem- poral in nature - with lower layers containing elements responding to more localized aspects of the perceptual field, and smaller, more localized groups of actuators. This kind of hierarchy certainly could emerge in an AGI system, but in CogPrime we have opted for a different route. If a CogPrime system is hybridized with a hierarchical sensorimotor network like one of those mentioned above, then the Atoms linked to the nodes in the hierarchical sensorimotor network will naturally possess hierarchical conceptual relationships, and will thus naturally grow hier- archical links between them (e.g. InheritanceLinks and IntensionalinheritanceLinks via PLN, AsymmetricHebbianLinks via ECAN). EFTA00624057

15 Emergent Networks of Intelligence artefact Fig. 15.2: A typical, though small, subnetwork of WordNet's hierarchical network. Once elements of hierarchical structure exist via the hierarchical structure of language and physical reality, then a richer and broader hierarchy can be expected to accumulate on top of it, because importance spreading and inference control will implicitly and automatically be guided by the existing hierarchy. That is, in the language of Chaotic Logic roe9 II and patternist theory, hierarchical structure is an "autopoietic attractor" - once it's there it will tend to enrich itself and maintain itself. AsymmetricHebbianLinks arranged in a hierarchy will tend to cause importance to spread up or down the hierarchy, which will lead other cognitive processes to look for patterns between Atoms and their hierarchical parents or children, thus potentially building more hierarchical links. Chains of InheritanceLinks pointing up and down the hierarchy will lead PLN to search for more hierarchical links - e.g. most simply, A -* B C where C is above B is above A in the hierarchy, will naturally lead inference to check the viability of A —> C by deduction. There is also the possibility to introduce a special DefaultInheritanceLink, as discussed in Chapter 34 of Part 2, but this isn't actually necessary to obtain the inferential maintenance of a robust hierarchical network. 15.3.2 Associative, Heterarchical Networks Heterarchy is in essence a simpler structure than hierarchy: it simply refers to a network in which nodes are linked to other nodes with which they share important relationships. That is, there should be a tendency that if two nodes are often important in the same contexts or for EFTA00624058