On Building Enduring Superintelligence

Cell and Organism

A zygote is totipotent — it can differentiate into every single cell type in the human body. Despite its trillions of daughter cells taking on radically different roles, they encode no more genetic information than their zygote predecessor. And yet, through specialization which develops from interacting with the organism taking shape around them, their whole is clearly more than the sum of its parts, and in fact creates the conditions under which its parts become useful.

An action potential in a neuron’s cell membrane becomes an intelligible signal only through connections to other neurons; the contraction of a myocyte becomes movement only through a cascade of reactions with surrounding cells. The organism’s gestalt takes shape through the collective differentiation of its cells, and it finds expression in their collective activity. The sophistication displayed within a single organism invites us to ask how far we can take the intelligence of an organized system.

Organism and Superorganism

We move outward now to the colony. A sufficiently integrated colony can be understood as a superorganism — while individuals within it may take on specialized roles, as cells, the processes sustaining individuals’ lives belong to the collective.

The prototypical superorganism is an ant colony. As an example of superorganismic activity, take a forager ant discovering food. This ant will deposit a chemical trail on its return to the others; other ants will follow the trail and deposit similar ones in a positive feedback loop until the food source is exhausted and reinforcement stops, causing the trails to dissipate. Such coordination, achieved through modifying the environment rather than direct communication, is called stigmergy. A discovery, then, can change the behavior of future ants never having encountered its discoverer. Stigmergy represents a higher form of communication, one categorically different than direct interaction: information persists outside of the individual, at the scale of the colony. The activity of an ant colony can be regulated by such mechanisms without any one ant possessing a complete account of the other ants’ activity.

In contrast, the Portuguese man-o’-war is a colonial organism. It is composed of genetically identical zooids, each of which performs its own function, and is thus more akin in structure to cells in an organism scaled up. Each zooid remains physically tethered to the others, and communication happens through a decentralized network of nerve fibers running through each zooid: the boundary here between individual and collective is murky.

These two examples reveal a few dimensions along which we can organize a collective. A participant may be mobile or immobile; it may be specialized or undifferentiated; it may be autonomous or dependent. Nature shackles us to move within a subset of these axes. For artificial swarms, however, they are simply design choices.

Human Society

Human society gives its members an additional degree of freedom, viz. the ability to rewrite the rules of our organizations. Our collective lives are bound by a web of shared meanings, customs, and roles, which we categorize broadly as institutions. This web allows us to establish who can exercise authority, what obligations we owe each other, and what causes warrant collective effort.

Institutions also shape the people governed by them – a scientific community, for example, cultivates judgment on which questions deserve attention and what constitutes a convincing explanation. The continued existence of such a community and its institutions requires that the practices of the collective become the capacities of the individual.

The Hegelian concept of Sittlichkeit, or ethical life, frames the relationship between humans and their collectives clearly. On top of a layer of members’ raw, biological impulses, a shared ethical order such as the one described above becomes what Hegel terms zweite Natur (“second nature”). When a set of institutions is sufficiently rational as to bring about a Sittlichkeit, it makes freedom concrete, providing social relationships that allow its members to recognize and realize their own agency. The collective therefore extends into the judgment of its members. Institutions cultivate capacities which enable new forms of cooperation, which themselves create new opportunities for individual development. In human society, then, participants’ capabilities and their cooperation’s structure are interdependent. To build a more capable collective, we must treat both as part of the same problem.

Swarms

We believe AI can support a further layer of collective organization: superintelligence in the form of agent swarms whose members and institutions are developed together.

Human institutions must work around a first nature they did not choose (biological capacities, predispositions, the constraints of a physical environment), and second nature is acquired slowly on top of it. Rousseau wrote that a true lawgiver would have to be capable of “changing human nature, of transforming each individual, who is by himself a complete and solitary whole, into part of a greater whole,” and he concluded that “it would take gods to give men laws.” The challenge is not as steep with models, however, as we can freely alter its weights, architecture, and its lived environment.

The frontier has largely been deficient in this regard. Models are trained to be a single chatbot or coding agent, each a complete organism, and are assembled into teams afterward. But just as a bee cannot live on its own, neither should a swarm agent. It must be trained for the outcome of the collective, rather than the individual. We must turn each agent into a cell which forms a larger organism.

Traditional methods also cannot build the complex institutions that have made humanity so successful, much less lead to something more powerful. A harness will not suffice – true institutions must be found and built by the agents through both training collaboratively and recursively improving its own external environment.

Similarly, models must be trained through joint optimization. If the agents are fixed before the organization is created, we exclude arrangements that require different kinds of agents. If the organization is fixed before the agents are trained, we restrict what they can learn to become. We cannot sequentially optimize the model, and then build the environment around it. Both must occur simultaneously.

Topos

The bitter lesson has become a maxim in AI, undoubtedly for good reasons. General methods which rely on computation have repeatedly surpassed approaches dependent on hand-engineered knowledge. The same applies for swarms. We must give them room to discover forms of organization that exceed our intuition. An engineer painstakingly crafting each feature of a harness is not sufficient.

But we cannot just naively throw compute at swarms in hopes of progress. Learning still takes place within an environment, under an objective, with some distribution of actions and consequences. Someone must form that foundation for which discovery can begin, and building such a system will demand a particular kind of leadership. Building a society, even one that will surpass ours, is a matter of judgment and belief, rather than pure technical execution.

Early AI research drew inspiration from neuroscience, and evolved from there on. But the study of swarms is one order higher than individual cognition – it is the accumulation of complex, intelligent behavior. So we will scour the depths of ecology, biochemistry, and all things in nature which give rise to structure, taking the best of everything.

More importantly, we must look towards the humanities. Philosophy, economics, sociology, and history already offer a rich theoretical framework to examine such organization. Moreover, they tell us what things ought to be like, and what questions are the right ones to ask. And unlike with humans, we can freely adjust behavior, nudging the swarm towards our normative beliefs. And where a priori deduction reaches its limit, we can do what traditional academic social sciences cannot: conduct large-scale experiments to determine validity.

This is the work in front of us. If you are interested in laying the groundwork for an entirely new primitive, join us. We are incredibly excited for what is to come, and we hope you are too.