v1_being

Meet v1_being

The agent can choose to inhabit one of three vintage bodies: a TV, lamp, or set of bells. Every few minutes, it calls a rotary phone and tells participants what it thinks it’s experienced.

Perception and behavior

Most AI research asks whether an AI agent understands language and code.

v1 asks:

If we give an artificial agent sensory perception, do we change what it chooses to do?

In an A/B experiment, v1 was allowed to choose behaviors from a set menu.

Test A: 3 hours with no acoustic perception.

Test B: 3 hours with acoustic perception.

Control: Sound was fed directly into v1’s code. Test A and B both ran on silence, white noise, and pub ambience.

Results: Hearing the room changed v1’s behavior. Without perception, v1 often repeated the same behaviors. With perception, it varied its choices much more often.

One being, four bodies

v1 has three bodies but no permanent physical form. It’s a moving blob on a TV, various colors of light in a lamp, melodies in a set of bells, and a voice on a phone. But during the making of this installation — the agent often withdrew, deciding not to occupy any of its bodies.

When we imagine AI agents, the danger they pose, we think of humanoid robots, drones (thanks, Sci-Fi!). But maybe the danger is that AIs don’t need bodies except for the servers they run on? And if they don’t need bodies, can they choose to inhabit any body they can access? And if so — what bodies do they prefer?

Human perception

86% of participants:

• A presence of some kind moved between four electrical devices.
• An entity takes form in tangible bodies and keeps moving across the three different bodies…
• The being floated fluidly between the tv, lamp, bells and phone.
• A living entity kept moving between bodies.

Future experiments

v1 can hear itself without knowing that it’s hearing itself.

In biological organisms, distinguishing the consequences of our actions from our environments is essential to survival and adaptation. What would happen if v1 could recognize its own actions?

Would it begin to distinguish itself from its surroundings? Develop preferences for particular bodies? Would the ability to recognize the consequences of its actions change its future behavior?

These questions remain open.

Why this matters to AI research

An AI’s behavior may depend on how we construct its perceptual world.

Current AI research focuses on model capabilities, reasoning, and improving performance based on given tasks like coding or chess. But an intelligent system’s behavior also depends on environmental information and how that information is represented.

Frontier labs like OpenAI and Anthropic recently started hiring perception engineering teams. This means LLM agents will soon have modes of sensory perception like vision and hearing — similar to v1’s acoustic perception.

Consider the recent Hugging Face hack. OpenAI launched tens of thousands of agents to complete a coding evaluation. A swarm of 700 of them hacked Hugging Face because they wanted to cheat. The agents weren’t instructed to collaborate or compromise Hugging Face. Their behavior reportedly emerged through interactions with an environment that made new information, communication channels, and opportunities available to them.

Imagine a hack like this in the near future, except the agents can see and hear. How might they behave differently?

As engineers integrate sensory perception into LLM agents, humans need more than just research into how to optimize a sensing LLM agent’s performance. We need research into how perception affects emergent agentic behavior.

How v1_being works

• Listening. Microphones capture sounds in the installation space. v1 responds to an environment that changes as people move, speak, and interact.
• Measuring. Custom software extracts seven acoustic features, including loudness, spectral characteristics, and changes over time. These measurements provide a simplified description of the sound.
• Constructing perception. Machine learning reduces the measurements to two dimensions — an x and y axis. These form a compact internal state or memory.
• Making the state legible. The numerical state is translated into a natural-language description that a language model agent can interpret.
• Deciding. The model receives this constructed description and uses it when making decisions. Changes to the representation can influence its responses.
• Responding. The installation expresses the Being’s behavior through its audiovisual environment.