A window

Achieving synthetic general intelligence is likely impossible without decoupling the models from human interfaces. While there is a hope that existing models will learn the nature of reality as a by-product of their interaction with human-generated data, their inputs and outputs conform by design to our biology, and these data points present a limited perspective on the true state of things. Such decoupling raises alignment concerns as our directions can have unexpected outcomes as the model translates the human interface (even if it has one) into its interface with general reality. Unlike feral children who share human biology, an autonomous intelligence can evolve unpredictably and devise abstractions incompatible with ours. Models capable of more than integration and repetition should have a window into general reality, and we need to design this window so that we get to experience the view.