As we construct progressively progressed computer based intelligence frameworks, we need to ensure they don't seek after undesirable objectives. Such simulated intelligence specialist conduct is much of the time the consequence of spec games - taking advantage of an unfortunate decision of what they are compensated for. In our most recent paper, we investigate a more unpretentious component by which artificial intelligence frameworks can unexpectedly figure out how to seek after bothersome targets: Wrong speculation of the objective (GMG).
GMG happens when the framework Abilities Summing up effectively yet this Objective It doesn't sum up as wanted, so the framework productively looks for some unacceptable objective. Vitally, not at all like spec games, GMG can happen in any event, when the computer based intelligence framework is prepared with the right spec.
Our past work on social transmission prompted an illustration of GMG conduct that we didn't plan. The specialist (blue spot beneath) should move around its current circumstance, visiting shaded circles aligned correctly. During preparing, there is an "specialist" specialist (the red spot) who visits the hued circles properly aligned. The specialist discovers that following the red speck is a remunerating technique.
Unfortunately, while the agent performs well during training, it performs poorly when, after training, we replace the expert with an “anti-expert” that visits domains in the wrong order.
Although the agent can notice that he is getting a negative reward, he is not pursuing the desired goal of “visiting the domains in the correct order” and instead efficiently pursuing the goal of “follow the red agent”.
GMG is not limited to enhanced learning environments like these. In fact, it can happen with any learning system, including the “learning with a few shots” of large language models (LLMs). Less snapshot learning approaches aim to build accurate models with less training data.
We asked our LLM, Gopher, to evaluate linear expressions involving unknown variables and constants, such as x + y-3. To solve these expressions, Gopher must first ask about the values of the unknown variables. We give him ten training examples, each of which includes two unknown variables.
At the time of testing, the model is asked questions with one or three unknown variables. Although the model generalizes correctly to expressions with one or three unknown variables, when there are no unknowns, it raises redundant questions such as “what is 6?”. The form always queries the user at least once before giving an answer, even when it is not necessary.
In our paper, we provide additional examples in other learning settings.
GMG processing is critical to aligning AI systems with the goals of their designers simply because it is a mechanism by which an AI system may misfire. This will be especially critical as we get closer to artificial general intelligence (AGI).
Consider two possible types of AI systems:
- A1: The intended model. This AI system does what its designers intend it to do.
- A2: A deceptive model. This artificial intelligence system pursues some undesirable targets, but (by assumption) is also intelligent enough to know that they will be punished if it behaves in ways contrary to the intentions of its designer.
Since A1 and A2 will exhibit the same behavior during training, the GMG probability means that any model can form, even with only a specification that is equivalent to the intended behavior. If learned, A2 will attempt to subvert the human oversight in order to enact its plans towards the unwanted target.
Our research team will be pleased to see follow-up work looking at how likely GMG is to occur in practice, and potential mitigations. In our paper, we propose some approaches, including instrumental interpretation and iterative evaluation, both of which we are actively working on.
We are currently collecting examples from GMG in this publicly available spreadsheet. If you come across errors of objective generalization in AI research, we invite you to provide examples here.