Knowledge of graders can mediate selection for grader-targeting reasoning through RL

TL;DR: We insert knowledge of specific grading processes via SDF. We then show that models RL’d on environments with these graders exhibit grader-targeting reasoning, and generalise to optimising held-out environments with graders that were described in the SDF mix but not trained on during RL. Figure 1: Outline of our experiments. (Left) We use SDF to instil knowledge of grading on different subsets of environments. (Middle) We then perform RL on one half of the set of environments, which are In-Distribution (ID) for RL. We observe that models with ID grader knowledge exhibit faster learning

• #knowledge
• #selection