Action plans are not generated from scratch for each movement, but features of recently generated plans are recalled for subsequent movements. This study investigated whether the observation of an action is sufficient to trigger plan recall processes. Participant dyads performed an object manipulation task in which one participant transported a plunger from an outer platform to a center platform of different heights (first move). Subsequently, either the same (intra-individual task condition) or the other participant (inter-individual task condition) returned the plunger to the outer platform (return moves). Grasp heights were inversely related to center target height and similar irrespective of direction (first vs. return move) and task condition (intra- vs. inter-individual). Moreover, participants' return move grasp heights were highly correlated with their own, but not with their partners' first move grasp heights. Our findings provide evidence that a simulated action plan resembles a plan of how the observer would execute that action (based on a motor representation) rather than a plan of the actually observed action (based on a visual representation).