Following De Ruiter (2000), I propose that there is no such thing as a lexical affiliate for every gesture. I suggest interpreting gestures by their conceptual affiliates. The existence of conceptual rather than lexical gesture affiliates is supported by empirical data from a perception study with German native speakers. They linked gestures in video clips without sound to their accompanying speech that was in a separate audio clip. The manifold lexical connections people made could be united when regarding them as parts of conceptual perceptions. Also, the conceptual affiliates are closely connected with the theme-rheme structure of utterances. This connection further supports the intricate relationship of speech and gesture. The phenomenon of conceptual affiliates implies a less restricted and more natural perception process of co-expressive gesture and speech than the long-held idea of lexical gesture equals.