Neighborhood walkability contributes to older adults’ walking. However, associations vary depending on the neighborhood definition applied as well as between objective and perceived walkability measures. Therefore, this study aimed to comparatively assess walkability indices for commonly used pedestrian network buffers and perceived neighborhood areas. A total of 97 adults aged ≥65 years answered a written physical activity questionnaire and 69 respondents participated in face-to-face interviews that involved mental mapping, i.e., to draw perceived neighborhood delineations on paper maps. Hierarchical regression analyses were used to compare the contribution of walkability indices for pre-set buffers and self-defined neighborhoods to older adults’ walking after adjusting for covariates. Results show that older adults’ self-defined neighborhoods are significantly larger, less home-centered, and more walkable than commonly used buffers. Furthermore, the variance accounted for in neighborhood walking increased from 35.9% to 40.4% (ΔR2 = 0.046; p = 0.029), when the walkability index was calculated for self-defined neighborhoods rather than pre-set buffers. Therefore, the study supports that geometric differences between pre-set buffers and older adults’ spatial ideas of perceived neighborhoods have a significant influence on estimated walkability effects and that exposure areas should be matched with the spatial dimension of outcome variables in future research.