The use of Personal Voice Assistants (PVAs) such as Alexa and the Google Assistant is rising steadily, but there is a lack of research investigating common issues and requests of PVAs in the context of smart home control. We address this research question with an online survey (n = 65), using a qualitative evaluation of users’ problems and improvement requests. Our analysis leads to a partly hierarchic clustering of issues & recommendations for interaction capabilities of PVAs into seven basic categories, allowing us in turn to derive implications and to condense them into design guidelines for future Human-Agent Interaction (HAI) with PVAs. Specifically, we formulate and elaborate the concepts Authentication & Authorization, Activity-Based Interaction, Situated Dialogue, and Explainability & Transparency as key topics for making progress towards smooth interaction with smart homes.