Coping with stress and making appropriate adjustments are important issues in the pressurizing modern environment. The city image is the representation of selection and exhibition and establishes collective memory in society. The city image is the interactions between residents and environments and the reflection of the living quality of a city, which influences the emotion of people. This study aims to characterize the genius loci and investigate the pattern between perceived naturalness, artifacts, and visual stimulation. The architectural buildings are the main part of the city image and influence people’s emotions and feelings. Understanding whether the design and index framework of building affect and change the user’s feelings trigger deeper psychological responses and stress relief is critical. We investigate the effect of 20 buildings’ images (public buildings, condominiums, and townhouses) and performed semi-structured interviews with 50 subjects to understand the stress-relief city image. The Semantic Differential Method with Five Points Image Scale was performed to examine the sensitivity of the collective memory. The evaluation focused on the concept of various styles, the visual of city images, and the Stress-Relief of city building form to obtain the forms of Stress-Relief Building Image. The results reveal the connection between architecture and people’s emotions and provide specific and clear references for the architecture designers.