Local adaptation to low temperature is essential for woody plants to against changeable climate and safely survive the winter. To uncover the specific molecular mechanism of low temperature adaptation in woody plants, we sequenced 134 core individuals selected from 494 paper mulberry (Broussonetia papyrifera), which naturally distributed in different climate zones and latitudes. The population structure analysis, PCA analysis and neighbor-joining tree analysis indicated that the individuals were classified into three clusters, which showed forceful geographic distribution patterns because of the adaptation to local climate. Using two overwintering phenotypic data collected at high latitudes of 40°N and one bioclimatic variable, genome–phenotype and genome–environment associations, and genome-wide scans were performed. We detected 75 selective regions which possibly undergone temperature selection and identified 14 trait-associated SNPs that corresponded to 16 candidate genes (including LRR-RLK, PP2A, BCS1, etc.). Meanwhile, low temperature adaptation was also supported by other three trait-associated SNPs which exhibiting significant differences in overwintering traits between alleles within three geographic groups. To sum up, a possible network of cold signal perception and responses in woody plants were proposed, including important genes that have been confirmed in previous studies while others could be key potential candidates of woody plants. Overall, our results highlighted the specific and complex molecular mechanism of low temperature adaptation and overwintering of woody plants.