Pollen fertility is a primary regulator of grain yield and is highly susceptible to cold and other environmental stress. We revealed the roles of rice cell wall invertase gene PWIN1 in pollen development and chilling tolerance. We uncovered its preferential expression in microspores and bicellular pollen and identified its knock-down and knock-out mutants. pwin1 mutants produced a higher proportion of abnormal pollen than wild-type plants. The contents of sucrose, glucose, and fructose were increased, while ATP content and primary metabolism activity were reduced in the mutant pollen. Furthermore, the loss of function of PWIN1 coincided with an increase in SnRK1 activity and a decrease in TOR activity. Under chilling conditions, pwin1 mutants displayed significantly reduced pollen viability and seed-setting rate, while overexpressing PWIN1 notably increased pollen viability and seed-setting rate as compared with the wild-type, indicating that PWIN1 is essential for rice pollen development and grain yield under cold stress. This study provides insights into the molecular mechanisms underlying rice pollen fertility during chilling stress, and a new module to improve chilling tolerance of rice at the booting stage by molecular design.