Inflorescence and spikelet development determines grain yields in cereals. Although multiple genes are known to be involved in regulation of floral organogenesis, the underlying molecular network remains unclear in cereals. Here, we report that the rice (Oryza sativa) microRNA OsmiR396d and its OsGRF targets, together with OsGIF1, are involved in the regulation of floral organ development via OsJMJ706 and OsCR4. Transgenic knockdown lines of OsGRF6, a predicted target of OsmiR396d, and overexpression lines of OsmiR396d showed similar defects in floral organ development including open husks, long sterile lemmas and altered floral organ morphology. These defects were almost completely rescued by overexpression of OsGRF6. OsGRF6 and its ortholog OsGRF10 were the most highly expressed OsGRF family members in young inflorescences, and the grf6/grf10 double mutant displayed abnormal florets. OsGRF6/10 localized to the nucleus, and electrophoretic mobility shift assays revealed that both OsGRF6 and OsGRF10 bind the GARE element in the promoters of OsJMJ706 and OsCR4, which were reported to be participated in the regulation of floral organ development. In addition, OsGRF6 and OsGRF10 could transactivate OsJMJ706 and OsCR4, an activity that was enhanced in the presence of OsGIF1, which can bind both OsGRF6 and OsGRF10. Together, our results suggest that OsmiR396d regulates the expression of OsGRF genes, which function with OsGIF1 in floret development via targeting of OsJMJ706 and OsCR4. This work thus reveals a miRNA-mediated regulation module for controlling spikelet development in rice.