Selaginellaceae, originated in the Carboniferous and survived the Permian–Triassic mass extinction, is the largest family of lycophyte, which is sister to other tracheophytes. It stands out from tracheophytes by exhibiting extraordinary habitat diversity and lacking polyploidization. The organelle genome-based phylogenies confirmed the monophyly of Selaginella, with six or seven subgenera grouped into two superclades, but the phylogenetic positions of the enigmatic Selaginella sanguinolenta clade remained problematic. Here, we conducted a phylogenomic study on Selaginellaceae utilizing large-scale nuclear gene data from RNA-seq to elucidate the phylogeny and explore the causes of the phylogenetic incongruence of the S. sanguinolenta clade. Our phylogenetic analyses resolved three different positions of the S. sanguinolenta clade, which were supported by the sorted three nuclear gene sets, respectively. The results from the gene flow test, species network inference, and plastome-based phylogeny congruently suggested a probable hybrid origin of the S. sanguinolenta clade involving each common ancestor of the two superclades in Selaginellaceae. The hybrid hypothesis is corroborated by the evidence from rhizophore morphology and spore micromorphology. The chromosome observation and Ks distributions further suggested hybridization accompanied by polyploidization. Divergence time estimation based on independent datasets from nuclear gene sets and plastid genome data congruently inferred that allopolyploidization occurred in the Early Triassic. To our best knowledge, the allopolyploidization in the Mesozoic reported here represents the earliest record of tracheophytes. Our study revealed a unique triad of phylogenetic positions for a hybrid-originated group with comprehensive evidence and proposed a hypothesis for retaining both parental alleles through gene conversion.