Molecular technologies have revolutionized our classification of animal mating systems, yet we still know very little about the genetic mating systems of many vertebrate groups. It is widely believed that anuran amphibians have the highest reproductive diversity of all vertebrates, yet genetic mating systems have been studied in <1% of all described species. Here, we use single nucleotide polymorphisms to quantify the genetic mating system of the terrestrial breeding red‐backed toadlet Pseudophryne coriacea. In this species, breeding is prolonged (approximately 5 months), and males construct subterranean nests in which females deposit eggs. We predicted that females would display extreme sequential polyandry because this mating system has been reported in a closely related species (P. bibronii). Parentage analysis revealed that mating success was heavily skewed towards a subset of males (30.6% of potential sires) and that nearly all females (92.6%) mated with one male. In a high percentage of occupied nests (37.1%), the resident male was not the genetic sire, and very few nests (4.3%) contained clutches with multiple paternity. Unexpectedly, these results show that sequential polyandry is rare. They also show that there is a high frequency of nest takeover and extreme competition between males for nest sites, but that males rarely sneak matings. Genetic analysis also revealed introgressive hybridization between P. coriacea and the red‐crowned toadlet (Pseudophryne australis). Our study demonstrates a high level of mating system complexity, and it shows that closely related anurans can vary dramatically in their genetic mating system.