The split appeared to have taken place between 360,000 and a million years ago, say the researchers from the University of California, Los Angeles. These ancient humans had babies with the ancestors of present-day Africans, much as Neanderthals reproduced with the ancestors of modern Europeans, wrote geneticists Arun Durvasula and Sriram Sankararaman.
DNA from this archaic population makes up between 2% and 19% of modern West Africans' genetic ancestry, they said.
Well-established research, the study says, has established the existence of Neanderthal DNA in modern European populations, and Denisovan DNA in Oceanic populations.
The UCLA researchers said that "while several studies have revealed contributions from deep lineages to the ancestry of present-day Africans, the nature of these contributions remains poorly understood." This is in part because of sparse fossil records in Africa and the difficulty of obtaining ancient DNA.
The scientists from UCLA overcame these challenges to find the "ghost" of the hominin tribe by using computer modeling techniques on modern DNA.
They built "genome-wide maps of archaic ancestry" across four West African ethnic populations living in three countries: Nigeria, Sierra Leone and the Gambia.
Researchers compared 405 genomes of West Africans with Neanderthal and Denisovan genomes, and worked out whether there had been interbreeding among an unknown hominin whose ancestors split off from the human family tree before Neanderthals. Data suggests this may have involved multiple populations.
Professor Joel D. Irish, a bioarchaeologist at Liverpool John Moores University, told CNN there were probably many different ancient human populations.
"I think at one time, there'd have been all sorts of populations, with genetics different enough to look a bit different," he said.
"Everybody tends to mate with everybody. I think we're going to find more and more of these 'ghost' populations coming up."
After extracting the first genomes from the bones of Stone Age hominins, scientists discovered in 2010 that the early ancestors of Europeans, Asians and Americans reproduced with Neanderthals.
Last month, researchers from Princeton's Lewis-Sigler Institute for Integrative Genomics said they also detected Neanderthal ancestry in Africans for the first time.
Their data indicated that a wave of modern humans left Africa for Europe about 200,000 years ago and interbred with Neanderthals, before migrating back to Africa.
It cast doubt on the widely held "out of Africa" theory of human migration, meaning that modern humans originated in Africa and dispersed to the rest of the world in a single wave between 60,000 and 80,000 years ago.