This makes the climate the biggest driver of internal displacement for the period, with the world's poorer countries at the highest risk, despite their smaller contributions to global carbon pollution compared to richer nations.
People are seven times more likely to be internally displaced by floods, cyclones and wildfires than volcanic eruptions and earthquakes, and three times more likely than by conflict, according to the report released Monday,
The issue is one of a raft of topics set to be discussed at the United Nations Climate Change Conference COP 25, which starts on Monday in Madrid.
Oxfam is calling on the international community to do more to fund recovery programs for poorer countries affected by the climate emergency, which is set to intensify as extreme weather events are projected to increase in both severity and frequency.
Low- and lower-middle income nations, such as India, are more than four times more likely to be affected by climate-fueled displacement than high-income countries like Spain and the US, according to the report.
Geography also plays a role, with about 80% of those displaced living in Asia.
Small island developing states (SIDS), such as Cuba, Dominica and Tuvalu, are particularly badly affected, making up seven of the top 10 countries with the highest rates of displacement from extreme weather disasters between 2008 and 2018.
People living in SIDs are 150 times more likely to be displaced by extreme weather disasters than those living in Europe, according to the report, which analyzed 2008-18 data from the Internal Displacement Monitoring Centre.
Internal displacement has a social cost as well as a financial one, according to Tim Gore, Oxfam's head of policy on climate and food justice.
"It's always the poorest, the most vulnerable, and women in particular, that are worst affected," Gore, who worked on the report, told CNN.
"This kind of displacement really tears at the social fabric of communities."
There is more acute risk in those countries where extreme weather and conflict combine, such as Somalia, said Gore.
Sudden extreme weather events such as cyclones grab a lot of attention, but slow onset phenomena like rising sea levels also have an impact, he added.
For example, floods affecting agricultural land in low lying coastal areas can leave it unusable for farming, pushing inhabitants to leave the area for good.
Oxfam is calling on world leaders to reduce emissions as fast as possible.
Developing countries are also set to push for support from developed countries through a financial mechanism to deal with loss and damage.
First discussed at a climate summit in Warsaw in 2013, such a mechanism will involve richer countries financially helping poorer countries to deal with the impact of climate change.
"Nobody has been prepared to talk about money and so that's one of the critical issues that will be on the table in Madrid," said Gore.
"Ultimately somebody is going to have to pay the price for these impacts and at the moment that price is being paid by the poorest communities in the world."
And while current data shows lower risk in developed nations, projections suggest that is set to change.
"Rich countries are not immune either from the threat of displacement," said Gore.
"Climate change is not going to discriminate."
Bob Ward, policy director at the Grantham Research Institute on Climate Change and the Environment, told CNN that increasing numbers of internally displaced people can be attributed in part to a growing population living in high-risk areas.
Displacement can also be a measure of success in some cases, he added, citing early warning systems that allow people to get out of danger before an extreme weather event hits, avoiding the major loss of life that has occurred in the past.
Ward also emphasized that climate-fueled displacement is a security issue.
"Although it's hard to show that climate change is in itself creating political instability and conflict, the way in which the national security community describes climate change is as a threat multiplier," he said.
"When you get large populations displaced that's when you get instability and conflict."