BlackRock and Vanguard are the biggest investors in private prisons

New York(CNN Business) Prison operators have been thrust into the spotlight recently since many of them run the controversial immigrant detention centers near the US border with Mexico.

Two of these companies, CoreCivic (CXW) and GEO Group (GEO), are publicly traded -- so many may hold them in an index fund or ETF as part of their portfolio or retirement account like a 401(k) or IRA without even knowing it.

Vanguard and iShares owner BlackRock (BLK) are the two largest holders of CoreCivic and GEO.

Both stocks are held mainly in passive funds that mimic indexes that include the companies, such as the S&P Mid Cap 400, Russell 2000 and the benchmark real estate MSCI US REIT Index, as opposed to funds where managers choose to buy them.

State Street Global Advisers (STT), which runs the popular SPDR series of index funds and ETFs, Prudential's (PRU) QMA unit and mutual fund giant Fidelity are also significant holders of CoreCivic and GEO in various mutual funds and ETFs.

It's not yet clear if big investment firms will distance themselves from the prison industry, similar to what iShares and several other fund managers did with gun stocks American Outdoor Brands (AOBC) (formerly Smith & Wesson) and Sturm Ruger (RGR) in the wake of a series of mass shootings in America during the past few years.

Vanguard said in a statement to CNN Business that the prison stocks make up a "very modest" portion of the company's portfolios, adding that it would be impractical to eliminate all controversial stocks from funds.

"We believe it would be exceedingly difficult to manage our funds effectively and efficiently while seeking to address the many social, political, and environmental concerns of our 20 million clients and the broader global community," the company said.

BlackRock/iShares did not respond to a comment about whether it would look to exclude CoreCivic and GEO from index mutual funds and ETFs.

iShares has a socially responsible ETF tied to the S&P Mid Cap 400 index called the MSCI KLD 400 Social (DSI) fund which does not hold GEO and CoreCivic.

State Street, Prudential and Fidelity were not immediately available for comment about their funds' ownership of CoreCivic and GEO.

But the controversy surrounding the migrant crisis is having a broad impact on the financial services industry.

Many big banks, including Wells Fargo (WFC), JPMorgan Chase (JPM) and, most recently, Bank of America (BAC), are cutting back on business relationships with CoreCivic, GEO and other privately held owners of prisons and detention centers.