(CNN) House Speaker Paul Ryan reiterated his support Wednesday for Donald Trump at a closed door meeting with House Republicans and asked his colleagues to unite behind the presumptive Republican nominee, according to several members who attended the session.
Ryan's backing of Trump comes a day after he blasted the businessman's remarks about a federal judge as a "textbook definition of a racist comment."
Trumps comments have sparked a fury on Capitol Hill where many Republicans worry that Trump's racially tinged criticism of Judge Gonzalo Curiel could ultimately hurt down-ballot GOP candidates competing in congressional races this year.
Though Ryan didn't back away from his endorsement, he said he would continue to speak out when he believed the billionaire businessman makes inappropriate statements.
Ryan spokesman said the speaker did not "urge" the party to fall in line behind Trump but stressed the importance of unity.
Ryan "just reaffirmed his disagreement with the comment, respecting that the people have elected Donald Trump as our Republican nominee and what we're going to do is we're going to use our position in a positive way to help America out," said Rep. Tom Reed, R-N.Y.
Trump tried to quell some of the outrage on Tuesday, releasing a statement saying his past remarks had been misconstrued, and Trump avoided the topic all together during a speech he gave later that evening.
Rep. Bill Flores, R-Texas, chairman of the Republican Study Committee, the largest conservative group in Congress, said Trump needs to "quit spending time bashing judges." He will vote for Trump, he said, but, "I don't endorse people that bash judges -- based on his ethnic heritage."
Rep. Mike Coffman, R-Colorado, who is facing a competitive race, has not endorsed Trump, and has said the presumptive nominee's comments on Curiel were "inappropriate."
"I just disagree with his tone. I just don't think you know his tone as far as Hispanics, his tone to women, and quite frankly, his tone to veterans. As a Marine Corps combat veteran," Coffman said.
Rep. Adam Kinzinger, R-Illinois, agreed with characterizations of Trump's comments as racist.
"I haven't endorsed him yet. And each day I say I want to get to the point of endorsing the Republican nominee. This makes it hard," Kinzinger said.
Ryan has faced criticism on both sides of the GOP civil war over Trump. The National Review's Jonah Goldberg on Wednesday slammed Ryan's endorsement of the billionaire businessman, saying his backing came without moving Trump any closer to conservatism.
"Because Trump did nothing to earn Ryan's endorsement, the presumptive nominee may conclude that he needn't negotiate with the GOP establishment; he can just count on its eventual submission," Goldberg wrote.