(CNN) Dr. Ronny Jackson, the embattled pick to lead the Veterans Affairs agency, will continue fighting for his nomination, according to senior administration officials, despite allegations of misconduct and hints from President Donald Trump that he could withdraw himself from contention.
The White House feels Jackson is being "railroaded" by the claims he oversaw a toxic work environment and drank to excess. Instead, the officials insisted Jackson would push back forcefully and said Trump was not wobbling over his selection. Trump met with Jackson on Tuesday in the Oval Office after saying it was "totally (Jackson's) decision" whether to continue the confirmation process.
The White House began publicly defending Jackson after the meeting, distributing praise from former administrations and claiming the raft of allegations came from a disgruntled ex-employee.
"Dr. Jackson's record as a White House physician is impeccable. He has improved unit morale, received glowing reviews and promotions under Republican and Democrat presidents," read a statement from a White House official, who declined to be named.
That confidence aside, Jackson's viability as a nominee appears in question. Multiple Republican and Democratic lawmakers have said the allegations are troubling, and his nomination hearing was postponed indefinitely.
Earlier Tuesday, Trump conceded the one-time White House doctor faces major problems.
"He is a high-quality person. It's totally his decision. So he'll be making a decision. I don't want to put a man through a process like this. It's too ugly and too disgusting," Trump said. "So, we'll see what happens. He'll make a decision."
Trump spoke by phone with Jackson earlier on Tuesday, two White House officials said, and told him it was up to him whether to withdraw.
"I told Admiral Jackson just a little while ago, what do you need this for? This is a vicious group of people," the President said. "What do you need it for?"
Trump said if he was Jackson making the decision, he would step aside.
"If I were him, I wouldn't do it," Trump said.
Trump was speaking as concerns mount over Jackson's nomination. Senate Veterans Affairs Committee members are assessing allegations from whistleblowers that have told the panel about Jackson's questionable behavior including excessive drinking and a "toxic" work environment under his leadership, according to two former White House medical staff members who have spoken with the committee.
Both sources who spoke with CNN told the committee about behavior they observed while working in the White House medical unit.
Trump said Tuesday he hadn't heard particular allegations against Jackson but defended him and said he was being unfairly treated by the press and Democrats, though the concerns about Jackson also come from Republicans.
Jackson, Trump insisted, "is one of the finest people I have ever met" but acknowledged he lacks prior background in running a government agency.
"There's a lack of experience," he said.
Though the White House publicly defended Jackson on Tuesday, several officials have privately conceded that the President's nominee to lead the Veterans Affairs department is in trouble. Officials are aware in detail of the allegations lobbed against Jackson, and are continuing to discuss what to do going forward.
But other officials fear Jackson's nomination is doomed. One said that having the White House defend him instead of simply pulling his nomination is worse, predicting more allegations will come out and the beloved West Wing physician will end up irrevocably tarnished.