Washington(CNN) Former FBI Director James Comey is "not worried about any tapes" of conversations between him and President Donald Trump, a source familiar with the matter told CNN Friday, adding that "if there is a tape, there's nothing he is worried about" that could be on it.
Friday morning, Trump issued a thinly veiled threat to Comey, apparently suggesting there are possibly recorded conversations between the two men that could be leaked to counter the former FBI director if necessary.
"James Comey better hope that there are no 'tapes' of our conversations before he starts leaking to the press," Trump tweeted.
On Friday, CNN's Jake Tapper reported that Comey was "taken aback" by Trump's request for a personal assurance or pledge of loyalty at a dinner shortly after he took office. Comey refused, saying he could not make such a pledge, but he promised to always be honest with the President.
White House press secretary Sean Spicer on Friday rejected the idea that Trump asked Comey to pledge his loyalty. "I think the President wants loyalty to his country and to the rule of law," Spicer said.
The source said of the dinner between Comey and Trump on January 27 that it came at Trump's request and was one-on-one. Trump told NBC he thought Comey asked for the dinner, but the source denied the President's account, saying that if Comey had wanted to dine with Trump, he would have had to put the request in through the deputy attorney general.
"It doesn't work that way. It's ludicrous," the source said.
A few days before the dinner, Comey had already been told by the President that he would stay on the job, the source said.
"It is absolutely untrue that Jim asked to have dinner or that he asked to have his job," the source said. "That is a complete fabrication."
As CNN has previously reported, Comey's refusal to pledge loyalty is one of the reasons the President fired him, the other being the acceleration of the probe into alleged collusion between Trump's presidential campaign and Russia, another source close to Comey has said.
The source said Comey knew it was possible he would lose his job, especially when he did not agree to pledge loyalty and later with the fallout over refuting the President's claims that President Barack Obama ordered the wiretapping of Trump Tower, but he didn't expect it when he was fired.
The source said Comey is likely not going to testify next week to Senate intelligence. As of Friday morning, Comey had not responded to the request.
"He loved his job and the mission," said the source, who added, "he'll be fine."
The source also confirmed previously reported accounts that Comey learned of his firing by catching it on TV -- initially, Comey thought it was a joke, and teased agents around him for teasing him. Then someone pull him aside and told him it was true, the source said.
The source said Comey thought at least he would get a phone call, adding, "the way (Trump) did it was remarkable."
The initial, official White House version of how Comey came to be fired was that deputy Attorney General Rod Rosenstein, fresh on the job, wrote a memo expressing concern about the way Comey had handled the Hillary Clinton email investigation.
But mounting evidence suggests Comey was actually fired because of the Russia probe.
Sources have told CNN that Trump's decision to ax Comey was made after he grew increasingly frustrated with him following a congressional hearing last week in which he said he was "mildly nauseous" over the idea that he helped sway the 2016 election.
Trump's surrogates, including Vice President Mike Pence, have struggled to keep up with the shifting narrative on how and why the decision was made, and Trump tweeted Friday it was "not possible" for his team to recount details and talking points with "perfect accuracy."
On Thursday, Trump, discussing the firing of Comey, told NBC News that he was frustrated by the ongoing investigation and believed it was motivated by Democrats' fury at losing the election.
Trump told NBC's Lester Holt: "And in fact when I decided to just do it, I said to myself, I said 'you know, this Russia thing with Trump and Russia is a made-up story, it's an excuse by the Democrats for having lost an election that they should have won.'"
Trump also reiterated in the interview Thursday that Comey told him on three separate occasions that he was not under investigation for his campaign's alleged ties to Russian intelligence officials.
"He told me that ... and I've heard that from others," Trump told Holt.
The President said Comey told him he was not under investigation once during a "very nice dinner" and two other times over the phone -- comments consistent with the letter Trump sent Comey informing him that he had been fired: "While I greatly appreciate you informing me, on three separate occasions, that I am not under investigation, I nevertheless concur with the judgment of the Department of Justice that you are not able to effectively lead the Bureau," Trump wrote.
There are still questions, however, as to whether Comey actually did tell Trump he was not under investigation at the dinner on January 27 and two more times by phone. Those close to Comey tell CNN they find it hard to believe he would speak about an ongoing investigation. Still, the conversations were one-on-one, making it impossible for the sources to know for sure.
One source was unwilling to tell CNN what was said about the investigation except that what was discussed was more nuanced than the way the President described it.
Senate Judiciary Committee Chairman Chuck Grassley said as well in remarks at a committee markup Thursday that he and ranking member Dianne Feinstein "heard nothing that contradicted the President's statement" when Comey briefed them about the targets of the various investigations. Grassley added: "It would not be appropriate for me to reveal those details before the professionals conducting the investigations are ready."