(CNN) Harvey Weinstein's defense attorney doesn't want to represent him anymore and has filed to withdraw from the sexual assault case, according to a source.
Attorney Jose Baez filed papers Friday asking the court to let him withdraw from Weinstein's case, marking the first time in his career he's asked to withdraw from a case, a source close to the case tells CNN.
The move comes after the fallen movie mogul hired a different attorney to communicate with Baez, the source said.
"First, Mr. Weinstein has engaged in behavior that makes this representation unreasonably difficult to carry out effectively and has insisted upon taking actions with which I have fundamental disagreements," Baez said in the filing, according to the source.
The source said the hiring of another attorney has severed the bonds of trust between Baez and Weinstein.
"Mr. Weinstein has deliberately disregarded our fee agreement and has had outside counsel indicate that he intends to continue to do so," Baez noted in the filing, according to the source.
A Weinstein spokesman declined to comment.
Baez has defended Casey Anthony and Aaron Hernandez, winning acquittals for them both.
He is not the first attorney to have withdrawn as Weinstein's counsel in the criminal case. Ben Brafman left the case in January, and Baez and Harvard University lawyer Ronald Sullivan took over. Sullivan withdrew last month.
Attorney Pamela Mackey, who famously defended former NBA player Kobe Bryant against sexual assault charges, also withdrew from defending Weinstein, as did Duncan Levin, a former federal prosecutor.
Weinstein is accused of raping a woman in a New York hotel room in 2013 and forcibly performing oral sex on another woman at his Manhattan apartment in 2006. Weinstein has denied all accusations of nonconsensual sex.
He faces five felony charges, including two counts of predatory sexual assault, one count of criminal sexual act in the first degree, and one count each of first-degree rape and third-degree rape. He has pleaded not guilty to all charges.
His trial is set to begin September 9.