The official also said Broadwell did not have a security clearance from the CIA.
Another official said Broadwell, who is an officer in the Army reserve, did have some kind of security clearance and that there are no issues with Broadwell having unauthorized access to classified information.
Petraeus' resignation also presents challenges to the congressional inquiry into the Benghazi attack.
CNN has confirmed that Petraeus recently traveled to Libya to meet the CIA station chief to discuss the attack. He was scheduled to testify before a congressional committee this week on the assault and the U.S. government response to it.
That now will not happen, but it is possible that he could be summoned by Congress to testify later.
Some Republicans have criticized the administration's response to the Benghazi attack and have speculated that Petraeus' departure was linked to the congressional inquiry.
Rep. Peter King, R-New York, said elements of the story "don't add up." He called Petraeus "an absolutely essential witness, maybe more than anybody else."
However, a senior U.S. official said Petraeus' departure wasn't connected to the Benghazi hearing.
"Director Petraeus' frank and forthright letter of resignation stands on its own," said the official, who spoke on the condition of anonymity because of the sensitivity of the topic. "Any suggestion that his departure has anything to do with criticism about Benghazi is completely baseless."
Congressional leaders are calling for an explanation of why they weren't notified sooner of the FBI's inquiry when it became clear Petraeus was involved.
Leaders of the House Intelligence Committee are expected to meet Wednesday with acting CIA Director Mike Morell and FBI Deputy Director Sean Joyce to discuss the Petraeus investigation and congressional oversight.
Sen. Diane Feinstein, the Democratic chair of the Senate Intelligence Committee, said on "Fox News Sunday" that she would "absolutely" push for answers.
"I mean, this is something that could have had an effect on national security," she said. "I think we should have been told. There is a way to do it. And that is, just to inform the chair and the vice chairman of both committees, to -- this has happened before, not with precise, same things, but, none of the four of us have ever breached that confidentiality."
On Monday, Feinstein told NBC that her concern about the situation "has actually escalated the last few days."
"...A decision was made somewhere not to brief us, which is atypical," Feinstein said. "This is certainly an operationally sensitive matter. But we weren't briefed. I don't know who made that decision."
The FBI investigation began when Kelley went to FBI officials to complain that Broadwell was sending harassing e-mails to her, a U.S. official told CNN. Kelley received the worrisome e-mails in May, an official said, describing the messages as along the lines of "stay away from my guy," but not explicitly threatening.
According to a source with knowledge of the e-mails, the messages accused Kelley of untoward behavior with some generals at MacDill Air Force Base in Tampa, Florida where Kelley did volunteer work.
The e-mails detailed the "comings and goings of the generals and Ms. Kelley," said the source, who declined to speak on the record because of sensitivity of the investigation.
Among those believed to be referenced in the e-mails was Petreaus. Because parts of Petreaus' schedule were not public, the e-mails raised questions about whether the sender of the e-mails had access to his private schedule or other sensitive information.
The content of the e-mails was first reported by NBC News.
At one point, Petraeus told Broadwell to stop sending the e-mails, a U.S. official said. It was not clear whether his request was made during or after his affair with Broadwell.
During the investigation, other communications surfaced between Petraeus and Broadwell, a married mother of two, according to a U.S. official.
Petraeus used a personal account to e-mail Broadwell, not his CIA account, a U.S. official said.
The FBI interviewed Petraeus, said the official, who stressed that the CIA director was never the target of the investigation and his communications were never compromised.