DENVER -- The family of Andrew Speaker, the 31-year-old Atlanta attorney in isolation at a Denver hospital with a rare and dangerous form of tuberculosis, is speaking out.

In an exclusive interview with ABC's Good Morning America, Cheryl and Ted Speaker and Robert and Betsy Cooksey said they are "in hell" and "we just want to get out of hell."

They said they are frustrated that while on the surface it appears as though Speaker was reckless in his decision to travel to and from Europe, they claim that the Centers for Disease Control and Prevention gave them no indication that Speaker was contagious with his drug-resistant TB.

Speaker's parents and in-laws said he never would have sat on the long-haul, trans-Atlantic flights if he thought he was contagious.

Both his father and new father-in-law, who works for the CDC, attended Andrew's meeting with health officials and claim the doctors said they would "prefer" Speaker not travel but that he was not contagious. They also claim to have the audiotape that proves it.

Ted Speaker said that they asked time and time again if his son was contagious. Each time, they said the answer was no.

"They said on tape that it was not contagious ... at least three times," said Ted Speaker.

Ted Speaker said he records most important conversations because he has hearing loss from his years as a helicopter pilot in Vietnam. He plans to release the tape at some point and said he believes it will confirm the family's side of the story.

But the CDC said just a few days before the Speakers and Cookseys left for the European wedding, a letter was sent to Speaker indicating he should not travel. The letter, however, never made it to Speaker.

While in Europe, the CDC called and told Speaker to isolate himself in Rome.

"They said, 'You will need to turn yourself in to health authorities in Rome and face quarantine until this is cured. We don't know that there's a cure. They're doing experimental stuff in Denver,'" Cheryl Speaker said.

The families claim that's when the CDC walked away from the problem. The CDC now said they were going to help.

"I can assure you that if we had one once, any smidgen that one once that that there was any plan to bring him home, that's all we needed. We had nothing," Betsy Cooksey said.

The families said they tried to reach high government officials and congressmen but instead an official with the CDC, Dr. David Kim, told them there only recourse was (to spend) $100,000 of their own money to get a private plane.

"And we said to Dr. Kim, 'We don't have $140,000.' He said to us on the telephone, 'So?' That's exactly what he said. And when he said that I knew that his ethics were down in the dumps," Ted Speaker said.

"We're not rich people. We're civil servants. I'm a school teacher. My husband works in government," Betsy Cooksey said.

Speaker's father-in-law, Robert Cooksey, responded to allegations that his work with tuberculosis could have led to the TB case.

"It's impossible for me to have given him tuberculosis," Cooksey said.

Andrew Speaker's mother said of Cooksey, "Do you think that this man, who is an expert in TB, do you think that this man, if he had thought that there was any danger whatsoever to his only child and his only grandchild, that he would've allowed this to happen?"

The CDC said it has made contact with half of the 292 passengers on Speaker's May 12 flight from Atlanta to Paris. All of the 26 passengers who sat in the five rows near him have been notified.

A spokesman National Jewish said Andrew Speaker is doing fine and could be at the Denver facility for two months. William Allstetter said that two of three sputum tests have come back negative. If a third sputum test comes back negative, Speaker will be deemed relatively noncontagious, Allstetter said.

Speaker has XDR-TB, a drug-resistant form of TB. He is taking a variety of antibiotics and after several weeks on the drugs, doctors will evaluate if surgery is needed to remove the infected portion of his lung. The infected portion is about the size of a tennis ball, his doctor said.

The federal government has lifted its quarantine on him but the state handed down its own, which will keep speaker in isolation until further notice.

So far, Speaker has undergone three sputum tests at National Jewish. Test results for the first two show were negative. Doctors are still awaiting results from the third test.

The patient's primary physician, Dr. Gwen Huitt said, "Three consecutive negative smear tests indicate that a patient is relatively non-contagious."

In a news release e-mailed early Monday, hospital spokesman William Allstetter said, "Generally, multidrug resistant TB patients at National Jewish are allowed to leave their rooms periodically for short periods of time, wearing an N95-rated mask, to walk outside briefly with an escort and get some fresh air. Drug-resistant TB patients who do venture outside are kept far from patients or any other community members, and pose no threat of infecting others."

The release goes on to say, "Mr. Speaker’s case is far from the typical case at National Jewish, both because of the disease and the media interest. No decision has been made about when or if he will leave his room. We want to emphasize that Mr. Speaker has been completely cooperative with his physicians in making plans for his treatment and movement."

Dr. Huitt said she saw the patient briefly Monday. "He was once again on his exercise bike with our physical therapist. He ate breakfast. He was in very good spirits."

When asked about his medication, Huitt said, "I believe he's up to four (pills) today, so we gradually increase them over a few days."

She said he's experienced no side effects from the drugs so far. "He's tolerating everything very well and hopefully it will stay that way."

Speaker was the first person put under federal quarantine since 1963.