COLORADO SPRINGS, Colo. -- Citing a lack of time, Focus on the Family founder James Dobson withdrew Tuesday from the team overseeing counseling for the Rev. Ted Haggard, the evangelical pastor who was fired amid allegations of gay sex and drug use.

"Emotionally and spiritually, I wanted to be of help -- but the reality is I don't have the time to devote to such a critical responsibility," Dobson said in a statement.

There was no immediate word on Dobson's replacement. The other two members of the team, Pastors Jack Hayford of The Church on the Way in Van Nuys, Calif., and Tommy Barnett of First Assembly of God in Phoenix, declined to comment.

Haggard was forced out as senior pastor of the 14,000-member New Life Church in Colorado Springs on Saturday after a former male escort alleged they had sex repeatedly and that Haggard sometimes was on methamphetamines during their trysts.

In a statement read at the church Sunday, Haggard confessed to unspecified "sexual immorality," accepted responsibility for his actions and asked forgiveness.

After the statement was read, some members of the congregation offered Haggard their forgiveness.

"Of course I forgive him whole heartedly, as I know God has forgiven him," Emily Raesides, a member of Haggard's church, said.

However, the counseling process, called restoration, could take years, said H.B. London, vice president for church and clergy at Focus on the Family, a Colorado Springs-based ministry. All this despite the some warm sentiments from his parish.

In a statement issued before Dobson's decision, Barnett said the counseling would focus on "personal restoration" and not returning Haggard to the ministry.