Senate Chaplain Rev. Barry Black will no longer attend an inaugural prayer breakfast being hosted by religious conservatives, after a petition criticizing the event garnered thousands of signatures online.

House Chaplain Rev. Patrick J. Conroy, however, will attend the event. His office says the Jesuit priest will deliver a quick prayer, and won't "stay too long."

The breakfast, which will take place just hours before President Barack Obama is publicly into office Monday, features special guests Pat Robertson, a televangelist whose commentary has managed to offend gays, Haitians and feminists alike, Jonathan Cahn, a preacher who believes some of Obama's actions are apocalyptic, and Joseph Farah, the editor of WorldNetDaily who has long pushed the birther conspiracy theories about the president's citizenship.

[VIDEO: Little Known Inauguration Facts]

"The official chaplains of the U.S. House and Senate are supposed to be nonpartisan pastors who offer spiritual support and guidance to our nation's elected leaders," reads an online petition against the breakfast started by Faithful America, a progressive faith advocacy group. "But on Inauguration Day, they're scheduled to speak at a right-wing event in Washington alongside some of the most hateful and racist figures on the Christian right."

The petition has 12,0000 signatures as of this writing.

Black, a former chief of U.S. Navy chaplains, previously dropped out an evangelical conference in 2007 after he was pictured next to conservative columnist Ann Coulter in promotional material, the New York Times reported. The Senate Chaplain said at the time he felt his appearance would conflict with the job's "historic tradition of being nonpolitical, nonpartisan, nonsectarian."

[PHOTOS: Inaugurations Throughout History]

When reached by phone Friday, Black's office said the Senate Chaplain had decided against attending the inaugural breakfast, but declined to say if the controversy was the reason. Black's name is still listed as a featured guest on the event's website.

Also on the list: Republicans Rep. Michele Bachmann of Minnesota and Sen. Roy Blunt of Missouri. Neither lawmaker responded to a request for comment about their participation. An email sent to the breakfast's organizers was also not returned.

Elizabeth Flock is a staff writer for U.S. News & World Report. You can follow her on Twitter or Facebook or reach her at eflock@usnews.com.