A video circulating on social media shows guards escorting Israel's deputy director general for Africa, Sharon Bar-li, out of the AU assembly taking place in the Ethiopian capital Addis Ababa.
A spokesman for the Israeli foreign ministry described the incident as "severe", noting Bar-li was "an accredited observer with an entry tag".
"It is saddening to see the African Union taken hostage by a small number of extremist states like Algeria and South Africa, which are driven by hatred and controlled by Iran," the spokesman alleged.
African states should "oppose these actions, which harm the African Union movement and the entire continent," the spokesman said.
The African Union has not responded to an AFP request for comment about the incident.
Asked about Israel's accusations that South Africa and Algeria were behind the move, South African President Cyril Ramaphosa's spokesman Vincent Magwenya told AFP at the summit: "They must substantiate their claim."
Israel attained observer status at the AU in 2021 after decades of diplomatic efforts, drawing protest from powerful members including South Africa and Algeria, which argued that it flew in the face of AU statements supporting the Palestinian people.
Last year, unease flared over the accreditation of Israel as an observer at the AU, with Palestinian Prime Minister Mohammed Shtayyeh urging it to be withdrawn.
The 2022 summit suspended a debate on whether to withdraw the accreditation and a committee was formed to address the issue.
The row erupted when Moussa Faki Mahamat, head of the African Union Commission, accepted Israel's accreditation, triggering a rare dispute within a body that values consensus.
The AU has not said whether Israel's status would be up for discussion at this year's summit.