Abu Muhsin al-Masri, an Egyptian national believed to be the group's number-two in the Indian sub-continent, was targeted in eastern Ghazni province, Afghanistan's intelligence agency said in a statement.
The National Directorate of Security did not provide further details about the operation or when it was carried out.
Masri's killing was an indication of the relationship between the Taliban and al-Qaeda, Afghanistan's Interior Minister Masoud Andarabi claimed on Sunday without providing details.
"The killing of one of the key Al-Qaeda member, al-Masri by the National Directorate of Security demonstrates close ties of the Taliban with the terrorist groups that are operating against the Afghan government and its people," he tweeted.
"They still keep close relations with the terrorist groups and they are lying to different sides."
The Taliban government's sheltering of al-Qaeda was the original reason for the US invasion of Afghanistan after the September 11, 2001 attacks.
Read also: Taliban endorses 'wise man' Donald Trump ahead of US presidential election
In a landmark deal between the United States and the Taliban in February, the Taliban agreed to not allow Afghan soil to be used by foreign extremists including al-Qaeda in return for Washington withdrawing its troops.
Masri, who also goes by the name Husam Abd-al-Ra'uf, is on the FBI's Most Wanted Terrorists list.
A US warrant for his arrest was issued in December 2018, after he was charged with providing support and resources to a foreign terrorist organisation and plotting to kill US nationals, according to the FBI.
A source at the Afghan intelligence agency, who did not want to be named, told AFP that an aide to Masri who was "in contact with the Taliban" was also detained during the operation in Ghazni province, a volatile region where Taliban fighters are very active.
Meanwhile, on Sunday the death toll from a bomb blast claimed by the Islamic State group at a higher education centre in Kabul rose to 24 - most of them students.
Interior Ministry spokesman Tariq Arian said another 57 people had been wounded in Saturday's attack at the centre in western district of the capital.
Students were seen on Sunday morning salvaging books from nearby accommodation that was also damaged in the blast
Residents in several districts of western Kabul belong to the minority Shia Hazara community, and are often targeted by Sunni extremists of IS.
Violence rages in Afghanistan as peace talks continue in Qatar between the Taliban and the Afghan government in a bid to end the country's long-running war.