Panetta visited the National September 11 Memorial park and museum at the World Trade Centre site here as the US readies to commemorate the 9/11 anniversary.
"They continue to plan attacks and I do not think we can take anything for granted," Panetta told reporters after visiting Ground Zero.
"The potential for that kind of attack remains very real," he said adding that Americans must be vigilant against this threat.
He, however added that with some of al-Qaeda's top leaders, including Osama bin Laden now dead, the terror outfit has weakened in the decade following the 9/11 strikes that killed about 3000 people.
"We can take great pride that since 9/11 we have achieved very significant success in going after Al-Qaeda, in going after Al-Qaeda’s leadership," he said adding that it is critical to maintain pressure on the terrorist organisation.
Panetta cautioned that sections of the Al-Qaeda outfit remain active outside Afghanistan and Pakistan in places like Somalia and Yemen.
"Yemen has risen to the top of the list," he said.Panetta added that the arrest of senior al-Qaeda leader Younis al-Mauritani in Quetta, Pakistan "is a tribute to the Pakistanis who worked with us in this effort."
"It is an encouraging sign that we have gotten the Pakistanis to work with us in this kind of effort," he said.
Accompanied by New York Mayor Michael Bloomberg, Panetta toured the 8-acre memorial, which centers on the sites of the former towers.
It has now been transformed to square, granite reflecting pools, each about an acre in size.
Each pool is fed by four 30-foot waterfalls that descend from ground level, and the pools drain into a 'center void' at the bottom of each.
The theme of the pools is 'reflecting absence'.The site also has the 'Survivor Tree', a pear tree that was rescued from the World Trade center site after the terrorist attacks.
Names of all the people who died in the attacks have been inscribed in bronze panels that border the reflecting pools at the site.
Panetta said the terrorist attacks of a decade ago brought the nation together in a commitment that such horror "will never happen again."
"As tragic as 9/11 was, we have drawn tremendous inspiration (from it)," he said, adding that those who attacked the US in an attempt to weaken the country actually made it stronger