ISI head Lt Gen Ahmed Shuja Pasha sat in on the key meeting held in Washington on Friday, the Dawn newspaper quoted diplomatic sources as saying.
Pasha was not in Zardari's entourage that arrived in the US on Thursday.
Pasha travelled alone to the US, participated in the President's meeting with Panetta but "stayed away from other activities", the report said.
The ISI chief's visit to the US took place against the backdrop of a court in Brooklyn issuing notices to Pasha, his predecessor Lt Gen Nadeem Taj and Lashker-e-Taiba leaders including Hafiz Muhammad Saeed in connection with a lawsuit filed by relatives of two Jewish victims of the 2008 Mumbai attacks.
The Pakistan government has said it will protect the interests of all officials named in the lawsuit but skirted the issue of defending private individuals like Saeed.
Zardari left Washington on Saturday evening at the conclusion of his visit during which he attended a memorial service for late Special Envoy Richard Holbrooke and met President Barack Obama with Pakistan's Ambassador Husain Haqqani.
Though Obama came to the meeting with his entire national security and counter-terrorism team, Zardari did not take any other official with him, causing wild speculation in the media about the purpose and contents of his talks with the US President, the Dawn reported.
Zardari also met other US officials and lawmakers but his meeting with the CIA chief was considered the most important after the one with Obama at the White House.
After the Zardari-Obama talks, the White House issued a statement that said the two leaders focused on shared efforts to fight terrorism and to promote regional stability.
No statements were issued after Zardari's meeting with the CIA chief, which led to speculative reports that Panetta had come with plans to expand drone attacks to areas in and around Quetta where the Americans believe senior Afghan Taliban leaders are hiding.
Pakistani and American officials are refusing to disclose the topics discussed at the meeting.
"We do not discuss intelligence matters," said Ambassador Haqqani.