Khan also said that besides the Kashmir issue, any terror activities in Kashmir or a 26/11 type attack has the potential of taking the relations back to "square one".
It is the "only issue" that is stopping the two countries from getting closer, he said.
"It's a core issue because without Kashmir you will always have a possibility of going back to square one no matter how much confidence-building measures you have," Khan said in an interview to Karan Thapar for CNN-IBN's "Devil's Advocate" programme.
"If something happens in Kashmir...maybe something happens from within Kashmir in India, some sort of terrorist attack – I'm scared that like Mumbai, certainly whatever happens comes back to square one," he said, referring to the possibility of another terror attack disrupting bilateral relations.
Responding to a question about President Zardari's remarks three years ago that the Kashmir issue be set aside for resolution by a future generation while India and Pakistan worked on CBMs, Khan said: "I certainly agree with that line of thinking. "I think that the more confidence we build with each other, the more trade we have, the more trust we develop and the moment militancy or intelligence agencies' roles disappear and the more we solve our issues on the dialogue table, that's the only way forward".
Though Zardari had said in an interview three years ago that the Kashmir issue should be set aside for resolution in the future, he subsequently made no reference to such an approach after facing criticism from Kashmiri groups and hardline elements. Asked how he would try to resolve the Kashmir issue, Khan replied: "Well, whatever the solution is, first of all, it should not be enforced through militancy.
"So whatever policy we have, it should be political dialogue and I also believe we should have other CBMs along the way". He said there should be greater trade, more linkages and people-to-people contacts on both sides of Kashmir.
Khan said the two countries should deal with the Kashmir issue in a "more civilised way" and suggested India could make a start by "withdrawing its army from Kashmir" if it "has confidence that there is no militancy coming from Pakistan".
Khan contended that the Kashmir issue hinges on the "rights of the Kashmiri people" and they should play a key role in finding a solution to the dispute. Khan remarked that his opposition to all military operations, including those within Pakistan, had led to some people describing him as "pro-Taliban".
He said he had called for the withdrawal of Pakistani soldiers from the restive tribal areas. "The solution does not lie in military operations...," Khan said.
Responding to another question whether his Pakistan Tehrik-e-Insaf party's policies would be dictated by anti-India leaders like Shireen Mazari, Khan said such matters would be decided through dialogue within the party.
The party, as a whole, objects to military operations everywhere, including Kashmir, he added. "Any political party has all sorts of people, all sorts of views come in...Shireen Mazari is a nationalist (but) Imran Khan has never been anyone's puppet ever," he said, adding that he would not become the premier if he could not decide his party's policies.
On the Afghan issue, Khan said Pakistan's perception of the relations between India and Afghanistan were closely linked to the evolution of India-Pakistan ties.
India's relations with Afghanistan would not matter if ties between New Delhi and Islamabad were "friendly" and based on mutual trust, he said. "But if it's perceived (as hostile) as it is right now by a military establishment which is basically controlling the policy... they will always be scared of being squeezed from two fronts," he said.
Asked about Pakistan's policy of using Afghanistan for strategic depth, he said: "I think this policy is wrong specifically because they don't understand the Afghan mindset.
"Afghans have never been controlled by anyone. They are the most independent-minded people which is why they fought every foreign occupation. Anyone who tries to control them would fail".