The government was not immediately threatened, because it holds a majority in the more powerful lower chamber, but the result makes it more difficult to pass laws and will force it to seek new coalition partners.
The election result -- the first ballot box test since Kan's party swept to power under a previous leader in a landslide poll last summer -- complicates his ambitious reform plans for the world's number two economy.
When Kan took office a month ago as Japan's fifth prime minister in four years, he pledged to restore the nation's vigour after two decades of economic malaise and to whittle down a huge public debt mountain.
The one-time leftist activist also promised to strengthen the social safety net for the rapidly ageing society and raised the prospect of tax hikes to pay for it all -- a gamble that backfired badly on election day.
If Kan, the 63-year-old former finance minister and self-declared "son of a salaryman", or man of the people, was looking for a strong mandate from Japan's more than 100 million eligible voters, he was left disappointed.
His Democratic Party of Japan (DPJ) will hold no more than 113 out of the 242 seats in the House of Councillors -- far short of the 122 seats needed for a majority -- according to an exit poll by public broadcaster NHK.
Other television stations forecast even worse results for the coalition government, which now includes one other small party, meaning it will have trouble pushing laws through the bicameral parliament.
Instead, Kan's government will have to engage in coalition talks to seek the support of smaller parties, with pundits pointing at the Buddhist-backed New Komeito Party and the year-old Your Party as the most likely contenders.
It's a disappointing state of affairs for the party that less than a year ago took power in what was widely hailed as an electoral earthquake that ended more than half a century of almost unbroken conservative rule.
When the DPJ took power last September under former premier Yukio Hatoyama, it promised to end the murky backroom politics of the business-friendly Liberal Democratic Party (LDP) and the powerful state bureaucracy.
Hatoyama pledged to return power to the people, make the capitalist powerhouse a kinder, gentler society, and forge less subservient ties with the United States, Japan's main security ally since the end of World War II.