Marc Fonbaustier, 46, France's consul-general in the Chinese semi-autonomous territories, was forced to leave his job after Beijing sought his removal just over a year after taking the top-ranking post, the source said.
Another source in Hong Kong said the theft occurred at a private members' club.
According to a report in the Apple Daily newspaper, the wine was worth 50,000 Hong Kong dollars (6,440 US dollars).
Police and the French consulate in Hong Kong both declined comment today.
"We stand by the statement of the spokesman for the foreign ministry last week," a consulate spokeswoman told AFP.
On Thursday, the foreign ministry said an "administrative investigation" had begun, adding that Fonbaustier's unspecified actions would "likely not meet the requirements of professional conduct for a French diplomat".
"He is suspended from duty," ministry spokesman Bernard Valero said.
"An administrative investigation has been started based on the circumstances behind the decision."
Fonbaustier, who is married with three children, is an alumnus of the Ecole Nationale d'Administration, France's top school for civil servants. He is a former second counsellor at the French embassy in New Delhi.