The man, identified only as Nang, then hanged himself.
Police said the crime, which took place Thursday in south-eastern Svay Rieng province near the border with Vietnam, was among the worst the country had seen in years.
Provincial police chief Prach Rim told the Phnom Penh Post newspaper the victims were related to Nang.
"The attacker had raped his sister-in-law twice before, and he tried to rape her again, but was apparently stopped by his family," he said.
"They then fought together, and the attacker locked the doors of the house and stabbed his family members to death before hanging himself," Prach Rim said.
Police said Nang murdered his wife, his 2-year-old son, his mother-in-law and two of his sisters-in-law. His daughter and nephew were injured, as was the sister-in-law he tried to rape.
Last month a drunken soldier murdered three people and injured four during a rampage in nearby Kampong Cham province after arguing with his wife.
The head of one prominent local human rights organisation said he was worried the issue of domestic violence is not being taken seriously in parts of Cambodia.
"It seems everyone sees this kind of violence as normal," said Thun Saray, the head of Adhoc.