The men, some English and some from eastern Europe, were found in "filthy and cramped" conditions. Police believe some may have been held for as long as 15 years, The Telegraph reported.
More than 200 officers swooped on the Greenacres travellers' site in Leighton Buzzard during the early hours of Sunday and found the large group of men living in squalid conditions.
The raid was launched as part of a long-running investigation by Bedfordshire Police which suggested the men were being held against their will in poor conditions at the site, and forced to work for no pay.
The five suspects, all residents of the site, were detained on suspicion of slavery offences and are being held in custody at police stations across Bedfordshire and Hertfordshire, police said.
Weapons, drugs and money were also found at the site, the BBC quoted police as saying.
"The men we found at the site were in a poor state of physical health and the conditions they were living in were shockingly filthy and cramped," Detective Chief Inspector Sean O'Neil said.
"We believe that some of them had been living and working there in a state of virtual slavery, some for just a few weeks and others for up to 15 years.
"They're recruited and told if you come here we'll pay you 80 pounds a day, we'll look after you give you board and lodgings.
"But when they get here, their hair is cut off them, they're kept in in some cases [in] horse boxes, dog kennels and old caravans, made to work for no money, given very, very small amounts of food," he said.
"That's the worse case. Some are treated a little bit better but they were told they could not leave and if they did they would be beaten up and attacked," O'Neil said.