The family has claimed that they were guaranteed regular contact with Mercy James, when the 52-year-old adopted her from an orphanage in June 2009, reported Daily Mail online.
But they have not seen her since, despite Madonna having made two return visits to Malawi with Mercy and the family have now enlisted the help of Malawi's leading civil rights group CILIC to prepare a legal case.
"I believe they have a case in law because there appears to have been a verbal contract between them and Madonna's representatives. I am preparing a letter which will appeal to Madonna’s lawyer Alan Chinula to intervene on the family's behalf and ask Madonna to kindly let Mercy meet her family," said Emmie Chanika, director of CILIC.
The child, whose 16-year-old mother died five days after giving birth, was raised by her grandmother and uncles, but placed in the care of the Kondanani Children;s Village when they could no longer look after her.
Mercy's grandmother Lucy Chekechiwa, said the family always intended to bring Mercy back to live with them once she was six, when they believed her immune system would be strong enough to tackle the country's endemic diseases.
"The baby needed feeding and the orphanage offered me a wet-nurse to take care of that. We know Madonna gave a lot of money to the orphanage, and the people there persuaded us to let her have our child," said the 72-year-old.
Mercy was the second Malawian child adopted by Madonna, who has two older children from two previous relationships, 14-year-old Lourdes and Rocco, ten. In 2008 she adopted one-year-old David Banda from an orphanage in the capital city Lilongwe. He was taken back to meet his father Yohane Banda for the first time in three years in March 2009.