Sri Lanka was reeling at 107 for eight chasing 240 to win when Mathews and Malinga combined in the 26th over. Together they put on the highest ninth-wicket partnership ever in an ODI and ensured a tie before Malinga was run out for 56 with the total at 239-9.
No 11 Muttiah Muralitharan calmly hit the winning runs with 34 balls to spare - a glanced boundary off Shane Watson in the next over - leaving Mathews unbeaten on 77.
It was a sixth consecutive defeat for Australia in all formats of the game. Long ranked No 1 in test and one-day cricket, Australia has now lost three consecutive tests - one to Pakistan and two to India - before an ODI defeat in India and a lopsided loss to Sri Lanka in Perth on the weekend in a Twenty20 International.
The Australians have two more ODIs against Sri Lanka before the five-test Ashes series against England starts in Brisbane next month.
In front of a Melbourne Cricket Ground crowd that seemed to feature more Sri Lankans than anyone else, Mathews hit eight boundaries and a six from 84 balls to run the match around. Malinga blazed two sixes and six boundaries from 48 balls in a brilliant cameo that netted his highest ODI score.
The Sri Lankan tailenders beat the ODI record 126 partnership for the ninth wicket which India's Kapil Dev and Syed Kirmani had held since the 1983 World Cup.
After winning the toss and electing to bat, Australia reached 239 for eight from its 50 overs with Mike Hussey remaining unbeaten on 71 and Thisara Perera doing the damage for Sri Lanka with a haul of 5-46 from eight overs.
Xavier Doherty completed a run out and then took four wickets on his ODI debut to have the visitors reeling at 84-5 before Mathews started to assert his influence on the match.