Scott Morrison has denied equating the conditions faced by First Fleet convicts with the experience of Indigenous Australians.
But the prime minister argues it is important to recognise the stories of all Australians on January 26.
"Australia is more than 25 million stories - more than 25 million - and each of us here can trace our stories back into our experience in Australia," he told reporters in Brisbane on Friday.
"All the stories are important, all the stories should be respected and on Australia Day there is an opportunity to do that."
Indigenous Olympic legend Cathy Freeman lashed the prime minister for his initial remarks.
"You can't compare the experiences of those 12 (sic) ships that first arrived to this country to what their arrival meant for all generations of Australia's First Nations people," she posted to Twitter.
But Finance Minister Simon Birmingham leapt to his leader's defence.
Asked whether it was fair to compare scurvy to genocide, Senator Birmingham said he did not want to focus on the past.
"That was more than 200 years ago, convicts being brought out under forced orders from the United Kingdom," he told the ABC.
"I don't want to reflect on what was happening more than 200 years ago in terms of the individual circumstances for many individuals.
"They were pretty rough times for lots of people."
Senator Birmingham said he wanted people to concentrate on what Australia had achieved and the potential for greater success.
"We will achieve that greater success by bringing people together not by dividing them, by embracing the Indigenous heritage of this national and the multicultural heritage of this nation, and by celebrating it together and not trying to segment one over the other."
Home Affairs Minister Peter Dutton has fired a warning shot for people planning to protest on January 26.
Thousands of protesters planning to march in nationwide "Invasion Day" rallies have been told police will not tolerate criminal behaviour and will enforce coronavirus restrictions.
"If people are going to protest, then they need to do it within the law," Mr Dutton told the Nine Network.
"They need to do it peacefully and people need to abide by the health directions."
Mr Dutton said state and federal governments had spent the past year trying to protect remote Indigenous communities from coronavirus.
"We don't want to see an outbreak and particularly amongst Indigenous Australians," he said.
"A lot of us have put in a lot of work over the course of this last 12 months to make sure that Indigenous Australians are protected from the virus.
"We don't want all that success unwound."
January 26 marks the raising of the Union Jack for the first time in 1788 after the British First Fleet's arrival in Botany Bay the previous week.
For many, it is a day of mourning that signals the European invasion of the continent after more than 60,000 years of Indigenous occupation.
Australian Associated Press