Gunshots were heard afterwards, but it was not immediately clear who fired them. One report said they came from a slum area where protesters scattered.
The unrest came on the first full day of office for newly-appointed Prime Minister Gerard Latortue.
It comes amid reports that Mr Aristide is due to travel to Jamaica next week.
Jamaica visit
Jamaican Prime Minister PJ Patterson, was quoted by the Associated Press as saying that Mr Aristide wanted to be reunited with his two young daughters.
Mr Aristide has spent the last two weeks in exile in Central African Republic after he was forced to flee Haiti.
His daughters were reportedly sent to New York for their safety.
"Mr Aristide has expressed a wish to return temporarily to the Caribbean with his wife and to be reunited with their two children," said Mr Patterson in a statement.
He said Mr Aristide was not seeking political asylum in Jamaica but was finalising plans "for permanent residence outside of the region". He did not specify where.
Mr Patterson said Mr Aristide would arrive early next week and stay for between eight to 10 weeks, AP reported.
Cars set ablaze
In Haiti, police in riot gear watched over around 1,000 demonstrators as they approached the presidential palace in Port-au-Prince on Thursday.
They chanted slogans against the United States and the new government being put together by Mr Latortue.
The BBC's correspondent in Port-au-Prince, Daniel Lak, said the crowd smashed shop and car windows as they fled.
Although gunfire was heard there were no reports of injuries, our correspondent added.
The protesters say they do not recognise Mr Latortue, who arrived back in the country from exile late on Wednesday and has begun meeting officials.
Weapons hunt
Mr Latortue - a former foreign minister and UN official - has called for national reconciliation and an end to the dictatorships of the past.
France and the US have sent 2,500 troops to Haiti to restore order, and they are due to be joined by an advance guard of Canadian forces on Friday.
However, our correspondent says large numbers of Haitians fear the country faces violence for many weeks to come.
Earlier on Thursday, peacekeepers began hunting down illegal weapons with their first major disarmament operation in a slum neighbourhood in Port-au-Prince.
They raided a house where they had information a cache of weapons was stored, but found nothing.