The Work and Pensions Secretary was one of the main speakers at a conference examining changes to the benefits system in Edinburgh on Wednesday.
As he started to speak one man shouted at the Tory that the Westminster coalition is creating a new poll tax with its reforms.
Later two partially sighted people were escorted out of the conference venue after they too heckled Mr Duncan Smith.
As he began his address, one protester demanded:
“Why are you here in Scotland? We have a different philosophy, we’ve not elected you and your Tory cohorts.”
Referring to the fact there are two giant pandas in Edinburgh Zoo while Scotland has just one Conservative MP, the man said:
“We don’t want you, we don’t need you. We’ve got more pandas than you. You’ve only got one MP.
“We want a different Scotland, one that cares for people who are in distress and poverty. You are going to make millions of people homeless, people that desperately need help. You’re making the rich richer and the poor poorer.”
He told Mr Duncan Smith:
“You are creating your new poll tax, that’s what you’re going to do. We’re going to see the end of you, back to England where you belong, you ratbag.”
The Work and Pensions Secretary insisted that the welfare system has a key role in “providing effective support for the most vulnerable and helping those who have fallen on hard times to get back on their feet”.
Mr Duncan Smith said: “This Government, I believe, is on the side of a welfare that does just that.”
As he spoke he was then heckled by two partially sighted protesters: a man with a guide dog and a woman carrying a white stick.
The pair, who gave their names as Jonathan Smith and Charlie Sabenfox, were both escorted from the city centre hotel where the event, organised by Capita Conferences, was taking place.
As they shouted at him, Mr Duncan Smith told them:
“If you listen to what I am saying, you will understand the reality is that this country is not cutting welfare, it is managing the growth at a lower level.”