As he started to speak one man shouted at the Tory that the Westminster coalition is creating a new poll tax with its reforms.

Later two partially sighted people were escorted out of the conference venue after they too heckled Mr Duncan Smith.

As the Work and Pensions Secretary began his address, one protester demanded:

“Why are you here in Scotland? We have a different philosophy, we’ve not elected you and your Tory cohorts.”

Referring to the fact there are two giant pandas in Edinburgh Zoo while Scotland has just one Conservative MP, the man said:

“We don’t want you, we don’t need you. We’ve got more pandas than you. You’ve only got one MP.

“We want a different Scotland, one that cares for people who are in distress and poverty.

“You are going to make millions of people homeless, people that desperately need help.

“You’re making the rich richer and the poor poorer.”

He told Mr Duncan Smith:

“You are creating your new poll tax, that’s what you’re going to do. We’re going to see the end of you, back to England where you belong, you ratbag.”

The Work and Pensions Secretary insisted that the welfare system has a key role in “providing effective support for the most vulnerable and helping those who have fallen on hard times to get back on their feet”.

Mr Duncan Smith said: “This Government, I believe, is on the side of a welfare that does just that.”

As he spoke he was then heckled by two partially sighted protesters: a man with a guide dog and a woman carrying a white stick.

The pair, who gave their names as Jonathan Smith and Charlie Sabenfox, were both escorted from the city centre hotel where the event, organised by Capita Conferences, was taking place.

As they shouted at him, Mr Duncan Smith told them: “If you listen to what I am saying, you will understand the reality is that this country is not cutting welfare, it is managing the growth at a lower level.”

One of the more controversial changes being introduced is cutting the housing benefit of tenants in council and housing association properties if they are deemed to have spare bedrooms.

This cut has been dubbed the bedroom tax by the UK Government’s opponents.

Mr Duncan Smith said: “I’m not saying that ending the spare room subsidy will not present difficult cases.”

But the Government has allocated an extra £370 million in discretionary housing payments to local authorities, including £10 million to councils in Scotland, to help manage that transition, he said.

“These are changes that you simply can’t walk away from. These are changes that are required to get the housing system back into balance.

“We all benefit from having a safety net to fall back on in hard times but, equally, when welfare doesn’t work we all also feel it.”

According to Mr Duncan Smith, 4.5 million people of working age, including 450,000 in Scotland, are “trapped” on out-of-work benefits. Across the UK, 3.7 million households are workless, of which 367,000 are in Scotland, and 1.8 million children are in households where no one is in work, including 145,000 youngsters in Scotland.

This is a “tragic waste of human potential”, he said.

“As well as the social cost, I think we must also acknowledge this entrenched dependency weighs heavily on the public purse. Across the UK we spend over £200 billion annually on benefits, tax credits and pensions.”

The Tory minister also used his speech to say that an independent Scotland could have an impact on state pensions in Scotland

“Across the UK, our ability to support those in retirement is something we should be proud of. By shouldering the responsibility on broad shoulders, even in difficult times, the coalition has been able, I believe, to pledge support for UK pensioners right now.”

But Scotland in future would have fewer working-age people supporting pensioners. Currently, there are 32 working-age people supporting every 10 pensioners both in Scotland and the UK as a whole, he said.

In Scotland it is forecast that just 23 workers will support every 10 pensioners by 2060, he said.

This could result in, “roughly, an enormous 20% increase in Scotland’s welfare spending”.

Mr Duncan Smith claimed: “Extra money will be required to meet Scotland’s demographic pressures, although oil and gas revenues are projected to decrease significantly over the next decade. The question simply remains: higher taxes or more borrowing?”

Also speaking was Scottish Housing and Welfare Minister Margaret Burgess.

She said the UK Government will “plough on regardless” with welfare reforms “in the full knowledge of the impact it will have on our most vulnerable citizens”.

Cuts in child tax credit and working tax credits last year hit 110,000 families in Scotland, while others have been hit by changes to child benefit.

“What is clear is that these changes are impacting significantly on individuals and hard-working families who are already struggling to make ends meet,”

Ms Burgess said.

She also criticised the “unfair” changes to housing benefit and called for control over this to be devolved to Scotland.

From April 1, those in receipt of housing benefit may have their payments cut if they have additional rooms.

Ms Burgess said:

“In five days’ time we will see this one-size-fits-all reform introduced to working-age households across Scotland, regardless of disability, family circumstances, economic conditions or availability of smaller houses.

“The Scottish Government has been consistently clear in its opposition to this policy. It is unfair and divisive. It will hit some of our most vulnerable groups hardest.”

It is “just another example of the vacuum between UK housing benefit policy and Scottish Government housing policy”.

While housing is devolved to Scotland, housing benefit is reserved for Westminster.

That set-up “does not work”, she said.

“They have to be in the same administration, and I think it should be here in Scotland.”

The Herald Scotland