Even in 1968, the so-called Rivers of Blood speech was extraordinarily controversial. Powell himself was summarily sacked from the Tory front bench, never to return; in liberal circles, he became a pariah. Yet few people now care to remember that the speech made him an enormous folk hero, the tribune of Britain’s silent white majority. Hundreds of dockers marched on Westminster to show their support, workers in Wolverhampton and the Black Country walked out to demand his reinstatement, and opinion polls consistently put support for Powell at more than 70 per cent. Within two weeks of the speech, Powell had been sent more than 43,000 letters, almost all of them praising him, and for years afterwards he was easily the most admired politician in the country.

His views might seem abhorrent to modern readers. And yet there is little doubt that, at the time, he spoke for the vast majority of the British public.

One of the ironies of the Rivers of Blood controversy was that it overshadowed the 20th anniversary of that supremely iconic moment in the history of immigration to Britain, the voyage of the SS Empire Windrush from Kingston, Jamaica to Tilbury, where its 492 passengers disembarked on 22 June 1948. Each had paid £28 10s for the chance to escape the poverty and limited horizons of the West Indies, and to build a new life in the imperial mother country.

• Read more | Bound for Britain: your guide to the arrival of the HMT Empire Windrush

Hints of the hostility to come

Contemporary press reports treated the newcomers as a diverting oddity, not a threat: the Express described trilby-hatted Jamaicans clapping and cheering on the dock before sitting down and tucking into roast beef, Yorkshire pudding and spotted dick. Yet not everybody welcomed the newcomers to Britain’s shores, and even in 1948 there were hints of the anxiety, hostility and prejudice that would lead to Enoch Powell’s speech two decades later.

As the Windrush approached, some civil servants had talked of moving its passengers on to Kenya, while a group of Labour back-benchers even wrote to Prime Minister Clement Attlee complaining that “an influx of coloured people” would “cause discord and unhappiness among all concerned”. Attlee ignored them; like many politicians of the day, he was attached to the ideals of imperial brotherhood. All Britain’s subjects, he wrote, should be “freely admissible to the United Kingdom”. But it was not a view that found much favour with the general public, and by the late 1950s, when West Indians were arriving in Britain at the rate of around 50,000 a year, the extent of public hostility was becoming ever more apparent.

Was Britain a racist country in the 20 years between the Windrush’s arrival and Powell’s speech? The answer is rather more complicated than a simple yes or no. Polls found that most Britons disapproved of overt colour prejudice, while as late as 1956, two out of three still supported the principle of unrestricted Commonwealth immigration. Yet the unspoken prejudices that had underpinned the British Empire died hard. Children grew up with stories of imperial derring-do against brutal savages; even a progressive politician like the Labour Chancellor Hugh Dalton could refer to the colonies as “diseased nigger communities”. Many hotels, restaurants, dance halls and landlords operated a colour bar; indeed, as more immigrants arrived, the colour bar often became more severe. In the mid-1950s, more than eight out of ten London landladies said they would refuse to house students who were “very dark Africans or West Indians”, while a survey in Birmingham found just 15 landladies out of a thousand who would let rooms to non-white families. “No Coloureds”, the signs used to read, a sentiment with which millions of people evidently agreed.

Generalising about the immigrant experience is of course extremely difficult. After all, by 1961 the total black and Asian population had risen to 337,000, and ten years later, despite various attempts to curtail it, the immigrant population hit 650,000. Some people clearly thrived in their new homeland, making new friends and progressing rapidly up the career ladder. But they were probably the exceptions. Implicit colour bars in many walks of life, from British Railways platforms to Bristol buses, meant that highly qualified immigrants had to settle for ill-paid, backbreaking jobs: one Pakistani customs inspector, for example, worked as a railways cleaner. “The jobs we got were always the worst, even if we were educated people who could read and write much better than the people who were in charge,” he later remembered. “They knew I had been an inspector of Customs, but that didn’t matter.” Discrimination at work or in housing was bad enough. What was worse, however, was the growing undercurrent of racist violence that finally spilled over in the Nottingham and West London riots of August 1958, when knife-wielding white teenagers, bored in the summer heat, went “nigger hunting” in the dilapidated streets where immigrants had settled. On the worst night of the trouble, some 700 people – among them dozens of women and children – rampaged through the streets of Notting Hill, chanting “Kill the niggers!” One black student, cornered by pursuers, took refuge in the doorway of a greengrocer’s shop; only the intervention of the grocer’s wife saved him from what may have been a savage beating.

• Read more | The great British civil rights scandal: the Bristol bus boycott

Contemporary observers recognised that the fighting was a shocking illustration of the prejudices that bubbled under the apparent civility of British society in the 1950s. In a front page cartoon in the Daily Mirror a few days later, Adolf Hitler’s ghost addresses a “racialist thug” in Teddy Boy regalia. “Go on boy,” the late Führer is saying. “I may have lost that war but my ideas seem to be winning.”

After Notting Hill, “racialism” disappeared from the headlines. But it did not disappear from the streets, or from the hearts and minds of millions of people. And while the prejudices of the 1950s can be explained as a hangover from the days of imperial superiority, or as a historically common reaction to people who seemed “different”, in the years that followed racism fed on the alienation of working-class Britons who felt betrayed by Harold Wilson’s government and were horrified by the nation’s apparent decline into economic penury. Indeed, as the 1960s wore on, and even though immigrants were arriving at a slower rate, the politics of race became more intense. And nowhere was this more apparent than in Enoch Powell’s political heartland, the West Midlands.

Tough times for scapegoats

The West Midlands became associated with the politics of racism for two reasons. The area had attracted a disproportionate share of black and Asian immigrants in the mid-1950s, when its engineering and manufacturing firms were booming. Ten years later when the economic pendulum had swung against Britain’s exporters, the firms of the West Midlands were particularly hard hit. And in the grimy, derelict industrial towns of the Black Country, the immigrant communities – who had moved into the cheapest and therefore least salubrious parts of town – made perfect scapegoats. “Vote Labour for more nigger-type neighbours” read campaign stickers in the parliamentary election in Smethwick in 1964.

The Conservative candidate Peter Griffiths, a fierce critic of immigration, won the seat, beating Labour’s foreign affairs spokesman Patrick Gordon Walker in one of the century’s most shocking electoral upsets. And although the Smethwick campaign was a rare episode of openly racist electioneering, it exposed the seething, unsettling resentments that lay beneath the froth and bubble of Harold Wilson’s supposedly Swinging Britain.