Instead, the first big confrontation between North and South was a chaotic battle outside Washington, near Bull Run creek, on 21 July 1861. The cavalry on both sides seemingly operatedat random, certainly without any proper co-ordination with infantry attacks. Troops mistook units on their own side for the enemy. After an inconclusive few hours of fighting, the Union army was sent into a panic-stricken retreat by a Confederate attack.

The losses in this first great conflict were tiny compared to the carnage of later battles, yet at the time casualty figures of 1,982 Confederate troops and 2,896 Union soldiers shocked both sides profoundly. While southerners rejoiced at victory, northerners were forced to confront for the first time the scale of the undertaking they had so blithely embraced.

In grand strategic terms, the Union needed to be on the offensive in order to conquer the South. Yet the South took the offensive whenever it could. General Lee invaded Maryland and Pennsylvania in the summers of 1862 and 1863, taking the war onto northern soil, in part because there was huge popular pressure on both sides to be seen to be on the attack.

However, technological innovations gave defending forces much greater tactical strength. Whereas in the Mexican War the army was still using smooth-bore muskets, by 1861 the use of rifled muskets and new conical-shaped bullets called Minié balls greatly increased the accuracy of firepower from a longer range. Towards the end of the war, entrenchments and barbed wire – notably in the long siege of Petersburg – made the conflict resemble the Western Front in the First World War.

Nevertheless, offensives against well-defended positions could still succeed when commanders not only had a numerical advantage but were also prepared to be persistent and flexible – as Grant and Sherman proved in 1864, and as British generals on the Western Front learned after 1916.

From the moment war began, abolitionists argued that a conflict caused by slaveholders could only be ended by ending slavery, the “taproot of the rebellion”. But other northerners vowed they wouldn’t support an ‘abolition war’. The official line from Lincoln was clear: this was a war to restore the Union, with no other aim.

Yet the reality on the ground in the south meant the Union army had to make de facto decisions about whether to encourage the dismantling of slavery. Wherever there was a Union military presence in a slave state, enslaved people sought sanctuary.

Some Union generals sent them back to their ‘owners’. Others allowed them to stay, and refugee camps grew up around military camps. It was General Benjamin F Butler, in command of a Union-held enclave in Virginia, who found a way of protecting runaway slaves without publicly challenging the official line that the Union did not seek emancipation. In the summer of 1861, he announced that any fugitive slave who sought refuge with his forces would be held as “contraband of war”.

This phrase deftly turned the argument that slaves were property against southerners. Just as horses or guns, if captured, could legitimately be impounded since they were likely to be of military value to the enemy then so too ‘human property’, likely to be used to dig fortifications or supply the Confederate army, could be seized – and effectively freed. Contraband became the normal term to describe runaway slaves for the rest of the war. As the debate about emancipation raged in the north, the reality was always that, intentionally or otherwise, the Union army was an instrument of emancipation.

Most of the press attention was on the eastern theatre of the war in Virginia. But in the first phase of the war, during the winter and spring of 1861–2, there was little action in the east as General-in-Chief George B McClellan, a man of enormous self-confidence who rejoiced in the moniker “the Little Napoleon”, painstakingly drilled and built up his troops.

Meanwhile in the west, Union commanders made big gains in a plan to force Confederate forces out of Kentucky and Tennessee, and then take control of the Mississippi River. In particular, General Ulysses S Grant was stunningly successful at putting this strategy into practice. He captured Fort Henry and Fort Donelson, opening up the southward-flowing Tennessee and Cumberland Rivers to the Union. These brilliant successes were followed, on 6–7 April 1862, by a major battle at Shiloh in south-western Tennessee. Grant’s forces were surprised by a Confederate force under Albert Sidney Johnston and PGT Beauregard but, in the bloodiest battle of the war to that point, the Union army held its ground, helped by the timely arrival of reinforcements.

The outcome of Shiloh was that a Confederate counter-offensive had been thwarted, albeit at heavy cost. Grant was initially criticised for his part in the battle, but when Lincoln was urged to remove him from command, he replied, “I can’t spare this man, he fights.”

The greatest danger of war between Britain and the USA came from a conflict over the rights of British shipping. The US Navy tried to seize British merchant vessels bound for neutral ports near the Confederate coast, such as the Bahamas or Cuba, on the grounds that cargo was then to be transferred to southern blockade-runners. On occasion, the US succeeded. Britain had done much the same to American shipping during the Napoleonic Wars, and at the time the US had protested fiercely (the issue was one of the triggers of the war of 1812). Now the roles were reversed.

The conflict came to a head on 8 November 1861, when sailors from the USS San Jacinto boarded a British ship, RMS Trent, 300 miles east of Havana, and removed two Confederate envoys, James Mason and John Slidell, en route to Europe to press Britain and France for support. The British government was furious at the violation of its flag and there was talk of war. The diplomatic row was defused after US secretary of state William Seward apologised and released the envoys, insisting as he did so that the case proved that the British had finally accepted the United States’ conception of neutral shipping rights.

The larger issue was what role Britain might play in the war. Northerners were frustrated by British recognition of the South as a belligerent power (although Britain never gave the Confederacy diplomatic recognition) and angry about blockade-running ships and a couple of naval vessels that were built in British ports. The Confederacy for its part hoped the cotton embargo would precipitate European intervention of the kind that had tipped the balance for the rebels in the War of Independence. It never came.

In the Peninsula Campaign of the spring of 1862, Union commander George B McClellan launched the Army of the Potomac in what he hoped would be the decisive move against the Confederacy. Rather than taking the direct approach due south, troops were sent by sea to the mouth of the James River from where they approached the Confederate capital Richmond from the east in a bid to evade Confederate defences. McClellan, who was fiercely opposed to emancipation, hoped to fight a limited war according to the “highest principles known to Christian civilisation”. At first the plan seemed to go well as Confederate forces fell back. But then General Robert E Lee took field command of the Army of Northern Virginia for the first time. An undemonstrative taciturn man, Lee was to about to prove himself one of the wiliest, most courageous and most effective commanders of the war. He believed the Confederacy could counter the manpower advantage of the Union army only by seizing and keeping the initiative.

In a stunning series of victories known as the Seven Days Battles (25 June–1 July), Lee’s leadership transformed the Confederacy’s position in one week, forcing McClellan’s army back. Thereafter officers and men in the Army of the Potomac developed what almost amounted to an inferiority complex in the face of Lee’s army, a spell that was only partially broken a year later at Gettysburg. McClellan’s star waned after the Peninsula Campaign and with him the idea the war could be fought in a ‘limited’ way.

In September 1862, Lee launched the first of his two grand raids into the north. In optimistic moments, Confederate leaders hoped Lee’s invasion might persuade Maryland slaveholders to support the South and foreign powers to recognise it, but at the very least they wanted to prove the North could never subdue the South militarily.

The ‘invasion’ culminated in a battle at Antietam Creek near Sharpsburg, Maryland, on 17 September. The conflict was a fluid, confused and messy affair, with friendly fire compounding the difficulties of communication on a large battle area where no one had more than a partial view of the fighting. Particular spots on the battlefield acquired an especially gruesome reputation, including Miller’s cornfield, which changed hands six times in just a few hours, and ‘Bloody Lane’, a sunken road from which the rebel South held the attacking forces of the North at bay for over three hours in late morning.

By nightfall, at the cost of around 23,000 casualties, the Confederate line had been pushed back a few hundred yards. Still, it was a victory for the Union, although to Lincoln’s frustration McClellan failed to pursue Lee’s forces after the battle.

The limited Union victory at Antietam was to be the final battle of the first phase of the war. Just a few days later, on 22 September 1862, Lincoln issued a proclamation stating that if, by 1 January 1863, the rebel states of the Confederacy had not returned to the Union, the United States would, from that date onwards, regard slaves held in rebel areas as free. This Preliminary Emancipation Proclamation had an incendiary effect. It was an ultimatum to the South: return to the Union within 100 days with slavery intact, or face total destruction.

Confederate leader Jefferson Davis called it “the most execrable measure in the history of guilty man”. The three-month delay was intended to send a clear message that emancipation was a tool of war rather than an end in itself. Like a riot policeman giving notice that a mob was about to fired upon if it did not disperse, Lincoln wanted to give the appearance of due process. But no one had any illusions: the President had tied the Union’s fate to emancipation.

On 1 January 1863, the president duly issued the Emancipation Proclamation. It applied only to those areas of the United States that were still in arms against the government and much of the document was taken up with a list of counties in rebel states that, because they were no longer under rebel control, were exempted from the proclamation. Lincoln took a political and strategic risk in coupling together the fate ofthe Union with the fate of slavery.

In some Union regiments there were near-mutinies at the news. But abolitionists rejoiced that at last the day of jubilee was at hand.

Union soldiers commonly used terms such as ‘darkie’ and ‘nigger’ in their letters. Even proudly antislavery soldiers exhibited an unquestioning racism. There was no contradiction in holding racist views while also thinking that a war against secession was inherently a war against slavery, and that the Confederacy was a repressive society that challenged American values of freedom and opportunity.

Encounters with runaway slaves had a dramatic impact on some Union soldiers. Black people were exotic and fascinating to rural farm boys from the north. In addition, many soldiers interpreted their encounters with freed slaves in the light of what they had heard and read of the cruelties of slavery. Private Chauncey Cooke wrote to his mother in Wisconsin about “a toothless old slave with one blind eye” who told him horrific stories of his wife and children having been sold, of whippings and being hunted by bloodhounds when he tried to escape. The stories, Cooke wrote, were “just like the ones in Uncle Tom’s Cabin and I believe them”.

Some Union troops were convinced by evangelical preachers and antislavery propaganda that expunging the sin of slavery would redeem their country in God’s eyes. Some simply wanted black troops to be placed in the front line instead of them. Most were probably convinced by the much more pragmatic case that if the rebels hated emancipation, then it must be a good thing, a weapon to strike at the heart of southern society.

After Antietam, General Ambrose E Burnside replaced George B McClellan as the commander of the Army of the Potomac. He assumed command of the army’s 120,000 men on 7 November 1862, and President Lincoln urged him to launch a fresh assault on Richmond immediately.

Burnside’s immense army was well supplied and even had hot-air balloons for surveillance, so he should have been able to outgun the Confederates. However it seemed that his troops could not out-manoeuvre the enemy. General Burnside’s plan was to cross the Rappahannock river above the town of Fredericksburg, which lay on the direct route from Washington to Richmond. A delay in the arrival of pontoon bridges meant that before the Union army had even crossed the river, General Lee had time to concentrate his troops on the heights behind Fredericksburg.

The Army of the Potomac eventually made it across the river on 13 December 1862, but could get no further. In repeated assaults up the gentle rise of Marye’s Heights, line after line of the North’s soldiers were cut down. The Union suffered more than 12,500 casualties in the day-long battle, and gained almost no ground.

Once the news of Burnside’s defeat at Fredericksburg broke, it seemed to inaugurate the bleakest period of the entire civil war for the North. President Lincoln summed up the Union’s perspective in one succinct line: “If there is a worse place than Hell, I am in it,” he told a visitor when the reports of the debacle came through.

On 30 April 1863, yet another Union commander, Joseph Hooker, crossed the Rappahannock River and tried to fight his way to Richmond. Hooker’s thinking was more subtle than Burnside’s – he crossed the river to the west of Fredericksburg with the aim of attacking the Confederates from the side – but the plan failed in its execution. Audaciously, Lee, ably supported by General ‘Stonewall’ Jackson, divided his army in the face of superior numbers and outflanked Union troops. In a three-day battle, Hooker made error after error, mistaking Jackson’s flanking movement for a retreat and finally abandoning the one piece of high ground from which artillery could be used effectively in wooded, undulating country.

The relief of victory for the South was tempered by the death of Jackson, who was accidentally fired on by his own men, prompting an outpouring of grief that transformed the dead general into a martyred hero. In retrospect, the battle of Chancellorsville was the high-tide mark of Confederate military success. It was certainly the apogee of Lee’s military career. Once again, his army had out-smarted and out-fought a larger, better-equipped Union force. But Hooker’s inept leadership had been a major factor in the southern victory, and such incompetence could not be relied upon indefinitely. The Confederates suffered huge losses and the difference in the size of the armies could be disguised by smart tactics only for so long.

In the first days of July 1863, two simultaneous military victories for the Union seemed to turn the tide in the war. The Confederates had launched another raid into the north, this time up the Shenandoah Valley and into the state of Pennsylvania. General George Meade, the latest commander of the Army of the Potomac, led the forces that finally defeated Lee in a fight at Gettysburg during 1–3 July.

Lee’s army retreated back into Virginia but a spell of invincibility had been broken. Lee’s army never fully recovered from Gettysburg, in spirit or in numbers. His officer corps and command structure were hit especially hard. Never again in the war would Lee be able to rely on his officers as he had done in 1862. In 1864, he had to appear in person on the front lines to rally troops, which was something that he hadn’t done in previous years.

Meanwhile in the west, General Grant scored a major breakthrough. On 4 July, after more than six weeks, of failed assaults, Grant accepted the surrender of Confederate forces in the fortified river town of Vicksburg, the last Confederate stronghold on the Mississippi. With New Orleans having fallen the previous year after an assault from the sea, the South was now split into two halves.

The final phase of the American Civil War: all out war

With fighting concentrated in the southern states, the Union army waged all-out war, aimed at destroying the Confederacy’s morale and crushing its capacity to wage war. As the conflict entered its final phase and the South put up determined opposition, both sides suffered huge losses…

Northerners on the home front may have been far from the sound of gunfire, but they still felt the impact of conflict. Few northerners opposed the war at the outset but, by 1862, two issues – emancipation and conscription – aroused fierce opposition. To many Democrats Lincoln was a tyrant, crushing the freedoms of Americans by pressing them into an abolitionist crusade.

The most dramatic incident of violent discontent took place in New York City on 13–16 July 1863. In some of the worst rioting in American history, thousands of workers, most of them poor Irish immigrants, rampaged through the city in a howl of rage against attempts to implement the draft. They targeted the visible property of the rich and the offices of Lincoln-supporting newspapers. They also launched indiscriminate attacks on the black population of the city, lynching them and burning an orphan asylum. The rioters’ targets reflected their perception of the Republicans destroying the white man’s republic as they had known it. The total death toll is unknown but was probably over a hundred. Order was restored by troops who marched north after the battle of Gettysburg.

In retrospect, the riots were a watershed. In mainstream magazines such as Harper’s, images of white men rioting to avoid fighting for the North in New York were deliberately contrasted with black Union soldiers launching a heroic failed assault on Fort Wagner near Charleston.

The old race-based conception of citizenship, to which many northerners still clung, had never faced such a severe challenge.

The most famous of Lincoln’s speeches was a two-minute address to dedicate the military cemetery at Gettysburg on 19 November 1863. Lincoln was not even the main speaker – he wound up proceedings after a two-hour oration from Edward Everett. But it’s Lincoln’s words that endure: “Four score and seven years ago, our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal.” The effect of this was to date the origins of the republic to the Declaration of Independence of 1776, with its grand preamble authored by Jefferson and appealing to the universal ideal of equality, rather than the more prosaic Federal Constitution of 1787. Lincoln was implying that the Constitution merely gave form to the nation, and that the nation mattered not as an end in itself, but as an embodiment of the ideals of equality and liberty. Echoing in secular language the Christian idea of a trial of faith, Lincoln went on to claim the civil war was a test of “whether that nation or any nation so conceived and so dedicated can long endure”. In little under 300 words, he went on to explain why the struggle and the sacrifice had a dignity and a purpose of universal and transcendent significance: “That this nation under God shall have a new birth of freedom, and that government of the people, by the people, for the people shall not perish from the earth.”

• Read more | The American Civil War in three films

Lincoln’s eloquence was noted at the time, but the reputation of his Gettysburg address has grown over the years, as Americans have sought to find an uplifting meaning in the slaughter of the war.

General Ulysses S Grant took command of all Union armies in early 1864 and adopted a harsh strategy designed to destroy the Confederacy’s capacity to wage war. The spring campaign of 1864 differed from those of the previous three years in two ways. Firstly, Union forces now had an even greater advantage of numbers in most confrontations. In addition, rather than retreating after a setback, Grant ordered his men forward. Experienced soldiers used to long periods in camp followed by occasional terrifying battles now faced what seemed like continuous marching and fighting.

In the battle of the Wilderness, which took place from 5 to 7 May near the site of Lee’s victory at Chancellorsville in Virginia, Union forces failed to dislodge Lee from his position. Sparks from the muzzles of thousands of rifles set the woods on fire and thousands of wounded men burned to death. In two days of fighting, the Union army suffered more than 17,000 casualties and the Confederates around 11,000.

At the battle of the Wilderness and in the bloody engagements of the following three months, the losses on the Union side were horrendous, but proportionally less than the South suffered. Grant, dubbed Butcher Grant by some in the north, had made the grim calculation that his Union army could withstand its losses more easily than the Confederacy.

Time and again as Grant attacked through rural Virginia, Lee moved his army quickly before establishing powerful defensive lines. The action moved in an arc east and south as Grant tried to get through Lee’s defences and Lee manoeuvred to keep his heavily outnumbered and outgunned army between Richmond and Union forces. At Spotsylvania Court House on 12 May 1864, the South repulsed the North in a particularly vicious battle. Both sides fought intensely, especially around the ‘bloody angle’, a U-shaped line of Confederate trenches that, by the end of the day, were filled with a mixture of blood, mud and corpses. Spotsylvania was essentially yet another defeat for the North. Grant had attempted to seize an important crossroads; Lee had beaten him to it and had then successfully, albeit bloodily, held off the Union’s assault.

But Grant refused to treat it as a defeat. Instead, he attempted another large-scale flanking movement to try to get between Lee and Richmond. Again, Lee anticipated the move. On 3 June, the Army of the Potomac was hurled against well-entrenched Confederate fortifications at Cold Harbor. More than any other civil war conflict, Cold Harbor was a harbinger of the horrific first day of the battle of the Somme. Most of the 7,000 Union soldiers who fell that day did so in less than an hour, as wave after wave of attacking troops were cut down.

The Army of the Potomac moved south from Cold Harbor and crossed the James River in another attempt to force Lee out from his trenches by flanking his army and seizing the key railroad junction at Petersburg, Virginia, just south of Richmond. Once again, before an assault took place, Lee realised what was happening and marched his army to thinly defended Petersburg with astonishing speed. Thousands of Union soldiers fell in futile efforts to dislodge the rebels from their earthwork entrenchments.