Governor Ralph Northam announced it would come down amid national protests after the death of George Floyd.
The statue became a focal point for this activism and crowds cheered as a crane removed it on Wednesday.
Memorials to leaders of the pro-slavery, Confederate states – whose capital was Richmond – have long stirred controversy.
While the removal of these statues is often done with little fanfare, authorities broadcast Wednesday’s removal on social media. Large crowds also gathered at the site.
Earlier in the week, Mr Northam called the statue “a monument to the Confederate insurrection”.
The 21ft (6.4m) statue – which dates to 1890 – was taken to a secure facility until a decision is made about what to do with it.
The statue’s 40ft (12m) pedestal, which is still covered in graffiti from the 2020 protests, will remain in place. Local officials have said that it will stay there until the area is “reimagined.”
The governor’s plans to remove the statue in 2020 were delayed by two separate lawsuits by Richmond residents opposed to its removal.
Last week, however, Virginia’s Supreme Court rejected the lawsuits, paving the way for the statue to be removed.
“This is an important step in showing who we are and what we value as a Commonwealth,” Governor Northam said.
Hundreds of statues of General Lee and other famous Confederate figures exist throughout the US.
“There’s no other country in the world that erects monuments to those who took up arms against their country,” Mayor of Richmond, Levar Stoney, told BBC News.
Since a wave of protests engulfed the US following George Floyd’s death at the hands of a Minneapolis police officer, more than a dozen statues have been removed in Richmond alone.
Former President Donald Trump released a statement on Wednesday night decrying the “complete desecration” of the statue.
“Our culture is being destroyed,” he wrote.
A seismic shift
Analysis by Anthony Zurcher, North America reporter
In a moment fraught with symbolism, the statue of the leader of the Confederate Army has come down in the city that was once the capital of the Confederacy.
For many in the South – the white South – Lee was a hero and a rallying cry. For some, he was a symbol of regional pride. For others, he was an icon of resistance – first, to the occupying Union armies in the aftermath war, but later to attempts by the US federal government to impose desegregation and civil rights protections on recalcitrant Southern states.
It was no coincidence that most of the confederate statues were erected not in the immediate aftermath of the war, but in the early 20th Century, as the white supremacist Ku Klux Klan gained power and southern legislatures enacted Jim Crow laws disenfranchising blacks.
Lee’s removal reflects the seismic shift in attitudes toward the Confederacy in much of the US South – especially Virginia, where changes in demographics and political perspectives have been particularly pronounced.
Now the city must determine how to fill the empty lots and graffiti-covered pedestals where these statues once stood – fashioning a new history in place of the dismantled one.