Berm, I've read and studied poetry my whole life, so hear me out on this. I'll include the whole third verse to provide context.
And where is that band who so vauntingly swore
That the havoc of war and the battle's confusion
A home and a country should leave us no more?
Their blood has washed out their foul footsteps' pollution.
No refuge could save the hireling and slave,
From the terror of flight and the gloom of the grave;
And the star-spangled banner in triumph doth wave
O'er the land of the free, and the home of the brave!
So the first four lines are directed at the British who "vauntly", meaning excessive boasting, swore that they were unleashing Hell and they would leave us without a country, A home if you will. The fourth line is a HUGE middle finger stating "their blood has washed away their foul footsteps' Pollution". He called the British a pollution upon our country's shores. Then we come to the part where he says "No refuge could save the hireling and slave", this is a continuation of those polluting British, their hired contractors and slaves, so what he's saying here is that even the hired help "wasn't safe from the gloom of the grave" I highlighted the word could in the above verse to show past tense as it's important in the fact that many people stating the verse is racist, isn't showing the verse in it's entirety. He didn't write the word "can" which would mean a continued pursuit of slaves, he wrote the would "could"
Where we, as Americans, go wrong is not showing how important African, Asian, and Latino people were to the very fight of our Nation, and still are today. Keep in mind when reading this third verse that there are black, yellow, and brown people chasing down and killing that "pollution" upon our shores including the contractors and slaves. The Red stripes on the flag that doth wave stand for Hardiness and Valor, and if you read the stories of Charles Ball and WIlliam Williams and many other heros of color you'll see that this Third verse isn't inclusive of all slaves, just the slaves that polluted our shores in support of the British.