I believe that the damage isnt based on your damage dealth, rather your actual damage. that is, you have 500 damage, the reflect is based on that 500 damage. And yess, it is spell damage, and yes, it stacks forever.
Ever gotten into a fight with Nanny using a melee hero who had no spell damage reduction? you will die.
Nanny has nearly 400 armor, like 95% reduction. You deal 2000 a hit. He is reflecting about 200 every time you hit him. Running some quick math based completely on, "damn that hurts" you will see that it isnt .5% (10% of 5%) AKA 10 damage, but really close to 10% without any other reduction save your magic damage reduction.
And yes, to somewhere out there asking, Nannys spiked armor is the reason you always get destroyed by him. His damage is pathetic. A ranged hero with scepter of the primarch will gain life while auto attacking him. Get spell damage reduction before you attempt the fight, or you will die.