Damage Reflection?

Does the Barrier Spikes reflect only melee damage? Also,will skills like smash reflect the smash damage back to the enemy or will the base attack be used?

Only contact skills (aka melee). It reflects a percentage of the damage dealt. If you hit the spikes for 100 and it has 50% reflect, you take 50 damage.

what if a monster theoretically windmilled the barrier? would the reflect damage ignore wm invincibility?

Of course.