Put your car on scales and see what the affect is of adding your ballast in the legal position. It is not as far rearward as you would like but is still better, IMO, than weight that moves around during cornering, braking, etc.
Adding a tube, as suggested, puts the weight up high and is not what you want either. Our cars can not be exactly corner balanced but get close and having the weight low and between the axles is better than higher and behind them. For that reason, putting the spare tire back in is a poor solution.
At the end of the day, how much it matters I do not know. It matters some but????? The driver is more important than much of what we do to our cars so the important thing is to do what gives you confidence and go out and have FUN!
Big Dog