The inverse square law also applies to the amount of energy you have to apply to deflect an object... so as you say, the nearer the deflection point is, the amount of energy increases as the square. At some point the required energy exceeds the objects structural integrity, at which point it's basically cosmic buckshot.

Hm.. why do I get the feeling that band between where we can't deflect it, and where we can't calculate how to deflect it, is a lot more narrow than anyone realises...