No, you have it right.
The long and short of it is the social stigma surrounding VanDwelling.
Instead of whining about how you're percieved, do your part to change it for the better. Practice "no-trace" camping. Don't overstay an area not meant for that, etc. There are lots of things...