I've been hiking, backpacking, road tripping and camping alone for my whole life, and though I've had some fear at times, it has been minimal.
I have had a few "things happen to me" in life that have helped give me a sense of "what to look out for", and I'm also a highly intuitive person with a strong "gut sense" and the ability to trust it. I've had some brief run-ins or crossed paths with (a very few) nefarious or sketchy folks and this has helped me learn what it looks like when danger crosses your path.
All that said, I don't think there's been a single time when I've talked to my mother about my camping or road trips, when she hasn't said, "Aren't you scared to go alone? I'd be too scared to do that." Some people are totally out of their element doing nearly anything alone. It isn't just fear of baddies which has kept my mother from this: it's that she is very dependent and needs someone to tell her what to do and where to go. She is also very heavily a creature of routine and would feel ill at ease without her regular routine, that keeps her living in a rather small world. She doesn't have it in her to just get up and say "Hmmmm...I wonder what new place I could go to today?" But she is content in her small world.
Applying common sense, one can realize that bad folks who want to harass or mess with someone, are not likely to be out in the middle of nowhere, where they may not find anyone to mess with. There are some bad folks who do harass or stalk people out in the boonies, as others have referred to hearing stories about, but even so, I would expect that they would not just go out into areas where the chance of coming across someone was very small, unless they had other business out there.
If I'm out in the woods or wherever and someone approaches who makes me feel ill at ease, I would just leave. I'm not going to confront them or argue about who got here first and so it's my campsite. Generally, for those who are simply bullies or territorial, or you inadvertently camped in a spot where the local "buddies" group goes out every week and drinks beer and does target shooting all night, they are content if you just get up and leave. That makes them feel that they won, they bullied you out of "their" spot. That's one of the prime benefits of car-camping in all its forms, actually: that it's much easier for car campers to pull up and leave and find another spot, than for tent campers to do that.
I'd also add that I think you're much safer "out in the middle of nowhere" than you are camping in a rest stop, a Walmart, or other parking lot or public place.