It would be interesting to know more about those State Department ratings. NOT to suggest they're unreliable, just, for example, there's a big difference between tourist areas and back streets, also between seasoned travelers and clueless folks who treat traveling like one giant Disney theme park.
From a different angle, I've been looking at crime maps for the town I'm thinking of moving to (in the States), and thinking "omg those are all red zones" and then I looked up the town I'm currently living in (very peacefully and feeling reasonably secure) and guess what I'm in a red zone already. How to interpret this info and use it wisely = a challenge.
A lot of border areas do have high crime but the criminals also tend to give tourists a pass because people recognize that if the tourists go away the money dries up. A lot comes down to common sense, just like it would in New Orleans or New York or any other attractive but high-crime US destinations.
Not sure where you could find lots of people speaking English in a /non/touristy area. Belize maybe?