Micro fibrer towels are not all rough feeling some have soft fibers.
However I just use the yoga mat towel sizes of micro fiber. They have the rougher texture.
The issue you are having is thinking they sghould be used with the same rubbing motion as cotton bath towels. All you have to do to make them work nicely is stop doing that. Dont rub, just blot the towel against you . No more sandpaper issues.
Although I must admit I dry my back, by holding the two far ends of the micro fiber towel and pulling it back and forth. What a lovely exfoliating back scratcher it makes, it feels just great to do that. Also great for exfoliating feet, knees and elbows.
Maybe you will revise your thinking about that type of towel and maybe you won't. Does not matter to me either way.