Rule of Thumb: if you can't leave it the way you found it or better, don't go, stay away.
I nearly always try to remove something that does not belong that I did not contribute. It only takes a second and feels good to do your tiny part. Doesn't have to be just the wilderness either, it can be...