In terms of 'wanker', I think it may have derived from "too much of a good thing" or "be careful what you wish for". While doing something for self pleasure might be good, a 'wanker' is seen as doing things ONLY for self good, and not what benefits anyone else.
Someone who only masterbates is really not pleasuring anyone else, especially a partner. So, a person can be sexual, and enjoy a well-rounded sexual life, including pleasing others, but a wanker is ONLY in it for him (or her) self.