It has to do very much with the feelings that you express with the phrase.
When you say "στο καλό" and in your mind you want to express "goodbye", then people will perceive it in this kind and positive way.
When you add the "άει" and say "άει στο καλό", you have an expression that Greeks will not use for something positive. Add some angry feelings and what you have is the more polite version of "άει στο διάολο" (get lost, go to hell).
You may say "άντε στο καλό", in a good positive way and people will still perceive it as a friendly goodbye.