Yes, in such a case your friend could have said: Η υπομονή σου σκάει γάιδαρο. You were patient, (and maybe persistent too).
Insult/compliment, it depends on the feelings of the one who expressed it. If your friend felt guilty for being so late, then it is a compliment and appreciation for your patience. If your friend wanted to avoid you, make you bored and cause you to leave, then it is an insult.