This kind of expressions have no specific grammatical background because they are informal and every day language.
"Του λόγου σου" means "you personally".
"Του λόγου"= genitive case of the noun "ο λόγος"= speech.
It has totally different meaning in this sentence.
---------------------------------------------------------------------------------------------------------------------------
Vasiliki Baskou, Instructor/Director, https://learn-greek-online.com.