your actions did not harm, frighten, or control him, they wouldn’t fit my definition of violence. He labels you as violent in order to shift the focus to what you do wrong, which will just lock you more tightly in his grip. However, I do recommend that you not assault him again, as he might seize on it as an excuse to injure you seriously.
