In my training to become of non-violence trainer we addressed this issue of self-defense. The difference lies in "the cause". It is different to suffer for a cause than to suffer at the hands of an attacker.
Non-violence is about a cause, not neccesarily big social change, could be personal and small. It doesn't mean you are a door matt.