I do have direct experience with this dynamic; and I remember clearly feeling very put out by it.
I’ve become a master in conflict diffusion; and I don’t see that in any way being a “coward”. Anger is so easily accessible, I’m not about to jump in and create further tension unless someone is threatening my family or anyone I love.
I’m not quite sure how this little diatribe could be considered in any way helpful other than me empathizing with you greatly.
Be the type of man you would want to see in the world, lead by example. By all means kick ass and take names if anyone tries to hurt you – but use it as a tool, not a reaction.