There was nothing pure in my actions, I was
just being an ass!
Then the volition behind your actions was impure, you planted a bitter seed.
Harsh speech is at times necessary, but I believe you are confusing harsh speech with impure negative speech.
Harsh definition: unpleasant rough or jarring to the senses.
U Ba Khin ran a meditation campus, to attend one agreed to observe noble silence and follow the strict meditative schedule. If a student decided to break the rules and talk or did not care to meditate they just wanted to goof around, then he was known to bellow at these individuals with great intensity, these students were not only disturbing others but they were not helping themselves. Students would learn very quickly to follow the course guidlines and benifited greatly in the end. His bellows were for the students benefit not to insult them or cause them harm.
As a father I often must use harsh speech with my kids, if their actions are about to cause them danger I bellow, but to protect them from harm, not to hurt or harm them.
Zealots reaction was pure he was pointing out that you were being an ass and therefore the accumulated Karma is light. Don't confuse him by pretending your intentions had a loftier goal.
Where I come from your behavior would warrant an apology.