Yes, good point.
How about rolling the 'edit' and 'remove' into one option and simply use the edit dialog to delete the term? Slower, two more clicks, but safer.
Lots of delete operations have 'Are you sure?' clicks, this one doesn't, so by putting the delete option into the edit dialog you are adding a moment to think about what you are deleting, rather than going D'oh! when you miss-click as I have done.
Nick