If its a massive pain for your use-case then please feel free to submit a feature request for it. we could include a setting that forced “delete” permission requirements for that action… although generally speaking I think this might prove to be confusing.
It is more likely that we would need the “assignment” itself to have permissions - again might be confusing 
But anyway, if it is a major pain we’d be happy to consider a solution - just create a topic in the features category and reference this post.