Disabling the labels system

Is there a way possible to disable the labeling system? I have a use case where we are not planning using it. I know that tags and categories can be disabled, but there is no admin disabling for labels, can this be done programatically?

If not, is there going to be a negative impact if I start hiding all the labeling stuff via CSS?