Layers > Delete > Unused Layers

Deletes all layers with no objects on them.