You can remove unwanted changes to the palette by using the Reset button.
Resetting restores the palette to its original state, or to its state before you last clicked Apply.