To delete a marker from a Marker set, select the name of the marker in the Model column, and press Delete on the keyboard.
If you click Delete in the Marker Set menu, you will delete the entire Marker set.