You can remove specific elements from a selection while keeping the other elements selected.
To deselect an element, while pressing Shift, click the element. Other selected elements remain selected.
The selection count on the status bar (
) updates to display the new number of selected items. (See
Determine How Many Elements Are Selected)