To deselect an element, while pressing Shift, click the element. Other selected elements remain selected.
The selection count on the status bar () updates to display the new number of selected items. (See How Many Elements Are Selected?)