You can only restore a selection while Modify is active. If you switch to a different tool, the selection is cleared.
To restore a selection (of one or more elements) that you have cleared, while pressing Ctrl, press the Left Arrow on the keyboard. Alternatively, you can right-click anywhere in the drawing area, and click Select Previous.