Drag horizontally in all modes to slide the view forward and backward in time. Drag vertically to slide the view up and down in value.
If you have a middle mouse button, pressing it and dragging instantly pans in Track View, just as it does in viewports. (If you are in Maya mode, use Alt + the middle mouse button to pan.) Use Ctrl+Alt + the middle mouse button to zoom (Maya mode: Alt + the right mouse button).
You can use the wheel on a wheel mouse to zoom and pan the Track View Key window. Drag with the wheel to pan, and roll the wheel to zoom.
To pan in the Key window: