You can redefine the boundary of a layout viewport by using the VPCLIP command. You can either select an existing object to designate as the new boundary, or specify the points of a new boundary. The new boundary does not clip the old boundary, it redefines it.
A nonrectangular viewport consists of two objects: the viewport itself and the clipping boundary. You can make changes to the viewport, the clipping boundary, or both.