Use these options to set what happens when you select
.Select the shape type that you want to map the UV border to (Square or Circle).
Turn on Automatic to map the UV border automatically in a shape that approximates the best use the texture space while maintaining the world-space relationships between edges.
Determines how closely the border mapping will match the selected Border Target Shape. Use the slider or enter a value between 0 and 1 to control the blend of the existing UV border with the specified Border Target Shape. A value of 0 produces a UV border mapping that is closest to the specified target shape (Square or Circle). A value of 1 produces a border that closely represents the world-space relationships between edges, but can result in concave areas along the border—these will produce overlapping interior UVs once relaxed.
If you have a piece of non-manifold geometry, any particular UV may be in more than one border/shell, so it does not uniquely determine a flat piece of mesh to relax. Map UV Border will find a boundary and map it, but you can't control which one it will get.
To work around this you can cut the UVs along the non-manifold edges to make sure that they're “flat”, or use Cleanup to eliminate the non-manifold areas first.