Use these options to set what happens when you select .
Select the shape type that you want to map the UV border to (Square or Circle).
Turn on Automatic to map the UV border automatically in a shape that approximates the best use the texture space while maintaining the world-space relationships between edges.
Determines how closely the border mapping will match the selected Border Target Shape. Use the slider or enter a value between 0 and 1 to control the blend of the existing UV border with the specified Border Target Shape. A value of 0 produces a UV border mapping that is closest to the specified target shape (Square or Circle). A value of 1 produces a border that closely represents the world-space relationships between edges, but can result in concave areas along the border—these will produce overlapping interior UVs once relaxed.

If you have a piece of non-manifold geometry, any particular UV may be in more than one border/shell, so it does not uniquely determine a flat piece of mesh to relax. Map UV Border will find a boundary and map it, but you can't control which one it will get.
To work around this you can cut the UVs along the non-manifold edges to make sure that they're “flat”, or use Cleanup to eliminate the non-manifold areas first.