Prior to relaxing any interior UVs on a shell you should attempt to untangle the border UVs on the exterior of the UV shell. You can do this using the Map UV Border feature and with the Straighten UV Border feature.
When you use Map UV Border you can move the UVs on the border of a UV shell out to the edges of the 0 to 1 texture space range (square), or in a circle inscribed within the 0 to 1 range. Mapping UVs so they use the entire 0 to 1 texture range is useful when you need to maximize the texture space required for a texture map.
You can optionally scale the edges proportionally as you move the border UVs. This is useful for untangling borders before you use Relax UVs to untangle interior UVs.
To map border UVs to a square or circle
The UV you select is important as it will determine where the UV shell will be mapped to the 0,0 origin of the UV texture space. If the resulting UV border mapping appears skewed, you may wish to undo it, and select another UV and map again.
If you are using the UV Editor, you can select .
The border UVs are mapped based on the settings you made. The interior UVs will remain unchanged. You can untangle the interior UVs using the Relax UVs feature.