VRED comes with an integrated web server, which offers a Python API, Web API, and low latency video streaming. While the Python API and Web API use HTTP endpoints, the stream is provided via WebSocket for full-duplex communication.

When someone mentions VRED Stream, they are talking about the Stream JPG and Stream H264 features. These are live image streams of a VRED viewport. In the VRED Stream, you can navigate in the scene, use touch sensors, and interact with HTML 5 content.

The streaming image data is encoded as JPEG or H264. While H264 seems more natural in such a scenario, JPEG shouldn't be underestimated. It requires more bandwidth, but is pretty fast, requires no separate encoder for VRED, and needs no special decoder on the client-side. Additionally, the OpenH264 v2.x codec supports a maximum resolution of 3840 x 2160 only.

How to Access the VRED Stream Options

Your device or machine must be able to reach the PC VRED is running on. If using localhost, the browser must be running on the same machine as VRED.

1. Open a VRED scene.

2. Select Edit > Preferences > WebInterface and ensure Enable Web Server is activated.

   Note the port number. The default is 8888.

3. Click OK.

4. Open a web browser on your device.

5. Enter either localhost:8888 or YourIP:8888.

6. Select Stream JPEG or Stream H264.

Connection

The stream endpoint is {schema}://{host}:{port}/socketstream. E.g., this is a URI to connect to a local instance using TLS:

wss://localhost:8888/socketstream

The binary type of the WebSocket is arraybuffer.

Incoming Data

VRED sends two messages for each frame through the WebSocket connection. The first is a JSON serialized string containing the stream format and some additional information. The second is a binary containing the frame image data.

{
    "format": {
        "codec": "h264",
        "width": 1024,
        "height": 1320,
        "fps": 60,
        "bitrate": 8000,
        "quality": 85,
        "fixed": 0
    },
    "clients": 1
}

Name	Type	Description
format.codec	string	Stream codec used for the binary image data. Currently, supported codecs are jpeg and h264.
format.width	number	Width of the image data.
format.height	number	Height of the image data.
format.fps	number	Limit of maximum frames per second.
format.bitrate	number	Bitrate used for H264 codec.
format.quality	number	Image quality used for JPEG codec.
format.fixed	number	A boolean value that indicates whether the stream format is fixed and can't be changed.
clients	number	Number of clients connected to the same VRED instance.

The binary image data contains the frame number in the first 4 bytes. This needs to be extracted before the image data is sent to the decoder. Additionally, the frame number needs to be sent back to VRED to synchronize the frames. Find more information in Outgoing Data.

Outgoing Data

Starting from version 2021.3, VRED can automatically manage the number of frames a client is able to handle. For this, VRED needs feedback from the client to know which frame was rendered. The string message should look like this sync(${frame}).

In addition to the sync, navigation commands, in particular, can be sent to the VRED via this channel. Find a table of all commands below:

Command	Parameters	Description	Example
dblclic	button: number, keyModifier: number, x: number, y: number	Mouse double click	-
keypress	keyCode: number, keyModifier: number, x: number, y: number	Key press or key down	keypress(-10, 0x04000000, 483, 281)
mousedown	button: number, keyModifier: number, x: number, y: number	Mouse button down	mousedown(-10, 0x02000000, 483, 281)
mouseup	button: number, keyModifier: number, x: number, y: number	Mouse button up	mouseup(-10, 0, 483, 281)
mousewheel	delta: number, keyModifier: number, x: number, y: number	Mouse wheel change	mousewheel(-10, 0, 483, 281)
sync	frame: number	Synchronize frames between VRED and the client.	sync(128)
update	-	Forces VRED to render a new frame.	update()

Parameters explained:

• button value represents the mouse button: Left button = 0, Middle button = 1, Right button = 2
• keyCode represents the unmodified decimal ASCII value of the pressed key
• keyModifier value is set bitwise for each key modifier pressed: Alt key = 0x08000000, Ctrl key = 0x04000000 and Shift key = 0x02000000
• x and y represents the mouse position relative to the stream image size

Every interaction causes VRED to render at least one new frame. Further frames depend on the render settings. E.g., VRED will send multiple frames when antialiasing is enabled.

Change Format

The stream format can be changed through a separate endpoint /stream/param?. E.g., this is a URI to change the format using full HD and JPEG encoding to a local instance using TLS:

GET https://localhost:8888/stream/param?width=1920&height=1080&format=jpeg&quality=85&fps=30

Find a table of all parameters below:

Name	Type	Description
format	string	Stream codec, can be jpeg or h264.
width	number	Width of the stream image.
height	number	Height of the stream image.
fixed	boolean	Indicates that the resolution is fixed and can't be changed until a fixed = false is sent.
fps	number	Limits the maximum number of frames sent to the clients per second.
bitrate	number	Bitrate for an H264 codec that is usually a value between 2000 and 16000.
quality	number	Image quality for a JPEG encoder with a value between 1 for best compression and 100 for best quality.