What is RemoteViz? Basic Concepts Getting Started with RemoteViz API (HelloCone) Main Service Settings Service Listener RenderArea RenderArea Listener Scene Graph
Client - Display a Scene Build and Run the Example Demo in Docker User Interaction Client-Service Communication Frames Encoding Secure Connection
RemoteViz is an image streaming toolkit. RemoteViz enables true web-based 2D/3D visualization that runs on a server and displays in the user's web browser. As images are rendered by the visualization software, RemoteViz streams them to the browser, using video encoding algorithms (H.264, VP9) to minimize required bandwidth, and GPU accelerated encoding to maximize performance. Remote desktop technologies can be used to "lift and shift" an existing desktop application into the Cloud, but that solution has many limitations. RemoteViz is designed to support visualization as a microservice in a modern web-based application. RemoteViz has many features but primarily solves three key problems.
First, to be interactive, a visualization service like a 3D viewer must deliver at least twenty images per second to the user's machine and each image may contain millions of pixels. That's a lot of data. RemoteViz manages the encoding and transmission of images to minimize bandwidth, maximize frame rate, and maintain image quality. Video encoding formats like H.264 and VP9 can dramatically reduce the bandwidth requirements. And using GPU acceleration can optimize performance, both for encoding (on the server-side) and for decoding (on the browser-side) the image stream. RemoteViz has no minimum bandwidth requirement and can adapt to the current network conditions by prioritizing either the image quality or the frame rate.
Third, and finally, RemoteViz allows web-based visualization applications to take full advantage of Cloud-native services. RemoteViz can be used, for example, to quickly implement a 3D viewer as a standalone web-based application. But RemoteViz is really designed to integrate visualization into a web-based application using a modern microservice architecture. For example, we provide examples showing how to integrate a RemoteViz visualization service with commonly used services like authentication and load-balancing. A RemoteViz service can be deployed in a public or private Cloud or on-premises. RemoteViz is compatible with Docker, Kubernetes, and other tools.
As described in the previous section, we believe RemoteViz is the best solution because it enables the implementation of a Cloud-native application on the server-side, provides state of the art image streaming between the server and browser, and integrates with the local user interface on the browser side.
But first, you might ask: Why implement a visualization application as a Cloud-native service using server-side rendering and image streaming? There are many general reasons for implementing an application in the Cloud, such as Cloud-native services, scalable resources, global access, and easy updating of application code. We donÃ¢Â€Â™t need to repeat those here. There are also specific reasons for using server-side rendering.
Number one is data. To use a local rendering solution, some portion of your applicationÃ¢Â€Â™s data must be transferred to the user's machine. with server-side rendering, your data stays in the Cloud. Only rendered images are transferred to the user's machine. Many applications need to keep data on the server for security. Medical applications generally need to keep data on the server for privacy. Oil and gas applications generally need to keep data on the server because of the size. A single seismic volume may be hundreds of gigabytes or even a terabyte of data. Keeping one copy of the data in the Cloud allows users to share while reducing storage expense and time-consuming copies. Data stored in the Cloud can be efficiently streamed to the services that need it using high speed networks. Visualization libraries, like Open Inventor, are designed for this scenario. Using "tiled" data storage, Open Inventor allows random access to any part of a large volume data set and automatically computes the data blocks needed to render the best possible image based on the server hardware, camera position, image size, and other factors.
Number two is hardware capability. When you use a local rendering solution, the application is limited to the CPU, GPU, and memory resources on the user's machine. Local rendering can be a good solution for applications with small data sets and relatively simple rendering requirements. For applications with larger data sets, the memory on the local machine may limit the amount of data that can be rendered or prevent the user from using full resolution data. For applications that require more complex rendering, like volume ray-casting, the GPU on the local machine may limit the image quality or even prevent the user from using some features. Using server-side rendering with RemoteViz streaming, users can view and interact with their data on almost any device. On the server-side, the application or the user can decide what kind of hardware instance to use. Users can have the capacity and rendering quality of a high-end workstation without the acquisition and maintenance costs.
RemoteViz image streaming works with any visualization library. As an initial step in the migration to the Cloud, some customers have even ported their existing applications to use as a "head-less" render engine. Commands from the local user interface are passed to the render engine and RemoteViz is used to stream the rendered images to the userÃ¢Â€Â™s browser. However, it is worth noting that the Open Inventor visualization library includes a highly optimized integration of RemoteViz streaming. For example, using Open Inventor, it is not necessary to upload each rendered image from the GPU before compression and encoding can be done. Using Open Inventor, RemoteViz is able to compress each rendered image on the GPU and only upload the resulting, much smaller, block of data. Open Inventor is also a powerful visualization and image processing library with many features specifically designed to increase the value of oil&gas and medical applications.
This guide is intended to walk you through creating your first basic RemoteViz application. It will not make you a RemoteViz expert or teach you how to write and deploy a real web application. However, we mention some "next steps" topics and provide links to more information about using RemoteViz.
Even though we use the Open Inventor library to provide visualization for our example program, RemoteViz can be used with any visualization library.
Note: The example code assumes that you are using Open Inventor version 10.6 or higher. RemoteViz is available for older versions but there will be some differences in the method signatures.
There are three components of a RemoteViz application: the service, the client, and a web server. The RemoteViz Service is basically an Open Inventor application with no viewer that runs on the server-side. It is in charge of data access, computing, and rendering. It renders the scene and sends it to the client via a web-socket connection. In most cases, RemoteViz Client is a web browser page hosting a render area. The client accepts the images and displays them on a render area. The client also captures the user input events and sends it to the service to be handled as Open Inventor events. In a production environment, a web-server hosts the client and manages web-socket communication between the client and the server.
|1. Service is in charge of executing the Open Inventor code
- Interprets events from the client
- Renders the scene using Open Inventor
- Sends images to the client
2. Web Server serves the web-page containing the render area
- Communicates between the service and the client
3. Client receives images and displays them
- Captures user requests & system events
- Sends them to the RemoteViz service
A RemoteViz application, by default, runs in Inventor Service mode and uses Open Inventor render engine. However, you may use your own render engine in a RemoteViz application by using the Independent Service run mode.
Important note : RemoteViz is not usable as Windows Service. Indeed, Windows creates the service processes in an isolated session that does not have access to the video driver (no OpenGL context found). This means that any attempt that a service makes to render graphics fails.
RemoteViz is protected by the Open Inventor license key mechanism limiting its usage to specified computers or network environments based on commercial agreements. RemoteViz uses floating license. One RemoteViz license is required to start the service. (An SDK license for development/debugging or a Runtime license for deployment.) Moreover, if the Service uses Open Inventor for rendering, then one Open Inventor license plus one license of each used extension is required to start the service. (An SDK license for development/debugging or a Runtime license for deployment.) For more details about floating server license configuration, please refer to the Open Inventor licensing section.
To use an Open Inventor extension with RemoteViz, please refer to the documentation of the method
In this section, you will learn the basics of programming using the RemoteViz API. We will walk through how to create a simple rendering service. We will then implement listener classes to handle notifications from the service and the render area.
This guide basically implements the "HelloCone" RemoteViz example program. You can find it in the $(OIVHOME)/examples/source/RemoteViz directory.
First, include the header files for Service, ServiceSettings, and ServiceListener classes.
Now add the following using statement before the main function. Note that all RemoteViz classes are defined inside a namespace, so the using directive allows RemoteViz classes to be more conveniently used without RemoteViz.Rendering as an explicit qualifier.
In the main( ) function, define settings such as IP address and port number of the rendering service.
If the service IP and port number are passed as command line arguments, then you need to access the values from argument vector (argv).
Next, you have to create an instance of
Use the method
Usually, you will attach a listener to the service object before using it. The listener receives notifications from the rendering service and responds accordingly. (More on the ServiceListener and HelloConeServiceListener classes in the next section.)
Before using the service, you must first open it with
Notice that after the program exits the main loop due to some interruption, you must close the rendering service using
So far you have defined a rendering service but it does not render anything. Next, we will implement a
The service listener listens for events and notifications from the RemoteViz service and allows an application to respond. In your application, you will implement a listener class derived from the base class
The listener is notified of events from the service and a typical sequence of calls to the service listener is:
onConnectedClient()- Client object has been created and is connected to the service onPendingCreateRenderArea()/ onPendingShareRenderArea()- RenderArea object is about to be created or shared onInstantiatedRenderArea()- RenderArea object has been created onInitializedClient()- Client is running and bandwidth calibration has finished Application is running ... onDisposingRenderArea()- Connection to RenderArea has been closed so it is to be disposed onDisposedRenderArea()- RenderArea object has been disposed onDisconnectedClient()- Client object has been disposed
Usually, you will define a listener class derived from
RemoteViz.Rendering.RenderArea> renderArea) override
/// Implementation in next section
This class also provides access to the touch manager for managing gesture events from touch screen devices. Use the
A RenderArea listener receives notifications from a
- Connection: Triggered when a connection has been created, initialized and disposed
- Rendering: Triggered at the start/end of rendering and when an image is sent to the client
- Input events: Triggered on mouse, keyboard and touch events
- Messages: Triggered when a message is sent from the client
To manage these events and notifications, you need to implement a listener class derived from the base class
onOpenedConnection()- Connection object has been created onInitializedConnection()- Connection is fully initialized
- While application is rendering ...
onPreRender()- Scene will be rendered onPostRender()- Scene has been rendered onSendingFrame()- Rendered image will be sent to the client
onClosedConnection()- Connection object has been disposed
Usually you will attach a listener object to the
Next, we will create a scene graph for rendering and assign it to the RenderArea inside this virtual method.
Classic Open Inventor examiner viewers are not available for RemoteViz applications, so we will use a
RemoteViz.Rendering.RenderArea> renderArea) override
// Instantiate a SceneExaminer to interact with the camera
SceneExaminer *examiner = new SceneExaminer();
To manage events from touch screen devices, register gesture recognizers using the touch manager.
RemoteViz.Rendering.RenderArea> renderArea) override
// Add recognizers for gesture events
In this section, we will create a web-viewer for client application using the RemoteViz Client API. We will first define HTML header and body, then implement a method to initialize a render area and connect to the service.
You need to add the following code sample to define an HTML header in the index.html. The first line specifies the
Here, you have to declare the web application as HTML5 by using the
Next, we will define a minimal HTML body that contains a couple of
Now let's implement the
Next, add a listener to capture predefined messages/events from the RemoteViz service. Only events related to connection and network calibration states are captured by the service handler. For a complete list of predefined messages, please see
The following code sample defines a handler function
Finally, to connect the client application to the RemoteViz service, we will use the
The service URL string must contain the address and port of the RemoteViz service. The address can be a domain name or an IP address. The URL may contain a render area identifier and a query string composed of a series of field-value pairs.
The following URL, for example, includes a query string requesting specific height and width.
"TheCone" in the above URL specifies the identifier (Id) of the requested render area. If no render area with specified Id exists, the RemoteViz service will create one and
Now that we have covered the basics of programming with the RemoteViz API and implemented a minimal rendering service as well as a client application, let's run a RemoteViz example locally.
To build the demo example using Visual Studio on Windows, you will need to ...
- Open the solution (.sln) file located in $(OIVHOME)/examples/source/RemoteViz/HelloCone/HelloConeRenderingService/
- Select either Debug or Release configuration and build the RemoteVizHelloConeRenderingService project.
The service executable is created in the default output directory $(OIVHOME)/examples/bin/arch-Windows-x86_64-msvc15-$(Configuration)/RemoteViz/.
To start the rendering service, navigate to the output directory (using either File Explorer or PowerShell) and run RemoteVizHelloConeRenderingService.exe. If successful, you will see a message saying "... Rendering Service is running".
Notice that the service is using the IP address 127.0.0.1 and port 8080 as defined in main( ) function.
Once the service is running, let's navigate to Clients/HTML5/ and open index.html in your browser. If the render area is successfully created, you should first see the "Network Calibration" message and then an image of a gray cone on the render area.
Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries - anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment.
Containers running on a single machine share the same operating system kernel; they start instantly and use less RAM. Images are constructed from layered filesystems and share common files, making disk usage and image downloads much more efficient.
Docker can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build, users can create an automated build that executes several command-line instructions in succession.
We provide the Dockerfile necessary to build a container image that can run a demo. A linux distribution is required and NVIDIA graphics card too.
To run a RemoteViz demo, an OpenGL context is required. To get an OpenGL context in a Docker container, you have to install the Nvidia container toolkit on the host machine and use the Open Inventor headless package.
Run the RemoteVizHelloCone demo in a Docker container Be sure you have copied your password.dat in OIVHOME/License folder and set the Open Inventor environment variables: OIVHOME, OIVARCH.
Go to the Docker folder in OIVHOME/examples/source/RemoteViz/Docker. Open the ReadMe.txt file and follow the instructions.
Usually, the client applications are highly interactive, and they must provide some way for the user to interact with the scene. We use a
SceneExaminer is either in NAVIGATION mode (the default, similar to viewing mode) or SELECTION mode. The user must press the ESC key to toggle between the two interaction modes. In NAVIGATION mode, user input events are automatically handled to modify the camera, and these events are not sent to the application scene graph. In SELECTION mode, all events are sent to the application scene graph, and the user input events can be captured and processed appropriately by the application on the server-side using
You may also capture the user input events in the
The following code sample implements a listener that is triggered when a MouseUp event is received from the client. For a complete set of listener methods that you may override, please see
RemoteViz.Rendering.Connection> sender,int x,int y,
// Application implementation
// Return true to process the event; forward it to be handled by the scene graph
// Return false to ignore the event.
The RemoteViz service and the client application can communicate by exchanging application specified messages, which can either be a binary encoded or text messages. The support of binary encoded messages is particularly useful when using tools like
To send a message from the client application to the RemoteViz service, use the
Likewise, to send a message from the service to the client applications, we will use the
The following code sample adds a listener on messages received from the service.
When a client connects to the RemoteViz service, by default, RemoteViz will use PNG encoding for still frames and JPEG for interactive frames. In the latter case, it may be acceptable to use (for example) lossy JPEG encoding to maximize performance, then switch to loss-less PNG encoding for the "still" frame when the user interaction is finished.
Video encoding (H.264 or VP9) generally provides better performance (frames per second) than image encoding (JPEG and PNG). However, this depends on many factors. Video encoded frames usually require less bandwidth, but the encoding of each frame can take more time.
To define what encoder to use for still and interactive frames, you need to access a
You may use the
Note that, currently, when using video encoding, the interactive and still encoders must be the same, e.g. both H264_NVENC.
RemoteViz uses the WebSocket protocol to enable communication between the client and the service. In a production environment, it is highly recommended to use secure WebSocket. Using a secure WebSocket connection improves confidentiality and reliability because the connection is encrypted with Transport Layer Security (TLS), which reduces the risk of a variety of attacks such as man-in-the-middle tampering with your data.
The WebSocket Secure (WSS) protocol is to WebSocket protocol (WS) what HyperText Transfer Protocol Secure (HTTPS) is to HyperText Transfer Protocol (HTTP). Like HTTPS, WSS requires a TLS certificate issued to a fully qualified domain name such as www.openinventor.com.
To enable a secure connection between the client and the rendering service, you need to use the
If the private key is protected by a passphrase, you can provide the passphrase using the
To connect the client application to the RemoteViz service using a secure connection, you will use the
The last step is to setup a web server like Apache or Nginx to serve the web page index.html over your domain name and an HTTPS connection using your TLS certificate. The web server has also to be configured as a reverse proxy to redirect WSS incoming connections to the rendering service. Further information about
Class Summary Class Description ClientRepresents a client application instance using RemoteViz. ClientSettingsSettings that define a
Connection ConnectionParametersField-value pairs included in the URL when the client requested a connection. ConnectionSettingsSettings that define a
EncodedFrameThis class provides the features of an encoded frame that is sent to the client. FrameEncodersDefines a pair of encoders that are used to encode still and interactive frames. HTTPHeadersThis class represents the header configuration options for an HTTP request and response. HTTPRequestThis class encapsulates an HTTP request message received by the service. HTTPResponseThis class encapsulates an HTTP response message. IFrameEncodingPolicy KeepFrameQualityPolicyThis class provides a calculation policy which allows to manage the FPS (frames per second) and the quality of interactive frames sent from Remoteviz service depending on the network bandwidth allocated to the
KeepFramesPerSecondPolicyThis class provides a calculation policy which allows to manage the FPS (frames per second) and the quality of interactive frames sent from Remoteviz service depending on the network bandwidth allocated to the
MetricsListenerThis class can be overridden by an application to monitor the service. MonitoringThis class enables to manage metrics listeners that monitor states and performance of the service. NetworkPerformanceThis class manages measures of service quality of the network. RenderAreaDefines the rendering area for Open Inventor rendering. RenderAreaHardwareSettings to setup hardware for a render area. RenderAreaListenerThis class can be overridden by an application to receive notifications from a
ServiceDefines the RemoteViz rendering service. ServiceListenerThis class can be overridden by an application to receive notifications from the rendering
ServiceSettingsSettings that define the rendering
UriThis class represents a Uniform Resource Identifier (URI).
Enum Summary Enum Description EncodedFrame.EncodingFormatsEncoding format. FrameEncoders.EncodersFrame encoder. FrameEncoders.StatusFrame encoder status. HTTPResponse.HTTPStatusEnum for the HTTP status codes. Monitoring.MetricTypesTypes of metrics. ServiceSettings.ExtensionsEach enumeration represents an Open Inventor extension. ServiceSettings.RunModesThe run mode is used to change the behaviour of the RemoteViz execution in three environments. ServiceSettings.SecurityProtocolsEach enumeration represents a security protocol.