Virtual Reality: History Curs 7 Arhitecturi SW pentru RV Networked VE Virtual Reality: History Pentru dezvoltarea unui sistem complet, o medie de timp satisfacatoare este de 12 luni. Pentru dezvoltarea unui sistem complet, o medie de timp satisfacatoare este de 12 luni. Teamwork-ul joaca un rol fundamental. Teamwork-ul joaca un rol fundamental. Dezvoltarea unei aplicaii complete de RV este o activitate care necesit manipularea mai multor tehnologii i rezolvarea sarcinilor in anumite limite de timp. Sistemele de RV sunt complexe Virtual Reality: History Complexitatea sistemelor de RV crete Cresc cunotinele noastre cu privire la mecanismele de percepie uman Progresele tehnologice sunt lente dar constante: performanele generale ale sistemului sunt n cretere i blocajele sunt eliminate Complexitatea tot mai mare a sistemelor rezultate Virtual Reality: History Componente multiple Un sistem modern de VR este o colecie de mai multe module de nalt performan Fiecare component are un domeniu de cercetare propriu, i necesit tehnologii dedicate, instrumente i abordri specifice. Virtual Reality: History Grafica 3D "Interactivitatea" necesit un load foarte mare poligoane; Multe tehnici i algoritmi exist (sau sunt dezvoltate) pentru manipularea optim a bazelor de date foarte mari cu date geometrice; Virtual Reality: History Collision Detection It used to be a big thing in VR. Collision detection algorithms searches for virtual objects interferences Collision detection algorithms searches for virtual objects interferences Algorithms are almost optimal very mature research fieldAlgorithms are almost optimal very mature research field There is little point in collision detection without physic simulation There is little point in collision detection without physic simulation Revived a bit lately: GPU-based collision detection Revived a bit lately: GPU-based collision detection Virtual Reality: History Physics simulation Interaction between virtual objects need to be simulated with virtual physics Objects in a VE should react to stimulus (gravity, external forces, collisions) Compound objects need to be considered, and mechanical effects of joints between rigid bodies simulated. Well known simulation models, but again the hard task is simulation in real-time. Opportunities & challenges: Parallel computing and dedicated co- processors. Real-time simulation of deformable objects. Virtual Reality: History Positional sound Sound sources can be: Sound sources can be: Sampled: playback of a pre-existing files (with additional, post processing effects) Synthesized: sound is generated on-line Audio sources usually needs to be positioned in 3D space (and they move around) Audio sources usually needs to be positioned in 3D space (and they move around) Even when the sources are static the listener is moving Even when the sources are static the listener is moving Virtual Reality: History Interaction Devices Management Users may interact with a system using various types of devices and technology Users may interact with a system using various types of devices and technology Interaction can be 3DOF, 6DOF, force enabled, thought enabled and more Interaction can be 3DOF, 6DOF, force enabled, thought enabled and more New forms of interactions are emerging such as using portable BCI devices New forms of interactions are emerging such as using portable BCI devices Complex devices require dedicated APIs and access paradigms Complex devices require dedicated APIs and access paradigms Virtual Reality: History Haptic feedback Haptic-enabled manipulaton involves the users body perception-action feedback loop: still a vastly unexplored research field with many technological challenge. Advanced software techniques and hardware devices need to be developed to achieve realistic haptic rendering of rigid and soft objects Haptic interfaces are usually custom objects and depends on the applications Virtual Reality: History Inertia feedback In some VR systems the user is placed on a moving platform Stewart platform: six independently actuated legs, where the lengths of the legs are changed to position and orient the platform. Virtual Reality: HistoryNetworking No man is an island communication between remote VR nodes can be very important. No man is an island communication between remote VR nodes can be very important. Interleaved data streams: audio, video, geometry transformations, haptics and more. Interleaved data streams: audio, video, geometry transformations, haptics and more. Virtual Reality: History Developing VR apps: the big picture VR is really about interaction. From a software point of view, interaction is a time-critical activity VR is really about interaction. From a software point of view, interaction is a time-critical activity Consequence: any VR application is a collection of high performance, low latency, dedicated tasks Consequence: any VR application is a collection of high performance, low latency, dedicated tasks Tasks are not independent. There are cross-tasks implications that should always be considered Tasks are not independent. There are cross-tasks implications that should always be considered Virtual Reality: History What is the state of a VR application? 3D meshes In theory: a collection of information about the virtual world In theory: a collection of information about the virtual world In practice: a collection of data structures and variables In practice: a collection of data structures and variables VR World state VR World state Positions Orientations Forces Velocities Accelerations Virtual Reality: History The state evolves over time advancing in a discrete sequence of steps and following the simulation computation. advancing in a discrete sequence of steps and following the simulation computation. t0t0 t0t0 t1t1 t1t1 t2t2 t2t2 t3t3 t3t3 Virtual Reality: History Why does the state keep evolving? Mainly to accommodate the effects of two things: Mainly to accommodate the effects of two things: External events (mainly due to user input/tracking); Internal events (timers, network events) VR World state VR World state Button_A_pressed! Button_A_pressed! Joy_1_MOVED! Joy_1_MOVED! MOUSE_OVER! MOUSE_OVER! Time_01! Time_01! Time_02! Time_02! Time_03! Time_03! Virtual Reality: History Q: How frequently does the state evolve? A: Fast enough to keep up with the fastest activity taking place inside the simulation. A: Fast enough to keep up with the fastest activity taking place inside the simulation. Consequence: the internal update rate of the simulation depends on the VR system configuration and the tasks performed Consequence: the internal update rate of the simulation depends on the VR system configuration and the tasks performed t0t0 t0t0 t1t1 t1t1 t2t2 t2t2 t3t3 t3t3 Simulation step Virtual Reality: History Loops in a typical VR application ~ 60 Hz Hz >1kHz ~10Hz ~30Hz Virtual Reality: History The graphics loop The state of a VR simulation need to be displayed at regular intervals (framerate) on a monitor, HMD, CAVE, etc Update frequency = monitor refresh Anything more would be wasted State(t) 60Hz Virtual Reality: History The collision detection loop If collision checking is not fast-enough some collision could occur undetected If collision checking is not fast-enough some collision could occur undetected State(t) ??? Checking if moving objects collide with each other Virtual Reality: History The physical simulation loop Objects responds to physical laws and interact Objects responds to physical laws and interact Update frequency needs to occur at least at the same speed as display rate, but stability issues or haptic rendering often requires much higher speeds Update frequency needs to occur at least at the same speed as display rate, but stability issues or haptic rendering often requires much higher speeds State(t) >=100Hz Virtual Reality: History Real-time physics in practice (I) A world is a collection of bodies Bodies from different worlds never interact Each rigid body has: a mass (and moment of inertia) needed for motion calculation obtainable by composing elementary solids a shape or geometry needed for collision detection obtainable by composing elementary solids Note: we are considering rigid-bodies physics Virtual Reality: History Real-time physics in practice (II) Physics materials are used to describe surface properties of objects var material = sim.getMaterial(idx); material.restitution = 0.1; material.staticFriction = 0.8; material.dynamicFriction = 0.5; material.apply(); Virtual Reality: History Real-time physics in practice (III) Simple objects are connected using a collection of physical joints Simple objects are connected using a collection of physical joints Many types of joints are available Many types of joints are available Sometimes joints can be brokenSometimes joints can be broken Virtual Reality: History Real-time physics in practice The physics loop: 1)Apply all the external forces to the scene objects 2)Ask the Physics engine to compute the evolution of the scene during a certain time interval 3)Get the results back Update the scene-graph Update the scene-graph *optionally* : control some external device *optionally* : control some external device 123 Virtual Reality: History The haptic loop(I) Haptic refresh rate is much higher than graphics refresh rate! Haptic refresh rate is much higher than graphics refresh rate! Stiffness in haptic rendering is influenced by loop speed -> collision detection speed Stiffness in haptic rendering is influenced by loop speed -> collision detection speed State(t) >=500Hz Virtual Reality: History The haptic loop (II) 123 * Reading the device velocity can be useful too 1)Read the device current position* 2)Depending on that position compute a force 3)Ask the device to apply that force Position(x,y,z) Force(fx,fy,fz) Virtual Reality: History So, does this mean that physics must run at 500 Hz too? Well..., depends: 1)For optimal and more accurate computation and rendering of forces, yes. 2)To reduce computational workload virtual coupling can in some situations be used. 500 Hz State(t) 500 Hz? Physics Virtual Reality: History The idea behind virtual coupling A virtual spring is attached between the VR object and the tip of the haptic interfaceA virtual spring is attached between the VR object and the tip of the haptic interface The object is considered inamovable between two consecutive physics loopsThe object is considered inamovable between two consecutive physics loops 500Hz computation is only performed to compute (and apply) the virtual spring elongation forces500Hz computation is only performed to compute (and apply) the virtual spring elongation forces 100 Hz State(t) 100 Hz Physics Virtual Coupler 500 Hz Virtual Reality: History The inertia feedback loop Some VR applications use also a motion platform Some VR applications use also a motion platform Device control is typically performed at around 100Hz Device control is typically performed at around 100Hz State(t) 100Hz Virtual Reality: History Controlling a motion platform The platform controller waits for: The platform controller waits for: x, y, z (floating point) x, y, z (floating point) Roll, Pitch, Yaw Roll, Pitch, Yaw The system has a high mechanical inertia The system has a high mechanical inertia Position commands takes a while to get executed Virtual Reality: History Computing perceived acceleration A physic engine can be used to compute position, velocity and acceleration of the head proxy of the user. A physic engine can be used to compute position, velocity and acceleration of the head proxy of the user. Data is passed to a Washout filter to get transformedData is passed to a Washout filter to get transformed Washout filter Position, Velocity, Acceleration Position, Velocity, Acceleration Virtual Reality: History The data flow Position, Velocity, Acceleration Position, Velocity, Acceleration Washout filter x, y, z x, y, z Roll, Pitch, Yaw Roll, Pitch, Yaw Physics engine >=100Hz >=100Hz 60Hz 60Hz Virtual Reality: History From acceleration to inclination For low-frequency accelerations, the washout filters tries to use gravity to simulate forces.For low-frequency accelerations, the washout filters tries to use gravity to simulate forces. Washout filter Position, Velocity, Acceleration Position, Velocity, Acceleration Virtual Reality: History Network communication loop Communication between remote VR nodes can be very important Communication between remote VR nodes can be very important Update frequency depends on the type of communication (LAN,WAN), interaction (with or without haptics) etc Update frequency depends on the type of communication (LAN,WAN), interaction (with or without haptics) etc State(t) Node 2 State(t) Node 1 10Hz 500Hz Virtual Reality: History Conclusion: loops are everywhere ~ 60 Hz Hz >500Hz ~10Hz ~30Hz Requisite I for Integration: A precise, possibly deterministic, fine control on all the loops Requisite I for Integration: A precise, possibly deterministic, fine control on all the loops Virtual Reality: History Variables are shared between loops Requisite II for Integration : A good framework takes care of data synchronization Requisite II for Integration : A good framework takes care of data synchronization Graphics Physics Haptics VR World state Virtual Reality: History Same object, multiple representations Haptic shapes and and physics shapes Physics shapes only Virtual Reality: History Events: loops cant model everything ~30Hz VR_EVENT_ 1 VR_EVENT_ 3 VR_EVENT_ 2 VR_EVENT_ 4 Virtual Reality: History A better general overview Events and loops are orthogonal concepts: Requisite III for Integration : a good framework should handle both. Events and loops are orthogonal concepts: Requisite III for Integration : a good framework should handle both. Virtual Reality: History So, writing VR applications can be hard Usually requires excellent C++ skills. Usually requires excellent C++ skills. Hi-performance tools and libraries need to be properly handled (otherwise performances might suffer).Hi-performance tools and libraries need to be properly handled (otherwise performances might suffer). Plenty of tools available, but mixing them is not trivial. Redundancy in files/data structures/code waste a lot of resources. A LOT of work goes into the framework. Plenty of tools available, but mixing them is not trivial. Redundancy in files/data structures/code waste a lot of resources. A LOT of work goes into the framework. 1/30/ Software architectures for modern Virtual Environments Virtual Reality: History When we discuss networked nodes we can be referring to: Physical connections Physical connections - the wires between the computers Logical connections Logical connections - how the messages flow trough the nodes Virtual Reality: History Nodes communicates over a network Virtual Reality: History Fundamentals of Data Transfer (1) Network bandwidth Network bandwidth the rate at which the network can deliver data to the destination host (bits per second, bps) the rate at which the network can deliver data to the destination host (bits per second, bps) Network reliability a measure of how much data is lost by the network during the journey from source to destination host a measure of how much data is lost by the network during the journey from source to destination host types of data loss: dropping: the data does not arrive dropping: the data does not arrive corruption: the content has been changed corruption: the content has been changed Virtual Reality: History Fundamentals of Data Transfer (2) Network latency the amount of time required to transfer a bit of data from one point to another the amount of time required to transfer a bit of data from one point to another one of the biggest challenges: impacts directly the realism of the NVE experience one of the biggest challenges: impacts directly the realism of the NVE experience we cannot do much to reduce it we cannot do much to reduce itorigins speed-of-light delay speed-of-light delay endpoint computers, operating systems scheduling endpoint computers, operating systems scheduling the network itself, routers the network itself, routers Virtual Reality: History Fundamentals of Data Transfer (3) Network protocol a set of rules that two applications use to communicate a set of rules that two applications use to communicate with each other A protocol defines packet formats: understanding what the other endpoint is saying packet formats: understanding what the other endpoint is saying packet semantics: what the recipient should do when it receives a packet packet semantics: what the recipient should do when it receives a packet error behaviour: what to do if (when) something goes wrong error behaviour: what to do if (when) something goes wrong Virtual Reality: History BROADCAST based architectures In the ideal large-scale NVE design all nodes can communicate with others directly Advantages: Simple to implement Simple to implement Players can join at any time Players can join at any timeDisadvantages: Only supported by LANs Only supported by LANs System management and update a nightmare System management and update a nightmare eventually we cannot scale out eventually we cannot scale out Virtual Reality: History Client-server architectures Each player sends packets to other players via a server Advantages: Works over a WAN Works over a WAN easy to have reliable communication easy to have reliable communication Server can compress multiple packets into a single packet Server can compress multiple packets into a single packetDisadvantages: Nodes are server-dependent: server failure brings the system down Nodes are server-dependent: server failure brings the system down Server slows down the message delivery Server slows down the message delivery Virtual Reality: History PEER to PEER architectures Peer-to-peer: communication goes directly from the sending player to the receiving player (or a set of them) Peer-to-peer: communication goes directly from the sending player to the receiving player (or a set of them)Advantages: It works over WAN It works over WAN We manage to avoid servers We manage to avoid serversDisadvantages: Nodes need to know network configuration Nodes need to know network configuration Each node has to send same data multiple times Each node has to send same data multiple times Virtual Reality: History Multiplayer Server-Network Players can locate in the same place in the NVE, but reside on different servers Real World Virtual World Real World Virtual World Server-to-server connections transmit the world state information Server-to-server connections transmit the world state information Each server serves a number of client players Each server serves a number of client players Virtual Reality: History Internet Protocol Low-level protocols used by hosts and routers Low-level protocols used by hosts and routers Guides the packets from source to destination host Hides the transmission path (phone lines, LANs, WANs, wireless radios, satellite links...) Nodes are identified by 4 numbers ( ) Nodes are identified by 4 numbers ( ) Note that addresses can be Public (WAN) or Private (LAN) (more on this later) Note that addresses can be Public (WAN) or Private (LAN) (more on this later) Virtual Reality: History Internet Protocol IP is organised in layers. IP is organised in layers. Applications access Transport layer (UDP or TCP APIs) Applications access Transport layer (UDP or TCP APIs) As we move down the layers data gets more and more encapsulated As we move down the layers data gets more and more encapsulated Virtual Reality: History Traversing Multiple nodes Virtual Reality: History IP Transmission Techniques Unicasting Unicasting Single sender - single receiver Broadcasting Broadcasting Single sender but all nodes in the network are receivers Multicasting Multicasting Single sender but one or more receivers have joined a multicast group Virtual Reality: History TCP & UDP Transmission Control Protocol (TCP/IP) Point-to-point connection Reliable transmission using acknowledgement and retransmission Stream-based data semantics Big overhead Transmission and receiving NOT immediate Slower because of re-transmissions User Datagram Protocol (UDP/IP) Lightweight data transmission Differs from TCP connectionless transmission best-efforts delivery packet-based data semantics Packets are easy to process Transmission and receiving immediate Applications rarely use the IP directly but the protocols that are written on top of IP - Transmission Control Protocol (TCP/IP) - User Datagram Protocol (UDP/IP) Virtual Reality: History Selecting an NVE Protocol 1 Multiple protocols can be used in a single system. Which protocol should I use in my NVE? Using TCP/IP reliable data transmission between two hosts reliable data transmission between two hosts packets are delivered in order, error handling packets are delivered in order, error handling relatively easy to use relatively easy to use point-to-point limits its use in large-scale NVEs point-to-point limits its use in large-scale NVEs bandwidth overhead bandwidth overhead Virtual Reality: History Selecting an NVE Protocol 2 Using UDP/IP Lightweight Lightweight offers no reliability nor guarantees the order of packets offers no reliability nor guarantees the order of packets packets can be sent to multiple hosts packets can be sent to multiple hosts more complex services have to be implemented in the application more complex services have to be implemented in the application serial numbers, timestamps recovery of lost packets positive acknowledgement scheme negative acknowledgement scheme