A presentation of a single stream is expected to be compatible to its nominal presentation. For a presentation integrating multiple streams, time constraints among the stream objects define temporal synchronization requirements. Many educational digital library applications require delicate temporal synchronization between streams with skew limited at 80 msec. For example, a video stream of medical surgery integrated with an audio stream of explanation require precise synchronization between what are seen and heard.
An extreme case is the lip synchronization which defines the temporal relationships between an audio and visual streams for the particular case of human speaking. For such streams, experimental experience has demonstrated that abrupt skipping or pausing can result in presentation hiccups. Such hiccups may dramatically degrade presentation effectiveness, especially for lecture presentations where lip synchronization must be maintained.
We have observed that robust scheduling algorithms can be designed for the above temporal synchronization requirements. In such algorithms, the recovery of a presentation from asynchrony should be accomplished by gradually accelerating or retarding presentation components, rather than abruptly skipping or pausing the presentation.
Figure 2(a)
We designed a platform for initial experiments, as shown in Figure 2(a). The Real-Time Protocol (RTP) and buffer threads together read data from the network and insert it into the respective buffers.
During the presentation of the streams, each stream reports to the scheduler its current progress. The scheduler in turn reports to each stream the rendition rate required to maintain the desired presentation. The individual streams must follow this rendition rate.
We have conducted experiments for the presentation scheduler. Various QoS parameters were considered. The presentation scheduling must be performed so that the constraints defined by these parameters are met.
Figure 2(b) shows jitter between streams with synchronization points at a 250 msec interval. Audio and video stream rates were then superimposed; These vary between 20%.
The CPU was heavily loaded at three points, as indicated. While there is a momentary increase in jitter, the streams quickly return to synchronization. Under a load, the video stream is retarded, since the video decoding process requires significant CPU time. To compensate, the algorithm slows the audio stream and accelerates the video stream, thus reducing the jitter to zero. The user perceives only a momentary loss of synchronization, with no hiccups within the streams. Synchronization between streams is improved when the synchronization granularity is reduced. Figure 2(c) shows a plot of stream progress against time in milliseconds under a constant load.
Following the system model defined in Figure 2(a), we propose to develop a framework for presentation scheduling algorithms. Let the nominal rendition rate for a stream si be denoted by Rnsi and Rsi(t) be the rendition rate of si at a time instant t. The progress
psi(t) of a stream si at time t is the presentation time of granules displayed to date, assuming the nominal presentation rate. Let t0 be the start time of the stream. Various QoS parameters must be considered. For example, the three basic QoS parameters, maximum rate change csi, maximum instantaneous drift dsi and synchronization jitter j will result in the following constraints:
where si and sk are different streams.
We enforce these constraints by determining the progress of each stream at set synchronization points and assigning a value to Rsi(t) such that the constraints are met at the next point. The synchronization points occur at regular intervals D ts. In general, suppose we have n streams s1,..., sn. At each synchronization point, we can calculate a range for psi(t+ D ts) to satisfy constraints (1) and (2). We can then determine the progress of each stream at synchronization point t+ D ts which satisfies constraint (e) for all synchronized streams.
The selection of a progress rate at the next synchronization point can be guided by a variety of approaches. We may opt to minimize rate fluctuations by selecting a progress rate which is closest to the nominal rate. Alternatively, a progress rate may be chosen which would result in the least deviation from t+ D ts. Since close synchronization between the streams is of great importance, the minimization of jitter would be the highest priority. This would be the case for a lecture presentation where lip synchronization must be maintained. The detailed algorithms will be formulated to handle various cases.