Stick: A Harmonious Fusion of Buffer-based and Learning-based Approach for Adaptive Streaming

Off-the-shelf buffer-based approaches leverage a simple yet effective buffer-bound to control the adaptive bitrate (ABR) streaming system. Nevertheless, such approaches in standard parameters fail to always provide high quality of experience (QoE) video streaming services under all considered network conditions. Meanwhile, state-of-the-art learning-based ABR approach Pensieve outperforms existing schemes but is impractical to deploy. Therefore, how to harmoniously fuse the buffer-based and learning-based approach has become a key challenge for further enhancing ABR methods. In this paper, we propose Stick, an ABR algorithm that fuses the deep learning method and traditional buffer-based method. Stick utilizes the deep reinforcement learning (DRL) method to train the neural network, which outputs the buffer-bound to control the buffer-based approach for maximizing the QoE metric with different parameters. Trace-driven emulation illustrates that Stick betters Pensieve by 3.5% - 9.41% with an overhead reduction of 88%. Moreover, aiming to further reduce the computational costs while preserving the performances, we propose Trigger, a light-weighted neural network that determines whether the buffer-bound should be adjusted. Experimental results show that Stick+Trigger rivals or outperforms existing schemes in average QoE by 1.7%-28%, and significantly reduces the Stick’s computational overhead by 24%-61%. Meanwhile, we show that Trigger also helps other ABR schemes mitigate the overhead. Extensive results on real-world evaluation demonstrate the superiority of Stick over existing state-of-the-art approaches.

[1]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[2]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Inference , 2016, ICLR.

[3]  Cisco Visual Networking Index: Forecast and Methodology 2016-2021.(2017) http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual- networking-index-vni/complete-white-paper-c11-481360.html. High Efficiency Video Coding (HEVC) Algorithms and Architectures https://jvet.hhi.fraunhofer. , 2017 .

[4]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[5]  Xin Jin,et al.  Can Accurate Predictions Improve Video Streaming in Cellular Networks? , 2015, HotMobile.

[6]  Te-Yuan Huang,et al.  A buffer-based approach to rate adaptation: evidence from a large video streaming service , 2015, SIGCOMM 2015.

[7]  Iraj Sodagar,et al.  The MPEG-DASH Standard for Multimedia Streaming Over the Internet , 2011, IEEE MultiMedia.

[8]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[9]  Carsten Griwodz,et al.  Commute path bandwidth traces from 3G networks: analysis and applications , 2013, MMSys.

[10]  Peter Stone,et al.  Behavioral Cloning from Observation , 2018, IJCAI.

[11]  Pieter Abbeel,et al.  An Algorithmic Perspective on Imitation Learning , 2018, Found. Trends Robotics.

[12]  Luca De Cicco,et al.  ERUDITE: a deep neural network for optimal tuning of adaptive video streaming controllers , 2019, MMSys.

[13]  Yi Sun,et al.  CS2P: Improving Video Bitrate Selection and Adaptation with Data-Driven Throughput Prediction , 2016, SIGCOMM.

[14]  Ali C. Begen,et al.  An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP , 2011, MMSys.

[15]  Tim Kraska,et al.  Park: An Open Platform for Learning-Augmented Computer Systems , 2019, NeurIPS.

[16]  Vyas Sekar,et al.  Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE , 2012, CoNEXT '12.

[17]  Yue Zhang,et al.  Buffer-Based Reinforcement Learning for Adaptive Streaming , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[18]  Bowen Zhou,et al.  A Structured Self-attentive Sentence Embedding , 2017, ICLR.

[19]  Bruno Ribeiro,et al.  Oboe: auto-tuning video ABR algorithms to network conditions , 2018, SIGCOMM.

[20]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[21]  Federico Chiariotti,et al.  D-DASH: A Deep Q-Learning Framework for DASH Video Streaming , 2017, IEEE Transactions on Cognitive Communications and Networking.

[22]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[23]  Ramesh K. Sitaraman,et al.  From theory to practice: improving bitrate adaptation in the DASH reference player , 2018, MMSys.

[24]  Andreas Schmidt,et al.  Cross-Layer Effects on Training Neural Algorithms for Video Streaming , 2018, NOSSDAV.

[25]  Van Jacobson,et al.  BBR: Congestion-Based Congestion Control , 2016, ACM Queue.

[26]  Ramesh K. Sitaraman,et al.  BOLA: Near-Optimal Bitrate Adaptation for Online Videos , 2016, IEEE/ACM Transactions on Networking.

[27]  Yuan Tang,et al.  TF.Learn: TensorFlow's High-level Module for Distributed Machine Learning , 2016, ArXiv.

[28]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[29]  Hari Balakrishnan,et al.  Mahimahi: Accurate Record-and-Replay for HTTP , 2015, USENIX Annual Technical Conference.

[30]  Lifeng Sun,et al.  Comyco: Quality-Aware Adaptive Video Streaming via Imitation Learning , 2019, ACM Multimedia.

[31]  Hongzi Mao,et al.  Neural Adaptive Video Streaming with Pensieve , 2017, SIGCOMM.

[32]  Christian Timmerer,et al.  A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP , 2019, IEEE Communications Surveys & Tutorials.

[33]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[34]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[35]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[36]  Yuandong Tian,et al.  Real-world Video Adaptation with Reinforcement Learning , 2019, ArXiv.

[37]  Lifeng Sun,et al.  Tiyuntsong: A Self-Play Reinforcement Learning Approach for ABR Video Streaming , 2018, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[38]  Bruno Sinopoli,et al.  A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP , 2015, Comput. Commun. Rev..