Operating system and file system monitoring: A comparison of passive network monitoring with full ke

System monitoring is the process by which information about computer systems and users of those systems is collected. It is carried out typically to assist in improving the operation of current systems and to aid in the development of future computer systems. This thesis compares and contrasts two system-monitoring techniques: full kernel instrumentation, where information is obtained through the placing of instrumentation code into the computer's operating system (kernel), and passive network monitoring where data associated with one or more computer systems attached to a network are extracted by monitoring the communications tra c exchanged by those computers with others on the network. In order to achieve this comparison and the contrast of the two system-monitoring techniques, two such systems were implemented, operated in tandem and the results then compared. A kernel-instrumentation system was ported from an earlier revision of the UNIX operating system to a more up-to-date version. This involved developing a working knowledge of the kernel system in general and the development of programs to process the records generated by the kernel instrumentation into a format that could be then be compared with the passive network monitoring system. The passive system was substantially complete as received although some support software was written. Processing software also had to be written for summarising the records of each technique and for producing a comprehensive analysis of the trace record. An understanding of the rule base of the post-processing software was achieved by thorough investigation including instrumentation and trial operation. A simpli ed cache simulator was constructed which, although it could not be integrated into the passive network monitoring software, aided in an understanding of the cache system. viii Full kernel instrumentation has been a popular technique because of its comprehensive nature but it su ers, among other things, from the necessity to have the kernel source-code available, from the need to make changes to the system being monitored and from the impact it can have on that system. In contrast, passive network monitoring can be conducted in a non-invasive, platform-independent manner which involves no changes to the operation of the monitored machine. Passive network monitoring has the potential to be used in place of full kernel instrumentation for many tasks, and even though it was unable to give results comparable with full kernel instrumentation in all cases, it is able to give good predictions of many values when compared with those derived from full kernel instrumentation. This was particularly true in areas related to the writing of data. This thesis also notes discrepancies between the result from the two system-monitoring methods and discusses ways in which those discrepancies can be reduced or eliminated. ix Acknowledgements Firstly, I would like to thank my parents, Alan and Dorothy Moore, and my brother, Nicholas, for their endless and unwavering love and encouragement. Mere words cannot begin to express my appreciation and love. I could not have had a better combination of mentors than my supervisors, Tony McGregor and Jim Breen, They are both true and steady friends who willingly give their time and guidance. This thesis would have been at least di erent and, perhaps, impossible without access to a number of programs. The starting point of this thesis was when Matt Blaze made the original rpcspy/nfstrace suite of software available to me. He then passed on changes and additions as well as words of advice and encouragement. I am more than grateful to him. Songnian Zhou and Chris Siebenmann gave me access to the original snooper code and gave unstintingly of their time to answer many questions and to o er advice. Michael Dahlin willingly assisted me by supplying changes his work group had made to rpcspy/nfstrace. My thanks to you all. I am grateful to Randy Appleton, Mary Baker, Charles Briggs, Dan Eaves, Rick Floyd, Simon Hill, Rick Macklem, John Ousterhout, Alan Rollow and Margo Seltzer for unhesitatingly giving of their enthusiasm, sought-after papers, ideas and the conrmation that the work of this thesis was worthwhile. I owe an eternal debt to the sta of the Department of Robotics and Digital Technology and to the users of the machines used for testing for putting up with crashing clients and low performance networks. x My thanks to Andrew Lysikatos for organising access to the Hewlett Packard network analyser and to Kathy Ching for giving so much of her time to assist in obtaining the much-needed source code which made this thesis possible. I extend thanks to my friends, in particular Michelle Judson, Cameron Blackwood, Rik Harris and Chris Beecroft, for making the past years both interesting and enjoyable and to Cameron and Chris for proof-reading. Finally, special thanks to Ralphe Neill, who willingly read through and commented on early revisions of this thesis. xi Chapter

[1]  Matt Blaze,et al.  Dynamic hierarchical caching in large-scale distributed file systems , 1992, [1992] Proceedings of the 12th International Conference on Distributed Computing Systems.

[2]  Matthew Addison Blaze Caching in large-scale distributed file systems , 1993 .

[3]  Mahadev Satyanarayanan,et al.  Using belief to reason about cache coherence , 1994, PODC '94.

[4]  Rick Macklem,et al.  Lessons Learned Tuning the 4.3BSD Reno Implementation of the NFS Protocol , 1991, USENIX Winter.

[5]  Bjarne Stroustrup,et al.  C++ Programming Language , 1986, IEEE Softw..

[6]  Derek L. Eager,et al.  Disk cache performance for distributed systems , 1990, Proceedings.,10th International Conference on Distributed Computing Systems.

[7]  Mary Baker Fast crash recovery in distributed file systems , 1994 .

[8]  Michael D Kupfer Performance of a Remote Instrumentation Program , 1985 .

[9]  John K. Ousterhout,et al.  Why Aren't Operating Systems Getting Faster As Fast as Hardware? , 1990, USENIX Summer.

[10]  Hossein H. Ghannad,et al.  A trace-driven study of CMS file references , 1991, IBM J. Res. Dev..

[11]  Michael Dahlin,et al.  A quantitative analysis of cache policies for scalable network file systems , 1994, SIGMETRICS.

[12]  Alan Jay Smith,et al.  Disk cache—miss ratio analysis and design considerations , 1983, TOCS.

[13]  Rick Floyd Short-Term File Reference Patterns in a UNIX Environment, , 1986 .

[14]  A. L. Narasimha Reddy,et al.  An Evaluation of Multiple-Disk I/O Systems , 1989, IEEE Trans. Computers.

[15]  J. Howard Et El,et al.  Scale and performance in a distributed file system , 1988 .

[16]  Paul Anderson Effective Use of Local Workstation Disks in an NFS Network , 1992 .

[17]  Alan Jay Smith,et al.  Efficient (stack) algorithms for analysis of write-back and sector memories , 1989, TOCS.

[18]  SmithAlan Jay Long term file migration: development and evaluation of algorithms , 1981 .

[19]  James J. Kistler Disconnected Operation in a Distributed File System , 1995, Lecture Notes in Computer Science.

[20]  Randy H. Katz,et al.  An Analysis of File Migration in a UNIX Supercomputing Environment , 1993, USENIX Winter.

[21]  Matt Blaze NFS Tracing By Passive Network Monitoring , 1992 .

[22]  Mark Shand Measuring System Performance with Reprogrammable Hardware , 1996 .

[23]  Mendel Rosenblum,et al.  The LFS Storage Manager , 1990, USENIX Summer.

[24]  Jim Griffioen,et al.  Reducing File System Latency using a Predictive Approach , 1994, USENIX Summer.

[25]  Daniel A. Reed,et al.  File archive activity in a supercomputing environment , 1993, ICS '93.

[26]  Margo I. Seltzer,et al.  File System Performance and Transaction Support , 1992 .

[27]  Andrew R. Cherenson,et al.  The Sprite network operating system , 1988, Computer.

[28]  Larry Wall,et al.  Programming Perl , 1991 .

[29]  Mahadev Satyanarayanan,et al.  Andrew: a distributed personal computing environment , 1986, CACM.

[30]  Mendel Rosenblum,et al.  The design and implementation of a log-structured file system , 1991, SOSP '91.

[31]  Alan Jay Smith,et al.  A File System Tracing Package for Berkeley UNIX , 1985 .

[32]  Robbert van Renesse,et al.  Experiences with the Amoeba distributed operating system , 1990, CACM.

[33]  Kester Li Towards A Low Power File System , 1994 .

[34]  Carla Schlatter Ellis,et al.  Directory Reference Patterns in Hierarchical File Systems , 1989, IEEE Trans. Knowl. Data Eng..

[35]  Alan Jay Smith Analysis of Long Term File Reference Patterns for Application to File Migration Algorithms , 1981, IEEE Transactions on Software Engineering.

[36]  Brent B Welch,et al.  Naming, State Management, and User-Level Extensions in the Sprite , 1990 .

[37]  David A. Patterson,et al.  A case for networks of workstations (now) , 1994, Symposium Record Hot Interconnects II.

[38]  Michael D Kupfer An Appraisal of the Instrumentation in Berkeley UNIX 4.2BSD , 1985 .

[39]  Jeffrey C. Mogul,et al.  A Better Update Policy , 1994, USENIX Summer.

[40]  M. Satyanarayanan,et al.  A Usage Pro le and Evaluation of a Wide-Area Distributed File System , 1994 .

[41]  Mahadev Satyanarayanan,et al.  Disconnected Operation in the Coda File System , 1999, Mobidata.

[42]  John K. Ousterhout,et al.  A trace-driven analysis of name and attribute caching in a distributed system , 1992 .

[43]  Sanjeev Setia,et al.  Analysis of the Periodic Update Write Policy For Disk Cache , 1990, IEEE Trans. Software Eng..

[44]  Mary Baker,et al.  Measurements of a distributed file system , 1991, SOSP '91.

[45]  Michael K. Molloy,et al.  ILMON: a Unix Network Monitoring Facility , 1986 .

[46]  Dan Walsh,et al.  Design and implementation of the Sun network filesystem , 1985, USENIX Conference Proceedings.

[47]  Andrew S. Tanenbaum,et al.  A Comparison of Two Distributed Systems: Amoeba and Sprite , 1991, Comput. Syst..

[48]  Mary Baker,et al.  The Recovery Box: Using Fast Recovery to Provide High Availability in the UNIX Environment , 1992, USENIX Summer.

[49]  Mirjana Spasojevic,et al.  A Usage Proole and Evaluation of a Wide-area Distributed File System , 1994 .

[50]  Wilson C. Hsieh,et al.  The logical disk: a new approach to improving file systems , 1994, SOSP '93.

[51]  John A. Kunze,et al.  A trace-driven analysis of the UNIX 4.2 BSD file system , 1985, SOSP '85.

[52]  John Wilkes,et al.  A trace-driven analysis of disk working set sizes , 1993 .

[53]  Brent B. Welch,et al.  The File System Belongs in the Kernel , 1991, USENIX MACH Symposium.

[54]  Sanjeev Setia,et al.  Optimal Write Batch Size in Log-Structured File Systems , 1994, Comput. Syst..

[55]  Mary Baker,et al.  Non-volatile memory for fast, reliable file systems , 1992, ASPLOS V.

[56]  John R. Nicol,et al.  Technological perspective on multimedia computing , 1991, Comput. Commun..

[57]  R. S. Fabry,et al.  A fast file system for UNIX , 1984 .

[58]  Jeffrey C. Mogul,et al.  Measured capacity of an Ethernet: myths and reality , 1988, CCRV.

[59]  David R. Cheriton,et al.  Decentralizing a global naming service for improved performance and fault tolerance , 1989, TOCS.

[60]  Brent B. Welch Measured Performance of Caching in the Sprite Network File System , 1991, Comput. Syst..

[61]  K. K. Ramakrishnan,et al.  Performance Analysis of Mass Storage Service Alternatives for Distributed Systems , 1989, IEEE Trans. Software Eng..

[62]  Mahadev Satyanarayanan,et al.  Long Term Distributed File Reference Tracing: Implementation and Experience , 1996, Softw. Pract. Exp..

[63]  Bruce E. Keith Perspectives on NES File Server Performance Characterization , 1990, USENIX Summer.

[64]  Robert Metcalfe,et al.  Ethernet: distributed packet switching for local computer networks , 1988, CACM.

[65]  B. Clifford Neuman,et al.  Kerberos: An Authentication Service for Open Network Systems , 1988, USENIX Winter.

[66]  Mahadev Satyanarayanan,et al.  The Influence of Scale on Distributed File System Design , 1992, IEEE Trans. Software Eng..

[67]  Mahadev Satyanarayanan,et al.  A study of file sizes and functional lifetimes , 1981, SOSP.

[68]  Douglas W. Clark,et al.  Measuring VAX 8800 performance with a histogram hardware monitor , 1988, [1988] The 15th Annual International Symposium on Computer Architecture. Conference Proceedings.

[69]  John H. Hartman,et al.  The Zebra striped network file system , 1995, TOCS.

[70]  John Wilkes,et al.  UNIX Disk Access Patterns , 1993, USENIX Winter.

[71]  A. Retrospective,et al.  The UNIX Time-sharing System , 1977 .

[72]  M. Satyanarayanan,et al.  Long Term Distributed File Reference Tracing: Implementation and Experience , 1996 .

[73]  Rafael Alonso,et al.  Long-Term Caching Strategies for Very Large Distributed File Systems , 1991, USENIX Summer.

[74]  J. Feder The UNIX system: The evolution of UNIX system performance , 1984, AT&T Bell Laboratories Technical Journal.