Optimizing Applications on Cisco Networks

Team LiB
Next Section
Book Cover
• | Table of Contents
• | Index
Optimizing Applications on Cisco Networks
By Mike Hicks
Publisher | : Cisco Press
Pub Date | : November 11, 2004
ISBN | : 1-58705-153-2
Pages | : 384

Maximize the performance and availability of network applications and the network infrastructure with application performance management

  • Learn and understand the business case for application performance management

  • Understand how to identify applications critical to your business model

  • Identify basic application characteristics and how they can be optimized for transmission

  • Understand the principle of proactive planning in order to maintain the delivery criteria necessary for your business critical applications

  • Improve service delivery and outsourcing decisions by utilizing meaningful SLAs

  • Identify and solve network application performance problems, saving valuable time and money

  • Includes real-world case studies that cover design and performance issues

The astounding growth of computer networking in recent years has been characterized by the struggle of network managers to keep pace with the demand for network resources. As more and more users, running more and more applications, demand increased network bandwidth and performance, network managers find themselves adding expensive capacity in an effort to satisfy the needs of their customers. Yet networks remain congested, and applications continue to run slowly. Moreover, new aggressive applications, an ever expanding set of business stakeholders, and demands for greater reliability and security compound this struggle.

Optimizing Applications on Cisco Networks takes a practical look at how application performance management can assist in the end-to-end application delivery model by proactively identifying hot spots within an organization. This book covers network dependent application delivery, with an emphasis on optimizing the underlying infrastructure, procedures, and methodologies to aid and measure delivery. The book also covers applications, delivery techniques, and QoS protocols.

Use Optimizing Applications on Cisco Networks to identify and profile business critical applications, define and establish service-level criteria, and ensure that these system configurations are continually assessed to maintain the desired level of service.

This book is part of the Networking Technology Series from Cisco Press, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.

Team LiB
Next Section


Table of Contents

Team LiB
Book Cover
• | Table of Contents
• | Index
Optimizing Applications on Cisco Networks
By Mike Hicks
Publisher | : Cisco Press
Pub Date | : November 11, 2004
ISBN | : 1-58705-153-2
Pages | : 384

Copyright
About the Author
 | About the Technical Reviewers
Acknowledgments
Icons Used in This Book
 | Command Syntax Conventions
Introduction
 | Approach
 | Who Should Read This Book
 | How This Book Is Organized
Part I: A Comprehensive Approach to Managing Networked Business Applications
 | Chapter 1. The Case for Application Performance Management
 |  | Why Application Performance Is Important
 |  | A Life Cycle Approach to Managing Networked Applications
 |  | The Benefits to the Enterprise
 |  | Summary
 |  | End Note
Part II: Aligning the Network's Business and Technical Requirements
 | Chapter 2. Understanding Your Business
 |  | Defining Your Business-Critical Applications
 |  | Determining Total Cost of Ownership
 |  | Defining Your Requirements
 |  | Prioritizing Your Applications
 |  | Business and Technical Alignment Methodology
 |  | Summary
 | Chapter 3. Detailing the Business Transaction
 |  | Understanding Profiling Terminology
 |  | Classifying Applications
 |  | Profiling Methodology
 |  | Baseline Overview
 |  | The New Application
 |  | The Deployed Application
 |  | Optimizing the Application
 |  | Verifying Application Delivery
 |  | Summary
 |  | End Note
 |  | References Used in This Chapter
 | Chapter 4. Service Level Management
 |  | Defining SLAs
 |  | Assessing Your Requirements
 |  | Profiling Case Study
 |  | Defining What to Monitor
 |  | Summary
 | Chapter 5. QoS and MPLS: Tools to Manage Application Performance
 |  | Background
 |  | Allocating Network Resources
 |  | Traffic-Handling Mechanisms
 |  | Provisioning and Configuration Mechanisms
 |  | MPLS
 |  | Traffic Engineering
 |  | Queuing Mechanisms
 |  | Integrated Services
 |  | Resource Reservation Protocol
 |  | Differentiated Services
 |  | Summary
 |  | End Notes
 |  | References Used in This Chapter
Part III: Deploying and Monitoring Network Applications
 | Chapter 6. Application Deployment
 |  | Baselining Your Infrastructure
 |  | Allocating Quality of Service Resources
 |  | The Pilot
 |  | Verification Process
 |  | Predictive Modeling
 |  | Big Bang vs. Staged Rollout
 |  | Building a Contingency Plan
 |  | Reporting
 |  | Summary
 |  | End Note
 |  | Reference Used in This Chapter
 | Chapter 7. Beyond the Boundaries
 |  | Business Requirements Overview
 |  | Assessing Demarcation Points
 |  | Recognizing Your Limits
 |  | Meeting Service Needs
 |  | Assigning the Correct QoS System
 |  | Summary
 |  | End Note
 |  | References Used in This Chapter
 | Chapter 8. Monitoring the Delivery
 |  | The Pyramid Model
 |  | Classifying Management Systems
 |  | Operating Multiple Systems
 |  | Defining Metrics
 |  | Collecting Metrics
 |  | Reporting Mechanism
 |  | The Delivery Circle
 |  | Summary
 |  | End Note
 |  | References Used in This Chapter
Part IV: Planning and Troubleshooting Network Applications
 | Chapter 9. Proactive Planning
 |  | Baselining Performance Levels
 |  | The Dynamic Enterprise
 |  | Diagnosing Problems
 |  | Validating Changes
 |  | Enabling Technologies
 |  | Scenario Modeling
 |  | Capacity Planning
 |  | Reviewing SLAs
 |  | End-to-End Delivery
 |  | Summary
 |  | Reference Used in This Chapter
 | Chapter 10. When Applications Fail
 |  | QoS Troubleshooting Tools
 |  | Diagnostic Tools
 |  | Troubleshooting Methodology
 |  | Identifying Responsibility
 |  | Resolving Performance Problems
 |  | Redefining Delivery Requirements
 |  | Redefining Service Level Criteria
 |  | Summary
 |  | End Note
 |  | References Used in This Chapter
Part V: Practical Implementations
 | Chapter 11. Business Aligning Case Studies
 |  | Case Study 1: Profiling the Business Transaction
 |  | Case Study 2: Defining SLAs
 |  | Case Study 3: Optimizing the Business Transaction
 |  | Case Study 4: Implementing QoS in a VoIP Environment
 |  | Summary
 |  | Reference Used in This Chapter
 | Chapter 12. Optimizing Application Delivery in Storage-Based Networking, Wireless LANs, and an End-to-End Model
 |  | Storage-Area Networks
 |  | Wireless LANs
 |  | Complete End-to-End Model
 |  | Summary
 |  | End Notes
 |  | References Used in This Chapter
Part VI: Appendixes
 | Appendix A. QoS-Related Requests For Comments
 | Appendix B. Guide to QoS Standards
 |  | IETF QoS Standards
 |  | ITU QoS Standards
 |  | ETSI QoS Standards
 | Glossary
Index

Team LiB


Copyright

Optimizing Applications on Cisco Networks

Mike Hicks

Copyright © 2005 Cisco Systems, Inc.

Published by:
Cisco Press
800 East 96th Street
Indianapolis, IN 46240 USA

All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the publisher, except for the inclusion of brief quotations in a review.

Printed in the United States of America 1 2 3 4 5 6 7 8 9 0

First Printing November 2004

Library of Congress Cataloging-in-Publication Number: 2003107989

Warning and Disclaimer

This book is designed to provide information about optimizing applications on Cisco networks. Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied.

The information is provided on an "as is" basis. The author, Cisco Press, and Cisco Systems, Inc., shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book or from the use of the discs or programs that may accompany it.

The opinions expressed in this book belong to the author and are not necessarily those of Cisco Systems, Inc.

Trademark Acknowledgments

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Cisco Press or Cisco Systems, Inc., cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Feedback Information

At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with care and precision, undergoing rigorous development that involves the unique expertise of members from the professional technical community.

Readers' feedback is a natural continuation of this process. If you have any comments regarding how we could improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us through e-mail at feedback@ciscopress.com. Please make sure to include the book title and ISBN in your message.

We greatly appreciate your assistance.

Corporate and Government Sales

Cisco Press offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales.

For more information, please contact: U.S. Corporate and Government Sales 1-800-382-3419, corpsales@pearsontechgroup.com.

For sales outside the U.S. please contact: International Sales international@pearsoned.com

Publisher | John Wait
Editor-in-Chief | John Kane
Executive Editor | Jim Schachterle
Cisco Representative | Anthony Wolfenden
Cisco Press Program Manager | Nannette M. Noble
Production Manager | Patrick Kanouse
Development Editor | Julie Rentz
Project Editor | Sheila Schroeder
Copy Editor | Keith Cline
Technical Editors | Henry Benjamin, Venkat Kankipati, Joshua Saul, Peter Welcher
Team Coordinator | Tammi Barnett
Composition | Interactive Composition Corporation
Indexer | Larry Sweazy
Cover and Book Designer | Louisa Adair
Proofreader | Kathy Bidwell

Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100

European Headquarters
Cisco Systems International BV
Haarlerbergpark
Haarlerbergweg 13-19
1101 CH Amsterdam
The Netherlands
www-europe.cisco.com
Tel: 31 0 20 357 1000
Fax: 31 0 20 357 1100

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
www.cisco.com
Tel: 408 526-7660
Fax: 408 527-0883

Asia Pacific Headquarters
Cisco Systems, Inc.
Capital Tower
168 Robinson Road
#22-01 to #29-01
Singapore 068912
www.cisco.com
Tel: +65 6317 7777
Fax: +65 6317 7799

Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on the Cisco.com Web site at www.cisco.com/go/offices.

Argentina • Australia • Austria • Belgium • Brazil • Bulgaria • Canada • Chile • China PRC • Colombia • Costa Rica • Croatia • Czech Republic • Denmark • Dubai, UAE • Finland • France • Germany • Greece • Hong Kong SAR • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia • Mexico • The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Scotland • Singapore • Slovakia • Slovenia • South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela • Vietnam • Zimbabwe

Copyright © 2003 Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, the Cisco Systems Verified logo, Cisco Unity, Follow Me Browsing, FormShare, iQ Net Readiness Scorecard, Networking Academy, and ScriptShare are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, The Fastest Way to Increase Your Internet Quotient, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, LightStream, MGX, MICA, the Networkers logo, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, SlideCast, SMARTnet, StrataView Plus, Stratm, SwitchProbe, TeleRouter, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.

All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0303R)

Printed in the USA

Dedications

To Shirley and Charlotte, thank you for your love, support, and laughter.


About the Author

Mike Hicks is currently an independent consultant and has been involved in the world of application performance management (APM) for many years. He has authored numerous white papers on the subject, in addition to the following two books:

  • Managing Distributed Applications: Troubleshooting in a Heterogeneous Environment (Prentice Hall, 2001)

  • Application Performance Management: An Introduction (Compuware, 2001)

Trained as a network engineer at the Dover Harbour Board in England, Mike worked as network manager at the Canterbury and Thanet Health Authority, and then moved to Siemens, where he was seconded to Ford Motor Company as a network specialist, maintaining the company's European network as well as connections to its U.S. headquarters. Later he became network manager for the European Space Agency (ESA), maintaining and optimizing its operations network based in Darmstadt, Germany, as well as designing and supporting temporary networks to support ESA space missions and NASA missions with ESA payloads.

Since moving to Australia, he has worked as a network consultant for dozens of major corporate clients, including Cisco Systems, Singapore Airlines, IBM, BP, ANZ Bank, Telstra, BT, and Telecom NZ. In addition, Mike works closely with many infrastructure vendors in the area of application profiling and management.

Mike was able to combine his network engineering background knowledge with his considerable practical experience in the APM world to produce a mix of technical and business requirements that make it possible to define the optimum delivery of the application.


About the Technical Reviewers

Henry Benjamin, CCIE No. 4695, holds three CCIE certifications (Routing and Switching, ISP Dial, and Communications and Services). He has more than 10 years of experience with Cisco networks and recently worked for Cisco in the internal IT department helping to design and implement networks throughout Australia and Asia. Henry is senior technical consultant with a large firm based in Australia.

Venkat Kankipati has been with Cisco Systems since 1997. Currently, he is a software development manager, and his experience spans a wide range of internetworking protocols and technologies, including security, network performance, and manageability. Prior to joining Cisco, Venkat worked for Bay Networks, Inc. He holds a masters degree in computer science from the University of Massachusetts, a masters degree in computer applications from the University of Madras in India, and a bachelors degree in science from the University of Bombay in India.

Joshua Saul, CCIE No. 9905, CISSP, is a senior engineer for a large multinational broadcasting company. He has previously consulted for GE Capital, Pfizer, Morgan Stanley, HSBC, Barnes & Noble.com, and several large financial firms. He is responsible for Cisco solutions incorporating fully redundant load-balanced application delivery and intrusion detection systems. He has a BBA in management from George Washington University and is currently pursuing an MBA/MS at Fordham University with a concentration in information and communication systems and finance. He lives with his wife, Turandot Saul, M.D., in New York City.

Dr. Peter J. Welcher, CCIE No. 1773, CCIP, Cisco Systems instructor No. 94014, has a Ph.D. in math from MIT. He started out teaching math at the U.S. Naval Academy. While there he also bought and maintained UNIX systems, wrote a book on a programming language, and wrote a major computer program in C. He changed careers in 1993 to teach a wide variety of the Cisco courses for Mentor Technologies, formerly Chesapeake Computer Consultants, while also doing network consulting whenever possible. Pete is now doing high-level network consulting with Chesapeake Netcraftsmen, with tasks including network design, security, quality of service (QoS), wireless, and IP telephony for several major enterprise customers. He has reviewed a large number of books for Cisco Press and other publishers and has authored or managed development of several courses for Cisco and others. Pete writes articles for Enterprise Networking Magazine. He can also sometimes be found presenting his own seminars at East Coast Cisco offices on topics ranging from campus network design to WLAN security. You can find his articles and seminars at http://www.netcraftsmen.net/welcher.


Acknowledgments

Application performance management holds a special place in my mind. Throughout my professional life, I have had the pleasure of working with and alongside many great engineers who have all played a part in the formation of my ideas and methodologies. Thank you to you all.

I want to give special recognition to Moore Literacy Agency, and in particular to Mike Meehan, Claudette Moore, and Debbie McKenna, who have been incredibly supportive and helpful during this project.

Thanks to Jim Schachterle, his professional and good-humored approach to this project has made it a pleasure to work with him.

Particular thanks to Julie Rentz, who was critical to my ramblings becoming a coherent set of text and artwork and a subsequently completed manuscript. Thanks also to the technical editors, Joshua Saul, Henry Benjamin, Venkat Kankipati, and Peter Welcher, whose feedback was valued.

I also want to thank the following organizations for their assistance and general understanding and appreciation of the APM world: Cisco Systems, Compuware, Mercury Interactive, and OPNET Technologies.

Finally, I want to thank Dave Manning, who has provided a sounding board for ideas, rants, and general abuse during the writing of this book.


Icons Used in This Book

    Team LiB


Command Syntax Conventions

The conventions used to present command syntax in this book are the same conventions used in the IOS Command Reference. The Command Reference describes these conventions as follows:

  • Boldface indicates commands and keywords that are entered literally as shown. In actual configuration examples and output (not general command syntax), boldface indicates commands that are manually input by the user (such as a show command).

  • Italics indicate arguments for which you supply actual values.

  • Vertical bars (|) separate alternative, mutually exclusive elements.

  • Square brackets [ ] indicate optional elements.

  • Braces { } indicate a required choice.

  • Braces within brackets [{ }] indicate a required choice within an optional element.


Introduction

The astounding growth of computer networking in recent years has been characterized by the struggle of network managers to keep pace with the demand for network resources. As more and more users, running more and more applications, demand increased network bandwidth, network managers continually find themselves adding more expensive capacity in an effort to satisfy the needs of their customers. Even still, networks remain congested, and applications continue to run slowly. New breeds of aggressive applications that use networks compound this struggle.

In general terms, performance of networked applications depends on complex interactions among applications, servers, and networks. IT organizations need a detailed, quantitative understanding of these interactions to efficiently and cost-effectively troubleshoot and deploy applications. Determining whether business applications are functioning properly from the end-user perspective is the single most important challenge faced by IT in delivering acceptable service for business-critical applications.

Application performance management (APM) is both a business concept and an evolving technology. Conceptually, APM is based on the recognition that network users are concerned only with end-to-end services. In today's business world, it is not practical to take the view that one person can control or manage an entire end-to-end scenario. The key to APM is to present, view, and share performance data from the user, through the network, and to the server in a language and view "common" to all parties.

In an e-commerce system example, if a user clicks a link to place an order, and the system does not respond within 10 or 15 seconds, the user is likely to go elsewhere. Exactly where the system performance shortfall is (such as the end user's ISP, a misbehaving router, or insufficient bandwidth) is of no concern to customers. APM begins with organizations defining which of their business processes are critical. For example, an organization might say, "Our web page must be available to customers." They would then define service objectives such as, "The web page must be available and load within 15 seconds, 98 percent of the time on weekdays and 90 percent of the time on weekends." If there are no clear-cut service objectives in the first place, management is meaningless, and performance evaluation is irrelevant.

After an enterprise has decided to look at the network from the outside, in terms of business objectives, the IT department is faced with linking those objectives to the technical realties.

Unfortunately, the traditional network management tools available to IT (whether they are aimed at troubleshooting, configuration, fault management, or even security) tend to be reactive rather than proactive. By the time they provide an alarm, it is too late from the point of view of the disrupted business process.

This book takes a practical look at how APM can assist in the overall management process by proactively identifying hotspots to various groups (such as server, desktop, and network) within an organization. By completing an overall view specific to a business transaction, readers will be in a position to define and meet business requirements relevant to their applications, which will subsequently optimize their overall delivery processes.


Approach

As enterprises grow more dependent on their networks for business, application delivery becomes more important. Unfortunately, traditional network and systems management tools are designed only to provide information about particular devices or particular segments of a network. By intensively collecting and then correlating the information these tools provide, the reader should be able to move up from the network-related information to the application, drawing some conclusions about the type of performance end users will experience. However, it is never easy to identify all parts of the delivery system.

This book takes a practical look at how APM can assist in the end-to-end application delivery model, by proactively identifying hotspots within an organization. Optimizing Applications on Cisco Networks covers network-dependent application delivery, with an emphasis on optimizing the underlying infrastructure, procedures, and methodologies to aid and measure delivery. The book covers the details of the applications, transmission, delivery techniques, and QoS protocols by identifying and profiling business-critical applications and defining and establishing service level criteria and steps to ensure that these system configurations are continually assessed to maintain the desired service.


Who Should Read This Book

This book is aimed at those enterprise IT managers whose responsibility it is to ensure the delivery of critical applications (for example, network administrators, network engineers, network analysts, network managers, application deployment staff, network planners, and so forth). It is not necessary for the IT professional to have wide ranging experience across all aspects of the delivery system (network, server, application). Their background can be equally drawn from either development and/or operations, because information is presented in a language common to both.

A secondary audience includes senior management (for example, CIOs, CFOs, COOs, IS department heads, and so forth) who need to understand the association between business and technical objectives. In both cases, the reader need not have a technical background to be able to use the technical and strategic direction sections in the decision-making process.


How This Book Is Organized

Although this book could be read cover to cover, it is designed to be flexible and allow you to easily move between chapters and sections of chapters to cover just the material that you need more work with.

The book is separated into five logical parts that cover the areas of defining your business requirements, application deployment, and troubleshooting. The whole story is put together with several case studies at the book's conclusion.

Part I, "A Comprehensive Approach to Managing Networked Business Applications," provides a context for the book, explaining how suggested best practices support each other, and where they fit in terms of who, when, where, and why.

  • Chapter 1, "The Case for Application Performance Management"To solve performance problems effectively or to forecast computing and networking resource requirements proactively, you must understand how applications consume system and network resources. Determining whether business applications are functioning properly from the end user's perspective is the single most important challenge faced by IT in delivering acceptable service for business-critical applications. This chapter addresses the need for application performance management (APM), going on to cover how APM can be applied through every aspect of the application life cycle, and the benefits such an implementation can bring.

Part II, "Aligning the Network's Business and Technical Requirements," covers the relationship between business and technical requirements to successfully align the business model.

  • Chapter 2, "Understanding Your Business"To be successful in today's competitive marketplace, organizations need an advantage, the kind they can get using Internet technology to solve critical business challenges. By addressing what defines a business-critical application, this chapter discusses business solutions that span industries and geography.

  • Chapter 3, "Detailing the Business Transaction"This chapter covers the details of how to profile the application identified and defined in the previous chapter. This chapter helps readers characterize those parts of the business transaction that should be handled and monitored, identifying which parts are active and time sensitive (and which parts are not), ensuring that protocols and policies are correctly applied. It covers profiling applications in a lab environment (predeployment) and applications already in operation that need to be optimized to improve the delivery.

  • Chapter 4, "Service Level Management"Service level management (SLM) involves the task of defining specific performance agreements expected from the network. The agreement defines certain performance metrics used to measure the actual service level obtained from the network against the stated level of service in a service level agreement (SLA). It is a common agreement between the provider of a service and the recipient of the service. The metrics for network performance can include response time and availability. This chapter shows that by using a combination of application performance data and the establishment of a baseline, SLM can be used to assist in defining (and subsequent monitoring of) the SLA.

  • Chapter 5, "QoS and MPLS: Tools to Manage Application Performance"This chapter discusses classifying of applications and assigning the correct QoS mechanism to each. It also explains the emergence and importance of label switching and Multiprotocol Label Switching (MPLS) in a network infrastructure. It explains the problems associated with conventional IP routing procedures and introduces the idea and importance of label switching to QoS.

Parts III, "Deploying and Monitoring Network Applications," and IV, "Planning and Troubleshooting Network Applications," look at two perspectives of the application story. Part III addresses application deployment, as well as the basics of application performance monitoring. This part covers aspects related to the deployment of new applications, as well as setting up the overall performance management systems.

  • Chapter 6, "Application Deployment"This chapter covers how to build up an accurate baseline. It then discusses using this baseline to ascertain the impact on the environment, including rolling out an application and projecting upgrades to both application and infrastructure.

  • Chapter 7, "Beyond the Boundaries"This chapter discusses approaches to situations in which you only operate (own) part of your delivery mechanism and must rely on third-party organizations to deliver your application to both your customers and internal staff. The chapter covers ways to work within these parameters.

  • Chapter 8, "Monitoring the Delivery"Having gone to the trouble of optimizing your application, you need to ensure that the delivery criteria you intended and expect is initially achieved and then maintained. This chapter addresses how to monitor the delivery, discussing what metrics should be referenced and how that data should be reported on to achieve your desired result.

Part IV assumes the reader is now working with a deployed application in a production environment and covers aspects of ongoing planning in such an environment, as well as the inevitable troubleshooting side of application performance.

  • Chapter 9, "Proactive Planning"This chapter covers proactive planning to ensure that you maintain the delivery criteria your business-critical applications require.

  • Chapter 10, "When Applications Fail"To manage faults, you need to discover, isolate, and correct problems. This chapter discusses how the user can employ Cisco best practices that facilitate fault discovery (using, for example, the system monitoring commands) to isolate problems with the system test commands and resolve problems with other commands, including debug commands. This chapter then lays out a troubleshooting methodology in which you can use this information. It discusses the importance of identifying and assigning responsibility for problem resolution, discussing the general default root approach instead of using predetermined metrics to allocate responsibility.

Taking all the lessons thus far, Part V, "Practical Implementations," presents case studies that combine both design and performance issues.

  • Chapter 11, "Business Aligning Case Studies"This chapter is composed of case studies reinforcing the material discussed in Chapters 1 through 10. The first case study focuses more on the business side of the process and shows how to define and profile the business-critical application. It finishes up with a description of the characteristics learned, and discusses how they can be used in the ongoing process and future troubleshooting exercises. The second case study shows how to take the information and relate it to produce a realistic SLA. It covers subjects such as metric definition, collection points, and thresholds to demonstrate good, poor, and bad performance. The third case study addresses the optimization of a single transaction in a banking environment. Finally, the fourth case study addresses the addition of QoS mechanisms to optimize the deliver of a voice-based application.

  • Chapter 12, "Optimizing Application Delivery in Storage-Based Networking, Wireless LANs, and an End-to-End Model"Basic design is considered very important in any optimization methodology. No amount of optimization will compensate for a poor base design. This chapter concentrates on the design issues and considerations with methodology approaches for storage-area networks, wireless LANs, and an end-to-end model.


Part I: A Comprehensive Approach to Managing Networked Business Applications

Chapter 1 The Case for Application Performance Management


Chapter 1. The Case for Application Performance Management

This chapter includes the following topics:

  • Why application performance is important

  • A life cycle approach to managing networked applications

  • The benefits to the enterprise

Network-based applications provide unprecedented opportunities for organizations to reach more customers with ever more useful services. These services, and the networking and computing systems they run on, are critical for market success. The applications boost productivity dramatically and increase the flexibility and responsiveness of the organizations that use them.

These critical pieces are not just routers, servers, and word processors. They are banking systems, worldwide airline reservation systems, e-retailing, multimedia-based applications, and the likemission-critical applications that are made up of numerous components and deployed across a distributed enterprise or many geographical regions.

These applications, fundamentally different from their predecessors, have additional dependencies on more systems spread over an ever-wider geographical area, occasionally crossing organization boundaries. Their functions are spread over the entire network, and they exploit many different technologies. Although these new technologies offer far superior services to a larger user base than ever before, at the same time these systems must be more reliable, scalable, and available. Nonavailability of applications has far greater repercussions on a business today.

All these components from PCs (clients), to networks, to applications, to operating systems (OS), to servers, to applications have to work with each other in a coordinated manner.

Statistics about specific routers, servers, or utilities that support these business systems are certainly helpful, but they do not tell you whether a business application is measuring up to its potential. To make the latter happen, you need to know whether business transactions are being completed successfully, if transaction response times are satisfactory, who is using an application, and so on. In other words, the ability to achieve excellent management and integration of critical resources is a determinant of the success or failure of an enterprise's business model and its initiatives.

Determining whether business applications are functioning properly from the end user's perspective is the single most important challenge faced by IT in delivering acceptable service for business-critical applications.

The requirement to understand application performance has led to a methodology and process widely known as application performance management (APM). Despite all the technically sophisticated ways in which network and system resources can be measured, end users really only perceive two things about an application's performance:

  • Availability Percentage of time that the application is available to give a user service

  • Responsiveness Speed or performance of the service

By measuring and delivering against these requirements, and by tuning the combination of application architecture and underlying infrastructure, you can assist in the improvement of the business process.

This chapter discusses the theory behind the end-to-end story, how APM enhances the full spectrum of the IT process, from operational tasks to planning initiatives. It includes the reasoning behind why there is a need to view the delivery model across client, network, and server, and what is actually meant by the term APM in the context of the enterprise.

This chapter addresses how APM can be applied through every aspect of the application life cycle, enabling users in network operations, engineering, planning, and application development to be far more effective in optimizing performance and availability of their networks and applications.

In addition to roles and responsibilities of IT organizations, the chapter discusses the approaches relevant to a time-horizon perspective. These processes range from short-term activities (such as daily recurring operations or emergency response) to long-range strategy.

It then elaborates on the story behind APM, by expanding on the questions that can be addressed with an APM strategy. Finally, it goes on to discuss what benefits the enterprise can expect to achieve by the implementation of an APM strategy.


Why Application Performance Is Important

APM is the practice of optimizing network service response time. It also entails managing the consistency and quality of individual and overall network services.

To help you assess why application performance is important, this section describes the following topics:

  • Managing the entire process

  • Isolating performance problems

  • Sharing information in a common format

  • Establishing and monitoring service level agreements

Managing an End-to-End System

APM is often talked about in terms of an end-to-end system. End-to-end is the entire process, from one discrete end (for example, a client station) through the network and transmission infrastructure to the application server. It is the entire path required for the application conversation to exist. The following exemplifies the end-to-end process:

  1. Initiation on the client station (such as when a user presses an enter key)

  2. Travel through the various network components

  3. Arrival onto the application final destination (such as a server)

  4. Follow-up interaction as required (such as a response from the server to the client)

Business applications that support mission-critical activities (e-commerce, enterprise resource planning [ERP], supply-chain management, human resources, manufacturing, logistic systems, and so forth) not only consume huge amounts of bandwidth, they are often very sensitive to network delays.

Because these applications typically affect the operation and success of the entire business, it is essential that IT organizations be able to measure and manage end-to-end performance to keep their network environment running optimally.

User productivity and perception are key measures of the success of business computer applications. For better or for worse, transaction response times and application availability are the key indicators of user productivity and satisfaction.

Unlike traditional system and network management, performance management focuses on that gray area between network up and network down.

Most organizations do know when their network is up or down, but they don't really know what is happening on it in terms of application performance from an end-user perspective and whether a significant new application will perform acceptably once deployed over the network.

Isolating Performance Problems

Increased investment is not always required to significantly improve performance. Simply throwing money at a problem is not always the answerfor example, upgrading the CPUs on a server to gain better performance will only have an effect if the CPU is the bottleneck. Equally wasteful is the approach of simply installing a faster network interface, such as from 100 MB to 1 GB, without understanding the application characteristics. In many cases, if you just invest blindly you might well make the situation worse.

The confounding thing about performance management is its holistic nature. If you cannot easily determine where the delay is (server, client, or network), you cannot expect to find the root cause. Without the ability to isolate performance problems within the end-to-end path, troubleshooting and optimization efforts are just stabs in the dark, and might become quite costly. Throwing network bandwidth at a response-time problem, for instance, might not necessarily improve performance. Although the overall application architecture can influence how well the application performs on a network and how efficiently the application uses network resources, the actual design and implementation details are the key variables. Some variables are within your control, and some are not. These variables include the following:

  • Amount of traffic used in application conversations across the network

  • Degree to which clients and servers are geographically separated

  • Amount of network capacity available to the application

  • Amount of delay on network paths

  • Ability to prioritize mission-critical network traffic

  • Overall availability of key network paths

  • Efficiency of clients' and servers' network transport settings

In any case, application performance must be examined together with the network performance to control the quality of performance delivered to the end user.

Performance management is a perpetual task. It requires continued monitoring of key indicators to ensure that service levels are maintained.

To achieve network application optimization, the application and the constituent network should be evaluated in detail (preferably prior to rolling out the application), and wherever possible implemented as a methodology in the early stages of application development.

Sharing Information in a Common Format

In typical corporations, individual IT departments are highly focused on network infrastructure, systems, databases, or applications. This level of focus, although necessary, leads to finger pointing when performance issues arise.

Successful performance management initiatives require a cooperative and generalized focus on the entire picture. You need to understand the interaction and dependencies relevant to the successful delivery of the application from an end-to-end perspective.

The best way to ensure that a more cooperative approach exists is to ensure that information pertinent to the application performance be provided in a common format, allowing all parties to understand the respective parts that make up the delivery model.

In taking the common format approach, you are also never biasing the results to any one particular group rather showing it as a complete delivery system, allowing each separate area of responsibility to understand the impact their section has while still appreciating the importance and functional requirements of another area.

Establishing and Monitoring Service Level Agreements

Performance management is important for establishing and monitoring service level agreements (SLAs). Service level management (SLM) is a compelling concept for business, because if done right, it can supply proof points demonstrating that applications are meeting the business requirements. Not only is it important for an internal department to provide quantifiable feedback, it can be equally important for an enterprise that outsources their IT and has a need to ensure that their contractual requirements are being met.

The SLA can be taken as a baseline of how an application is delivered. Or if the dependencies of an environment do not match the SLA, it can help in identifying what is possible given the existing system, effectively saying this is the best service you will be able to achieve. This service definition could then be expanded to a cost-justification process. The SLA demands a specific performance that can be achieved only through extensive upgrades to the delivery system. With this information, the business can make accurate cost-related decisions to upgrade or not.


A Life Cycle Approach to Managing Networked Applications

Business and IT environments are in a constant state of flux. Businesses are constantly reinventing themselves in response to new market opportunities. Acquisitions and deregulation often force change upon organizations. IT departments must always be prepared to respond to these commercial changes while simultaneously getting up to speed on new technologies that can enhance business practices.

From business practices to applications architectures, a business that believes it has the ultimate answer tends to set things in concrete. For a changing business, strategic applications are always in a state of evolution. Iterative application development and deployment is a commonly held way of building and refining applications. IT architectural initiatives need to be in sync with business goals.

Performance management of applications means different things to different people, but it is this constant striving for application-delivery utopia that drives the application life cycle process.

Application life cycle is far more than simply writing code, testing, and then deploying; it takes into consideration the constant quest for application delivery that enables users in network operations, engineering, planning, and application development to be far more effective in optimizing performance and availability of their networks and applications.

Enterprise APM Needs

Think of the IT department as having two primary functions. First, IT is responsible for technology-driven initiatives to create new revenue opportunities, automate processes, and save money. This is IT's interesting side; the other is much more sedate. Second, IT is also responsible for keeping the trains running on timefor example, making sure applications are available and delivering acceptable performance to business users.

This book is about availability and performance management, part of the nonsexy side of IT but a key to IT operations excellence. Without these disciplines, the whiz-bang side of IT is worthless.

To solve performance problems effectively or to forecast computing and networking resource requirements proactively, you must understand how applications consume system and network resources. Due to the magnitude of this problem, even on a single system, network managers make attempts to divide the system resource-consumption aggregate into smaller, more manageable units often called workloads, applications, or subsystems.

To get the maximum value out of the characterization of application workloads, these workloads should map onto the application that the end user sees (for example, order entry, inventory control, receivables, customer information) and the task the application performs to support end-user business needs. These business transactions are the automatic processes of end-users' work, with the following characteristics:

  • They are discrete. (Their beginning and end are bounded.)

  • They can be mapped onto a business function, which in turn can be used as a forecasting element when applied to an organizational business situation.

The first characteristic is important because it determines where to start and end the monitoring of resource-consumption data surrounding a transaction. The second characteristic allows management processes such as SLM, capacity planning, and accounting and cost-recovery mechanisms to be applied based on business needs rather than on some arbitrary technology-related measure. This second characteristic thus enables the end user to relate resource consumption and the required capacity to key business factors.

Establishing an APM Strategy

Having explored the real-world need to understand, track, and monitor application performance as an indicator to overall business success, you should start to establish a strategy that can be put in place to facilitate a meaningful APM operation.

The aim of any APM strategy should have the ultimate goal of optimizing the application delivery so that it adds real benefit to the overall business model. To reiterate an earlier statement, to effectively troubleshoot or optimize application flow, you must be able to pinpoint the most probable cause of the delay. Simplistically, you can divide an application flow into covering three distinct areas, as follows:

  • Client

  • Network

  • Server

By taking this simple divide approach, shown in Figure 1-1, you will be in a position to answer questions relevant to each part of the delivery system, such as the following:

  • At what point is my application slowing up?

  • Is the application taking the best route for its characteristics?

  • Why are the users all complaining at 9 a.m. every Monday?

  • Is my mission-critical application ready for deployment?

  • Are my SLAs realistic for our business requirements?

  • Are we using the best quality of service (QoS) settings?

  • Should we upgrade our infrastructure to improve performance?

Figure 1-1. Division of End-to-End Delivery System

[View full size image]

Following this division, and with the proper tools, methodologies, metrics, and data, you can answer the following:

  • How will a single application transaction perform on the existing network?

  • What impact will the projected application utilization have on the network?

  • What adjustments can be made to the application to improve its network performance?

  • What upgrades need to be made to the network to deliver the desired performance levels?

  • What network and application performance metrics must be maintained to ensure user satisfaction?

By concentrating on the application delivery, you begin to build up an accurate understanding of the overall delivery structure. This understanding of the process helps to close the gap between application developers and support staff. By demonstrating interaction between the applications, infrastructure components, and protocols, you can resolve faults more quickly, but more importantly you can optimize the whole delivery system, including the application.

Areas of Responsibility

In an overall application-delivery story, there are many different areas, each with different roles, functions, and responsibilities for each part of the delivery model (such as help desk, network engineers, network planning, and so on). This section defines the relative roles, which are then used throughout the book.

For the purposes of describing the IT teams, use the generic organizational chart in Figure 1-2.

Figure 1-2. Generic IT Organizational Chart[1]

[View full size image]

Naturally, some IT organizations have a different structure from the one presented in Figure 1-2. However, most IT organizations comprise the following basic functions:

  • First-level help desk This group is the first point of contact for end users to report IT-related trouble tickets. This group can escalate the problem to a specialized support group for difficult issues that they are not able to resolve immediately.

  • Specialized support This group has in-depth support skills to support the analysis and identification of root causes of performance problems as they relate to applications, servers, network infrastructure, and workstations.

  • System administration This group installs, operates, and manages information systems equipment such as distributed servers, data centers, and workstations.

  • System planning This group plans the overall information systems infrastructure, including the planning and provisioning of servers to meet business needs.

  • Network operations This group manages the network operations center, and is responsible for the day-to-day monitoring and management of the network infrastructure.

  • Network engineering This group implements hardware and software changes to network devices, and ensures all network devices are configured properly according to the organization's policies.

  • Network planning This group designs the network infrastructure.

  • Network security This group has the responsibility of preventing intrusion into the network as well as unauthorized access to network and system resources by internal users.

  • Application-development teams This group has responsibility for the architecture, design, development, customization, and testing of applications.

  • Application-deployment teams This group has responsibility for ensuring that applications are deployed successfully. The teams are staffed with skills from across the organization as appropriate for particular application deployments.

Implementation Triggers

Through the various stages of an applications life on your systems, you will face different requirements and have to implement different processes relevant to APM. Compelling situations, such as the completion of function testing of a new application, will generate specific requirements; these events are known as triggers. These triggers can be allocated to the following categories:

  • Application performance troubleshooting/analysis

  • Application assessments (networkability/application impact)

  • Predictive performance analysis

These triggers will include both reactive and proactive scenarios; and although these issues can be triggered at any time, you will find different scenarios and issues will arise at specific timelines within an application's life cycle. These processes range from short-term activities (such as daily recurring operations or emergency response) to long-range strategy (such as capacity planning). Figure 1-3 represents these processes relevant to a time-horizon perspective.

Figure 1-3. APM Processes Relevant to a Time-Horizon Perspective

[View full size image]

Note that these processes continue across the entire continuum of an application life cycle. Although represented as a linear time scale, an application life cycle should be considered to be a constant process.

Application Performance Troubleshooting/Analysis

The first trigger is that of application performance troubleshooting and analysis. This will occur as soon as an application performance problem is reported or in situations where it is deemed the application performance is detrimental to the overall business performance.

This troubleshooting trigger can be deemed to be very reactive; by the very fact that it appears at the beginning of the time scale, you can assume that you have little or no time to plan for its occurrence. This is most likely where you will find yourself initiating your APM strategy. Chapter 10, "When Applications Fail," discusses this in more detail; in short, the benefits that can be derived by applying an APM strategy that covers the end-to-end delivery system to cope with the reactive troubleshooting trigger include the ability to quickly determine root cause of poor performance (server, network, or client). This will ultimately assist in expediting speed to resolution for performance issues.

Application Assessments

Having moved away from the immediate issues, you can begin to move to the next trigger and the first of the proactive APM statements. These include the areas of application assessments, including impact and networkability studies.

When analysts talk about application networkability, what they are referring to is evaluating whether an application will perform as required when deployed on the network. This can apply equally to new application deployments (both in-house and off-the-shelf installations) as well as enhancements to existing applications, including version upgrades. APM implemented correctly at this trigger will benefit you by minimizing performance surprises when the application is deployed. It will also assist in conveying to the application developer's critical changes that need to be made prior to deployment, which will subsequently avoid potential application rework in a post-deployment mode (or late-stage predeployment).

Understanding how an application is going to perform in isolation is obviously only half the story. The other half is that of coexistence, understanding how the implementation (or upgrade) of an application will impact the existing applications. As previously mentioned, this trigger occurs at the same time as the networkability assessment and so is influenced by the same conditions. However, the outcome and benefits differ slightly, but are nonetheless important.

Understanding the application characteristics in accordance with transport system's underlying infrastructure and other applications will assist in the following:

  • Enabling you to implement cost-effective infrastructure upgrades and therefore prevent overengineering

  • Determining whether the cost of supporting the application (for example, network and server resources) outweighs the benefit of the application to the business

  • Ensuring that the network performance and existing applications are not adversely impacted

For these areas to become an effective part of an overall APM strategy, they must be undertaken proactively. Chapter 9, "Proactive Planning," discusses this in more detail; for now, it is important to understand the importance and timing of the trigger.

Predictive Performance Analysis

The final long-term trigger centers on the area of capacity planning. All enterprises should consider regular planning to accommodate business growth, but it can also be triggered earlier in the life cycle through scenarios such as planning for network expansion due to merging organizations.

This APM strategy should cover all aspects of the end-to-end delivery model, and should incorporate contingency-type planning for failures. If implemented correctly, this strategy will bring the following benefits to an organization:

  • Allowing for the provision of sufficient capacity with enough lead time

  • Avoiding capacity-shortage crisis situations

  • Preventing expensive overprovisioning

Part IV, "Planning and Troubleshooting Network Applications," covers this subject in more detail.

Underpinning all these aspects should be the constant revision, monitoring, and reporting of the delivery system, represented in Figure 1-3 by the Unified data management bar. This constant review of the delivery model is one of the most important aspects of the APM story and is one of the main themes throughout this book.


The Benefits to the Enterprise

APM is important to the overall delivery and optimization of the application-delivery model. By implementing an APM strategy, an enterprise can expect to achieve certain benefits.

The first benefit is that through the optimization of the business-critical applications you can have a direct effect on the business bottom line. Although a significant benefit, results will become apparent only over time. In addition, note that it would require constant optimization to ensure the business continues to reap the full benefits.

A second, more tangible benefit is a simple case of making the most of what you already have. Having gone through the process of building up a significant network infrastructure, you will be constantly challenged to show return on investment. It would be unwise to continually have to return to the budget table asking for more money to upgrade each time an improvement in performance is required. So rather than going through a hefty system upgrade process, it may be that you can gain significant increase in the overall performance simply by optimizing the systems you already have to perfectly match the characteristics and requirements of the business-critical applications.

There are also associated benefits inherent in the creation of an APM strategy and model. A full understanding of the delivery model will provide valuable baseline data that can subsequently be used in troubleshooting procedures. By providing visibility and an understanding of the level of applications on the network, including major flows, host servers will ultimately expedite speed to resolution for future faults.

These direct and associated benefits are discussed throughout this book.


Summary

Enterprises are dependent on their IT infrastructures to support their core business activities. APM generates significant business benefits across the IT organization, by providing information and acceleration to a broad range of IT staff.

Performance management has to be viewed from an application perspective, and applications must be tracked end to end through the entire infrastructure, including client devices, routing protocols, network configurations, servers, and associated components.

APM is the step-by-step process of iteratively investigating, determining, and implementing optimizing solutions using a proven methodology and known triggers. While optimizing an end-to-end system, it is important to know when to tune, what to tune, and how much optimization it requires, and when optimization efforts should stop and planning begin. Specific goals need to be set relevant to each department and stage in the application life cycle, enabling information to be shared to enhance the overall application delivery.

Through the various stages of an application's life on your systems, you will face different requirements and have to implement different processes relevant to APM. These triggers can be allocated to application performance analysis, application assessments (networkability/application impact), and predictive performance analysis.

By defining and implementing an effective APM strategy, you can provide your organization with specific benefits, including improved business performance, lower IT infrastructure costs (including recurring costs), reduced downtime, and faster problem resolution.


End Note

  1. Deploying Intelligent Network Management in the Enterprise © 2002 OPNET Technologies, Inc.


Part II: Aligning the Network's Business and Technical Requirements

Chapter 2 Understanding Your Business

Chapter 3 Detailing the Business Transaction

Chapter 4 Service Level Management

Chapter 5 QoS and MPLS: Tools to Manage Application Performance


Chapter 2. Understanding Your Business

This chapter includes the following topics:

  • Defining your business-critical applications

  • Determining total cost of ownership

  • Defining your requirements

  • Prioritizing your applications

  • Methodology

To be successful in today's competitive marketplace, organizations need an advantage, the kind they can get using Internet technology to solve critical business challenges. The modern IT professionals need to be as focused on business goals as they are on the technology. Customer-facing technology solutions have moved IT up the business value chain. The Internet has brought more and more applications into the business-critical space. Emphasis on customer satisfaction has made user availability of business-critical applications more important than system, network, and server availability.

The starting point for defining application requirements is understanding the customer experience. Higher availability of user services is the single most pressing challenge that IT departments face today.

In today's enterprise, applications are caught in the middle between supporting the business and relying on the infrastructure. What you need to do is to take an application view of the world. Taking that one step further, you need to view those applications whose performance is critical to your business models.

Applications considered business critical today include the following:

  • Financials

  • Transactions processing

  • Enterprise resource planning (ERP)

  • Customer relationship management (CRM)

  • Logistics

All currently are, or will need to be, made available to users, customers, and suppliers, including internal audiences, on a 24-hour basis.

The adaptation challenge to meet customer needs concerns all levels of the business, not just IT. Moreover, companies looking to fulfill their IT requirement need an end-to-end solution, one that understands how to manage the wider implications of technology in business.

Therefore, the IT companies that enable businesses to implement IT solutions need to provide more than just technical help and expertise. They must facilitate the integration of IT into the business. In short, the modern IT company has become as much of a strategic business adviser to organizations as accountants, law firms, and traditional management consultants.

This chapter discusses what constitutes a business-critical application, including the impact and criticality to the business model. It begins by discussing the components of the application that are important to the overall business objective.

Then the chapter covers the indirect costs of a network, money spent on system design, installation, administration, and support, along with intangibles such as lost revenue resulting from the failure of mission-critical network functions. After outlining the basis for total cost of ownership (TCO), the chapter goes on to discuss how each new user, application, and peripheral added to or removed from a network affects its TCO.

Having defined business-critical applications from a functional perspective, the chapter discusses aligning the technical requirements covering end-to-end requirements, including which infrastructure is best suited to transporting in the short and long term. It then discusses how to take the business and technical requirements learned in the previous section and begins to plan for application priorities to ensure that as a whole the business objectives are fulfilled.

Finally, the chapter presents guides for creating a customized methodology to assist with aligning the technical parameters with the business requirements, as well as a template to use as a starting point for the definition of a business-critical application.


Defining Your Business-Critical Applications

Traditionally, a business-critical application has been defined as an application that is essential in the business process. For example, data-centric applications such as a bank's transaction processing system or an airline's reservation system have always been considered core business-critical systems. Their impact is conventionally measured and described either in terms of a loss of revenue or employee productivity.

More and more applications are now entering the business-critical space. This reality has in many enterprises already altered the role and function of IT. In essence, a business-critical application can still be defined as those in which system failure of a user service leads to lost revenue, customer dissatisfaction, and/or lost productivity. However, the demarcation lines of impact to the business are becoming blurred. Different services, even within the same corporation, require different criteria to define what is business critical.

The definition very much depends on the consequences of service interruption for the customer and for the business. Poor performance can equal loss of productivity for internal employees, partners, and even the IT department as they try to troubleshoot problems. It also impacts the company's revenue if customers cannot order, partners cannot get information needed to sell products, and salespeople cannot place orders. Today poor performance can even affect a company's stock price. Consider an individual customer satisfaction example where a user has a bad experience. The user subsequently posts his or her experience on the Internet. As a result, the stock market reacts to what people have read instead of the application's actual performance. In another example, if you have an outage that impacts your business, investors can lose faith in the company and punish the stock. Poor performance can have far-reaching consequences.

Each case will essentially be unique, but a good starting point is to initially list the business' key area of focus. For example, a financial institute is primarily concerned with applications that impact its business revenue, whereas a government department is more concerned with customer satisfaction. Table 2-1 shows some general business areas and associated key focus areas.

Table 2-1. Generalized Key Focus Areas by Business

Mfg.

Finance

Gov.

Retail

Media

Impact on customer satisfaction

X

Impact on business revenue

X

X

X

Impact on productivity

X

In reviewing Table 2-1, it is important to understand that these areas do not necessarily represent the only areas of focus. For example, a financial institution is not only interested in revenue, they are also concerned with customer satisfaction. This table represents the main area of focus and is designed as a starting point. This quick-and-simple approach allows the definition process to start expediently.

As you begin to populate the grid, it might be that your organization has a need to fulfill several areasfor example, the financial institution also derives income from an online share-trading application aimed at home users, so here they would also be concerned with customer satisfaction, in an effort to retain clients. In this instance, you should begin to assign a loading to the activity.

Loading is defined as importance to your business objective. This can be a simple priority scheme, developed through discussion with the relevant business stakeholders, comprised of the following steps:

Step 1. | Each application is assigned a figure based on its importance to the business objectives, such as 1 (very important), 2 (somewhat important), or 3 (not important).
Step 2. | Stakeholders are asked to assign a score relevant to their rating of the application's importance.
Step 3. | For each application in the list, find the mean value.
Step 4. | Place the applications in rank order, and show the results to the stakeholders.
Step 5. | Discuss reasons for items with high standard deviations.
Step 6. | The stakeholders may insert removed items back into the list after discussion.
Step 7. | Repeat the ranking process among the stakeholders until the results stabilize.

The ranking results do not have to be unanimous, but should have a consensus that all can live with. Two passes are often enough, but four are frequently performed for maximum benefit.

This loading enables you to take into account business policies as well as direct revenue streams. For example, it may be a corporate directive to move focus to the home user, in an effort to increase your market share in this area.

Having defined your key area (or areas) of focus, you should start assigning associated applications against this function. Start with the main application families, such as People Soft, Oracle, and so on. From here you can start to drill into the specific functions or transactions within that application, remembering to retain focus on your key area.

When you investigate a specific function, you are focusing on a business transaction. Later when you perform the profiling on the application, you cover which elements make up this function. (Profiling is dealt with in more detail in Chapter 3, "Detailing the Business Transaction.") For example, entering a customer detail into a CRM system can be one specific function.

By defining the specific functions within an application family, you achieve the following two objectives:

  • Isolating the most important aspects of your application-delivery model, enabling you to concentrate on a very specific model when trying to optimize delivery.

  • Beginning to create a unique definition to be used in your own service level agreement (SLA). (This objective is dealt with in more detail in Chapter 4, "Service Level Management.")

A point to take into account when defining your specific functions is timing. Consider the work practices of your users. For example, a bank teller processing withdrawals and deposits in a bank's front office will most likely have her busiest period of the day during the lunch hour, because customers use their own lunch break to visit the bank. So on its face, this relatively simple transaction may not appear to have significant relevance to the key focus area; however, a combination of the timing and frequency ensures its importance to your business process. This specific definition is a critical part of the process. You will achieve far greater optimization when concentrating on a specific transaction than when trying to achieve a cover-all approach by simply hitting the whole application family.


Determining Total Cost of Ownership

Determining the cost of owning and operating your business network requires more than totaling hardware and software expenditures. Your analysis also needs to include the value and expenses associated with installation, configuration, and maintenance. These associated costs greatly affect the total amount of money your company will spend on its network, otherwise known as its total cost of ownership (TCO).

TCO takes into account the indirect costs of a network (money spent on system design, installation, administration, and support) along with intangibles such as lost revenue resulting from the failure (or slow delivery) of mission-critical applications, through to missed opportunities resulting from the diversion of resources from other projects. Figuring indirect costs allows a company to take into account the price tag on lost productivity suffered from system crashes, ineffective repairs, and recurring problems.

Each new user, application, and peripheral added to or removed from a network affects its TCO. The larger the network, the more an organization should expect to spend on system design and installation. Larger networks run more protocols and add complexity, so TCO grows in a nonlinear fashion. Failure to appreciate this is often a problem in fast-growing networks. As your network expands, you are also going to increase the potential cost of lost productivity. This increase is because any delivery defect on a large system will ultimately affect many more users.

There is no definitive answer to calculating TCO impacts, because it will always depend on each specific environment. However, having said that, several areas should be considered, including the following:

  • Original cost of the computer, network infrastructure, and software

  • Hardware and software upgrades

  • Maintenance (including scheduled downtime)

  • Technical support

  • Training

TCO is important. The ultimate goal for a business is to make money. In competitive markets, where net profit matters most, businesses increase their bottom line by either increasing revenues or reducing operating costs by improving production.

Operations managers are responsible for reducing business operating costs. One of these operating expenses is the cost of ownership of the application-delivery system (computer hardware, network infrastructure, and applications) used by the business. Many hidden costs are associated with application delivery and maintenance and often are not recognized until well after the application has been deployed. By then, it is too late, and a poor purchasing decision could result in costing the company hundreds of thousands of dollars per year that was not budgeted for.

Do not limit yourself to reviewing TCO relative to existing deployments, because by merging networks (for example, converging separate voice and video networks into your data network), you could realize a significant reduction in TCO. This approach is likely to realize significant savings in the following:

  • Telecommunications costs (installation and ongoing costs)

  • Support costs (with a single network being cheaper to maintain with one data network support staff, for instance)

  • Future equipment costs

To gain the benefits of TCO concepts and methodologies, enterprises should conduct routine TCO evaluations that provide the following:

  • Management oversight Important as they are, analytical tools are not the sole indicators of the success of technology initiatives, nor do they necessarily validate good management or stewardship of resources. Such analyses, however, can provide consistent quantifiable data for managers to use to understand the impact as well as the cost of technology initiatives.

  • Documentation of direct and indirect costs An organization cannot manage what it cannot measure. Understanding and consolidating direct and indirect costs tend to lead to lower overall spending. Alternatively, it may lead to increased efficiency in meeting goals at the same level of spending.

Direct costs are visible and commonly understood. They are generally incurred by the technology support group (formal or "official" support) or represent capital purchases (hardware, software, and so on). The end-user communities incur generally hidden, indirect costs. Indirect costs are driven by the labor time associated with the following types of activities:

  • Peer support The time spent by users asking questions of other users and the time spent by users responding to questions such as, "How do I change my settings to use a different printer?"

  • Downtime The idle time spent by users because the network, or their computer, is down, or slow.

Although these costs are considered "soft," they do, in fact, cost the organization money. They also measure the value provided by the formal technology support organization.

When total costs are understood, organizations can analyze efforts and begin to measure technology effectiveness. For example, by identifying all computer applications used throughout the organization, IT managers can uncover duplications and inefficiencies.

When new technologies or applications are considered, IS departments should use the TCO to model and anticipate the initial and continuing costs. What-if scenarios can provide insight into the relative impacts of multiple options for implementing and supporting technology initiatives.

Organizations should be realistic about the time frame used in measuring TCO by establishing a time period that reflects the maturity and stability of the technology. For example, when measuring TCO for a CRM system software initiative, spreading the initial costs over a five-year or even a seven-year period is appropriate. For assessing TCO for a technology that changes rapidly (for example, personal digital assistants), a three-year or even two-year time frame might be a better fit. Before investing in any technology initiative, a thorough investigation that includes total costs over the expected lifetime of the technology will enable the organization to make an informed decision on the cost versus the benefits of the proposed implementation.

Organizations should routinely conduct TCO evaluations to understand the true overall spending on technology and to manage and assess technology investments and initiatives in the context of business goals. A comprehensive TCO analysis will also create a level of external-stakeholder confidence in IS department decision making and performance.

There is no magic number for TCO. Although a high TCO will significantly impact the business bottom line, a low TCO is not desirable if it is accompanied by low customer satisfaction or is created by reducing the effectiveness or availability of services to clients (internal or external). Ultimately, you should aim at a TCO that fits within your technical and business requirements.


Defining Your Requirements

Having defined your business-critical applications from a functional perspective, you now must align these with technical requirements, including which infrastructure is best suited to transporting in the short and long term.

This definition will ultimately lead into the actual profiling of the application (which is dealt with in more detail in Chapter 3). At this stage, you must create a blueprint for that profiling exercise. The blueprint includes the following steps:

Step 1. | Define your technical requirements.
Step 2. | Take a transaction in its purest form.
Step 3. | Introduce delay.

When defining your technical requirements, you look to delineate a number of metrics. (In this context, metrics are measurable indicators that alone and together impact application performance.) These will ultimately be used to monitor and report on the application's performance, but for now are used just to set the conceptual requirements.

Your main concern is application response time, so the requirements should basically focus on that area. You are going to be concerned with what is going to detrimentally affect your response time.

Next take a transaction in its purest form running internally on the server. At this point, you are restricted to the internal process and operating system of the server.

Moving to the next stage of the same function deployed on a high-speed same-segment LAN, you start to introduce factors with the framing on the network and any inherent delays on the communication system between the client and server. These may well be negligible; as you start to build your deployed system, however, you start to add various delaying factors.

Ultimately, the application will be restricted by technical parameters surrounding it, such as server configuration, network bandwidth, and so on. The desired response time should be based on these technical restrictions of the application. These should include factors such as the following:

  • Will the application be deployed to LAN- or WAN-based users?

  • What bandwidth are you expecting to be present in any WAN connections?

  • What other applications will also be using the infrastructure (includes network and system)?

  • Is the application sensitive to latency?

  • Is the application sensitive to bandwidth variations?

  • What protocol ports will be required?

Answers to these questions should assist in defining your technical requirements. For example, the interactive voice application of Figure 2-1 will be deployed on a central server that users will access via a 256-kbps Frame Relay link, which they will share with the Internet traffic. It would be unrealistic to believe the system can respond within 2 ms, so you can say your desired goal is to have the application respond within 30 ms. You also know that this application is sensitive to latency because it is a voice application, and voice does not react well to excessive delay. Do not worry too much at this stage if you cannot accurately determine which applications are sensitive to latency and bandwidth; this sensitivity will become apparent during the profiling stage.

Figure 2-1. Coexisting Applications

[View full size image]

Remember at this stage you are only setting desired goals; they do not need to be definitive. They should just be technically what you would like to see. Attempting to achieve these results will help you shape what infrastructure should be deployed. You assess the reality of these goals when you move into the profiling and business alignment of your application. By the end of this process, you should have a conceptual understanding of how well the application is likely to perform.


Prioritizing Your Applications

If you walk into any department at any company and ask which of their applications are critical to their operation, you are likely to be met with the same response: They are all important. In fact, you will often hear that their applications are the most important to the whole business. To a degree, this response may be true, because it takes all the cogs to operate successfully; if you take that end-user expectation and subsequently assign every application a high priority, however, the end result will be that all applications ultimately suffer a poor response. That type of prioritization defeats the objective. Taking the business and technical requirements learned in the previous section, you could begin to plan for application priorities (and, in fact, prioritize each part) to ensure that as a whole the business objectives are fulfilled.

Your task is to ensure that your networked applications use your infrastructure in the most optimum way while ensuring that they are delivered to the end user seamlessly, with a perceived response time that facilitates business. With that in mind, you need to take your applications and prioritize not only against their importance to the business but also relative to the application's specific requirements.

You must again compile a list of business units and match them against your original list of key focus areas. The next stage is to match specific technical requirements.

Different applications have different requirements regarding the handling of their traffic in the network. Applications generate traffic at varying rates and generally require that the network be able to carry traffic at the rate at which they produce it. In addition, applications are more or less tolerant of traffic delays in the network and of variation in traffic delay. Certain applications (or transactions) can tolerate some degree of traffic loss, whereas others cannot. For example, Voice over IP (VoIP) is extremely sensitive to delays, whereas something such as a file transfer can quite happily experience delays without ultimately affecting the overall delivery target. You can express these requirements relatively simply using the following parameters:

  • Bandwidth The volume and rate at which application traffic should be carried.

  • Latency The delay an application can tolerate in delivering a packet of data. Note that a predictable delay can sometimes be accommodated in a delivery model. However, unpredictable delay will ultimately cause significant performance issues.

  • Jitter The variation in latency.

  • Loss The percentage of lost data.

If infinite network resources are available, all application traffic can be carried at the required bandwidth with no latency, jitter, or loss. However, this scenario is seldom the case in the real world, where you may also have the impact of an application being designed with different environments and settings in mind (for example, an application built and tested on a LAN but deployed over a WAN). So your job in optimizing has to be the allocation of resources to meet the business requirements.

Prioritization is a means to an end, in as much as you are looking to meet the service needs of your business-critical applications and users. Your goal is not necessarily to optimize service to each application and user individually. Rather, your goal is to maximize the utility of the network across all applications and users. To this end, your task is to provide mechanisms to control and allocate resources among applications and users. By prioritizing your applications based on their importance to your business model, you can begin to describe a policy mechanism to be used to distribute resources.

You do this by taking the application's characteristics and assigning resources based on each of them. For example, you ensure that an application sensitive to latency waits for a minimal amount of time during its transmission across the network. In the profiling section in Chapter 3, identification of these areas of an application's characteristics is addressed.

Your ultimate aim is to ensure that each application is allocated the correct resource when it needs it. However, this allocation may not always be possible. So what you do is take your business-critical applications (previously defined) and start allocating resources that ensure their optimal delivery. Then you move on to the next level of applications, each time ensuring that the characteristics of each application and subsequent resource allocation do not detrimentally impact the applications that have gone before.

In assigning these priorities, you must consider additional parameters, such as time of day.


Business and Technical Alignment Methodology

You need a methodology that enables you to align technical and business requirements, to make changes that will optimize the flow of an application through the infrastructure in the most cost-effective manner, and to define some meaningful way to judge its performance. The establishment of a clearly defined set of steps will assist you in making the correct decisions.

The process can be as simple as reactively responding to specific business requirements, or as complex as forming a committee that meets on a weekly basis to evaluate the operation from top to bottom. The latter does seem a little extreme, but to be effective the methodology must consider the entire system (business and technical) in all phases. The methodology should include the following steps:

Step 1. | Evaluate the baseline.
Step 2. | Establish performance goals.
Step 3. | Compare goals relative to baseline data.
Step 4. | Reiterate until the goals are realistic.

The major feature lacking in most methodologies is the establishment of attainable goals. Without a reasonable goal, you will never know whether you have met business/customer expectations.

Before launching into any application optimization, you need to meet with a selection of your customers and the business stakeholders and agree on some reasonable performance goals. The most important thing about setting goals is they must be both attainable and quantifiable. It is pointless to say, "We want the application to run faster." Without identifying what to change, effort is needlessly expended on the wrong parts of the delivery system. It is essential to understand what aspect of the application (transaction) needs to run faster. You need to know how long it takes now, and what specific improvement is sought, to have a positive effect on the business model. For example, the application needs to return with the result of a query within two seconds, not five.

This first stage is the establishment of a baseline. The importance of a baseline is discussed in more detail in Chapter 3. For now, it is important that you understand the key focus of the business (as derived in earlier sections of this chapter). This focus gives you the business requirements. For this data to have significant relevance in the optimization process, you must match it to the existing environment to get an understanding of what is happening today.

This technical baseline data should be collected from your production environment, derived from your existing management tools such as Cisco Works. Cisco Works gives you an understanding of the configuration and infrastructure. On top of this, you need to understand the application flows. You can derive data relevant to the application paths and so forth from sources such as the Cisco NetFlow utility or third-party products such as Compuware's Network Vantage products. (NetFlow gives a detailed, time-based view of application usage over the network, enabling network managers to gain detailed understanding of customer/user utilization of network and application resources.)

The next stage is to settle on some performance goals. When establishing these performance goals, you must consider what performance is required to improve your business. For example, at present it takes a lawyer three minutes to enter a single client's billable hours into the online billing system. Arguably, by increasing this performance to one minute the lawyer will spend less time entering data and, in effect, more time billing. This decrease in time is a quantifiable improvement.

After these performance desires have been decided, you should compare these goals with the existing baseline data. Note that this first pass is not intended to be a detailed analysis and optimization planning session. It should just be aimed at coming up with realistic performance goals. These goals should be in the form of a statement that defines the current performance and the desired performance. The output from this session can be in the form of a document that covers the relevant details of the application transaction, its current performance and its desired performance. Table 2-2 shows a sample of such a form.

Table 2-2. Sample Performance Goal Form

Application

Transaction

Response Time

Required Response Time

Business Justification

SAP

Stock Search

90 sec

8 sec

Remove wait time for clients and reps

Oracle

Outstanding leave

34 sec

4 sec

Reduce impact in HR department to review outstanding leave

Oracle

Customer order

10 sec

5 sec

Allow more sales to be processed

It is also important that these desired goals be realistic. It is pointless to set expectations that will never be met. Not all applications have to have a subsecond response time, for example. From an alignment perspective, the important thing is to sit down and try to understand what the business is trying to achieve. From there you can help set reasonable goals.

Therefore, it is important to include the baseline data into this alignment process because it gives a clear indication of the performance currently being achieved within your environment. Through discussion and review, the technical area and business units should be able to settle on a realistic objective for the optimization process.


Summary

No matter how extensive or simple your business plans are, it is critical to think about the performance of the applications you depend on. All the money spent on web servers, network hardware, and bandwidth will not make your business successful if your applications and the infrastructure that supports these applications are not available and performing when you need them.

Poorly performing applications can have far-reaching consequences. For example, poor performance equals loss of productivity for internal employees, partners, and even the IT department as they try to troubleshoot problems. Today, poor performance can even affect a company's stock price.

In this economy, applications are caught in the middle. They support the business and they rely on the infrastructure. Because applications are the glue between the business and infrastructure, your focus should start with them. The business itself may not be aligned with applications or the infrastructure; part of the optimization process should include ensuring this alignment takes place. It is critical to understand which applications and indeed which parts of the application are important to the business model. This understanding must be based on a technical and business-unit perspective, including setting expectations and ensuring that the performance goals are realistic and subsequently met.

TCO is an important aspect of your business model. By understanding TCO, you potentially can reduce your operating costs and more fully understand the use of your infrastructure. You can use this data in an overall optimization model.


Chapter 3. Detailing the Business Transaction

This chapter includes the following topics:

  • Understanding profiling terminology

  • Classifying applications

  • Profiling methodology

  • Baseline overview

  • The new application

  • The deployed application

  • Optimizing the application

  • Verifying application delivery

Before considering application optimization, you have to be able to understand what it is you are dealing with. Consider stocking up for a big family meal. The first thing you have to do is pay a visit to your cupboards to see what you have in stock before you can make a judgment on what it is you need to replace or replenish. You would not consider going out and blindly buying up goods, because it would be wasteful, and in all honesty, inefficient. When you optimize your applications, you must go through the same process. You need to assess what you already have, and from there you can make a decision on what needs to be added (hardware, software quality of service [QoS] mechanism) and what needs to be replenished (such as routing path, application architecture, and so on).

This chapter covers the details of how to profile the application identified and defined in Chapter 2, "Understanding Your Business." This chapter begins by defining the terminology used throughout this book. Then it characterizes those parts of the business transaction that should be handled and monitored, identifying which parts are active and time sensitive (and which parts are not), to ensure that protocols and policies are correctly applied. It covers profiling applications in a lab environment (predeployment) and applications already in operation that need to be optimized to improve the delivery.


Understanding Profiling Terminology

Before getting into the technical aspects of profiling, this section lays out the terminology used throughout this chapter and the rest of this book. It includes both industry and Cisco-specific terminology and in each case defines and explains the reasoning behind the definition of the term; it also cites examples of the term.

Latency

Latency is the time it takes for a packet to cross a network connection (as seen at Layer 3, the network layer), from sender to receiver, and the period of time that a frame is held by a network device before it is forwarded.

Two of the most important parameters of a communications channel are its latency, which should be low, and its bandwidth, which should not limit application flow. Latency is particularly important for a synchronous protocol where each packet must be acknowledged before the next can be transmitted[1]. Figure 3-1 shows a graphical representation of this description of latency.

Figure 3-1. Latency

[View full size image]

In Figure 3-1, the time elapsed between the leading bit of a frame passing point A and the same bit passing point B can be described as latency.

Bandwidth

Bandwidth is the range of frequencies, expressed in kilobits per second (kbps), that can pass over a given data transmission channel within a network. The bandwidth determines the rate at which information can be sent through a channelthe greater the bandwidth, the more information that can be sent in a given amount of time. Bandwidth is usually measured in bits per second. A full page of English text is about 16,000 bits. A 56-kbps modem can easily move 16,000 bits in less than 1 second.

The term is often used erroneously to mean data rate or capacitythe amount of data that is, or can be, sent through a given communications circuit per second[1]. When reviewing your bandwidth for a particular path, always consider it to be restricted by the smallest part of that complete flow. For example, if the ingress and egress parts of the path are connected by a 100-Mbps connection, but the pipe in the middle is a 256-kbps frame circuit, your effective end-to-end bandwidth for that path is restricted to 256 kbps.

Jitter

Jitter is a variance in latency for a particular flow or an interpacket delay variance. A variation in this delay can seriously impact latency-sensitive applications such as voice (for example, digital phone distortion hindering the conversation). Many of these applications have been designed to adjust for a certain amount of latency; when that latency randomly varies during a transmission, however, it is hard for the application to adjust, and hence distortion results.

Application (or TCP) Turns

Application (or TCP) turns are critically important when characterizing the application. Application turns are the number of application-level request/response pairs during the exchange as determined by going through a set of frames.

Consider a trip to the grocery store. Each trip you make, you return with one item, and on your return to your house place it on your shelf, and then return to the store for the next item. This example is fine if you happen to live next door to the store, but becomes an issue when you have to get in your car to travel five miles to the store each time. By collecting more goods on each trip, essentially doing your weekly shopping, you make more efficient use of your time and resources.

The same theory applies to application turns, as shown in Figure 3-2. Each time you have to go back to the server for a response adds to your overall response time. Packaging these requests improves your efficiency and your overall response time. The conversation shown on the left of Figure 3-2 represents a turn ratio of 1:1 (that is, a response for every individual request). With this ratio, it is difficult to fill fat, slow pipes effectively, thus making the application conversation in this case a latency-bounded transaction. The conversation exchange shown on the right of Figure 3-2 represents a turn ratio of 4:1 (that is, multiple responses to a single request). This is a more effective way to utilize bandwidth. However, it will be effective only on large bandwidths and could potentially become bandwidth constrained on small pipes with low latency.

Figure 3-2. Application Turns

[View full size image]

Protocol/Network Analyzer

A protocol/network analyzer is any device that captures and interprets the network traffic between two or more connected computer systems. The traffic can then be decoded so that it is possible to see what processes are occurring. Ethereal is an example of a protocol/network analyzer and can be downloaded from http://www.ethereal.com/.

Because they enable you to examine the flow of traffic, you can use protocol analyzers to find out where problems (such as bottlenecks or the failure of a network device) are on a LAN.

Advanced Protocol Analyzer

Advanced protocol analyzers can also provide statistics on the traffic that can help to identify trends that may in the future lead to further problems with the network.

Application Analyzer

Application analyzers are specific modules with decode functionality within the analyzer software that provide visibility to the details of the network traffic at the upper layers, making specific decodes of applications such as Oracle Financials possible. It is important to be able to view the traffic at this level to fully understand the process involved in the communication procedure, which, in turn, enables you to identify and pinpoint the root cause of performance problems.

Conversation

Conversation refers to an exchange of dialogue between two parties. In networking terms, the term describes all traffic flowing between source and destination end nodes (such as client/server) to carry out the execution of a transaction.

Flow

Flow is a subset of all packets, having the same optimization requirements, passing through a device. A flow may consist of traffic from a single conversation or may aggregate traffic across multiple conversations; however, the traffic must all be in the same direction.

Application Task or Transaction

An application task or transaction is a basic unit of user activity within the context of an application. For example, reading e-mail, making a calendar entry, or obtaining records from a database are all tasks.

Cisco Service Assurance Agent

Cisco Service Assurance Agent (SAA) is embedded software within Cisco IOS devices that performs active monitoring. Active monitoring is the generation and analysis of traffic to measure performance between Cisco IOS devices or between Cisco IOS devices and network application servers. Active monitoring performance measurements include the following:

  • Network delay or latency

  • Packet loss

  • Network delay variation (jitter)

  • Availability

  • One-way latency

  • Website download time

SAA can be used to measure network health, verify service level agreements (SLAs), assist with network troubleshooting, and plan network infrastructure. SAA is supported on almost all Cisco IOS devices and in almost all Cisco IOS router images. Enterprises and service providers routinely deploy SAA for network performance statistics within IP networks that utilize QoS, voice over IP (VoIP), security, virtual private network (VPN), and Multiprotocol Label Switching (MPLS). SAA provides a scalable and cost-effective solution for IP service level monitoring and eliminates the deployment of dedicated active monitoring devices by including the probe capabilities within the Cisco IOS architecture.

Probe

A probe is a discreet device that nonintrusively listens and collects the network traffic for use in the profiling or monitoring process. The probe describes both the physical device and the collection software (see agent), and is considered part of the network management system. Examples of probe technology include NAI Distributed Sniffer, Compuware NetworkVantage, Netscout, and Statscout.

Agent

In the performance optimization model, an agent is the part of the system that performs information preparation and exchange on behalf of an analyzer. The use of agent in this context refers specifically to collection software placed on a client machine to observe and collect its network traffic and send it back to the console for viewing and analysis. Although its functionality is the same as that of the probe, the word agent is used to define the difference between a discrete standalone probe and the embedded software on hosts.

Application Thread

An application thread is an entire sequence of frames on the network that constitutes a single, complete action by an application task. The thread includes the command and response data. Examples of application threads include the following:

  • SELECT, INSERT, UPDATE, and EXECUTE statements in a database environment

  • HTTP GET or PUT requests for web client interfaces

  • OPEN, READ, and WRITE functions in a file server environment


Classifying Applications

The general term used when discussing the optimal delivery of application data is QoS. As discussed in Chapter 1, "The Case for Application Performance Management," a generalized approach to application delivery is not the best approach; there is also the issue of wasting resources by trying to optimize an application where optimization will have minimal effect. A lot of IT managers have gotten into the habit of overtuning application performance, carrying out tuning at a micro level. This not only frustrates the end users, with system downtime and interruptions, it also does not tend to result in any significant return in performance, relative to the amount of effort and time put in.

There is a need to prioritize tuning tasks and components that can be changed (or implemented) based on the effort invested. As discussed in Chapter 2, when assigning goals for your performance expectations, you need to be able to quantify your tuning goals and subsequent performance increase. This quantification is essential to avoid unnecessary optimization.

Chapter 2 introduced the factors that impact application performance: latency, bandwidth, loss, and jitter. You will use two of these key factors (latency and bandwidth) when classifying your applications. By taking this relatively simple approach, you can quickly assess the broad characteristics of the application and make specific improvements to significantly improve the application performance.

Latency-Bounded Applications

As mentioned previously, latency is the time taken for the leading bit in a frame to travel from one network location to another.

Several factors can impact the measure of latency, including the following:

  • Distance between locations

  • Routing path

  • Router/switch transit time

  • Router/switch queue congestion

  • Carrier oversubscription

You are trying to assess the time it takes for the leading bit to traverse a connection, so the physical distance between the locations is certainly going to have an impact.

As well as the physical (hardware) path, the logical (software) path the frame takes through the network also impacts the delay factor. The logical path is determined by the routing protocol. By understanding how this flow is established, you can classify the application as latency sensitive or not.

Networks interconnect hosts using a variety of network devices, including host network adapters, routers, switches, and hubs. Each of these contains network interfaces. The interfaces interconnect the various devices via cables, fibers, or wireless connections. Network devices generally use a combination of traffic inherent to the hardware infrastructure and routing information to send data from one to another. Each interface can send and receive traffic at a finite rate. If the rate at which traffic is directed to an interface exceeds the rate at which the interface can forward the traffic, congestion occurs.

Network devices may handle congestion by queuing traffic in the device's memory until the congestion subsides. In other cases, network equipment may discard traffic to alleviate congestion. As a result, applications experience varying latency (as traffic backs up in queues or interfaces) or loss.

By understanding the factors that can impact latency, you can begin to classify applications as latency boundedapplications that will be impacted by latency. These applications have the following characteristics:

  • Many turns per transaction An application that uses an excessive number of turns will have the effect of latency multiplied for each required turn.

  • Small packet sizes Small packets result in fragmentation of the application data into a large number of frames. The effect of latency is multiplied by the number of frames sent.

  • Protocol window size limit constraining turn ratio As with small packet sizes, a small advertised window results in a large number of frames multiplying the effect of latency.

  • High protocol overhead to transport application data A high protocol overhead reduces the available space for application payload; therefore, there are more frames sent to carry all the application data, multiplying the impact of latency.

  • Rate held lower than available bandwidth Although insufficient bandwidth can impact latency, adding bandwidth will not significantly improve performance if flow is constricted by other latency constraints.

Figure 3-3 shows a subset of a conversational exchange of a latency-bounded application. The application has to transfer 300 kb of data. The nature of this application is such that each frame sent to the server must be acknowledged individually, giving a saw-tooth transfer pattern. Each round-trip time (the time interval between the moment a request is sent and the moment a response is received, RTT) is 35 ms. Out of that 35 ms, 20 ms can be directly attributed to the time the server takes to process the response. In general a 20-ms server process time is relatively low. In this situation, however, the fact that the conversation requires 5 requests to transfer a relatively small amount of data means that the server response time must be increased by a factor of 5. Therefore, even before considering any potential bandwidth constraints, this exchange will be subject to the following delay:

Figure 3-3. A Latency-Bounded Application Conversational Exchange

Bandwidth-Bounded Applications

The second general area of classification is applications that are bandwidth boundedapplications that are affected by bandwidth constraints. When considering an end-to-end path, the bandwidth is always going to be effectively capped at the smallest part of that path. It is irrelevant that your two end connections are made up of high-speed LAN connections if the WAN link connecting them together is a slow 64-kbps connection; 64 kbps will ultimately be the bandwidth for the conversation.

Applications that are bandwidth bounded have the following characteristics:

  • Few turns per transaction

  • Larger windows

  • Less time sensitive

At this stage it may seem a very simplified approach to define an application as either latency or bandwidth bounded. However, this simplified approach enables you to easily classify your transactions, which will facilitate administration, monitoring, and quantification of your optimization process.


Profiling Methodology

The establishment of a set of clearly defined steps will assist you in working through the profiling scenario. This methodology is not intended to be cast in stone. It is intended as a guide to provide you with the key features and a proposed order to follow. When you understand all the features, you can adopt your own techniques and execute the steps in any order that you are comfortable with.

Profiling an application and its proposed environment can assist in many ways and ultimately save both time and money.

When an application is deployed over a network, it may cause additional traffic on the links and congestion on intermediate devices and servers. This impact in turn may affect applications that have been previously deployed. If the application is deployed on a large number of client machines, its own behavior can be affected because of increased use of network resources. By recording this data, you will be in a position to quantify and cost-justify the need for new technology or greater capacity.

Ideally, this profiling process should take place prior to a new deployment; however, this timing is not always possible. This methodology can easily be adjusted to meet the requirements of a deployed application and environment.

The profiling methodology can be divided into two parts. Figure 3-4 displays the steps in the methodology. The first part is the capture and profiling of the application transaction. This is done in an effort to understand and classify the application. It also provides an opportunity (if required) to optimize the application prior to deployment (as discussed in more detail later in this chapter).

Figure 3-4. Profiling Methodology

[View full size image]

The second distinct part of the methodology is the merging of the existing network statistics with this application transaction. This is the section where you assess the various impacts (both network on application and application on network).

Your first stage is to collect performance statistics gathered from the existing network. (In a proposed design, these baseline figures have to be estimated, or taken from another site.) This performance baseline represents the current state of the network in the model and enables you to judge the network impacts when you place your application transaction in that environment.

Define Base Transaction

Before moving to capture, the first step is to define the base transaction. This is done to ensure that no time is wasted modeling unnecessary or irrelevant processes. The definition of the base transaction should be one that is a repeatable, necessary part of the business-critical application, such as an initial system logon.

Determine Application Status

Step two determines whether the application is in a preproduction testing environment or still in the planning stage. If it is in preproduction testing, you should be able to capture the transaction in a lab environment. If it is still in the planning stage, you must simulate it based on its proposed operation characteristicsfor example, a browser front end making specific queries to a database.

Build Baseline Environment

Having determined the applications status, the next step is to create a baseline environment for this application to operate in. It is not essential to replicate your entire environment, but at the very least this environment should include the minimum operational infrastructure, such as servers, a single client, and the proposed infrastructure (including link specifications). It should also, if possible, include performance statistics gathered from the production network; the performance baseline should represent the current state of your production network in your environment.

You can obtain this data from a combination of environments. You can use the base configuration files to create something of a skeleton template on which to overlay the associated traffic loads. OPNET IT Guru is an example of a tool that can take these configuration files and build a representation of your environment. Then, by using traffic statistics taken from your environment, you can begin to add a projected load to the environment.

This data is reflective of a production environment; if your system is being deployed in a brand new environment, this data is unlikely to be available. In that situation you must estimate based on predicted loads. A tool such as OPNET IT Guru can assist in these areas as well. The base infrastructure model can still be carted either from representative configuration files, or just by redesigning using the specific infrastructure models. Then you can estimate user distributions and associated loads, which you can manually configure to represent background traffic.

Characterize Application

You now have the transaction and baseline environment defined. The next step is to characterize the transaction from a technical level, as discussed earlier in this chapter. As part of this step, you define specific attributes of the transaction, such as packets sent, payload size, and number of turns.

This information assists in tracking the transaction within the simulated model and enables you to optimize the virtual environment to ensure the optimum application delivery.

The importation of the transaction to your model can be handled in a number of ways. Taking the characterization, you can match these key attributes on top of a template transaction within the modelfor example, matching the number of application turns and processing times on the relevant servers. It is also possible to take the trace file and import it into your model. Tools such as OPNET IT Guru and Compuware Predictor enable you to import the captured transaction directly into a virtual environment.

Simulate/Model the Effects of Users Traffic

Next, you introduce your captured application transaction traffic. You can model application traffic explicitly or analytically. No hard-and-fast rule determines which method is best. Table 3-1 compares these approaches.

Table 3-1. Comparing Explicitly and Analytically Modeled Application Traffic

Explicitly Modeled

Analytically Modeled

Simulate individual packets of the application and its detailed protocol behavior

Simulate application performance based on known behavior

Very accurate results

Results provide enough information

Slower

Quicker

If the objective of your study is to analyze the impact of the application on another application or on another instance of the same application, you should model a single instance of the impacted application explicitly and the rest analytically. The rule of thumb is to model an application explicitly if you want to study its behavior in detail and obtain detailed application response times.

View the Results

The final part of the methodology outputs reports in two forms. The first form represents the application characteristics. (This form provides important baseline data that you can use during future capacity-planning troubleshooting activities.) The second form identifies constraints and determines infrastructure specifications. Figure 3-5 depicts this process. Note that this can be an iterative processthat is, you can review the application relevant to specific changes.

Figure 3-5. Profiling Process

After you have finished the model, the results will also leave you with a set of baseline details and application traits and characteristics that should form part of your operational handover documentation.


Baseline Overview

This section discusses the value of a baseline and covers a general approach to baselining (including the type of data to be collected). There is no single best method for baselining. The data and the area where it is collected will ultimately be driven by your specific requirements.

The Value of a Baseline

It is extremely important to build up a baseline of your environment before embarking on an optimization process. In addition to needing quantifiable goals, you need a starting point; otherwise you will be making changes blindly. Not only does the baseline give you a starting point to compare and report on tangible improvements, it also plays an ever-present part of your evolving optimization process.

If you know what something looked like before it broke, you may be able to assess what made it break in the first place. The thing to consider here is the inherent power in having baseline information. For example, if an office building were to be destroyed in a fire, you would be able to rebuild it to its former glory. Why? Because you not only have access to the blueprints detailing how each girder is to be placed, you also have graphical representations, in the form of photographs, drawings, and even scale models; so in short, you can fix it because you know what it was like before it was broken.

Consider the baseline to be your blueprint. It will tell you the state of your environment before you make any changes. This data is very useful in assessing your goals. It gives you an indication of what is possible with your current infrastructure as well as clues into potential bottleneck areas and other areas of concern.

A baseline should be a moving target, in as much as you should constantly update the blueprint with production data. The reason for this dynamic baseline is that the enterprise environment is always evolving; new users are introduced, new hardware is added, new versions of operating software are deployed, and so on.

By maintaining an up-to-date baseline, you will be in a position to compare a known working situation. It is the best position to proactively identify problems, improve your reactive troubleshooting environment, and expedite your speed to resolution.

If you only do one thing to optimize your application delivery model, create a baseline for your environment.

Producing a Baseline

Now that you understand the importance of a baseline, you need to assess what type and how much information you need to produce a baseline for your environment. A good rule of thumb is to include everything you would need to rebuild your delivery system for your business-critical applications. The exact amount and depth of your documentation will vary depending on the complexity of your network and the resources you have to create/collect the documentation.

The objective of the baselining process is to determine the current status of network devices and then compare that status to standard performance guidelines.

Because of the large amount of data and the amount of time it takes to analyze the data, you must first limit the scope of a baseline. This enables you to quickly build a template and process for a future baseline. The most logical, and at times the most beneficial, place to start is with the core of the network. This part of the network is usually the smallest and requires the most stability.

The steps required to build your baseline are as follows:

Step 1. | Compile a hardware, software, and configuration inventory.
Step 2. | Verify that the Simple Network Management Protocol (SNMP) Management Information Base (MIB) is supported in the devices.
Step 3. | Poll and record specific SNMP MIB objects from the device.
Step 4. | Collect application data.
Step 5. | Analyze the data to determine thresholds.
Compile a Hardware, Software, and Configuration Inventory

It is extremely important that you compile an inventory of hardware, software, and configuration for several reasons. First, Cisco SNMP MIBs are, in some cases, specific to the Cisco IOS release that you are running. Some MIB objects will have been replaced with new ones or completely eliminated. The hardware inventory is most important after the data is collected because the thresholds you need to set after the initial baseline are often based on the type of CPU, amount of memory, and so on, on the Cisco devices. It is also important to know the current configurations, because you may want to change device configurations after your baseline to tune buffers and so on.

The most efficient way to do this part of the baseline for a Cisco network is with Cisco Works 2000 Resource Manager Essentials (Essentials). If correctly installed in the network, Essentials should have the current inventories of all devices in its database. You just need to look at the inventories to see whether there are any issues. If Essentials is not installed in the network, you can use the UNIX command-line tool snmpwalk from a UNIX workstation to find the Cisco IOS version.

Verify That the SNMP MIB Is Supported in the Devices

Now that you have an inventory of the devices you want to poll for your baseline, you can begin to choose the specific object identifier (OID) you want to poll. It saves a lot of frustration if you verify, ahead of time, that the data you want is actually there.

There are two common ways to poll the MIB OID to make sure it is available and functioning. One way to do this is to use a MIB walker from your network management system (NMS) platform, such as HP OpenView Network Node Manager (NNM) or Cisco Works Windows, and enter the OID you want to check. Another easy way to poll the MIB OID is to use the UNIX command snmpwalk, as shown in Example 3-1.

Example 3-1. Polling the MIB OID
             nsahpov6% cd /opt/OV/bin
             nsahpov6% snmpwalk -v1 -c private 172.16.71.80
               .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1
cisco.ciscoMgmt.ciscoProcessMIB.ciscoProcessMIBObjects.cpmCPU.cpmCPUTotalTable.cpm
  CPUTotalEntry.cpmCPUTotal5min.1 : Gauge32: 0

Poll and Record Specific SNMP MIB Objects from the Device

There are several ways to poll MIB objects and record the output. Off-the-shelf products, shareware products, scripts, and vendor tools are available. All front-end tools use the SNMP get process to obtain the information. The main differences are in the flexibility of the configuration and the way in which the data is recorded in a database.

Now that you know the OID is supported in the router, you need to decide how often to poll it and how to record it. Cisco recommends that the CPU MIB be polled at 5-minute intervals. A lower interval increases the load on the network or device, and because the MIB value is a 5-minute average anyway, it would not be useful to poll it more often than the averaged value. It is also generally recommended that baseline polling have at least a 2-week period so that you can analyze at least two weekly business cycles on the network.

Collect Application Data

Having gained an understanding of the infrastructure data, you now need to understand the application data flow. As in the baseline polling, it is important to obtain data over at least a 2-week period. Because you are dealing with application traffic, it is important to use a tool that collects information that details the source and destination of a conversation, time of day, and amount of data transferred. Usually, you will collect this type of data by using third-party probe-based tools such as Compuware Network Vantage. Restrict your collection to areas critical to your business applications to reduce the amount of data you have to process. You also use these in the monitoring process, which is discussed in detail in Chapter 8, "Monitoring the Delivery."

Analyze Data to Determine Thresholds

Now that you have some data, you can begin to analyze it. This phase of baselining determines the threshold settings you can usethey are an accurate measure of performance or fault, and will not set off too many alarms when you turn on threshold monitoring. One of the easiest ways to analyze the data is to import it into a spreadsheet, such as Microsoft Excel, and plot a scatter chart. This method makes it very easy to see how many times a particular device would have created an exception alert if you were monitoring it for a certain threshold. It is not advisable to turn on thresholds without doing a baseline, because this may create alert storms from devices that have exceeded the threshold you have chosen.


The New Application

Network and computer resources, including both physical assets and leased services, comprise a large portion of IT expenditures. Application impact assessment involves understanding the impact of the application on the network and/or on other applications, followed by forecasts of application performance under alternative scenarios.

When an application is deployed on a large scale, some sections of the network may experience significant traffic loads, whereas other sections may be underutilized. Although business-critical applications must be supported with adequate infrastructure, to maintain cost efficiency you should avoid overengineering. Applications can also stress the buffers on the intermediate hops and can impact the servers. An application may impact other applications by causing additional delays. Impact assessment enables you to evaluate your network and application infrastructure options and make cost-saving choices with confidence. Impact assessment should also include what-if scenarios where the network and application parameters can be altered to improve the performance of the application.

You will see as you move to the deployed application that the profiling methodology is a good approach to impact analysis because it takes you through steps to capture and identify an application. The major issue you will confront when assessing a new application is that you have relatively little baseline data to use in your study.

You can deal with this issue in several ways. You can just stick a finger in the air and take a guess at what the application response time will be and subsequently what you can do to optimize its delivery. Or you can utilize a modeling tool and, using your baseline data, build a virtual representation of your production environment; by so doing, you will gain a clear understanding of what impact your new application will have on the environment.

The best advice from a technical perspective is to use a modeling tool to replicate your production environment. However, modeling is often overlooked when building a project plan, because it is believed that the guessed-at figures will provide enough of an estimate and can be slightly adjusted as the rollout occurs. This approach is fundamentally flawed. The majority of applications assessed in this manner fail to meet their expected performance requirements when deployed.

Virtual-Environment Modeling Methods

Ideally, you should view the application in a virtual environment. With modeling, you can again take two approaches: analytical modeling or discrete-event simulation (DES). Historically, these approaches were considered mutually exclusive techniques, presenting the trade-offs shown in Table 3-2.

Table 3-2. Comparing Virtual-Environment Modeling Approaches

Analytical Modeling

DES

Uses mathematical representations of behavior and performance for network, applications, and protocols. It is inherently fast because of its static natureit evaluates mathematical formulas.

Models a system dynamically as a set of interacting entities. It mimics your applications and network explicitly; it evaluates each change, each packet transfer, each retransmission timeout, and so on.

Compuware Predictor, Application Expert, and OPNET Quick Predict.

OPNET IT Guru.

Short simulation runtimes.

Long simulation runtimes.

Small memory requirements.

Large memory requirements.

Can address only a limited number of network and application questions.

Can evaluate a wide range of network and application characteristics.

Assumption and approximation result in a loss of accuracy.

Accurate results.

Analytical modeling uses mathematical representations of behavior and performance for your network, applications, and protocols. It is inherently fast because of its static nature, meaning, essentially, that it evaluates mathematical formulas. Explicitly modeling involves using the DES model. The DES environment provides highly detailed models that explicitly simulate packets and protocol messages. The models in the discrete-event environment execute the protocol in much the same way as a production environment. Protocol behavior under different network conditions can be studied in detail. Studies such as convergence time, route oscillations, the impact of timers, or application performance can be conducted most effectively using DES (because such studies require packet-by-packet simulation).

Analytical modeling attempts to provide estimates of link (or virtual circuit) utilization and network latency. Simulations are fast, but analytical modeling does not necessarily address several important network problems adequately, including the following:

  • Application response time Executing a single transaction across a network may require many exchanges between client and server. Some of these data transfers may occur concurrently, and some may be "blocking," causing the application to wait until the previous transaction is completed. In addition, protocols impose their effects through mechanisms such as segmentation and windowing. All of these factors alter the number of packets transferred and their timing. In contrast to this modeling requirement, analytical models allow only calculation of latency through the network for an individual packeta so-called ping delay. However, there is no generally valid way to map estimates for a ping delay into estimates for application response time.

  • Protocol effects Many of the special performance-oriented features that protocol designers and network programmers have incorporated are too dynamic to be accounted for by a static mathematical network model.

  • Other types of statistics Routing algorithm convergence time and overhead, packet-loss ratios in Frame Relay networks, delay variation for multimedia services, and so on may be of interest, but are not generated by analytical models.

The DES technique simulates each change in network devices, links, and end systems, as they would actually occur over time. DES models protocol interactions in detail to reproduce the different sources of delay that exist in the real world. This technique produces accurate results and addresses many of the network issues that cannot be addressed analytically. Unfortunately, conducting DES poses several challenges. The simulations can take a long time to run (hours or even days for large networks with high traffic volumes). You can reduce the runtimes of simulations by using smaller network models, but results do not necessarily scale correctly.

This method of simulation generally requires significant time and memory. Every explicit event has to be simulated. An event that takes 10 ms to occur in the real world could in theory be made up of a million different events, each requiring simulation, effectively extending the runtime out from 10 ms to 30 seconds.

Both approaches add value: Analytical modeling of networks provides the advantage of quick results, whereas DES caters to more complex scenarios. You will always find champions for both approaches who offer a convincing argument why their preferred approach is best. Ultimately, however, there is no one single answer. In assessing which method to use, you must consider which adds the most benefit to your business environment. In addition, you need to understand what it is you're trying to achieve and associated timelines.

Before leaving the subject of the virtual environment, remember that the information you build here provides valuable information for your baseline picture.

Assessing New Application Impact Example

Having made a decision on which method of virtual environment to use, you can move to the process of assessing the impact. Referring back to the methodology section earlier in this chapter, the first thing you need to do is capture the transaction. In the case of the new application, your preference would be to capture the application as it is tested. It is not always necessary to capture the application in the exact deployed state (after all, that is where the virtual environment comes in), but it is important to capture the interaction (to understand the characteristics).

If it is not possible to capture this applicationfor example, you are required to perform a sizing exercise prior to the application completionyou should gain an understanding from the application designers of the functional aspects of the application. This understanding may consist of something like the following:

Step 1. | Client logs on using a web interface.
Step 2. | Check the application components. The application is a People Soft front end with an Oracle back end.
Step 3. | Client makes a request for record information.
Step 4. | Client logs off.

Although this is a very simple example, it will enable you to gain an understanding. From this understanding, you can determine the protocols that will be used by the client, and you can understand the type of query information that will be used. With this data, you can use either library files (most simulation tools contain library files that enable you to use standard type of applications such as database interaction, HTTP transfer, FTP, and so on) that match your function description or you can capture production data from another similar system (if one exists).

The accuracy of these models depends on the characteristics of the application interaction and how much information you can obtain.

Understanding the characteristics is the key aspect that has to be taken from profiling a new application, because this aspect will become a significant factor when you move to optimize and assign various QoS techniques. In Chapter 5, "QoS and MPLS: Tools to Manage Application Performance," you will see the different mechanisms that are best suited to individual characteristics. The task of the profiler is to understand those characteristics to ensure that the correct mechanism or method is used.


The Deployed Application

The deployed application presents a slightly different view. In this case, you are less likely to want to use the full-blown simulation technique, because the magnitude of the data makes it too complicated (although it is still a valid method). However, the same principle and methodology to profiling still applies.

In this situation, you are dealing with a known entity, in as much as you know the application and you know the business function and transaction. What you are trying to do in this instance is again understand the application's characteristics.

In an effort to get the cleanest view of the application, move your focus to something of a controlled environment, referred to here as a test environment. Note that at this stage your purpose in capturing data is only to profile an application. Troubleshooting of applications with incriminating profiles is discussed in Chapter 10, "When Applications Fail."

The test environment needs to include a client machine, the server(s) supporting the application, a local network connecting these together, the application agent/analyzer, and the application. Developers must be able to execute the transaction across a local network from a client machine. Figure 3-6 illustrates the test environment.

Figure 3-6. Test Environment

The application agent/analyzer captures the transaction's data packets from the network to perform its analysis. To determine how to optimize the system, you can apply the information on user requirements you learned in Chapter 2. By combining this captured data with these requirements, you can begin to identify the efforts needed to meet these goals.

You can capture data to assess application impact in a lab or in a production environment. After you have established which environment you are using, you must decide on the location of capture points within your network.

Comparing Lab Versus Production Environment for Capturing Data

An important advantage to using a lab is that you are not affecting the corporate network; a lab allows maximum flexibility in introducing capture devices and running experiments of your choice.

Second, using a lab enables you to capture application characteristics without encountering network problems, such as packet loss or unusual delays. Because your initial goal is to characterize the application independent of the network on which it is installed, this ability is an important benefit. This approach minimizes the variability introduced by the network, yielding a more accurate characterization of the application.

A third advantage in using a lab is that you can isolate the application of interest. The application capture will contain data relating to a single transaction and will be free of noise from other applications or other instances of the same application. Some applications share a common path/connection across select tiers, and data from multiple transactions are multiplexed along the path. For example, certain three-tier web applications have a common TCP connection on the back end between the web server and the database server. All client requests are sent on the same connection. In a production environment, in which multiple clients access the database continuously, it becomes difficult to isolate one transaction. A lab solves this problem, because you can have a single client.

Unfortunately, the lab approach has an important limitation with respect to the measurement of server response time: The servers in the lab do not necessarily experience workloads similar to those in the production environment. Typically, the workloads, and therefore the response times, are lower in the lab. You can deal with this issue in two ways:

  • Create additional load on the servers in the lab environment to emulate the actual servers' processing delays.

  • Allow the server in the lab to have a light load, but account for this limitation in your application analysis.

As an alternative to the lab approach, you can use the production environment to obtain empirical data about your application. The chief advantage of using the actual environment is that the server response time reflects resource contention on the server. A drawback to using the production environment is the difficulty in isolating an application.

Locating Capture Points

When capturing application traces, the capture device can be located anywhere between the application tiers involved. As long as the capture device is either at the initiation, termination, or transient path of the conversation, it will be captured. When reviewing the data, it is important to be aware of your capturing positions, because they impact the way the data is reflected.

This section covers some generic network designs and discusses the relevant capture points. It gives you a good starting point for your unique environment.

If all the application tiers are located on the same LAN segment, such as a hub, the latency on this network is effectively zero (negligible). The bandwidth is the bandwidth of the shared segment. A single capture is sufficient in this case because the latency is negligible and all the packets or messages are visible. If you are using an analyzer, you can connect it to the central hub as shown in Figure 3-7. If you are using the software agent, you can install the agent on the middle tier (web server), assuming all traffic is visible at the middle tier.

Figure 3-7. Simple LAN

If the tiers are on the same LAN and are separated by more than one hop, such as on a switched LAN environment, the capture device should be placed on a segment where all traffic is visible. In the three-tier example illustrated in Figure 3-8, the capture device can be placed on the hub or the network tap connected to the web server. If the web server is connected directly to a switch, you can connect the capture device to a mirror port on the switch. If you are using the agent, install the agent on the web server or on a separate computer that you can use in a similar fashion as the analyzer. The client/web server traffic and the web server/database server traffic will be visible at this location. A single capture is adequate in this situation, assuming that latencies are negligible on a LAN.

Figure 3-8. Tiers on Different LAN Segments

[View full size image]

If any of the application tiers are on remote segments separated by appreciable latencies, you should capture at multiple points. This strategy enables you to trace the frames through the network. When placing the capture agent, be aware of the basic functional aspects of the conversation. For example, in Figure 3-8 the conversation involves three tiers (initiation on a client that communicates to a web server, which, in turn, communicates with the back-end database). If you just concentrate on the client and back-end database, you will not be able to trace the conversation, because the conversation will not contain common data. (Note that the web server is deemed the endpoint for the conversation between the database server; conversations appear to stop at the web server and then a new conversation begins between the web server and the database.) So by concentrating on the web server, you will be able to see the conversation process from the end-user perspective. You may also need to profile the back-end conversation as a separate exercise to complete an accurate characterization.

The characterization process of an application conversation should take into account the defining elements of that conversation. For example, the client sends an initial request to a web-based server. This server then forwards the specific request for information to a second database server. The requested data is returned to the web server, which then renders the screen for the client. So initially you can say this conversation includes exchanges between the client and the web server and the web server and the database.

By identifying the physical systems involved, you will also be able to determine their geographical position. You should then use this information to determine the physical circuit size linking the sites together.

Next you need to start understanding the detail within that conversation. For example, does the client make a lot of small requests for each separate part of information? Or does it make one request and expect a whole screen of information in return? By understanding this aspect, you will be in a position to determine the number of application turns involved.

You may want to consider the size of the frames that make up this request. Are they larger in one direction than the other? For example, are they larger when the server is responding to the client or when the client is requesting information from the server?

Finally, you need to understand how much data is involved in this exchange. This should then be further defined into payload data and protocol overhead.

From this simplified example, you are now in a position to make some statements about the conversation. You can clearly state the following:

  • The tiers involved in the conversation exchange

  • The geographical position of each tier

  • The minimum bandwidth between tiers

  • The amount of data exchanged (and direction) during the conversation

  • The size and frequency of the requests between the tiers

  • The amount of protocol overhead within the frames

You can gather most of this information by using a data analyzer (such as NAI Sniffer, Compuware Application Vantage, OPNET ACE) to observe the conversation. This will provide you with the breakdown of frames, application turns, and size of data transferred. The information concerning geographic location and minimum bandwidth is more likely to be obtained by referring to site configuration documentation. Figure 3-9 shows placement points for the application analyzers in a multitier environment. The placement at specific aggregation points will enable you to observe the relative conversation frames.

Figure 3-9. Tiers on Different Remote Segments

[View full size image]


Optimizing the Application

Having characterized your application, you should now have a better understanding of how it interacts and how the conversation flows, which part talks to which part and in what order. You should also have a better understanding of how it is made upfor example, how much overhead is used, how many turns are involved, what bandwidth is used, and how much latency it encounters.

Taking this data and applying it to your simple classification criteria (from earlier in this chapter), you can begin to apply the first of your optimization steps to your latency-bounded and bandwidth-bounded applications.

Optimizing the Latency-Bounded Application

Your main aim in optimizing a latency-bounded application is to decrease the time required for a frame to go from one network location to another. You may want to perform the following simple optimization methods (or suggestions) based on the application characteristics:

  • Reduce the number of turns per transaction By eliminating unnecessary calls/queries and session connection setups, you can reduce the number of turns.

  • Reduce the conversation interaction By just implementing more caching of highly used calls, you can reduce the conversation interaction.

  • Ensure the application does not issue connection resets By increasing maximum TCP window size and configuring reasonable timeout parameters (basically making the application more in tune with its environment), you can ensure the application does not issue connection resets.

  • Ensure more data is carried in each frame By increasing the amount of data carried in each frame, it becomes possible to reduce the number of frames required to transport the payload data.

  • Apply some refinements to your network These may include the following:

    - Optimize routing paths

    - Upgrade routers, offload congestion sources

    - Protect traffic with QoS differentiation

    - Challenge/change oversubscription and/or suboptimal routing paths on carrier's network

    - Relocate servers

These improvements can all be implemented accurately because you now understand the characteristics of your business-critical application. Applying QoS to differentiate traffic is certainly something that will be considered; at this stage, however, your task is just to identify the key areas of the application. Chapter 5 discusses these QoS mechanisms and how to apply them to your application characteristics.

Optimizing the Bandwidth-Bounded Application

The bandwidth-bounded application is a little simpler to address. Your main aim in optimizing such an application is to effectively reduce the amount of bandwidth used, or at least control its use to provide a more deterministic response time. Immediate improvements could be made by the following:

  • Physically reducing the amount of data on all or part of the wire

  • Reducing unnecessary interactions by reducing the number of extraneous calls

  • Reducing window size

  • Increasing the capacity

  • Relocating the servers to centralized sites

Physically Reducing the Amount of Data on All or Part of the Wire

The reduction of data on all or part of the wire can be done by introducing caching (or improving existing caching).

Reducing Unnecessary Interactions by Reducing the Number of Extraneous Calls

Reducing the number of extraneous calls can also reduce unnecessary interactions. If you need to search a database for information, for example, use database mechanisms to reduce the amount of data retrieved instead of blindly pulling down the whole database. This strategy may include things such as a localized or reduced address book in an e-mail system or the use of indexing in a database.

Reducing Window Size

Remember, you are trying to improve the use of bandwidth across the network, so areas that were of concern in the latency-bounded application can be used as an optimization tool in bandwidth-bounded applications. By reducing window size on a transmission, you can ensure that the data is transmitted at a constant flow rather than the bursty nature that marks its current character. This constant flow should also stem retransmissions should they occur. Note that you have to be careful not to adjust the window size to such a point that the application becomes latency bounded. To achieve reliability, the sending device transmits a packet and then waits for an acknowledgment (ACK) before transmitting another. The window (or sliding window) technique allows the sender to transmit multiple packets before waiting for an ACK. The window size can be defined as the amount of data allowed to be outstanding (to be sent) at any given point in time.

In a TCP sliding-window operation, for example, the sender might have a sequence of bytes to send (numbered 1 to 10) to a receiver who has a window size of 5. The sender would place a window around the first 5 bytes and transmit them together. It would then wait for an acknowledgment.

The receiver would respond with an ACK = 6, indicating that it has received bytes 1 to 5 and is expecting byte 6 next. In the same packet, the receiver indicates that its window size is 5. The sender then moves the sliding window 5 bytes to the right and transmits bytes 6 to 10. The receiver responds with an ACK = 11, indicating that it is expecting sequenced byte 11 next. In this packet, the receiver might indicate that its window size is 0 (because, for example, its internal buffers are full). At this point, the sender cannot send any more bytes until the receiver sends another packet with a window size greater than 0.

In theory, the TCP window size should be set to the product of the available bandwidth of the network and the RTT (or latency) of data going over the network. Using a cross-country OC-3 link (approximately 60 ms based on a total 6000-mile round trip) creates an available data window of 155 Mbps x 60 ms = 1163 kb. A DS3 satellite connection (540-ms round trip) creates an available data window of 45 Mbps x 540 ms = 3038 kb.

Too large a window can cause the sender to keep transmitting (and therefore waste resources) if the receiver goes down. This is because the sending machine would not be expecting an ACK until it had sent the entire window. Another issue with having the window too large is that routers along the path may become flooded, causing congestion and retransmissions. Too small a window can cause excessive exchanges of windows and ACKs on the network, with each transmission subjected to latency.

Increasing the Capacity

It is cost-prohibitive and inefficient just to increase capacity, but at least now you are armed with relevant information to make an accurate prediction of what bandwidth is needed to support your required response times.

Relocating the Servers to Centralized Sites

The baseline data (along with the modeling) enables you to maximize use of your existing bandwidth by making your decisions based on realistic data. For example, you can perform an accurate cost comparison to relocate a server against ongoing costs of increasing capacity.

These improvements can all be implemented accurately because you now understand the characteristics of your business-critical applications. The one other area, implementing some form of QoS mechanism to differentiate the service, is discussed in Chapter 5.


Verifying Application Delivery

Having gone through the first process of optimization, consider the lessons learned from the characterization and performance goal setting to verify that application delivery is actually meeting your business expectations, and assess that nothing else is affected. Verification ensures that all the infrastructure and applications are behaving as expected.

Essentially your verification is going to come from monitoring your environment and the application performance. The fact that you have built up a definition of your business-critical applications and then profiled those means you have gathered all the baseline data to begin to form your SLA strategy. Specific tools and methodologies are covered in Chapter 8; for now, to verify application delivery, you need to do the following:

Step 1. | Identify the metrics to be monitored.
Step 2. | Consider the time period for monitoring.
Step 3. | Monitor the metrics.
Step 4. | Amalgamate the findings into a common report.

Identifying the Metrics

From the application profile, you can begin to decide on the metrics to monitor. Remember what you are trying to achieve is a framework that ensures that the agreed levels of availability and performance are met. Service level monitoring is never a substitute for good design. If you do not understand your network traffic patterns and fail to optimize your application flow, you will never achieve great performance.

You must decide which metrics to measure. Certain conditions must be in place to achieve your desired result. The application begins on the server, traverses the infrastructure, and ends up delivered to the client station. These are essentially the metric points you need to measure to provide enough meaningful data, not only to report on correct operation, but also to troubleshoot in the event of performance problems. Chapter 8 discusses defining metrics.

You use the constant reference point(s) just to monitor performance. Correct selection of these points is imperative. Selecting the wrong points or oversimplifying the measurements will diminish your ultimate goal.

By identifying the overall traffic flow, you can dramatically reduce the number of metrics needed to predict system behavior. Instead of facing the complexity and cost of trying to gather every possible metric and derive key performance indicators (KPIs) from the aggregate, you can use the virtual model to identify points in a system which in themselves act as aggregators for system faults.

A simple example might be packet retransmission. At a data level, there is a host of conditions, such as dropped frames, which may increase the quantity of retransmitted packets at the application level. At the service level management (SLM) layer, by tracking rising retransmissions you can aggregate data-level conditions.

Reducing the number of metrics does not just reduce the cost of implementing an SLM environment, it also makes it easier to draft straightforward, reportable, and enforceable SLAs.

Considering the Time Period

You must consider the period of monitoring. At this step, the business objectives start to play a part. There is little point in setting and monitoring service levels over a large period of time outside of normal business hours.

Consider, for example, an SLA that states System A will be available 99.5 percent of the time. Based on a 24 x 7 monitoring period, this availability would mean no more than 3.65 hours of downtime on that system in 1 month. But this availability could conceivably happen in a single 3-hour period in the middle of the working day. The SLA would be satisfied, but the potential impact to business would be huge.

If you measure the same availability (99.5 percent) based on office hours only, you reduce the allowable downtime period to no more than 65 minutes in 1 month. There would still be an impact on business, but it would be far less than in the other scenario.

Monitoring the Metrics

By identifying key metrics, you have already started the monitoring process. The monitoring of these metrics and the reference points provides a means of proactively tracking the application's performance. Any deviation, either positive or negative, relative to the baseline measurement will be easily recognized, facilitating quick action.

Amalgamating the Findings

The reporting function of SLM serves dual purposes: verifying performance and providing data for troubleshooting. The information obtained from the various performance metrics enables you to more easily find the causes of problems and fix them.

Chapter 8 covers in more detail this method of using metrics to supply reports and data.


Summary

Initially you need to define which applications are critical to the success of your business. This functional classification provides a focus to begin the technical profiling phase. As well as profiling the specific application, you also begin to profile the overall network environment. This information begins the formation of your baseline, and will not only help you understand what you can achieve through optimization, but will also provide you with a blueprint for the future.

The best way to profile application conversation is to physically monitor the application to identify trends and characteristics over a period of time.

Having observed the application, you are in a position to classify it. At this stage, keep the classification relatively simple, identifying areas such as bandwidth used, application turns, and window size.

Ultimately, you use the profile you have defined to verify your application performance (via monitoring) and to set realistic and achievable SLA criteria.


End Note

  1. Free On-line Dictionary of Computing (FOLDOC). http://www.nightflight.com/foldoc/index.html


References Used in This Chapter

  • Cisco Systems, Service Assure Agent, http://www.cisco.com/en/US/products/sw/iosswrel/ps1830/products_feature_guide09186a0080087a78.html

  • OPNET Technologies, Simulation Technology Options http://www.opnet.com/support/home1.html


Chapter 4. Service Level Management

This chapter includes the following topics:

  • Defining SLAs

  • Assessing your requirements

  • Profiling case study

  • Defining what to monitor

Application delivery now drives the business bottom line and represents the gauge of success or failure for the support organization. Monitoring and reporting this delivery becomes a necessity. Taking this as the basis of service level management (SLM), a service level agreement (SLA) can be defined as a set of criteria that must be met to satisfy the customer's business requirements.

In this chapter, you not only learn the importance and limitations of SLAs, you also learn how to do the following:

  • Profile Define your required service levels based on both business and technical criteria

  • Monitor performance Collect data from key monitoring points to ensure that the required levels are actually met


Defining SLAs

The SLA defines performance metrics used to measure the actual service level obtained from the network against the performance characteristics expected from the network. It is a common agreement between the provider of a service and the recipient of the service. The metrics for network performance can include response time and availability.

SLAs vary, but typically include the following provisions:

  • Availability The percentage of time the service is available; for example, 98 percent availability. This availability requirement can be written to either include or exclude scheduled downtime. For example, this may be 98 percent availability excluding 4 hours maintenance downtime per week.

  • Predictability Network delay, jitter, and packet loss for the specific class of services; for example, average 100-ms round-trip delay between branches during business hours, with less than 3 percent packet loss for data virtual private network (VPN) services.

  • Reparation mean time to repair Mean time to repair (MTTR) is the most common measure of maintainability. It is the average time required to perform corrective maintenance on all the removable items in a product or system. This kind of maintainability prediction analyzes how long repairs and maintenance tasks will take in the event of a system failure. MTTR measures the time between the service interruption and service restoration. MTTR includes problem diagnosis and problem repair. For example, the MTTR for the main access router is 4 hours; this includes time to alert, diagnose, exchange faulty parts (if necessary), and restore service.

  • Service Customer service is the elapsed hour(s) and process time it takes for customer service to respond and deal with alerts; for example, respond to network problems within 2 hours.

  • Credit Reimbursement schedule for violating the guaranteed service level; for example, 15 percent credit toward the next month's service fee.

A successful SLA offering requires network performance monitoring for the following activities:

  • Capacity planning To guarantee a certain service level, it is extremely important to assess the capacity of the network, the traffic mix, and the traffic pattern. Only by monitoring the network over time can one provide the optimal service level guarantees.

  • Service validation Any guarantee requires validation. A network provider has to continuously monitor the service level to make real-time adjustments and to plan for future services. In addition, customers can ensure that their expectations are met by reviewing the measurements. Offering real-time measurement to customers is a valuable marketing tool for promoting service step-ups.

  • Real-time troubleshooting Just as the traffic in the network is dynamic and constantly changing, so is the service level of the traffic. Real-time monitoring ensures that the network provider identifies and resolves any problem in a timely manner. Real-time monitoring also gives users more confidence in their expectations.

Constructing an SLA

Having profiled the application and subsequently optimized it, as described in Chapter 3, "Detailing the Business Transaction," you are now in a position to state the best-case delivery criteria. Constructing an SLA requires planning and care. Although the process can vary among companies, there are often basic elements in common. These are outlined in the following 10 steps:

Step 1. | Assess whether an SLA is appropriate for the application; for example, using jitter as an SLA metric for VoIP.
Step 2. | Get management commitment.
Step 3. | Designate SLA managers.
Step 4. | Educate the parties involved about SLAs.
Step 5. | Assess current services.
Step 6. | Gather customer feedback.
Step 7. | Ensure agreement about the agreement; create a draft.
Step 8. | Solicit feedback.
Step 9. | Complete pre-implementation activities, such as establishing tracking mechanisms and conducting pilots.
Step 10. | Implement and manage the agreement.

Most important is the first step. It will be a waste of time for all concerned if you just pick an arbitrary figure upon which to base the SLA. Equally you should not take the figure from one single source (for example, a database manufacturer's recommendation); instead, you should profile the business transaction and take a performance figure based on the end-to-end conversation process, and based on characteristics pertaining to the network in question.

SLAs are being transformed into a business process. Although it is easy to state that SLAs should be written to serve business objectives, in reality the alignment is not always obvious. Rather than just technical objectives, the methodology you take to arrive at the SLA needs to encompass the overall delivery view, including the technical aspects that underpin the delivery.

Importance and Limitations of SLAs

Although an SLA is an excellent expectations-managing mechanism, it is important to manage your own expectations of what it can realistically accomplish. Unfortunately, some people view an SLA as a complaint-stifling mechanism or a quick fix to a troubled relationship; however, using it for such purposes creates more problems than it solves. Instead, think of an SLA as the following:

  • A communications tool The value of an agreement is not just in the final product; the very process of establishing an SLA helps to open up communications.

  • A conflict-prevention tool An agreement helps to avoid or alleviate disputes by providing a shared understanding of needs and priorities. If conflicts do occur, they tend to be resolved more readily and with less gnashing of teeth.

  • A living document This is one of its most important benefits. The agreement is not a dead-end document consigned to the Forget Forever file. On a predetermined frequency, the parties to the SLA review the agreement to assess service adequacy and negotiate adjustments.

Keep in mind that SLAs are not a universal remedy. They will not magically transform unhappy users into satisfied customers or infuse senior management with glowing confidence in IT's performance. Nevertheless, internal SLAs are a key step toward greater user satisfaction and alignment of IT operations with business goals. By focusing on business requirements and presenting IT performance in business terms, SLAs play an important role in demonstrating IT's value to the business.

Like other business and technical processes, SLAs are not perfect at their inception. They are refined and improved over time. The SLA life cycle is as follows:

Step 1. | Create the SLA package (service, plus service level, plus cost).
Step 2. | Monitor and enforce the SLA.
Step 3. | Assess and refine the service levels.
Step 4. | Analyze and reassess the business unit's service level requirements.

This is a continuous process that leads to improvements in both the level of service that the business unit receives and the efficiency of its delivery.

Having too many SLAs makes tracking and discussing performance more difficult, so start the SLA process by asking yourself where you most need SLAs. By dividing an SLA into different performance levels, such as basic, enhanced, or premium, CIOs can easily show customers the resource and cost trade-offs required to achieve different levels of support, empowering the customer in customizing their SLA.


Assessing Your Requirements

The issue with most SLAs is that they are not based on realistic deliverables. You can meet the goals of an SLA without the business application actually providing a benefit to the business. For example, if you have a warehouse packing system to pack inventory and print out delivery documentation only once every 4 hours, it is not necessary to have an SLA specifying a 1 ms response time.

To assign realistic requirements, consider the following two criteria categories:

  • Business What is really necessary to ensure you continue to meet your business objectives?

  • Technical What is achievable in your environment today?

Business Criteria

The business criteria category requires going back to the understanding of the business process. A customer-facing application (such as a bank teller system) requires faster response times than an unattended warehouse packing system.

It is important to include the business stakeholders in the decision process because they have the best understanding of the required response times. (You can include them by holding regularly scheduled SLA meetings between IT and the stakeholders.) Together with the baseline data, it is possible to combine response time requirements with the existing infrastructure. Not only does this help to assign some form of ownership within the business unit, it could also assist in justifying further system infrastructure investment.

When assigning importance to each SLA, you can invoke the process discussed in the "Defining Your Business-Critical Applications" section of Chapter 2, "Understanding Your Business." Your requirements need to be built around the specific functions of the applications that are most critical to your business operations.

Application performance management (APM) begins in the same way as SLM, with organizations defining what business processes are critical. For example, a company might say, "Our web page must be available to customers." It would then define service objectives such as, "The web page must be available and load within 15 seconds, 98 percent of the time on weekdays and 90 percent of the time on weekends." If there are no clear-cut service objectives in the first place, management is meaningless.

Even when your goal is enterprise-wide SLAs, it is best to begin the process by establishing agreements for one or two mission-critical services, such as network availability or help desk response, which can be closely monitored. This is also an opportunity to experiment with different tools that can help you automate and track the process. Later, with this experience under your belt, you can extend the process to groups of services that encompass an entire business process, such as manufacturing or human resources. When getting started, remember that after you have the right procedures and the right toolset in place, implementing your second SLA will not be nearly as daunting as your first.

Meet with the department heads and elicit their input. When they refer to service quality, what do they mean? What level of service do they really expect? Are they willing to pay a little bit more for a higher standard of service? Agreement is sometimes difficult to reach; if you do not have a thorough understanding of what the business units need from a service, however, you will never be able to provide it.

Cost is a critical aspect of service levels and SLAs. The higher the level of service, the more it will cost to support. Business managers are used to making trade-offs between price and performance and should be able to choose between a premium, more costly service and less-expensive options that cover the basics. This works to IT's advantage because associating costs with service levels allows for more accurate capacity and budget forecasts.

Technical Criteria

Having defined the business criteria, now consider which technical characteristics relate to the business criteria. After you have determined these, identify from where you can collect metrics that represent this criteria. From a technical perspective, you want to determine which parts of the application are important from a monitoring/profile perspective.

In the "Classifying Applications" section of Chapter 3, you learned about the characteristics that make up an application. Concentrate on these characteristics as you look for a series or combination of metrics to facilitate simple monitoring. Consider things such as the following:

  • Number of turns per transaction

  • Number of frames/bytes you expect per transaction

  • Ports/sockets used

  • Window sizes used

  • Application's path through the network

Apart from the technical aspects of the application characteristics, such as the physical number of bits and bytes, an important factor to consider is the ongoing monitoring of these statistics. You want to set up a monitoring system that has negligible impact on the application's performance and overall network load.

The initial aspect of profiling an individual application transaction can essentially be done in isolation as an individual, unique type of activity; therefore, it is not so critical if you collect a large amount of information because you are going to use the data for reference, not to constantly poll the systems.

The tracking of these systems as an ongoing process is something of a different proposition. Take care when tracking and collecting stats. You may be tempted just to turn on all available statistics and collect from all sources; in this case, however, more is not better than less.

Suppose that you do collect a large amount of data. Pulling this data back across the network not only significantly impacts the network load, it also significantly impacts the infrastructure processors, potentially taking away valuable resources by relaying meaningless data back to a management system that would be better served forwarding frames. Therefore, it is better to get specific information instead of blindly polling or collecting every metric available.

Base your technical considerations on facts taken directly from production data. Remember that the existing data is failing to meet your business objectives. You need to optimize, and the SLA objective should be based on a post-optimization projection. To achieve this, implement the optimization process and then reprofile the transaction, or use predictive modeling tools such as OPNET IT Guru or Compuware's Application Expert.

The output in Figure 4-1 compares an application before and after the implementation of optimized SQL code. The optimization's intent was to tighten up the query, ensuring that instead of searching a total of 1000 rows of information, the query only searched 100 rows.

Figure 4-1. Simulation Tool Result of SQL Code Optimization

[View full size image]

Modeling is an effective tool. When able to visualize dramatic improvements because of their work, application developers are more likely to optimize their applications.

Identifying Key Data Collection Points

To get specific information, you must identify key points or critical paths in the application transmission to gather your data. The traditional broadcast domain or segment of Ethernet that was commonly used to monitor no longer exists. It was a shared media in which any one port would give users (through an analyzer) a complete picture of what was happening for all traffic. In today's switched environment, the broadcast domain (VLAN) may span multiple switches, closets, and buildings. Unicast traffic, and sometimes multicast traffic, is segregated to only the ports that need to see it.

Traditionally, you could use a traffic analyzer to look at all traffic on a segment and determine which devices were the busiest, overloaded, underutilized, and so forth. In a switched environment, that overall picture does not exist anywhere in the network. The traffic on each port of a VLAN differs (except for ports with only one active devicethe switch). Even if you use a SPAN port, it gives a picture of what exists on only a single switch. This picture is only part of the same VLAN, which may coexist on multiple switches.

Although the broadcast rate still imposes a limit on the growth of a VLAN (as it did on shared media), it is only part of the picture. Because it is generally not practical to monitor every single switched port (to get a complete picture), you need to identify, document, and maintain only the critical switched portsports that are vital to the success of the network operation.

Examples of critical switched port connections include the following:

  • File/application servers ports

  • Router ports

  • Trunk ports

  • Mahogany Row ports (those executives whose ports should never go down)

Client ports (with the exception of Mahogany Row) should never be considered critical. Trying to maintain management data for that many ports can be overwhelming and can overload switch resources. Receiving alerts whenever users turn off their PCs does not provide useful data in comparison with getting a warning when a trunk or router port goes down in an unplanned fashion.

After you have identified the critical ports, document them, physically mark them, and maintain them through moves/adds/changes. If you do not, network operators run the risk, because of an undocumented change, of reacting to once-critical port conditions that are now noncritical.

Collecting Data

After you have identified your key points, you must collect the data. Table 4-1 compares the two main collection methods used by network application monitoring tools: external probe and embedded agent.

Table 4-1. Comparing Collection Methods

External Probe

Embedded Agent

Does not interfere with network traffic

Interferes with network traffic

Provides accurate performance measurements

Provides very accurate performance measurements

Additional cost and maintenance

More cost-effective

You can measure network performance by placing specialized external hardware probes in the network. These probes can either actively send out synthetic packets to other probes or passively observe packets flowing across the probes.

The major benefit of the external probe method is that external probes do not interfere with network traffic. (Note that a passive probe results in negligible interference, but, by design, active probes place traffic on the network.) In addition, external probes generally provide an accurate reading of network performance. However, adding hardware probes to the network means additional costs and maintenance. To measure performance levels in the overall network, it is necessary to deploy these probes in many locations. A typical SLA requires performance measurement between customer premises equipment (CPE) and the core network; therefore, it is necessary to deploy these probes at each CPE location.

Another collection method, a software probe, embeds a software agent in the routers. This method eliminates the need to deploy and manage hardware probes and is therefore more cost-effective. Because the agents reside in the routers, they have precise knowledge of router delay and can therefore provide very accurate performance measurements. If you have outsourced your infrastructure, however, you do not necessarily have access to the embedded software. If you have a multivendor environment, the critical path of your network cannot be covered fully by embedded software. Cisco Service Assurance Agent (SAA) is a performance-measurement agent embedded in routers powered by Cisco IOS Software.

Measuring Methods

You can calculate the actual measurements in two ways. First, however, you must decide whether you need to passively or actively monitor the data. Your enterprise structure, such as link speeds, and business considerations, such as system criticality, will influence your decision.

The passive method observes the existing packets moving across the network. By correlating the sources, destinations, and time stamps of the packets, a passive monitoring tool can provide the measurements of the actual traffic. Because it only observes the traffic, it does not interfere with the actual operation. However, processing all packet headers is extremely resource intensive and can be done only in a specialized external probe. In addition, because the passive method only looks at the actual traffic, it cannot be used for future service planning.

In contrast to the passive method, the active method (for example, Cisco IOS SAA) generates synthetic packets. This is analogous to testing a traffic delay on the highway by driving across it. Although the active method does inject additional traffic into the network, the added traffic is typically negligible and the added traffic load completely predictable. In addition, synthetic traffic enables a network provider to plan new services by generating the new type of traffic into the network and assessing the performance.

It may be necessary to know when a customer-facing application fails out of hours. Under a passive system, you will only be alerted when a user attempts to connect the next working day. By employing an active system, you can identify the problem out of hours and a resolution can be put in place before any user comes back online.

Cisco NetFlow Example

The Cisco NetFlow feature is an example of a passive device that allows detailed statistics of traffic flows to be collected for SLAs, capacity planning, billing, and troubleshooting functions. You can configure NetFlow on individual interfaces to gather information about traffic passing through those interfaces. The following types of information are part of the detailed traffic statistics:

  • Source and destination IP addresses

  • Input and output interface numbers

  • TCP/UDP source port and destination ports

  • Number of bytes and packets in the flow

  • Source and destination autonomous system numbers

  • IP type of service (ToS)

NetFlow should be activated based on your traffic-flow patterns, network topology, and architecture. Key deployment considerations include the following:

  • Utilize NetFlow services as an edge-metering and access-list performance-acceleration tool and do not activate on hot core/backbone routers or routers running at very high CPU utilization rates.

  • Understand application-driven data-collection requirements. Accounting applications may only require originating and terminating router-flow information, whereas monitoring applications may require a more comprehensive (data-intensive) end-to-end view.

  • Understand the impact of network topology and routing policy on flow-collection strategy. For example, avoid collecting duplicate flows by activating NetFlow on key aggregation routers where traffic originates or terminates, and not on backbone routers or intermediate routers that would provide duplicate views of the same flow information.

  • Utilize NetFlow export data for measuring transit traffic usage (traffic neither originating nor terminating on your network) of network resources for accounting and billing purposes.

NetFlow data gathered on network devices is exported to a collector machine. The collector performs functions such as reducing the volume of data (filtering and aggregation), hierarchical data storage, and file system management. Cisco provides NetFlow Collector and NetFlow Analyzer applications for gathering and analyzing data from routers and Cisco Catalyst switches. There are also shareware tools such as cflowd that can collect Cisco NetFlow UDP records.


Profiling Case Study

In the profiling stage, you start by mapping out the exact service level required (to achieve and maintain the business objectives); otherwise the effort is pointless. You must take into account both the business and technical requirements for delivering the desired result. SLM will never be a substitute for good design. If you do not understand your network traffic patterns, and fail to optimize your application flow, you will never achieve great performance.

Mapping business-critical transactions across supporting infrastructure components is a necessary step in SLM. There are automated tools for discovering components and applications, such as Compuware Network Vantage. SLM solutions should enable you to set parameters against these existing tools. Some SLM software provides topological insights into the path of transactions across the network itself, operating systems, web servers, applications servers, and database servers.

To effectively map a transaction, the IT group needs to meet with someone from the business unit to gather information about what end users do with the system. For example, end users may perform search queries on an inventory management database, or retrieve customer account information from a customer relationship management (CRM) system. SLM solutions provide the detailed information about the network, application, and systems layers of the infrastructure regarding the time it takes between the end users typing in requests until the resulting data appears onscreen.

In the case of a bank, SLM concentrates on the business transaction performed by the teller. The business objective here is to have the teller wait no more than 10 seconds for each operation requested by the customer. The bank wants to eliminate those embarrassing silences during which the teller has to wait up to a minute for the screen to supply correct data.

This business objective becomes your starting point. You then take into context the underlying network infrastructure, the mainframe, the transmission protocols (for example, Data Link Switching [DLSw]), and the application's reaction to this combined picture. Figure 4-2 illustrates the bank teller end-to-end system.

Figure 4-2. Bank Teller End-to-End System

[View full size image]

With newly deployed applications, you have already obtained this data during development and deployment. You now just input that data into your profile. With established applications, you have to profile the application in relation to the specific environment.

The best way to do this is to physically monitor the application in detail to identify trends and characteristics over a period of time. You can also use this data to build a model that emulates application activities. This is called a synthetic transaction and can be used in operation systems to proactively understand the health and relative performance of the relevant application transactions. You can use this software code to repeat the business-user transaction over and over again, simulating an end-user activity but using (and therefore experiencing) the end-to-end transmission path.

Figure 4-3 shows the collected data for a Microsoft Exchange system over a period of time.

Figure 4-3. MS Exchange Family Group Traffic

[View full size image]

This data was collected using Compuware Network Vantage. This tool was used because it enables you to view all systems and associated information exchanges in a single view. The data represents the interaction between two Exchange servers. For each time period, you can see the systems authenticate against each other (MS Netlogin and security authentication), and then exchange data over a server message block (SMB). As you can see, it has left a fairly distinct pattern, on which you will base your application profile. The first thing you will notice is the uniform nature of the exchange, and then the relative values associated to each process (for example, authentication). This is your first indication that the process you want to observe is relatively consistent. Next you need to understand the various parts that make up this process.

One objective of SLM is to identify a unique signature or characteristic that you can use as a trigger point in subsequent monitoring. A single metric can provide a unique reference point from which to view and analyze application performance.

Having established the profile of the application, its path through the network, and the pattern it leaves behind, you can start to assign demarcation points for monitoring performance.

Traditionally, the responsibility is assigned to the respective IT groups; for example, the network demarcation point ends at the server network interface card (NIC) when the server team takes over. In many outsourced systems, however, those lines have become harder to define. In addition, the emphasis on application delivery has led to more confusion as to which department can take responsibility. Therefore demarcation points have to be clearly defined to ensure that no one single part of the conversation process is overlooked.

For example, referring back to the bank teller system, the mainframe department's demarcation point is as soon as the data leaves the mainframe bound for the network. The network group then picks up responsibility as the data traverses through the system, to the client and back.


Defining What to Monitor

When the SLA requirements, metrics, and penalties are in place, you need to measure performance and demonstrate compliance. The goal is to establish a process that does this effectively and efficiently. An organization realizes no net gain if the increased productivity from improved service performance is offset by burdensome compliance procedures. The process of monitoring, defining, generating, and distributing service reports on a regular basis is comprised of routine tasks that should be automated to minimize overheads.

Certain conditions must be in place to achieve the desired result. In the bank teller example, the application begins on the server, traverses the infrastructure, and ends up delivered to the client station. These are essentially the metric points you measure to provide enough meaningful data, not only to report on correct operation but also to troubleshoot in the event of performance problems.

Rather than face the complexity and cost of trying to gather every possible metric and derive a key performance indictor (KPI) from the aggregate, use the information gained in the profiling exercise to identify points in the system that act as natural aggregators (or reference points) for performance or system faults. A simple example may be packet retransmission. At the low level, a host of conditions may increase the rate of retransmitted packets. From a diagnostic view, this must be investigated; at an SLM level, however, this alone may trigger some proactive measure.

Correct selection of constant reference points is imperative. Selecting the wrong points or oversimplifying the measurements will diminish your ultimate goal.

Reducing the number of metrics does not just reduce the cost of implementing an SLM environment, it also makes it easier to draft straightforward, enforceable SLAs.

When defining what to monitor, where to monitor, and the locations and positions of the natural metric aggregators, the final thing to consider is the period of monitoring. This is where the business objectives start to play a part. There is little point setting and monitoring service levels over a large period of time outside of normal business hours or during scheduled maintenance windows.

Manually tracking SLA performance metrics such as network availability is tedious and prone to error. In addition, if someone outside IT (in finance, for example) is calculating the same SLA metrics, the two groups may not agree. Software tools that automate the measuring process save time and reduce error and conflict.

In SLM software, you can define segments such as the following:

  • Operational systems Designed to monitor performance against a given business parameter and, if a failure occurs, identify the technical reasons why a business specification was not met

  • Analytical systems Designed to relate technical metrics to higher-level business objectives, either in real time or on a historical basis

  • Diagnostics systems Designed to help with historical information (understanding why service levels were not met) and with system design (modeling systems against future service level objectives)


Summary

Application delivery is effectively the gauge of success or failure for the support organization. Monitoring and reporting this delivery becomes a necessity, and SLAs provide a mechanism to validate the performance.

SLAs require that you define specific performance characteristics expected from the network. In general, SLAs include provisions such as availability, predictability, MTTR, and reimbursement.

You should consider the SLA a living document. Build mechanisms for review and change. This will allow the SLA to work with the business objective and will assist in alleviating conflicts between delivery groups.

You should also consider the SLA a tool for monitoring application delivery; however, do not think of an SLA as a remedy for poorly performing applications or badly designed networks.


Chapter 5. QoS and MPLS: Tools to Manage Application Performance

This chapter includes the following topics:

  • Background

  • Allocating network resources

  • Traffic-handling mechanisms

  • Provisioning and configuration mechanisms

  • MPLS

  • Traffic engineering

  • Queuing mechanisms

  • Integrated services

  • Resource Reservation Protocol

  • Differentiated services

Mission-critical applications, such as Oracle, Citrix, Microsoft Exchange, and the new breed of web-based applications, must perform well to ensure success in today's fast-paced environment. Bottlenecks may occur in many places, including the network. These bottlenecks often occur even though you have budgeted what you thought was more than adequate bandwidth for each application.

This is generally caused by a misperception of the application characteristics, including areas such as the base transaction (for example, number of turns, payload size, and so on), or by something more fundamental such as an increase in the number of users or an increase in the number of times each transaction is run.

Quality of service (QoS) is an umbrella term for a number of techniques that intelligently match the needs of specific applications to the network resources available. This is accomplished by accurately identifying the application descriptors and then allocating an appropriate amount of network resources such as bandwidth by relative application priority.

The result is that applications identified as business critical can be allocated the necessary priority and bandwidth levels to run efficiently. Applications identified as less than critical can be allocated best-effort bandwidth and will thus run at a lower priority.

This chapter does not attempt to detail the various intricacies and algorithms associated with the various protocols and mechanisms; such detail would draw focus away from the goal of optimizing application delivery. Instead, this chapter describes the various protocols and techniques used in the application delivery model. The chapter then covers allocation of network resources relative to application needs and the associated mechanism and protocols used by the infrastructure to deal with these needs. Finally, the chapter addresses the various characteristics of queuing and forwarding protocols engaged in the process.

If you do require specific protocol details, refer to the relevant organizations and standards, which are listed in Appendix A, "QoS-Related Requests For Comments," and Appendix B, "Guide to QoS Standards."


Background

QoS encompasses several technologies that enable selected applications to receive preferential access to shared resources, thus ensuring a specified level of performance for these applications. Pertinent performance parameters include jitter, throughput, latency, and packet loss. There are various approaches to allocating resources to attempt to ensure that these performance parameter values stay within acceptable ranges for specific applications or classes of applications.

When multiple applications are running simultaneously over the same network infrastructure, resources must be shared. When the Internet was first developed, connectivity was the primary objective. Most of the traffic was either file transfer or e-mail; the primary parameter by which to measure network performance was throughput.

With the rapidly growing role of multimedia and collaboration in current and future applications, connectivity alone is no longer adequate; emerging applications require interactive support from the network infrastructure. As a result, performance parameters such as packet delay, packet loss, and jitter are becoming just as important as throughput.

To support the diverse requirements of applications that will share tomorrow's network infrastructure, a portion of network resources must be allocated either to individual applications or to categories of applications.


Allocating Network Resources

Devices that provide QoS support do so by intelligently allocating resources to submitted traffic. This resource allocation is a configurable function of the network administrator. Under congestion, for example, a network device might choose to queue the traffic of applications that are more latency tolerant instead of the traffic of applications that are less latency tolerant. As a result, the traffic of applications that are less latency tolerant can be forwarded immediately to the next network device. In this example, interface capacity is a resource that is granted to the latency-intolerant traffic. Device memory is a resource that has been granted to the latency-tolerant traffic.

To allot resources preferentially to various types of traffic, it is necessary to identify different traffic.

It is important to understand the characteristics of the applications that need protection. Some applications tend to be sensitive to latency or packet loss, whereas others are considered aggressive because they are bursty or bandwidth hogs. If the application is bursty, is it a constant burst or a small burst? Is the packet size of the application large or small? Is the application TCP or UDP based?

You then need to associate the application relative to its characteristics with certain resources. This is typically achieved as follows:

  1. Traffic arriving at network devices is identified in each device and is separated into distinct flows via the process of packet classification. In broad terms, a flow can be described as a stream of packets or frames passing through a network device that belong to the same conversational process and have the same QoS requirements.

  2. Traffic from each flow is directed to a corresponding queue.

  3. The queues are then serviced according to some queue-servicing algorithm. The queue-servicing algorithm determines the rate at which traffic from each queue is submitted to the network, thereby determining the resources that are allotted to each queue and to the corresponding flow.

Thus, to provide network QoS, it is necessary to provision the following in network devices:

  • Classification information by which devices separate traffic into flows

  • Queues and queue-servicing algorithms that handle traffic from the separate flows

To provide preferential service to a type of traffic, it must first be identified. Second, the packet may or may not be marked. Marking allows changing the QoS level of the packet based on classification or policing. These two tasks make up classification. Classifying is the mechanism by which packets are separated for different treatment, possibly different output queues. During the process of accepting, routing, and transmitting a packet, a networking device can classify the packet a number of different ways. Classification can include marking the packet, which usually happens on the boundary of a network under a single administrative control, or classification can occur on each hop individually.

Common Cisco methods of identifying flows include access control lists (ACLs), policy-based routing, committed access rate (CAR), and network-based application recognition (NBAR). In network devices, classifiers are used to separate traffic into flows. These flows are then directed to the appropriate queues.

Because of the bursty nature of, for example, video and data traffic, sometimes the amount of traffic exceeds the speed of a link. At this point, what will the network device do? Will it buffer traffic in a single queue and let the first packet in be the first packet out? Or will it put packets into different queues and service certain queues more often? Congestion-management techniques address these questions. Techniques include priority queuing (PQ), custom queuing (CQ), weighted fair queuing (WFQ), class-based weighted fair queuing (CBWFQ), random early detection (RED), and weighted random early detection (WRED).

Note that QoS mechanisms do not create resources. They merely reallocate existing resources among different traffic classes. When certain traffic classes are allocated more resources, other traffic flows are allocated fewer resources. However, certain applications require additional resources for their traffic flows, whereas others operate satisfactorily with fewer resources. Given the lower available bandwidth in WAN environments, QoS mechanisms are generally more prevalent in the WAN than the LAN scenario.


Traffic-Handling Mechanisms

Any QoS-enabled network relies on fundamental traffic-handling mechanisms in the network elements through which data is to be carried. These elements are predominantly switches and routers. The traffic-handling mechanisms classify traffic as belonging to one flow or another. Then the queuing mechanisms allot more or fewer resources to traffic based on the particular flow classification.

Networks interconnect hosts using a variety of network devices, including host network adapters, routers, switches, and hubs. Each of these contains interfaces; these interfaces interconnect the various devices via cables, fiber, and/or wireless. Network devices generally use a combination of hardware and software to forward traffic from one interface to another.

Each interface can send and receive traffic at a finite rate. If the rate at which traffic is directed to an interface exceeds the rate at which the interface can forward the traffic onward, congestion occurs. Network devices may handle this condition by queuing traffic in the device's memory until the congestion subsides. In other cases, network equipment may discard traffic to alleviate congestion. As a result, applications experience varying latency (as traffic backs up in queues at interfaces) or traffic loss.

The capacity of interfaces to forward traffic and the memory available to store traffic in network devices (until it can be forwarded) are the fundamental resources required to provide QoS to application traffic flows. Mechanisms internal to network devices determine which traffic gets preferential access to these resources. These are the fundamental traffic-handling mechanisms used to provide QoS to the networked applications.

You can subdivide the various traffic-handling mechanisms necessary to implement a QoS system into two categories:

  • Those pertaining to the network devices

  • Those pertaining to the actual routes themselves

Queuing Algorithms

This chapter's discussion of the network devices is primarily concerned with the queuing algorithms themselves, such as the following:

  • First in, first out (FIFO)

  • Priority queuing (PQ)

  • Class-based queuing (CBQ)

  • Weighted queuing schemes

Because queues are not of infinite size, they can fill and overflow. When a queue is full, any additional packets cannot get into the queue and are dropped. This is called a tail drop. The issue with tail drops is that the network device cannot prevent these packets from being dropped (even if they are high-priority packets).

Therefore, a mechanism is necessary to do the following two things:

  • Try to make sure that the queue does not fill up, so that there is room for high-priority packets

  • Allow some sort of criteria for dropping packets that are of lower priority before dropping higher-priority packets

WRED achieves both of these requirements.

Routing Protocols

When considering the traffic-handling mechanism, do not neglect the fundamental aspects of your network design. Critical to the overall delivery model is the path taken through the network. For this path, you need to rely on the underlying routing protocols, such as Enhanced Interior Gateway Routing Protocol (EIGRP), Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), and so on, as well as consider protocols such as Multiprotocol Label Switching (MPLS) to enable the most effective path (from a quality and business perspective) is taken.

It is very important that your routing protocol is configured to enable the most optimum path through the network. This may require some additional weighting or tuning to the standard routing protocols to ensure that the most preferential path for the characteristics of your application will be taken.

In standard routing protocols, best-effort services are obtained by computing lowest-cost paths. The cost, in this case, can be defined on a single administratively assigned link metric (such as hop count and highest available bandwidth).

When you are considering the optimization and ultimate aim of the application-delivery model with respect to QoS, you need to consider a routing protocol model that supports an alternative path selection. As well as the traditional metrics, you need to consider the addition of link metrics such as latency and jitter to your model. It is also important to understand that the metrics may well fluctuate with respect to the characteristics of the different applications.

You can assign all the queuing algorithms and high-speed architecture available, but in the end, if you fail in providing this basic transit through the network, you will not achieve your desired result.


Provisioning and Configuration Mechanisms

To effectively provide network QoS, it is necessary to affect the provisioning and configuration of the traffic-handling mechanisms across multiple network devices. Provisioning and configuration mechanisms can be categorized as top-down (pushed) or signaled. Table 5-1 compares these mechanisms.

Table 5-1. Top-Down vs. Signaled Mechanisms

Pushed Mechanisms

Signaled Mechanisms

Push configuration information from a management console down to network devices

Carry QoS requests (and implicit configuration requests) from one end of the network to the other, along the same path traversed by the data that requires QoS resources

Initiated on behalf of one or more applications by a network management program

Initiated by an application's changes in resource demands

Determining appropriate classification criteria to use can be quite challenging

Provides robust bindings between classification information and users and applications

In top-down provisioning, a network management system is used to push traffic-handling configuration to a set of network devices. Typically, queuing mechanisms are configured on device interfaces. Then classification criteria are configured to determine which packets are directed to different queues in the device. Classification criteria may classify packets based on source and destination IP addresses and TCP ports and the IP protocol or QoS protocol (such as 802.1q in a switch environment) marks in packet headers.[1]

Determining the appropriate classification criteria to use can be quite challenging. Network administrators would like to use QoS to assign resources to the traffic of certain applications or users, rather than fields in packet headers such as IP addresses and TCP ports. Top-down provisioning systems attempt to assist the network administrator by creating bindings between applications and TCP/UDP ports and between users and IP addresses. Unfortunately, these are often unreliable. Applications may use transient ports or may source multiple traffic flows (requiring differing QoS) on the same port. Users' IP addresses may change (or may be translated) as a result of Dynamic Host Configuration Protocol (DHCP) configurations, or general mobility. In addition, encryption may encrypt IP ports, rendering them useless as classification criteria.[2]

Signaled configuration is generally initiated from applications on hosts. These applications inform relevant devices along the traffic path regarding the nature of the application traffic and request admission of the application flows to certain QoS requirements. One clear benefit of host-based signaling is that it provides robust bindings between classification information and users and applications. Beyond this, host-based signaling brings topology-aware dynamic admission control.

Pushed and signaled approaches are not mutually exclusive, and in most cases can be found deployed together on the same network, applied to different application types to meet different delivery requirements. Figure 5-1 shows both approaches.

Figure 5-1. Basic Operation of Pushed vs. Signaled Configuration Systems

[View full size image]


MPLS

MPLS and related technologies offer the potential to improve network performance. The basic concept of label switching is very simple. In a network not employing label switching, the method to relay data from one end to the other is very similar to the method used by the postal system. That is, an address is examined by the relaying infrastructure. This address determines how the infrastructure forwards the data.

Label switching is different. Instead of using a destination address to make the routing decision, a number (a label) is associated with the data. This number is then referenced and used to route the data to the recipient. So label switching can improve the speed of data forwarding through the network.[3]

Label switching does not have anything to do with certain aspects of QoS categories, such as raw bandwidth; however, label switching can be a valuable tool to combat latency and jitter, two important operations for delay-sensitive traffic, such as video and voice. Because label switching is designed to speed up relaying of traffic in a network, it follows that the technology will reduce latency and improve jitter. Label switching on its own is not going to solve delay and variable delay problems. If you are connected to a low-bandwidth network, label switching will not provide you any more bandwidth, but label switching will improve delay and jitter problems significantly.

Although it can be argued that Asynchronous Transfer Mode (ATM) never quite reached the acceptance and penetration its supporters originally predicted, many of its concepts remain quite capable. One of these is the concept of switching a discrete unit of data along a predetermined circuit.

In conventional IP routing, a router performs the following process:

  1. Inspects the destination address of each packet as it arrives at the router

  2. Applies a routing lookup to determine the interface to forward the packet to, with the ultimate aim of reaching its destination

In ATM cell switching, the following occurs:

  1. A sender requests a virtual circuit (VC) to a certain destination.

  2. ATM switches along the path from source to destination establish the VC to the destination, and then send an identifier to the sender.

  3. The sender attaches the identifier to each cell transmitted to the destination. As the cell traverses the path, each device retrieves the identifier and uses it to establish which interface to forward the cell to (or out of). This identifier remains valid for the life of the VC (as long as the flow is required).

Many vendors took up this approach and began devising schemes where IP routers possessed the ability to establish temporary VCs for what could be described as commonly used paths. There were subtle differences between the implementations but essentially routers along a given path would cooperate and assign a label (or tag) relevant to a particular path. Hosts or routers close to the sending host would then participate by tagging packets with the appropriate label to facilitate its quick passage through the network. This became commonly known as label switching.

The MPLS standard, published by the Internet Engineering Task Force (IETF), evolved from the Cisco Tag Switching implementation. At its conception, it was originally intended to improve the performance of traditional IP routing. However, implementations of MPLS seen today are not essentially intended to offer significant performance advantages over traditional IP routing. MPLS provides significant benefits to networks with pure IP architectures as well as those with IP and ATM or a mix of other Layer 2 technologies.

MPLS technology is key to scalable virtual private networks (VPNs) and end-to-end QoS, enabling both efficient utilization of existing networks to meet future growth as well as rapid fault correction of link and node failure. The technology also helps deliver highly scalable, differentiated end-to-end IP services with simpler configuration, management, and provisioning for both Internet providers and subscribers.

In summary, MPLS can be defined as a multiprotocol switching technology that uses labels to determine how packets are forwarded through a network.

MPLS uses label switching to forward data through the network. A small, fixed-format label is inserted in front of each data packet on entry into the MPLS network. At each hop across the network, the packet is forwarded based on the value of the incoming interface and label, and dispatched to an outward interface with a new label value.

The path that data follows through a network is defined by the transition in label values, as the label is swapped at each label switch router (LSR). Because the mapping between labels is retained at each LSR, the path is determined by the initial label value. Such a path is called a label switched path (LSP).

At the ingress to an MPLS network, each packet is examined to determine which LSP it should use and hence what label to assign to it. This decision is a local matter but is likely to be based on factors including the destination address, the QoS requirements, and the current state of the network. This flexibility is one of the key elements that make MPLS so useful.

The set of all packets that are forwarded in the same way is known as a forwarding equivalence class (FEC). One or more FECs may be mapped to a single LSP. Figure 5-2 shows two data flows from host A: one to host F, and one to host C. Two LSPs are shown.

Figure 5-2. Basic Operation of an MPLS Network

[View full size image]

LSR W is the ingress point into the MPLS network for data from host A. When it receives packets from host A, LSR W determines the FEC for each packet, deduces the LSP to use, and adds a label to the packet. LSR W then forwards the packet on the appropriate interface for the LSP.

LSR X is an intermediate LSR in the MPLS network. It simply takes each labeled packet it receives and uses the pairing (incoming interface, label value) to decide the pairing (outgoing interface, label value) with which to forward the packet. This procedure can use a simple lookup table and, together with the swapping of label value and forwarding of the packet, can be performed in hardware. This allows MPLS networks to be built on existing label switching hardware such as ATM and Frame Relay. This way of forwarding data packets is potentially much faster than examining the full packet header to decide the next hop.

In the example, each packet with label value 20 will be dispatched out of the interface toward LSR Z, bearing label value 27. Packets with label value 17 will be relabeled with value 15 and sent toward LSR Y.

LSR Y and LSR Z act as egress LSRs from the MPLS network. These LSRs perform the same lookup as the intermediate LSRs, but the (outgoing interface, label value) pair marks the packet as exiting the LSP. The egress LSRs strip the labels from the packets and forward them using Layer 3 routing.

So, if LSR W identifies all packets for host C with the upper LSP and labels them with value 20, they will be successfully forwarded through the network.

Note that the exact format of a label and how it is added to the packet depends on the Layer 2 link technology used in the MPLS network. For example, a label could correspond to an ATM virtual path identifier/virtual circuit identifier (VPI/VCI), a Frame Relay data link connection identifier (DLCI), or a dense wavelength-division multiplexing (DWDM) wavelength for optical networking. For other Layer 2 types (such as Ethernet and PPP), the label is added to the data packet in an MPLS shim header, which is placed between the Layer 2 and Layer 3 headers.

MPLS also provides a flexible and elegant VPN solution based on the use of LSP tunnels to encapsulate VPN data. The basis of any MPLS solution for VPNs is the use of LSP tunnels for forwarding data between edge routers that border on a given VPN. By labeling the VPN data as it enters such a tunnel, the LSR neatly segregates the VPN flows from the rest of the data flowing in the service provider backbone.

MPLS traffic engineering (TE) enables an MPLS backbone to expand on the basic traffic-engineering capabilities of Layer 2 ATM and Frame Relay networks. Traffic engineering is enabled through MPLS mechanisms that allow traffic to be directed through a specific path, which may not necessarily be the least-expensive path.

MPLS is sometimes used synonymously with QoS, but more accurately it is a QoS-enabling technology that forces application flows into connection-oriented paths and provides mechanisms for traffic engineering and bandwidth guarantees along these paths.[4]

Furthermore, when an MPLS network supports differentiated services (DiffServ), traffic flows can receive class-based admission, differentiated queue servicing in the network nodes, preemption priority, and other network treatment that provide a basis for QoS guarantees.


Traffic Engineering

Traffic engineering (TE) is the process of routing data through the network according to a management view of the availability of resources and the current and expected traffic. The goal in traffic engineering is to select paths to optimize the use of the networks available resources, to in effect engineer its delivery (for example, reduce congestion or meet latency or QoS targets). The class of service and QoS required for the data can also be factored into this process.

In a theoretical model, the following four steps must be undertaken to achieve traffic engineering:

Step 1. | Measure network traffic.
Step 2. | Characterize network traffic.
Step 3. | Model network traffic.
Step 4. | Move traffic to its desired location.

To measure, characterize, and model traffic for the entire Internet is an immense task that would require resources far in excess of those available. It is not practical to implement, and prior to MPLS you would have had to understand the characteristics and the traffic models of the entire Internet to perform TE.[5]

On a smaller scale, TE may be under the control of manual operators where managers monitor the state of the network and route the traffic or provision additional resources to compensate for problems as they arise. Alternatively (and preferably), automated processes reacting to information fed back through routing protocols or other means may drive TE.

One of the automated approaches and, in fact, main uses for MPLS is to allow improved TE for the Internet service provider (ISP) backbone networks. MPLS TE replaces the need to manually configure the network devices to set up explicit routes. Instead, you can rely on the MPLS TE functionality to understand the backbone topology and the automated signaling process. The focus of TE for MPLS networks is both the measurement and control of the traffic. The latter operation deals with the process to ensure the network has sufficient resources to support the QoS requirements.

In its simplest form, TE attempts to optimize QoS to meet user needs by making the best use of network resources to support those needs. In an environment that does not have sufficient network resources (link speed and processing power) to support all the users QoS requirements all the time, TE must shape the user traffic.

This means putting in place mechanisms to determine how the network supports the different classes of user traffic. (A full section later in this chapter is devoted to the discussion of queuing mechanisms.) The next issue to consider in a TE model is routing.

The routes between the endpoints are set up by routing protocols, such as OSPF and BGP. These protocols are not designed for router and resource optimization. They are in fact biased toward the shortest path, although it should be noted that BGP is far more complex than just shortest path. They build a route based on the topology of the network, not on the resource requirements defined under TE exercise.

MPLS TE automatically establishes and maintains paths across the backbone by using Resource Reservation Protocol (RSVP). The path that MPLS uses is determined by the resource requirements (including availability) and network resources, such as bandwidth. This is defined as constraint-based routing, in which the path for a traffic flow is the shortest path that meets the resource requirements (constraints) of the traffic flow.

MPLS TE is built on the following Cisco IOS mechanisms:

  • IP tunnel interfaces From a Layer 2 standpoint, an MPLS tunnel interface represents the head of an LSP. It is configured with a set of resource requirements, such as bandwidth and media requirements, and priority. From a Layer 3 standpoint, an LSP tunnel interface is the head end of a unidirectional virtual link to the tunnel destination.

  • MPLS TE path calculation module This calculation module operates at the LSP head. The module determines a path to use for an LSP. The path calculation uses a link-state database containing flooded topology and resource information.

  • RSVP with TE extensions RSVP operates at each LSP hop and is used to signal and maintain LSPs based on the calculated path.

  • MPLS TE link management module This module operates at each LSP hop, does link call admission on the RSVP signaling messages, and does bookkeeping of topology and resource information to be flooded.

  • Link-state IGP (IS-IS or OSPF, each with traffic engineering extensions) These interior gateway protocols (IGPs) are used to globally flood topology and resource information from the link management module.

  • Enhancements to the SPF calculation used by the link-state IGP (IS-IS or OSPF) The IGP automatically routes traffic onto the appropriate LSP tunnel based on tunnel destination. Static routes can also be used to direct traffic onto LSP tunnels.

  • Label switching forwarding This forwarding mechanism provides routers with a Layer 2-like capability to direct traffic across multiple hops of the LSP established by RSVP signaling.

One approach to engineering a backbone is to define a mesh of tunnels from every ingress device to every egress device. The MPLS TE path calculation and signaling modules determine the path taken by the LSPs for these tunnels, subject to resource availability and the dynamic state of the network. The IGP, operating at an ingress device, determines which traffic should go to which egress device, and steers that traffic into the tunnel from ingress to egress.

A flow from an ingress device to an egress device might be so large that it cannot fit over a single link, so a single tunnel cannot carry it. In this case, multiple tunnels between a given ingress and egress can be configured, and the flow is load shared among them.


Queuing Mechanisms

In any QoS system, the role of the network devices is to ensure the packets get forwarded as efficiently as possible. That may be in order of priority or just as quickly as can be. Delays in this process start to occur when these devices cannot actually forward data at the desired rate and have to begin placing the data into queues for future forwarding. If these queues are not efficiently serviced, you begin to see some form of congestion as potential packet drops occur.

One way that network elements handle an overflow of arriving traffic is to use a queuing algorithm to sort the traffic, and then determine some method of prioritizing it onto an output link.

Cisco IOS Software includes the following queuing tools:

  • First-in, first-out (FIFO) queuing

  • Priority queuing (PQ)

  • Custom queuing (CQ)

  • Flow-based weighted fair queuing (WFQ)

  • Class-based weighted fair queuing (CBWFQ)

  • Low-latency queuing (LLQ)

Each queuing algorithm was designed to solve a specific network traffic problem and has a particular effect on network performance.

Queues and queue-servicing algorithms are critical elements of traffic handling in providing network QoS. Queuing happens only when the interface is busy. As long as the interface is idle, packets will be transmitted without special treatment. Regular queues invariably employ the FIFO principle: The packet that has been waiting the longest is transmitted first. When the queue is full, and additional packets come in, tail drops happen.

More sophisticated queuing mechanisms usually employ several queues. Packets are classified by user-configurable means and then placed in the appropriate queue. Then, when the interface is ready to transmit, a queue from which the next packet will be transmitted is selected as per the queuing algorithm. Note that special queuing mechanisms have effect only when it is not possible immediately to transmit a packet over the output interface. If the interface is idle and there are no queued packets, the new packet is transmitted immediately.[6]

When evaluating the suitability of a queuing scheme, consider the following factors:

  • Type of scheme Should be appropriate for the task. Some schemes are designed to share available capacity optimally, whereas others are designed to control the impact of a particular type of traffic on the network.

  • Use of a dropping scheme This is a mechanism by which you can effectively control the length of a queue, and in doing so can successfully define the latency.

  • Different mechanisms You are looking at a dynamic instance. Therefore, you may need to apply different mechanisms to different parts of the journey.

Queuing schemes provide QoS by controlling the forwarding capacity or bandwidth available to certain traffic flows.

This section separates queuing mechanisms into the following three broad categories related to serving general application characteristics requirements:

  • FIFO

  • Work-conserving

  • Non-workconserving

Technically speaking, there are really only two categories, because FIFO is essentially classed as a work-conserving queuing discipline. For the purposes of this discussion, however, it is treated as a third category.

Work-conserving describes a queuing mechanism that saves the forwarding device work. When a device has packets in one of its queues, it wants to do as little work on it as possible. Holding it in a queue effectively means that the device must perform additional work on the packet. So a network device can be considered busy whenever the service queue contains queued requests. A work-conserving mechanism will always transmit one packet, provided there is at least one packet inside the queue.

A non-workconserving mechanism is one that expects the device to perform extra work, such as holding the transmission of packets on the output link even if some packets are waiting inside the router buffers. This is done to ensure delay and jitter can be guaranteed, which is imperative in delay-sensitive applications such as voice.

Based on this description, you can see that FIFO is in fact the simplest work-conserving queuing discipline. FIFO queuing is the most basic queuing strategy; packets are just transmitted in the same order they come in.

FIFO

In the context of QoS, you expect queuing mechanisms to enhance the delivery model by giving specific traffic priority over other traffic. Because FIFO queuing cannot alter the order in which the packets are forwarded, the service experienced by any particular flow is going to depend entirely on the order in which the packets arrive, as shown in Figure 5-3. FIFO also has the potential problem of delaying short frames behind longer frames. This is caused by the longer frame filling the queue.

Figure 5-3. FIFO Queuing Example

[View full size image]

Figure 5-3 shows three connections to a single router. Links A and B contain traffic coming from a priority application and site; however, the available link speeds are 128 kbps and 64 kbps, respectively. The less-important traffic is seen coming in on interface C and has been connected using a T1 (1.54-Mbps) circuit. The data arrives more frequently on this higher-bandwidth circuit than the data of higher-priority links A and C. The result is that the traffic arriving via link C will experience better service than the lower-bandwidth connections.

FIFO has been included for completeness, and does have a part to play in the overall delivery system. However, the FIFO queuing scheme is incapable of controlling the service allotted to a traffic flow and as such has limited use in supporting QoS.

Work-Conserving Queuing Mechanisms

All the Cisco IOS queuing tools listed at the beginning of this section are work-conserving queuing mechanisms.

Priority Queuing

Priority queuing (PQ) ensures that important traffic gets the fastest handling at each point where it is used. It was designed to give strict priority to important traffic. PQ can flexibly prioritize according to network protocol (for example IP, IPX, or AppleTalk), incoming interface, packet size, source/destination address, and so on. In PQ, each packet is placed in one of four queues (high, medium, normal, or low) based on an assigned priority. Packets that are not classified by this priority list mechanism fall into the normal queue. During transmission, the algorithm gives higher-priority queues absolute preferential treatment over low-priority queues.

PQ can greatly improve latency for high-priority packets, but it does have FIFO-style problems of delaying short packets behind long ones. It also has issues of fairness within each queue. In addition, it may well starve some flows completely, preventing acknowledgment frames from returning in time, ultimately causing the entire VC to collapse. It also has the potential to increase jitter in lower-priority packets. Therefore, PQ is a relatively efficient method for prioritizing a few flows.

Custom Queuing

Custom queuing (CQ) was designed to allow various applications or organizations to share the network among applications with specific minimum bandwidth or latency requirements. In these environments, bandwidth must be shared proportionally among applications and users. CQ handles traffic by assigning a specified amount of queue space to each class of packets and then servicing the queues in a round-robin fashion. The queuing algorithm places the messages in 1 of 17 queues (queue 0 holds system messages such as keepalives, signaling, and so on) and is emptied with weighted priority. The router services queues 1 through 16 in round-robin order, de-queuing a configured byte count from each queue in each cycle. This feature improves fairness; a protocol stack that tries to push too many packets through will simply overflow its private queue instead of grabbing bandwidth from others when the line is under stress. The round-robin approach also avoids complete starvation of any flows. However, fairness still depends on packet size. Because one packet is taken on each cycle, flows using large packets can get much more bandwidth than flows with small packets.

Flow-Based Weighted Fair Queuing

For situations in which it is desirable to provide consistent response time to heavy and light network users alike without adding excessive bandwidth, the solution is flow-based weighted fair queuing (commonly referred to as just WFQ). WFQ is a Cisco premier queuing technique. It is a flow-based queuing algorithm that creates bit-wise weighted fairness by allowing each queue to be serviced fairly in terms of byte count. For example, if queue 1 has 100-byte packets and queue 2 has 50-byte packets, the WFQ algorithm takes two packets from queue 2 for every one packet from queue 1. This makes service fair for each queue: 100 bytes each time the queue is serviced.

WFQ ensures that queues do not starve for bandwidth and that traffic gets predictable service. Low-volume traffic streams (which comprise the majority of traffic) receive increased service, transmitting the same number of bytes as high-volume streams. This behavior results in what appears to be preferential treatment for low-volume traffic, when in actuality it is creating fairness.

The weighted portion of WFQ comes from the use of IP precedence bits to provide greater service for certain queues. Using settings 0 to 5 (6 and 7 are reserved), WFQ uses its algorithm to determine how much more service to provide to a queue. In practical implementations, weights generally are not used on a per-conversation basis, but to bias certain aggregate flows.

WFQ is efficient in that it uses whatever bandwidth is available to forward traffic from lower-priority flows if no traffic from higher-priority flows is present. This is different from strict time-division multiplexing (TDM), which simply carves up the bandwidth and lets it go unused if no traffic is present for a particular traffic type. WFQ works with both IP precedence and RSVP (described later in this chapter) to help provide differentiated QoS as well as guaranteed services. WFQ is generally used for applications that need latency guarantees, such as real-time speech.

Class-Based Weighted Fair Queuing

Class-based weighted fair queuing (CBWFQ) extends the standard WFQ functionality to provide support for user-defined traffic classes. For CBWFQ, you define traffic classes based on match criteria including protocols, access control lists (ACLs), and input interfaces. CBWFQ enables a network administrator to create minimum guaranteed bandwidth classes. Instead of providing a queue for each individual flow, a class is defined that consists of one or more flows. Each class can be guaranteed a minimum amount of bandwidth.

Low-Latency Queuing

In addition, a low-latency queue (LLQ) may be designated, which essentially is a priority queue. Note that this feature is also referred to as priority queue class-based weighted fair queuing (PQCBWFQ). This mechanism provides a very fair and efficient distribution of bandwidth between classes, and it works well in high-speed circuits (such as T1 and above). It is normally used to allocate bandwidth on high-speed backbones.

Table 5-2 shows general application characteristics and QoS implementation guidelines.

Table 5-2. QoS Implementation Guidelines[7]

Characteristic

Guideline

Application that is delay or loss sensitive (voice and real-time video)

Do not use WRED, traffic shaping, fragmentation (FRF-12), or policing. For this kind of traffic, implement LLQ and use a priority queue for the delay-sensitive traffic.

Application that is consistently bursty or is a bandwidth hog (FTP and HTTP)

Use WRED, policing, traffic shaping, or CBWFQ to guarantee bandwidth.

Application that is TCP based

Use WRED because lost packets will cause retransmissions. If the traffic is UDP based and does not retransmit, do not use WRED. Use policing if you need to rate-limit the application; otherwise let the packets tail drop.

Non-WorkConserving Queuing Mechanisms

Non-workconserving queuing schemes are typically used to control the impact of traffic on the network by limiting the rate at which it is forwarded to the network. These mechanisms are often referred to as traffic-shaping mechanisms.

Traffic shaping attempts to adjust the transmission rate of packets that match a certain criteria. Holding packets in a buffer and releasing them at a preconfigured rate accomplishes this. Non-workconserving schemes or traffic-shaping schemes shape traffic according to a set of parameters. They are typically used to shape traffic either at the traffic source (as near to it as possible) or in network devices on the periphery of the network. This is because in the core the link speeds tend to be higher as is the number of flows that need to be shaped. This impacts the amount of processing required in these devices.

In many environments, implementing a relatively simple LLQ or PQ scheme is sufficient. However, these queuing methods still pass traffic as fast as they can; their goal is to service some types of traffic before others. Sometimes you actually need to slow traffic down, which you can do with traffic shaping. This is most common in Frame Relay environments, where one access circuit is much faster than another, but it can be used on most other types of interfaces. Shaping can also help your Voice over IP (VoIP) network by regulating the speed of nonvoice traffic and normalizing the rate of flow for your voice traffic.

All traffic, real time or not, is bursty to some degree. Shaping is used to control the impact of bursts on the network. It is not used to completely eliminate bursts. To account for the bursty nature of traffic, the following two models are used:

  • Leaky bucket

  • Token bucket

The theory behind this shaping is to smooth out a traffic flow and reduce traffic clumping. This is essentially done by controlling the volume of traffic entering the network.

Both models operate in a similar way. In the leaky bucket method, the traffic arriving at the queuing stage is inserted into the bucket from the top. A hole in the bottom of the bucket allows traffic to leak out at a fixed rate. This is the rate at which the forwarding stage forwards the traffic.

In the token bucket method, the bucket is filled with tokens at a fixed rate. Each time the forwarder is ready to forward a packet from a queue, it must also remove a token from the bucket. When no tokens are available, the forwarder must either drop packets or delay them.

The rate at which the data is forwarded is controlled by a combination of the rate (at which the tokens are entered in token bucket, and the rate at which packets leak in leaky bucket) and the depth of the bucket itself.

Figure 5-4 depicts a token bucket model with data being forwarded at a controlled (shaped) rate. The rate is controlled by the rate at which the tokens are placed in the bucket and the depth of the bucket. In this case, if the forwarder is unable to obtain a forwarding token it drops the packet to maintain the rate.

Figure 5-4. Traffic Shaping with a Token Bucket Model

[View full size image]

So, in certain scenarios it may be necessary to drop packets to maintain the flow. By controlling the timing and selection of packet discard, packets can be dropped fairly.

In its simplest form, discards work on the last-in, first-out principle. That is, packets are dropped from the tail of the congested queue when there is no queue space left. This is known as tail drop, and it is not the fairest way. A bursty source flow can overflow a queue resulting in the dropping of a well-behaved flow. The other thing to note about tail drop is that it is a reaction to an already overloaded or congested device. In a perfect world, you would look to avoid that congestion by detecting the buildup early and taking corrective action before any bottlenecks occur.

Congestion-avoidance techniques monitor network traffic loads in an effort to anticipate and avoid congestion at common network and internetwork bottlenecks before it becomes a problem. These techniques are designed to provide preferential treatment for premium-class (priority) traffic under congestion situations while concurrently maximizing network throughput and capacity utilization and minimizing packet loss and delay. WRED and DWRED are the Cisco IOS QoS congestion-avoidance features.

Random early detection (RED) is a congestion-avoidance mechanism that takes advantage of TCP's congestion-control mechanism. The RED algorithms are designed to avoid congestion in networks before it becomes a problem. RED works by monitoring traffic load at points in the network and stochastically discarding packets if the congestion begins to increase. The result of the drop is that the source detects the dropped traffic and slows its transmission. RED is primarily designed to work with TCP in IP Internet work environments.

Weighted RED (WRED) generally drops packets selectively based on IP precedence. Packets with a higher IP precedence are less likely to be dropped than packets with a lower precedence.

Distributed weighted random early detection (DWRED) is a high-speed version of WRED that runs on VIP-distributed processors. The DWRED algorithm provides functionality beyond what WRED provides, such as minimum and maximum queue depth thresholds and drop capabilities for each class of service.

Flow-based WRED is a feature of WRED that forces WRED to afford greater fairness to all flows on an interface in regard to how packets are dropped.

Drop schemes are best suited to traffic types that understand the implication of a dropped packet and act accordingly. TCP senders handle packet drops as a routine indication of congestion, and will slow down and resend dropped packets in response. UDP, on the other hand, will not necessarily slow down in response to congestion and will not resend dropped packets.

Therefore, you must understand your application characteristics to choose the best method to optimize application delivery.


Integrated Services

Integrated services (IntServ) is not actually a traffic-handling mechanism but rather a framework for defining services. The original Internet service model is very simple, and it does not provide any support for QoS or bandwidth sharing in the network.

IntServ, described in RFC 1633, has two services defined within this framework. These are the guaranteed service and the controlled load service, to support real-time and other performance-critical applications. The guaranteed service promises to carry a certain traffic volume with a quantifiable, bounded latency. The controlled load service agrees to carry a certain traffic volume with the appearance of a lightly loaded network.

IntServ services are usually associated with the RSVP signaling protocol. Each of the IntServ services define admission control algorithms, which determine how much traffic can be admitted to an IntServ service class at a particular network device, without compromising the QoS. IntServ services do not define the underlying queuing algorithms to be used in providing the service.

Traditional IP service provides the same QoS to every packet. IntServ expands this service model so that applications can choose an appropriate service.

Important elements of IntServ include the following:

  • Parameters for forwarding mechanisms that are appropriate for real-time information

  • A setup protocol that allows establishing special forwarding treatment for real-time information flows (RSVP)

  • A transport protocol for real-time information (RTP/RTCP)

  • Controlled link sharing, which addresses the sharing of link bandwidth (for example, during congestion)


Resource Reservation Protocol

RSVP is an IETF Internet standard (RFC 2205) network layer signaling protocol for allowing an application to dynamically reserve network bandwidth. RSVP enables applications to request a specific QoS for a data flow. Hosts and routers use RSVP to deliver QoS requests to the routers along the paths of the data stream and to maintain router and host state to provide the requested service, usually bandwidth and latency. RSVP uses a mean data rate, the largest amount of data that the router will keep in queue, and minimum QoS to determine bandwidth reservation.

WFQ or WRED act as the workhorse for RSVP, setting up the packet classification and scheduling required for the reserved flows. RSVP is used by the following:

  • Transmitting applications To describe their data traffic characteristics

  • Receiving applications To describe their QoS requirements

  • Routers To deliver QoS requests to other routers along the path of a flow

RSVP treats data flow from receiver to sender as logically independent from the flow from sender to receiver. Accordingly, a reservation for data from sender to receiver is independent from a reservation from receiver to sender. Because RSVP establishes a reservation for simplex flows, reservations for traffic can be made from any or both directions. RSVP is a hop-by-hop QoS signaling protocol. This means that RSVP messages are transmitted from one node to another through all RSVP-aware nodes along the data path. Figure 5-5 shows the reservation setup mechanism.

Figure 5-5. RSVP Reservation Process

[View full size image]

The following steps show the event sequence for RSVP resource reservation in a unicast scenario:

1.
The transmitting sender host usually knows the characteristics of the traffic it generates, such as data rate and the deviation from data rate. As it transmits data, the sender's RSVP module periodically sends RSVP path messages, which do the following:

- Describe the traffic generated by the sender

- Create a state on each intermediate RSVP-aware node along the data path

Path messages are sent with the destination address of the receiver host and are routed as data sent to that host.

2.
Path messages create a path state in each router that is traversed. Through this path setup mechanism, all devices along the path become aware of their adjacent RSVP nodes for data flow.

3.
When a local RSVP module notifies a receiver host application that an RSVP path message has been received, the receiver host application decides whether resources should be reserved.

4.
When a decision is made to request network resource reservation, the host application sends a request to the local RSVP module to assist in the reservation setup.

5.
The RSVP protocol then carries the request as Resv messages to all nodes along the reverse data path to the data source. The reservation is made on a hop-by-hop basis, and each intermediate node checks for sufficient resources and decides whether the request can be granted. If the reservation is successful, a Resv state is created, and the reservation request is forwarded to the previous hop in the data path.

6.
The receiver can request a notification about the reservation status. In this case, when the sender receives the Resv message from the receiver, it sends the receiver a Resv confirmation message.

7.
If the receiver sends any data, it will start sending path messages to the sender. In this case, Steps 1 to 6 are repeated with the receiver acting as the sender and the sender acting as the receiver.

Fundamental RSVP concepts include the following:

  • Specific quantity of resources Reserved and available to each traffic flow.

  • Soft state RSVP state times out automatically unless it is periodically refreshed.

  • Receiver orientation Receivers control the amount of resources reserved on their behalf.

  • Scalable multicast support RSVP was designed with this in mind.

  • Sessions Basic identifiers by which traffic flows and reservations are correlated.

  • Policy and security Support for these is built in to RSVP via the inclusion of objects that identify and authenticate users and applications.


Differentiated Services

DiffServ is a Layer 3 QoS mechanism. DiffServ is a complement to the IntServ model. It does not require end-to-end signaling. It classifies packets to one or a small number of aggregated flows based on the setting of a bit in the Type of Service (ToS) field of the IP header, called the DiffServ codepoint (DSCP). (DSCP is 6 bits wide, allowing coding for up to 64 different forwarding behaviors. The DSCP field was originally the IP precedence bit and is still backward compatible.) Typically, hosts or routers sending traffic into a DiffServ network mark each transmitted packet with the appropriate DSCP. Routers within the DiffServ network use the DSCP to classify packets and apply specific queuing or scheduling behavior (known as a per-hop behavior or PHB) based on the results of the classification.

DiffServ is rule based. Therefore, it is a unique mechanism for policy-based network management. Networks can be managed by appropriate network policies, applying current network technology and resources while considering in-house traffic and upstream and downstream networks.

The DiffServ framework offers the same kind of classification system. Based on network policies, different kinds of traffic can be marked for different kinds of forwarding. Resources can then be allocated according to the marking and the policies.

A set of contiguous DiffServ-compliant networks containing DiffServ-compliant nodes is a DiffServ domain. Service agreements between networks define service level commitments, traffic flow constraints, and internetwork DSCP marking rules. At the different network boundaries in the domain, ingress and egress routers ensure inbound and outbound packets are accordingly marked and that traffic complies with the service level agreements (SLAs).

Each network's marking rules determine the DSCP to be applied to each packet. Packet marking might be done by a DiffServ-compliant network device, by a traffic classifier entity in the network, or at the first-hop router in the network. In the DiffServ-compliant network and at boundary routers, forwarding would follow the DSCP prioritization rules. Routers and DiffServ-compliant Layer 3 switches may make delay/drop decisions on lower-class packets to ensure delivery of higher-class packets and, if available, may select faster, higher-quality routes for the highest classifications. Marking entities can change the DSCP marking as packets enter or leave the network based on the upstream and downstream marking rules defined in the SLA, ensuring packet-marking integrity within the network.

As mentioned previously, DiffServ is designed to work together with IntServ networks. DiffServ includes services that can both satisfy quantitative performance requirements (such as peak bandwidth) and relative performance requirements (such as class differentiation).

Services can be constructed by a combination of the following:

  • Setting bits in an IP header field at network boundaries (autonomous system boundaries, internal administrative boundaries, or hosts)

  • Using those bits to determine how packets are forwarded by the nodes inside the network

  • Conditioning the marked packets at network boundaries in accordance with the requirements or rules of each service


Summary

Network QoS involves the capability to control traffic-handling mechanisms in the network to serve the requirements of the applications in accordance with the business policies. These traffic-handling mechanisms are controlled by a combination of long-term provisioning and short-term configuration. Pushing provisioning or configuration information from a management console to the network equipment exerts this control.

QoS provides differentiated services, which provide higher priority to flows, or guaranteed services that provide an assured service level. Both of these are in contrast to best-effort service, which is provided by what is generally considered a lack of QoS. FIFO provides best-effort service. In such, flows are not differentiated and are serviced on a first-come, first-served basis.

Guaranteed services are generally provided by RSVP, although CBWFQ could be considered a form of guaranteed service. RSVP is a signaling protocol that signals the network to provide guaranteed services for the entire path of the packet. CBWFQ differs in that it guarantees service onto an interface.


End Notes

  1. Microsoft, A Short Overview of QoS Mechanisms and Their Interoperation, http://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/featusability/qosmech.mspx

  2. Microsoft, A Short Overview of QoS Mechanisms and Their Interoperation, http://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/featusability/qosmech.mspx

  3. Black, U., MPLS and Label Switching Networks, (c)2001, pp.1,2. Reprinted by permission of Pearson Education, Inc., Upper Saddle River, New Jersey

  4. Fineberg, Victoria. QoS Support in MPLS Networks, http://www.webtorials.com/abstracts/QoS%20Support%20in%20MPLS%20Networks.htm

  5. Converge Network Digest, MPLS Traffic Engineering tutorial, http://www.convergedigest.com/tutorials/mpls5/page2.asp

  6. Iljitsch van Beijnum, BGP: Building Reliable Networks with the Border Gateway Protocol, Copyright © September 2002 O'Reilly, Inc. Reprinted with permission of O'Reilly, Inc.

  7. Cisco Systems, Implementing Quality of Service, http://www.cisco.com/warp/public/105/wantqos.html


References Used in This Chapter

  • Cisco Systems, QoS (Quality of Service), http://www.cisco.com/en/US/tech/tk543/tech_topology_and_network_serv_and_protocol_suite_home.html

  • Cisco Systems, QoS Congestion Avoidance, http://www.cisco.com/en/US/tech/tk543/tk760/tech_protocol_family_home.html

  • Cisco Systems, QoS Congestion Management, http://www.cisco.com/en/US/tech/tk543/tk544/tech_protocol_family_home.html

  • Cisco Systems, Quality of Service Networking, http://www.cisco.com/univercd/cc/td/doc/cisintwk/ito_doc/qos.htm

  • Microsoft, Quality of Service Technical Overview, http://www.microsoft.com/windows2000/techinfo/howitworks/communications/trafficmgmt/qos.asp

  • Microsoft, Quality of Service Technical White Paper, http://www.microsoft.com/windows2000/techinfo/howitworks/communications/trafficmgmt/QoSOver.asp


Part III: Deploying and Monitoring Network Applications

Chapter 6 Application Deployment

Chapter 7 Beyond the Boundaries

Chapter 8 Monitoring the Delivery


Chapter 6. Application Deployment

This chapter includes the following topics:

  • Baselining your infrastructure

  • Allocating quality of service resources

  • The pilot

  • Verification process

  • Predictive modeling

  • Big bang vs. staged rollout

  • Building a contingency plan

  • Reporting

This chapter covers how to build up an accurate baseline using Cisco management tools and background data. It then discusses how to use this baseline to ascertain an application's impact on the environment at rollout and at application and infrastructure upgrades. Moving on, it discusses the allocation and deployment of quality of service (QoS) settings.

This chapter builds on what you learned in Chapter 3, "Detailing the Business Transaction," about how to optimize and profile the application. Specifically, this chapter covers the methodology and issues germane to rolling out that application. It discusses aspects such as staging a pilot program to check out work practices and verifying that a full rollout will not adversely affect your environment. The chapter then discusses steps that you should take if you incur problems, and how to report and quantify details of a successful rollout. The chapter then examines the importance of including a disaster and contingency plan as part of a deployment strategy.


Baselining Your Infrastructure

Networks are built by interconnecting various devices (such as routers and switches) running a variety of protocols. Protocols are configured with a number of parameters that enable them to behave in a certain way. These protocol configurations influence the traffic patterns in a network. Understanding these patterns from a unified baseline view is the first step in a successful application deployment.

The collection of baseline data is very important to the overall application deployment. The accuracy and pertinence of your baseline will dramatically affect the deployment project. Baselining is a technique used to characterize the existing IT infrastructure prior to application rollout and, subsequently, to track the application's performance.

Prior to application deployment, you must characterize existing network paths, servers, and databases that new applications will share. When these applications have their own dedicated servers, baselining of the servers' performance is the primary requirement. When the new application will reside on existing servers, however, the shared servers should also be baselined relative to their existing applications.

For the baselining effort to yield meaningful results, you must do the following:

  • Collect just the data that is pertinent to the application deployment For example, ignore nonessential or nondependent functions such as name resolution.

  • Collect the data at an acceptable frequency Network and application usage tends to be bursty. If you collect data points too infrequently, the bursts of activity are averaged out.

  • Collect the data for an acceptably long period of time An acceptable period of time takes into account items such as month end. A typical period to cover normal aspects of an enterprise activity is 1 month.

  • Determine the tolerances to design for Do you want to guarantee that a user never experiences degradation? Or do you allow for some degradation in the interest of cost?

Baselining is key to a successful strategy and forms the basis for pilot and contingency planning. It provides a constant reference point in the verification process and is used in an ongoing manner to assess the success of deployment against desired results.


Allocating Quality of Service Resources

Traditional network devices provide resources to network traffic on a first-come, first-served basis. Devices that provide QoS support do so by intelligently allocating resources to certain traffic (as discussed in Chapter 5, "QoS and MPLS: Tools to Manage Application Performance"). To treat traffic preferentially, you must identify traffic and associate it with certain resources. This section discusses the processes and reasoning behind this allocation.

To facilitate the accurate delivery of applications, various QoS features and functions are introduced to the environment. Simplistically, QoS can be described as a measure of performance for a data transmission system that reflects its transmission quality and service availability. To realize QoS, you must implement various protocols that provide an expected delivery profile to meet the application's delivery criteria relevant to the path and infrastructure (for example, a move away from best effort to the predictable delivery the applications demand).

The specific QoS requirements for given applications are generally determined by three parameters: latency, jitter, and loss. Depending on the type of application to be implemented over a Cisco network infrastructure, the importance and effects of these considerations can differ.

Latency is the amount of delay that affects all types of communication links. Delay on networks is usually measured in milliseconds (ms). Whereas delay is the time it takes to get a unit of information from source to destination through a network, jitter is the variance of delay.

For example, although one-way latency for voice or videoconferencing should be no more than 150 to 200 ms, streaming video can tolerate up to 4 to 5 seconds of latency. Similarly, jitter is not an issue for streaming video, but it should be limited to 30 ms for voice and videoconferencing applications.

Bandwidth effects will also differ. For example, to support videoconferencing, the minimum bandwidth guarantee is the size of the videoconferencing session plus 20 percent.

Having gone through the process of identifying traffic and applying delivery criteria and relative queuing algorithms as discussed in Chapter 5, the next task is to assign these algorithms and polices to the conversation flows. You can do so manually by predetermining the conversation flow and relative characteristics, and then pre-assigning the allocation of resources to meet the demand.

However, even network management teams that have well-planned QoS policies find that implementing and enforcing them throughout a large enterprise network is a daunting, if not overwhelming, task. Even with advanced, intelligent network devices and the latest management tools, the fact remains that in today's networks, the configuration is often very complex. This configuration is often done by hand, through a command-line interface (CLI), one device at a time.

Few network administrators have the time or experience to correctly implement end-to-end QoS policy in every device. This shortfall often results in inconsistent policy implementation and a lack of dynamic application control. Clearly, there is a need for a simpler, higher-level way to implement policy without requiring a detailed understanding of the QoS mechanisms and how they interoperate.

For successful deployment of a robust QoS framework, consider a QoS policy system. For example, Cisco QoS Policy Manager (QPM) provides a framework for automated policy deployment. QPM delivers differentiated services across network infrastructures for converged voice, video, and data applications. Network administrators can use QPM as a complete system for centralized QoS monitoring, policy control, and automated policy deployment.

In its simplest form, a QoS framework can enforce a delivery policy based on a combination of the application characteristics and current network conditions.

A policy is a set of predefined configuration data that specifies the condition of an application transfer. The policy will be in place and triggered by a match (or even mismatch) with the criteria. If you want to give the CEO's data transfer high priority during office hours, for example, you first need to reference the central directory to ascertain the CEO's logon information, address range, and so forth. Then you choose one of the following two possibilities:

  • You could just set the QoS settings for the perceived transmission of the CEO's data.

  • You could track the CEO's information flow and see whether in fact you need to tweak the infrastructure to guarantee the response the CEO expects.

Understanding Policy Management Architecture

This section further explores QoS resource management by illustrating policy management architecture and associated policy domains.

The primary policy management architectural components are the policy decision point (PDP), policy enforcement points (PEPs), and the policy repository, which is typically stored in a directory.

The purpose of the Common Open Policy Service (COPS) protocol is to exchange policy information between a PDP and its clients, PEPs. A PEP is a router or other device that handles IP traffic and implements policy-based admission control for data flows. The PDP, which has a view over the whole network area through its PEPs, decides whether to admit a specific data flow to enter. There is at least one PDP per domain. The PDP makes its decisions to admit a flow based on information held on a number of authentication and directory systems. Communications with these systems is achieved using the Lightweight Directory Access Protocol (LDAP).

The protocol employs a client/server model, where the PDP is the server and the PEPs are the clients. It uses a persistent TCP connection as its transport, thus there is no need for reliability mechanisms in the protocol itself. To achieve a high level of security, the messages can be signed with a key and a secure hash function. Alternatively, IPSec can be used to externally cipher the IP packets carrying COPS information.[1]

COPS is in its basic incarnation a very simple protocol, designed to be easy to extend and to provide a wide range of functionality. A PEP sends messages in the form of requests, updates, and deletes to the PDP, which replies with decisions.

To make its decisions, the PDP contacts the repository for the rules and conditions established by the network manager and decides, based on current network conditions, how traffic and access rights should be enforced.

After the PDP has made a decision about how to treat network traffic, it communicates this behavior to the PEPs (routers, switches, and gateways) via COPS. Each PEP has separate network interfaces, enabling network managers to set a fine granularity of policy management rules or to create batches of rules that apply to particular applications, users, or locations. Figure 6-1 shows the logical flow.

Figure 6-1. Policy Enforcement and Decision Architecture

The separation of the PEP from the PDP is a logical one based on functionality; it is not necessarily a physical separation. In other words, the separation of PEP and PDP does not preclude the possibility of having them built in to the same device.

A network can consist of multiple distributed PEPs. This will speed up the implementation of the policy. A policy-enabled network is likely to need fewer PDPs.

Three main functions are assigned to the architecture: decision making, enforcement, and metering.

Decision making is the function of the PDP and involves the following steps:

1. | Retrieving policy
2. | Interpreting policy
3. | Detecting policy conflicts
4. | Making policy decision requests
5. | Obtaining policy elements (conditions) from PEPs
6. | Determining which policy is relevant
7. | Applying policy
8. | Returning results (possibly with other policy elements, such as error information)

It also involves sending policy elements to the PEP asynchronously, based on policy updates or requests from external entities. The COPS policy-provisioning client message was created for this purpose.

Enforcement involves the PEP applying actions according to PDP decisions, relevant policies, and current network conditions.

Metering is an ongoing active or passive examination of the network and its constituent devices, checking network health, whether policies are being satisfied, and whether clients are taking unfair advantage of network services.

Decision making by PDP uses static and/or dynamic data to determine whether a type of policy is being satisfied, and if not, what steps are needed to satisfy that policy. Metering by PEP is the auditing of policy compliance to verify that the policy users properly implement policies.

The policy architecture is built on top of this framework, and will determine where and how the policy is implemented to the network elements. It is important to understand how policy information will be passed on to the elements.

The following two main components pass information to the network elements:

  • Resource Reservation Protocol (RSVP) Passes information from the PDP to the network elements capable of receiving this protocol

  • Simple Network Management Protocol (SNMP) Sets commands used to assign configuration to network elements

It is important to ensure that the infrastructure components do not receive too many changes. Although you want the system to be dynamic in adjusting to the needs of the user population, you need to ensure that it is not to the detriment of the infrastructure components. Constantly downloading changes to the system configuration will induce latency to the overall system throughput. If the infrastructure constantly has to review and change its associated tables and queuing information, packets will be delayed. This has a domino effect because application delay errors are reported back to the PDP and then the PEP is requested to make more changes.

Sharing Responsibility with Policy Domains

Today's networks are heterogeneous, incorporating a variety of technologies and devices to meet specific requirements. Therefore, applying end-to-end QoS requires signaling that is transparent to technologies and devices. Devices share the responsibility of delivering priority traffic by dividing up the QoS tasks. Administrators can define QoS policy domains within the network to indicate which devices take up which tasks. The most common instance of a policy domain is a demarcation between a LAN and a WAN. The device (usually a router or access server) that links the LAN and WAN becomes a QoS policy domain edge device, whereas those within the LAN and WAN backbones are core devices, as shown in Figure 6-2.

Figure 6-2. Policy Domains and Device Classification

[View full size image]

QoS edge devices are responsible for the following activities to monitor, recognize, and classify traffic:

  • Admission control Permits/denies application traffic

  • Classification Sets a priority level (high/medium/low)

  • Proxy signaling Signals IP precedence/differentiated services or an IP RSVP bandwidth allocation request along the delivery path

  • Policing Verifies that only the allocated bandwidth is used

  • Traffic shaping Manages reactive congestion

  • Network-based application recognition Recognizes application

QoS core devices are responsible for the following to expedite transport of high-priority traffic without congestion or delay:

  • Scheduling

  • Congestion avoidance

  • QoS signaling propagation


The Pilot

In most large-scale deployments, the enterprise will run a pilot to assess the suitability and other criteria of an application deployment. The process of distributing and installing an application throughout an entire organization is not the same as a pilot, which is where you provide the application to a smaller group of users to identify and evaluate problems that might occur during the actual full-scale deployment.

The primary purposes of a pilot are to demonstrate that your design works in the production environment as you expected and that it meets your organization's business requirements. Conducting a pilot reduces your organization's risk of encountering problems during full-scale deployment. To further minimize your risk during deployment, you might want to conduct multiple pilots consisting of separate pilots for different transmission technologies or operating systems, or you might want to conduct a full-scale pilot in phases.

The pilot participants provide feedback about how well the application is performing, and whether it is meeting their expectations and requirements. The deployment management team uses this feedback to resolve issues that arise or to create a contingency plan. Ultimately, the pilot leads to a decision to proceed with a full deployment or to slow down so that you can resolve problems that could jeopardize your deployment.

Figure 6-3 is a flowchart illustrating the steps involved to successfully develop and implement a pilot plan.

Figure 6-3. Pilot Program Flowchart

[View full size image]

The pilot process is iterative. You deploy the application on a limited number of servers in a controlled environment, evaluate the results, fix problems, and deploy an updated release (including network infrastructure requirements and configurations) of your pilot, repeating this process until you are satisfied that you are ready for a full deployment.

The first step in planning your pilot is to define what you want to accomplish (the objectives or goals) and what you plan to include and exclude (the scope). Be sure to align the pilot objectives and scope with those for the deployment project as a whole, as defined in the master project plan. Ensure that the pilot plan includes an opportunity for the team to evaluate features identified in the project plan as high priority, to ascertain that they successfully meet all of your business objectives.

Define the scope of the pilot by clearly stating which services and features will be included and which will not. When you list the transactions and features you plan to include in the pilot, also state how you expect them to perform. Describe the areas of functionality that the pilot implementation affects, and note to what extent they are affected, and in which situations they are affected. Avoid the mistake of making pilot testing too easy. Test as many situations that might arise in production as you possibly can.

Do not expect to test every feature or service during the pilot. Focus on processes that present the greatest risk and events that are most likely to occur. Prioritizing the transactions to be tested in the pilot is particularly important if your team plans to conduct multiple pilots. When multiple pilots are planned, start small and gradually increase the scope of successive pilots.

If the pilot cannot cover certain aspects of your design, describe them. For example, if your organization plans to use satellite technology to reach some of the branch offices, you may want to simulate satellite transmission characteristics as opposed to using real satellite circuits in the first stage of the pilot. Then, having refined the application to meet the characteristics, you perform a pilot over real circuits.

Also, specify the duration of the pilot, in terms of either time or of the criteria to be met. Pilot plans include a number of documents that provide procedures for successfully rolling out the pilot in a business environment. Your pilot plan should include the following documents:

  • Training Describes what the pilot participants need to know before they begin the pilot and how you plan to train them

  • Support Describes how problems that arise for participants during the pilot will be resolved

  • Communication Explains how you will keep participants informed about what is happening in the project

  • Evaluation Describes how you plan to obtain feedback from participants

The feedback can be used to assess design changes that might be required before the pilot is rolled out to the rest of the organization. You should also include the following:

  • Risk and contingency Describes potential risk factors, including how you plan to assess and diffuse them

  • Backup and recovery Describes how to back up data, system, and infrastructure configurations, how to test those backups, and how to roll back to previous configurations if problems arise that might affect business operations or participants' access to data

  • Schedule Includes milestones at which you plan to evaluate and make necessary changes to the pilot

The pilot plan you create helps to ensure the smooth deployment and completion of your pilot project by clarifying the tasks and roles of the release management team and the pilot participants.

When the pilot is complete, obtain feedback to evaluate the pilot's success from a variety of sources, including participants, pilot release management and support teams, and other observers.

Review the parts of the pilot that were successful and the types of problems that were reported so that you can revise and improve your plan. Determine whether the pilot met the success criteria you defined before the pilot began. If you established metrics to measure your success, evaluate how the pilot measured up. Use the feedback to evaluate whether the delivered design meets the design specification, as well as the business requirements.


Verification Process

This section examines how to verify, using a combination of live data collection as well as a virtual environment, that the application will actually meet the expected results.

The default form of verification is the user's response. The application is the key business driver, and as a consequence the end user of that application is critical to the business model. The users of the application are best placed to tell you whether the application performs as they expect. However, these comments come after the fact. The application has already been deployed, and as a result any discrepancy in the delivery expectation places you in a reactive position.

Providing enough information in a predeployment scenario puts you in a proactive position to effectively verify your expected performance delivery. There will never be a 100-percent certainty that your predeployment verification will mimic the production environment, but it can be very accurate. Ultimately, however, you receive true verification from the users.

The methods of predeployment verification depend on your rollout strategy. For example, you could choose to run a pilot program that is comprised of a small section of real users. The feedback from these users would provide good data for use in predicting the performance after the application is fully deployed. To make this truly effective, you must take information from the different geographical and transmission types, or at least use the data in your predictive model (see the section "Predictive Modeling" for details). The verification process should form part of your predeployment testing strategy.

A successful predeployment testing strategy is an integral part of the pilot process. You must follow these steps:

Step 1. | Establish your goal.
Step 2. | Verify that all functions of the application work as expected.
Step 3. | Verify that all functions work correctly under stress.
Step 4. | Verify the delivery systems ability to cope with long-term constant usage.

Initially, you need to establish your goal, such as what are you looking to verify? What functions? And what results are you expecting? This data will already have been defined in your pilot plan and will form the evaluation part of the methodology. (Refer to Figure 6-3.)

You can collect this data, using data collection techniques (see Chapter 8, "Monitoring the Delivery," for details) from the pilot. This data may on its own sufficiently satisfy your requirements verification goal. It is more likely that this information is just a baseline for use in your virtual environment (see this chapter's "Predictive Modeling" section for details).

Then you need to verify that all the functions of the applications work as expected. This includes verifying response times relative to expectations.

The next stage is to verify that all the functions operate correctly under the following two different types of stress:

  • Load Loading the systems, both servers and network

  • Fault condition Introducing failures into the system or severely loading parts of the delivery infrastructure

Although these tests could be performed at the same time, you should carry them out independently, so as not to confuse the results.

Stress testing can be undertaken by carrying out destruction testing on a live/pilot system. Destruction testing is a test in which various loads and fault conditions are artificially placed on a delivery system to stress it to the breaking point. Although this can be carried out to a degree on a pilot system, you will only see a true result by performing the destruction test on your production environment, putting you in a reactive position. However, this is an ideal situation in which to use your virtual model (see this chapter's "Predictive Modeling" section) to provide a virtual replica of your production environment, where you will be able to proactively test multiple scenarios to ascertain the predictive results under different fault situations.

The last test should cover the long-term stability and will verify the delivery system's capability to cope with long-term constant usage. It is recommended that this test go beyond the scope of your expected daily-usage plans. As before, you can run this test on either the physical pilot environment or, preferably, on your virtual environment. The virtual environment is preferable in this context because it enables you to run a simulation independent of the pilot or production system. This is important because during the final stages of the predeployment testing, you may have cause to test various parts of the delivery system; this would obviously have an impact on the long-term stability test. By providing the verification in the virtual environment, you can build up a detailed understanding of the system over a simulated period of time.


Predictive Modeling

One of the most effective ways to observe how these applications will manifest themselves in a production environment is to take a virtual model, which accurately reflects the production environment (including infrastructure architecture, background applications, traffic, and so forth), and import the application characteristics in the form of an application trace into that model. Combining predictive simulation and a comprehensive understanding of network technologies, intelligent network management software reduces the considerable cost, labor, and risk involved in the management of IT resources, deployment of applications, and delivery of communications services.

There are several steps to the creation of a virtual environment, as follows:

Step 1. | Define goals.
Step 2. | Consider the underlying infrastructure.
Step 3. | Overlay network traffic on top of infrastructure.
Step 4. | Refer back to the goal of the simulation to decide which traffic flows should be modeled and which can be safely ignored.
Step 5. | Validate and analyze application response times.
Step 6. | Experiment with what-if scenarios.

The first step in the creation of the virtual environment is to define goals, understanding what it is you are trying to achieve. As with the pilot phase, you should have a clearly defined objective before embarking on the model-building process.

Although not out of the question, you would not initiate a model by building a whole network. You should just define and build the parts of your enterprise that will be affected by an application deployment or upgrade. By taking this divided approach, you can provide quicker, more defined results than would be immediately possible by trying to create a large model.

If the objective of the study is to analyze a client/server application, for example, it is necessary only to identify the network devices and path constituting the single connection between the client and the server. The effect of the remaining portions of the network will be taken into account by representing the traffic that crosses, and therefore affects, the path of interest. The advantage of this topology is that it limits the focus of the study to the main objects of interest: the client and the server. Because other devices are not explicitly modeled, the simulation is very efficient. There is no loss of accuracy, however, because the effect of the remaining network on the client/server traffic is taken into account.

Having defined the boundaries of your network, you must consider the underlying infrastructure. To build the network topology, consider all sources of information available to you for your network. These may include router configuration files, device Management Information Bases (MIBs), network management platforms, traffic management systems, spreadsheets, and proprietary databases. To gather this baseline information, you should leverage discovery provided by network management tools. The more complete your view, the more accurate your model will be.

Having built your base network, you need to overlay the network traffic to complete your model. Characterization of network traffic plays a crucial role in simulation. Traffic information can potentially affect results and conclusions drawn from the model.

Traffic can be described in two forms: background and explicit. Background traffic is the traffic that, although involved in the exchange, is not directly implicated. It is just occupying the same infrastructure, and in doing so may impact the application's performance. It is important to include the background traffic to be able to build a complete picture of what, who, and when performance will be affected.

Explicit traffic refers to the exchange of conversation data directly pertinent to your study. This includes all dependent applications and tiers in the process. Whereas background traffic can just report end-to-end flows, including source, destination, and application type information, explicit traffic is required to include more application information such as the client processing time, server processing time, client request sizes, and server response sizes.

Background traffic can be obtained using tools, such as Cisco NetFlow, that capture traffic flows between source-destination pairs.

Explicit traffic requires protocol analyzers (such as Network Associates Sniffer) that can capture all packets in the network and provide a line-by-line decode of each packet.

Traffic in a network has complex characteristics. An important step in modeling traffic is understanding and defining traffic flows. On a given network, it is useful to classify the different types of traffic, determine the direction of traffic flow, and take measurements. The following questions may assist you in understanding your network traffic:

  • What are the primary applications that generate traffic?

  • Can you classify traffic by protocol (TCP, UDP, IP, and so forth)?

  • Where are the servers located on the network?

  • What is the general direction of traffic (toward the server, local to LANs, crossing the backbone, and so forth)?

  • Is there bidirectional conversational traffic (HTTP downloads, FTP transfers, and so forth)?

  • Are the applications multitier in nature?

  • Is there any unaccounted traffic (nightly backups, e-mail replication, and so forth)?

After you have a good understanding of network traffic, it is important to refer back to the goal of the simulation and decide which traffic flows should be modeled and which can be safely ignored. Continuing with the original example, if the objective of the predictive study is to analyze the performance of a client/server application, the flows along the path of this application are important and need to be captured accurately. In addition, the client may be located on a LAN segment and may be contending for medium access with many users. The effect of this contention needs to be considered.

For a backbone, it is important to capture all flows going across the backbone only. You can ignore local LAN segment traffic flows that do not traverse the backbone can. Before importing traffic to the model, identify all possible flows in the network and list the ones that are important and therefore should be modeled.

After validating and analyzing the application response times, you can perform a wide range of what-if scenarios by changing attributes, network configurations, and any of the model's inputs. Simulation gives you the power to swap protocols, technologies, servers, and client attributes. Moreover, simulation enables you to experiment without having to disrupt your production network.


Big Bang vs. Staged Rollout

There are many schools of thought regarding application deployment. Some favor long-scale phased deployment; other solutions require an overnight transformation. This section discusses the pros and cons of each approach (see Table 6-1), including the necessity of each particular approach.

Table 6-1. Comparing Staged to Big Bang Deployment

Long-Scale, Staged Deployment

Overnight, Big Bang Deployment

Small groups of users converted at a time.

The application deployed as one; all users simultaneously cut over to the new system.

Minimal disruption to the production environment.

Large-scale disruption to production environment.

Any performance issues that occur can be quickly isolated and rectified without any large-scale disruption to the wider user population.

Performance issues may result in large-scale disruptions to user population.

Allows for a staggered training, meaning that you potentially have fewer users out of the workforce at any single time.

Requires mass training, resulting in large numbers out of the workforce at a time.

Potentially a need to keep two systems running, often incurring additional bandwidth and system requirements.

Only single system running at one time, so no additional bandwidth or system requirements.

Additional cost of having a help desk support two separate groups of users, including potential ongoing maintenance costs for maintaining two separate systems.

There is no need to support two concurrent systems.

Issue of data correlation, where there is a need to cross-reference data across two systems, may add unnecessary complexity into the system.

No need to implement temporary infrastructure and systems to facilitate support and data correlation between two relatively disparate systems.

More focused planning, allowing for variations to be implemented more easily and with more speed.

Requires comprehensive planning, a lot of which will have been ironed out during the pilot phase (see the section "The Pilot" for details) and coordinated in the contingency-planning process (see the section "Building a Contingency Plan" for details).

The method of deployment will depend on the size of the project, and there is no one-size-fits-all approach. However, specific drivers may affect deployments. For example, the year 2000 issue forced many deployments to be carried out with an unmovable deadline, which in many cases meant that the big bang was the most effective and timely way to achieve the desired result.

In this situation, it is imperative that the contingency plan is very expansive, because you want to minimize any disruption to your business model.

Other projects, such as the deployment of a brand new application (such as a new version of Microsoft Office), may, by design, be deployed to a small subset of users before making an assessment of the relevance for a full-scale rollout. In this situation, the staged approach is more than adequate, although the contingency plan is still important and can actually be less comprehensive.


Building a Contingency Plan

Part of every deployment plan should include a contingency to cope with the potential that everything may not go entirely to plan. This section discusses what a contingency plan should include and examines how you can modify this plan to have a useful disaster recovery tool.

The scope of deployment problems that may result stretches from the application not meeting the expected performance levels through to the application having an adverse effect on existing production applications or infrastructure.

In each case, the organization must be prepared to begin a series of actions to rectify the situation. In an extreme case, this means returning a system to its original status and backing out of the planned deployment. This is where the baseline data collected earlier becomes of paramount importance, because it allows the system to be returned to a known working state.

The contingency plan is comprised of the following criteria:

  • Technical Deals with the specifics of the application-delivery model and includes physical configuration and build of the delivery system

  • Business Ultimately controls the timing of any contingency plan because it contains information pertinent to business model impact

Acquiring the Technical Criteria

The more complete your technical baseline is, the better you will understand your delivery system (and its associated weaknesses).

Part of the contingency plan should include vendor specification coupled with expectations of performance. This should be made up of test data, or at the very least vendor lab benchmarking. Benchmarking refers to vendor testing in a controlled environment. You need to consider that the software vendor may have delivery criteria that needs to be adhered to in order to guarantee their expected performance. This may include the following:

  • Server sizing (such as processor speed, number of CPUs, memory, and so forth)

  • Bandwidth to desktop (for example, 10 Mbps to desktop)

These requirements need to be built in to the contingency plan (in case your environment does not meet them). You must carry out some form of predeployment testing, preferably in a virtual environment, to build a definitive list of requirements and expectations. Doing so enables you to understand any potential bottlenecks before you actually implement the contingency plan.

Collection of all this baseline data also enables you to build in the various aspects of the contingency plan. For example, by understanding how an application will manifest itself in your environment, you will be in a better position to understand the potential bottlenecks and be able to test workarounds in this virtual environment.

Therefore, the technical criteria needs to include the following items, in detail:

  • Expected performance of transactions to X users (where X is the number of users in pilot or modeling testing)

  • Required performance of transactions to Y users (where Y is the expected number of deployed users)

  • Actual performance of existing business-critical transactions

  • Software vendor infrastructure specifications (including version number and required patches)

  • Transmission infrastructure (including configuration and model numbers)

You use this data when implementing the workarounds to either improve performance or back out of the deployment. Note that the technical aspect of the contingency plan includes more baseline data than actual workarounds. If the application has gone through predeployment testing, any performance issue will be somewhat of a surprise. It is then that you use this baseline data to ascertain whether there are any noticeable differences in what is expected in the deployment (including configuration and software levels) and what has actually been deployed. From here, the data is used to build up a series of workarounds.

Deciding on the Business Criteria

The application drives the business bottom line. Just as a poorly performing application causes issues in production, a failed or poorly deployed rollout can equally debilitate the business. With that in mind, an important step in building the contingency plan is to ensure that there is a business unit owner for each group involved in the planning process.

The key contacts for each business unit should collectively form the company's contingency planning team and be responsible for creating plans, communicating within their units, planning maintenance, and training regarding when the contingency plans should be triggered.

This group should also transition into a rapid response team that deals with problems that arise before or after the deployment. The team should include representatives from all the disciplines in the enterprise, such as IS, facilities, and operations, as well as representatives from individual business units.

In many cases, employees rather than executives will enact contingency plans, so it is important that employees are informed about the plans that are in place and roles that they may be asked to play.

After the contingency planning team has been formed, it must first prioritize the business infrastructure with regard to importance to the business processes, including systems and key suppliers. To begin the process of contingency planning, companies must first decide, at the highest level, what areas of the company must be functional following an application deployment, and then create a plan that provides backup options for the systems, departments, people, and technologies within those areas.

When assessing the importance of a given system, companies also need to consider how long they can afford to have that system down. If downtime of more than 30 days will cause a significant slowdown in the business, the company needs a solid plan for coping with the problem and a trigger date clearly defining when the plan will kick in.

Often contingency planning involves the gross simplification of a process rather than a technical solutionfor example, the use of a manual system to record client information. Alternatively, sending data input to a different physical location for processing may meet the business requirements while a resolution is instigated.

Part of contingency planning is simply looking for ways to minimize the risk of failure. This is where predeployment testing and verification play an important part. Every contingency plan needs to include some cross-reference to the baseline and predeployment testing. This assists in expediting speed to resolution.

Just like collecting baseline data, building a contingency plan is also time well spent, because it has numerous uses and benefits throughout the delivery system. The framework and workarounds built in to your contingency plan will form the basis of your disaster recovery plan and should continue to form an important part of your delivery system.

Building a Disaster Recovery Plan

For a business, a disaster is defined as an event that interrupts business services to the point that an alternative service is required to prevent business loss. Disaster recovery is the process of creating, testing, and maintaining a business plan to recover from any disaster. A disaster can be anything from a 2-second power outage to a natural disaster such as a flood or earthquake.

Generally, disaster recovery comprises those procedures, policies, and products that allow for the timely resumption of the computer-based elements of an enterprise's business processes following a significant large-scale interruption in service. The fundamental assumption behind a disaster recovery plan is that the building where the computers reside is neither usable nor accessible. In this regard, disaster recovery is a particular discipline within the general category of remote-site recovery.

There are six fundamental phases in the development of a disaster recovery plan (DRP), as follows:

  • Justify costs.

  • Assess risks.

  • Select strategies.

  • Create procedures.

  • Test the procedures.

  • Keep pace with changes.

Cost justification will vary from organization to organization, but will be a major input in deciding how large a scale the DRP will encompass. Each organization must assess how much per hour it will cost if a system fails. This figure can then be weighed against the cost of implementing the DRP infrastructure.

In the risk-assessment phase, the business should identify the disasters that could potentially affect them, and then consider the implications of the resulting downtime. This is often called business-impact analysis.

There are basically two kinds of recovery strategies to choose from in the strategy-selection phase: redundant data center or host-site services.

With the redundant data center strategy, the customer duplicates its data center at a full or reduced scale in a different geographical location. The organization buys or leases all the recovery equipment, buildings, and networks and assumes all recovery responsibilities. This strategy makes sense if the organization has recovery planning expertise in-house and the data center is large.

The other strategy is the use of hot-site services offered by a third-party provider. This recovery alternative makes sense for companies with little or no recovery expertise and a small data center.

After the appropriate strategy has been chosen, the actual recovery procedure is created. The procedure should include instructions on how to deal with various aspects of the network such as loss of routers, communication links, and so on. Also, include a specification identifying who arranges for repairs or reconstruction and how the data-recovery process occurs.

It is critical that the recovery procedure is tested frequently. At a minimum, the procedure should be tested annually. The actual testing frequency is determined by many factors, including personnel changes and changes in the systems or network. It is also important that different people are involved in the testing so that a disaster does not take out the only recovery expertise in the company.

Keep pace with the changes. Do not assume that a DRP is set in stone. Your DRP should evolve to match the changes in your production environment (such as software version updates).

Investment and total commitment are required to implement an enterprise-wide plan. Contingency planning, and its successor DRP, is a necessity in today's rapidly expanding networked world. As IT becomes a more integral aspect of business, the financial and competitive advantages of a sophisticated recovery strategy become apparent.


Reporting

Regardless of what data is recorded or reported, it must be communicated in an audience-appropriate format. For example, executive management will be most concerned with how the deployment (good or bad) impacts the financial standing of the business. Operations and IT personnel will need detailed technical information so that they can understand the implications of the applications, ultimately providing them with baseline data enabling them to fix future problems. Reporting objectives are discussed in Chapter 8.


Summary

When deploying an application, you must build an accurate baseline pertinent to the systems and users affected by the application deployment. The baseline will help you understand the potential bottlenecks in the existing environment, and in doing so assist in defining where and when QoS resources should be allocated.

The baseline data forms the backbone for your pilot, contingency planning, and disaster planning. In addition, the pilot plan should include a number of documents that provide procedures for successfully rolling out the pilot in a business environment.

One of the most effective ways to observe how these applications will manifest themselves in a production environment is to simulate the application in a virtual model. Doing so enables you to assess the impact on the production environment and issues behind the application deployment before rollout.

In general, there are two approaches to application deployment: a big bang approach and a staged approach. The method of deployment depends on the size of the project, and there is no one-size-fits-all approach. However, specific drivers may affect deployments.

The success of an application deployment depends on the response from the actual users. Therefore, you need to define quantifiable objectives to measure the success of the deployment (as reported by the users). Remember to keep these objectives simple and in line with the business requirements. Good, clear communication is fundamental to gaining and retaining support for an application deployment.


End Note

  1. Benzekri, A., F. Barrère, F. Grasset, and Y. Raynaud. An Extended Enterprise Security Policy Management Architecture. http://www.hpovua.org/PUBLICATIONS/PROCEEDINGS/8_HPOVUAWS/Papers/Paper05.1-Benzekri-AnExtended.pdf


Reference Used in This Chapter

  • OPNET Technologies, Simulation Technology Options, http://www.opnet.com/support/home1.html


Chapter 7. Beyond the Boundaries

This chapter includes the following topics:

  • Business requirements overview

  • Assessing demarcation points

  • Recognizing your limits

  • Meeting service needs

  • Assigning the correct QoS system

You may only operate (own) part of your delivery mechanism, relying on third-party organizations to deliver your application to both your internal staff and your customers. This chapter discusses approaches to this situation, covering ways to work within these confinements.

It begins by discussing how and where to establish your demarcation points. This not only helps to define your management structure, it also expedites fault-condition resolution by having clearly defined areas of responsibility.

It then discusses using this demarcation point information to recognize when you have reached the limits of your responsibility in the continuing optimization and application performance-monitoring (APM) model.

Next, it discusses how to incorporate business criteria into the APM process and introduces the concept of business service management.

Considering the newly assigned demarcation points, the chapter revisits the concept of assigning the quality of service (QoS) mechanism learned in Chapter 5, "QoS and MPLS: Tools to Manage Application Performance," and Chapter 6, "Application Deployment."

Finally, it discusses the importance of reporting, ensuring that the information is aimed at the correct target audience, and that it is clearly presented for all parties across the delivery system.


Business Requirements Overview

As organizations increase their dependence on IT, business requirements become more critical and the adage "If you improve IT, you improve the business" can be applied. Business and IT alignment is critical to any IT department wanting to understand and report business contextual information for the delivered IT service, but it is often outside the context of traditional network management systems.

This situation is further complicated when you have to work within the confines of an outsourced networkcore parts of the delivery system are under the control of a third-party organization. This may be as simple as a carrier who is providing point-to-point circuits, or down to comprehensive outsourcing contracts that include, for example, all application servers and client desktops.

The trick is just to apply the same rules and processes on smaller sections of your environment. As discussed, your first task is to define your objective. In this case, your objective is to clearly identify where responsibilities begin and end for relevant parties. You have then identified key transition parts in your overall delivery system.


Assessing Demarcation Points

Having established the profile of the application, its path through the network, and the pattern it leaves behind, you can start to assign demarcation points for monitoring performance. These fall where the responsibilities of the customer and IT service provider (in-house or outsourced) start and stop.

Where you draw the lines depends on the following:

  • Business arrangement (such as outsourcer or internal service provider)

  • Application-delivery requirements

In an ideal world, no demarcation lines would exist. Application support would be delivered on an end-to-end basis, with the application treated as a whole, and a single entity would be responsible for every aspect of the delivery process, from the server through the network and subsequently on to the client and the desktop.

In the majority of cases, this is not achievable, because many different skill sets are involved in each step of the delivery process, and this may not be the most optimum method from a business perspective.

Figure 7-1 illustrates a simple network scenario, including a WAN, access circuits, and LAN equipment and services. From this schematic, you can start to set demarcation points for network responsibility.

Figure 7-1. A Simple Example of Demarcation Points

[View full size image]

Figure 7-1 shows an example of defining boundaries for the measurement of application delivery, making it possible to develop a clear set of procedures for performance deterioration.

The demarcation point represented by the inner circle (circle number 1) is that of the traditional telecommunication services, where responsibility ends at the edge of the network just as it enters the customer tail (access) service.

This demarcation point is useful from a carrier perspective, but has little relevance for end users, because they just see the network as a transmissions path.

In circle number 2, the telecommunication's responsibility has been taken a little further to cover all aspects of the WAN delivery up to the point of delivery at the customer premises.

Circle number 3 expands the responsibility and covers the point at which the LAN interface is presented to the customer site.

Finally, circle number 4 covers the client/server LAN area, concentrating on the delivery to the relevant hosts.

Each of these areas can be delivered as either an individual or a combination approach to the delivery model. For example, you may want to just purchase a link from a carrier. In that scenario, you simply apply the demarcation point at the end of the link, define metric collection at that point, and effectively monitor and report on your application delivery relevant to that endpoint. (For details, see Chapter 8, "Monitoring the Delivery.")

You may then want to assign more of the responsibilities to different departments within your own organization. For example, the network delivery team assumes responsibility for the delivery mechanism from where they pick up delivery from the carrier, to the point at which it is delivered to the interface of the internal switch. Responsibility then moves to the LAN or server administrators.

For example, Acme Electronics is an electronics retail firm, and they have recently moved to sell their goods online (in addition to their traditional storefront operations). Their business objectives are straightforward. They intend to migrate 80 percent of their sales revenue from their retail outlets to the web-based offering. In addition, they want to ensure that each customer will have to wait no longer than 8 seconds for online confirmation of an order.

Acme's network has recently been outsourced, and at the time they moved to the web service, they had a network carrier providing connection between their web server and the warehouse database system.

Acme's responsibilities include the web server, warehouse database system, associated LAN systems, and connection to the Internet. The carrier's responsibility includes the carriage of data between the web server and the warehouse database system.

After you have defined these areas of responsibility, you can start to assign equipment within each of these areas to the responsible parties. You may break an area down to whole equipment (such as a router or switch) or down to a specific interface. This demarcation is sometimes made easier by understanding who owns the equipment. In an outsourced agreement, for example, all network infrastructure may be supplied by the outsource company; therefore, demarcation points for that path or process clearly reside where the outsource equipment meets the customer equipment. It should be added that, in these scenarios, the cabling is quite often overlooked and can cause a contention if the fault is found to reside between the customer and outsourced equipment. Therefore, any agreement should clearly state cabling and interface conditions.

From Acme's perspective, this can be further defined, with the server administrators taking responsibility for the web server and database server (including software and hardware) and the network engineers taking responsibility for the LAN components of the network.

By taking your profile and reference point, and then matching it to your demarcation point, you are in a position to write the service level agreement (SLA), bearing in mind three basic rules:

  • Keep it simple If it is too complex, the SLA will never be agreed to or implemented.

  • Stick to business It is no good having a perfect SLA unrelated to the actual business objective.

  • Minimize overhead You are in business to make money, not administer an SLA.

When defining demarcation points, consider aligning with metric collection locations. This will be dealt with more in the "Defining Metrics" section of Chapter 8, but the concept should be considered at this point. The theory is as follows. A demarcation point is a strategic connection point in the conversation transmission. By observing specific conditions (metrics) at this point, you can gain a quick and inductive view of the application's performance. In a sense, you are looking for these demarcation points to become the natural aggregators of an application's performance issues.

One of Acme's business objectives is to have no user waiting for more than 4 seconds to receive confirmation of an order. For the order to be confirmed, the web server must receive notification from the warehouse database that the product is in stock. So in Acme's network, a strategic connection point is the warehouse-facing network connection on the web server. From here they can track the request from the web server to the warehouse database, and the response from the warehouse server to the web server.


Recognizing Your Limits

At some point, you will either reach the limit of your responsibility or maybe even the limit of your skill level.

At this point, you need to make a decision on the state of the application. Consider the following scenarios:

  • Scenario 1 You have met the business delivery requirements.

  • Scenario 2 You have optimized as far as you can given the infrastructure and application profile, but have not met the business delivery requirements.

In Scenario 1, you have achieved the objective of meeting the business delivery requirements. (In Acme's case, every order response from the warehouse server is less than 4 seconds.) You can consider the issue resolved. (As discussed, the optimization process is a constantly evolving process.)

In Scenario 2, you have effectively gone as far as you can go. You have applied all QoS strategies to improve the application architecture, but the application performance is still not meeting the business requirements. At this stage, you must take steps to escalate the situation further. How you do this depends on what level of optimization was achieved. You can determine this by studying the metrics collected at the relevant demarcation points.

Each metric reveals application performance status; this in turn indicates areas that need improvement, such as an increase in bandwidth or even server CPU speed. You can find details about how to collect and apply these metrics in Chapter 8, "Monitoring the Delivery," and in Chapter 10, "When Applications Fail."

In Acme's case, the response time received at the metric point reveals that the warehouse server is taking 7 seconds to respond. Investigation by Acme revealed that the requests are being received at the warehouse database in good time, but the database server is taking an excessive time to process the response.

Assuming these metrics do not bring up more areas for you to optimize, your decision must then be to escalate. The metrics will identify the area of responsibility (more specifically the third party or vendor) that needs to address some issues. By establishing this audience, you will be in a position to deliver a clear and concise message conveying the true impact to your application-delivery model, enabling the audience to make recommendations for improvement. You can then verify these using your virtual environment, as described in Chapter 2, "Understanding Your Business."

Acme was able to identify the most probable cause of concern to be the warehouse server architecture. This conclusion was reached by simple refection on their metric collection. The web server showed it was sending and processing the requests in time, and the speed across the network (when a response was received) showed no congestion, drops, or latency, so by elimination, the area of concern and responsibility was the warehouse database server. The database administrators confirmed that the database was optimally tuned, so the last area to address was the server architecture. This is a simplified example, but it helps to illustrate how the observations at specific demarcation points ultimately assist in identifying the most probable area of concern.


Meeting Service Needs

An accurately defined and monitored APM system can assist in the ultimate aim of effective delivery of the application, while providing sufficient data feedback into management tools that will ultimately assist in optimizing delivery.

Delivery criteria cannot be concerned solely with technical parameters, but must meet business objectives. The following three major factors differentiate an organization:

  • Product or service offerings

  • Customers

  • Delivery to market

Together these factors make up the organization's unique value proposition. Each organization's mission is to create and maintain the systems and processes to support its value proposition.

Business systems and processes operate under a combination of time, cost, and quality requirements that make it essential to use IT. IT reliability and flexibility are often the critical make-or-break issues behind business initiatives. Companies across a broad spectrum of the economy rely on IT to create competitive advantage and improve productivity, with business initiatives such as supplier and supply-chain management, business and logistics systems, customer relationship management, and e-commerce. IT supports data-driven decision making and synchronizes real-time activity between the managers, operators, and users of the business systems and processes.

To provide a better service to its customers and users, an enterprise needs to evaluate and manage technology at a business process level. This provides a business-oriented view of the enterprise's information systems, translating the traditional metrics into demonstrable measures that make sense to the company's senior executives.

Key business services supplied by an enterprise to its customers quite often rely on a number of business processes. For example, acceptance of a customer's product order will probably entail the delivery of product information to a company's website, the checking of stock availability, and the process of credit authorization. Each of these three processes will rely on combinations of multiple technology resources, including network devices, system platforms, applications, and databases.

For an organization to ensure its product ordering is functioning correctly, it has to collect and relate a large amount of availability, fault, status, and performance information that extends across the whole array of management systems. The information has to be put into context across the relevant business processes and then consolidated across business units and possibly companies.

IT organizations were traditionally managed based on the technical capabilities of underlying infrastructure such as networks, systems, storage, and applications. This silo-based approach to management, where each of the technologies is managed separately, evolved over time as various technologies were developed.

Today, most organizations have recognized that this approach is no longer viable. It fails to take into account the interrelated nature of individual components, and most importantly does not adequately address the business priorities of the firm. In response, business executives and IT leaders have begun to demand a more holistic approach for managing IT resources. This has led to the development of systems that take in the technical metrics and traps and assess them relevant to the unique business criteria defined by each organization, as discussed in Part II, "Aligning the Network's Business and Technical Requirements." This approach is known as business service management (BSM).

BSM is the logical progression of network and systems management and fits nicely into the emerging requirement to report business value. BSM brings into line IT services with the business processes they enable, and then manages these services in a way that is consistent with business priority. Many benefits result from this, including the following:

  • Investment in service delivery is appropriate to business benefit and overcapitalization is avoided.

  • Problem resolution is prioritized in line with business requirements.

  • Service-improvement programs are initiated only if there is a real payback to the business.

  • Business process owners are aware of service disruption and can take action to mitigate the consequences to the business.

  • Executives understand the value of their IT service provider.

To be accountable for the delivery of service, IT must manage simultaneously across business processes, management systems, and different business units, enterprises, and/or organizations. As the IT infrastructure has become so important to the enterprise's ability to conduct business, its complexity has increased dramatically, making it imperative for the business to understand the technology issues and consequences involved.

To gain this understanding, enterprises must view the infrastructure from a business service perspective. To achieve this, each line of business must be accessible with a degree of granularity that allows access to each individual technology element for diagnosis and repairfor example, access to an element manager to view infrastructure availability or an application analyzer to view the raw data of an application conversation. Lines of business services now tend to cut across the traditional management systems and business processes. They can also span organizations and divisions. An application such as supply-chain management can range across the functional organizations of sales, marketing, purchasing, manufacturing, and customer service.

A BSM system complements an enterprise's existing management systems to achieve this level of management sophistication. It operates as a top layer and provides a single, centralized method of command and control for the entire IT infrastructure.

This alignment is very simple in theory, and has been discussed at some length in Part II of this book; however, many organizations have been frustrated in their attempts to achieve it.

A major factor is that the IT department is not aware of business strategies at a detailed enough level. They do not know what business strategies to align with.

Business activities that require IT support appear to be practically thrown over the wall, because IT becomes involved only after decisions are made and it is implementation time. The IT department will receive an application rolled out on their environment (or to roll out) with little or no information pertinent to its delivery criteria or its criticality to the overall business model.

Without a detailed business strategy, IT's ability to plan well is extremely limited, and the best IT can do is to execute rapidly and attempt to architect flexible enough processes and systems to accommodate change with little or no notice.

For IT management to succeed in becoming a key participant in business strategies, there must be open communications with the business groups for such alignment. Only then will the IT department be able to develop detailed metrics for managing internal IT processes and tracking workload fluctuation, resource availability, and service quality pertinent to the business requirements.


Assigning the Correct QoS System

Where the entire end-to-end delivery system is under single control, you can use the simple approach to assigning QoS protocols and configurations discussed in the "Allocating Network Resources" section of Chapter 5. The implementation of this chapter's systems depends on how much control you havefor example, a priority set by your system at an edge could effectively be reclassified in the core, rendering your original configuration redundant. There is an element of cooperation and agreement dependent on your individual organization's unique arrangement with the third-party system.

QoS solutions are achieved through the use of traffic conditioners in the end-to-end network. The traffic-conditioning model implemented in RFC 2475, "An Architecture for Differentiated Services," includes classifier, policer (meter), marker, shaping, and dropping mechanisms. Cisco routers use many traffic-conditioning implementations to provide these functions. The methods used depend on the objective of the QoS policies implemented and the traffic present in the network.

The methodologies discussed so far in this book remain central to the overall system management and ultimate APM infrastructure. Central to these methodologies is the profile of the application. You must understand the application characteristics and dependencies, and then assign QoS based on a combination of those characteristics and the business delivery requirements.

The traffic-conditioning mechanisms used are as follows[1]:

  • Classifier Determines what classes packets belong to by inspecting various fields within the packet header including, but not limited to, source address, destination address, protocol identifier, source port, and/or destination port. The Cisco Network-Based Application Recognition (NBAR; see Chapter 6) can be used to further classify traffic based on Layer 4 through Layer 7 information within the protocol data unit. Classification is most often done using access control lists, but NBAR can be leveraged to police file-sharing applications and worms, which typically do not use well-known port numbers.

  • Meter Measures the speed of a traffic stream. The metering process can be used to affect the operations of the marker and shaper/dropper. Traffic that conforms to the committed information rate (CIR) is treated differently than the traffic that exceeds the CIR. A dual leaky-bucket algorithm is used with policing mechanisms that meter traffic. This enables the administrator to configure a committed burst (Bc) and excess burst (Be) rate that can be controlled with different exceed and violate actions. Policing can be used to re-mark and/or drop traffic.

  • Marker Responsible for turning on prioritization bit values in the Layer 2 and/or Layer 3 headers as follows to signify the importance of traffic:

    - IP traffic employs the IP Precedence or DSCP fields.

    - Layer 2 traffic depends on the technology used.

    - Ethernet has three 802.1p priority bits in the 802.1 header that can be marked.

    - Frame Relay includes a Discard Eligible (DE) bit, which can be marked to signify that the frame should be dropped when the service provider experiences congestion.

    - ATM has a Cell Loss Priority (CLP) bit that is similar in use to Frame Relay.

  • Shaper Limits bandwidth used on a link by queuing traffic that exceeds the set rate. Shaping is implemented only on egress links and proves especially useful in Frame Relay hub-and-spoke architectures. Frame Relay hubs may be running at DS-3 speeds (44.736 Mbps), whereas small sites may only have a fractional DS-1 access pipe of 128 kbps. The central site (hub) can flood the remote site link with excess traffic and not use resources properly. Framing would shape the traffic stream from the central site to a maximum rate of 128 kbps to the remote site.

  • Dropper Discards out-of-profile packets. This mechanism proves very useful in metropolitan Ethernet environments where a service provider will give the customer a Gigabit Ethernet handoff, but only allow 1 Mbps of bidirectional traffic based on the customer contract. The service provider would have the benefit of instantly provisioning. Dropping mechanisms can be deterministic in the case of weighted random early detection (WRED) or undeterministic in the case of tail drop.

All nodes in an internetwork perform traffic forwarding, queuing, and congestion-avoidance (WRED) procedures. Other QoS mechanisms are used, depending on the node's physical location within the network. As a general rule, edge nodes (ingress and egress) perform classification, marking, policing, shaping (only on egress), and dropping. A core router's main function in the network is to forward packets at high speeds. The CPU- and memory-intensive tasks associated with edge device QoS functionality would burden core routers.

You need to consider how the demarcation lines within your environment theoretically reshape your delivery topology, remembering QoS is deployed primarily at the enterprise LAN-WAN boundary. Based on the demarcation points, there may be a virtual shift in the delivery model.

Table 7-1 identifies various Cisco QoS mechanisms and their application to the QoS building blocks.

Table 7-1. Cisco IOS Traffic-Conditioning Mechanisms

Traffic-Conditioning Mechanism

Examples

Classification

Modular QoS CLI (MQC) IP to ATM

(Class of service) NBAR

QoS Policy Propagation over BGP (QPPB)

Route maps

ACCESS control lists

Marking

Committed access rate (CAR)

Class-based marking

QPPB

Route maps

Metering

Weighted fair queuing (WFQ)

Class-based WFQ (CBWFQ)

Priority queuing (PQ)

Custom queuing (CQ)

Weighted round-robin (WRR)

Modified-deficit round-robin (MDRR)

CAR

MQC policing

Class-based low-latency queuing (LLQ)

Shaping

Generic traffic shaping (GTS)

Frame Relay traffic shaping (FRTS)

Virtual circuit (VC) shaping

Dropping

WRED

Flow-based weighted random early detection (FRED)

CAR


Summary

In a system with noncentralized control, your first objective after having established the profile of the application, its path through the network, and the pattern it leaves behind is to assign demarcation points for monitoring performance.

Demarcation points are strategic connection points in the conversation transmission. By observing specific conditions (metrics) at these points, you can gain a quick and inductive view of the application's performance. The demarcation points become the natural aggregators of an application's performance issues.

The collected metrics provide the background and data to assess when you have taken the optimization as far as possible within the confines of your control. They will then assist in reassigning responsibility or escalating requirements and issues to third-party organizations.

When assigning QoS protocols in a system with noncentralized control, consider the virtual boundaries imposed by the different areas of control (such as a carrier-controlled core network).


End Note

  1. Hartmann, Dennis. Introduction to QoS. Global Knowledge. http://knowledgestorm.inc.com/search/viewabstract/inc/67554/index.jsp


References Used in This Chapter

  • Cisco Systems, Planning for Quality of serviceCiscoWorks Policy Manager, http://www.cisco.com/en/US/products/sw/cscowork/ps2064/products_user_guide_chapter09186a0080191f03.html

  • RFC 1513, Token Ring Extensions to the Remote Network Monitoring MIB, S. Waldbusser, http://rfc.net/rfc1513.html, September 1993

  • RFC 1757, Remote Network Monitoring Management Information Base, S. Waldbusser, http://rfc.net/rfc1757.html, February 1995

  • RFC 2021, Remote Network Monitoring Management Information Base, S. Waldbusser, http://rfc.net/rfc2021.html, January 1997

  • RFC 2457, An Architecture for Differentiated Services, S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, W. Weiss, http://rfc.net/rfc2475.html, December 1998


Chapter 8. Monitoring the Delivery

This chapter includes the following topics:

  • The pyramid model

  • Classifying management systems

  • Operating multiple systems

  • Defining metrics

  • Collecting metrics

  • Reporting mechanism

  • The delivery circle

This chapter discusses approaches to collecting application-profile characteristics using Cisco management tools. It discusses the classification of systems and the terms associated with this area of network management, going on to cover the ITUT FCAPS recommendations.

This chapter introduces the concept of an operational support system (OSS) for the enterprise, focusing on their interoperability and associated processes to provide the IT manager with the total view of the environment.

It goes on to discuss the definition and subsequent collection of metrics using protocols and applications such as SNMP, RMON, NBAR, and NetFlow.

Reporting the performance of the application is critical to the business model. The chapter discusses the hierarchical approach of collect once and use many for the various metrics.

Finally, it discusses the need to constantly review and refine the monitoring strategy to keep the process and systems in line with the business requirements and objectives.


The Pyramid Model

As more technology and resources are added to a network, more users and applications come to depend on it. It becomes a business enabler for an enterprise with utility-like expectations. It becomes a business service. The details of how the network works, what is broken, or what new capacity needs to be added are of little concern to end users who expect zero latency and infinite bandwidth. Although networks are often blamed for business-process failures, frequently the problem is poorly written applications, inadequate system resources, or simple human error, not to mention unannounced changes in user behavior or usage demands.

Historically, network managers used tools to deflect blame. As the network evolves to utility status, the goal can no longer be to deflect blame. A proactive service approach is needed, integrated across the infrastructure disciplines of network, system, and application management. Tools and processes must focus on establishing the root cause for problem resolution, no matter whose fault it is. The reengineering of network services must result in clearly defined business-relevant services tied to cost, with predictable and guaranteed service levels.

There have always been business processes in place. The difference is that the success of these business processes is now inextricably intertwined with IT services. In some cases, the business process is the IT service. Once upon a time, network management stood distinctly separate from computer system administration and applications management. New business models that are dependent on the performance of the IT infrastructure have changed this, requiring a new breed of business-savvy IS managers with end-to-end responsibility.

Using people, processes, and tools, the IS organization must link infrastructure elements (application, network, system) together to create the concept of an IT service that supports the business processes. The IS organization must learn to communicate in terms of business metrics, translating bits and bytes, latency, and jitter to service levels and customer satisfaction. Like any other business unit, the IS organization needs to meet a budget and be able to present the cost/benefit for any required investments.

Layered techniques can help IS organizations understand where products from the network management market fit in the process of synchronizing network resources with business objectives. This chapter builds up the pyramid of Figure 8-1 to achieve the top layer of business management.

Figure 8-1. Tools and Processes Pyramid Model

The purpose of the network management investment is to ensure that business initiatives are enabled and encouraged. Each investment must be viewed in the context of how it delivers greater enterprise speed, agility, innovation, adeptness, and customer-centricity. It is vital to seek out business goals and synchronize the network infrastructure with these goals.

You must define network services in coordination with business-unit requirements and put the instrumentation in place to measure service quality. You must understand the connection between network components and business processes to ensure you are gathering the right performance data, reporting on the correct service levels, and offering relevant guarantees.


Classifying Management Systems

Every vendor on the market has a solution that will "solve all your problems." This section discusses the importance of an enterprise operational support system (OSS) and defines the different management systems (such as analytical, diagnostic, reporting, monitoring, configuration, and so forth) and their relevant functions.

OSS

The term OSS traditionally referred to systems that perform service management, inventory, engineering, planning, and repair functions for communications service providers and their networks. Today's real-world requirements to manage the delivery of the application in line with the business objectives means that an OSS approach strategy is also a fit for almost all organizations looking to align their business requirements with their delivery infrastructure.

Making an application work the way it should depends entirely on how the application is implemented and its criticality to the underlying business goal of increased efficiency. OSSs are meant to optimize process efficiency. You cannot overlook the time it takes for the user to physically enter data and complete common tasks. Much of the work, such as port assignments in a provisioning application, can be fully automated, but often the business process requires that a person validate the assignment before it is sent to the network.

One of the more difficult aspects of any OSS implementation involves gathering and making sense of user and process requirements so that they can be reflected in the system's configuration. It is actually quite a challenge asking users to document their daily jobs to get them to use an application they may never have seen before. User-interface deployments often struggle or fail because of bad communication or poor planning.

In every aspect of any OSS application, it is clear that human factors, the deciphering and definition of the process, are key. An application is a tool, a sophisticated machine designed to do a job for people. That job only gets done well if the machine is oiled properly and has the right materials to work with. For OSSs, the right materials are process definitions, interface requirements, and permissions that are necessary because people are part of the system. It is too easy to be lost in functionality details and overlook whether those functions are accessible in a way that is dynamically configurable but also highly controlled.

Clarifying Network Management Classification Terms

As advances have been made in fault management, root-cause solutions, and other types of advanced analytics, the market has become increasingly confused by a variety of terms that often mean different things to different people. The following list defines some of these terms:

  • Troubleshooting This is diagnostic procedures directed at isolating a problem, whether it is fault related or performance related. It is a nontechnical term, freely in use, and has for years foreshadowed the continuum between fault and performance management.

  • Event management This might sound fairly basic and obvious, but events can range from simple pings to see whether the IP node is up or down to more complex polling for events that can reflect a variety of conditions based on policies. For instance, threshold conditions can be set so that events can also touch on performance-related conditions to proactively anticipate a probable failure.

    Thresholds can be combined with time-sensitive conditions (for example, buffer utilization at a condition of 80 percent over 6 hours). Finally, alerts can be created that reflect relatively sophisticated insights into device behavior. For example, if buffer utilization is X, CPU utilization is Y, and memory is at N, a high-level alert can be triggered based on rules or models for a relationship between these XYN metrics.

    Sometimes, these alerts are based on out-of-the-box insights into specific device or application behaviors, and at other times, they are based on customized rules that reflect unique operational conditions. Documentation of these conditions and resulting thresholds as they occur will make the rule creation more effective.

  • Event correlation A process that enables correlation across multiple event conditions to isolate problems.

  • Alarm suppression The policy-based suppression of alarms, typically a part of event correlation and event automation. Alarm de-duplication, or the elimination of duplicate alarms, is one type of alarm suppression. Single-point-of-failure notification is often achieved through event correlation.

  • Downstream alarm suppression A classic approach to fault management in routed rather than switched networks, capturing the hierarchic idea that devices downstream from a failed router or failed server also will appear to fail. However, downstream alarm suppression is no longer necessarily a good metric given the growing prevalence of switched or flat networks. Advanced topologies are dealing with this confusion, and most good root-cause solutions integrate connectivity information sensitive to physical layer connections.

  • Root-cause analysis This typically refers to isolating a single point of failure, often achieved through event correlation. This single point of failure can be a device, or a component of a device, or an application, or a line. Failure can mean metrics for availability (IP node up or down) or more complex behavioral aberrations, including those sensitive to performance, user experience, and service impact.

Classifying Management Systems by Function

Begin classification of management systems by dividing the functional areas of system management. The International Organization for Standardization (ISO) helped simplify the partitioning of network management activities by identifying five major areas (also know as FCAPS, for the initials of each functional area). This categorization is a functional one and does not describe the business-related role of a management system within the network. The idea of FCAPS stems directly from the ITUT recommendations and describes the following five different types of information handled by management systems, described in the previous section:

  • Fault

  • Configuration

  • Accounting

  • Performance

  • Security

Fault Management

The goal of fault management is to detect, log, notify users of, and (to the extent possible) automatically fix network problems to keep the network running effectively. Because faults can cause downtime or unacceptable network degradation, fault management is perhaps the most widely implemented of the ISO network management elements.

Fault management involves the following steps:

Step 1. | Determine the symptoms and isolate the problem.
Step 2. | Fix the problem and test the solution on all important subsystems.
Step 3. | Record the detection and resolution of the problem.
Configuration Management

The goal of configuration management is to monitor network and system configuration information so that the effects on network operation of various versions of hardware and software elements can be tracked and managed.

Each network device has a variety of version information associated with it. Configuration management subsystems store this information in a database for easy access. When a problem occurs, this database can be searched for clues that may help solve the problem.

Accounting Management

The goal of accounting management is to measure network utilization parameters so that resources used by individuals or group users on the network can be regulated appropriately. Such regulation minimizes network problems (because network resources can be apportioned based on resource capacities) and maximizes the fairness of network access across all users.

Appropriate accounting management involves measuring utilization of all important network resources. Analysis of the results provides insight into current usage patterns, so that usage quotas can be set. Some correction will be required to reach optimal access practices. From this point, ongoing measurement of resource use can yield billing information as well as information used to assess continued fair and optimal resource utilization. For example, a service provider using Compuware's Network Vantage can collect application usage and performance data of traffic traversing a particular link over a single month. This data can subsequently be used to provide an itemized bill, showing which groups used the link and how much data they transferred. Figure 8-2 shows a sample of the output.

Figure 8-2. Application Usage in Charge-Back Representation

[View full size image]

Performance Management

The goal of performance management is to measure and make available various aspects of network performance so that application performance can be maintained at an acceptable level. Examples of performance variables that might be provided include network throughput, user response times, and line utilization.

Performance management involves the following three main steps:

Step 1. | Performance data is gathered on variables of interest to network administrators.
Step 2. | The data is analyzed to determine normal (baseline) levels.
Step 3. | Appropriate performance thresholds are determined for each important variable so that exceeding these thresholds indicates a network problem worthy of attention.

Each of the steps just described is part of the process to set up a reactive system. When performance becomes unacceptable because of an exceeded user-defined threshold, the system reacts by sending a message. Performance management also permits proactive methods. For example, network simulation can be used to project how network growth will affect performance metrics. Such simulation can alert administrators to impending problems so that counteractive measures can be taken.

Security Management

The goal of security management is to control access to network resources according to local guidelines so that the network cannot be sabotaged (intentionally or unintentionally) and those without appropriate authorization cannot access sensitive information. A security management subsystem, for example, can monitor users logging on to a network resource and can refuse access to those who enter inappropriate access codes.

Security management subsystems work by partitioning network resources into authorized and unauthorized areas. Access to network resources is usually inappropriate for people outside of the company. For internal network users, access to information originating from a particular department may be inappropriate. Access to human resources files, for example, is inappropriate for most users outside the human resources department.

Security management subsystems perform several functions. They identify sensitive network resources (including systems, files, and other entities) and map them to user sets. They also monitor access points to sensitive network resources for logging occurrences of inappropriate access.

Element Management

An element management system (EMS) manages one or more of a specific type of network element (NE). Typically, the EMS manages the functions and capabilities within each NE, but does not manage the traffic between different NEs in the network. To support management of the traffic between itself and other NEs, the EMS communicates upward to higher-level network management systems (NMS).

At the lowest layer of the pyramid model, tools from network-device manufacturers, such as Cisco Systems CiscoWorks, achieve element management. Note that in this context CiscoWorks basic functionality is being referenced. It is understood that many associated modules will mean that CiscoWorks occupies a place in various stages of the overall process.

These tools are specific to managing individual network components or groups of similar components and often provide photorealistic device views for ease of use.

Most element management architectures use the same basic structure and set of relationships. End stations (managed devices such as computer systems and other network devices) run software that enables them to send alerts when they recognize problems (for example, when one or more user-determined thresholds are exceeded). Upon receiving these alerts, management entities are programmed to react by executing one or more actions, including operator notification, event logging, system shutdown, and automatic attempts at system repair.

Management entities can also poll end stations to check the values of certain variables. Polling can be automatic or user initiated, but agents in the managed devices respond to all polls. Figure 8-3 shows this basic functionality.

Figure 8-3. Automated and Unsolicited System Polling

[View full size image]

Agents are software modules that first compile information about the managed devices in which they reside, then store this information in a management database, and finally provide it (proactively or reactively) to management entities within NMSs via a network management protocol. The following five types of network management protocols are available to manage Cisco equipment at the bottom level of the pyramid model:

  • Telnet

  • Simple Network Management Protocol (SNMP)

  • Remote Monitor (RMON)

  • Syslog

  • Cisco Discovery Protocol (CDP)

Telnet

As defined in RFC 854, Telnet, also known as a command-line interface (CLI), allows direct login into a switch or router to gain access to configuration and monitoring commands. Telnet is an application in its own right, and the ability to directly log in to a device itself provides a base knowledge that the network is operational.

SNMP

As defined in RFC 1157, there are currently three versions of SNMP: SNMP1, SNMP2, and SNMP3. SNMP is based on the concept of an SNMP manager communicating with one or more SNMP agents, using SNMP. SNMP Get, GetNext, and Set operations are performed by the SNMP manager to the agent to either retrieve or set management variables supported by the SNMP agent. Management information is usually equally available from SNMP and through Telnet.

The following are the main components of the SNMP protocol:

  • Agents and managers The SNMP protocol defines methods that network agents (SNMP applications that reside on network devices and that follow the rules of SNMP) use to communicate with network managers (also called network management systems or NMSs)

  • Management Interface Bases(MIBs) Specifications containing definitions of management information that allow remote monitoring, configuration, and control of networked systems.

  • Protocol data units (PDUs) SNMP agents or managers can perform five types of functions: GetRequest, GetNextRequest, GetResponse, SetRequest, and Trap.

The SNMP model for a managed network includes one or more managed nodes, each of which is running an application called an agent, and at least one management station that can process information collected by the SNMP agents.

Some applications can act as both an agent and a manager, sending information to other managers and also managing information received from other agents. Management applications can query agents, asking for particular information about the device on which the agent is running, and agents can answer the queries. Agents can also send events to management applications without being queried. These types of events are called traps.

To answer the questions sent by the management agent or to organize the information that it may send in the form of traps, the agent relies on a database of information about the network device and its components or about the network itself. This database is called a MIB.

A MIB is a collection of objects that model the components of a device or a network. In theory, a MIB can model any object in existence. MIBs are defined in text files that use an arcane computer language called Abstract Syntax Notation One (ASN.1).

MIBs define a hierarchy of objects that is organized in a tree structure. Each object in the tree is identified and located by means of an object identifier (OID).

An object identifier consists of a series of numbers, separated by dots. For example, .1.3.6.1.4.4.1.1.120.1.1.1.3 is the OID for a data element called ManagerRowStatus in the MIB used with the Cisco Networking Service (CNS) notification engine.

Each object type in a MIB is defined by an OBJECT-TYPE construct in a MIB file. The first four numbers of an OID are almost always .1.3.6.1. The OBJECT-TYPEs for these are as follows:

  • 1iso International Organization for Standardization (ISO).

  • 3org Objects under ISO are organizations recognized by the ISO.

  • 6dod Department of Defense (DoD), which developed the original Internet (ARPANET).

  • 1internet A code that the DoD assigned to something called the "Internet community."

An alternative way of writing the path represented by an OID is to give the name of each OBJECT-TYPE, followed by its number in parenthesis. For example, you could also write .1.3.6.1 like this:

iso(1)org(3)dod(6)internet(1)

Take a look at the OID for the ManagerRowStatus data element used with the CNS notification engine (.1.3.6.1.4.4.1.1.120.1.1.1.3), which can also be written as follows:

iso(1)org(3)dod(6)internet(1)private(4)private(4)enterprises(1)stratacom(1)rtm(120) rtmproxy(1)trapsConfig(1)trapConfigTable(1)TrapConfigEntry(3),

Note that the OID corresponds to a hierarchy of objects defined in the rtm.mib file, located in the following CNS notification engine directory:

/opt/ssng/mib

This MIB file defines how the real-time management (RTM) proxy module in the CNS notification engine system processes traps that it will forward to the SNMP agent and which the SNMP agent will forward to your OSS.

To enable managers and agents to communicate with each other, SNMP defines the following five PDUs:

  • GetRequest Retrieves the value of specified management information. A single message can retrieve one or more items.

  • GetNextRequest Retrieves the OID and value of the MIB instance that comes one number after the instance of management information provided in the request.

  • GetResponse The response to a Get, GetNext, or Set request.

  • SetRequest Modifies the value of one or more MIB instances.

  • Trap Reports an event on a managed system.

Remote Monitoring

Based on the SNMP technology, the network remote monitoring capabilities are supported by a dedicated RMON data-collection and monitoring engine residing in a device. Communicating (for example, setting collection rules and receiving notifications from the engine) is performed through SNMP. RMON addresses ISO Layers 1 and 2, whereas RMON2 addresses ISO Layers 3 and above.

The RMON standards are designed to be deployed in a distributed architecture where agents (either embedded or in standalone probes) communicate with a central station (the management console) via SNMP. The RFC 1757 RMON standard organizes monitoring functions into nine groups to support Ethernet topologies, and adds a tenth group in RFC 1513 for Token Ring-unique parameters. Fast Ethernet link monitoring is provided in the framework of the RFC 1757 standard, and Fiber Distributed Data Interface (FDDI) ring monitoring is provided in the framework of both RFC 1757 and RFC 1513.

The RFC 2021 RMON specification drives remote monitoring standards beyond the Media Access Control (MAC) layer to the network and application layers. This setup enables administrators to analyze and troubleshoot networked applications such as web traffic, NetWare, Notes, e-mail, database access, Network File System (NFS), and others. RMON alarms, statistics, history, and host/conversation groups can now be used to proactively monitor and maintain network availability based on application layer trafficthe most critical traffic in the network. RMON2 enables network administrators to continue their deployment of standards-based monitoring solutions to support mission-critical, server-based applications.

Syslog

Cisco devices issue unsolicited notifications to a management station using the syslog protocol. Although similar in nature to SNMP traps, the syslog protocol is only used for event notification. The CISCO-SYSLOG-MIB is being implemented on Cisco devices as an alternative to issuing syslog messages, to enable any SNMP manager to receive all events through SNMP.

Cisco Discovery Protocol

The CDP is a Cisco proprietary Layer 2 protocol that is media and protocol independent and runs on all Cisco-manufactured equipment, including routers, bridges, access servers, and switches.

A Cisco device enabled with CDP sends out periodic interface updates to a multicast address to make itself known to neighbors. Because it is a Layer 2 protocol, these packets (frames) are not routed. Use of SNMP with the CDP MIB allows network management applications to learn the device type and the SNMP agent address of neighboring devices and to send SNMP queries to those devices. CDP uses the CISCO-CDP-MIB.

Operations Management

Tools at the operations management layer understand the relationships and dependencies between network components. They provide features such as discovery of network topology, status monitoring, fault management, and troubleshooting. They provide a reactive solution after a fault has occurred.

At the operations management layer of your network management model, current network troubleshooting practices require the human network manager to understand topology and device interconnections to correctly determine the source of a problem represented by multiple symptomatic events. This manual inference process relies heavily on the skills of IT staff members and often resembles the proverbial search for a needle in a haystack.

As IS organizations strive to move up to higher levels of IT management process maturity, they must move from the manual inference process to automated root-cause analysis (RCA). RCA tools correlate events and data from multiple sources to find the root cause of problems and trigger notification and corrective actions. To be successful, network RCA software must be able to discover accurate Layer 2 and 3 topology information and keep that topology continually updated. Without discovery, the network manager would be required to manually define and input correlation rules regarding device interconnections, eliminating the potential advantages of automated RCA.

Service Management

Building up to the service management layer of the network management model, you see the introduction of tools that collect performance data and include features such as baselining, trend analysis, historical usage analysis, service level reporting, and, in some cases, interfaces to billing systems. Business applications make use of many IT resources, and it is not always easy to track which resources come together to provide a service, and thus measure the service level achieved.

Also at the service management layer are tools that measure end-to-end response time. Network probe solutions are popular with network operations staff to audit external network services and prove internal service levels. Client capture solutions will be most useful to IS organizations supporting internal users where they have control over the desktops because the download of a desktop agent is required.

Synthetic transactions are useful with internal users and in simulating external users outside IS organization control. In all cases, response-time information is collected into a database and then analyzed for service level violations and trend reporting.

Business Management

At the business management layer, enterprises need tools that can correlate IT components with business services, and then relate component problems to the affected business service. Understanding the business impact of network faults and outages helps prioritize network support efforts. Some enterprises have already built this process manually, using tactical tools and business interviews to determine the service topology.

There are two general approaches to this area, bottom up and top down. From the bottom-up perspective, tools extend the network discovery process to identify the systems attached to a network component, and then discover which applications are running on those systems using functions such as snooping into network packets or port-scanning systems for active applications. Connection to an external provisioning or customer service database can also help reveal the business impact.

From the top-down perspective, starting with in-depth discussions between the IS organization and business units, it is possible to define the high-priority business processes or services and then construct a mapping of those services to their IT supply chain of underlying network, system, and application infrastructure components.

This mapping can be done using object-oriented model defining layers of abstraction between individual components and logical business services. By linking to existing tools that monitor the status and performance of components, a summarized picture of the overall health of the end-to-end service is achieved.

Basic principles of good management, such as using a disciplined naming convention, can help make the link between IT components and business services. Well-thought-out names will facilitate the selection, copying, and assignment of IT components to service views.

When it was a straightforward client/server connection, it was an easy decision just to throw resources at any problem. There were only a few options: upgrade the CPU or increase the network bandwidth. Today's architectures are much more complex, with multiple options such as outsourced network connections, firewalls, load balancers, and back-end servers. Enterprises cannot throw resources at all these places. They must decide what should be changed, and by how much.

Network modeling and simulation perform what-if scenarios in the safety of a lab environment (or virtual environment) to determine the most efficient network architecture to support business applications. Part of the capacity-planning process, modeling and simulation help ensure optimal network design to support current business applications and reliably forecast future performance based on an understanding of business demands, including additional users and new applications.

Network modeling and simulation is a complex exercise that requires a high level of training and experience. As an organization, you have to consider whether you have the need and skill set to perform this task internally. This exercise will add considerable value to the overall delivery. If you think the internal skill set is either not present or in fact desired, you should hire external consultants on a project basis as needed for a major network redesign or the rollout of a new application.


Operating Multiple Systems

Large transport and data networks generally include equipment from several sources, thus freeing the organization from dependence on a single vendor. However, this is achieved at the cost of more complex networking and network management.

Moreover, each vendor's equipment and graphical user interface (GUI) requires different operations, methods, and training. There is a compelling need for a central view of the whole network. No single product from any single vendor is likely to answer all your requirements in all areas. However, there are some tightly coupled solutions within single brand portfolios, and through vendor partnerships, that begin to answer most of these demands. This section discusses the amalgamation of information from the relevant functional areas to provide a single centralized, dashboard view of the operation.

Initially, you should seek out those areas of investment that most match your requirements. An effective assessment of what your management requirements are is always the most important first step in an effective deployment. This should include not only your business requirements but also your expected workflow. Figure 8-4 shows a conceptual business model and workflow.

Figure 8-4. Business Management Model

[View full size image]

There are two schools of thought as to how all these components should be brought together. One is the typical approach for IP networks, which keeps all the intelligence in the network and relies heavily on traditional element management and vendor-aligned solutions. The second locates some of the intelligence that relies on end-to-end network knowledge and some of the heavy coordination work in a central OSS instance in the network.

Figure 8-4 covers the basic elements of the FCAPS model. Having defined your basic areas of functional requirements, you should then begin to match specific solutions with each area. It is not the intention of this text to detail a specific solution, because the requirements and size will differ vastly from organization to organization.

Before assembling the system, maintain a close relationship between the requirements at each level and the systems used to meet those requirements. Overlap may occur, but keep it to a minimum.

The next element to consider is the ability to share information where these overlaps occur. Systems may either exchange information using a common protocol or they may write to a common database. The database system serves as the data warehouse that provides the basis of all historical reports as well as performance measurements. An OSS should determine which of the following features are required for this system:

  • Commercial relational database management engine, such as Structured Query Language (SQL), to allow for tailored platform

  • Large-scale, high-performance disk storage with scalable growth path

  • Data integration point of all database functionality provided through a common, best-in-class relational database

The operations and tracking system handles processing, data distribution, and actions based on information received from the enterprise collection system. An organization should determine which of the following features are required for this system:

  • Resourcing of trouble ticket, inventory, security, performance, and configuration management processing

  • Centralized event and notification system

  • Linking to all external actions and event processing

An ideal solution will enable you to take this business model information and view it in a dashboard view, not replacing the individual components, but merely providing a centralized view of the whole system. This approach will also enable you to retain investment in existing management platforms, which become feeders to the umbrella system.


Defining Metrics

When a new application is being rolled out or an existing one upgraded, the application owners will have some preconceived ideas concerning expected delivery metrics. These may have been obtained through predeployment testing or just based on a combination of gut feeling and experience. This section discusses the definition of key metrics in an effort to match business with technical requirements.

Choosing the Right Number of Metrics

Traditionally IT engineers have focused on the old adage that if you cannot measure it, you cannot manage it. This in turn has led to a situation where they have decided to measure everything they can possibly think of. This may be considered a safe approach, and may work for a small LAN-based system. When you try and implement the same strategy to a large enterprise environment, however, you will start to run into the following issues:

  • The sheer amount of data you have to correlate is imposing.

  • Additional overhead on the network potentially impacts your transmission medium when trying to retrieve these metrics.

  • Load on the infrastructure devices diverts them from their primary focus of moving packets.

  • You are dealing with a dynamic environment. Historically, metrics are put in place because of past events and are rarely revisited to see whether they are still relevant.

Balance Metrics Between Business and Technical Objectives

When optimizing your application delivery, your goal is to maintain an alignment with the business objectives. This leads to one of the biggest complaints from the business users. Metrics tend to concentrate on the technical parameters. When confronted by performance complaints from the business users, the technical answer is along these lines: "Network utilization is 38 percent, so I cannot see why you have a performance issue."

There is essentially nothing wrong with the technical approach as long as you maintain relevance to the business requirements. This will not only assist in the delivery process, but will also facilitate accurate reporting to the relevant business units.

Although many network management organizations have effective alarm techniques to notify technicians about a device failure, it is much more difficult to define and implement an assessment process for the end-to-end application performance. Therefore, although the network operations center (NOC) can respond quickly to a downed router or switch, the network conditions that may undermine network performance and affect user perception may easily go unnoticed until that perception becomes negative. However difficult, implementing an assessment process can provide immense benefit to both the business organization and network management.

Performance management is an umbrella term that incorporates the configuration and measurement of distinct performance areas:

  • Gathering network baseline data

  • Availability

  • Response time

  • Accuracy

  • Utilization

  • Capacity planning

Gathering Network Baseline Data

When defining the metrics, ensure that you do not make unrealistic expectations of your network performance. This usually comes from misunderstanding the details of networking protocols or the applications. Often, poor performance is not the fault of the network, but rather a result of poor application design. The only way to document and measure an application's performance is to have a baseline of the network's performance prior to installing the application.

Perform a baseline of your network before and after deploying a new solution (application or IOS change) to determine whether the solution initially meets performance and availability objectives and benchmark capacity. A typical router/switch baseline report would include capacity issues related to CPU, memory, buffer management, link/media utilization, and throughput. You may also include other types of baseline data, depending on the defined objectives in the concept of operations. For example, an availability baseline would demonstrate increased stability/availability of the network environment. Perform a baseline comparison between old and new environments to verify solution requirements.

Another specialized baseline is the application baseline. It can be valuable in trending application network requirements. This information can be used for billing and/or budgeting purposes in the upgrade cycle. Application baselines can also be important when considering an application's availability relevant to delivery criteria or qualities of service per application.

This chapter covers baselining in just a general way. For more details about baselining, see Chapter 3, "Detailing the Business Transaction." The key now is in defining the metrics. There is no one answer. What you are looking for is something of a natural aggregator that will enable you to collect from as few points as possible, while still providing the underlying data for the relevant IT department to drill into, should the metric recorded breech the predefined threshold.

For example, the journey for a businessman to his office is approximately 20 miles. A set of traffic lights is situated at about 10 miles into the journey. Over a period of time, the businessman observes that when he arrives at these traffic lights at 8 a.m., he arrives in the office on time at 8:30 a.m., and any deviation in his arrival at the traffic lights (either earlier or later) has an effect on his arrival time into the office.

Many factors can affect his journey time, such as the load on the highways, the weather (people have a tendency to drive slower in poor conditions), whether he has to stop for gas, and so forth. All these factors can be measured by various means, but simply by observing the arrival time at the traffic lights you can build an accurate prediction of his arrival time at the office.

So in this example you measure the other metrics that affect this journey, such as fuel load and road conditions, but your trigger threshold to review is the single metric of arrival time at the traffic lights. The trigger is initiated by a time deviation of plus or minus 5 minutes from the target time of 8 a.m.

The same principle that applies to the businessman applies to the enterprise. It is only relevant to report on factors pertinent to the journey (whether that be a businessman's journey on the road or a business-critical application's journey on the transmission path). This is where the baseline becomes very important, because it provides an understanding of the relevant portions of the applications journey.

The baseline document should give you enough information to understand the flow of business-critical application conversations, and ultimately provide a set of traffic parameters defining a traffic flow that is both quantitative (for example, bandwidth, delay, jitter) and qualitative (such as packet loss) in nature. From here you are able to select metrics that relate to the business activities your system is serving. In turn, you assist in avoiding too many unnecessary measurements creeping in. Figure 8-5 shows a simple network with an associated application conversation path.

Figure 8-5. Conversation Path and Metric Collection Points

[View full size image]

Figure 8-5 shows an application conversation between the application and database server at HQ to the branch users at Site 2.

To collect the conversation metrics, a probe is positioned at Point A. Here the switch is configured to monitor the ports connected, the application servers and the database server, and the ingress/egress interface on the switch (Point B).

At Site 2, a second probe is configured to monitor the ingress/egress interface to the branch site (Point C). This is the optimal configuration for monitoring at this site. It will allow the probe to observe all the business-critical conversations, those traveling between Site 2 and the servers at HQ, but will not be flooded with intra-Site 2 traffic.

The location of the two probes (one at HQ switch and one at Site 2 switch) will provide coverage and details of the application conversation, and will show you details such as packet drops and latency. In addition, it will reveal the response times at specific parts of the journey (for example, at the HQ switch or the Site 2 switch). This will assist in any required troubleshooting or optimization.

To complete the metric collection, statistics are collected from infrastructure traversed in this business-critical path. Metric collection from nonessential parts of the network is not necessary, and would only confuse the picture. For example, the traffic flowing between the users and servers at Site 3 has no impact on the business-critical application.

Availability

One of the primary metrics used by network managers is availability. Availability is the measure of time for which a network system or application is available to a user. From a network perspective, availability represents the reliability of the individual components in a network.

Measuring availability, for example, requires coordinating the help desk phone calls with the statistics collected from the managed devices. However, availability tools cannot determine all of the reasons for failure.

Network redundancy is another factor to consider when measuring availability. Loss of redundancy indicates service degradation rather than total network failure. The result may be slower response time and a loss of data due to dropped packets. It is also possible the results will show up in the other areas of performance measurement such as utilization and response time.

Response Time

Response time is the best measure of your customer's network use and can help you gauge your network's effectiveness. No matter what the source of the slow response is, users get frustrated as a result of delayed traffic. In distributed networks, many factors affect the response time, including the following:

  • Network congestion

  • Nonoptimal route to destination (or no route at all)

  • Underpowered network devices

  • Network faults such as a broadcast storm

  • Noise or cyclic redundancy check (CRC) errors

Whenever possible, measure response time as it appears to users. Users perceive response as the time from when they press Enter or click a button until the screen displays. This elapsed time includes the time required for each network device, the user workstation, and the destination server to process the traffic.

Many solutions are available for measuring response time. Although the levels of decoding they provide vary, the following approaches should enable you to assemble source and destination details and application type and response times relative to the collection point, the server, and the client:

  • Network devices and servers You can use the network devices and servers to measure response time.

  • Internet Control Message Protocol (ICMP) You can use ICMP to measure transactions; although it does not take into account any delays introduced into a system as the upper layers process it. This approach solves the problem of understanding how your network is performing, but only at a very low level and should be used only as an indicative guide.

  • Timing the response to pings At a simplistic level, you can measure response time by timing the response to pings from the network management station to key points in the network, such as a server interface, the endpoint of a service provider connection, or key user IP addresses. The problem with this method is it does not accurately reflect user perception of response time between the user machines and destination machines. It just collects information and reports response time from the network management station's perspective. This method also masks response time issues on a hop-by-hop basis throughout the network.

  • Cisco IOS Service Assurance Agent (SAA) An alternative to server-centric polling is distributing the effort closer to the source and destination you want to simulate for measure. Use distributed network management stations and implement SAA functionality. By using SAA on routers, you can measure response time between a router and a destination device such as a server or another router. You can also specify a TCP or UDP port, thereby forcing traffic to be forwarded and directed in the same manner as the traffic it is simulating.

  • Application-aware probes These can either be third-party discrete systems or embedded systems within the infrastructure. These devices can observe the application traffic passing by. As long as the conversational flow initiates on, terminates on, or uses the collection point as a transient path, the conversation will be recorded.

  • Protocol discovery Network-Based Application Recognition (NBAR) provides a special protocol-discovery feature that determines which application protocols are traversing a network at any given time. It captures key statistics associated with each protocol in a network. These statistics enable you to understand the characteristics of the application performance.

Utilization

Utilization measures the use of a particular resource over time. The measure is usually expressed in the form of a percentage in which the usage of a resource is compared with its maximum operational capacity.

Through utilization measures, you can identify congestion (or potential congestion) throughout the network. You can also identify underutilized resources. Utilization is the principle measure to determine how full the network pipes (links) are. Measuring CPU, interface, queuing, and other system-related capacity measurements enables you to determine the extent to which network system resources are being consumed.

High utilization is not necessarily bad. Low utilization may indicate traffic flows in unexpected places. As lines become overutilized, the effects can become significant. Overutilization occurs when there is more traffic queued to pass over an interface than it can handle. Sudden jumps in resource utilization can indicate a fault condition.


Collecting Metrics

No canned solution to metric data collection meets every organization's unique requirements. However, every organization shares several common factors that affect the number and level of metric details in the process and procedure build-out.

Measuring the overall performance of the operation through prescribed business and operational metrics allows the organization to baseline its performance and effectiveness.

Performance baselining involves taking measurements at regular intervals, over a long enough period of time to encompass the normal and busy times. For example, you might find high utilization at certain times every day, but month-end business activity increases utilization to a new, higher overall level. Your baseline needs to identify these critical activity peaks. The frequency at which you take samples and the ways in which you interpret the raw data will directly influence the end result.

There are many methods of collecting data for use in your optimization model.

Data-Collection Methods

The first data-collection method is to make use of the statistics and data that is embedded in the infrastructure. This includes aspects of the polled data, such as RMON and SNMP, as well as traditional methods, such as Telnet. Next is the data available from software features and hardware. This is also effectively embedded in the infrastructure. Finally, analyzers or probes deployed on a combination of sample and key points will provide the final piece of the puzzle, giving an end-to-end perspective of the application-delivery model.

Collecting Data with Telnet

You can collect all the performance data you need for polling and monitoring by initiating a Telnet session to the network devices and issuing various CLI commands. Traffic and error statistics for each interface, as well as critical data on the general health of the devices, are available through the CLI. This method does not scale for large numbers of devices or for continuous monitoring of any number of devices. However, it is an effective tool for obtaining real-time data when troubleshooting a problem. It is also a necessary tool for obtaining data that is not available via SNMP. You can use scripting languages such as Expect and Perl to obtain this data on a periodic basis or in response to the detection of some fault condition.

Collecting Data with SNMP

The most common method for collecting performance data is to use SNMP. Through SNMP polling, you can collect RMON MIB, MIB-2, and other enterprise-specific data, including traffic statistics for your critical ports. This data may also include enterprise-specific data for overall switch performance, such as backplane utilization.

Commercial NMS platforms are common methods of managing a network through SNMP. Typically, you can use the polling services in such a package to periodically poll your switches for specific information and store the data in some type of database. The database may be simple flat files or a relational database management system (DBMS).

Generating reports from this data may consist of importing the flat files into a spreadsheet or using the reporting utilities of the DBMS. You may also opt for creating your own SNMP tools using a scripting language such as Perl, which is also useful for creating reports from flat files.

Specialized management packages are similar to the more general NMS products mentioned previously, but they specialize in a certain type of data, such as RMON or MIB-2 traffic statistics. These packages generally produce numerous basic reports, with options for customizing them for your needs.

Deciding What Data to Poll

After you have the means to gather the data, you must decide what data to poll. There may be more than one MIB object that contains the same data. For example, the RMON etherStatsTable, the MIB-2 ifTable, and the dot3 MIB all contain objects that provide similar traffic and error statistics on Ethernet ports.

Cisco recommends using the etherStatsTable for most polling purposes. Many commercial applications already poll the etherStatsTable for this data and have ready-made tools for processing the data and generating reports. It is also easy to take advantage of the RMON history-collection features to reduce polling overhead. Note that some Catalyst Cisco IOS versions allow the setting of RMON thresholds only on objects in the etherStatsTable. Because the objective is to baseline objects on which RMON thresholds can be set, the etherStatsTable is a good first choice.

Some traffic statistics that are useful to baseline include the following:

  • Total traffic (etherStatsOctets)

  • Multicast traffic (etherStatsMulticastPkts)

  • Broadcast traffic (etherStatsBroadcastPkts)

Some error statistics to monitor include the following:

  • Error frames (etherStatsCRCAlignErrors)

  • Ethernet fragments (etherStatsFragments)

  • Jabbers (etherStatsJabbers)

Monitoring Ethernet collisions in a switched network is usually useful only on shared segments.

Cisco devices supported are available in the public MIBs area on the Cisco website. (Search for "MIB" at www.cisco.com.) The v1 and v2 directories contain ASN.1 MIBs complete with object definitions in SNMP v1 or SNMP v2 syntax, respectively. Most NMS stations can compile these files directly into a MIB database with no modification. The OID and schema directories are provided for other NMS stations that require the MIB data in this format. The OID directory is also handy for finding the full OID of a particular object if you are writing your own scripts to collect this data.

SNMP polling for switches is similar to SNMP polling for routers, but the amount and type of MIB variables polled for differ. A large number of interface or port MIB variables will not be polled using SNMP, because of the density of the ports on switches and the unknown state of switch ports directly connected to PCs or workstations that may be turned off at any time. SNMP proves very useful for monitoring switch and trunk port health.

NetFlow data enables extensive near-real-time network monitoring capabilities. Flow-based analysis techniques may be used to visualize traffic patterns associated with individual routers and switches as well as on a network-wide basis (providing aggregate traffic or application based views) to provide proactive problem detection, efficient troubleshooting, and rapid problem resolution.

The purpose of polling needs to be determined before discussing what to poll for. Polling devices have the following three major purposes:

  • To determine the availability of a device (monitor polling)

  • To determine whether an error condition has occurred or is approaching (threshold polling)

  • To analyze data for trends or performance measurements (performance polling)

Monitor Polling

The purpose of monitor polling is to detect when network changes have occurred and to generate an immediate alarm. Monitor polling detects hard errors such as a device not responding or an interface status change. Any alarm generated by monitor polling should be acted on immediately.

If your network management system is monitored on a 24x7 basis, a monitor polling alarm should create a visible and audible alarm on the system, so that the operator can take immediate action. If you do not have your network management system monitored on a 24x7 basis, it is recommended that any monitor polling alarm generate an alarm to the appropriate person, possibly via pager or e-mail.

Monitor polling applications are included with most commercial SNMP platforms and SNMP-based management applications. The purpose of monitor polling applications is to analyze the data points and generate an alarm when necessary.

Threshold Polling

The aim of threshold polling is to determine when escalating error conditions are occurring and take action before performance is severely impacted. A wide variety of problems originally surface as increased error conditions. These conditions may eventually become a hard error, or may present themselves as phantom problems that come and go. Threshold polling is a tool to detect these problems.

To implement threshold polling, the first decision is to determine which MIB variables to poll. After the threshold MIB variables have been determined, you must establish baseline values for these variables. You can do this by starting a polling process on the MIB variables for a period of time (1 week, for example).

You must then review the data collected. Baselining should be performed during peak traffic patterns to be most representative of unusual conditions. The more static your network is (only yearly adds, moves, and changes, for instance), the less baselining you need to perform; the more dynamic your network is (monthly adds, moves, and changes, for instance), the more baselining you need to perform. With this data in hand, determine what values would be out of specification. A rule of thumb is to set thresholds 10 percent to 20 percent higher than the maximum value seen.

The threshold values for any particular MIB variable may be applied uniformly across all switches, or they could be customized for groups of switches that have similar characteristics (such as core and distribution).

Another decision to be made is what kind of notification is appropriate for threshold violations. Threshold violations are not indications of hard errors, so immediate notification is usually not in order. Logging all threshold values and reviewing on a daily basis is usually the most appropriate means of notification.

It is very important that repeated threshold violations be investigated. It can then be determined whether a problem has occurred that can be corrected, or whether the violations are a result of the threshold values being too low for the circumstances.

Performance Polling

The aim of performance polling is to gather data that can be analyzed over time to determine trends and to aid in capacity planning. As with threshold polling, the determination of what MIB variables to poll for is the first consideration. For performance polling, individual data points (raw data) are stored intermittently on the polling machine. Depending on what polling mechanism you are using, the data could be in either a raw format or in a relational database.

To best keep the data manageable, the raw data should periodically be aggregated and stored in another database or file to be kept for future reporting. The raw data can be kept for a period of time for backup purposes, but eventually this data should be purged and only the aggregate data kept. The last step in the process is to produce reports from the aggregate data that will be periodically reviewed for trending or capacity-planning purposes.

Sometimes you will not be able to obtain all the by utilizing infrastructure-polled data. In those cases, it is worth considering the placement of external analyzers or probes to enable the collection of relevant application data.

The majority of sites now use switch technology, preventing the shared listening environment that was historically available for obtaining data. When presented with this scenario, you can make use of the Switched Port Analyzer (SPAN) or Remote Switched Port Analyzer (RSPAN) features, sometimes called port mirroring or port monitoring.

SPAN selects network traffic for analysis by a network analyzer such as a SwitchProbe device or other RMON probe. SPAN mirrors traffic from one or more source ports on any VLAN, from one or more VLANs, or from the sc0 console interface to a destination port for analysis. (See Figure 8-6.)

Figure 8-6. SPAN Configuration[1]

In Figure 8-6, all traffic on Ethernet Port 5 (the source port) is mirrored to Ethernet Port 10. A network analyzer on Ethernet Port 10 receives all network traffic from Ethernet Port 5 without being physically attached to it.

The RSPAN configuration shown in Figure 8-7 has all the features of SPAN plus support for source ports and destination ports distributed across multiple switches, allowing remote monitoring of multiple switches across your network.

Figure 8-7. RSPAN Configuration[1]

The traffic for each RSPAN session is carried over a user-specified RSPAN VLAN that is dedicated for that RSPAN session in all participating switches. The SPAN traffic from the sources is switched to the RSPAN VLAN and then forwarded to destination ports configured in the RSPAN VLAN. The traffic type for sources (ingress, egress, or both) in an RSPAN session can differ in different source switches, but is the same for all sources in each source switch for each RSPAN session. Do not configure any ports in an RSPAN VLAN except those selected to carry RSPAN traffic.

In Figure 8-7, the user ports A1, A2, B1, B2, and B3 are the ports to be monitored at the source switches. The monitored traffic from these ports is transported back to the centralized analyzer located at the data center via the RSPAN VLAN. This RSPAN VLAN is initially distributed across the source switches, and each source switch data is sent via ports A3 and B4 to the corresponding interfaces C1 and C2. At the intermediate switch, the VLAN streams are unified and transported to the data center switch via interface C3, arriving at the data center switch on interface D1 where it is sent to its final destination of D2 and the connected analyzers.

NetFlow Data

NetFlow data gathered on network devices is exported to a collector machine. The collector performs functions such as reducing the volume of data (filtering and aggregation), hierarchical data storage, and file system management. Cisco provides NetFlow Collector and NetFlow Analyzer applications for gathering and analyzing data from routers and Cisco Catalyst switches. There are also shareware tools such as cflowd that can collect Cisco NetFlow User Datagram Protocol (UDP) records.

Although the richness of data provided by NetFlow, required for planning, monitoring, and accounting applications, is valuable to your model, you should carefully plan NetFlow deployment. This is something that should be designed after review of your baseline study. Key deployment considerations include the following:

  • NetFlow services should be used as an edge metering and access list performance acceleration tool, and not activated on core/backbone routers or routers running at very high CPU utilization rates.

  • Understand your application-driven data-collection requirements. Accounting applications may only require originating and terminating router flow information, whereas monitoring applications may require a more comprehensive (data-intensive) end-to-end view.

  • Understand the impact of network topology and routing policy on flow-collection strategy. For example, avoid collecting duplicate flows by activating NetFlow on key aggregation routers where traffic originates or terminates and not on backbone routers or intermediate routers.

NetFlow is an input-side measurement technology that should be deployed on appropriate interfaces on edge/aggregation or WAN access routers to gain a comprehensive view of originating and terminating traffic to meet customer needs for accounting, monitoring, or network planning data. The key mechanism for enhancing NetFlow data volume manageability is careful planning of NetFlow deployment. NetFlow should be deployed incrementally (interface by interface) and strategically (such as on well-chosen routers), instead of widespread on every router in the network.


Reporting Mechanism

You do not want to bore the customer or management with myriad reports. They just want to know whether the system is running correctly. This section discusses the various levels of hierarchical reports and their respective audiences (such as senior management, engineers, and so forth).

It is important for the business owners to be able to view and understand the performance of the applications. Therefore, the reports must be presented in terms relative to the business and not in the more traditional terms of bits and bytes. This is not to say that you should exclude bits and bytes information from your reporting structure.

The reporting system should provide select operational and status information for clients and end users.

The many data-delivery options depend on the accuracy and nature of the collected data. The basic features in a reporting system should include fully secured access to a wide range of richly formatted information for all subscribing users based on either web or client technology.

Business owners are really only concerned with performance relevant to their requirements. For example, an online retailer is interested only in the processing of orders within a maximum time limit, not that the carrier bearer circuit is carrying 45 Mbps of data and is running at a utilization rate of 32 percent. In addition, they want to see this information in as simple a form as possible, preferably an easy-to-read graphic rather than a list of figures.

Presentation and context is analogous to goal seeking in human beings. This is an area of tremendous investment. GUIs and reports should support users of a specific background who have to make decisions. In other words, create a contextual, role-based presentation.

The reporting function should serve dual purposes: verifying performance and providing data for troubleshooting. The information obtained from the various metrics on application performance provides a leg up for finding the causes of performance problems and fixing them.

In general, the owners of applications just want to see a clear view with a traffic-light approach: green, good; yellow, within tolerance but with issues; and red, business objectives not being met.

To achieve this, you need to amalgamate the metrics to provide that single view. At the same time, make the data accessible to the various groups responsible for the specific elements of the application-delivery process.

This method of using metrics to supply reports and data to various groups (illustrated in Figure 8-8) also meets the demand of the business, in which the application-delivery process is no longer isolated to a single department, but is collectively the responsibility of many groups. This method can also help to eliminate finger pointing when a performance issue arises.

Figure 8-8. Hierarchal Report Model

[View full size image]

Amalgamating all the metrics in a single common report also enables the disparate groups to view the data relevant to the other groups, thus expediting resolution and ultimately improving overall application delivery.

The presentation of the reports will differ from organization to organization and should be tuned to meet the business objectives. This means customizing the reports with meaningful data such as known names as opposed to IP addresses and so forth.

To provide the most effective use of your data, your reporting system should be able to address the following questions:

  • Does the GUI support an appropriate range of operational functions, from operator to administrator, to manager, and to director?

  • Does the GUI provide a common frame of reference for both the service desk or help desk user and the operations center? Similarly, does the GUI help enable common decision making across domain expertise (typically network, application, and systems specialists)?

  • Do the reports support or model business-related impact? Are the reports appropriately designed for more senior executives, up to and including, in some cases, CIO- and CFO-level users?

  • Do the reports bridge the gap between component management, infrastructure management, and the delivery of services with service level agreements in place?


The Delivery Circle

A newspaper publisher occupies something of a unique place in the manufacturing industry, because at the end of each day the publisher has produced a product, and then the next day the publisher must start from scratch on a completely different product. The publisher must constantly review and refine their offering to ensure that each new product is received with the same enthusiasm as the last (and thus maintain sales). You can apply the same theory to your delivery system. You need to constantly monitor and observe the delivery process to ensure that your end product continues to meet expectations. The ever-changing demands of the end-to-end system ensure that the optimization will not be a single-occurrence event.

After you have defined your metrics, you enter into a constant evaluation process, continually reviewing the location and type of data you are collecting. This review should be linked to your overall management policy and built in to the ongoing procedures.

The application life cycle traditionally covered only the development process. As enterprise software has expanded to encompass more business-critical and mission-critical activities, however, the technology for building and deploying applications has become more sophisticated, and the inclusion of delivery data has become an important part of the process.

Data collected during the optimization process is fed back into the system. This baseline facilitates future development and provides trending information for use in capacity-planning studies.

One of the most difficult aspects of software development and deployment is the efficient sharing of information among different groups and teams. This is also one of the most crucial factors in success. Effective communication can ensure that the right tasks are successfully accomplished, that issues are addressed quickly, and that project status can be discerned rapidly and easily.

Effective communications is one of the central benefits of monitoring the application delivery and producing a set of hierarchical reports.

Metrics and traffic flow will not stay fixed forever. Part of the monitoring strategy should include a procedure to review changes in performance relevant to changes in application characteristics. Comparing the baseline data with the new profile easily does this. Any discrepancy will show, and changes can quickly be made to the delivery system. This may mean changing the nature of the metrics, moving a probe or analyzer, or even adjusting expectations to match the new characteristics.


Summary

Network management tools, good performance data, and the transformation of data into information are not enough to position the network as a business service. It requires a strategic link with the business to understand what metrics are critical, which ones really make a difference, and where the business impact is. Forging links with the business is critical to positioning the network as a business service.

The baseline document should give you enough information to understand the flow of business-critical application conversations, and will ultimately provide a set of traffic parameters defining a traffic flow that is both quantitative and qualitative in nature. Measuring the overall performance of the operation through prescribed business and operational metrics allows the organization to baseline its performance and effectiveness.

Polling devices have three major purposes: to determine the availability of a device, to determine whether an error condition has occurred (or is approaching), and to analyze data for trends or performance measurements.

It is important for the business owners to be able to view and understand application performance. This means the reports must be presented in terms relative to the business and not in the more traditional terms of bits and bytes.

Having defined your metrics, you will be entering into a constant evaluation process that requires you to review continually the location and type of data you are collecting. This review should be linked to your overall management policy and should be built in to the ongoing procedures.


End Note

  1. Cisco Systems, Configuring SPAN and RSPAN, http://www.cisco.com/en/US/products/hw/switches/ps700/products_configuration_guide_chapter09186a008007fa3a.html


References Used in This Chapter

  • Cisco Systems, Cisco Network Monitoring and Event Correlation Guidelines, http://www.cisco.com/en/US/products/sw/cscowork/ps2393/prod_technical_reference09186a00800a9e2f.html

  • Cisco Systems, Baselines Process: Best Practices White Paper, http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008014fb3b.shtml

  • Cisco Systems, Performance Management: Best Practices White Paper, http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008014fbf3.shtml

  • Cisco Systems, Cisco Service Assure Agent, http://www.cisco.com/en/US/products/sw/iosswrel/ps1830/products_feature_guide09186a0080087a78.html

  • Cisco Systems, Network Management System: Best Practices, http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a00800aea9c.shtml


Part IV: Planning and Troubleshooting Network Applications

Chapter 9 Proactive Planning

Chapter 10 When Applications Fail


Chapter 9. Proactive Planning

This chapter includes the following topics:

  • Baselining performance levels

  • The dynamic enterprise

  • Diagnosing problems

  • Validating changes

  • Enabling technologies

  • Scenario modeling

  • Capacity planning

  • Reviewing SLAs

  • End-to-end delivery

Networking is dynamic. Things will change, including software versions and usage levels. For application monitoring to be effective, businesses must first define their performance objectives (acceptable response time limits) for each application transaction type. Proactive planning ensures maintenance of the delivery criteria required of business-critical applications. (That is, application response times will be maintained as user numbers increase.)

This chapter begins by discussing aspects of proactive troubleshooting, using baseline-type data. It then discusses the dynamic nature of the enterprise environment.

Moving on, it discusses the approach and reasoning behind validating changes in your environment.

In addition to the respective tools used in end-to-end monitoring systems, certain enabling technologies and methodologies are desirable. This chapter defines and introduces those technologies.

It goes on to discuss the use of a virtual network environment that replicates the behavior of the enterprise network, including its routers, switches, protocols, servers, and individual applications.

A chain is as strong as its weakest link, and a distributed environment's performance is often set by its weakest component. Capacity planning is an active process of measuring and monitoring networked resources to ensure delivery of business-critical applications will be maintained. This chapter discusses practical and meaningful approaches to capacity planning.

It goes on to discuss using service level agreements (SLAs) as a performance indicator, covering the review process and assignment of penalties. It includes a summary of assigning SLA criteria.

Finally, it discusses the vision of an end-to-end system, and what is realistically achievable (and practical) in an enterprise.


Baselining Performance Levels

Performance test results, gained by testing an application's performance prior to its live debut, can provide appropriate baselines unique to a specific application environment. For example, if test results show that a typical purchase transaction consistently requires 45 seconds to process, then 45 seconds would become the logical baseline to measure against in the live environment. After establishing acceptable levels of application performance, organizations possess key information from which to draw comparisons.

Baselining is the key to ascertaining a network's load and performance capabilities going forward; this is known as capacity planning. It enables network administrators to understand their current level of capacity and performance, a requirement for understanding the impact of new network application deployment or additions to deployed applications. Baselining should be done on at least a quarterly basis to identify capacity and trends.

Performance baseline objectives can be further normalized within the live environment to more accurately represent response times based on times of day and/or geographic location.

At the start of business, for example, most organizations experience a peak of network traffic as users begin the day by logging on to systems. This increase in load likely represents a peak for the day, with general usage recorded at a much lower load. It is unwise to use the peak data in isolation without considering that the peak is a once-a-day occurrence. Therefore, the baseline should show a peak at the start of the business day diminishing to a lower level throughout the day.

It may also be necessary to take into account different geographic locations and work practices. For example, start of business on the East Coast is 3 hours ahead of start of business on the West Coast, so the load experienced because of users logging on to a centralized system will see different peaks with regard to users on the East Coast and West Coast.


The Dynamic Enterprise

The constant moving target of your environment is very important to the overall objective of application performance optimization. If everything remained static, you could just apply predetermined quality of service (QoS) or delivery criteria to the conversational flow, make the configuration and sizing considerations, and then walk away to the next site. Because of the dynamic nature of the environment, however, you must continually re-evaluate and monitor the delivery.

The dynamic environment factor creates an incentive to move the operational objectives from a reactive mode (wherein an organization reacts to the report of a fault or problem) to a proactive mode (wherein potential issues and problems can be recognized in advance and changes affected to ensure that the user performance perception is never impacted).

You must consider the fact that your monitoring system needs to change to accommodate fluctuations in traffic flow, user population, and new infrastructure. Some infrastructure changes may even have embedded intelligence (such as Remote Monitoring [RMON], Cisco IOS Service Assurance Agent [SAA], and so forth) to assist with monitoring the system.

Part of the performance assessment should always include an evaluation of what data has been collected. This should be validated (see the "Validating Changes" section later in this chapter for details) to ensure it still represents metrics relevant to your business objectives.

The constant re-evaluation of performance data relevant to business objectives and SLAs ensures the monitoring system in place will continue to provide value to the business model.


Diagnosing Problems

Proactive troubleshooting can provide valuable information to facilitate forward planning. The process involves the following steps:

Step 1. | Define what you are trying to achieve.
Step 2. | Build a virtual environment. (This is an optional step in the process.)
Step 3. | Apply traditional troubleshooting methodology (changes to model).
Step 4. | Validate (verify) changes.
Step 5. | Use results to draw conclusions. (Apply to production.)

Consider the benefits a virtual environment can offer when used to perform a large portion of application performance monitoring (APM). Such a tool, which replicates the behavior of the entire network, can benefit network managers, network planners, and support staff when used to diagnose difficult problems, validate changes before implementation, and plan for future scenarios (including growth and failure).

As the central nervous system that links all facets of an organization's operations, the IT infrastructure presents a complex performance management challenge: the constant measurement and analysis of correlated information spanning the entire organization. The primary reason for monitoring application performance is to be alerted immediately to performance abnormalities in the live environment. For example, perhaps dramatically increasing traffic volumes are stretching system capacity. This increase could happen as a result of a major news event related to the company, a widespread advertising campaign, or business growth. Immediate knowledge of abnormalities provides the opportunity to take corrective action before all users are affected and the issue grows more complex. Base alerting methods on the seriousness of the performance problem and automate corrective actions where appropriate (for example, rebooting a problematic web server or rerouting traffic to another resource).

True proactive application management begins by capitalizing on automated processes designed to diagnose, alert, and resolve common performance issues around the clock. In certain scenarios, however, it is more practical just to work on a problem away from the production network. In these situations, the use of a virtual environment that reflects your production system can provide a very realistic fault-diagnosis solution.

Several factors can degrade the performance of an application, including the following:

  • Increased server utilization

  • Decreased server performance due to new server OS

  • Increased traffic on the network

  • Degraded reliability of the network

  • Unfavorable protocol configuration

Diagnosing the problem involves identifying the cause of end-to-end performance problems (network, server, or the application) and validating proposed fixes to solve them.

It is wrong to believe that using a virtual environment will solve all performance issues. As in all cases, the first step is to define what you are trying to achieve. It is tempting when you have a hammer to view all problems as nails, but it is impractical to use the virtual environment for everything.

For example, diagnosing poor infrastructure performance in a virtual environment can help IT organizations avoid future problems, whereas diagnosing a specific application transactional slowdown is better suited to data analyzers and monitoring toolsets.

The troubleshooting methodology when using the virtual environment is the same as that in the production world (see Chapter 10, "When Applications Fail"), but with the additional assurance of meeting service levels and satisfying end users. For a tool to provide a virtual replication of your environment, it must be able to re-create normal operations of your environment.

Networks are built by interconnecting various devices (routers, switches, and so forth) running a variety of protocols. Protocols are configured with a number of parameters that enable them to behave in a certain way. Traffic patterns in a network are influenced by these protocol configurations. This environment lends itself to the virtual environment approach. For example, simple changes to routing configurations can significantly impact traffic patterns. You can see this impact by applying attribute changes to a virtual replica of the production environment. Therefore, the virtual environment makes diagnosing and troubleshooting complex network problems, such as congestion, both feasible and expedient.

When using a virtual environment to carry out this analysis and troubleshooting, consider the following two steps:

Step 1. | Create a unified view in the virtual environment visualizing the IP network topology, traffic paths, and protocol configuration (routing domains).
Step 2. | Calculate the routing and forwarding tables and data-plan restrictions and settings.

To obtain a unified view of the network, you must collect information from various sources and import this information into the toolset. This information can come from router configurations, element management systems, network management systems, performance management systems, traffic management systems, and so forth.

To define what information you will use to create your virtual environment, you initially must define the problem you are trying to solve. Although it is theoretically possible to re-create your whole environment in this virtual world, it is somewhat irrelevant to include sections or paths that play no direct or significant part in the problem area under investigation. Their inclusion will only slow down the process and potentially create confusion and divert attention away from the immediate issues.

By taking this information from your working environment, you provide the best opportunity to mirror your production system. It is also possible to build a virtual model from scratch using a toolset's inherent architectural library and default settings. However, aside from the time needed to build what could potentially be a complex system, this approach introduces the potential for error, which in turn could result in a less-than-accurate picture of your production system, rendering future diagnoses and predictions based on this model at best inaccurate and at worst invalid.

To define and understand routes and subsequent traffic flows, the virtual environment must be able to understand the network configuration. This is important because the toolset needs to replicate (as closely as possible) the activity within the production system. The next step, after having created the unified view, is to calculate the routing and forwarding tables, including the data-plane (infrastructure configuration) restrictions and settings such as access control lists (ACLs) and QoS settings.

Having created these paths for the application data, you can begin to import the combination of what can be termed application and background data.

Application data refers to packet-by-packet traffic, in which the virtual environment simulates each packet-related event (packet created, packet queued, packet transmitted, and so on). Application traffic modeling provides the most accurate results because it models all protocol effects. However, this also results in longer simulation runs. This traffic is generally imported from application trace files.

Background traffic refers to traffic that impacts the performance of application traffic by introducing additional delays and background load. It is represented as load, in contrast to the application data's packet-by-packet representation. Because each packet that produces traffic on the network is not explicitly modeled, using background traffic can speed up simulations considerably. The methodologies for importing this data are specific to each solution, and you should refer to the relevant vendor for details.

The use of the background and application data in your model will be determined by the problem definition that you isolated prior to building the virtual environment, when you defined what you were hoping to achieve. For example, you may be troubleshooting a specific transaction on a server located in San Diego with a client in Australia. You would essentially use application data for the specific transaction, taking into account the server architecture and applications in San Diego, but also include background traffic for the specific links and services that make up the conversational (or potential) journey between Australia and North America.

At this stage, you should have a virtual representation of your production environment, or at least the part that you are investigating. You now apply the traditional troubleshooting methodology as you would to a production system (see Chapter 10). Figure 9-1 shows the creation and troubleshooting flowchart.

Figure 9-1. Creation of Virtual Model and Troubleshooting Flowchart

[View full size image]

In addition to providing the means to diagnose and rectify performance issues, the virtual environment provides the means to carry out predictive and what-if scenarios. (See the "Capacity Planning" section later in this chapter for details.)

If used correctly, the virtual environment occupies an important part of your overall management strategy. For optimum use, the toolset should be able to replicate your system architecture, import configuration details from your production environment, and represent (and import) a combination of application and background traffic.


Validating Changes

Any change to the overall delivery mechanics, either following an optimization or after a proposed change (such as new architecture, infrastructure failure testing, application or system upgrades, and so forth), requires validation.

Validation is the process of checking that the change or introduction of a new element (such as a new application, additional users, and so forth) had the impact you predicted. Figure 9-2 shows a validation process.

Figure 9-2. Validation Flowchart

You begin by defining what the impact will be. This ultimately determines whether (and how) the change is implemented. You can make this prediction in several ways, as follows:

  • A combination of common sense and vendor specification Simplistic as it may seem, this is your most important tool in validation. You should have an opinion on where the perceived impact will have a negative or positive effect. Your common sense and intuition may come from your experience with the real system or with the technologies at hand.

  • Crude mathematical calculations This boundary analysis is a simple calculation that you expect the impact to definitely be lower or higher than. For example, the user population increases by 10, so the theoretical load on the link to the server will increase by a magnitude of 10.

  • Your virtual environment This is always the preferred method because it provides you with an accurate baseline as a reference point.

Regardless of the approach you choose, validation is a key step before using results to draw conclusions. Validation is a step that is generally performed repeatedly during the course of application optimization as you continue to add enhancements.

By verifying fundamental behaviors of your delivery system with each change, you can maintain a high degree of confidence and easily identify which particular changes are responsible for new, unexplained behavior. In contrast, if you make many simultaneous changes before examining the system's behavior, you will then be unable to determine which changes are responsible for your observations, and to what extent each change has contributed.

The next procedure in the validation process is the comparison between expected and realized behavior. Measurement is the most common approach to validation and the one most people think of first. This can be done only if the real system, or some portion of it, is actually accessible. If the system is not currently in place, you can use the pilot approach (as discussed in Chapter 6, "Application Deployment"), in which part of the system can be observed in a controlled environment.

The measurement tools you choose should be those currently being used to measure the delivery, as described in Chapter 8, "Monitoring the Delivery." It is important that you use these installed tools and metrics because this validation process will also form part of the verification of your SLA and collection points.

If all is well, your expectations will meet the real performance results. If the results fail to meet your expectations, you must review the expectations and, if necessary, adjust the model or calculations accordingly. Making individual changes rather than many changes simultaneously is the most sound strategy. By so doing, you enable yourself to identify (and explain) what the significant changes were in optimizing system performance.


Enabling Technologies

You must fully understand your end-to-end delivery system to comprehend end-to-end approaches for diagnosing problems, validating changes, and planning ahead for growth and high availability. Enabling technologies available are as follows:

  • Application performance troubleshooting The diagnosis of application performance problems. Such troubleshooting includes a methodology (see Chapter 10) as well as data taken from the management system (see Chapter 8).

  • Configuration validation The validation of all system configurations (not in isolation, but a whole delivery-system view). Configuration validation identifies errors that result from misconfigurations or protocol conflicts. Products such as OPNET Technologies NetDoctor provide a means to check key configuration details of a delivery system.

  • SLA compliance analysis The fulfillment of performance results against performance requirements. One of the main objectives of an SLA is continuous improvement. Certainly the early detection and rapid resolution of issues improves operational efficiency. However, these activities are still reactive in nature. The service must already be unacceptable (via the SLA threshold) or have begun deteriorating (detected by intelligent baselines) to trigger action. If the service is in a steady but inefficient state, it will not be noticed. An SLA compliance analysis should provide a mechanism to quickly discover these inefficiencies and identify opportunities for improvement. In addition to tracking performance against a static SLA threshold, it is important to understand how current performance compares to past performance. User expectations are set by their previous use of an application.

  • Failure analysis The assessment of an application failure. This includes root-cause analysis (the identification of the most probable cause of the problem) as well as steps or processes to rectify and ensure the situation does not reoccur. For further details, see Chapter 10.

  • Discrete-event simulation (DES) A simulation technique that models a system dynamically as a set of interacting entities. DES mimics your applications and network explicitly; it evaluates each change, each packet transfer, each retransmission timeout, and so forth. For example, an application conversation is simulated on a packet-by-packet basis. DES should be used to simulate specific interactions relevant to the end-to-end model, not an overview of the whole process. DES represents an important part of the end-to-end story because it provides valuable information for both capacity planning and the verification process.

  • Hybrid simulation A combination of DES and analytical modeling to provide estimates of link (or virtual circuit) use and network latency. Simulations are fast. This is important because it enables you to complete complex studies within a reasonable time.


Scenario Modeling

In Chapter 6, the subject of predictive modeling with a virtual network environment as a means to understanding how applications will manifest themselves in your production environment was introduced. Replicating the behavior of the entire network, including its routers, switches, protocols, servers, and individual applications, is valuable in the proactive planning arena. Working in the virtual network environment, network managers, network planners, and operations staff are more effectively empowered to diagnose difficult problems, validate changes before they are implemented, and plan for future scenarios (including growth and failure).

The delivery network is like a complex, semi-organic machine that requires calibration and fine-tuning to perform at optimal levels under all conditions, especially those involving higher-than-normal traffic patterns. Like a racecar engine, it must not splutter and stop when its fuel-intake valve is open any more than a website should hesitate in response to a few dozen (or a few thousand) visitors. The performance monitoring solution has outfitted the network with data capture points that provide the infrastructure engineers with the information they need to respond to problems when they occur, and hopefully to keep them from occurring in the future. Scenario modeling provides a means for the same engineers to build and apply changes to the network in a virtual environment to determine the network's optimum performance.

By integrating these two critical needs (data capture points and scenario modeling), the IT department will be equipped with the tools to ensure application and network performance from the end-user perspective, under all conditions.

The key to useful performance analysis is to extract relevant information from the network, estimate the impact on user productivity, and provide actionable results. Although most everybody agrees on this objective, there is a wide range of solutions to this problem.

Network performance deals with parameters such as latency, packet loss, and available bandwidth. Analysis of user experience, on the other hand, should focus on delays that impact user productivity.

To understand the performance implications of a new application, you could build a virtual representation of how the application will behave upon deployment over the network. If there is one particularly useful vantage point for establishing the model, it is the network.

Specialized tools, such as Compuware's Application Expert and OPNET's ACE, enable you to capture application-level transactions on the network and determine how much of the measured response time can be attributed to the client, server, network, and so forth.

You cannot always directly measure the delays experienced by an application across a network path (for example, an application in early deployment or pre-rollout). In these scenarios, a model can assist with the understanding of the impact to your environment. In addition, it can assist in the assessing and setting of SLAs.

A model must calculate the delays, but you cannot know the exact amount of congestion or packet loss without specific measurements. Therefore, a serious approach to modeling must include both an analytical model and measured, collected data. Using a model in a simulated environment and armed with knowledge of performance metrics, some statistical tools, baseline data, and an application model, you can deliver a valuable assessment of the performance issues related to application deployment. The goal is to achieve initial performance levels and maintain user satisfaction throughout the life cycle of the application.

A simulator is a piece of software that generates test conditions that approximate the actual system. By creating a model of the network topology in conjunction with actual network snapshots (packet traces), you give the simulator a basis for approximating how the application will perform over the existing network. The simulator mimics the traffic flow and queuing dynamics present in the existing network.

With the appropriate input data and application of sound techniques, it is possible for the simulator to show the network dynamics before and after the new application is deployed. With this information, you can determine what network upgrades are needed and the implications of not upgrading the network. In the case of a change to a deployed application, you can determine infrastructure changes required (including geographical server location) to continue to meet the performance criteria.

The virtual environment represents a more cost-effective approach than trying to replicate the entire enterprise-wide system in a lab environment (see Chapter 6), and the use of a modeling tool proves a delivery system's functionality.

It is important to remember that you need to focus on the systems under question. Your objective is to build a model for use in the simulator software that is equivalent to a real system, existing or proposed. However, equivalent is a subjective term that must be defined more precisely. Equivalence means that your model behaves in some sense like the real system. Nevertheless, for practical reasons, models are usually limited to representing only certain aspects of the system of interest. Therefore, it is up to you to define equivalence for your modeling project.

The model must answer the questions of interest. You want to use the model to help you study a particular set of issues. You need to define those issues clearly before you start your modeling effort. Knowing which questions are important will enable you to exercise good judgment in including or omitting certain features in your model. Employment of simulation requires extreme care if an accurate result is to be achieved. The following are some of the challenges involved with simulation:

  • Building a model that accurately reflects current network dynamics

  • Accurately representing the projected network traffic in the model

  • Understanding the limitations of the simulator in use and the limitations of simulation in general

Creating a virtual representation of your situation gives you an environment from where you can understand and track changes to your baseline model, a reference point from which to validate any proposed changes to the environment. A beneficial side effect of this model is that you now have an environment away from the production system where you can trace persistent issues.

Using the model, you can understand how the applications interact with the infrastructure and you can make changes to improve or resolve problems. You can then simulate these effects in this nonproduction environment before making recommendations to changes to the production environment. This is a methodology that is recommended for some of the more complex situations. You would not want to resort to the model to resolve or study every issue that occurs on your system, such as issues that relate to infrastructure of software faults. A model will always replicate behavior as it is expected to operate. Therefore, a virtual environment will not help you in any situation in which a system is effectively breaking the rules.


Capacity Planning

Although some view capacity planning as an art form, you can approach it scientifically. To properly analyze a distributed environment, view it as a series of connected components (computers, peripherals, software, networks, and so forth) and break these down into their individual, measurable parts. For example, client machines are composed of CPUs, memory, software, busses, and peripheral devices. Each of these can be monitored for its effect on the entire configuration's total performance. As with the old saying about a chain being as strong as its weakest link, a distributed environment's performance is often determined by its weakest (or busiest) component.

Capacity planning is an active process of measuring and monitoring networked resources. Its purpose is assuring appropriate infrastructure capacity levels to support existing and future business functions. Capacity planning provides both IT and business benefits. It can reduce resource expenditures in today's complex infrastructure environments, improve application infrastructure availability, and help align IT with business priorities. Capacity planning, when implemented as a continuous process, becomes a business resource capable of helping ensure business continuity. In addition, capacity planning gives IT a mechanism for providing business managers with insight into the technology resources on which they depend. Understanding capacity issues can put IT and business management in control of optimizing resource allocation, ending complaints from end users and business departments, and stretching the IT budget.

Capacity intelligence enables prioritization and reallocation of existing applications, systems, and network components. For example, applications with lower priorities can be allocated lower-cost resources, such as lower-cost bandwidth, which can delay or eliminate the need for additional bandwidth expenditures.

Capacity planning also projects future resource needs, protecting the infrastructure from slowdowns as capacity demands increase. Capacity usage patterns can pinpoint underutilized infrastructure areas while predicting where additional resources will be needed. Resource management through capacity planning improves end-user satisfaction. For example, IT can proactively implement necessary resource upgrades before the infrastructure becomes stressed, improving the quality of business service delivery. Utilization forecasts and time-to-capacity reports can ensure sufficient time to procure resources, which is a less-expensive proposition than a last-minute rush to correct overutilization problems.

Altogether, capacity planning's vision of the future helps ensure required application delivery while improving the company's bottom line. Active capacity planning is a useful tool for all levels and functions of IT as well as business management.

An organization's need for capacity planning is driven by its need to cut infrastructure costs while maintaining the levels of service required by the business. Everyone in the IT management organization has a need for accurate capacity-planning information. CEOs want sufficient infrastructure capacity to accommodate the corporation's future, CFOs want to identify excess capacity and deploy resources efficiently, CIOs want to meet committed IT service levels, and IT operations managers want to size the infrastructure appropriately and cost-effectively through activities such as server consolidation. Those with direct capacity-planning responsibility want to provide accurate, timely information to help the IT department make the best decisions possible as it strives to balance spending and service levels.

Capacity planning is an extension of performance management. Ideally, infrastructure information flows from performance management's real-time capabilities through to capacity planning's long-term planning capabilities, balancing the needs of the business with the IT budget as the process unfolds.

At its simplest, capacity planning involves monitoring the infrastructure, analyzing the gathered data, and then acting upon it. For capacity planning to be most effective, it has to draw data from across the infrastructure, from network devices, servers, storage arrays, databases, and applications. Capacity planning is virtually impossible to do without a solution that automates the majority of this process, and ideally allows both real-time monitoring of the infrastructure for immediate, current capacity issues as well as traditional tracking and historical data capture.

The capacity-planning process includes the following five phases:

Step 1. | Monitor, measure, and track resource utilization.
Step 2. | Analyze utilization information.
Step 3. | Determine and implement actions.
Step 4. | Define real-time alerts.
Step 5. | Seek continuous improvement through both real-time monitoring and long-term planning exercises.

The initial phase involves the collection and creation of the baseline. Data is collected and processed into usable information from all infrastructure components. Consumption and performance is measured, historical trending is performed, and infrastructure capacity baselines are built.

Information generated in the first stage is used to analyze capacity's impact on both current operational needs and planned services. Excess capacity is identified for reallocation, and resources under stress are marked for corrective action. During this second phase, planners gain an understanding of application use and impact on infrastructure capacity, enabling them to make decisions mapped to business priorities.

For planned application rollouts, this stage is the place to model application capacity needs and to look at what-if scenarios to predict performance, availability, and the true infrastructure cost of the application.

During the third phase, necessary infrastructure resource actions to support current business goals and upcoming rollouts are determined and implemented. Taking action to support future services during this phase helps ensure that resources are purchased in a timely, cost-effective manner.

Capacity thresholds for the improved infrastructure are put into place in the fourth phase. Doing so harnesses the power of capacity-planning automation, to raise awareness of problem areas before internal service levels are compromised (proactive planning).

Capacity planning is not a one-time activity to leverage the potential of the infrastructure for the business; capacity planning needs to be seen as an ongoing activity with real business impact. To be most effective, capacity planning must have both real-time and long-term planning components. This can be considered the fifth phase.

Real-time monitoring and management through an automated capacity-planning solution removes both the guesswork of wondering whether the infrastructure is properly supporting applications and the potential surprise of service levels that fail to meet expectations, creating outages and brownouts. Iterative capacity exercises ensure the continuous availability of resources into the future.


Reviewing SLAs

Besides the straightforward contractual value of an SLA, the service level management (SLM) process provides a feedback mechanism to ensure the applications your business relies on continue at a high level of performance. This is essential to protect your investment in your application and infrastructure.

Constant monitoring and re-evaluation help you continually optimize and improve application delivery. However, you need to know what happens if your SLA goes bad and what the penalties are. This is a long-debated point.

A good SLA is critical to the success of any managed service. As a customer you might want the SLA to cover every conceivable risk, but the provider (internal or external) will only guarantee the services over which it has direct control. (For example, e-business covers what lies inside the firewall.) A good SLA should do the following:

  • Express a clear statement of purpose

  • Describe the services and their duration

  • Outline availability and performance criteria

  • Describe payment terms and termination conditions

  • Describe anything that might invalidate the contract, such as exceeding the agreed-upon number of supported users or applications

Taking into account the particular needs of the client organization, the agreement should reflect the fact that companies do not adopt managed services just to improve internal processes, but to monitor software that tracks and logs the transactions and metrics that have been set as critical to the business.

There is much more to SLM than SLAs. Many companies have service agreements with more than one supplier, as well as internal service agreements. The coordination of service providers should be made the responsibility of a senior manager within the company, because each supplier's responsibilities need to be coordinated with those of other suppliers. Identification, notification, and resolution among suppliers also must be managed. Certainly in these cases it is advisable to use a monitoring service to get early notification that users are experiencing difficulties.

After an application has been rolled out, maintaining service levels presents a challenge. Factors that impact service levels over time include the following:

  • User relocations

  • Application software upgrades

  • Increased demand on shared infrastructure from other new or existing applications

Because it is impossible to micromanage the user population, the ability to track performance indications over time is absolutely essential if you intend to manage and maintain service levels. The same tools that were used to baseline the network for application deployment also enable you to manage performance trends over time.

The tracking of network trends greatly assists in your ability to plan and justify infrastructure upgrades. Even with the ability to track trends, you must also be able to pinpoint performance problems as they arise and localize them to a specific cause (server, network, client, and so on).

The metrics identified in Chapter 8 provide the key to proactively identifying bottlenecks. When setting thresholds relevant to these metrics, it is important to set the trigger thresholds at a level below that of your expected SLA. The reasoning behind this is to ensure that should performance start to go outside your parameters, you can be alerted, make an assessment, and rectify the situation before it impacts your users.

Penalties become relevant in a number of situations. The first is the traditional model of an outsourcer providing a service to the organization. The second is more of an internal SLA. The IT department is essentially responsible for delivering the service to their user population. These users become their clients. In an effort to quantify performance as well as give some tangible target to measure achievement, the internal IT department creates SLAs by which they measure the service delivery. They need to be specific about when to expect payback for a service level violation (at the moment of violation or after a period of time).

Penalties are put into an SLA to give force to an agreement. Although there must be some penalty for nonconformance, just imposing a financial sanction is not always beneficial. Few IT managers would happily walk into a board meeting at the end of the fiscal year and report, "We constantly failed to deliver on the SLA, but we got this check back from the outsourcer."

On the same theme, it is equally important to set realistic, achievable service levels: The ultimate goal is to deliver the application, not trick the service provider into paying a penalty.

If you set up an SLA in conjunction with good network design, you should never have to invoke penalties. However, penalties should have teeth so that everyone has an incentive to make sure you never have to invoke them.

Penalties should be viewed as a way of securing commitment, not punishing the offender. They should be considered attention getters. When writing penalties, always put performance of the network over the financial rebate. Ultimately, you require a service that runs smoothly and performs reliably.

Rebates should be in line with losses suffered in their business; however, quantification of such a loss is often difficult to determine. This is essentially why the constant monitoring, and the drive toward proactive monitoring, is of the utmost importance. The ongoing evaluation of a system's performance in line with your predetermined metrics will facilitate performance maintenance.

This section's aim is not to provide a template for penalties; after all, many factors influence each individual case. However, in brief, a penalty should be designed to provide comfort that when the delivery process fails, the penalty is strong enough to ensure positive action on part of the service provider and not just something that can be written off the bottom line.


End-to-End Delivery

In today's marketplace, because the Internet has become an integral part of the distributed enterprise, the effect of QoS on the mission of the enterprise and on the applications that drive it is more critical than ever. QoS can affect every server, client, network, and application in every sector of the IT industry. The problems inherent in this area are neither trivial nor short term, yet at the same time the industry is demanding immediate solutions with guarantees for QoS levels.

One of the major challenges in this regard relates to the distributed IT paradigm and the requirement to provide QoS guarantees for end-to-end delivery of information. Performance management has to be viewed from an application perspective, and application transactions need to be tracked end to end through the entire infrastructure, including client devices, routing protocols, network configurations, server architectures, and other components. Integrated solutions that address this end-to-end requirement will have a big impact.

End-to-end management is the process of coordinating resources through the use of tools, such that the following goals can be achieved from the desktop through the network to the server:

  • Appropriate planning for growth

  • Sound operation

  • Minimal administration

  • Improved QoS (user-impact values)

  • Ease of evaluation/status of the network, desktop, and server

  • Minimal unscheduled downtime

Figure 9-3 shows an end-to-end system, a consolidated view of the entire conversational process.

Figure 9-3. Logical End-to-End System

Performance monitoring software systems have many capabilities that are fundamental for IT managers. Finding problems after they occur is difficult, expensive, and time-consuming. Users are rarely tolerant of slowdowns or failures when they impact their productivity. Traditionally, network managers have approached this issue in a piecemeal fashion. One management tool would outline how a desktop computer was functioning, a second would measure the efficiency of the router, a third would determine various parameters of the server, and a fourth would outline how a database is managing queries. Although this approach produces useful insights, it does not provide a comprehensive picture as to how end-to-end performance is impacted. The variance in equipment standards from different vendors obfuscates the process, making it more complicated and time-consuming. Tools that incorporate these disparate sources of information into a single report with an inclusive picture of the network are very valuable.

It is important to have an end-to-end view. If you cannot easily determine where the delay is (server, client, network), you cannot expect to find the root cause of the performance issue. Subsequently, without the ability to isolate performance problems within the end-to-end path, troubleshooting and optimization efforts are just guesswork.

To assign responsibility for a particular performance issue to the relevant group, it is imperative that performance management initiatives present a cooperative and generalized focus on the entire picture. This can be achieved by providing a unified end-to-end view. To provide the best optimization possible to your delivery model, you must be in a position to view the whole picture; application performance on the client and server must be examined together with the network performance to control the quality of performance delivered to the end user.

Starting Points to Building an End-to-End View

Traditionally, performance management vendors have approached the end-to-end process from the following three distinct functional areas:

  • Programmers insert markers or breakpoints within applications so that they write data to a file or communicate with another program at each crucial step to be monitored.

  • Infrastructure managers and systems collect all or selected parts of the traffic on the network via statistics on the infrastructure, correlate requests with replies, and infer application performance based on packet analysis.

  • Application managers install agents/probes at specific points in the conversation journey (as discussed in Chapter 8). The agent looks at application profiles and is either told or is able to successfully infer what to measure.

The advantage of the programmed breakpoint is that the original programmers know precisely what the right events for accurate and comprehensive instrumentation are, and there should be no doubt that the measurement is correct. Furthermore, the measurement can be as granular as needed. Applications that are not amenable to meaningful performance measurements based on generic measurements can be measured and controlled only with the custom-programmed management approach.

The disadvantage of this approach is that very few client/server or n-tier applications have been instrumented in this fashion. Even enterprises prepared to devote skilled programming resources to develop custom management solutions are out of luck if the application was not created with management hooks and application programming interfaces (APIs) in the first place. The other primary disadvantage of a custom-programmed performance solution is that skilled programming resources are rare, costly, hard to control, and usually slower than expected.

The traffic-inference approach has the big advantage of not requiring programmatic intervention in the target application. Probes or agents deployed on the local network close to the user may also be the simplest approach in terms of the number of computers and connections that must be modified.

Although traffic inference provides a relatively accurate view in isolation, the biggest disadvantage of traffic inference is a limitation to only the least common denominators of application performance. Specific application behaviors may not be detected (for example, multitiered server activity). Furthermore, performance bottlenecks in the client itself may not be detected.

Continuous monitoring of network delay is also essential for optimization and planning activities. There are several common methods for measuring network delay. Active methods include scheduling Internet Control Message Protocol (ICMP) pings or simulating TCP session connects. Passive methods include measuring TCP session connects or more general application packets. Of each of these methods, network delay measurements based on observing general application packets provide the most accurate representation of performance.

It is important to understand the network delay components to appreciate the merits and limitations of each approach. The network delay consists of the following five components:

  • Serialization

  • Queuing

  • Propagation

  • Processing

  • Protocol

Serialization or transmission delay is the time required to put all the bits in the packet on the transfer medium. Table 9-1 shows serialization delay's dependence on both the packet size and the link access rate.

Table 9-1. Round-Trip Serialization Delay by Byte Size

Packet Size

56-kbps Link Access Rate

256-kbps Link Access Rage

1.5-Mbps Link Access Rate

64 bytes

18.3 ms

4.0 ms

0.7 ms

1500 bytes

428.6 ms

93.8 ms

16 ms

TCP session connects involve 64-byte packets. As a result, measurements based on TCP session packets will generally underestimate the network delay experienced by the rest of the application. ICMP pings can be configured to assume any size, but the packet size is always the same in both directions. Most applications do not have this symmetry, which makes it difficult for ICMP to accurately capture the serialization delay experienced by the application. Note also that the default ICMP packet size is 64 bytes.

Queuing delay is the time the packet waits in a buffer for its turn to be transmitted. It depends on the serialization delay for the packets served ahead, the dimension of the buffers, the amount of congestion, and the configuration of the router or switch scheduling policies. Congestion can change dramatically in microseconds, but a TCP session may be open for seconds or hours or even days. Therefore, the queuing delay experienced by the TCP session connects can be significantly different from that of the main application. The same is true for any scheduled probe such as ICMP; the queuing delay even 60 seconds earlier may bear little resemblance to that experienced by the application. In addition, the router or switch may place ICMP packets in a special queue for preferential (either better or worse) handling.

During periods of congestion, ICMP packets may be preferentially dropped while the application packets wait, and therefore ICMP never measures the true delays. ICMP packets may be preferentially moved to the head of the queue and experience shorter delays; they may be selectively moved to the rear of the queue and experience longer delays (unless dropped).

Propagation or distance delay refers to the time it takes the packet to travel along the physical path. It is dependent only on distance and the type of medium. If the TCP session connects and ICMP packets travel the same physical path as the main application packets, the propagation delays will be identical. However, it is not guaranteed that the same paths will be traversed; if they are not, the ICMP measurement is rendered irrelevant.

Processing delay refers to the time it takes the router or switch to prepare the packet for delivery. Processing delay depends on a wide variety of factors, but it is normally insignificant. Note that TCP session connects may require more processing than the remaining packets in the flow, and ICMP packets may require less processing.

Protocol delay refers to the time the packet waits because of underlying protocols. In a shared medium, for example, the packet must wait for its node to acquire access. The effect of this delay varies greatly with protocol.

In summary, network delay measurements based on ICMP pings only reveal the delay experienced by ICMP pings at that moment in time. Network delay measurements based on TCP session connects only reveal the delay experienced by 64-byte packets at the time the session was established (seconds, hours, or even days ago). Of each of these methods, passively observing general application packets is the most effective means for measuring network delay because it reflects what users are actually seeing.

Although it should be stressed that the network agent approach is the best starting option (because this is most likely to provide you with indicative performance metrics that can be tracked and represented relative to business requirements), the ultimate target is to cover each of three key areas (client, network, and server) and provide collaboration between the collected data to present a unified view reflecting the performance on an end-to-end basis.

Monitoring the End-to-End System

Following on from that approach, the logical first step (assuming network monitoring is in place) is the client station. Desktop monitoring provides a diagnostic starting point for latency troubleshooting. The moment an unmet service level threshold is triggered, IT organizations can execute a health check of the business infrastructure to look for specific network and application failures.

There are two ways to measure the round-trip transaction response time experienced by end users. Synthetic transaction (active) measurement involves simulating transactions from a designated machine at defined intervals. Desktop (passive) monitoring involves deploying agents on desktops at the end user's location. These agents watch events or packets without slowing down the transaction.

In both cases, because monitoring is conducted from a user location, the response-time measurement accounts for all the intervening components of the infrastructure. Server, database, and network monitors effectively make assumptions for the client wait and processor times.

Active tools generate synthetic transactions from dedicated robot PCs, so service level measurements are taken from designated computers of known and constant configuration. The advantage is you can capture diagnostic data at the time a service level exception occurs, without having to re-create the problem to identify the cause. With passive measurements, independent agents are deployed on many representative desktops in a corporate network to validate the end-user experience for Windows, web, and other applications such as SAP. Figure 9-4 illustrates a combination of active and passive agents interacting with a central console.

Figure 9-4. Passive and Active Agents Reporting to Control Console

In Figure 9-4, the passive agent is placed on a user terminal (1). This agent will monitor the application performance at the user's desktop as the user makes requests and receives information from the server. The data recorded from this agent will also include things such as the user wait time. This is the time between requests that the user takes to digest the information returned on the screen before moving on. In the case of a robot agent (2), these wait times will be eliminated, because the agent has no need to understand the response for the server. It only needs to know it received one before moving on to the next request. Both systems will return common statistics regarding the application's performance to a centralized console (3) for reporting and correlation.

Both approaches have pros and cons. The active agent gives an uninterrupted view of the transaction and can potentially identify issues out of hours and as a result alert support staff to take action before the general population is affected. However, it has the disadvantage of creating additional load on the system resources, and takes additional hardware to run. If synthetic agents are used to repeatedly run the same transactions, the results may be cached either by the client or the server. This caching effectively invalidates the results because it is not representative of the real user experience. If the server is caching the information, it cannot be selectively disabled. If the transactions are randomized, the main benefit of synthetic agents (their determinism) is lost. This selective caching effect can render synthetic agents inaccurate for measuring server response time. Passively monitoring server performance for all transactions and all system users eliminates these problems and can provide a useful baseline for future performance. The passive agent has the disadvantage of being affected by human intervention; it is also reporting after an event (after the user has an issue).

Most organizations would benefit from a mix of agent technologies to monitor the user base. This also gives the advantage of being able to correlate performance between the robot and real users and in doing so assess whether the transaction monitoring is accurate and indeed reflecting the end-user experience.

The other important thing to consider is the physical deployment of such technology. It is impractical to deploy agent technology to each and every one of your desktops. After all, the logistics of deploying and keeping the relevant agent current (OS and so forth) and running may prove overly burdensome. (Remember, you never want to implement a management system that is in place for the sake of management; ultimately the aim is to improve application delivery.) More importantly, even if you could deal with the logistics and support issues involved, the sheer amount of data created would be excessive, and once again render the management process impractical.

The server side is very well covered, and visibility into specific application counters on each server is equally an important part of the overall process. They will provide insight as to how the business-critical applications are performing on the relevant server.

Having monitored the conversational journey, you are in a position to view and understand the various aspects of the application's journey and are in a position to present a single unified view of the business transaction's performance if required.

Figure 9-5 shows a simplified view of a network system. Clients exchange data across the network with the three servers (web server, application server, and database server). Performance statistics are collected at specific points on the application's journey, such as the client station response time (at the client), the WAN utilization (at the network), and server statistics such as delivery statistics, database utilization, and server performance (at the server). Although this data represents parts of the complete journey, it effectively is taken from a single-point perspective (such as client, network, or server). The data is then amalgamated in a single centralized database. Individual functional areas (for example, network or server teams) can view data relevant to their requirements (such as network performance or server load) from this centralized database; alternatively, this information can be passed to associated management systems such as a policy manager or business service management system for inclusion in additional reports or analysis.

Figure 9-5. Combined Monitoring Systems Cover the End-to-End View

[View full size image]


Summary

Proactive planning involves the constant evaluation and monitoring of your overall delivery system. By constantly reviewing and ascertaining delivery criteria of your business-critical applications, you will be in a position to identify potential issues before they seriously impact your production environment. This process of constant evaluation provides valuable data in the capacity-planning process. Capacity planning is an active process of measuring and monitoring networked resources. Its purpose is to ensure appropriate infrastructure capacity levels to support existing and future business functions.

In certain scenarios, it is more practical just to work on a problem away from the production network. In such situations, the use of a virtual environment that reflects your production system can provide a very realistic fault-diagnoses solution.

In typical corporations, individual IT departments are highly focused on network infrastructure, systems, databases, or applications. Viewing the delivery system from a centralized end-to-end position will assist in understanding the complete application journey. By doing so, you can determine where the delay is (server, client, network, and so forth). This assists in locating the root cause of performance issues. Without the ability to isolate performance problems within the end-to-end path, troubleshooting and optimization efforts are just guesswork.


Reference Used in This Chapter

  • OPNET TechnologiesSimulation Technology Options, http://www.opnet.com/support/home1.html


Chapter 10. When Applications Fail

This chapter includes the following topics:

  • QoS troubleshooting tools

  • Diagnostic tools

  • Troubleshooting methodology

  • Identifying responsibility

  • Resolving performance problems

  • Redefining delivery requirements

  • Redefining service level criteria

In the majority of enterprises today, it is rare to see a complete application failure. What normally occurs is that an application fails to perform to end-user expectations, with a resulting loss in productivity and frustration for the user.

Performance problems are seldom caused just by overloaded WAN or LAN links. Application transactions depend on many outside influences, and any one discrepancy in the process can cause the whole system to crumble. You need to profile the application (from a network point of view) to understand the client/server traffic necessary to complete the transaction.

This understanding of the process helps to close the gap between application developers and support staff. When they see the interaction between their applications and the infrastructure components and protocols, they then can work together to expedite resolution, eliminate the majority of finger pointing associated with cross-group issues, and (probably most importantly) provide baseline information. Armed with this information, developers can then scale their applications for deployment and provide network architects with realistic figures to design and upgrade their underlying infrastructure.

To manage faults, you need to discover, isolate, and correct problems. This chapter discusses how the user can employ Cisco best practices to facilitate fault discovery using the system monitoring commands to isolate problems with the system test commands and to resolve problems with other commands, including debug commands.

This chapter then introduces a simple troubleshooting methodology through which you use this information.

It covers the importance of identifying and assigning responsibility for problem resolution, discussing the general default approach of determining which area is responsible for resolving the performance degradation, and allocating responsibility instead of using predetermined metrics (as previously defined in Chapter 3, "Detailing the Business Transaction," and Chapter 8, "Monitoring the Delivery").

The chapter then moves on to discuss the capture of the application transaction as it traverses the infrastructure, defining application and architecture along the way, and discusses relevant capture points and approaches.

Finishing off, the chapter explains why the observed performance figures will ultimately not be improved. It discusses the review process required to redefine the delivery requirements and subsequently adjust expectations and measures.


QoS Troubleshooting Tools

As well as understanding and troubleshooting the application and infrastructure, you must also understand and, if necessary, troubleshoot quality of service (QoS) settings (for example, identify whether Layer 2 switches are discarding 802.1p tags or Resource Reservation Protocol [RSVP] packets are traversing RSVP-enabled routers). The tools and methods used for network management are fundamentally different from those used for network troubleshooting. Management tools usually depend on the operation of fundamental network services, such as packet routing and Domain Name System (DNS). Troubleshooting tools are for use when things are degrading.

Network Monitoring

Network monitoring solutions, sometimes referred to as framework systems because they can monitor the delivery framework from a network perspective, have been used for many years and are designed to monitor very large, heterogeneous networks 24 x 7 to provide the following:

  • Event management

  • Up/down status of servers and network connections

  • System logs

  • Network interfaces

  • Basic system performance

These solutions are used to determine system availability or to alert if performance thresholds have been exceeded.

Application Monitoring

Application-monitoring solutions vary widely in their coverage and depth. Some application monitors are designed to monitor all the interdependent parts that make up an application stack to provide event management (alerts and so forth) and store real-time and historical information. Others are designed to monitor specific parts of the application stack, often the database and network layers.

Although there is some overlap between framework and application monitoring, particularly with network and basic operating system monitoring, the purpose of application monitoring is to show availability by the following:

  • Application (by grouping all the pieces together)

  • Depth of application collection

  • Historical application performance

If a server or network connection fails, if a critical error appears in an application log, or if an object manager fails, the administrator will know which applications are affected.


Diagnostic Tools

Diagnostic solutions are launched as needed to diagnose performance problems of a specific layer or software component in real time. These solutions take frequent data collection samples of a broad array of metrics over a limited period of time. Diagnostic products are not intended to operate over long-term intervals. For example, data analyzers are limited by the size of their capture buffers. If you are looking at importing large amounts of traffic information, such as an entire day's worth of traffic on your LAN, or traffic going out to the Internet, use an application-monitoring tool. Network sniffers, application tracing products, debugging tools, and database diagnostic products are categorized as diagnostic tools.

Toolsets are divided into the following two groups within diagnostic tools:

  • Command-line interface (CLI) tools

  • Graphical user interface (GUI) tools

CLI Tools

The CLI tools category includes the following tools:

  • Cisco Service Assurance Agent (SAA)

  • Management Information Base (MIB)

  • Cisco IOS show commands

  • Cisco IOS debug commands

  • Data analyzers

SAA

SAA measures performance by sending one or more synthetic packets to a target IP device or a Cisco router. The packets are echoed back to the sender, similar to the functionality of ping. SAA uses the time stamp information to calculate performance metrics (such as jitter, latency, and response time). The processing delay in the source router is removed, which improves measurement accuracy. SAA can also assess packet-loss statistics with operations that send out multiple packets.

A target router that is running Cisco IOS Software can be configured as an SAA responder, which processes measurement packets and provides detailed time stamp information. The responder has intimate knowledge of Cisco IOS processing, so it can send information about the target router's processing delay back to the source SAA. This delay is removed during calculation to further improve accuracy.

Service level agreements (SLAs) often include class-based performance guarantees. SAA can be configured to monitor per-class traffic over the same link by setting the differentiated services code point (DSCP) bits.

An SAA operation can be scheduled to run at any point in time, or continuously over any time interval. SAA provides a proactive notification feature with a Simple Network Management Protocol (SNMP) trap. Each measurement operation can be configured to monitor against a preset performance threshold. When this threshold is crossed, SAA generates an SNMP trap to alert management applications. This feature can monitor the actual performance against what was promised in the SLA and notify the administrator of potential service level violations.

To further expedite problem resolution, SAA can be configured to automatically run a new operation when the threshold is crossed. This feature, combined with hop-by-hop measurement capability, allows for immediate real-time problem analysis when a problem arises. Measurement results can be retrieved with SNMP or from the Cisco IOS CLI.

MIB

In the world of routers, network management is based on the concept of a Management Information Base (MIB), a database of manageable items. In a typical MIB, you will find the system name, IP routing tables, and counts of the packets handled by each protocol, as well as counters and statistics specific to the performance of QOS.

The most popular protocol in use to manipulate MIBs is SNMP. A second version of SNMP has been developed. SNMPv2 is part of the Internet Standard Network Management Framework version 2 and is accompanied by nearly a dozen attending documents. RFC 1441 introduces the entire framework. Version 1 MIB corresponds roughly to version 1 SNMP and is documented in RFC 1155. MIB-II, the most widely implemented MIB, is documented in RFC 1213. RFC 1902 discusses version 2 MIB structure.

The MIB model is quite powerful. SNMP is a quality protocol, and many MIBs have been written for QoS settings. The most used MIB when troubleshooting QoS-related issues is the class-based QoS MIB, which provides read access to QoS configurations. This MIB also provides QoS statistics information based on the modular QoS CLI, including information regarding class map and policy map parameters.

This class-based QoS MIB is actually two MIBs: CISCO-CLASS-BASED-QOS-MIB and CISCO-CLASS-BASED-QOS-CAPABILITY-MIB.

It is not the intention of this book to list all available MIBs. Apart from the sheer number of them, they are also subject to change and enhancement, quickly rendering the text obsolete.

To locate and download MIBs for selected platforms, Cisco IOS releases, and feature sets, use the Cisco MIB Locator found at the following website:

http://tools.cisco.com/ITDIT/MIBS/servlet/index

In case the Cisco MIB Locator does not support the MIB information that you need, you can obtain a list of supported MIBs and download them from the Cisco MIBs page at the following website:

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

show Commands

Cisco IOS show commands are among your most important tools for understanding the network status and isolating problems in your enterprise. They are essential in almost any troubleshooting and monitoring situation. The following show commands will provide details on your QoS environment[1]:

  • show class-map Displays all class maps and their matching criteria, or a specified class map and its matching criteria.

  • show policy-map Displays policy configuration.

  • show policy-map [policy-map]class [class name] Displays the configuration for the specified class of the specified policy map.

  • show policy-map interface [interface name][vc][vpi/][vci][dlci dlci][input | output] Displays statistics for each class within a policy on an interface.

  • show queue[interface name] Displays weighted fair queuing (WFQ) configuration/statistics.

  • show interface fair-queue (On virtual IP [VIP]) for WFQ.

  • show interface[interface name] Displays interface statistics.

  • show queuing Displays queuing strategy per interface.

  • show queuing interface[interface name] Displays weighted random early detection (WRED).

  • show ip rsvp interface Displays RSVP-related interface information.

  • show call-rsvp-sync conf Displays configuration settings for RSVP synchronization.

  • show call-rsvp-sync stats Displays statistics for calls that attempted RSVP reservation.

  • show frame-relay fragment Displays information about the Frame Relay fragmentation taking place in the Cisco router.

  • show traffic-shape queue Displays information about the elements queued at the virtual circuit (VC) data-link connection identifier (DLCI) level. Used to verify the operation of IP Real-Time Transport Protocol (RTP) priority over Frame Relay. When the link is congested, voice flows are identified with a weight of zero. This indicates that the voice flow is using the priority queue.

  • show frame-relay pvc[dlci] Displays information such as traffic-shaping parameters, fragmentation values, and dropped packets.

  • show traffic-shape Displays shaping configuration.

  • show traffic-shape statistics Displays number of delayed packets statistic.

  • show interface fairqueue Displays information and statistics about WFQ for a VIP-based interface.

  • show interface random-detect Displays WRED on VIPs.

  • show interface rate-limit Displays conform/exceed statistics/configuration.

  • show access-lists rate-limit Displays committed access rate (CAR).

  • show ip nbar protocol-discovery Displays the statistics gathered by the Network-Based Application Recognition (NBAR) protocol-discovery feature.

  • show ip nbar port-map Displays the ports assigned by the user to the protocol. If no ip nbar port-map command has been used, the show ip nbar port-map command displays the default ports.

For full details and specific platforms, consult http://www.cisco.com and search on the specific platform number and the word show (for example, 7200 show commands).

debug Commands

Cisco infrastructure includes software to aid in tracking down problems with the network, server, or other network hosts. The privileged EXEC command debug enables the display of several classes of network events, such as emergencies, alerts, error warnings, and notifications. The debug command can provide a wealth of information about the traffic being seen (or not seen) on an interface, and error messages generated by nodes on the network.[1]

Many debug commands are available. Their usage and effectiveness depend entirely on the specific environment they are operating in. Use debug commands with caution. In general, it is recommended that these commands only be used under the direction of your router technical support representative when troubleshooting specific problems.

The following list details some of the more useful debug commands when determining issues relevant to your QoS configuration:

  • debug frame-relay packet Displays process-switched packets that have been sent on a Frame Relay interface.

  • debug frame-relay fragment [event | interface | type | number | dlci] Displays event or error messages related to Frame Relay fragmentation; it is only enabled at the permanent virtual circuit (PVC) level on the selected interface. The following list describes the Frame Relay fragmentation debug command options:

    - The Optional Event field displays event or error messages related to Frame Relay fragmentation.

    - The Optional Interface field displays fragments received or transmitted on the specified interface.

    - Enter the interface type for which you want to display fragments received and/or transmitted in the Optional Type field.

    - Enter the interface number in the Optional Number field.

    - Enter the DLCI value of the PVC for which you want to display fragments received and/or transmitted in the optional DLCI field.

  • debug priority Displays priority queuing output.

  • debug ppp multilink fragments Displays information about individual multilink fragments and important multilink events.

  • debug ip rsvp Displays RSVP messages. The following list describes the RSVP debug command options:

    - all RSVP messages for all categories

    - api RSVP application programming interface (API) events

    - database RSVP database debugging

    - dump-messages Dump RSVP message contents

    - fast-reroute RSVP fast reroute support for label-switched paths (LSPs)

    - filter RSVP filter information

    - handles RSVP database handles events

    - hello RSVP hello events

    - messages RSVP messages (sent/received via IP) debugging

    - msg-mgr RSVP Message Manager events

    - path RSVP PATH messages

    - policy RSVP policy information

    - proxy RSVP proxy API trace

    - rate-limit RSVP rate-limiting messages

    - reliable-msg RSVP reliable messages events

    - resv RSVP RESV messages

    - routing RSVP routing messages

    - signaling RSVP signaling (PATH and RESV) messages

    - snmp RSVP SNMP events

    - summary-refresh RSVP refresh and bundle messages events

    - timer RSVP timer events

    - traffic-control RSVP traffic-control events

    - wfq RSVP WFQ events

Enabling debugging can disrupt operation of the router when networks are experiencing high load conditions. This is because the router gives priority to output of the debug messages. Therefore, if logging is enabled, the access server may intermittently freeze up as soon as the console port gets overloaded with log messages.

Before you start a debug command, always consider the output that the command will generate and the amount of time it may take. For example, if you have a router with one basic rate interface (BRI), debug isdn q931 probably will not harm the system. However, doing the same debug on an AS5800 with full E1 configuration will probably generate so much input that the system may hang and stop responding.

By default, the network server sends the output from debug commands and system error messages to the console. If you use this default, monitor debug output using a virtual terminal connection rather than the console port. Possible destinations include the console, virtual terminals, internal buffer, and UNIX hosts running a syslog server. The syslog format is compatible with 4.3 Berkeley Standard Distribution (BSD) UNIX and its derivatives. Logging to the console produces very high overhead, whereas logging to a virtual terminal produces less overhead. Logging to a syslog server produces even less, and logging to an internal buffer produces the least overhead of any method.

Before debugging, look at your CPU load using the show processes cpu command to verify that you have ample CPU available. For example, if you have a Cisco 7200 router with an ATM interface doing bridging, depending on the number of subinterfaces configured, rebooting the router might use a lot of its CPU. The reason is that for each VC, a bridge protocol data unit (BPDU) packet needs to be generated. Starting debug commands during such a critical time may cause the CPU utilization to rise dramatically and result in a hang or network connectivity loss.

Note that when debug commands are running, you do not usually see the router prompt, especially when the debug is intensive. However, in most cases, you can use the no debug all, undebug all, or u all commands to stop the debugs.

For details and instruction on using debug commands, refer to the following document:

http://www.cisco.com/en/US/tech/tk801/tk379/technologies_tech_note09186a008017874c.shtml

Data Analyzers

Data analyzers (sometimes known as network or protocol analyzers) are a group of tools that capture and interpret the network traffic between two or more connected computer systems. The traffic can then be decoded so that it is possible to see what processes are occurring. Examples of tools in this category include Network Associates Sniffer, Compuware Application Vantage, OPNET ACE module, and Ethereal.

GUI Tools

GUI tools operate and deliver results in a user-friendly graphical format. Tools in this category include the following:

  • QoS Policy Manager (QPM)

  • 24 x 7 application monitoring (such as Compuware Network Vantage)

  • Modeling (such as OPNET IT GURU)

These tools are aimed at viewing and controlling elements within the conversation path.

QPM

Cisco QPM is an enabler of Cisco end-to-end QoS for enterprise networks. QPM delivers differentiated services across network infrastructures for converged voice, video, and data applications. QPM can be described as a complete system for centralized QoS monitoring, policy control, and automated reliable policy deployment.

QPM provides information enabling you to do the following:

  • Validate QoS settings and results with traffic analysis

  • Get real-time and historical reports for QoS troubleshooting

  • Control roles and privileges for policy view, modification, and deployment

Policy-based QoS management with CiscoWorks QPM makes it possible for enterprises to deploy consistent QoS controls. It also proves helpful for allocating bandwidth limitations for heavy-use applications. For example, an enterprise could use QPM to segregate time-sensitive, high-priority traffic. With QPM, the organization can prioritize business-critical functions during peak hours of operation.

24 x 7 Application Monitoring

Various solutions are currently available to address the needs of performance management for enterprise environments. These systems are capable of collecting, storing, and presenting data from network devices and servers. The web-based interface on most products makes the performance data accessible from anywhere in the enterprise. Examples include Compuware Network Vantage suite, Concord Network Health, and InfoVista Vista view.

An evaluation of these products will determine whether they meet the requirements of different users. Some vendors support integration with network management and system management platforms. For example, Compuware supports server agents to provide key performance statistics from application servers.

Each product has a different pricing model with base offering capabilities. (Consult the vendors for specific information.) Support for performance management features for Cisco devices such as NetFlow, RMON, and Cisco IOS Service Assurance Agent/Response Time Reporter (SAA/RTR) is available on some solutions. Concord also has support for Cisco WAN switches that can be used to collect and view performance data.

The SAA/RTR feature in the Cisco IOS Software can be used to measure the response time between IP devices. A source router configured with SAA can measure the response time to a router or other destination IP device. The response time can be measured between the source and the destination or for each hop along the path. SNMP traps can be configured to alert management consoles if the response time exceeds the predefined thresholds.

Observation of the availability of the interdependent components of the network, operating system, applications, and application integrations implies that the application should be functioning normally. However, these observations are indirect ways of measuring application performance. System components can look perfectly healthy even when an application becomes unavailable.

Only the measurement of user experience (taken from an end-to-end perspective) can provide the complete availability picture.

Indirect measurement is useful in determining performance and trends and in monitoring CPU, memory, and other resources that affect current and future performance and capital planning expenditures. It also enables administrators to go back in time to determine the state of the environment when a business transaction degraded or became unavailable.

The main things to look for when troubleshooting QoS issues are dropped packets and network bottlenecks that can cause delay and jitter. Performance can be measured for real-world scenarios through the configuration of SAA operations that are executed periodically. Performance metrics measured include round-trip response time, connect time, packet loss, application performance, and interpacket delay variance (jitter). The SAA enables you to receive notifications and perform troubleshooting and problem analysis based on these metrics.

Performance issues can be identified by observing the following conditions:

  • Interface drops

  • Buffer drops

  • Interface congestion

  • Link congestion

Each interface in the conversational path should be examined, and drops and congestion should be eliminated. In addition, round-trip delay should be reduced as much as possible. Pings between the endpoints will give an indication of the round-trip delay of a link. The round-trip delay impact is going to vary relative to the profile and requirements of your application. (See Chapter 3, "Detailing the Business Transaction," for details on profiling, and see Chapter 4, "Service Level Management," for details on assessing requirements.) For example, a voice call should not exceed 300 ms whenever possible. If the delay does have to exceed your predetermined value, efforts should also be taken to ensure this delay is constant, so as not to introduce jitter or variable delay.

Verification should also be made to ensure the Cisco IOS queuing mechanism is placing the application packets within the proper queues. Cisco IOS show commands, such as show queue [interface-name] or show priority, can assist in verification of queuing.


Troubleshooting Methodology

Now that you are familiar with the various tools and approaches, you need a troubleshooting methodology. The establishment of a set of clearly defined steps helps you think a problem through and hopefully prevent your jumping to conclusions. As your experience on a system grows, the time to run through the methodology also decreases.

Step 1: Define the Problem

Just as with defining an application's characteristics, the first step of troubleshooting a poorly performing application is to define the problem to be solved. Although some problems are identified based on complaints, many are identified only from observation over a period of time. The scenario in this section deals with the complaint situation, but the rules and methodology can be equally applied to long-term trending and proactive troubleshooting.

When presented with a scenario, it is easier if you can define clear boundaries for the problem. Noting all the symptoms and considering the associated causes for the problems should enable you to set these boundaries. Defining these boundaries narrows down the area to search for the most probable cause.

The best way to start defining these boundaries is by asking simple questions that highlight the who, what, and when of the problem. The following is a typical line of questions:

  • Has this issue always existed?

  • If not, how long has the problem been present?

  • Are all your applications affected?

  • Is this a new application?

  • If yes, has it ever worked?

  • Can you connect to any other systems?

  • Are all the users affected in a specific area?

  • Have any other sites reported similar issues?

  • Does the problem occur at any particular time of day?

  • Has anything changed?

The answers to these basic questions pinpoint which users are affected, how long they have been affected, and on what systems they are seeing the issue. With this information, you can focus analysis and data collection.

Step 2: Gather the Facts

In many ways, the data collection has already begun. It begins with the simple questions that enabled you to define your problem. The next stage is to collect some information from the network itself.

This gathering exercise may well be repeated over a smaller area or with different tools as you zero in on the problem area. In the majority of cases, the zeroing would be done via a data analyzer. The performance statistics collected in the network monitoring process (see Chapter 8) should already have provided you with an overall view of your system. Now use this big-picture data to view the details of the transaction from your recently defined probable area of concern.

There are two approaches to this information-gathering exercise. As discussed, you can use historic data to identify problem areas, or you can initiate a new collection of this big-picture data, effectively refreshing the databases of gathered data.

An alternative to re-collecting is to refine the filters on the devices to reduce the amount of visible data. In general, collect as much information as possible (within the boundaries of your target objectives, remembering it is counterproductive to the management system to collect data that is not pertinent to the overall delivery of your business-critical applications).

Surplus information can be discarded at a later date. The point is, the more you can see, the greater evidence you have, and the easier the case is to prove. It is worth noting that any information you gather here will be useful not only for the immediate problem, but as a baselining exercise for the network. It also shows you how applications (and protocols) manifest themselves in your network environment.

Understanding application behavior patterns is an important part of troubleshooting. Recognizing both normal and abnormal behavior patterns will expedite speed to resolution exponentially. In addition, understanding the complexities and dependencies of an application will give you a pretty good idea of what is likely to break in various scenarios.

Most applications (or protocols) have distinct characteristics that leave plenty of unique signatures across a network. These may be as simple as a predicted pattern of behavior (such as keepalive frames every 10 seconds) or an individual bit (flag) within a packet that identifies that packet as being the start of a login procedure. The programmers can set the identification and assignment of these signatures during the development phase of the application. Picking one of these points to act as an identifier is up to you. Whatever helps you identify normal or abnormal behavior is acceptable.

For example, Figure 10-1 shows an application window using Compuware's Network Vantage product. The view shows the Open Shortest Path First (OSPF) routing protocol as the top application. Consider this in context; OSPF is a link-state protocol. (This is a routing protocol that only sends a table update following a link-state change. It will also send out, by default, a complete database of routes every 30 minutes.) Under normal circumstances, you would expect to see some OSPF traffic (small exchanges between peer routers), but the size of this interaction indicates a routing issue on the network.

Figure 10-1. Abnormal OSPF Behavior

[View full size image]

It is important to point out that the data should always be viewed in context. Figure 10-1 provides a fairly easy example, because the traffic load was high, but it should always be viewed in relation to the expected network load as well as the time interval you are viewing. For instance, an Oracle traffic load of 35 GB is very high for an hour interval, but not bad when aggregated over a week's viewing.

Now review the application SQL Server. In Figure 10-2, it can be seen to be suffering from extreme response issues. Review the timing of this response issue. Referring back to the peak in routing traffic, the assumption can be made that this response hit resulted from a network integrity issue rather than an application server-related issue. It is important to remember that networks contain multiple paths and devices to ensure no single points of failure exist. A link failure not directly in the path of a conversation can adversely affect the entire routing infrastructure; and without this overall view of the world, it may be missed.

Figure 10-2. SQL Response Issue

[View full size image]

Step 3: Assess the Data

Having gathered the data, you are now in a position to eliminate entire areas as irrelevant to the problem area. In doing so, you can effectively simplify the situation down to the most probable cause being the server, client, or network.

For example, if only one application traversing a shared path is affected, you can effectively eliminate the network as a probable cause. A potential next step is to determine whether all clients experienced the situation. If the answer is yes, eliminate the client as a probable cause and concentrate attention on the server, or permissions on a common gateway such as a firewall.

This is a simplistic view of a problem, but simple elimination speeds up the resolution of a problem because it enables you to focus on the issues. Be aware that if you have a preconceived idea while assessing the data, you may focus your attention on the wrong area. Indeed, you may find yourself collecting and reviewing data to suit your definition instead of viewing the data relevant to the performance issue.

Step 4: Make a Plan

The plan you create should evoke the divide-and-conquer paradigm, in line with the most probable cause theory discussed in Step 3. Consider the most likely cause, and formulate a plan in which only one variable is changed at any one time. For example, if the most probable cause has been identified as a particular server, the next conclusion might be that the server is suffering from an overloaded processor. Stopping all the applications on that server would not tell you which one process was catalyst (which application was overloading the server). In that particular scenario, each process should be stopped while the effect is reviewed. Remember to always keep plans simple. This way they will remain easy to implement and assess.

Step 5: Implement the Plan

Now execute the plan you created. In most cases, this will be the first instance of the resolution loop. In complex scenarios, a single change might not resolve all the issues. This is why it is so important that the plan is simple and you change one variable at a time. Strive to make continuous progress toward a smaller set of possibilities. When implementing an action plan, remember to stick to the script, and do not make changes without at least mentally going through the methodology. Failure to do so will only lead to confusion and could delay the ultimate resolution (as undocumented/unplanned changes are neither recorded nor considered).

Step 6: Observe the Results

Having conceived and carried out the plan, the next step is to observe the results. After analyzing the results, you must decide whether you have solved the problem.

The most definitive result of a solved problem is a happy user. The speed of resolution relates directly to the tool you select to observe the issues. You would never use a cannon to kill a mosquito. Neither should you use a focused data analyzer when trying to observe the big picture.

As mentioned briefly in Step 5, if the problem is not solved after the execution of the plan, you must take the information, add it to the data already collected, and refine your action plan.

In some cases, you may continue this loop with no resolution. Unfortunately, at some stage you will have to decide whether to escalate the situation either to a third party or to a vendor. Never be afraid to escalate the problem. The important thing is that by having gone through this process, you are very clear on what the problem is not, and the arsenal of data collected will only ultimately aid in the resolution.


Identifying Responsibility

When the application is running slowly, the network team will point the finger at the application guys, who will then point the finger at the database administrator (DBA), who in turn will find some other area to blame; all the while, the business is suffering because of the degradation of the application's performance.

Many IT organizations are left in the dark when it comes to determining the scope of a performance problem. It is difficult to find out how many people are affected, where the problem is occurring, how serious it is, or its duration. Because the severity of the problem is unknown, it is hard for IT management to determine the resources needed to solve a particular problem.

Fault Processing

Accepting that the objective is to become more proactive than reactive, the enterprise will still see a high proportion of faults reported via a central help desk.

Upon receiving the performance issue report, the help desk will seek to assign it to a group. In the majority of enterprises, the help desk will have something of a default route for this process, based on nothing more than a feeling that generally the designated default area resolves the majority of performance issues.

Upon receipt of the trouble report, the designated group will check their particular silo and report back no fault found, returning the problem to the help desk. The help desk is the point of most sufferance because they are the ones who have to go through this bouncing of blame among the groups until they can assign the correct group. Figure 10-3 shows the buck-passing flowchart. In this scenario, the time to resolve the performance issue is the sum of all the time spent in each of the support groups as each passes the problem in turn.

Figure 10-3. Traditional Parallel Fault Processing

[View full size image]

Reporting Hierarchy

If the help desk becomes the owner of the end-to-end view as illustrated in Figure 10-4, reporting will be centralized, and the help desk will be in a position to identify the most probable cause of the problem, enabling them to correctly assign the fault and thus expedite time to resolution.

Figure 10-4. Proactive Fault Processing

[View full size image]

The biggest obstruction to accepting responsibility is the failure to use common language. In typical corporations, individual IT departments are highly focused on network infrastructure, systems, databases, or applications. This level of focus, although necessary, is generally accompanied by reporting in terms familiar only to each group.

One of the problems this causes is that when a silo reports on issues or performance specific to their area, the report may contain information pertinent to another group but becomes lost. This is why it becomes important for groups to report on performance specific to the business requirements, making the report and data available to all groups. One simple method is to ensure all information is entered into a central trouble ticket and not conveyed in private e-mails or documents. As long as the delivery criteria has been defined in line with the business requirements and in consultation with the relevant delivery groups, it should be possible to report as such. (See Chapter 4.)

When identifying responsibilities, you need to analyze the workflow data gathered during the analysis phase of the project. To determine who should be responsible for the various areas in which issues have been observed, an understanding of who currently has responsibility (created from the original SLA documentation) is needed. Sometimes the policy will change the normal workflow to assign an individual area overall delivery responsibilityfor example, during a new application rollout or infrastructure upgrade.

You need to define a data framework that your collected metrics and monitoring tools feed into. This framework will then be used for reporting and assigning responsibilities. Using business models based on aspects defined in your SLAs (such as business-critical applications, locations, and so forth, as defined in Chapter 3), you can quickly and easily view application performance monitoring (APM) information that narrows the focus on your applications and environment.

CNS Delay Reporting

After a business model has been defined, the reporting function should allow more of a simplistic root-cause analysis. Ideally, this is divided into three broad areas of application performance analysis: client, network, and server (CNS).

The CNS breakdown should enable you to understand whether the transaction/application problems reside on the client, network, or server, facilitating the assignment of the correct group to work on the problems. Figure 10-5 and the list that follows show the steps in an ideal reporting hierarchy.

Step 1.
Identify a fault relevant to one particular application family, such as Oracle Financials.

Step 2.
Classify the fault against a specific transaction or transactions (for example, a client address search).

Step 3.
Further classify, isolating the problem to a particular user location or locations (for example, Accounts department in London).

Step 4.
Identify the most probable cause of performance degradation from the following:

- Client All clients are experiencing issues across all applications.

- Network All application transfers across the network are experiencing issues.

- Server All users accessing a single server are experiencing issues.

- Application All users accessing a server have issues with single application.

Step 5.
Assign the problem to the responsible area (such as client, network, server, or application).

Figure 10-5. Fault-Responsibility Reporting Hierarchy

How this CNS view is tackled depends directly on the nature of metrics and toolsets used in each environment. Remember the initial aim is not to provide detailed analysis, but just to point to the most probable cause of a performance issue, enabling the correct resources to be allocated to resolving the issues as soon as possible.

If a performance bottleneck exists in the back end of the network, network-centric monitoring tools will help identify the location of the transaction delay. Because most core networks are high speed and low latency (assuming that latency issues will be quickly identified by your monitoring infrastructure), it is logical to turn your attention to server/client response times. By viewing response times relevant to the ingress and egress points of your network, you can start to isolate which areas are experiencing the greatest delay, such as client, web server, application server, or database server. This should then indicate which group is best positioned to determine the specific cause of the degradation.

Figure 10-6 shows a simple CNS response-time evaluation. Number 1 shows the overall end-to-end time. This includes all elements of the application's journey, including client, network, and server times. This time can be broken down as follows:

  • Find the network time (2) by placing two application monitors on the network itself (located at points A and B, respectively). The time taken for the application data to move between these two points represents network time.

  • Find the client wait time (3) by subtracting the time the application data leaves network measurement point A from the time the response is seen at the client.

  • Find the server wait time (4) by subtracting the time the application data leaves network measurement point B from the time the response is seen at the server.

Figure 10-6. CNS Delay Simple Methodology

Procedures and checklists should reflect the decisions made in assigning authority/responsibility via the signature blocks (or electronic acceptance) on them. It is important to have the people who will be signing off on any forms also be involved in their creation.

The original assignment of measured metrics plays a crucial factor in distributing responsibility to the associated group. In addition, it is important that all parties be able to view the data and metric collection in a common format. (See Chapter 8.) This enables all parties to agree and discuss the various aspects involved in the transmission and ultimate delivery of the application.

When assigning responsibilities, your initial diagnosis does not need a great deal of detail; in fact, such detail may negatively impact resolution time. In addition, you do not want the initial receptor just to hand off the fault to the same group each time. This does happen, and over time the group becomes expert in proving the fault is not their responsibility; although this is not a completely wasted process, it adds unnecessary delay to the problem-resolution process.


Resolving Performance Problems

Performance of networked applications depends on complex interactions among applications, servers, and networks. IT organizations need a detailed, quantitative understanding of these interactions to efficiently and cost-effectively troubleshoot and deploy applications.

Effective problem resolution is all about being on top of business-impacting issues early enough to address and correct them prior to any negative effects on operations, customer relationships, or finance. For IT, this generally relates to problems that arise in the underlying technological infrastructure. Users of IT servicesinternal, external, and business partnersall can be impacted by unexpected errors or by a failure to appropriately identify and resolve unanticipated technology issues. Elongated problem resolution timeframes represent significant costs to the organization regardless of industry. These costs are very tangible in terms of professional staffing expenses required to solve technology issues; they are less tangible, but still significant, in terms of potential lost sales and poor company image when the results of IT problems impact beyond the corporate walls.

Users expect consistent performance and rapid resolution of problems, but they typically do not report performance problems until after a significant period of time. In this situation, IT groups are not notified before or at the beginning of a performance slowdown. This not only allows bad service to linger, it puts IT in the unenviable position of not knowing that problems are occurring. Silo-oriented monitoring tools have proven to be ineffective early-warning solutions for application service because they do not measure end-user response time or provide any cross-domain intelligence.

It is important to be able to visualize the transaction flow, both at the application level and network packet level, enabling you to both identify and diagnose the cause of application availability and performance problems.

The process of resolving a performance problem becomes difficult and time-consuming when a technician must manually correlate data from multiple, nonintegrated sources. Even if it were easy to correlate data, it would be impossible to understand where the problem is without seeing what was abnormal during the time of the problem.

Having the ability to isolate performance problems in all tiers of an application (client, network, server, and database) helps IT organizations improve their troubleshooting processes and become more efficient. This is generally referred to as the big-picture view. This is something of a high-level view that enables the engineer to understand how the application traverses the infrastructure. This understanding can be performed manually and will involve correlating the baseline data to perform the unified view.

Alternatively, the collected data can be unified using modeling tools such as the OPNET IT Guru product that will allow the baseline data to be fed into a virtual environment. (See Chapter 6.) Modeling tools will enable the engineer to view and understand the application flows. This in itself may be sufficient to understand and diagnose issues, but it is also important to understand the flow to be able to correctly position the more detailed data analyzers to collect relevant data.

An alternative approach is to use the monitoring toolsets. An example of such a tool is Compuware's Network Vantage. This provides the high-level view of the application conversation process, enabling the engineer to identify bottlenecks and issues in more of a real-time environment.

Although there are many varieties of applications, the most prevalent can be classified in two categories:

  • Transaction or task oriented

  • Interactive and multimedia

Transaction- or Task-Oriented Applications

An application is transaction or task oriented if the application executes finite tasks or transactions. A task or transaction is a basic unit of user activity within the context of an application. For example, reading e-mail, making a calendar entry, or obtaining records from a database are all tasks. The start and end of a task is clearly defined. E-commerce, web, and database applications are examples of task-oriented applications.

Interactive and Multimedia Applications

Interactive applications such as X Windows or Telnet do not have clearly defined transactions. A user can log in to an X terminal and type commands. The commands generate responses. The user acts on the responses by issuing further commands. Hence, the application interacts with the user, and user "think time" is involved in the activity.

Multimedia applications such as streaming video or audio generally have the bulk of traffic in a single direction (server to client). Remote systems management, on-line radio broadcasts (one way voice), and computer-based training programs are examples of interactive and multimedia applications.

Application Architecture

Applications are implemented in a variety of architectures, such as two-tier or multitier architectures. For example, e-commerce/web applications are typically three-tier architectures and include a client, an application server, and a database server. Military applications may span more than three tiers. Commercial applications such as medical imaging may have arbitrary architectures based on their implementation. The key to effective data collection is to understand the architecture and capture traces accordingly.

Capture agents and protocol analyzers look at each packet and decode the information contained in the packet. This information is stored in a file called a trace file or a packet capture.

Where to Capture the Trace

When capturing application traces, the capture device can be located anywhere between the application tiers involved. The nature of application conversation means that as long as the analyzer is placed at the initiation, termination, or transient path of the conversation flow, the trace will be captured.

The key for detailed analysis is to be able to collect trace data as close to the source as possible. Architectures and conversation flows relevant to capture points are discussed in detail in Chapter 4.

How Many Traces to Capture

Long-term monitoring is also important in identifying trends, changes, and so forth. The two factors that determine whether you should capture a single trace or multiple traces are latency and visibility.

If there is high latency between any of the tiers, traces should be captured at both ends of the high-latency link. If this is not possible (for example, the responsibility of the remote site exceeds demarcation lines) or if the tiers are separated by low latency, a single capture is adequate.

If all the packets or messages are visible in a single location, a single trace is adequate. If the packets or messages are not visible at a single capture location, multiple traces should be captured. All aspects of the transaction must be captured in the combined set of trace files. Traces that contain common information (information that appears in all traces, such as logical name or Layer 3 address and so forth) can be related and therefore merged. Wherever possible, you should collect multiple traces simultaneously.

When to Capture Traffic Information

The timing of the capture is an important issue. In general, follow these rules:

  • Be consistent Take measurements at the same time that you perform a packet capture. Latency may vary with time depending on the traffic encountered in the network. The timing recorded in the trace is a function of the latency on the network at the time when the trace is captured.

  • Watch for packet loss If there is packet loss in the network, or you suspect that there is a possibility of packet loss (say on the Internet), ensure that you repeat the measurements until you get a trace that is representative of this condition.

Capturing Traces on a Busy Network

If you want to capture a multitier application on a busy network, in which a middle tier (such as the web server) is receiving multiple client requests per second, you can use several strategies, as follows:

  • Capture the application when no other client requests are hitting the servers You can do this most reliably in a lab environment or during off-hours.

  • Configure a load balancer to create a temporary clear channel for your application You may be in a production environment where a load balancer distributes requests from multiple web servers. In some cases, you can configure the load balancer temporarily to distribute requests to all servers except one. You can then capture on the excluded server while only your test client is hitting this server's specific address.

    Depending on the load balancer, the configuration change may be as simple as changing the HTTP port on the isolated server to a different port. Many load balancers automatically determine that they cannot connect to this server on the standard port and seamlessly distribute requests to the other web servers. The test client can then connect to the excluded server using the new port number. With this configuration, you can filter on the excluded server's address when capturing the application.

    This is often the easiest way to capture cleanly in a production environment. Historically, users have been willing to use this temporary and potentially simple technique. One benefit is that the back-end tier receives other transaction requests during the capture. The primary drawback is that loads on the other servers may increase temporarily.

  • Filter on all tiers to obtain only the traffic associated with a single client request In this strategy, you concentrate on the interaction between the target client and associated server. This requires careful filtering and understanding of the application characteristics; otherwise, you may filter out pertinent data.

  • Capture traffic between the first and second tiers to ensure that you see only one client's traffic Analyze each tier individually, such as the interaction between the client and the web server. To achieve this, capture the traffic between the client and the tier that initially receives the client's request. This is often a good initial strategy; it enables you to identify any bottlenecks in the interaction between the first two tiers. If the largest component of delay is application delay at the second tier, you can analyze the remaining tiers using one of the other capture methods.

    You can combine these methods when analyzing a particular application. For example, the first method (capturing in a lab environment or during network downtime) results in an end-to-end capture of the application; however, you do not see the effects of other users simultaneously hitting the middle tier. You might want to compare these results with the results from the last method (focusing on client-side traffic) to see whether the middle tier's processing delay changes significantly.

Fault-Conditions Guide

It would be impossible to try and identify every possible fault condition that may be experienced. Each case depends on myriad criteria. However, some of the more common problem areas are as follows:

  • Network integrity issues This is a large subject area and can cover many issues, and the issues can manifest themselves in many ways, such as the following:

    - High response times across all applications (indicative of retransmissions)

    - Multiple paths to the same destination

    - High counts of routing and Address Resolution Protocol (ARP) traffic

    - Network transmission delays (due to drops or congestions in routing infrastructure)

    - High counts of Internet Control Message Protocol (ICMP) traffic

  • Overloaded client/server Overloaded nodes exhibit many response issues across all applications. What is occurring is basically a blockage at the session layer of the device (such as Layer 5 of the Open Systems Interconnection [OSI] seven-layer reference model). The information is getting held up as the system processes something else. The machine will just exhibit many response issues across all applications resident on that particular server.

  • Inefficient application Inefficient applications show themselves in delayed processing of requests. This is different from the session layer issues in the overload situation, because in general the speed onto the network is good. The delay is actually incurred by a loop, lock, or inefficiency in the query. A loop occurs when a particular instruction within the application causes the application to continually repeat the same process or tries to start a nonexistent process (such as querying a nonexistent table of data). Data can become locked by another query, meaning that when your query attempts to access the same data it is temporarily unavailable.

    Other inefficiencies can manifest as a query trying to search too big a database, or one that does not use the network media to its fullest extent by sending small, multiple packets out onto the wire. Each time the conversation changes direction (an application turn, see Chapter 3), the packets incur a network delay while traversing the network. Mildly chatty applications suffer over high-latency networks (such as WANs), whereas very chatty applications can suffer even over low-latency networks (such as LANs). Interactive applications (such as Telnet) tend to be chattier than noninteractive applications. This is sometimes shown better during a profiling exercise.

  • Dependency failure Last, but by no means least, dependency failure is probably one of the hardest to spot. If you understand your environment and how it interacts, you can spot discrepancies in your scenario. It has as much to do with what you do not see as with what you do see. For example, if there is no DNS traffic on the network, the host cannot resolve the address for its destination server. Even if the application is running well, a connection is not likely to occur. Because the initiating process of the conversation is to resolve a name to an IP address, the lack of such data probably means there is no connection to the DNS server. Equally as valid, the lack of spanning-tree (IEEE 802.1d) traffic in a LAN environment, coupled with high transmission times, might indicate multiple paths and perceived network latency by end users.

Take time to think through the connection process. This might actually mean profiling (in the case of complex interactions, see Chapter 4) or just understanding the characteristics of the basic communication methods.


Redefining Delivery Requirements

If during the process of profiling or even modeling, you can see there is no possible optimization that can be made to any part of the overall delivery system, you will be faced with the prospect of redefining the delivery requirements. This may involve something simple such as changes to general work practices (such as start time for shifts) or something as complicated as a complete architectural change to the base application.

The whole objective of this book has been to optimize the delivery, enabling the enterprise to make the most of its infrastructure to ensure the business-critical applications initially meet the business delivery requirements and ultimately continue to improve performance as the infrastructure keeps up with the business models.

However, eventually you will reach a point where optimization to the existing infrastructure will make no difference. Alternatively, the business requirements may be beyond the scope of your environment. For example, an application requires a latency of 2 ms to meet business requirements; however, the network is restricted to a WAN link with a minimum latency of 55 ms. Aside from physically moving the two locations together, there is little optimization you can do to improve the situation.

In these situations, you have the following relatively simple technical decisions to make:

  • Upgrade the infrastructure (such as hardware, links, and so forth)

  • Realign the delivery requirements

Business justifications will determine which choice is right for your particular environment. One choice is just to upgrade the infrastructure. In making your cost justification, you must consider items such as downtime for installation, retraining for support staff, and the actual cost of the new hardware itself. (Link-speed upgrades have less of a cost impact than support issues, for example.)

When considering the options, you may find that a small change in delivery expectations has negligible impact on the delivery requirements. For example, a specific business transaction has an overall requirement to be complete in 20 minutes, but the current infrastructure at its absolute maximum can only guarantee a completion in 20 minutes and 5 seconds. In this case, you must assess the overall impact to the business, which could conceivably be very slight.

You must analyze the baseline data and understand the specific workflows from an end-user perspective. After all, a significant peak in load (accompanied by a reduction in response time) may occur during specific periods of the day. For example, a peak might occur as all the office workers log in to the system between 8 and 8:30 a.m. Perhaps another peak occurs just after lunch. With this information in hand, you may find that you can consistently meet your delivery requirements by adjusting the client's work practices (for example, a staggered start in the morning and staggered lunch hours).

Your effectiveness in these situations is going to be determined by the granularity and accuracy of your baseline data. It is also going to be important to use some modeling tools (see Chapter 6) to help confirm your decisions.


Redefining Service Level Criteria

Following the identification and subsequent rectification of an application issue, you need to hold a debriefing to ascertain what, if anything, could have been done to prevent the incident. Part of that process should include a review of the SLA.

Chapter 9, "Proactive Planning," discussed imposing penalties for nonconformance. In that discussion, it was argued that very few network managers would willingly walk into an end-of-year review stating that performance had not met expectations, but they had received a check from the outsources. Following that argument through, the SLA is not there as a trick to deceive the supplier; its intention is to provide a tool to maintain application delivery. This is why it is important to consider the SLAs in place during the debriefing. The review needs to consider the following two things:

  1. Does the SLA reflect achievable delivery requirements?

  2. Are the data collectors and metrics correct to measure the SLA?

The first consideration should determine whether the defined business requirements are actually achievable. Can the existing infrastructure and associated configurations deliver the desired result? If the answer is no, you must redefine the delivery criteria (see redefining delivery requirements) and then redefine the SLA to match these new requirements.

The second consideration should answer whether your monitoring system is adequate. You might find through this consideration that your monitoring system is inadequate in proactively identifying and subsequently inadequate in providing enough data to be able to rapidly resolve performance breaches.

If during this process you decide that certain information is missing, revert to the methodology defined in Chapter 8 to ensure that the insufficiency in monitoring and reporting are subsequently corrected.

Figure 10-7 shows the flowchart associated with the SLA review.

Figure 10-7. SLA Review Flowchart

[View full size image]

This review process should be central to the problem-resolution process. It is not intended as a search to appoint blame to any particular department. It is intended to confirm the dynamic delivery environment is adequately understood and continually tuned to ensure continued business benefits from the deployment. Remember, the environment is not set in stone; it will continue to change. This review will assist in keeping pace with those changes.


Summary

It is rare to see an application fail completely; what you generally experience is a slowdown in performance that directly affects end-user productivity. When this occurs, you need to manage faults to discover, isolate, and correct problems. Many network management tools enable you to isolate problems. Such tools include CLI commands, application monitors, and data analyzers. In addition, it is important to assign responsibility for the fault resolution to the correct area (for example, network or server).

Following the identification and subsequent rectification of the application issue, you must hold a debriefing to ascertain what, if anything, could have been done to prevent the incident. Part of that process should include a review of the SLA.

This review process should be central to the problem-resolution process. It is not intended to become a search to appoint blame to any particular department. It is intended to confirm the dynamic delivery environment is adequately understood and continually tuned to ensure continued business benefits from the deployment.


End Note

  1. Cisco Systems, Performance Management: Best Practices White Paper. http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008014fbf3.shtml


References Used in This Chapter

  • Cisco Systems, Performance Management: Best Practices White Paper. http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008014fbf3.shtml

  • Cisco Systems, Service Assure Agent. http://www.cisco.com/en/US/products/sw/iosswrel/ps1830/products_feature_guide09186a0080087a78.html

  • RFC 1441, Introduction to Version 2 of the Internet-standard Network Management Framework, J. Case, K. McCloghrie, M. Rose, S. Waldbusser http://rfc.net/rfc1441.html

  • RFC 1155, Structure and Identification of Management Information for TCP/IP-based Internets, M. Rose, K. McCloghrie http://rfc.net/rfc1155.html

  • RFC 1213, Management Information Base for Network Management of TCP/IP-based internets: MIB-II, M. Rose, K. McCloghrie http://rfc.net/rfc1213.html

  • RFC 190, Structure of Management Information for Version 2 of the Simple Network Management Protocol (SNMPv2), J. Case, K. McCloghrie, M. Rose, S. Waldbusser http://rfc.net/rfc1902.html


Part V: Practical Implementations

Chapter 11 Business Aligning Case Studies

Chapter 12 Optimizing Application Delivery in Storage-Based Networking, Wireless LANs, and an End-to-End Model


Chapter 11. Business Aligning Case Studies

This chapter includes the following topics:

  • Case Study 1: Profiling the Business Transaction

  • Case Study 2: Defining SLAs

  • Case Study 3: Optimizing the Business Transaction

  • Case Study 4: Implementing QoS in a VoIP Environment

This chapter is composed of case studies that reinforce the material discussed so far in this book.

The first case study focuses more on the business side of the process and shows how to define and profile the business-critical application. It finishes up with a description of the characteristics learned and how they can be used in the ongoing process and future troubleshooting exercises.

The second case study shows how to take the information and relate it to produce a realistic service level agreement (SLA). It covers subjects such as metric definition, collection points, and thresholds to demonstrate good, poor, and bad performance.

The third case study addresses the optimization of a single transaction in a banking environment.

Finally, the fourth case study addresses the addition of quality of service (QoS) mechanisms to optimize the deliver of a voice-based application.


Case Study 1: Profiling the Business Transaction

This case study describes a business function for a bank and covers the applications and relevant transactions that are pertinent to its success.

Transactions can be defined as an application's processing modules that exhibit similar resource requirements, characteristics, and user expectations. Distinct transactions were chosen to reflect the methodology. This case study focuses on banking (teller-based) transactions; however, these types of transactions can be found in almost all enterprises and remain applicable across business and scientific applications.

By using a bank as an example, it is possible to determine all the necessary considerations for identifying, profiling, and characterizing the business-critical transaction. The first step is to define the bank's activities relevant to business applications. This will enable you to subsequently define criticality within the business model.

Defining Base Transaction

A bank's main business-critical functions are those transactions involved with serving the customers; therefore, a bank is a service organization. You should focus your attention on the customer-facing activities that the bank determines are most critical to its business model.

Categorizing Business Transactions

It is important not to microdefine; that is, you should avoid characterizing too many small details within the transactions. Wherever possible, keep the functions simple and easily defined. For this purpose, the bank's customer-facing transactions can be split into the following three main categories:

  • Deposit system Money deposited into the bank system

  • Statement inquiries Requested information about state of accounts and so forth

  • Portfolio transactions Multiple requests, including moving money from accounts, withdrawing from accounts, and so forth

At one time or another, you may have found yourself in a bank line with someone who is balancing his or her bank statement, applying for a car loan, and trying to find a missing check from May 1989, all at the same time. Each transaction has different resource requirements (and response expectations). It is accepted that many institutions have now converted these face-to-face situations to electronic banking using a browserlike format. For the purpose of this initial categorization, however, assume a human teller is performing the functions.

First, take the simplest transaction, depositing a single check into an account. For this example, consider the teller to be specialized in depositing transactions. The teller has the appropriate resources to process the deposit within a response time acceptable to the customer. Other tellers process different transactions, such as safety deposit transactions (very physical, employing various resources), bank statement queries (very process intensive, requiring a lot of data), and financial portfolio transactions (multiple process, compute intensive, requiring high priority). Figure 11-1 shows each of the tellers defined as specializing in separate functions. Because each teller now has a specific function, the customers can queue at a specific location to receive action on their requirements.

Figure 11-1. Functional Teller Queues

Each of these transactional areas has the following subcategories:

  • Simple Requiring no additional work

  • Authorized Requiring external authorization to finalize task, and therefore requiring additional conversations with other systems

  • Complex Involving many parts and systems, each needing different resources and requirements to be executed

So in categorizing the types of transactions from a functional view, you have started to define, at a very basic level, the resource requirements. The next stage is to take these transactions and apply a business value against them.

Applying Transactions' Business Values

To define a transaction's criticality to the business, you need to consider frequencythe number of users who require the transaction and how often they require it. The frequency model not only allows a simple calculation to determine financial value of a transaction, but also assists in determining which transactions are important to the customers (which transactions will ensure repeat business for the bank).

Starting from a single-transaction monetary function, you can assume the least value to the business model is the statement query functions. The next assessment is slightly different. On the face of it, depositing money into the banking system is of high value to the bank. However, this does not consider the size and frequency of the transactions. For example, depositing a birthday check from Auntie is unlikely to swell the bank's profits too much. The portfolio transaction may involve more transactions, but has the potential to both bring and remove money from the bank.

When considering the frequency, note that the portfolio activities might include functions of both statement queries and depositing. This overlap will most likely occur when defining all business activities.

The theory behind categorizing transactions is to keep the categorization and definition as simple as possible. You do not want to find yourself in the position of having to profile and categorize hundreds of transactions. This is counterproductive not only from a profiling aspect, but also from an optimization approach. You will generally find that transactions tend to share a lot of similar functions and requirements, thus allowing them to be grouped together.

The term function should really be used to define a combination of user expectations and criticality to the business model. The numbers included for each activity should account for specific activities only. The frequency should include not only the perceived number of transactions but also the number of users performing the transactions, type of transactions being executed, and location of the users performing the transactions.

The understanding of the frequency will also determine how the profiling and subsequent monitoring systems are set up. This should include both numbers of expected users as well as physical locations that transactions will be executed from (considering the client position as opposed to the server position).

Because very early on in this process the bank was able to define its customer-facing applications as business critical, and then further subdivide to simple, authorized, and complex, you can assign further business value by reviewing utilization and frequency figures. Because key transactions have been defined, profiling can take place. The assignment of a business value is used further down the track when optimization techniques need to be applied to the system.

Figure 11-2 shows the distribution of users, physical connectivity, and predicted access points to the system. It is assumed that all branches, sub-branches, and Internet users perform all transaction types.

Figure 11-2. Physical Connection and Sample User Distribution

Figure 11-2 shows different link speeds connecting back to the centralized database and web servers. This may well have an effect on the characteristics and subsequent requirements for the relevant transactions. Taking this into consideration, you must profile the applications from a sample of each connectivity condition. It is not necessary to take a profile from all the sub-branches because it is expected that requirements and characteristics will be consistent across all sub-branches.

Table 11-1 shows the number of tellers per function at each branch, and Table 11-2 shows the normal number of transactions each teller is expected to carry out per function each day.

Table 11-1. Number of Tellers per Branch

Branch 1

Branch 2

Branch 3

Deposit tellers

30

25

20

Statement tellers

5

10

10

Portfolio tellers

25

15

20

Table 11-2. Number of Transactions per Teller per Site

Branch 1

Branch 2

Branch 3

Deposit tellers

120

100

80

Statement tellers

50

75

40

Portfolio tellers

300

200

200

Capturing a Trace

The first step in profiling a transaction is to capture a trace. Select a task or set of tasks that represents a typical user or exhibits the performance characteristics of the task (for example, retrieving records from a central database). From a profiling perspective, the complex transaction provides too many variables to produce a typical transaction. Although this is not critical during this initial phase, it must be considered when defining user expectations and SLA measurements. The profiling will therefore address the three transaction types for a simple and authorized perspective. Expectations and SLA figures will subsequently be produced based on a combination of these profiles.

Note that although the transaction types have been identified and documented, they should not be set in stone. There is always room to review both profiles and expectations should it be felt that the recorded data no longer provides an accurate or direct business benefit. This is also relevant following the introduction of new users, updated software versions, and so forth.

So after having identified what now amounts to six transactions to profile, the data collection can begin. This is performed using capture processes (called capture agents or probes), which can be installed and run on local or remote machines. Upon activation, the agents collect packet trace information and send that information to the remote site for analysis.

This can also be accomplished with a variety of network analyzers (such as Sniffer, OPNET ACE, Compuware Application Vantage, and so forth) that monitor the traffic on the communications link and save a trace of the packets that they detect. To accurately differentiate delays caused by the network and the end devices in a production environment, it is sometimes useful to analyze the application traffic from multiple places in the network.

You can do so in two ways. The first is by merging multiple trace files of the same application to automatically resolve transaction durations and identify delays caused by individual components. The second involves manually reviewing and identifying packets as they traverse the infrastructure.

Where to Capture Traces

The location of these capture points is going to be determined by the most probable conversational path. Capture agents (or protocol analyzers) can be located anywhere along the transaction data path. As long as the capture point can see the initiation, termination, or transient path of the conversation, it can record and report upon the data.

If we take the path of the conversation from Branch 2, it can be assumed that the path will enter the HQ site through the router and then the common switch serving the server farm. Figure 11-3 shows the perceived path.

Figure 11-3. Perceived Conversation Path Between Branch 2 and the Servers

Place the capture points close to the sample user at Branch 2 and at the switch at HQ as shown in Figure 11-3. To see the interaction between the client and server, the port on the switch is mirrored to reflect both ingress and egress ports involved in the conversation.

When capturing the transaction, it is important to exclude extraneous traffic; for the profiling exercise, you are only concerned with the characteristics of the transaction.

When to Capture a Trace

Business transactions are made up of multiple applications, systems, and so forth. A transaction refers to a specific function performed by an end user, such as requesting a quote for a product. The transaction starts when the user calls up the quote screen and ends when the quote is received back on the user's screen. This is not to be confused with a single transaction on a server, which refers to a single query and response; business transactions can be made up of many of these.

To get a clean transaction, it is important to know when the user starts and stops the transaction. You can determine this by speaking with the user on the phone or, preferably, by watching the user and physically stopping the capture when the transaction is complete.

Which method you choose depends on your location and that of the capturing point. Both methods are valid. Once collected, the trace file must be filtered and tuned to represent a clean transaction.

When capturing the transaction, it is normally best to start the capture just before the user begins work on the transaction. As shown in Figure 11-4, this ensures that no data relevant to the transaction is missed.

Figure 11-4. Capturing Transactions

The capture process should start just before the transaction begins and should stop just after the transaction ends. The timing of the capture is an important issue. In general, follow these rules:

  • Be consistent Take measurements at the same time that you capture the trace. Latency may vary with time depending on the traffic encountered in the network. Remember that the aim is to produce a clean baseline profile of the transaction.

  • Watch for packet loss If packet loss occurs in the network, or you suspect that there is a possibility of packet loss (say, on the Internet), ensure that you repeat the measurements until you get a trace representative of this condition. Packet loss may sometimes be unusual, and your second capture may not exhibit the same behavior as the first one. Also ensure that you place the capture device so that you can detect this loss.

  • Watch for unusual latencies Repeat measurements until you obtain approximately the same response time. If you measure a response time that appears unusually long, it may be due to an unexpected protocol effect or a stray delay caused by some device. For example, the first time you FTP a file, the destination address may not be resolved. The Address Resolution Protocol (ARP) will attempt to locate the destination, and this protocol may add an additional delay that may not appear the second time you capture the trace. As another example, taking a trace at noon across the Internet may give you extraordinary delays as opposed to taking the trace at midnight. In general, you may choose to take measurements at any time; however, ensure that you do it a few times until you obtain a fairly stable measurement.

  • Record the total capture time Record the start and stop time of the capture using your stopwatch. Note the response time of the transaction. You can use this information for subsequent validation of the imported data.

Cleaning the Trace

After collecting the data, you can remove from the trace the leading and trailing data not relevant to the transaction. This then gives a clean view of the transaction, and you can begin to understand its traits and characteristics. If you are capturing a trace of an Oracle transaction, for example, you might not be interested in broadcast traffic or in queries to non-Oracle servers.

NOTE

At this stage of the characterizations, it might be essential to perform some troubleshooting on the performance of the transaction. (For details, see Chapter 10, "When Applications Fail.") If this is the case, the optimized transaction should be reprofiled before moving on to assign SLA or delivery expectations.

Reviewing the Trace

Once cleaned, the next stage is to visually investigate and initially study the behavior of the application data. When reviewing the trace data, try to answer the following questions:

  • Does the application packet trace look like you expected it to look?

  • Does the amount of traffic look accurate?

  • Are the packet sizes what you expected them to be?

  • What is the general direction of traffic (client to server or server to client)?

  • Can you see the data in the data fields? Is it encrypted?

When you are happy with the collected data, you can begin to identify some of the traits and characteristics that will assist in tracking the transactions, from a performance perspective.

Remember that at this stage you are not looking to troubleshoot this transaction from a performance perspective; you are only looking to identify characteristics and traits. Therefore, you are looking to visualize the transaction in graphical form. The theory behind this is twofold. First, you will be able to understand the application characteristics that will subsequently allow it to be optimized for transmission across the network infrastructure. Second, you will have a known working picture of the transaction, ultimately enabling expedited speed to resolution of upcoming performance problems.

Merging Existing Network Statistics with This Application Transaction

By just comparing the known optimized transaction's picture against the fault scenario, you should be able to easily identify areas of discrepancy that will ultimately aid in identification and resolution of the performance problem. Figure 11-5 shows a single transaction in a normal operational mode (baseline) and the same transaction experiencing a performance issue. A comparison of the trace files leads to several observations, including the following:

  • Changes in overall transaction response time or file size

  • Identification of specific component responsible (for example, network, client, or server)

  • Relative percentage the component has contributed to the overall response time

  • Changes to the compositions of queries (such as graphic location or arrangement, or number of transmission errors such as retransmitted frames)

Figure 11-5. Comparison of Working vs. Broken Transaction

[View full size image]

This approach enables you to quickly determine discrepancies and identify possible areas for optimization.

The areas of interest for this profiling are as follows:

  • Overall transaction time Provides a representation of the time as experienced by the user (from the time the user presses the Enter key until the user receives a response at his or her station from the server) and thereby helps you understand user expectations.

  • Application architecture (two tier, three tier, and so forth) Assists in the tracking phase.

  • Frames and bytes per transaction Helps you understand the load on the network, and can be tracked for fluctuations.

  • Bursty or constant nature Assists in optimizing the delivery of this transaction.

  • Average bytes per frame Helps you understand the load on the network, and can be tracked for fluctuations.

  • Number of turns per transaction Assists in optimizing the delivery of this transaction.

  • Transport protocol used Assists in optimizing the delivery of this transaction.

  • Server address Defines which server the client communicates with, and is used in monitoring and, potentially, optimizing.

  • Specific repeatable queries Assists in monitoring by looking for any standard queries that will remain constant throughout any of the same transaction groups.

  • Port numbers Defines which ports the application communicates with, and is used in monitoring and, potentially, optimizing.

  • Data transferred (including payload data and overhead) Assists in understanding the load on the network, and can be tracked for fluctuations.

As you can understand from the preceding list, some of the characteristics are used in defining and tracking metrics relevant to the SLA, whereas others are used in understanding the nature of the application (to effect the best delivery mechanism to ensure efficient delivery of the transaction). For example, the data transferred can be actively tracked to assess breaches in load, which might be the result of retransmissions, lost data, and so forth; in contrast, the number of turns is used to understand the chattiness of the transaction.

The other area of interest, not so much for profiling but certainly for the SLA, is the frequency of usage. You can determine this manually by interviewing the user population or by studying traffic patterns, using long-term monitoring data in combination with the profile data to understand the usage pattern for the various transactions.

Overall Transaction Time

The first thing to record is the overall transaction time. In this case, you can see that the business transaction took approximately 257 seconds, or 4 minutes 17 seconds. On the face, this may appear a long time. Remember, however, that this is considering the complete business transaction time and at this stage does not include specific system or server response times. This is also in keeping with the visual and stopwatch check performed while collecting the sample data. Also note that at this stage the users and bank staff see this overall transaction time as matching their requirements and therefore acceptable. Figure 11-6 shows a bounce diagram view of a simple portfolio transaction taken from Branch 2. The collection for this transaction was performed using Compuware's Application Vantage product.

Figure 11-6. Bounce Diagram for a Simple Portfolio Transaction

[View full size image]

Transaction Conversation Involves Two Servers

Figure 11-7 shows the conversation process in the form of a logical map; this is known as a conversational map. Notice that the conversation for this transaction involves two servers. This transaction clearly shows the interaction between the two servers. This is due to the conversation processthe client makes contact with Server 1, and after receiving notification of the request from Server 1, Server 2 responds directly to the client. This may not always be the case, however; and although not essentially important from a profiling perspective, it becomes important when tracking and monitoring. The capture point determination will differ in these scenarios and will be dealt with in more detail in the following section. In this case, you can see the client station involved in conversations with Server 1 and Server 2. This data was captured from a single point (in this case, the client side).

Figure 11-7. Conversational Map for Transaction

[View full size image]

Nature of the Transaction Flow

Next, view the nature of the transaction flow. Given that this transaction requires user intervention, it is expected that the load will be somewhat bursty in nature. This is confirmed when the interaction is viewed against a timeline, as in Figure 11-8. Note that applications such as file transfers would have more of a constant nature to their transfer.

Figure 11-8. Bursty Nature of Traffic Load

[View full size image]

Average Bytes per Frame

View the average bytes per frame. This enables you to understand the conversational process. Ongoing monitoring assists not only in setting metrics, but also in tracking anomalies that may lead to degradation in performance. In addition, as mentioned previously, this information may be pertinent in optimizing application performance prior to setting the SLA. It will also play a key part when defining the transmission techniques and speeds to transport across wide-area links. Figure 11-9 shows the average bytes relative to the transaction for the conversation. You can see that the average bytes per frame between the client and Server 1 (database server) is approximately 414 bytes, whereas the client to the web server (Server 2) is 741 bytes. Both figures are in keeping with the expectations of the application. The database application server is operating in a Telnet mode, and the web server is using HTTP.

Figure 11-9. Average Bytes per Frame

[View full size image]

Application Turns

The application turns will enable you to determine the nature of the transaction (is it latency or bandwidth bounded?), which may prove useful in optimizing the delivery of the application. For details on application classification, see Chapter 3, "Detailing the Business Transaction." Figure 11-10 shows the number of application turns involved in the simple portfolio transaction. A TCP turn is defined as the number of request/response pairs induced by or attributable to the TCP stack. This information was collected and compiled using Compuware's Application Vantage product.

Figure 11-10. Application Turns

[View full size image]

From this information, it is now possible to define the main servers involved in the conversation, and therefore the relevant Layer 3 and logical names as well as the main ports involved (in this case, TCP port 23 and 80).

Port Numbers

A lot of applications may actually use dynamic port allocations, and therefore the usage of a defined TCP port to track application performance is going to be flawed. In these cases, it is important to understand something more about the application, including items such as a combination of addresses (such as client and server) or a more unique signature within the transaction. When using the word signature, you refer to a unique identification within the data. This may be as simple as a predicted pattern of behavior (such as a three-way handshake or an individual bit [flag] or string within the frame that will allow individual transaction identification).

Specific Repeatable Queries

With regard to individual queries that occur, you are looking for consistent repeatable queries within the transaction. It is pointless to use a query that is unique in structure to one user and one function. Remember that the idea is to build up a set of metrics that enables you to easily track application performance.

Example 11-1 shows the transaction detail used to post the portfolio form to the client desktop. In this case, you can see the multiple data-entry screens the server posts to the client. Apart from being standard to this transaction, they also form the key to this transaction, because it is these requests that will be required to service the user's request.

Example 11-1. HTTP Post
HTTP: [POST /bank/form_Portfolio.asp HTTP/1.1]
!Portfolio form is posted to client desktop
TCP: Sequence Number = 2164216224 -> 2164216901
TCP: Acknowldgement Number = 2713721088
TCP: Window Size = 63749
TCP: Duplicate TCP Frame = No
TCP: Missing TCP Length = 0
TCP: Overlap TCP Length = 0
HTTP: [POST /bank/form_Portfolio.asp HTTP/1.1]
HTTP: Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
  application/vnd.ms-excel, application/vnd.ms-powerpoint,
HTTP: Referer: http://Server_2/bank/form_Portfolio.asp
!Understand what and where the data is drawn from
HTTP: Accept-Language: en-au
HTTP: Content-Type: application/x-www-form-urlencoded
HTTP: Accept-Encoding: gzip, deflate
HTTP: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;
  bankIEv6sp1 Install Package v1.0)
HTTP: Host: ts
HTTP: Content-Length: 29
HTTP: Connection: Keep-Alive
HTTP: Cache-Control: no-cache
HTTP: Cookie: SandsSol%5FClipsEM=SessionID=&AccountName=Client
  ASPSESSIONIDQGQGIGEK=CBPPGIHBNCHODJELAKIEMPEJ
!Output omitted for brevity

You must consider two elements when assessing these response times. The first is the physical deployment of the screen to the user's desktop. This is served by the request in Example 11-1. Of equal importance is the user's response (the entering of data on the screen and posting for a response).

Given in this case that the users are prompted to enter specific information pertinent to this query, it is possible to track the user request for the information from the server. In this case, it is represented by the get command and is shown in Example 11-2. Note that the population of the data can be subject to user wait time, so this metric should be for the request for information only and not taken from the time the initial screen displays at the client.

Example 11-2. HTTP GET
HTTP: [GET /bank/form_Portfolio Contents.asp HTTP/1.1]
!Completed portfolio form is retrieved by the server
TCP: Sequence Number = 2224398400 -> 2224399005
TCP: Acknowldgement Number = 2785360490
TCP: Window Size = 64240
TCP: Duplicate TCP Frame = No
TCP: Missing TCP Length = 0
TCP: Overlap TCP Length = 0
HTTP: [GET /bank/form_Portfolio Contents.asp HTTP/1.1]
HTTP: Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
  application/vnd.ms-excel, application/vnd.ms-powerpoint
HTTP: Referer: http://Server_2/bank/form_Portfolio.asp?PID=11391&SID=773
HTTP: Accept-Language: en-au
HTTP: Accept-Encoding: gzip, deflate
HTTP: User-Agent:_Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;
  bank IEv6sp1 Install Package v1.0
HTTP: Host: Server_2
HTTP: Connection: Keep-Alive
HTTP: Cache-Control: no-cache
HTTP: Cookie: SandsSol%5 bank EM=SessionID=&AccountName=Client;
  ASPSESSIONIDQGQGIGEK=CBPPGIHBNCHODJELAKIEMPEJ
!Output omitted for brevity

Total Amount of Data Transferred

Finally, look at the total amount of data transferred in this query. Figure 11-11 shows the conversational map view with the relevant bytes transferred between the client and servers. By taking this information and combining it with the frequency and user deployment, it is possible to gain an indicative view of the predicted loads on the network infrastructure. You can use this information to review the conversational loads and to track and record any changes that may indicate performance issues.

Figure 11-11. Bytes Transferred

This process is then repeated for the remaining transactions (and potentially locations), and the information collected is recorded for use in the assigning of SLAs.


Case Study 2: Defining SLAs

Having compiled the profile of the transactions, you can now use the information to define and track the bank's SLAs. As discussed in Chapter 4, "Service Level Management," SLAs involve the task of defining specific performance characteristics expected from the network. The agreement defines certain performance metrics used to measure the actual service level obtained from the network against the stated level of service in the SLA.

In Case Study 1, you identified the conversation flow path through the bank's network. This information is valuable for assigning the location of the monitoring tools.

Defining the Location of Metric Collection Points

Your first step in assigning a usable SLA is to define the location of the metric collection points, where actual performance will be measured and correlated to the desired performance requirements. Remembering that you are looking to ensure the transaction is delivered to the end user in a business-effective time, the actual figures represented at each point of the collection become somewhat arbitrary. When the performance of the application degrades, the metric needs to show either a positive or negative movement that you can then act upon. The actual figure itself becomes simply a marker. At other points, the metrics should reflect more meaningful data, but this initial assignment is there to act as an aggregator pointthat is, a common point in the application's conversation flow.

You do not want to place collection devices at every point of the conversational process, because there would be too many extraneous devices, and they would be collecting too much data, causing the following problems:

  • Placing extra, nonessential load on the infrastructure It would be detrimental to the delivery of any application if the system deployed to measure its delivery caused an excessive nonproduction load on the infrastructure.

  • Introducing an artificial performance issue to the system In the case of embedded software on infrastructure devices, if too many counters are enabled they can potentially draw focus away from the device's main function of forwarding data.

  • Rendering data interpretation inconclusive To be of any use, data has to be interpreted and reported on. This, combined with the desire to move to a proactive world where performance issues are picked up as early as possible, means that data collection (or alerting) should be performed at regular intervals. Too much data retrieval can cause the collection periods to overlap, not in terms of physical collection but in terms of being able to correlate and produce useful reports from the data. What then happens in these situations is effectively the number crunching never reaches a conclusion, because each time it nears the end, a new batch of data is introduced to the equation. Attempting to solve this problem by delaying the collection then introduces the issue of potentially missing a problem.

To avoid these problems, try to cover the conversational path of all business-critical applications with a minimal amount of metric collection points. What you are looking to achieve is a natural aggregation of metric data through specific collection-point placement.

Figure 11-12 shows the transactional flows from each site to the centralized servers.

Figure 11-12. Conversation Flows

[View full size image]

The circles in Figure 11-12 represent the common convergence points in these conversation flows. There is one where the applications converge on the central site, and one where the branch users' traffic leaves their local site en route to the servers.

Notice a convergence-point location between the servers. This is in place to enable you to track the interserver traffic. Traditionally, two-tier applications were written for LAN environments, for which it was assumed that bandwidth and latency were not issues. The clients in such architectures consume large amounts of bandwidth and are chatty. When deployed over a WAN, the performance generally becomes unacceptable. In this case study, the client is a thin client. User I/O and graphics updates are sent to the client. In this situation, occasional interaction occurs between the servers to answer relevant queries. This interaction is not always possible to track end to end because the conversation is effectively terminated and restricted at the server. This is because the server conversations are in reality a new conversation. When an answer is achieved, the update is sent back to the client from the original server.

Interserver traffic should be tracked as a separate transaction because its application profile and characteristics will differ from those taken from the client perspective.

Defining Transaction Targets

The next step in creating the SLA is defining what constitutes the transaction, giving the transaction boundaries. In this case, the simple portfolio transaction can be defined as a single transaction that enables the users to transfer between two accounts and check the availability of funds in up to two other accounts. You can further define that from start to finish. For example, the transaction should not exceed 4 minutes 30 seconds (note that this may vary from branch to branch, depending on the baseline profile) to complete. Within that time of 4.5 minutes, the return of information should never exceed 2.5 seconds.

This is a very simplistic view and at this stage has not taken into account availability of the system. However, it allows the definition to be completed very quickly and in terms that are both acceptable and meaningful to the business requirements. It is also an achievable target based on the specific data obtained in the profile exercise.

Collecting the Data

Having defined targets, the next stage is to begin to track the metrics. The effective collection points have been defined. (Refer to Figure 11-12.) Note that in this example server and client counters are not included. Both have relevance in a complete end-to-end monitoring system (see Chapter 9, "Proactive Planning," for details on end-to-end systems), but this example focuses on the network delivery itself.

Figure 11-13 shows the various collection tools and their relevant locations.

Figure 11-13. Collection Points and Tools

[View full size image]

NetFlow gives very good information with regard to flow loads, and when it is used in conjunction with SAA, it is possible to build up an accurate picture of transaction performance. The third-party probe technology (Compuware Network Vantage) is included to add granularity to the collection. The additional probe technology enables you to monitor and track passively, enhancing the data collected from the embedded devices. It also provides something of a checks-and-balances mechanism in the monitoring environment.

You should set the thresholds for the relevant metrics with specific delivery criteria in mind (in this case, receiving the server address and relevant TCP port). The profiling activity assists in defining the trigger criteria, which will differ from scenario to scenario. The more unique your triggers are to a particular transaction, the greater the granularity you will be able to achieve. Trigger criteria can include the following:

  • TCP/UDP port numbers

  • Open file requests

  • HTTP URLs

  • RPC program numbers

  • IP addresses

  • IP multicast addresses

  • Microsoft RPC/DCOM

The performances of the transactions are going to be affected by a number of factors, including the following:

  • Congestion caused by overload

  • Dropped or lost frames

  • Latency

  • Server/Client processing time

These will also form the basis of your metric definitions.

Load Threshold

The load threshold is achieved very simply by taking the number of users, multiplying by the frequency, and finally applying to the profile load.

So in the case of the simple portfolio transaction, relative to Branch 2, the load threshold is determined as follows:

Further granularity can be achieved by dividing the load relative to the servers. In this particular scenario, that only becomes applicable to the metric collection points at the centralized servers, because up to that point the path taken is identical. However, by tracking each interaction separately, you can achieve greater insight into any potential performance degradation.

With regard to this transaction, the requests and responses are somewhat fixed and will only return a maximum value to any particular field. When defining other load metrics, be sure to set any threshold to accommodate the maximum load.

Having defined the threshold, it should be set approximately 0.5 percent above the expected load. This gives some flexibility in the transmission. The idea behind setting a threshold limit at each of the collection points is to make identification of performance issues easier. For example, an increase in load might indicate retransmissions; if that load is only seen increasing at Branch 2, however, you can assume it is an issue relevant to that one site.

Dropped or Lost Frames

Many of the metrics take on more meaning when viewed in conjunction with other metrics. That is the case with dropped or lost frames. In this case, you are looking at the frames offered versus the frames delivered.

The rate at which packets are lost can have a dramatic impact on application performance.

The threshold of this metric is again dependent on the specific application, and is subject to how sensitive the application is to dropped or lost frames. For example, a real-time application such as IP telephony will suffer if excessive packets are lost, whereas some applications can automatically recover and are not affected by small losses. Every application has a threshold, which becomes obvious during the profiling of the application.

Given that the portfolio transaction is based on HTTP and Telnet, it is assumed that it is not too sensitive to packet drops. Predictive tools are available to estimate the impact of packet loss to application performance, but in this scenario it is deemed sufficient to observe the difference between packets offered and packets received at the respective ends of a conversational system (such as WAN links). These, in conjunction with latency alerts, will assist in identifying SLA breaches. Note that this threshold ensures that the SLA is still met and not exceeded. It is important to set thresholds (for monitoring purposes) just below the exact threshold requirements to ensure you are alerted to degrading services before the performance falls to a point that violates the SLA.

Latency

As in the previous examples, the goal of profiling is to understand the application traits, including whether the application is susceptible to latency. (See Chapter 3 for details on application classification.) Determining the impact of latency on a transaction will almost certainly require external predictive tools. (See Chapter 6, "Application Deployment," for details on predictive tools.) These tools enable you to gain a more accurate measure of latency impacts, to more finely adjust thresholds, and so forth. In general, however, it will suffice to monitor and understand the following general latency characteristics of specific transmission systems:

  • Typical LAN 1 ms

  • Typical campus 10 ms

  • Moderate WAN 20 ms

  • United States coast to coast 30 ms

  • Satellite 250 ms

Note that these latency figures represent one-way communication from carrier to carrier. In the case of a satellite system, you are likely to see a latency of 250 ms to the satellite and a further 250 ms down from the satellite. Therefore, to determine the actual request-to-response time, you must factor in latency in both directions.

Server/Client Processing Time

Effective positioning of collection points enables you to accurately record and monitor the quantifiable effects on the network of the metrics previously discussed. When dealing with the client and server processing time, however, you face a different issue: Ignoring for the moment that it is possible to use any number of third-party software counters for recording client and server statistics, the concern at this stage is to be able to view the performance relative to the overall delivery.

Earlier in this section, you read about the location of collecting points at natural aggregators. From that discussion, you learned that the best location is at the ingress/egress points to the conversational path, relevant to each site. (Refer to Figure 11-12.) That statement remains true; however, notice the location of these devices relevant to the clients and servers. Potentially, a number of devices might reside between the collection points and their end nodes (for example, network switches).

Figure 11-14 illustrates the problem this presents. From the collection-device perspective, the next hop is the end node. The probe has no knowledge of the infrastructure that makes up that last part of the path. Therefore, it assumes that after the frame conversation has passed by, its next immediate step is the end node, and it attributes all remaining time to client or server processing.

Figure 11-14. Monitoring System Assumes the Final Destination Is One Hop Away

In Figure 11-14, the probe (based on its position) is aware only of the conversation heading to and terminating on the servers. It has no knowledge of the intermediate steps taken in the physical path.

What is needed is a single figure that is representative of the performance of either the client or servers processing at each of the collection point sites. This figure will be derived from the profiling exercise, but will be a general figure that applies to the average performance of all end nodes at each site relevant to that application.

In this example, during the profiling exercise it was found that if all transactions at Branch 2 showed an average client response time of 209 ms (remember this is initial response and is recorded at the branch office), the overall desired response time was achieved; subsequently, at the centralized server location, the figure relative to achieving the SLA was 250 ms.

Note that these are just example figures used here to enable the tracking of the SLA. In the case of a breach, you should expedite the search for the most probable cause of a delay. These figures are not intended to be reported on as part of the SLA.

The discussion of the transaction-definition phase mentioned that complex transactions are generally too individual to define a fixed profile. That remains true, but the information and thresholds gained from profiling the other more definable transactions will provide the basic thresholds for these transactions.

It is assumed that the complex application transaction is just made up of any number of the other two transactions, but in varying degrees and attributes. With that in mind, the deliverables for the overall performance remain the same, and therefore the metric collection and threshold settings for the other transaction should meet the complex transaction requirements.

In fact, the optimization and monitoring process is a constant moving target, with frequent reviews required to ensure continued business benefits. That said, never be afraid to include individual items from the complex transactions, such as individual queries or information requests, should you feel that it would provide a benefit.

Although the basis of this case study is to start out simple to quickly gain some control and visibility of the transaction performance, over a period of time it is not inconceivable to extend your scope of business-critical applications. As such, it may be beneficial to subdivide elements of the complex transactions.


Case Study 3: Optimizing the Business Transaction

Case Study 1 and Case Study 2 in this chapter discussed the bank's simple portfolio transaction. Initially, this transaction, the simple portfolio, was within acceptable levels to meet the business requirements. The time taken to complete the transaction met user expectations.

The profile revealed the following about the application:

  • Bandwidth requirements are relatively low.

  • There is zero tolerance for packet loss.

  • Bandwidth requirements are in the order of kilobits per second, depending on the data retrieved.

  • Initial response times vary from 500 ms to a few seconds.

  • Server resource requirements vary, depending on the specific query, in terms of CPU, disk, and memory.

The bank is keen to determine whether the transaction can indeed by optimized.

The first thing noted in the profile is that the transaction is not particularly bandwidth sensitive, so the simple approach of increasing the bandwidth will have a negligible impact on individual response times. Exercise caution when predicting potential loads in the merged case. Increased usage will ultimately result in greater bandwidth requirements. Inability to meet these requirements may well result in dropped packets that will detrimentally impact user performance.

Aside from providing a basic understanding of the application characteristics, the profile also revealed some potential bottleneck issues occurring between the client and Server 2. The issues centered on TCP windowing.

When an application is sending bulk data over a network, TCP window sizes must be large enough to permit TCP to send many packets in a row without having to wait for TCP acknowledgments (ACKs). TCP will send data only if the amount of sent-but-not-yet-acknowledged data is less than the minimum of the congestion control window, sender window, and receiver window sizes.

Checking Window Size

The first stage of the optimization process is to make sure the application is using windows that are larger than the bandwidth-delay product. A bandwidth-delay product measures the amount of data that will fill the pipe. This in turn is controlled by buffer space at the sender and receiver, which allows maximum throughput on the TCP connection over the path. Problems arise when a bandwidth-delay product is large.

You first need to check the send and receive windows for incompatibilities. This information is visible in the TCP protocol decode as WIN=XXXX. Note that applications can set the receive window size with a system call such as setsockopt. Figure 11-15 shows details of a frame with the window size highlighted in both a decimal and hex format.

Figure 11-15. TCP Window Size Within a Frame

[View full size image]

Checking for Retransmissions

Retransmissions are not excessive; however, in an effort to circumvent potential issues in the future, TCP extensions should be enabled. These include items such as fast retransmit/fast recovery and selective acknowledgments. (See Chapter 12, "Optimizing Application Delivery in Storage-Based Networking, Wireless LANS, and an End-to-End Model.")

Fast-retransmit and -recovery algorithms will help recover one packet loss per window without draining the pipeline. However, more than this will result in a retransmission timeout, forcing pipeline drain and a slow-start algorithm to begin again. (Random dropping schemes might negatively impact this application because they might cause more than one drop per window.) Although expanding the window will assist in improving the delivery for this application, exercise caution because expanding the window size to match pipe capacity increases the probability of packet drops. Figure 11-16 shows the points and type of optimization for a conversation between users at Branch 2 and the application servers located at HQ.

Figure 11-16. Optimization Settings for the Branch-to-HQ Conversation

[View full size image]

In Figure 11-16, point 1 indicates that the window size on the routers is set to ensure the bandwidth between the routers is fully used; note this is done to achieve optimum use of bandwidth between the routers. In addition, the throughput is maintained by using traffic shaping. Point 2 also applies to the routers and is the use of multiple queues to protect the data from packet drops. The priority queue is used to schedule between the queues. Point 3 applies to the LAN infrastructure, where speed and duplex settings should be fixed to avoid mismatch. Finally, point 4 is where you ensure the window size on the application server is set to match the end-to-end delivery criteria for your environment.

Checking Application Architecture

Having made recommendations for delivery optimization, you should address the application architecture itself. However, even though this is a latency-sensitive application, the number of application turns is not considered to be an issue. Figure 11-17 shows the conversation map with the application turns between the client and server. As you can see, the number of application turns involved in this conversation is already optimized. In situations where turns are many, you can achieve significant improvements in the response time.

Figure 11-17. Application Turns Already Optimized

[View full size image]

Checking Queries

Analysis of the queries reveals that, in general, the POST and GET statements for the browser are relatively quick, with no excessive wait times on the server being recorded. Table 11-3 reveals a 2.8-second time for one POST.

Table 11-3. Observation of Times for POST and GET

Name

Server

Duration

HTTP: [POST /clipsem/menu_Products.asp HTTP/1.1]

Server_1:80

2.828556400

HTTP: [GET /clipsem/menu_BasketContents.asp HTTP/1.1]

Server_1:80

1.456264000

HTTP: [GET /clipsem/form_notes.asp?PID=11391&SID+773 HTTP/1

Server_1:80

0.468190000

HTTP: [POST /clipsem/form_notes.asp HTTP/1.1]

Server_1:80

0.027468000

HTTP: [GET /clipsem/menu_BasketContents.asp HTTP/1.1]

Server_1:80

1.849363600

HTTP: [POST /clipsem/menu_Products.asp HTTP/1.1]

Server_1:80

0.193250000

However, analysis also reveals that the details for the form (such as product listings) that remain constant are being referenced from the remote server, which does not constitute a large time (2.8 seconds). Local referencing, which remains constant, would realize a savings of 1.9 seconds per transaction, as shown in Example 11-3. Note that that this does create an additional administration overhead to ensure product details are updated to the local machines, but the client agrees that this time savings constitutes a productivity improvement.

Example 11-3. Static Product List Is Referenced from a Remote Server
HTTP: [POST /bank/form_Portfolio.asp HTTP/1.1]
TCP: Sequence Number = 2164216224 -> 2164216901
TCP: Acknowldgement Number = 2713721088
TCP: Window Size = 63749
TCP: Duplicate TCP Frame = No
TCP: Missing TCP Length = 0
TCP: Overlap TCP Length = 0
HTTP: POST /bank/form_Portfolio.asp HTTP/1.1]
HTTP: Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
  application/vnd.ms-excel, application/vnd.ms-powerpoint,
HTTP: Referer: http://Server_1/clipsem/menu_Products.asp
!Product menu is being referenced from the remote Server_1. This is a relatively static

  product list and could be referenced locally.
HTTP: Accept-Language: en-au
HTTP: Content-Type: application/x-www-form-urlencoded
HTTP: Accept-Encoding: gzip, deflate
HTTP: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;
  bankIEv6sp1 Install Package v1.0)
HTTP: Host:Server_2
HTTP: Content-Length: 29
HTTP: Connection: Keep-Alive
HTTP: Cache-Control: no-cache
HTTP: Cookie: SandsSol%5FClipsEM=SessionID=&AccountName=Client
  ASPSESSIONIDQGQGIGEK=CBPPGIHBNCHODJELAKIEMPEJ
!Output omitted for brevity

Verifying Performance Increase

Having optimized the transaction, the next stage is to observe the changes and compare to see whether the optimization has in fact increased performance. Note that this change is carried out on a single sample branch to understand the impact prior to rolling out. This approach could also be achieved in a virtual environment to observe impacts in a nonproduction environment. (See Chapter 9.)

Using the same collection agents and methodology, including users agents and filters, ensures consistency when comparing results. Figure 11-18 shows the comparison of the optimized application with the original baseline figure. The top line represents the application performance before any optimization, and the lower line shows the same transaction following the optimization.

Figure 11-18. Comparison of Application Transaction With and Without Optimization Changes

[View full size image]

Deploying Changes and Updating SLA

The optimization represents a significant improvement in performance, so the next stage is to deploy the changes to all branches. The new baseline figure becomes the basis for the SLA, and as a result, you need to change the collection metrics to monitor the new specifications.


Case Study 4: Implementing QoS in a VoIP Environment

IP telephony is a fast-expanding market with many new users every day. The basic idea is simple. Instead of transmitting the voice over a telephone network, you sample the voice and send it as data packets over your regular computer network. The phones can either be dedicated IP phones or softphones that run on your desktop PC. The packets are treated just like any other data packet by the network and are propagated toward the receiver, who then decodes the sample and hears your voice.

For Voice over IP (VoIP) to be a realistic replacement for standard Public Switched Telephone Network (PSTN) telephony services, customers need to receive the same quality of voice transmission they receive with basic telephone services, meaning consistent, high-quality voice transmissions. Like other real-time applications, VoIP is extremely bandwidth and delay sensitive. For VoIP transmissions to be intelligible to the receiver, voice packets should not be dropped, excessively delayed, or suffer varying delay (jitter).

VoIP can guarantee high-quality voice transmission only if the voice packets, for both the signaling and audio channel, are given priority over other kinds of network traffic. For VoIP to be deployed so that users receive an acceptable level of voice quality, VoIP traffic must be guaranteed certain compensating bandwidth, latency, and jitter requirements. QoS ensures that VoIP voice packets receive the preferential treatment they require. In general, QoS provides better (and more predictable) network service by providing the following features:

  • Supporting dedicated bandwidth

  • Improving loss characteristics

  • Avoiding and managing network congestion

  • Shaping network traffic

  • Setting traffic priorities across the network

In this example, the bank has decided to replace its PSTN connectivity with a VoIP system. The initial deployment saw the installation of voice gateways at each of the branch sites. The connectivity between the WANs was provided using Frame Relay circuits. Figure 11-19 shows the deployment. In the cases of the branch and sub-branch sites, the access router also becomes the voice gateway device, but at HQ the number of users dictates that this be handled by a separate device.

Figure 11-19. VoIP Infrastructure Deployment

[View full size image]

The system was initially deployed without specific QoS settings for either the VoIP traffic or specific sites. The result of this was that at peak times for the bank, when many users were trying to access data from the HQ-based servers, the voice quality on the IP phones deteriorated to such a level that it became unusable. Figure 11-20 shows the increase in jitter for the VoIP traffic relative to the overall load on the link. The bar charts represent the outgoing and incoming traffic on the main HQ permanent virtual circuit (PVC). The single line represents the VoIP jitter observed for voice traffic on that link. Notice that the VoIP response is worse during these times of heavy load.

Figure 11-20. WAN Traffic Relative to VoIP Jitter

[View full size image]

In this scenario, the bank was using a Frame Relay WAN data-traffic infrastructure to carry voice calls between its headquarters and its branch offices. They had the following two options to improve the overall voice quality on this system:

  • Carry the voice and data on separate PVCs

  • Use the same PVC for voice and data traffic

When carrying voice and data on separate PVCs, they must still give the voice traffic priority by using a priority queuing (PQ) technique. (See Chapter 5, "QoS and MPLS: Tools to Manage Application Performance.") PQ enables you to assign PVC priorities of high, medium, normal, or low. PQ also allows PVCs to be queued at the main physical interface on the router so that high-priority traffic goes before medium-, normal-, and low-priority traffic. However, PQ does have a problem: The high-priority traffic can starve the other traffic of bandwidth. If you use Frame Relay traffic shaping correctly, however, you can minimize this problem, because each PVC will have a defined maximum transmission rate.

Frame Relay traffic shaping is the most common traffic shaping application in VoIP environments. Frame Relay scenarios usually have a hub-and-spoke network, where the hub link speed is higher than any of the remote link speeds. In some cases, the sum of the remote link speeds is higher than the hub link speed, causing oversubscription. Without Frame Relay traffic shaping, the hub may try to send at higher rates than the remotes can receive traffic, causing the Frame Relay network to arbitrarily drop traffic. However, the remotes could all send at an aggregate rate that is higher than what the hub can receive, again causing the Frame Relay network to arbitrarily drop traffic.

The Frame Relay network normally refers to the service provider (SP) network of WAN switches that provide the end-to-end PVC connectivity. Because the WAN SP cloud has no Layer 3 or above intelligence, it can drop VoIP traffic if contracts are violated. Therefore, you need to control transmission rates into a Frame Relay cloud so that you can control which packets get dropped and which packets receive priority servicing.

The bank needed to configure Frame Relay traffic shaping to ensure that speed mismatches at the remote and branch sites were handled correctly. For example, the Branch 2 site has a 2-Mbps connection into the Frame Relay network, and the remote sub-branch site has a 64-kbps access speed. Branch 2 has the capability to send at 2-Mbps speeds toward this single sub-branch site. The Frame Relay switches will buffer this traffic to a small extent, but then arbitrarily drop anything over 64 kbps. The bank needed to decide what should be dropped and what should be prioritized at the PVC endpoints.

Frame Relay traffic shaping allows the routers to send traffic into the Frame Relay cloud below a preconfigured rate. Any traffic over this rate is queued, and a queuing algorithm such as low-latency queuing (LLQ, see Chapter 5) can be used to make intelligent decisions on which packets should be sent. If the queues fill up, the packets are just dropped. If VoIP is given priority, and the total VoIP traffic is below the traffic-shaping rate, however, VoIP packets are serviced with low latency and are not dropped.

Even if queuing is working at its best and is prioritizing voice traffic, sometimes the priority queue is empty, and a packet from another class is serviced. Packets from guaranteed-bandwidth classes must be serviced according to their configured weight. (See Chapter 5.) If a priority voice packet arrives in the output queue while these packets are being serviced, the VoIP packet could wait a substantial amount of time before being sent. If you assume that a VoIP packet will need to wait behind one data packet, and that the data packet can be, at most, equal in size to the maximum transmission unit (MTU, 1500 bytes for serial and 4470 bytes for high-speed serial interfaces), you can calculate the wait time based on link speed.

For example, for a link speed of 64 kbps and an MTU size of 1500 bytes, you have the following wait time:

Therefore, a VoIP packet may need to wait up to 187.5 ms before it can be sent if it gets delayed behind a single 1500-byte packet on a 64-kbps link. VoIP packets usually are sent every 20 ms. With an end-to-end delay budget of 150 ms and strict jitter requirements, a gap of more than 180 ms is unacceptable.

You need a mechanism that ensures that the size of one transmission unit is less than 10 ms. Any packets that have more than 10-ms serialization delay need to be fragmented into 10-ms chunks. A 10-ms chunk or fragment is the number of bytes that can be sent over the link in 10 ms. You can calculate the size by using the link speed. For example, for a link speed of 64 kbps and a serialization delay of 10 ms, you have the following fragmentation size:

It takes 10 ms to send an 80-byte packet or fragment over a 64-kbps link.

On low-speed links where a 10-mssized packet is smaller than the MTU, fragmentation is required. Simple fragmentation is insufficient, however, because if the VoIP packet must wait behind all the fragments of a single large data packet, the VoIP packet still will be delayed beyond the end-to-end delay limit. For lower-speed links less than 1.2 Mbps, the bank needed to configure packet fragmentation to ensure that a VoIP packet need not wait behind a large packet. Fragmenting larger data packets to 10 ms of the link speed can maintain the maximum waiting period.

They also used Real-Time Transport Protocol header compression (cRTP) to make optimum use of the bandwidth. IP RTP header compression reduces the 40-byte IP + UDP + RTP header to 24 bytes, thereby reducing the bandwidth required per voice call on point-to-point links. The header is compressed at one end of the link and decompressed at the other end.

Note that to configure IP RTP header compression, you need to configure the ip rtp header-compression command under the serial interface, or the frame-relay ip rtp header-compression command under the Frame Relay subinterface. You can also configure the ip rtp compression-connections interface configuration command to set a maximum number of flows that will be compressed. Because cRTP can be processor intensive, you need to limit the number of compressed flows to prevent router performance degradation. Compressed RTP is recommended on low-speed links where bandwidth is scarce and there are few VoIP calls.

To provide high quality to VoIP over Frame Relay, the bank configured the following features:

  • Frame relay traffic shaping To configure a map class, use the following commands.

    - Set the frame-relay cir map-class configuration command to maximum transmit rate. (It should be the negotiated guaranteed rate from the service provider.)

    - Disable the frame-relay adaptive-shaping map-class configuration command and set the mincir command value to match the cir command value for best-quality voice.

    - Set the frame-relay bc map-class configuration command to 1/100 of CIR to allow traffic shaping to service packets at least every 10 ms.

  • Link fragmentation and interleaving (LFI) Fragmentation was used only on the 64-kbps links.

  • LLQ on Frame Relay PVC LLQ is applied under the map class for Frame Relay traffic shaping.

  • cRTP cRTP is applied under the Frame Relay subinterface; you should use cRTP only if the CPU utilization is low, and for a small number of calls, depending on the platform.

The following list shows configuration examples for enabling the QoS features to provide the required performance for the bank's VoIP system:

  • The following configuration creates the class voip for voice traffic that has been marked with IP Precedence 5 using one of the available marking methods:

    - class-map voip

    - match ip precedence 5

  • The following configuration creates the class webtraffic for web traffic that has been marked with IP Precedence 3 using one of the available marking methods:

    - class-map webtraffic

    - match ip precedence 3

  • The following configuration defines the QoS policy map llq: Class voip traffic gets priority and is limited to 64 kbps during congestion; class webtraffic packets are guaranteed 64 kbps. All other traffic shares the remaining bandwidth.

    - policy-map llq

    - class voip

    - priority 64

    - class webtraffic

    - bandwidth 64 class class-default

    - fair-queue

  • The following configuration enables Frame Relay traffic shaping. You must enable Frame Relay traffic shaping to handle speed mismatches and oversubscription. (LLQ per Frame Relay PVC also requires Frame Relay traffic shaping.)

    - interface Serial 0/1

    - no ip address

    - encapsulation frame-relay

    - frame-relay traffic shaping

  • The following configuration attaches traffic-shaping class voice to this Frame Relay PVC:

    - interface Serial 0/1.64 point-to-point

    - ip address 10.14.96.2. 255.255.255.252

    - frame-relay interface-dlci 128

    - class voice

  • frame-relay ip rtp header-compression configures cRTP to reduce the bandwidth requirements of each voice call.

  • frame-relay fragment 320 enables fragmentation with a fragment size of 320 bytes.

  • service-policy output llq attaches the llq QoS policy to the defined map class.

In this example, Frame Relay traffic shaping handles speed mismatches, fragmentation prevents VoIP packets from getting delayed behind large data packets, cRTP reduces VoIP bandwidth requirements, and LLQ provides priority to VoIP traffic and guarantees bandwidth to another class. These features were configured on both ends of the Frame Relay link. However, fragmentation was needed only for the sub-branch links, because they are less than 1.2 Mbps. Figure 11-21 shows the improved response time for the VoIP traffic during the high load periods.

Figure 11-21. Improved Response Time

[View full size image]


Summary

Profiling the application is the first step in defining a meaningful SLA that will bring quantifiable business benefits. When deciding what is important, you need to be ruthless. The bigger you try to make the exercise, the less likely you are to achieve a positive end result. This is an often-overlooked process in the overall optimization of the application delivery.

Transactions are traditionally defined as applications processing modules that exhibit similar resource requirements, characteristics, and user expectations. The best approach is to initially describe the functions in terms of human activity. Then having defined the functions, apply applications and computing systems. This will give you the basis of your essential business transactions.

When you begin to optimize an application for improved performance, consider the application's profile. Just increasing a network's bandwidth might not always positively impact the application's response time. Things to consider include the send and receive window sizes, the number of application turns (between client and server), and the potential for packet drops.

When the bank replaced its PSTN-based system with a VoIP system, they found that without applying QoS settings to the VoIP traffic, the application was at times unusable. VoIP traffic must be guaranteed certain compensating bandwidth, latency, and jitter parameters. QoS ensures that VoIP voice packets receive the preferential treatment they require. By using a combination of traffic shaping, priority queuing, and fragmentation, the bank improved the overall performance of the VoIP deployment.


Reference Used in This Chapter

  • Cisco Systems, Configuring QoS for IP Telephony, http://www.cisco.com/en/US/products/sw/cscowork/ps2064/products_user_guide_chapter09186a0080191f00.html


Chapter 12. Optimizing Application Delivery in Storage-Based Networking, Wireless LANs, and an End-to-End Model

This chapter includes the following topics:

  • Storage-area networks

  • Wireless LANs

  • Complete end-to-end model

For years, network performance problems were solved with additional bandwidth. For underperforming applications, the first choice often has been to add servers and processing power. Hardware and bandwidth are relatively inexpensive, but IT managers have a difficult time scaling fast enough to keep up with the exploding amount and diversity of application traffic. Web surfing, e-mail, and streaming video and audio can hog bandwidth and degrade the performance of mission-critical applications such as customer relationship management (CRM) and enterprise resource planning (ERP). Adding more bandwidth might only improve the response of nonessential applications; it does not guarantee that the bandwidth will be available to the applications that need it most.

This chapter concentrates on the design issues and considerations with methodology approaches for storage-area networks, wireless LANs, and an end-to-end model. Basic design is considered very important in any optimization methodology. No amount of optimization will compensate for a poor base design.

Each study can stand alone as isolated design considerations, or they can be merged to form a more holistic view of requirements for different scenarios.


Storage-Area Networks

Direct storage in any client/server storage model results in limitations in bandwidth, based on the maximum number of devices available on a bus, and concurrency of traffic. Attaching application servers and Fibre Channel (FC) storage devices to a central location creates a network of storage devices. This configuration, shown in Figure 12-1, allows the application servers to access storage devices in a network fashion. In other words, the devices, both server and storage, must have an address on the network, have the capability to log in, and have methods available to communicate with other devices within the network.

Figure 12-1. FC-Attached Storage-Area Network

[View full size image]

Storage-area network (SAN) is an emerging data communications platform that interconnects servers and storage at gigabit speeds. By combining LAN networking models with the core building blocks of server performance and mass storage capacity, SAN eliminates the bandwidth bottlenecks and scalability limitations imposed by previous SCSI bus-based architectures.

SANs are constructed with many new components from the storage world. The foundation is the FC switch, which provides the physical connection that allows any-to-any communication. Any-to-any communication implies that mixed operating systems and applications (such as different databases) can communicate and make use of the storage network. SAN switches provide the hardware and software foundations needed to facilitate the network; the hardware itself is comprised of ports that permit the connection of FC-based devices, such as storage arrays and servers.

Emerging standards and protocols deal with quality of service (QoS) on the FC switches and network, but for this example the optimization effort terminates at the FC switch.

Although this design example does not deal with the FC network, for completeness the connections operate as follows. To participate in the storage network, servers require a host adapter, similar to a network adapter, known as an FC host bus adapter (HBA). The HBAs supply the storage network drivers that allow the server to communicate with the switch and ultimately log in and communicate with storage devices.

FC is a layered connectivity standard that has the characteristics of a bus with the flexibility of a network. FC operates on a serial link design and uses a packet type of approach for encapsulation of the user data. FC transmits and receives these data packets through the node participants within the fabric. The packets shipped by FC are referred to as frames and are made up of header information that includes addresses, user data (encapsulating 2192 bytes per frame), and error recovery code (ERC) information.

Comparing Network-Attached Storage with Storage-Area Networks

One of the first questions addressed by enterprise customers is the one of network-attached storage (NAS) versus SANs. Each has different capabilities. This should be a consideration in providing optimization to the business-critical application. Much confusion surrounds the differences between a SAN and a NAS device. A NAS device is a self-contained, autonomous storage appliance connected directly to a network. Unlike SAN, which operates at a block level, NAS is a file-oriented storage solution that functions like traditional file servers in an IT infrastructure.

The storage resources of a NAS device are accessed via mapped network drives that talk to servers and clients through the Network File System (NFS) or Common Internet File System (CIFS) protocols, which are run over the IP protocol.

NAS is very simple to set up and maintain. The thin server technology used by most NAS devices is easy for network administrators of any skill level to understand and is very reliable. However, NAS is not a replacement for SANs.

The basic technical difference between SAN and NAS lies at the communication protocol level. Whereas NAS communicates via NFS and CIFS Fibre Channel, SANs transmit data over the Fibre Channel Protocol (FCP).

As the name implies, file-level protocols transfer data from storage units to servers in the form of files. Unlike a SAN, NAS solutions have their own file systems, which they manage independently within their storage enclosures. Although thin server units, in charge of their own file system management and user authentication, are convenient and typically easy to manage, a NAS unit can become a serious bottleneck in high-transaction environments.

In the case of FC, data is delivered in device blocks the same way an external redundant array of inexpensive disks (RAID) would deliver data to an attached server. In a SAN environment, the middleman, a thin server unit that typically manages NAS, is eliminated.

Furthermore, because a SAN does not have to use IP to transfer data to the servers, the conversion of data blocks into IP packets (which is done in NAS) does not have to take place, reducing latency.

Contrary to many beliefs, NAS is not replaced by SANs. Part of the confusion between these two technologies results from fact that both SANs and NAS can centralize data. NAS enables users to share data as single files in a single data repository, leveraging networking technology (Ethernet). Because NAS moves storage traffic off the SAN and onto IP networks, IT managers can use NAS to reduce the pressure in scaling their SAN infrastructures beyond their existing means, saving some infrastructure cost, although SANs by nature are extremely scalable. Today, SANs are best suited to serving the needs of transaction-intensive application servers, whereas NAS (because they do not require extra infrastructure) are best suited for the storage needs of low-end workstations and desktops.

Although NAS is cheaper to implement and has similar centralized storage capabilities, enterprises consider SAN implementations for performance reasons. Most IT managers would not want to run disk-intensive applications on a NAS device across the network. Some applications, such as databases, work best when the database application has direct high-speed access to their directories, data, and log files. NAS works well for file server access in most situations, but an organization should still use traditional server-attached or SAN storage subsystems to run disk-intensive applications such as transaction processing.

Many applications are driven by a need to share access to the same files, usually for read-only access. Internet-based applications, requiring file sharing with web page serving, are a good example. Shared data access is typically from many heterogeneous servers running concurrently. Each application wants access to the same web pages.

Consider media serving. Potentially, thousands of copies of the same movie or video clip are being delivered to an equal number of users at the same time, but not necessarily at the same point in the clip. It takes many servers to deliver this same clip over and over again, in most cases pointing to a single file. So if you are serving up a large collection of web pages, or video streams, to many users at the same time across a variety of client environments, network-based file access is required.

The primary reason to use NAS is if sharing occurs at the file levelmany, sometimes large, files need to be shared, typically by diverse servers. Examples of files that need to be shared can be found everywhere: network drives in an Windows NT environment, workgroup CAD files, and web pages shared by many web servers.

Matching Application Characteristics with Relative Storage Solution

Now that you understand the attributes and relative merits of each solution, the first consideration in optimizing delivery in a SAN environment is to match your application characteristics with the relative storage solution. Table 12-1 shows the basic storage components and their relative merits to types of application transactions.

Table 12-1. Simple Assessment of Application Merits for Storage Techniques

Workload Type

Direct Attached Storage

NAS

SAN

Batch

Use with care

Use with care

Optimum

Online transaction processing (OLTP)

Optimum

Problematic

Optimum

Web messaging

Requires significant tuning

Optimum

Use with care

Data centric

Problematic

Requires significant tuning

Optimum

Identifying Flows

Now that you understand which applications are best suited to SAN environments, you can set about optimizing their delivery. As well as understanding the applications that best suit this environment, from a functional point of view you also need to appreciate the impact these will have on a network environment. For example, the idea of a centralized storage area is a good one, but there is the consideration that centralized storage will create large data flows between all your sites.

Based on this initial consideration, the first step is to identify the flows. This identification process should include numbers of end nodes requiring communication with the SAN network and, probably more importantly, the path flows that will be used. Your identification of such has the following twofold effect:

  1. Identification of the flow will assist in assigning common points of the flow that can be used as metric collection points. (See Chapter 8, "Monitoring the Delivery.")

  2. You can then use these to determine whether the flow is taking the most optimal route. The optimal path might not be the same one that would naturally be identified by the dynamic protocols.

Figure 12-2 shows a subsection of an enterprise network. You can see remote branch nodes that require connection to the SAN network located at the HQ site. The users at this site also require access to a highly time-sensitive business-critical application server that is also located at the HQ site. In this case, the response requirements of the SAN traffic are not as great as those of the business-critical application; given the potentially large volume of traffic identified in the profiling stage (see Chapter 3, "Detailing the Business Transaction"), however, the SAN traffic might just overwhelm the business-critical application.

Figure 12-2. Traffic Flows from a Branch Office to HQ

[View full size image]

You should apply a higher priority to the business-critical application as it leaves the branch site, and subsequently assign a lower priority to the SAN traffic. In addition, you can take still more steps in an effort to ensure the SAN traffic receives the optimum delivery.

Optimizing SAN Traffic Delivery

The SAN traffic is transported by TCP; therefore, the following conditions will assist in optimizing the SAN traffic delivery:

  • Large window sizes

  • Traffic shaping or rate limiting

  • Selective acknowledgments

Also note that at an Ethernet level, the traffic may well benefit from the use of jumbo frames, because they will allow the Ethernet frame size to match that of the FCP. However, this does not apply to this example because the traffic is traversing a WAN pipe and will be subjected to fragmentation.

SAN traffic is synchronous by nature; therefore, to ensure the application makes the most of the bandwidth, it is best to fill as much of the pipe as possible (remembering that this will be restricted to ensure the business-critical application receives adequate resources to meet its requirements).

By characterizing the application flows, and then applying measures suited to the application data characteristics, it is possible to maintain the flow of data to the SAN network, and in doing so maintain the required delivery criteria. In this example, that means ensuring the delivery pipe is kept full and constant, while providing recovery techniques to ensure that an excessive amount of data is not flooded into the network.

TCP Window Scaling

SAN traffic can be described as a bandwidth-constrained application (see Chapter 3); therefore, you need to ensure a steady stream of data is transmitted. The best place to start is to ensure the TCP window is set to achieve maximum throughput. This will aid in ensuring the available bandwidth.

The TCP Window Scaling feature adds support for the Window Scaling option in RFC 1323, "TCP Extensions for High Performance." A larger window size is recommended to improve TCP performance in network paths with large-bandwidth and long-delay characteristics, which are called long fat networks (LFNs).

The window-scaling extension in Cisco IOS Software expands the definition of the TCP window to 32 bits and then uses a scale factor to carry this 32-bit value in the 16-bit Window field of the TCP header. The window size can increase to a scale factor of 14. Typical applications use a scale factor of 3 when deployed in LFNs.

The Cisco IOS window-scaling feature complies with RFC 1323. The maximum window size has been increased to 1,073,741,823 bytes. The larger, scalable window size will allow the SAN traffic to perform better over LFNs. When configuring, remember that both sides of the link must be configured to support window scaling; otherwise, the default of 65,535 bytes will apply as the maximum window size.

There is not a definitive figure for window size that will guarantee performance increase, and each configuration will depend on factors within the specific environment. In theory, the larger the window size, the larger the throughput. In general terms, the throughput will equal the window size divided by the round-trip time (RTT). It is important to note at this stage that wherever possible the RTT should be kept to a minimum. In many cases, this will depend on the network itself, but be influenced by the number of times the nodes need to exchange information. When profiling, wherever possible note the characteristics and make recommendations to decrease this interaction. (See Chapter 3.)

Traffic Shaping

There is a tendency to use too large a window size. (In this particular network, the TCP window size was set to 2 MB.) A too large window can impact the performance by allowing too much traffic onto the network, which has a similar effect on the routers' buffers, resulting in overflows, dropped packets, and repeated retransmissions of data. Therefore, it is essential to have a large enough number of buffers in routers to be able to absorb data bursts while keeping the queues nominally small. To achieve this, implement an active queue management mechanism such as weighted random early detection (WRED) on the routers.

Active queue management mechanisms do not eliminate packet drops, but instead drop packets at an earlier stage to solve the full-queue problem for flows that are responsive to packet drops as a congestion signal.

Selective Acknowledgment

The SAN TCP transmission may experience poor performance when multiple packets are lost from one window of data. With the limited information available from cumulative acknowledgments, the end node can only learn about a single lost packet per RTT.

Selective acknowledgment (SACK) is a strategy that corrects this behavior in the face of multiple dropped segments. With SACKs, the data receiver can inform the sender about all segments that have arrived successfully, so the sender needs to retransmit only the segments that have actually been lost. In this example, SACK is used to overcome the limitations of recovering from multiple lost packets by allowing a receiver to acknowledge nonconsecutive data and maintain the flow of data.[1]


Wireless LANs

As users experience the convenience of wireless connectivity, they are beginning to demand support for the same applications they run over today's wired networks. Because wireless bandwidth availability is restricted, QoS is increasingly important in 802.11 networks.

802.11 refers to a family of specifications developed by the IEEE for wireless LAN technology. 802.11 specifies an over-the-air interface between a wireless client and a base station or between two wireless clients. The IEEE accepted the specification in 1997.

The 802.11 family includes the following specifications:

  • 802.11 Applies to wireless LANs and provides 1- or 2-Mbps transmission in the 2.4-GHz band using either frequency hopping spread spectrum (FHSS) or direct sequence spread spectrum (DSSS).

  • 802.11a An extension to 802.11 that applies to wireless LANs and provides up to 54 Mbps in the 5-GHz band. 802.11a uses an orthogonal frequency division multiplexing encoding scheme rather than FHSS or DSSS.

  • 802.11b An extension to 802.11 that applies to wireless LANS and provides 11 Mbps transmission (with a fallback to 5.5, 2, and 1 Mbps) in the 2.4-GHz band. 802.11b uses only DSSS. 802.11b was a 1999 ratification to the original 802.11 standard, enabling wireless functionality comparable to Ethernet. 802.11b is also referred to as 802.11 High Rate or Wi-Fi.

  • 802.11g Applies to wireless LANs and provides 20+ Mbps in the 2.4-GHz band.

802.11b wireless LAN (WLAN) networks allow multiple devices to transmit data, voice, and video at data rates up to 11 Mbps. Although WLAN networks allow similar types of traffic to be transmitted over them, they differ from wired networks in the following ways:

  • WLANs operate as a shared medium, which means that communication on the WLAN is half duplex and that all devices within a single WLAN share the 11 Mbps.

  • WLAN bandwidth varies based on the distance that a WLAN client is from a WLAN access point (AP). The farther the distance, the lower the data rates at which traffic is supported.

  • Because all WLAN traffic is seen by all other WLAN devices (within range), additional security considerations need to be taken to ensure that traffic is not captured or manipulated by intruders.

Before examining a voice-traffic WLAN deployment, this section discusses the first activity that every network administrator should undertake: the completion of a site survey of the environment where the WLAN devices are deployed.

The same rules for deploying QoS in a wired network apply to deploying QoS in a wireless network. (See Chapter 5, "QoS and MPLS: Tools to Manage Application Performance.") The first and most important guideline in QoS deployment is to know your traffic. You should know your protocols, the application's sensitivity to delay, and the traffic bandwidth. QoS does not create additional bandwidth; it just gives more control of where the bandwidth is allocated. Voice traffic is probably the most familiar QoS application, and the most likely to benefit from the application of QoS mechanisms; however, other applications will require assistance.

WLAN Site Survey

The site survey should be performed to understand the number of APs required to provide radio frequency (RF) coverage. It should take into consideration which types of antennas provide the best coverage, as well as where sources of RF interference exist.

An important factor to keep in mind is that rarely do two physical environments have the same RF characteristics. Because of this, the discussion of RF is generalized in this chapter. Network administrators may need to adjust these parameters to their specific RF environment and requirements.

Seeking to serve as wide an area as possible, most network managers have focused on planning for adequate coverage. According to the 802.11 standard (this applies to 802.11a, 802.11b, and 802.11g), all connections must adjust their data rate upward or downward in response to changes in the RF environment. These environmental changes can manifest themselves as changes in signal levels, interference levels, or both. This variable data rate requires a trade-off between RF coverage and service levels, which has become a crucial design consideration. Figure 12-3 shows data-rate falloff for 802.11b as a function of distance from the AP.

Figure 12-3. Distance vs. Data Rate

[View full size image]

In a multistory, multitenant building, it is not enough for a network manager to just assign channels for the APs within the office. Because the RF radiation pattern from omnidirectional APs is analogous to a partially deflated beach ball, radio energy is radiated in a roughly spherical 360-degree pattern, which means the network manager must think three-dimensionally.

Interference can come from above, below, and through the wall from the company next door. Therefore, the network manager must have RF measurement and management tools.

With the high cost of early APs, designing WLANs for maximum coverage made sense. APs were positioned to cover the maximum amount of area with little or no overlap. Specific areas within buildings were targeted as wireless zones, analogous to today's public hotspots.

As the WLAN begins to cover multiple floors, multiple buildings, and hundreds of users, the complexity of the task exceeds the capacity of the manual tools currently available. Automated design tools are required that take into account different building materials, the characteristics of the APs, and a deep understanding of the three-dimensional nature of the planning problem.

Designing for coverage suffers from a lack of predictability in connection rates and, consequently, service levels. More importantly, as the number of users associated with an AP increases, and they begin to use different connection speeds, the entire system slows down. Data can be transferred only as fast as the slowest device, and if there is a mix of mobile devices running at different speeds (such as 1 Mbps, 2 Mbps, 5.5 Mbps, and 11 Mbps), the faster devices must wait for the slower devices to complete their transmissions before the channel is released.

The increase in WLAN usage is forcing a different perspective on design. WLANs must now be designed to provide predictable service levels to accommodate critical applications and isochronous data streams used for voice and video. The ability to meet service levels further implies that the network manager can control bandwidth, deliver a measure of QoS, and use policy management to support various service levels as mandated by the enterprise's business needs.

Service level requirements are driven by business requirements and by the applications that will be run. On a wired LAN, not everyone will need a 100-Mbps connection, nor does a pressing need for gigabit connections exist, because not many workstations can process that data rate. As a result, the network manager ends up with a mix of bandwidth requirements, with slower speeds going to peripheral devices or administrative tasks, whereas interactive users or those with graphic-intensive applications are assigned higher speeds.

Wired LAN switches can compensate for the speed differentials through internal buffering, and as a result, none of the users is aware that the speed differences exist. WLANs cannot compensate for different connections, and so all the users on an AP are affected when a slowdown occurs.

Although designing a WLAN to meet service levels rather than just provide coverage seems counterintuitive, installing enough APs to provide uniform, predictable bandwidth will result in greater overall coverage than that derived by planning for coverage alone. The aim is to design a seamless RF umbrella that delivers uniform signal levels to each mobile device regardless of location.

To ensure predictable service levels, data rates must not be allowed to fall below a minimum level. Therefore, minimum acceptable association rates for each mobile device must be specified. These rates should be based on user requirements, which implies that user-oriented policy-based management is necessary.

In a design based on service level, a large number of APs are used to form micro-cells. Although wired networks are fairly static, WLANs are organic entities that need to grow and change rapidly in response to their use. Providing mobility modifies the social dynamics of the enterprise and the way people work. The WLAN must respond to these dynamics as they emerge. Figure 12-4 shows two WLAN designs. The design on the right is the traditional design using just enough APs to provide basic coverage. The design on the left is a based on service level and uses more APs to ensure the strongest possible signal across the whole coverage area.

Figure 12-4. A Large Number of APs Assist a Design Based on Service Level

To react to changes in network utilization, the system must track user movements. Although this may raise some privacy concerns, this tracking is not intended to find out the behavior of specific individuals, but rather to develop an idea of usage patterns.

Managing the RF link is critical to meeting service levels. A highly desirable feature of WLAN systems is the ability to match AP location to physical building plans, thus creating a set of as-built network diagrams. WLAN systems can also convert design plans into configuration data for APs and other system elements and push that configuration data out to all devices automatically. This automation will be increasingly critical over time as enterprise WLANs increase in size.

The shared nature of WLANs can also benefit from policy-based management. Defining access groups with restrictions on access and bandwidth can assist in managing a scarce resource, namely the RF link. These policies can be further based on location and can define the type of encryption and authentication required, defining guest access as a means of controlling access by visitors to the enterprise network. Typically, this permission level allows only Internet access.

Continually measure the usage of APs to provide valuable information necessary to properly scale the WLAN as user traffic changes. The use of APs acts as a gauge to indicate when additional APs, access controllers, and Internet bandwidth are necessary. In addition, network monitoring should also keep an eye on sources of RF interference and raise flags when the interference is high enough to cause significant degradation in throughput.

The key to good performance in a WLAN is a good base design with the following key considerations:

  • Site surveys do not end after the APs are deployed. The management system must point the way to changes necessary after the network begins operation to ensure coverage areas remain uniform and that established policies can be met.

  • APs must be placed so that coverage areas overlap. This will provide sufficient signal strength to prevent data rate falloff as users move around the office.

  • The system must provide load balancing to accommodate conditions where one AP has more associated mobile devices than another. Load balancing must keep as uniform a load as possible across the RF umbrella.

  • Policies should be established that limit connection speeds to the highest available. This will provide a uniform connection experience for all users. Alternately, data rates can be kept lower, allowing fewer APs to be deployed while managing the user experience.

  • A design with closely packed APs will require more than three channels to be successful. As a result, service levelbased WLANs need to be based on 802.11a. Dual-band network interface cards (NICs) should be placed in mobile devices to allow access to hotspots and wired networks.

  • WLAN systems must provide management software that detects dead zones, and must provide "heat maps" depicting RF coverage areas.

  • RF management must be performed continuously, to allow rapid reaction to changing conditions within the WLAN and the associated RF environment.

During deployment and installation of a WLAN, consider the type and characteristics of applications that will be deployed in the environment. In some cases, interfaces with applications located on various hosts and servers can bring about major problems when using a WLAN. A relatively short loss of connectivity because of RF interference or poor coverage area causes some applications to produce errors. This occurs mostly with legacy applications that lack error-recovery mechanisms for wireless systems.

For example, a user may be using an inventory application by scanning items and entering total counts via a keypad on the scanner. If loss of connectivity occurs after scanning the bar code and before entering the count, the host-based application could log the user out without completing the inventory transaction. As a result, the application on the host may record an incorrect or invalid value for the inventory item.

To avoid these types of risks, carefully define the types of applications the wireless user devices will interface with. If needed, incorporate solutions such as wireless middleware to adequately handle recovery mechanisms related to WLANs. The middleware will often center on a WLAN-friendly application such as a browser interface that allows access to a back-end legacy system. The characteristics that lend themselves to a WLAN environment are similar to those of a latency-bound application. (See Chapter 3.) Traditionally, this has meant that an application that has few application turns will not be too bandwidth intensive, and importantly will have the capability to recover from session disconnect caused by dropped packets, intermittent network connectivity, and so forth.

Today, with their expansion into enterprise and vertical (retail, finance, education) markets, WLANs are used to transport high-bandwidth, data-intensive applications in conjunction with time-sensitive multimedia applications. This usage has led to the necessity for wireless QoS.

Supporting Voice Traffic

When most people think of WLANs, they generally only consider transferring data while using applications such as a web browser, e-mail client, file transfer, and so forth. It is possible, however, to use a WLAN as the transport system for carrying telephone traffic from mobile users as well.

Performance should be your primary consideration when installing a WLAN system that supports voice. 802.11b is only capable of running three uncompressed audio streams smoothly. As a result, it is important that the system compress the audio signals before transmission to increase the number of supported audio streams.

Another way to increase the number of audio streams is to use a higher-performing standard, such as 802.11a, for the WLAN backbone. 802.11a has the capacity to handle approximately four times as much voice traffic as 802.11b. The problem with 802.11a, however, is that other nonvoice users of the network may be equipped only with 802.11b. As a result, consider installing APs that include both 802.11a (for voice users) and 802.11b (for data users).[2]

Although Voice over IP (VoIP) does convert voice signals into IP data packets and converges them with data traffic, the requirements of voice and data are very different when transmitted over an IP network, as shown in Table 12-2.

Table 12-2. Comparing Data and Voice Traffic Characteristics

Data Traffic

Voice Traffic

BurstyTraffic can be sent in large or small bursts, depending on the application.

SmoothPackets are sent at consistent intervals with uniform packet sizes.

Bandwidth "greedy"Applications attempt to use as much bandwidth as the network allows.

BenignPackets only attempt to use the bandwidth necessary to send from end to end. VoIP does not use any windowing to determine data rates.

Packet-loss insensitiveRetransmission capabilities allow data to continue to work correctly even with packet loss on the network.

Packet-loss sensitiveTraffic is extremely sensitive to packet loss. Excessive loss degrades overall voice quality.

Delay insensitiveCan handle some packet delay without effecting overall performance.

Delay sensitiveVoIP can tolerate some amount of delay, but excessive delay or jitter degrade overall quality.

The network guidelines defined by Cisco Architecture for Voice, Video and Integrated Data (AVVID) for proper VoIP operation are as follows:

  • Delay Not to exceed 150 ms (one way)

  • Delay variation (jitter) Not to exceed 30 ms

  • Packet loss Not to exceed 1 percent

This presents a problem given the earlier statement regarding application characteristics of WLAN environments. As mentioned earlier, WLANs operate as a shared medium. This means that QoS on WLAN networks is more difficult than on wired networks because endpoints do not have dedicated bandwidth for sending and receiving traffic. Table 12-3 compares the characteristics of a wired and a wireless VoIP QoS.

Table 12-3. Comparing Wired and Wireless VoIP QoS Characteristics

Wired LAN VoIP QoS

WLAN VoIP QoS

Dedicated access per user or device (switched Ethernet, point-to-point [P2P] WAN).

Shared access to bandwidth.

Packets marked with 802.1p and IP Type of Service (ToS) or Differentiated Services (DiffServ).

Packets marked with 802.1p and IP ToS or DiffServ.

Can provide hard QoS.

Provides soft QoS.

QoS can be applied to upstream or downstream traffic.

QoS is currently available to downstream traffic from the AP, but few devices other than the Cisco 7920 provide upstream QoS toward the AP.

Can provide hard admission control.

Can only provide limited admission control.

Unlike networks with dedicated bandwidth, WLAN networks have to consider traffic direction with regard to QoS. Traffic needs to be considered as either upstream or downstream from the point of view of the AP, as shown in Figure 12-5.

Figure 12-5. Upstream and Downstream Traffic Patterns

[View full size image]

Radio downstream QoS refers to the traffic leaving the AP and traveling to the WLAN clients. Radio upstream QoS refers to traffic leaving the WLAN clients and traveling to the AP. No vendor support is currently available for radio upstream QoS features for WLAN clients. This support is specified in the 802.11e draft, but has not yet been implemented.

Ethernet downstream refers to traffic leaving the switch or router and traveling to the AP. QoS may be applied at this point to prioritize and rate-limit traffic to the AP. Ethernet upstream refers to traffic leaving the AP and traveling to the switch. The AP classifies traffic from the AP to the upstream network.

The application of QoS features may not be easily detected on a lightly loaded network. If latency, jitter, and loss are noticeable when the media is lightly loaded, it indicates only a system fault or that an application's latency, jitter, and loss requirements are not a good match for the network.

QoS features start to impact application performance as the load on the network increases. QoS works to keep the latency, jitter, and loss for selected traffic types within acceptable bounds.

By providing downstream prioritization from the AP, upstream client traffic is treated as best effort. A client has to compete with other clients for (upstream) transmission as well as compete with best-effort (downstream) transmission from the AP. Under certain load conditions, a client can experience upstream congestion, and the performance of QoS-sensitive applications may be unacceptable despite the QoS features on the AP.


Complete End-to-End Model

The importance of managing the application performance of enterprise networks has grown as the networks have evolved. One job of the network manager is to implement solutions that monitor and prioritize application traffic and optimize the network for the most critical applications.

Network and Application Performance Management

Network and application performance management have evolved over time as networks have become more complex. An enterprise network is a system of interconnected pipes with information flowing from one place to another. As the amount of traffic increases, these pipes can become congested, resulting in performance degradation. To ease this congestion, enterprises can use one or both of the following approaches:

  • Infrastructure-focused approach Eliminating congestion by provisioning enough capacity or adding bandwidth

  • Management-focused approach Managing the traffic that flows through these enterprise network pipes, effectively dividing the pipe so that some applications are ensured the bandwidth they need to perform effectively

Most enterprises require a combination of both approaches to guarantee network and application performance.

Infrastructure Scaling

Scaling the infrastructure is a common practice for addressing performance issues. Managers can add more bandwidth, more processing power, or both. Some enterprises even require distributed, geographically diverse data centers and networks. Load-balancing devices can share application requests and further improve performance. This approach also has the added feature of being a redundant solution, good for companies needing disaster recovery.

The major drawbacks to scaling the physical infrastructure are cost and effectiveness. Depending on location and link size, bandwidth may or may not be terribly expensive, but redundant architectures are expensive to build and maintain. In addition, they do not always work. Voice and video certainly need a minimum level of throughput, but additional bandwidth alone will not alleviate performance problems such as latency and jitter.

Traffic Management

Eventually, enterprises reach the limits of scaling their infrastructure and realize that the more effective strategy is efficient management. The primary methods for managing the available resources include performance management software and application traffic management.

Legacy network management tools, which are good at isolating faults, alert network managers to device failures. Most enterprises have now augmented their network management tools with a performance management software package, which is good at alerting network managers to performance issues and, in some cases, correlating these events with network management consoles. This provides a way to isolate whether a performance issue is being caused by a network problem. These tools provide a necessary and valuable function, but they only offer a reactive view. For example, they do not actually manage traffic; they alert administrators to a performance issue as it happens. It is still up to the administrator to isolate and fix.

Along with network performance management software, most enterprises and service providers should use a form of application traffic management. This approach includes a combination of detailed application discovery and performance monitoring, establishing QoS policies, traffic shaping, and compression.

Figure 12-6 shows the end-to-end system in its simplest form. It comprises three groups of elements: client, network, and server.

Figure 12-6. End-to-End System in Its Simplest Form

[View full size image]

To achieve overall optimization for this delivery system, it is essential to focus on the big picturethe interaction between these three groups of elements. Until now, this book's focus has been on optimizing the underlying infrastructure. To a degree that will remain the focus, but this section starts to explore the simple approaches to optimizing the overall system. At this point, note that optimization techniques for specific server and application architectures will remain recommendations, and specific experts should carry out any detailed tuning. The idea behind identifying areas within these systems that can be optimized is to assist in the development of an overall view with common information. It is not the network engineer's responsibility to carry out that optimization, but quick recognition, coupled with clear concise identification, will ultimately ease the overall process.

End-to-End Optimization Process

The method for achieving the overall end-to-end optimization remains a prioritized, orderly, goal-oriented, holistic approach. The process can be as simple as reacting to calls from users who identify a performance problem, or as complex as calling in a team of experts to evaluate a system from top to bottom. To be effective, however, it must consider the entire system in all three phases. Any reliable optimization methodology must include the following:

  • Baseline readings

  • Established performance goals

  • Structure and tracking of changes

  • Evaluation of effect of changes

  • Comparison with established performance goals

  • Reiteration until goals are met

Remember if the process is too complex, it is less likely that anyone will adhere to it. A good methodology should be simple, yet have the core components of problem solving. It must focus on specific issues (it is no good to say I want to optimize the application performance), and it must have a definite endpoint. Too many efforts fail because no definite endpoint is set. The best methodology will always consider the effect of changing one component on other affected components. It must be holistic to truly succeed. You cannot increase the priority and bandwidth allocation on one application without expecting it to impact others; alternatively, you cannot increase the buffer allocation on a server for a specific instance and not expect it to impact the other jobs on the system. It has to be a matter of balance.

Establishing a Baseline

The first step is to establish the baseline, and in doing so identify the most probable cause of the performance bottleneck. Remember that this is based on general assignment of client, network, or server.

The simplest way is to take the business-critical transactions (as defined in Chapter 3) and track their progress at various points of the transmission path. You can perform this monitor through specific application performance management (APM) tools such as NetFlow, or by using a synthetic-based transaction such as the Cisco Service Assurance Agent (SAA). (See Chapter 8.)

Emulating the Application Traffic

The SAA application performance monitor enables you to monitor the performance of network-hosted applications by emulating the traffic of these applications. The SAA application performance monitor can also emulate and measure the performance of SAA Frame Relay and path jitter operations.

Monitoring the performance of network-hosted applications enables service providers and IT departments to verify that applications are performing as needed, and to implement improvements as necessary.

Application performance monitoring operates differently from most SAA operations. Standard SAA operations allow the monitoring of protocol-based activity between a source and a destination, whereas application performance monitoring operations monitor software application transactions between a source and a destination.

Application performance monitoring provides a way of monitoring client/server applications between devices by emulating application transactions. When using application performance monitoring, one Cisco device is the application server node, and another is the application client node. Using SAA terminology, the transmitting device (the server) is the agent, and the receiving device (the client) is the responder. A single Cisco device can function as both a client and a server for network applications.

Applications are emulated through the use of downloadable scripts and configuration files. Application performance monitoring configuration files contain information about which script and scheduler files to run. These configuration files use the .cf filename extension. Script files are written in TCL and use the .scr extension. Execution of application performance-monitoring operations can be scheduled through the use of scheduler files, which use the .sch extension.

The following application performance monitoring script files are examples of scripts embedded on the router running the SAA:

  • frm.scr Script for emulating Frame Relay Monitor operations

  • iptv.scr Script for IPTV, NetMeeting, and Real Networks audio and video file emulation

  • ldap.scr Script for Lightweight Directory Access Protocol (LDAP) emulation

  • lotus_snd.scr Script for Lotus Notes "send e-mail" emulation

  • nntp.scr Script for Network News Transfer Protocol (NNTP) emulation, measures the time required for retrieving a news article using NNTP

  • pattern.scr Script for ACSII/binary/hex pattern-matching for data integrity

  • pjm-many.scr Script for emulating Path Jitter Monitor operations

  • pop3.scr Script for POP3 "retrieve e-mail" emulation

  • sap-auth.scr, sap-inv.scr, sap-login.scr, sap-pur.scr Scripts for SAP business-to-business database transaction emulation (SAP authentication processes, SAP invoice generation, SAP login process, and SAP purchase-order generation)

  • smtp.scr Script for Simple Mail Transfer Protocol (SMTP) "send e-mail" emulation

  • tcp.scr Script for RTT measurement over TCP/IP

  • udp.scr Script for RTT measurement over UDP

Identifying the Bottleneck

It is possible to identify the largest wait time using the transaction's progression. From here a quick assessment of the method or approach to optimization can be made.

Compare the overall response time to the demarcation point timings to identify the slowest part of a transmission. Your initial concentration should be centered in this simple example.

First, record the time at each demarcation point of Figure 12-6. Then by subtracting the times at each point, you can work out how long each element in the conversation took.

The total time for the client to send a business query and receive a response from the server was 4.4 seconds. The client took 0.8 seconds to process the request. The data was on the network for 0.5 seconds, and the server took 3.1 seconds to provide the data.

Handing Over the Problem

Having identified the server as the most probable cause of the performance bottleneck, do not just pass the problem over to the server group with a glib "it's not our problem" approach. That is counterproductive and will potentially extend the time to resolution for the performance issue.

Instead, hand over the problem with as much relevant information as possible. For example, take time to return data for a specific query. (The majority of database performance issues arise because of bad Structured Query Language [SQL] or a function that appears to take an excessive amount of time [such as initial logon].) This information will then enable the server team to focus their investigation and tuning efforts.

Tuning

In its simplest form, tuning means providing the correct amount of resources to the application so that users are not kept waiting for their data. Remember that supplying the correct resources does not mean just throwing more memory or CPUs into the server. It may mean redesigning parts of the system, or scheduling work and batch jobs appropriately to better use the existing resources in a uniform fashion.

It is beyond the scope of this book to detail or discuss specific server-based system tuning. The database administrator (DBA) is usually in the best position to determine where the system bottlenecks reside, and should be given pertinent information based on the end-to-end analysis to make the necessary tuning.

Application Classification

Having tuned the performance of the server, it is observed that the network could potentially give more performance improvements. At this point, consider applying some QoS parameters to the transmission path.

For simplicity, the applications can be classified into the following five groups, having different network and computing requirements:

  • Data transfers

  • Video/voice streaming

  • Interactive video/voice

  • Mission-critical applications

  • Web-based applications

The challenge is to align the network and computing services to the application's service requirements from a performance perspective.

Data Transfers

The data transfer class includes applications such as FTP, e-mail, and so forth. This class of applications tends to have zero tolerance for packet loss and high tolerance for delay and jitter. Typical acceptable response times range from a few seconds for FTP transfers to hours for e-mail. Bandwidth requirements in the order of kilobits per second are acceptable, depending on the file size and keeping response times in the order of a few seconds. Depending on the characteristics of the application (for example, size of files), disk I/O transfer times, along with network bottlenecks, can contribute cumulatively to delays.

Video/Voice Streaming

The video/voice streaming class includes applications such as Apple's QuickTime Streaming or Real Networks streaming video and voice products. This class of applications tends to have some tolerance for packet loss and medium tolerance for delay and jitter. Typical acceptable response times are in the order of a few seconds because the server can prebuffer multimedia data on the client to a certain degree. This buffer then drains at a constant rate on the client side, while simultaneously receiving bursty streaming data from the server with variations in delay. As long as the buffer can absorb all variations by not draining empty, the client sees a constant stream of video and voice. Typical bandwidth requirements are in the order of megabits per second, depending on frame rate, compression/decompression algorithms, and the size of images. Disk I/O and CPU can also contribute to delays. Large MPEG files need to be read from disks and compression/decompression algorithms.

Interactive Video/Voice

The interactive video/voice class of applications tends to have some tolerance for packet loss and low tolerance for delay and jitter. Typical bandwidth requirements are tremendous, depending on the number of simultaneous participants in the conference, growing exponentially. Due to the interactive nature of the data being transferred, tolerances are very low for delay and jitter. As soon as one participant moves or talks, all other participants need to immediately see and hear this change. Response time requirements range from 250 to 500 ms; this is compounded by the bandwidth requirements, with each stream requiring a few megabits per second. So in a conference of several participants, each participant will be pumping out his or her voice and video streams and will simultaneously receive other participants' streams.

Mission-Critical Applications

The mission-critical class of applications varies in bandwidth requirements, but tends to have zero tolerance for packet loss. Bandwidth requirements are in the order of kilobits per second, depending on the application. Response times vary from 500 ms to a few seconds. Server resource requirements vary, depending on the application, in terms of CPU, disk, and memory.

Web-Based Applications

The web-based class of applications tends to have low bandwidth requirements (unless large image files are associated with the requested web page) and is growing in CPU and disk requirements, as a result of dynamically generated web pages and web-based transaction applications. Response time requirements range from 500 ms to 1 second.

End-to-End QoS

To fully understand end-to-end QoS, it is useful to take a look at the details of the complete network path traversal, starting from the point where a client sends a request, traversing various network devices, and finally reaching the destination where the server processes the request. There are different classes of applications, having different characteristics and requirements. Several federated networks, combined with different traffic characteristics, make end-to-end QoS a complex issue.

Figure 12-7 provides a simplified high-level perspective of the components involved in an end-to-end packet traversal of an enterprise that does not have its own backbone service, but uses the services provided by an incumbent local exchange carrier (ILEC), competitive local exchange carrier (CLEC), or inter-exchange carrier (IXC). These can be called either network service providers (NSPs) or Internet service providers (ISPs) because almost all NSPs now also provide Internet access.

Figure 12-7. Overall Journey View

[View full size image]

It is important to remember that you can implement QoS in many different ways. Each domain has control over its resources and can implement QoS on its portion of the end-to-end path using different technologies. The following are the main domains:

  • Enterprise

  • NSP

  • ISP

Enterprise

Enterprises can control their own networks and systems. Enterprise networks can control their LANs. From a local Ethernet/Token Ring LAN perspective, IEEE 801.p can be used to mark frames according to priorities that in turn allow the switch to offer preferential treatment to certain flows. Similarly for VLANs, IEEE 802.q provides a Layer 2 marking scheme. For computing devices, there are facilities that allow processes to run at higher priorities, thus obtaining differentiated services from a process computing perspective. Figure 12-8 shows the marking schemes.

Figure 12-8. Packet-Marking Identification

[View full size image]

Network Service Provider

NSPs aggregate traffic and forward it within their own networks or hand it off to another NSP. NSPs can use technologies such as differentiated services or integrated services to prioritize handling of traffic within their networks. SLAs are required between NSPs to obtain a certain level of QoS for transit traffic. NSPs implement QoS at both the IP layer and ATM layer.

Internet Service Provider

Most ISPs still have ATM networks that carry IP traffic. ATM itself offers the following six types of QoS service:

  • Constant bit rate (CBR) Provides constant bandwidth, delay, and jitter throughout the life of the ATM connection

  • Variable bit rate (real time) (VBR[RT]) Provides constant delay and jitter, but variations in bandwidth

  • Variable bit rate (non-real time) (VBR[NRT]) Provides variable bandwidth, delay, and jitter, but very low cell-loss rate

  • Unspecified bit rate (UBR) Provides best-effort service, no guarantees

  • Available bit rate (ABR) Provides no guarantees, expects the applications to adapt according to network availability

  • Guaranteed frame rate (GFR) Provides a minimum frame rate, delivers entire frame or none, used for AAL5

Network Transmission Path

As you can see in Figure 12-7, a typical packet journey can begin on the enterprise server at Enterprise Network 1. Servers can get congested, even though network bandwidth from end to end may be completely clear. A typical example occurs when web pages are served dynamically by servlets; servers often reach a limit on the number of requests that can be handled.

The packet routes its way out of the enterprise network, toward the edge router connected to the service provider. Before reaching the edge router, the packet may have traversed several layers of switches, which typically include LAN switches, and then a backbone switch. QoS settings at the LAN side of the network campus are about managing transmit buffers in switches, as opposed to controlling bandwidth. This is because in theory, LAN-based networks generally do not suffer from bandwidth-allocation problems. Figure 12-9 shows a typical transmit buffer in a switch. By applying QoS settings at this level, you will be able to use multiple queues to prevent transmit-queue congestion and resulting drops, and thus keep latency and jitter low.

Figure 12-9. Representation of Queuing on Switch Transmit Buffers

The final device in Enterprise Network 1 of Figure 12-7 could be an egress customer premises equipment (CPE) device, such as a channel service unit/data service unit (CSU/DSU), which terminates the WAN link. Typically, these are relatively expensive links and tend to be the network bottleneck.

At this point is where QoS from a network perspective makes a significant amount of difference, by controlling what traffic is prioritized and sent over this relatively slow, shared link. The enterprise edge switch must implement some form of QoS by restricting the amount of traffic that is injected into the WAN link. At this point, the edge switch, during episodes of congestion, makes sure that premium packets get priority over less-important network traffic packets.

The packet then reaches the Tier 2 service provider network at B. The NSP will forward the traffic to the destination, via that NSP backbone provider, which then will forward to a peer backbone provider, either via a public peering point or a network AP. The destination-side backbone provider will then forward the packets to the destination Tier 2 provider or directly to a subscriber, depending on the IP address of the destination.

Finally, the destination will receive the packets from the server through the access networks, as shown in Figure 12-7.

From a non-realtime system computing perspective, unpredictable delays can result from limited CPU resources or disk I/O latencies, which degrade during heavy load. From a network perspective, many components add up to the cumulative end-to-end delay, such as physical distance, number of hops (including components such as firewalls), and overall link speed. Remember that the slowest pipe on the journey governs the throughput.

In general, QoS is needed to control access to shared resources during episodes of congestion. In this example, the shared resources are servers and certain links. Figure 12-10 and Table 12-4 show allocation of QoS settings on an end-to-end system.

Figure 12-10. Simple Overview of QoS Scheme Placement

Table 12-4. Allocation of QoS Settings

QoSHQ Access

QoSHQ Distribution

QoSWAN

QoSBranch

Speed and duplex settings

Classification trust on IP phones, VC systems, and content services

Multiple queues on IP phones and APs

Layer 3 provisioning for content distribution

Multiple queues on all ports, priority queuing for VoIP

WRED within data queues for congestion management

Low-latency queuing

Data traffic queue provisioning

Window and fragmentation control

Traffic shaping

Admission control

Speed and duplex settings

Classification trust on IP phones, VC systems, and content services

Multiple queues on IP phones and APs

Applying QoS Policies

Ensuring these policies are accurately deployed to a dispersed network can be a complex task. The best approach is to use an automated policy engine, such as the Cisco Quality Policy Manager (QPM). QPM deploys the QoS policies in a deployment group to your network devices. A deployment group can consist of a group of devices, such as edge switches or complete sites. When you deploy your QoS policies to network devices, QPM translates the policies into device commands and enters the commands through the device's command-line interface (CLI). You can choose to deploy your QoS configurations directly to network devices using Telnet. QPM automatically deploys your QoS configurations to configuration files. This deployment process does not configure the devices, but generates configuration files that can be sent manually to the devices. QoS configurations can be deployed to the device using any application that downloads configuration files to the devices.

Referring back to the simple client network server model of Figure 12-6, the final area of potential optimization is the client. The client aspect is along similar lines to the server, in as much as identifying the most probable cause of delay is generally where the network engineer's responsibility ends.

The client will differ from the server in that it is unlikely to be running network applications accessed by multiple users. Instead, it may have a subset of an application (such as a form) that is used to interact with the main application server. In general, issues that are associated with the server involve resource issues such as physical memory and processing loads caused by coexisting applications.


Summary

Storage area network is an emerging data communications platform that interconnects servers and storage at gigabit speeds. SAN traffic is synchronous by nature, and to ensure the application makes the most of the bandwidth it is best to fill as much of the pipe as possible. SAN traffic is transported by TCP, and the following conditions assist in optimizing its delivery:

  • Large window sizes

  • Traffic shaping or rate limiting

  • Selective acknowledgments (SACK)

The same rules for deploying QoS in a wired network apply to deploying QoS in a wireless network. The first and most important guideline in QoS deployment is to know your traffic. Know your protocols, application's sensitivity to delay, and traffic bandwidth. Remember that QoS does not create additional bandwidth; it just gives more control over where the bandwidth is allocated.

The best way to achieve an overall end-to-end optimization is to follow a prioritized, orderly, goal-oriented, and holistic approach. Such a process can be as simple as reacting to calls from users who identify a performance problem, or as complex as calling in a team of experts to evaluate a system from top to bottom. To be effective, however, the process must consider the entire system in all three phases: client, network, and server.


End Notes

  1. SACK Limitations, http://www.cisco.com/en/US/products/hw/ps4159/ps2160/products_configuration_guide_chapter09186a00801f96a6.html

  2. Voice over Wireless LANs Jim Geier, Principal Consultant, Wireless-Nets, Ltd. http://www.wi-fiplanet.com/tutorials/article.php/2171721


References Used in This Chapter

  • Cisco Network Design Considerations, http://www.cisco.com/en/US/netsol/ns340/ns19/ns24/networking_solutions_white_paper09186a00801c796d.shtml

  • Cisco Systems, Wireless Quality of Service Deployment Guide, http://www.cisco.com/en/US/products/hw/phones/ps379/products_white_paper09186a00801ca827.shtml#wlan_qos_voip

  • Deepak Kakadia, Sun Microsystems, Enterprise QoS Policy Based Systems & Network Management, http://wwws.sun.com/software/bandwidth/wp-policy/

  • Cisco Systems, Wireless Quality of Service Deployment Guide, http://www.cisco.com/en/US/products/hw/phones/ps379/products_white_paper09186a00801ca827.shtml#wlan_qos_voip

  • RFC 1323, TCP Extensions for High Performance, V. Jacobson, R. Braden, D. Borman, May 1992, http://rfc.net/rfc1323.html

  • Cisco Systems, User Guide for CiscoWorks QoS Policy Manager, http://www.cisco.com/en/US/products/sw/cscowork/ps2064/products_user_guide_book09186a00800e09f6.html

  • Cisco Systems, Product Guide Service Assurance Agent, http://www.cisco.com/en/US/products/sw/iosswrel/ps1830/products_feature_guide09186a0080087a78.html


Part VI: Appendixes

Appendix A QoS-Related Requests For Comments

Appendix B Guide to QoS Standards

Glossary


Appendix A. QoS-Related Requests For Comments

A Request For Comment (RFC) is an official document of the Internet Engineering Task Force (IETF) that specifies the details for protocols included in the TCP/IP family. These are an evolving series of reports and proposals for protocols. You can access RFCs from a number of websites. A source that provides a search engine can be found at the following website:

http://www.rfc-editor.org/rfc.html

The RFCs presented here in Table A-1 refer mainly to QoS-related issues.

Table A-1. QoS-Related RFCs

RFC No.

RFC Title

Status

3729

Application Performance Measurement MIB

3728

Definitions of Managed Objects for Very High Speed Digital Subscriber Lines (VDSL)

3705

High Capacity Textual Conventions for MIB Modules Using Performance History Based on 15 Minute Intervals

3670

Information Model for Describing Network Device QoS Datapath Mechanisms

3662

A Lower Effort Per-Domain Behavior (PDB) for Differentiated Services

3649

HighSpeed TCP for Large Congestion Windows

3644

Policy Quality of Service (QoS) Information Model

3630

Traffic Engineering (TE) Extensions to OSPF Version 2

~2370

3593

Textual Conventions for MIB Modules Using Performance History Based on 15 Minute Intervals

2493

3583

Requirements of a Quality of Service (QoS) Solution for Mobile IP

3578

Mapping of Integrated Services Digital Network (ISDN) User Part (ISUP) Overlap Signalling to the Session Initiation Protocol (SIP)

3564

Requirements for Support of Differentiated Services-aware MPLS Traffic Engineering

3545

Enhanced Compressed RTP (CRTP) for Links with High Delay, Packet Loss and Reordering

3540

Robust Explicit Congestion Notification (ECN) Signaling with Nonces

3511

Benchmarking Methodology for Firewall Performance

3496

Protocol Extension for Support of Asynchronous Transfer Mode (ATM) Service Class-aware Multiprotocol Label Switching (MPLS) Traffic Engineering

3487

Requirements for Resource Priority Mechanisms for the Session Initiation Protocol (SIP)

3483

Framework for Policy Usage Feedback for Common Open Policy Service with Policy Provisioning (COPS-PR)

3477

Signalling Unnumbered Links in Resource ReSerVation Protocol - Traffic Engineering (RSVP-TE)

3476

Documentation of IANA Assignments for Label Distribution Protocol (LDP), Resource ReSerVation Protocol (RSVP), and Resource ReSerVation Protocol-Traffic Engineering (RSVP-TE) Extensions for Optical UNI Signaling

3474

Documentation of IANA assignments for Generalized Multi-Protocol Label Switching (GMPLS) Resource Reservation Protocol - Traffic Engineering (RSVP-TE) Usage and Extensions for Automatically Switched Optical Network (ASON)

3473

Generalized Multi-Protocol Label Switching (GMPLS) Signaling Resource ReserVation Protocol-Traffic Engineering (RSVP-TE) Extensions

3465

TCP Congestion Control with Appropriate Byte Counting (ABC)

3449

TCP Performance Implications of Network Path Asymmetry

=BCP0069

3432

Network performance measurement with periodic streams

3398

Integrated Services Digital Network (ISDN) User Part (ISUP) to Session Initiation Protocol (SIP) Mapping

3393

IP Packet Delay Variation Metric for IP Performance Metrics (IPPM)

3390

Increasing TCP's Initial Window

2414, ~2581

3387

Considerations from the Service Management Research Group (SMRG) on Quality of Service (QoS) in the IP Network

3357

One-way Loss Pattern Sample Metrics

3346

Applicability Statement for Traffic Engineering with MPLS

3317

Differentiated Services Quality of Service Policy Information Base

3308

Layer Two Tunneling Protocol (L2TP) Differentiated Services Extension

3290

An Informal Management Model for Diffserv Routers

3289

Management Information Base for the Differentiated Services Architecture

3287

Remote Monitoring MIB Extensions for Differentiated Services

3276

Definitions of Managed Objects for High Bit-Rate DSL - 2nd generation (HDSL2) and Single-Pair High-Speed Digital Subscriber Line (SHDSL) Lines Processing

3272

Overview and Principles of Internet Traffic Engineering

3270

Multi-Protocol Label Switching (MPLS) Support of Differentiated Services

3260

New Terminology and Clarifications for Diffserv

~2474, ~2475, ~2597

3248

A Delay Bound alternative revision of RFC 2598

3247

Supplemental Information for the New Definition of the EF PHB (Expedited Forwarding Per-Hop Behavior)

3246

An Expedited Forwarding PHB (Per-Hop Behavior)

2598

3244

Microsoft Windows 2000 Kerberos Change Password and Set Password Protocols

3222

Terminology for Forwarding Information Base (FIB) based Router Performance

3210

Applicability Statement for Extensions to RSVP for LSP-Tunnels

3209

RSVP-TE: Extensions to RSVP for LSP Tunnels

3182

Identity Representation for RSVP

2752

3181

Signaled Preemption Priority Policy Element

2751

3175

Aggregation of RSVP for IPv4 and IPv6 Reservations

3168

The Addition of Explicit Congestion Notification (ECN) to IP

2481, ~2474, ~2401, ~0793

3155

End-to-end Performance Implications of Links with Errors

=BCP0050

3150

End-to-end Performance Implications of Slow Links

=BCP0048

3148

A Framework for Defining Empirical Bulk Transfer Capacity Metrics

3135

Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations

3124

The Congestion Manager

3097

RSVP Cryptographic Authentication Updated Message Type Value

~2747

3086

Definition of Differentiated Services Per Domain Behaviors and Rules for their Specification

3084

COPS Usage for Policy Provisioning (COPS-PR)

3006

Integrated Services in the Presence of Compressible Flows

2998

A Framework for Integrated Services Operation over Diffserv Networks

2996

Format of the RSVP DCLASS Object

2990

Next Steps for the IP QoS Architecture

2983

Differentiated Services and Tunnels

2963

A Rate Adaptive Shaper for Differentiated Services

2961

RSVP Refresh Overhead Reduction Extensions

2940

Definitions of Managed Objects for Common Open Policy Service (COPS) Protocol Clients

2914

Congestion Control Principles

=BCP0041

2884

Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks

2872

Application and Sub Application Identity Policy Element for Use with RSVP

2861

TCP Congestion Window Validation

2859

A Time Sliding Window Three Colour Marker (TSWTCM)

2816

A Framework for Integrated Services Over Shared and Switched IEEE 802 LAN Technologies

2815

Integrated Service Mappings on IEEE 802 Networks

2814

SBM (Subnet Bandwidth Manager): A Protocol for RSVP-based Admission Control over IEEE 802-style networks

2758

Definitions of Managed Objects for Service Level Agreements Performance Monitoring

2753

A Framework for Policy-based Admission Control

2752

Identity Representation for RSVP

+3182

2751

Signaled Preemption Priority Policy Element

+3181

2750

RSVP Extensions for Policy Control

~2205

2749

COPS usage for RSVP

2748

The COPS (Common Open Policy Service) Protocol

2747

RSVP Cryptographic Authentication

~+3097

2746

RSVP Operation Over IP Tunnels

2745

RSVP Diagnostic Messages

2724

RTFM: New Attributes for Traffic Flow Measurement

2723

SRL: A Language for Describing Traffic Flows and Specifying Actions for Flow Groups

2722

Traffic Flow Measurement: Architecture

2063

2721

RTFM: Applicability Statement

2720

Traffic Flow Measurement: Meter MIB

2064

2702

Requirements for Traffic Engineering Over MPLS

2698

A Two Rate Three Color Marker

2697

A Single Rate Three Color Marker

2689

Providing Integrated Services over Low-bitrate Links

2688

Integrated Services Mappings for Low Speed Networks

2682

Performance Issues in VC-Merge Capable ATM LSRs

2681

A Round-trip Delay Metric for IPPM

2680

A One-way Packet Loss Metric for IPPM

2679

A One-way Delay Metric for IPPM

2678

IPPM Metrics for Measuring Connectivity

2498

2676

QoS Routing Mechanisms and OSPF Extensions

2647

Benchmarking Terminology for Firewall Performance

2638

A Two-bit Differentiated Services Architecture for the Internet

2598

An Expedited Forwarding PHB

+3246

2597

Assured Forwarding PHB Group

~+3260

2581

TCP Congestion Control

2001, ~+3390

2549

IP over Avian Carriers with Quality of Service

~1149

2508

Compressing IP/UDP/RTP Headers for Low-Speed Serial Links

2501

Mobile Ad hoc Networking (MANET): Routing Protocol Performance Issues and Evaluation Considerations

2498

IPPM Metrics for Measuring Connectivity

+2678

2493

Textual Conventions for MIB Modules Using Performance History Based on 15 Minute Intervals

+3593

2490

A Simulation Model for IP Multicast with RSVP

2481

A Proposal to add Explicit Congestion Notification (ECN) to IP

+3168

2475

An Architecture for Differentiated Service

~+3260

2474

Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers

1455, 1349, ~+3168, 3260

2430

A Provider Architecture for Differentiated Services and Traffic Engineering (PASTE)

2415

Simulation Studies of Increased Initial TCP Window Size

2414

Increasing TCP's Initial Window

+3390

2386

A Framework for QoS-based Routing in the Internet

2382

A Framework for Integrated Services and RSVP over ATM

2381

Interoperation of Controlled-Load Service and Guaranteed Service with ATM

2380

RSVP over ATM Implementation Requirements

2379

RSVP over ATM Implementation Guidelines

=BCP0024

2330

Framework for IP Performance Metrics

2309

Recommendations on Queue Management and Congestion Avoidance in the Internet

2215

General Characterization Parameters for Integrated Service Network Elements

2214

Integrated Services Management Information Base Guaranteed Service Extensions using SMIv2

2213

Integrated Services Management Information Base using SMIv2

2212

Specification of Guaranteed Quality of Service

2211

Specification of the Controlled-Load Network Element Service

2210

The Use of RSVP with IETF Integrated Services

2209

Resource ReSerVation Protocol (RSVP) Version 1 Message Processing Rules

2208

Resource ReSerVation Protocol (RSVP) Version 1 Applicability Statement Some Guidelines on Deployment

2207

RSVP Extensions for IPSEC Data Flows

2206

RSVP Management Information Base using SMIv2

2205

Resource ReSerVation Protocol (RSVP) Version 1 Functional Specification

~+2750

2124

Cabletron's Light-weight Flow Admission Protocol Specification Version 1.0

2123

Traffic Flow Measurement: Experiences with NeTraMet

2064

Traffic Flow Measurement: Meter MIB

+2720

2063

Traffic Flow Measurement: Architecture

+2722

2001

TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms

+2581

1859

ISO Transport Class 2 Non-use of Explicit Flow Control over TCP RFC1006 extension

1857

A Model for Common Operational Statistics

1404

1856

The Opstat Client-Server Model for Statistics Retrieval

1810

Report on MD5 Performance

1633

Integrated Services in the Internet Architecture: an Overview

1564

DSA Metrics (OSI-DS 34 (v3))

1455

Physical Link Security Type of Service

+2474

1431

DUA Metrics (OSI-DS 33 (v2))

1404

A Model for Common Operational Statistics

+1857

1372

Telnet Remote Flow Control Option

1080

1349

Type of Service in the Internet Protocol Suite

+2474, ~1248, 1247, 1195, 1123, 1122, 1060, 0791

1323

TCP Extensions for High Performance

1072, 1185

1273

Measurement Study of Changes in Service-Level Reachability in the Global TCP/IP Internet: Goals, Experimental Design, Implementation, and Policy Considerations

1262

Guidelines for Internet Measurement Activities

1257

Isochronous applications do not require jitter-controlled networks

1254

Gateway Congestion Control Survey

1185

TCP Extension for High-Speed Paths

+1323

1152

Workshop report: Internet research steering group workshop on very-high-speed networks

1144

Compressing TCP/IP headers for low-speed serial links

1128

Measured performance of the Network Time Protocol in the Internet system

1110

Problem with the TCP big window option

1106

TCP big window and NAK options

1080

Telnet remote flow control option

+1372

1079

Telnet terminal speed option

1073

Telnet window size option

1072

TCP extensions for long-delay paths

+1323, 2018

1046

Queuing algorithm to provide type-of-service for IP links

1018

Some comments on SQuID

1016

Something a host could do with source quench: The Source Quench Introduced Delay (SQuID)

0996

Statistics server

0896

Congestion control in IP/TCP internetworks

0889

Internet delay experiments

0813

Window and Acknowledgement Strategy in TCP

0662

Performance improvement in ARPANET file transfers from Multics

0618

Few observations on NCP statistics

0612

Traffic statistics (December 1973)

0601

Traffic statistics (November 1973)

0594

Speedup of Host-IMP interface

0586

Traffic statistics (October 1973)

0579

Traffic statistics (September 1973)

0577

Mail priority

0573

Data and file transfer: Some measurement results

0566

Traffic statistics (August 1973)

0557

Revelations in network host measurements

0556

Traffic statistics (July 1973)

0538

Traffic statistics (June 1973)

0522

Traffic statistics (May 1973)

0509

Traffic statistics (April 1973)

0497

Traffic statistics (March 1973)

0482

Traffic statistics (February 1973)

0455

Traffic statistics (January 1973)

0449

Current flow-control scheme for IMPSYS

~0442

0443

Traffic statistics (December 1972)

0442

Current flow-control scheme for IMPSYS

~+0449

0422

Traffic statistics (November 1972)

0413

Traffic statistics (October 1972)

0410

Removal of the 30-Second Delay When Hosts Come Up

0400

Traffic Statistics (September 1972)

0392

Measurement of host costs for transmitting network data

0391

Traffic statistics (August 1972)

0378

0388

NCP statistics

~0323

0378

Traffic statistics (July 1972)

+0391

0323

Formation of Network Measurement Group (NMG)

~+0388

0210

Improvement of Flow Control

0059

Flow Control - Fixed Versus Demand Allocation

0019

Two protocol suggestions to reduce congestion at swap bound nodes

Use the following key to the Status column when referring to Table A-1:

  • [+xxxx] Rendered obsolete by RFCxxxx

  • [xxxx] Renders RFCxxxx obsolete

  • [~+xxxx] Updated by RFCxxxx

  • [~xxxx] Updates RFCxxxx

  • [=xxxx] In addition to RFCxxxx

Copyright © The Internet Society (1998). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.


Appendix B. Guide to QoS Standards

This appendix provides a listing of present standards and protocols that support and implement quality of service (QoS). You can obtain detailed information on each standard from the following list of organizations from which the standards originated:

  • Internet Engineering Task Force (IETF) http://www.ietf.org/

  • International Telecommunication Union (ITU) http://www.itu.int/home/

  • European Telecommunications Standards Institute (ETSI) http://www.etsi.org/


IETF QoS Standards

The IETF standards concentrate on the optimisation of the transmission:

  • Resource Reservation Protocol (RSVP)

  • Differentiated services (DiffServ)

  • Integrated services (IntServ)

  • Multiprotocol Label Switching (MPLS)


ITU QoS Standards

The ITU standards concentrate on the implementation details relating to a QoS framework.

Table B-1. ITU QoS Standards

ITU No.

Description

E.360.1

Framework for QoS routing and related traffic engineering methods for IP-, ATM-, and TDM-based multiservice networks

E.360.2

QoS routing and related traffic engineering methodsCall routing and connection routing methods

E.360.3

QoS routing and related traffic engineering methodsQoS resource management methods

E.360.4

QoS routing and related traffic engineering methodsRouting table management methods and requirements

E.360.6

QoS routing and related traffic engineering methodsCapacity management methods

E.360.7

QoS routing and related traffic engineering methodsRouting table management methods and requirements

E.361

QoS routing support for interworking of QoS service classes across routing technologies

E.800

Telephone network and ISDN quality of service, network management, and traffic engineering: Terms and definitions related to quality of service and network performance including dependability

G.1010

End-user multimedia QoS categories

H.460.9

Support for online QoS-monitoring reporting within H.323 systems

I.356 Amendment 1

New Appendix VSupport of Y.1541 QoS classes 0 and 2 in ATM-based networks

M.3341

Requirements for QoS/SLA management over the TMN X-Interface for IP-based Services

X.606.1

Information technologyEnhanced Communications Transport Protocol: Specification of QoS management for simplex multicast transport.

Y.1541 Amendment 1

Revised Appendix VI: Applicability of the Y.1221 transfer capabilities and IETF

Y.1541 Amendment 2

New Appendix XIConcatenating QoS values


ETSI QoS Standards

The ETSI standards concentrate on the provision and QoS pertaining to a telecommunication system.

  • ETSI NA (SPAN) ETR 003General aspects of Quality of Service (QoS) and Network Performance (NP)

  • ETSI STQ EG 201 769QoS parameter definitions and measurements; parameters for voice telephony service required under the ONP Voice Telephony Directive 98/10/EC

  • ETSI UG EG 202 009Quality of Telecom Services

    - Part 1: Methodology for identification of parameters relevant to the Users

    - Part 2: User-related parameters on a service-specific basis

    - Part 3: Template for Service Level Agreements (SLA)


Glossary

The term quality of service (QoS) broadly refers to a network's capacity to provide better service to selected network traffic for various technologies, including IP routed networks, Frame Relay, ATM, Ethernet, 802.1 networks, SONET, and so on. The majority of terms in this glossary refer to IP QoS. For further listings, refer to the following website:

http://www.cisco.com/univercd/cc/td/doc/cisintwk/ita/index.htm

1+1

A method of protecting traffic in which a protection channel exists for each working traffic channel. For optical systems, the protection channel fibers can be routed over a path separate from the working fibers. The traffic signal is bridged to both the working and protection transmitters so the protection signal can be selected quickly if the working channel fails.

1:n

A method of protecting traffic in which one protection channel exists for n traffic channels. Only one traffic channel can be switched to the protection channel at any given time.

1G mobile network

First-generation mobile network. Refers to the initial category of mobile wireless networks that use analog technology only. Advanced Mobile Phone Service (AMPS) is an example of a 1G mobile network standard.

100BASE-FX

100-Mbps baseband Fast Ethernet specification using two strands of multimode fiber-optic cable per link. To guarantee proper signal timing, a 100BASE-FX link cannot exceed 1312 feet (400 meters) in length. Based on the IEEE 802.3 standard.

100BASE-T

100-Mbps baseband Fast Ethernet specification using UTP wiring. Like the 10BASE-T technology on which it is based, 100BASE-T sends link pulses over the network segment when no traffic is present. However, these link pulses contain more information than those used in 10BASE-T. Based on the IEEE 802.3 standard.

100BASE-T4

100-Mbps baseband Fast Ethernet specification using four pairs of Categories 3, 4, or 5 UTP wiring. To guarantee the proper signal timing, a 100BASE-T4segment cannot exceed 328 feet (100 meters) in length. Based on the IEEE 802.3 standard.

100BASE-TX

100-Mbps baseband Fast Ethernet specification using two pairs of either UTP or STP wiring. The first pair of wires receives data; the second transmits data. To guarantee the proper signal timing, a 100BASE-TX segment cannot exceed 328 feet (100 meters) in length. Based on the IEEE 802.3 standard.

100BASE-X

100-Mbps baseband Fast Ethernet specification that refers to the 100BASE-FX and 100BASE-TX standards for Fast Ethernet over fiber-optic cabling. Based on the IEEE 802.3 standard.

1000BASE-F

A 1-Gbps IEEE standard for Ethernet LANs.

2B1Q

2 binary 1 quaternary. An encoding scheme that provides a 2-bits per baud, 80-kbaud per second, 160-kbps transfer rate. The most common signaling method on ISDN U interfaces. The 1988 ANSI spec T1.601 defines this protocol in detail.

2G mobile network

Second-generation mobile network. Refers generically to a category of mobile wireless networks and services that implement digital technology. GSM is an example of a 2G mobile network standard.

2G+ mobile network

Second-generation plus mobile network. Refers generically to a category of mobile wireless networks that support higher data rates than 2G mobile networks. GPRS is an example of a 2G+ mobile network standard.

24th channel signaling

See 2G mobile network

3G mobile network

Third-generation mobile network. Refers generically to a category of next-generation mobile networks, such as UMTS and IMT-2000.

4B/5B local fiber

4-byte/5-byte local fiber. Fiber channel physical media used for FDDI and ATM. Supports speeds up to 100 Mbps over multimode fiber.

6BONE

The Internet's experimental IPv6 network.

8B/10B local fiber

8-byte/10-byte local fiber. Fiber channel physical media that supports speeds up to 149.76 Mbps over multimode fiber.

802.x

A set of IEEE standards for the definition of LAN protocols.

A
access router

A router, at a customer site, that connects to the network service provider WAN. Sometimes termed a customer premises equipment (CPE) router.

ACL

access control list. ACLs are used for security inside of programs and operating systems. For example, Cisco uses ACL to control access across routers, and Windows NT uses ACLs for directory and file access.

administrative domain

A collection of network elements under the same administrative control and grouped together for administrative purposes. It is usually managed by a single corporate entity. For QoS enforcement purposes, a network domain refers to any domain that shares a common QoS policy. It may or may not overlap other kinds of domains such as IP or NT domains.

admission control

A policy decision applied initially to QoS requests for controlling the admission of network traffic from outside a given administrative domain. Admission control is closely tied to accounting and relies on source authentication. Contrast with policing, which occurs after a request is accepted and data is flowing.

AF

assured forwarding. A specific DiffServ behavior that divides IP packets into four separate per-hop behavior (PHB) classes. Using these classes, a provider may offer different levels of service for IP packets received from a customer domain. Each AF class is allocated a specified amount of buffer space and bandwidth.

ARP

Address Resolution Protocol. Internet protocol, defined in RFC 826, used to map an IP address to a MAC address. Allows host computers and routers to determine the data link layer address corresponding to the IP address in a packet routed through the LAN. Although the packet is addressed to an IP address, the LAN hardware responds only to data link layer addresses. The host or router with the destination IP address replies with its own data link layer address in an ARP response, which the forwarding host or router will use to construct a data link layer frame. The result is stored in cache memory so subsequent packets addressed to the same destination can be routed without an explicit ARP process.

ARQ

automatic repeat request. A method of checking transmitted data on high-speed data communications systems in which the sender encodes an error-detection field based on the contents of the message. The receiver recalculates the check field and compares it with the received field. If the fields match, a positive acknowledgment (ACK or PAK) is returned to the sender. If the fields do not match, a negative acknowledgment (NAK) is returned to the sender.

AS

autonomous system. A self-connected set of networks that are generally operated within the same administrative domain. See also administrative domain

ATM

Asynchronous Transfer Mode. A data framing and transmission architecture with built-in QoS capabilities, designed to carry voice, video, and data. ATM operates at Layer 2 of the OSI model and is a high-speed, connection-oriented, packet-switching, multiplexing architecture. Bandwidth is divided into fixed-size cells of 53 bytes each, including headers, which are allocated to services on demand. Bandwidth can be dynamically allocated. ATM can offer bandwidth rates of up to multigigabit bandwidth. Although relatively few native ATM applications exist, TCP/IP traffic can be sent over an underlying ATM layer. In principle, ATM could be used over the LAN, MAN, or WAN; in practice, ATM is prevalent in WANs and Internet backbones.

AVVID

See Cisco AVVID

B
backbone

A network linking together multiple domains, either enterprises or service provider.

bandwidth

Transmission capacity of a computer channel or communications line or bus, usually stated in bits per second (bps). Bandwidth indicates the theoretical maximum capacity of a connection, but as the theoretical bandwidth is approached, negative factors such as transmission delay can cause deterioration in quality.

bandwidth manager

A traffic manager deployed at congestion points that limits access to network resources. It often requires locating a proprietary hardware device directly on the network and may be an additional point of failure. It cannot coordinate multiple traffic flows or resolve conflicting QoS requests made by multiple clients, and is therefore not an end-to-end QoS solution.

bandwidth throughput

Average traffic throughput over a given time interval, expressed in bits per second (bps).

best-effort service

The default behavior of TCP/IP networks in the absence of QoS measures. TCP/IP nodes will make their best effort to deliver a transmission, but will drop packets indiscriminately in the event of congestion, managing the bandwidth, or assigning priority to delay sensitive packets. The Internet today is a good example of best-effort service. Best effort is suitable for a wide range of networked applications such as general file transfers or e-mail.

BGP

Border Gateway Protocol. An Internet routing protocol used to pass routing information between different administrative routing domains or autonomous systems. BGP does not pass explicit topology information. BGP is often used between Internet service providers (ISPs).

border router

Generally describes routers on the edge of an autonomous system (AS). Uses BGP to exchange routing information with another administrative routing domain. Can also describe any router that sits on the edge of a routing sub area, such as an Open Shortest Path First (OSPF) area border router.

broadcast storm

An overload condition in a network created by an incorrect packet broadcast onto the network that causes multiple hosts to respond all at once. Typically, the response contains equally incorrect packets, which causes the storm to grow exponentially in severity.

business-critical application

A networked application that is deemed critical to the success of the business organization.

C
CBQ

class-based queuing. A public domain QoS methodology for classifying packets and queuing them according to criteria defined by an administrator to provide differential forwarding behavior for each traffic class.

CBR

constant bit rate. Multimedia streams, audio, and video are examples of CBR applications because they send at a relatively steady data rate with constant bandwidth allocations. A class of service defined by ATM.

CBWFQ

class-based weighted fair queuing. Enables you to define traffic classes that are based on certain match criteria, such as access control lists, input interface names, protocols, and QoS labels.

CDP

Cisco Discovery Protocol. A device-discovery protocol that runs on all Cisco-manufactured equipment. Enables a device to advertise its existence to other devices and receive information about other devices in the network.

CDV

cell delay variation. In an ATM network, the variation in cell delay through the network.

CER

cell error ratio. In an ATM network, ratio of errored cells to transmitted cells. Measures the accuracy of cell transmission.

CFQ

class-based fair queuing. Per-class packet scheduling.

CIR

committed information rate. A minimum access rate the service provider commits to providing the customer.

Cisco AVVID

Cisco Architecture for Voice, Video, and Integrated Data.

Cisco IP phone

Telephone that provides voice communication over an IP network.

Cisco IP Telephony Solutions

A software and hardware product suite offering an IP alternative to traditional private branch exchanges (PBXs).

class

An abstraction that can be determined by different policy criteria such as IP packet header content (such as source or destination addresses, port numbers, or transport protocol), time of day, ingress point, and so forth. The definition of a class can differ at different locations on the network.

classifier

An entity that selects packets based on the content of packet headers according to defined rules.

CLEC

competitive local exchange carrier. A telephone company that competes with other local phone companies within an exchange, or calling area, to provide local telephone service.

CLI

command-line interface. A method of interacting with a device (for example, router or switch) by giving it lines of textual commands (such as a sequence of characters) either from keyboard input or from a script.

client

Node or software program (front-end device) that requests services from a server.

client/server computing

Term used to describe distributed computing (processing) network systems in which transaction responsibilities are divided into two parts: client (front end) and server (back end).

client/server model

The process of workload sharing between the client, the server, and the network. Examples include the name server/name resolver paradigm of the Domain Name System (DNS), as well as file server/file-client relationships, such as Network File System (NFS) and diskless hosts. See also DNS and NFS

CLP

cell loss priority. In ATM cells, a bit that specifies whether a cell can be discarded, if necessary; for example, if network congestion or policing occurs.

CLR

cell loss ratio. In an ATM network, the ratio of lost ATM cells to transmitted cells; measures the percentage of cells lost between two points in the network. Cell loss is typically caused by severe network congestion; it can also result from bit errors in the cell header.

CMR

cell misinsertion rate. In an ATM network, the total number of misinserted cells observed during a specified time interval divided by the time interval duration.

codec

coder-decoder. 1. A device that typically uses pulse code modulation to transform analog signals into a digital bit stream, and digital signals back to analog. 2. In Voice over IP, Voice over Frame Relay, and Voice over ATM, a software algorithm used to compress/decompress speech or audio signals.

codepoint

Codepoint markings are made in a new implementation of the IP version 4 type of service (IPv4 ToS) header called the DiffServ field (6 bits, reserving 2 for congestion notification), and are used to select a per-hop behavior (PHB). This marking takes place on the host, on a boundary, or on an edge device.

congestion avoidance

Congestion avoidance is the action a network takes to head off congestion before it can occur, anticipating circumstances in which flows or aggregated flows might no longer receive designated service levels due to excessive traffic loads at points in the network. An example is the constructive application of a drop policy such as RED to provide implicit feedback to host systems to reduce network traffic during congestion.

congestion control

Mechanisms that control traffic flow so that intermediate network devices and end stations are not overwhelmed.

congestion management

A mechanism at multiplexing points that imposes order when traffic exceeds network capacity for a flow or set of aggregated flows. It determines whether some packets must be discarded, and, if so, it preserves the more important packets. Queuing, scheduling, and traffic shaping are among the most popular techniques.

controlled load

Tightly approximates best-effort service under unloaded conditions, a high-level but nonguaranteed service. In the proposed IETF integrated services model, this level of service is designed for multimedia applications where time delay is not critical but quality of the delivery is important. This service is appropriate for applications such as one-way voice or video, but not for real-time applications.

converged network

A network that combines varied traffic types such as data, voice, and multimedia. Most analysts expect the converged network of the future to be based on Internet protocols.

conversation

The exchange of traffic that takes place between a client and server during the execution of an application transaction. Similar in nature to that of a human conversation, involving requests and responses.

COPS

Common Open Policy Service. A standard defining a simple protocol for provisioning QoS by outsourcing policy-based admission control over requests for network resources.

core router

A router on the network service provider WAN that has no direct connections to any routers at customer sites.

CoS

class of service. A category based on type of user, type of application, or some other criteria that QoS systems can use to provide differentiated classes of service. The characteristics of the CoS may be appropriate for high-throughput traffic, for traffic with a requirement for low latency, or just for best effort.

cRTP

compressed RTP. IP RTP header compression reduces the 40-byte IP+UDP+RTP header to 2 to 4 bytes, thereby reducing the bandwidth required per voice call on point-to-point links. The header is compressed at one end of the link and decompressed at the other end. See also RTP

CTD

cell transfer delay. In an ATM network, measures the peak and mean delay that cells experience while traveling from one point in the network to another.

customer router

A router at a customer site that is not directly connected to the network service provider WAN.

D
DCOM

Distributed Component Object Model. Protocol that enables software components to communicate directly over a network. Developed by Microsoft and previously called Network OLE, DCOM is designed for use across multiple network transports, including Internet protocols such as HTTP.

demarcation point

The point where the service provider's equipment meets a customer's equipment.

differentiated services boundary

The edge of a DiffServ domain, where classifiers and traffic conditioners are likely to be deployed. A DiffServ boundary can be further subdivided into ingress and egress nodes, where the ingress/egress nodes are the downstream/upstream nodes of a boundary link in a given traffic direction. A DiffServ boundary may be co-located with a host, subject to local policy.

differentiated services domain

A contiguous portion of the Internet over which a consistent set of DiffServ policies are administered in a coordinated fashion. A DiffServ domain can represent different administrative domains or autonomous systems, different trust regions, different network technologies (such as cell/frame), hosts and routers, and so forth.

differentiated services field

The IPv4 header ToS octet or the IPv6 Traffic Class octet when interpreted in conformance with the definition given in RFC 2474.

DiffServ

differentiated services. An IETF standard for a small, well-defined set of per-packet building blocks from which a variety of services may be built, thereby providing a framework for delivering QoS in networks.

DNS

Domain Name System. The system used to translate names of network devices into IP addresses.

DSBM

designated subnet bandwidth manager. A device on a managed subnetwork that acts as the subnet bandwidth manager for the subnetwork to which it is attached. This is done through an election process specified in the IETF SBM protocol specification.

E
e2e QoS

end-to-end QoS. The capability of a network to deliver service needed by a specific network application from end to end, with the capability to provide both class of service and reserved bandwidth for different types of network traffic.

edge device

A device, such as a router or a gateway, that is deployed at the border of an administrative domain. Such devices control traffic through one point only.

edge-to-edge QoS

Applies to QoS within a network that connects to other networks rather than hosts or end systems (the typical service provider network, for example), with some level of control over bandwidth, jitter, delay, and loss provided by the network.

EF

expedited forwarding. A per-hop behavior (PHB) in the DiffServ standard used to create a virtual leased-line service.

EPD

early packet discard. A congestion-avoidance mechanism generally found in ATM networks.

EtherChannel

Cisco EtherChannel technology is a trunking technology based on grouping several full-duplex 802.3 Ethernet links to provide fault-tolerant, high-speed links between switches, routers, and servers.

ETSI

European Telecommunications Standards Institute. An independent, nonprofit organization that produces telecommunications standards.

F
FEC

forward error correction. Sending redundant data so that if data loss occurs, data recovery is possible without retransmission.

Filterspec

Part of the Resource Reservation Protocol (RSVP) that is used by senders in a PATH message to identify themselves, and by recipients in RESV messages to select which senders they are interested in receiving messages from.

FIFO

first in, first out. Simple tail-drop FIFO queue used in (best-effort) IP service. The simplest queuing algorithm used in switches and routers because the first packets to arrive in a buffer are the first to leave. FIFO offers no configuration options; therefore, it is sometimes accompanied by other schemes such as random early detection (RED) and weighted fair queuing (WFD).

flow

A set of packets traversing a network element, all of which are covered by the same request for control of QoS.

flowspec

Information about the QoS that is required by the flow and contained in an RSVP message and that tells each router along a data path which resources it should reserve for that particular flow.

G
G.711

An audio-compression standard used for digital telephones on a digital PBX/ISDN. In G.711, encoded voice is already in the correct format for digital voice delivery in the PSTN or through PBXs. G.711 uses a bandwidth of 64 kbps. G.711-compliant devices can communicate with other G.711 devices, but not with G.723 devices. Described in the ITU-T standard in its G-series recommendations.

G.723.1

Describes a compression technique that can be used for compressing speech or audio signal components at a very low bit rate as part of the H.324 family of standards. This codec allows dissimilar communication devices to communicate with each other using a standardized communications protocol. Used for digital telephones on a digital PBX/ISDN, producing digital audio at either 6.4 or 5.3 kbps. Described in the ITU-T standard in its G-series recommendations.

G.729

ITU-T's standard voice algorithm. Describes the encoding/decoding of speech.

gatekeeper

1. Component of an H.323 conferencing system that performs call address resolution, admission control, and subnet bandwidth management. 2. Telecommunications: H.323 entity on a LAN that provides address translation and control access to the LAN for H.323 terminals and gateways.

The gatekeeper can provide other services to the H.323 terminals and gateways, such as bandwidth management and locating gateways. A gatekeeper maintains a registry of devices in the multimedia network. The devices register with the gatekeeper at startup and request admission to a call from the gatekeeper. In the Cisco CallManager, for example, the gatekeeper is a device that supports the H.225 RAS message set used for call admission control (CAC), bandwidth allocation, and dial-pattern resolution. There is one gatekeeper device per Cisco CallManager cluster.

gateway

The point at which a circuit-switched call is encoded and repackaged into IP packets. A gateway is an optional element in an H.323 conference and bridges H.323 conferences to other networks, communications protocols, and multimedia formats.

guaranteed service (hard QoS, quantitative QoS, reserved bandwidth)

A service level that attempts to guarantee a minimal delay for traffic delivery. In the IETF IntServ model, guaranteed service is intended for real-time applications, such as teleconferencing. Guaranteed service is an absolute reservation of network resources, typically bandwidth, which implies reservation of buffer space along with the appropriate queuing disciplines and so on to ensure that specific traffic gets a specific service level. Typically, guaranteed service is associated with a fine level of traffic classification, so that particular flows (or aggregates) have network resources reserved for them so that required guarantees can be met.

GUI

graphical user interface. User environment that uses pictorial as well as textual representations of the input and output of applications and the hierarchical or other data structure in which information is stored.

HI
H.323

A standard approved by the International Telecommunication Union (ITU) that defines how audio-visual conferencing data is transmitted across networks.

IEEE

Institute of Electrical and Electronics Engineers. Professional organization whose activities include the development of communications and network standards.

IEEE 802.1D

802.1p was incorporated into and superceded by the IEEE 802.1D standard.

IEEE 802.1p

An IEEE standard for improving support of time-critical and multicast-intensive applications across bridged LANs. A Layer 2 scheme that lets end stations request priority and network devices enforce it, using a tag in the packet header. Incorporated into and superceded by IEEE 802.1D.

IEEE 802.1Q

An IEEE standard for providing a virtual LAN capability within a campus network. It establishes a standard format for frame tagging (Layer 2 VLAN markings).

IETF

Internet Engineering Task Force. An engineering and protocol standards body that develops and specifies protocols and Internet standards, generally in the network layer and above. See www.ietf.org.

IntServ

integrated services. One of two architectures (the other being DiffServ) defined by the IETF. IntServ uses the Resource Reservation Protocol (RSVP) to explicitly signal the QoS needs of an application's traffic along the devices in the end-to-end path through the network. If every network device along the path can reserve the necessary bandwidth, the originating application can begin transmitting. Request For Comments (RFC) 2205 defines RSVP, and RFC 1633 defines IntServ.

IP Precedence/CBQ

A 3-bit value in the IP packet header meant to designate the relative priority of a packet, applied on a host, access router, or gateway, and then used by core routers. Values range from 0 to 7, but typically 6 and 7 are not used by applications because network control messages use these. For example, a brokerage firm might assign a higher IP Precedence/CBQ value to real-time stock trades than to e-mail to ensure that the trading gets expedited delivery. Same as ToS bits. The DiffServ codepoint has been designed to be backward compatible with IP Precedence.

IPSec

IP Security Protocol. Applicable to both IPv4 and IPv6, IPSec defines encryption, authentication, and key management. An extension header is added to the packet to identify those that are authenticated and encrypted.

IPT

IP telephony. Transport of telephone calls over the Internet, no matter whether traditional telephony devices, multimedia PCs, or dedicated terminals take part in the calls, and no matter whether the calls are entirely or only partially transmitted over the Internet.

IPv4

Internet Protocol version 4. The most widely deployed version of the Internet Protocol, IPv4 provides some basic traffic-classification mechanisms with its IP Precedence/CBQ and ToS header fields. However, network hardware and software traditionally have not been configured to use them.

IPv6

Internet Protocol version 6. An update to the Internet Protocol. Most of the refinements concentrate on basics such as expanding the IP address-numbering scheme to accommodate the growth of the Internet. However, IPv6 does include a class header field that is explicitly intended to designate a class of service (an extension of IPv4's IP Precedence/CBQ field).

ISO

International Organization for Standardization. Responsible for a wide range of standards, including those relevant to networking. ISO developed the popular OSI networking reference model. See also OSI

ISP

Internet service provider. Communications service company that provides Internet access and services to its customers.

ITU

International Telecommunications Union. Under the auspices of the United Nations. Develops radio (ITU-R) and telecommunications (ITU-T) standards. Prior to 1993, the ITU-T Standardization Sector was known as the Consultative Committee for International Telegraphy and Telephony (CCITT).

JKL
jitter

The distortion of a signal as it is propagated through the network, where the signal varies from its original reference timing, and packets do not arrive at its destination in consecutive order, or on a timely basisthey vary in latency. In packet-switched networks, jitter is a distortion of the interpacket arrival times compared to the interpacket times of the original transmission. Also referred to as delay variance. This distortion is particularly damaging to multimedia traffic.

keepalive message

A message sent by one network device to another indicating that the circuit between the two is still active.

LAN

local-area network. Data communications network connecting computers and related equipment, usually over an area not greater than 10 km.

LANE

LAN emulation. A set of services and protocols that provide for the emulation of LANs, using ATM as a backbone to allow connectivity between LAN and ATM attached end stations.

latency

Delay in a transmission path or in a device within a transmission path. In a router, latency is the amount of time between when a data packet is received and when it is retransmitted. Also referred to as propagation delay.

Layer 1

The physical network layer in the OSI model. Applies to basic wiring. An example of a Layer 1 protocol is 10BASE-T.

Layer 2

The data link layer in the OSI model. Examples include Ethernet, Token Ring, and HDLC. Layer 2 is divided into the Media Access Control (MAC) sublayer and the Link Layer Control (LLC) sublayer. Layer 2 is where LAN protocols such as Ethernet, Token Ring, and FDDI are defined.

Layer 3

The network protocol layer in the OSI model. Examples include IP and IPX. Layer 3 is the packet, or network, layer. A routing protocol, such as Routing Information Protocol (RIP), operates at Layer 3. Other Layer 3 protocols provide network signaling and other functions. Layer 3 protocols are implemented in routers and L3 switches.

Layer 4

The transport layer in the OSI model. It provides end-to-end connections across the network. An example of a Layer 4 protocol is TCP. Below the transport layer, each protocol operates in both end stations and intermediate network nodes. Transport layer (and higher) protocols operate only in end stations. Typical Layer 4 services include guaranteed delivery of data across a network and segmentation of large messages into packets small enough to be handled by the lower-layer protocols.

Layer 5

The session layer in the OSI model. It manages communications. For example, it sets up, maintains, and terminates virtual circuits between sending and receiving stations. It also sets up boundaries for the start and end of message and establishes how messages will be sent (for example, half or full duplex, and so on).

Layer 6

The presentation layer in the OSI model. It deals with the format and representation of data that applications use; specifically, it controls the formats of screens and files. Layer 6 defines such things as syntax, control codes, special graphics, and character sets.

Layer 7

The application layer in the OSI model. It serves as the window for users and application processes to access network services. This layer applies to the actual meaning as opposed to the format of the syntax of applications (see Layer 6) and permits communication between users.

LDAP

Lightweight Directory Access Protocol. A standardized TCP/IP protocol for access to a central X.500-based directory that is shared by many different services. LDAP repositories are not capable of issuing dynamic policy authorizations and do not have policy-enforcement capabilities. LDAP can be used as a vehicle for propagating policy information from policy servers to switches. LDAP is likely to play a central role in managing dynamic networks.

LDP

Label Distribution Protocol. A fundamental concept in Multiprotocol Label Switching (MPLS) is that two label-switching routers (LSRs) must agree on the meaning of the labels used to forward traffic between and through them. This common understanding is achieved by using the LDP. LDP is the set of procedures and messages by which LSRs establish label-switched paths (LSPs) through a network by mapping network layer routing information directly to data link layer switched paths.

leaky bucket

A traffic-shaping mechanism in which only a fixed amount of traffic is admitted to the network. The traffic is leaked into the network. Excess traffic is held in a queue until it can either be accommodated or must be discarded. See also token bucket

leased line

A private, dedicated telecommunications line reserved for a single customer, often used to connect sites in a WAN. The bandwidth on a leased line depends on the service; for example, a T1 line provides 1.544 Mbps in North America, and an E1 line provides 2.048 Mbps in Europe and other countries.

LLQ

low-latency queuing. A mixture of priority queuing (PQ) and class-based weighted fair queuing (CBWFQ). It is the recommended queuing method for Voice over IP (VoIP) and IP telephony. The priority queue takes all packets that meet the definition defined by a class map and applies a policy to them.

local exchange

A local telephone company that provides telephone service within an exchange, or calling area. Also known as local carriers or local exchange carriers (LECs).

local loop

Telecommunications connection between a home or business and the local telephone exchange or central office.

M
MAC

Media Access Control. Lower of the two sublayers of the data link layer defined by the IEEE. The MAC sublayer handles access to shared media. See also MAC address

MAC address

Standardized data link layer address that is required for every port or device that connects to a LAN. Other devices in the network use these addresses to locate specific ports in the network and to create and update forwarding tables and data structures.

MAN

metropolitan-area network. The interconnection of networks in a city into a single larger network (which may then also offer efficient connection to a WAN). It is also used to mean the interconnection of several LANs by bridging them with backbone lines. The latter usage is also sometimes referred to as a campus network.

mechanism

A specific algorithm or operation (such as a queuing discipline) that is implemented in a node to realize a set of one or more per-hop behaviors.

media stream

The information content carried on a call. Refers to what is actually transmitted and received over the line, and can be read or written by a media stream application program interface (API).

MIB

Management Information Base. A database of managed objects accessed by network management protocols.

Microsoft NetMeeting

A virtual meeting application from Microsoft.

MPLS

Multiprotocol Label Switching. Architecture for integrating a mechanism for label swapping with Layer 3 routing to accelerate packet forwarding. Forwarding decisions are based on fixed-length labels inserted between the data link and network layer headers to increase forwarding performance and path-selection flexibility. The outcome is to make an IP-routed network more connection oriented in nature, where traffic is routed along a labeled path in the topology.

MPOA

Multiprotocol over ATM. An ATM forum standard for the routing of multiple protocols across an ATM substrate. It enables ATM services to be integrated with existing local-area networks (LANs) that use Ethernet, Token Ring, or TCP/IP protocols. The goal of MPOA is to allow different LANs to send packets to each other via an ATM backbone. Unlike other techniques, such as LAN emulation (LANE), which operates at Level 2 of the OSI reference model, MPOA operates at Level 3. MPOA requires a route server that manages the virtual circuits. The first version of MPOA supports only IP traffic.

MQC

modular QoS CLI. Simplifies the configuration of QoS on Cisco routers and switches by defining common command syntax and the resulting set of QoS behaviors across platforms.

N
NBAR

Network-Based Application Recognition. A classification engine that can recognize network applications that dynamically assign TCP or UDP port numbers. After the application is recognized, the network can invoke specific services for that particular application.

network element

A networking device, such as a router, a switch, or a hub, where resource-allocation decisions have to be made and the decisions have to be enforced. The element is therefore potentially capable of exercising QoS control over data flowing through it.

Network Management Forum

A nonprofit, global consortium of service providers, equipment suppliers, and software developers for improving the management of public networks and services.

NFS

Network File System. A commonly used distributed file system that allows remote file access across a network.

O
OID

object identifier. A unique designator of attributes of managed resources used in SNMP network management. An OID is a unique number that defines an object in the MIB. The numbers that comprise the OID describe a path to the object through a tree hierarchy, which is often called an OID tree or a registration tree. When the SNMP agent wants to access a specific object, it traverses the OID tree in the MIB file to find the object.

OSI

Open Systems Interconnection. In the 1980s, the European-dominated International Organization for Standardization (ISO) began to develop its (OSI) networking suite. OSI has two major components: an abstract model of networking (the basic reference model, or seven-layer model, see Layers 1 through 7) and a set of concrete protocols.

OSS

operations support system. Traditionally, a term used by telecommunications companies to manage, monitor, and control the telecommunications networks. Billing, customer-care systems, directory services, network element, and network management are all operational support systems. Now also applied to enterprise environments to refer to the interaction of multiple management systems to manage the overall delivery of applications.

overprovisioning

A way to address current limitations of best-effort networks by provisioning more bandwidth than expected network peak requirements. Overprovisioning increases the probability, but does not guarantee the quality, of transmission of time-sensitive and bandwidth-intensive applications. Overprovisioning is most costly for the WAN.

P
packet classification

A methodology for organizing packets into a group useful for QoS. Classification may be done over a range of granularities, from groups of aggregated flows to individual flows or even subflows. Typically, classification is done in a way similar to defining access lists (that is, based on some contents of the packet header). In this case, a packet may be classified by information in the L2, L3, or L4 headers (source/destination addresses, port numbers, subarea address, applications, user, as well as various Layer 2 attributes). Classification can also be done based on information within the packet payload. Classifications can be broad for aggregated flows, such as traffic destined for a subnetwork X, or as narrow as a single flow or even subflow.

patch

A small addition to the original software code, written to bypass or correct a problem, and also provided between software releases.

PDP

policy decision point. The point where policy decisions are made. Responsible for handling policy decisions on behalf of policy-enforcement points (PEPs).

peering agreement

A reciprocal agreement that lets Internet service providers (ISPs) share backbone links so traffic can reach destinations beyond the ISP's management domain. Agreements between network service providers to allow traffic to transit each other's networks.

PEP

policy-enforcement point. A port on a network device where the policy decisions are actually enforced.

Performance Monitor

A Microsoft Windows server application that displays server activities in real time.

PHB

per-hop behavior. The forwarding treatment given to a specific class of traffic, based on criteria defined in the DiffServ field. Routers and switches use PHBs to determine priorities for servicing various traffic flows.

PHB group

A set of one or more PHBs that can only be meaningfully specified and implemented simultaneously, due to a common constraint applying to all PHBs in the set, such as a queue-servicing or queue-management policy. A PHB group provides a service building block that allows a set of related forwarding behaviors to be specified together (such as four dropping priorities). A single PHB is a special case of a PHB group.

policing

Packet-by-packet monitoring function at a network border (ingress point) that ensures a host (or peer or aggregate) does not violate its promised traffic characteristics. Policing means limiting the amount of traffic flowing into or out of a particular interface to achieve a specific policy goal. Policing typically refers to actions taken by the network to monitor and control traffic to protect network resources, such as bandwidth, against unintended or malicious behavior. Traffic shaping may be used to achieve policing goals or to do congestion management.

policy

The combination of rules and services where rules define the criteria for resource access and usage in order to manage the bandwidth made available to specified traffic. A policy dictates a number of conditions that must be met before a specified action can be taken.

policy control

The application of rules to determine whether access to a particular resource should be granted.

policy domain

The part of a network subject to policy. Policy applies to one domain, and policy domains do not overlap. Note that a policy domain is not the same as a DNS domain or Windows NT domain.

policy element

Subdivision of policy objects; contains single units of information necessary for the evaluation of policy rules. A single policy element carries user or application identification, whereas another policy element may carry user credentials or credit card information. Examples of policy elements include identity of the requesting user or application, user/app credentials, and so forth. The policy elements themselves are expected to be independent of which QoS signaling protocol is used.

policy object

Contains policy-related info, such as policy elements, and is carried in a request or response related to a resource-allocation decision.

policy resolution

When a policy server or other policy decision point is attempting to apply a policy rule to address a particular situation, it may find a number of possible policies for which all the matching criteria fit. In such cases, additional criteria should be introduced, such as rule ordering or rule specificity, to provide an unambiguous answer. Policy resolution may be user assisted or automated.

policy server

A server that authorizes QoS requests received from routers or gateways and coordinates bandwidth usage on multiple network devices to ensure consistent end-to-end service throughout the data-path. A policy server ensures that packets receive the appropriate QoS, based on a set of policies defined by the network administrator. (Cisco QPM and QDM are examples of policy servers.)

premium service

In DiffServ terms, premium service is a peak-limited, extremely low-delay service, resembling a leased line. At the network edge, where a premium class is first created, it must be either shaped or policed to a rate with no more than a two-packet burst. A policer for premium service is set to drop packets that exceed the configured peak rate. For this service, the peak rate of the premium class aggregate across any boundary must be specified, and the rate must be smaller than the link capacity.

profile

The bandwidth and burst requirements for a given class of service (or application), either at the source site or between a source and destination site.

provisioned service

A service for which network resources are allocated ahead of time, in accordance with a service contract.

PVC

permanent virtual circuit. Exhibits the characteristics of a leased-line connection over a packet-switched, connection-oriented network and remains established.

Q
QoS

quality of service. A collective measure of the level of service delivered to the customer. QoS can be characterized by several basic performance criteria, including availability (low downtime), error performance, response time and throughput, lost calls or transmissions due to network congestion, connection setup time, and speed of fault detection and correction. Service providers may guarantee a particular level of QoS (defined by a service level agreement, SLA) to their subscribers. QoS-enabled hardware and software solutions sort and classify IP packet requests into different traffic classes and allocate the proper resources to direct traffic based on various criteria including application type, user or application ID, source or destination IP address, time of day, and other user-specified variables.

QoS policy

A set of actions a network takes to configure and signal for a particular QoS to be provided to a particular traffic classification.

QoS signaling

The means for transmitting QoS requests and parameters between devices or applications to deliver a QoS requirement across the network. Either in-band signaling (for example, IP Precedence or 802.1p) or out-of-band signaling (RSVP) is used to indicate that a particular QoS is desired for a particular traffic classification. IP Precedence and RSVP are the two most useful signaling mechanisms, because they take advantage of the end-to-end nature of Layer 3 protocol and the growing ubiquity of IP.

QoS strictness

Three broad categories of QoS from end to end or edge to edge, are best effort, differentiated, and guaranteed. The strictness of the QoS describes how tightly the service can be bound by specific bandwidth, delay, jitter, and loss characteristics. For example, the delay, loss, and jitter characteristics can be offered to within tight tolerances on a constant bit rate (CBR) service, whereas they are much harder to bound on a typical Internet IP connection.

QPM

QoS policy manager. Delivers differentiated services across network infrastructures for converged voice, video, and data applications. Network administrators can use QPM as a complete system for centralized QoS monitoring, policy control, and automated reliable policy deployment.

queuing

A method for metering the flow of traffic by placing packets in holding queues and retransmitting them according to a sorting algorithm, typically a simple first-in, first-out (FIFO) formula. Queues of different sizes can be used to assign levels of importance according to class of service (CoS) designations. Queues that overflow, typically discard packets to reduce network congestion.

R
RAPI

RSVP application programming interface. An RSVP-specific API that enables applications to interface explicitly with an RSVP process.

RED

random early detection. A congestion-avoidance algorithm developed in the early 1990s and built on the base-level TCP behavior of automatically slowing transmissions when packet loss is detected. RED tries to anticipate congestion by monitoring a queue on a router. When the specified threshold is reached, it randomly discards packets. This is an implicit signal that the originating applications should slow their transmissions before congestion becomes severe. Unlike CBQ, WFQ, and SFQ, RED does not require flow state in routers.

reservation

Part of a resource that has been dedicated for the use of a particular traffic type for a period of time through the application of policies.

resource

Something of value in a network infrastructure to which rules or policy criteria are first applied before access is granted. Examples of resources include the buffers in a router and bandwidth on an interface.

round-robin queuing

An algorithm that services each queue in a predefined sequence. For example, it might empty 1500 bytes apiece from queue 1 (high priority), queue 2 (medium priority), and queue 3 (low priority), servicing each in turn.

RSPAN

Remote SPAN. A feature of Cisco switches that extends the monitoring capabilities of an existing network analyzer to remotely monitor source ports and destination ports that are distributed across multiple switches, enabling you to monitor the traffic on one switch using a device on another switch. See also SPAN

RSVP

Resource Reservation Protocol. An IETF standard that allows an end device and a network to negotiate specific QoS characteristics. Using RSVP, an application signals a request to reserve resources along a route from source to destination. RSVP-enabled routers then schedule and prioritize packets. A reservation for the required bandwidth is allowed or denied depending on the current network conditions.

RTCP

Real-Time Control Protocol. Monitors the QoS of an IPv6 RTP connection and conveys information about the ongoing session. See also RTP, IPv6, QoS

RTP

Real-Time Transport Protocol. A network protocol used to carry packetized audio and video traffic over an IP network. See also cRTP

S
SAA

Service Assurance Agent. Embedded software within Cisco IOS devices that performs active monitoring.

segment

1. Section of a network that is bounded by bridges, routers, or switches. 2. In a LAN that uses a bus topology, a continuous electrical circuit that is often connected to other such segments with repeaters. 3. Term used in the TCP specification to describe a single transport layer unit of information. The terms datagram, frame, message, and packet are also used to describe logical information groupings at various layers of the OSI reference model and in various technology circles.

server

1. Node or software program that provides services to clients. 2. In network addressing, a concentrator, data switch, or host computer being accessed.

service

A description of the overall treatment of (or a subset of) a customer's traffic across a particular domain, path, or end-to-end network. In DiffServ, service descriptions are covered by administrative policy, and services are constructed by applying traffic conditioning to create behavior aggregates that experience a known per-hop-behavior (PHB) at each node within the DiffServ domain. Multiple services can be supported by a single PHB in concert with a range of traffic conditioners.

service provider

Controls a network infrastructure and may be responsible for the charging and accounting of services.

SFQ

stochastic fair queuing. A hash function used to map flow to one of a set of queues.

signaling

Communications between devices to set up calls and tear them down.

SLA

service level agreement. A contract between a service provider and customer defining provider responsibilities in terms of network levels (for example, throughput, loss rate, delays, and jitter).

SNMP

Simple Network Management Protocol. A UDP-based network management protocol used predominantly in TCP/IP networks. SNMP can be used to monitor, poll, and control network devices.

soft QoS (qualitative QoS)

A quality of service that does not consist of a 100-percent guarantee of one of the performance parameters (bandwidth, latency, jitter, and so forth), but delivers that performance with some percentage probability and, in general, gives better than best-effort service.

SONET

Synchronous Optical Network. North American transport standard for optical networks with speeds from approximately 52 Mbps (OC-1) to 10 Gbps (OC-192).

SPAN

Switched Port Analyzer. Feature of the Cisco switches that extends the monitoring capabilities of existing network analyzers into a switched Ethernet environment. SPAN mirrors the traffic at one switched segment onto a predefined SPAN port. A network analyzer attached to the SPAN port can monitor traffic from any of the other catalyst switched ports. See also RSPAN

standalone router

A router that is considered in isolation from all other routers in the network. The standalone router is seen as the only router on the WAN, so it can be used for test or demonstration purposes.

static policy

A policy put in place as a service is provisioned and then not changed dynamically by changing application conditions.

Subnet Bandwidth Manager

An IETF standard for handling resource reservations on shared and switched IEEE 802-style local-area media.

SVC

switched virtual circuit. Virtual connection set up only for the duration of a single communications session. In contrast, a permanent virtual circuit (PVC) remains available at all times.

T
TCA

traffic-conditioning agreement. An agreement specifying classifier rules and any corresponding traffic profiles and metering, marking, discarding, and/or shaping rules that are to apply to the traffic streams selected by the classifier. A TCA encompasses all the traffic-conditioning rules explicitly specified within an SLA, along with all the rules implicit from the relevant service requirements.

TCP rate control

A technology implemented at network end points that attempts to regulate the introduction of traffic into the network.

throughput

In data transmission, throughput is the amount of data moved from one place to another in a given time period.

token bucket

A traffic-shaping mechanism in which a predetermined amount of tokens in a bucket represent the capacity allowed to each class of traffic. Packets are forwarded until they exhaust their supply of tokens. When the token supply is exhausted, packets may be discarded or delayed until the bucket is replenished. This controls the transmit rate and accommodates bursty traffic. In some systems, a customer's token supply might correspond to a service fee. See also leaky bucket

ToS

type of service. A field within an IP header that can be used by the device originating the packet, or by an intermediate networking device, to signal a request for a specific QoS level. ToS uses 3 bits to tell a router how to prioritize a packet, and 1 bit apiece to signal requirements for delay, throughput, and reliability. ToS is also known as IP precedence bit format and the IP Precedence field.

traffic conditioner

An entity that performs traffic-conditioning functions, and which may contain meters, policers, shapers, and markers. Traffic conditioners are typically deployed in boundary nodes, not in internal nodes of a domain.

traffic conditioning

Control functions that can be applied to a behavior aggregate, application flow, or other operationally useful subset of traffic, such as routing updates. These may include metering, policing, shaping, and packet marking. Traffic conditioning is used to enforce agreements between domains and to condition traffic to receive a differentiated service within a domain by marking packets and by monitoring and altering the temporal characteristics of the aggregate where necessary.

traffic profile

A description of the temporal properties of a traffic stream such as rate and burst size.

traffic shaping

A group of techniques that attempts to regulate or meter the flow of packets through the network. See also leaky bucket, token bucket

traffic stream

An administratively significant set of one or more microflows that traverse a path segment. A traffic stream may consist of the set of active microflows that are selected by a particular classifier.

V
VC

virtual circuit. A logical connection between two network nodes that acts as though it is a direct physical connection, even though it may physically be packet based. The term is used most frequently to describe connections between two hosts in a packet-switching network.

VLAN

virtual local-area network. A networking architecture that allows end systems on topologically disconnected subnetworks to appear to be connected on the same LAN. Similar in functionality to bridging.

VoIP

Voice over IP. A technology for transporting integrated digital voice, video, and data over IP networks.

VPN

virtual private network. A private data network that makes use of the public telecommunication infrastructure, maintaining privacy through the use of tunneling protocol and security procedures. For example, a company could contract with an ISP to set up a VPN to use the Internet to connect two geographically separated sites, rather than set up a dedicated WAN or use a leased line.

W
WDM

wavelength-division multiplexing. A mechanism to allow multiple signals to be encoded into multiple wavelengths.

WFQ

weighted fair queuing. Per-flow packet scheduling in network elements that automatically categorizes traffic flows into high and low priority, based on volume of packets seen by a router or switch. Low-bandwidth traffic has effective priority over high-bandwidth traffic, and high-bandwidth traffic shares the transmission medium proportionally according to assigned weights. Like class-based queuing (CBQ), WFQ is designed to prevent any one traffic type from entirely eclipsing another. By default, WFQ favors lower-volume traffic flows over higher-volume ones (for example, a routine e-mail over a large FTP download).

WLAN

wireless local-area network. WLANs use radio frequency to transmit and receive data over the air.

WRED

weighted random early detection. A congestion-avoidance and QoS mechanism for TCP/IP-based networks.

X
XML

eXtensible Markup Language. A standard maintained by the World Wide Web Consortium (W3C). It defines a syntax that enables you to create markup languages to specify information structures. Information structures define the type of information (such as subscriber name or address), not how the information looks (bold, italic, and so on).

X terminal

Terminal that enables a user to simultaneously access several different applications and resources in a multivendor environment through implementation of X Windows.

Reference Used in Glossary

  • Cisco Systems, Internetworking Terms and Acronyms. http://www.cisco.com/univercd/cc/td/doc/cisintwk/ita/index.htm


Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]


SYMBOL

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

802.11 networks, QoS


A

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

ABR (available bit rate)
Abstract Syntax Notation One (ASN.1)
access control lists (ACLs) 2nd
access. [See also security]
storage-area networks
transactions
accountability, delivery of service
accounting
accounts, transactions
acknowledgments (ACKs) 2nd
ACLs (access control lists) 2nd
active agents
Address Resolution Protocol (ARP)
administration. [See also management]
APM
benefits of
end-to-end systems
isolation
lifecycles
monitoring SLAs
overview of
sharing information
CRM systems
oversight
QPM
SLAs
system
TCO
tools
configuration
DiffServ
FIFO
IntServ
MPLS
networks resource allocation
non-workconserving queuing
provisioning
queuing
RSVP
TE
traffic-handing mechanisms
work-conserving queuing
agents 2nd
capture
embedded
enabling
profiling
SAA 2nd 3rd
agreements (SLAs)
Cisco NetFlow
defining
delivery
management
monitoring
profiling
requirements
alarms
algorithms
queue-servicing
queuing 2nd [See also queuing]
alignment, business/technical methodologies
allocation
ports
QoS
resources 2nd
amalgamation
analysis
business-impact
capacity planning
failure
modeling
predictive performance
queries
root-cause 2nd
SLA compliance
systems
TCO
thresholds
triggers
virtual environments
analyzers
data
network
profiling
any-to-any communication
APIs (application programming interfaces)
APMs (application performance management) 2nd
benefits of
end-to-end systems
isolation
lifecycles
monitoring SLAs
overview of
services
sharing information
application programming interfaces (APIs)
application-aware probes
application-development teams
applications 2nd
analyzers
architecture 2nd
assessment
bandwidth-bounded
baselining
business-critical
capture points
characterizing
classification 2nd
coexisting
data flow
delivery 2nd
deployment
end-to-end models
interactive
latency-bounded 2nd
mission-critical
monitoring 2nd
probes
status
storage solutions
threads
turns 2nd
web-based
workloads
architecture
applications 2nd
delivery requirements
hub-and-spoke
policy management
ARP (Address Resolution Protocol)
ASN.1 (Abstract Syntax Notation One)
assessment
applications
demarcation points
new application impact
assignment
associated applications
demarcation points
QoS
ATM (Asynchronous Transfer Mode)
authentication
authorized transactions
automation
capacity planning
polling
availability
defining
delivery
available bit rate (ABR)
average bytes per frame, viewing


B

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

background
data
traffic 2nd [See also traffic]
backplane utilization
balancing metrics
bandwidth
application consumption of
increasing
priority
profiling
windows
base networks, building
base transactions, defining 2nd [See also transactions]
baselining 2nd
businesses
collection
environments
establishing
infrastructure
performance
technical baselines
BC (committed burst)
Be (excess burst)
benchmarking
benefits of
APM
TCO
Berkeley Standard Distribution (BSD)
BGP (Border Gateway Protocol)
big bang deployments
big-picture view
bits, DE
Border Gateway Protocol (BGP)
bottlenecks, identifying
boundaries, defining transactions
BPDU (bridge protocol data unit)
branches, transactions
bridge protocol data unit (BPDU)
broadcast traffic (etherStatsBroadcastPkts)
BSD (Berkeley Standard Distribution)
BSM (business service management)
building
baseline environments
baselines
business applications
business baselines
business criteria category (SLAs)
business management
business objectives
business requirements
business service management (BSM)
business transactions
optimizing
profiling
business-critical applications
defining
priority
requirements
TCO
technical alignment
business-impact analysis
bytes
transfers
viewing


C

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

calculations
measurements
SPF
TCO
TE
validating
capacity
increasing
planning 2nd 3rd
capturing
capture points
data
traces 2nd
traffic
transactions
CAR (committed access rate)
case studies
QoS
SLAs
transactions
optimizing
profiling
categorizing business transactions
CBR (constant bit rate)
CBWFQ (class-based weighted fair queuing) 2nd
CDP (Cisco Discovery Protocol)
Cell Loss Priority (CLP)
cell switching, ATM
central site (hub)
channel service unit/data service unit (CSU/DSU)
characterizing applications
CIFS (Common Internet File System)
CIR (committed information rate)
Cisco Discovery Protocol (CDP)
Cisco IOS
show commands
traffic-conditioning mechanisms
Cisco NetFlow 2nd
Cisco Networking Service (CNS) 2nd
Cisco Tag Switching implementations
Cisco Works
class-based weighted fair queuing (CBWFQ) 2nd
classification 2nd 3rd
applications 2nd
devices
management systems
clean transactions
cleaning traces
CLEC (competitive local exchange carrier)
CLI (command-line interface) 2nd 3rd
client, network, and server (CNS) 2nd
clients
overloading
processing time
CLP (Cell Loss Priority)
CNS (Cisco Networking Service) 2nd
CNS (client, network, and server) 2nd
coexisting applications
collection
baselining
key date points
methods
metrics
points
collisions, Ethernet
command-line interface (CLI) 2nd
commands
debug
frame-relay ip rtp header-compression
get
ip rtp compression-connections
ip rtp header-compression
show
show processes cpu
snmpwalk
committed access rate (CAR)
committed burst (Bc)
committed information rate (CIR)
Common Internet File System (CIFS)
Common Open Policy Service (COPS)
common sense
communication
deployment
directory systems
SLAs
comparing trace files
competitive local exchange carrier (CLEC)
complex transactions
compliance, SLA analysis of
computing resources, forecasting
Compuware Application Vantage
configuration
baselining
CLI
management
metrics
QoS 2nd
RSPAN
SPAN
storage-area networks
TCO
thresholds
validation
conflict-prevention tools, SLAs
congestion, troubleshooting
congestion-avoidance (WRED) procedures
connections
any-to-any communication
bandwidth-bounded applications
critical switched ports
latency
WLANs
consistency
APM
transaction captures
constant bit rate (CBR)
constraint-based routing
consumption, monitoring
contingency plans, deployment
control consoles, reporting to
conversation
capture points
paths 2nd
conversational maps
COPS (Common Open Policy Service)
core devices, QoS
core networks, CPE measurements
correlation of events
CPE (customer premises equipment) 2nd
CQ (custom queuing) 2nd
credit, SLAs
critical switched port connections
CRM (customer relationship management)
cRTP (Real-Time Transport Protocol header compression)
CSU/DSU (channel service unit/data service unit)
custom queuing (CQ) 2nd
customer premises equipment (CPE) 2nd
customer relationship management (CRM)
customer service, SLAs


D

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

data analyzers
data collection [See also baselining]
data flow, application
data link connection identifier (DLCI)
Data Link Switching (DLSw)
data transfers
data-collection method
database administrator (DBA)
database management system (DBMS)
DBA (database administrator)
DBMS (database management system)
DE (Discard Eligible) bits
debug commands
defining
base transactions 2nd
business-critical applications
demarcation points
location of metric collection points
metrics
routes
SLAs
transactions 2nd
delay
calculations
CNS reporting
distance
introduction of
measurement of
ping
processing
protocols
queuing
SLAs
transmission
deleting leading/trailing data
delivery
applications
business objectives
management
monitoring
collecting metrics
defining metrics
identifying cycles
operating multiple systems
reporting mechanisms
pyramid model
responsibilities
traffic
troubleshooting
verifying
demarcation points, assigning
dense wavelength-division multiplexing (DWDM)
dependency failures
deployment
application-deployment teams
applications
baselining
big bang
contingency plans
domains
infrastructure
NetFlow
pilot
policy management architecture
predictive modeling
profiling
QoS resource allocation
reporting
staged
TCO
teams
verification
VoIP
WLANs
deposit systems
DES (discrete-event simulation) 2nd
design
baselining
CLI
management
metrics
QoS 2nd
RSPAN
SPAN
storage-area networks
TCO
thresholds
validation
WLANs
development of application-development teams
devices
capture points
classification
configuration
core
CPE 2nd
edge
NAS
networks
PEPs
storage-area networks
DHCP (Dynamic Host Configuration Protocol)
diagnostics [See also troubleshooting]
CLI
GUI
proactive troubleshooting
systems
tools
validating changes
differentiated services (DiffServ)
DiffServ
DiffServ codepoint (DSCP)
direct costs, TCO
direct sequence spread spectrum (DSSS)
directory systems communication
disaster recovery. [See also troubleshooting]
planning
tools
Discard Eligible (DE) bits
discovery, protocols
discrete-event simulation (DES) 2nd
distance delay
distributed weighted random early detection (DWRED)
distribution, transactions
division of end-to-end systems
DLCI (data link connection identifier)
DLSw (Data Link Switching)
documentation
domain policies
downstream alarm suppression
downtime
dropping
frames
schemes
traffic
DSCP (DiffServ codepoint)
DSSS (direct sequence spread spectrum)
DWDM (dense wavelength-division multiplexing)
DWRED (distributed weighted random early detection)
dynamic environments, proactive planning
Dynamic Host Configuration Protocol (DHCP)


E

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

edge devices, QoS
edge nodes
efficiency, OSS
EIGRP (Enhanced Interior Gateway Routing Protocol)
element management system (EMS)
embedded agents
EMS (element management system)
emulation, traffic
enabling
debugging
technologies
end-to-end delivery systems
enabling technologies
planning
end-to-end management
end-to-end models
end-to-end networks
end-to-end QoS
end-to-end systems
APM
division of
end-users [See also users]
endpoints, TE
engineering, networks
Enhanced Interior Gateway Routing Protocol (EIGRP)
enterprise APMs [See also APMs]
enterprise networks
enterprise operational support system
environments
lab
production
testing
virtual 2nd
VoIP
equivalence
ERC (error recovery code)
errors [See also troubleshooting]
essential applications. [See also applications]
defining
priority
requirements
TCO
technical alignment
Ethernet collisions
etherStatsTable (polling)
ETSI (European Telecommunications Standards Institute)
evaluation, TCO
events
excess burst (Be)
expenses, TCO
explicit traffic
extensions, TE
external probes


F

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

failures. [See also troubleshooting]
analysis
TCO
fault
fault-conditions
management
processing
solutions [See also troubleshooting]
FC (Fibre Channel)
FCP (Fibre Channel Protocol)
FDDI (Fiber Distributed Data Interface)
FEC (forwarding equivalence class)
FHSS (frequency hopping spread spectrum)
Fiber Distributed Data Interface (FDDI)
Fibre Channel (FC)
Fibre Channel Protocol (FCP)
FIFO (first in, first out)
files
flat
NFS
trace
first-level help desks
flat files
flow
application data
conversational paths
identifying
profiling
SANs
transactions
flow-based WFQ
flow-based WRED
flow-latency queuing (LLQ) 2nd
flowcharts
pilot deployments
SLAs
troubleshooting
validation
focus, key areas of business
forecasting resources
forwarding 2nd
forwarding equivalence class (FEC)
Frame Relay
frame-relay ip rtp header-compression command
frames
average bytes per
dropped/lost
FC
latency-bounded applications
threads
windows
framework systems [See also monitoring]
frequency hopping spread spectrum (FHSS)
frequency of transactions 2nd
functions. [See also commands]
business transactions 2nd
investigating
management systems
verification
future equipment costs


G

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

get command
get process (SNMP)
GET statements
GFR (guaranteed frame rate)
goals
lack of in methodologies
performance
setting
guaranteed frame rate (GFR)
GUI (graphical user interface) 2nd


H

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

handing over problems
hardware
baselining
external probes
HBA (host bus adapter)
help desks
hierarchies, reporting
high utilization
hooks, management of
host bus adapter (HBA)
hub-and-spoke architecture
hybrid simulation


I

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

ICMP (Internet Control Message Protocol) 2nd
idle time, TCO
IETF (Internet Engineering Task Force) 2nd 3rd
IGPs (interior gateway protocols)
ILEC (incumbent local exchange carrier)
impact of business-critical applications
impact of new applications
implementation. [See also configuration]
QoS 2nd
triggers
increasing
capacity
performance [See also performance]
incumbent local exchange carrier (ILEC)
inefficient applications
information technology (IT)
management
organizational charts
infrastructure
baselining
planning
scaling
VoIP
installation
TCO
WLANs
inter-exchange carrier (IXC)
interaction
applications
reduction of
video/voice
interconnection, SANs
interfaces
APIs
CLI 2nd
FDDI
GUI 2nd
IP tunnel
interior gateway protocols (IGPs)
International Organization for Standardization (ISO)
International Telecommunication Union (ITU)
Internet Control Message Protocol (ICMP) 2nd
Internet Engineering Task Force (IETF) 2nd 3rd
Internet service providers (ISPs) 2nd 3rd
Intserv (integrated services)
intuition
IP (Internet Protocol) tunnel interfaces
ip rtp compression-connections command
ip rtp header-compression command
ISO (International Organization for Standardization)
isolation
APM
events
performance [See also performance]
ISPs (Internet service providers) 2nd 3rd
IT (information technology)
management
organizational charts
ITU (International Telecommunication Union)
IXC (inter-exchange carrier)


J

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

jitter
priority
profiling
QoS parameters
SLAs
VoIP


K

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

kbps (kilobits per second)
key areas of focus 2nd
key data collection points
kilobits per second (kbps)
KPIs (key performance indicators) 2nd


L

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

lab environments
label switch router (LSR)
label switched path (LSP)
label switching [See also MPLS]
latency
jitter
priority
profiling
QoS parameters
traces
transactions 2nd
latency-bounded applications 2nd
LDAP (Lightweight Directory Access Protocol)
leading data, deleting
leaky buckets
levels, baselining
LFNs (long fat networks)
lifecycles, APM
Lightweight Directory Access Protocol (LDAP)
limitations of responsibilities/skills
link-state IGPs
links
DLSw
management
RF
LLQ (low-latency queuing) 2nd
load thresholds
loading
locations
capture points
metric collection points
long fat networks (LFNs)
long-term monitoring
long-term triggers
loss
priority
QoS parameters
SLAs
transaction captures
lost frames
LSP (label switched path)
LSR (label switch router)


M

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

MAC (Media Access Control)
maintenance. [See also troubleshooting]
MTTR
TCO
management
accounting
APM 2nd
benefits of
end-to-end systems
isolation
lifecycles
monitoring SLAs
overview of
sharing information
BSM
business
configuration
CRM systems
EMS
end-to-end models 2nd
events
fault
hooks
IT organizational
links
operations
oversight
performance 2nd
policies
proactive application
pyramid model
QPM 2nd 3rd
RF links
security
service
SLAs
SLM
system 2nd
TCO
tools
configuration
DiffServ
FIFO
IntServ
MPLS
network resource allocation
non-work-conserving queuing
provisioning
queuing
RSVP
TE
traffic-handling mechanisms
work-conserving queuing
traffic
users
Management Information Bases (MIBs) 2nd 3rd
Management Interface Bases (MIBs)
managers (SNMP)
mapping
conversational
priorities
services
markers
marking 2nd
matching demarcation points
maximum transmission unit (MTU)
mean time to repair (MTTR)
measurements
availability
bandwidth
capacity planning
CPE/core networks
delay
external probes
latency
methods
NetFlow
Media Access Control (MAC)
merging statistics
messages, debug
metering
methodologies
business and technical
profiling
troubleshooting
methods
collection
data-collection
measurements
of deployment
metrics. [See also requirements]
average bytes per frame
balancing
collecting
collection points
defining
monitoring
profiling
SLAs
Cisco NetFlow
defining
delivery
management
monitoring
profiling
tracking
MIBs (Management Information Bases) 2nd 3rd
MIBs (Management Interface Bases)
milliseconds (ms)
mirroring ports
mission-critical activities
mission-critical applications
models
analytical
business management
delivery responsibilities
DES
end-to-end
predictive modeling
pyramid
scenario
simulation 2nd
virtual
virtual-environment methods
modification
baselining
CLI
management
metrics
QoS 2nd
RSPAN
SLAs
SPAN
storage-area networks
TCO
thresholds
validation
monitoring
agents
APM 2nd
applications 2nd
average bytes per frame
capacity planning
delivery
classifying management
collecting metrics
defining metrics
identifying cycles
multiple systems
pyramid model
reporting mechanisms
demarcation points
end-to-end delivery systems
Ethernet collisions
long-term
metrics 2nd
networks
period of
polling
ports
predictive modeling
real-time troubleshooting
resource-consumption data
SLAs 2nd 3rd
toolsets
MPLS (Multiprotocol Label Switching)
ms (milliseconds)
MTTR (mean time to repair)
MTU (maximum transmission unit)
multicast traffic (etherStatsMulticastPkts)
multimedia applications
multiple systems, operating
Multiprotocol Label Switching (MPLS)


N

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

NAS (network-attached storage)
NBAR (network-based application recognition)
NE (network element)
near-real-time network monitoring capabilities
NetFlow 2nd 3rd
network element (NE)
Network File System (NFS) 2nd
network interface cards (NICs) 2nd
network management systems (NMS) 2nd
Network Node Manager (NNM)
network operations center (NOC)
network service providers (NSPs)
network-attached storage (NAS)
network-based application recognition (NBAR)
networkability
networks
analyzers 2nd
BSM
CNS delay reporting
devices
engineering
enterprise
forecasting
latency
monitoring
operations
pipes
planning
resource allocation
security
SLAs
Cisco NetFlow
defining
delivery
management
monitoring
profiling
requirements
SP
statistics
TCO
transmission paths
unified views of
VPNs
WLANs
NFS (Network File System) 2nd
NICs (network interface cards) 2nd
NMS (network management systems) 2nd
NNM (Network Mode Manager)
NOC (network operations center)
nodes
non-workconserving queuing mechanisms
notification, single-point-of-failure
NSPs (network service providers)
number of metrics, selecting
numbers, ports


O

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

object identifier (OID) 2nd
OBJECT-TYPE constructs
objectives, loading
objects, polling
OID (object identifier) 2nd
Open Shortest Path First (OSPF)
operating costs, TCO
operating multiple systems
operational objectives
operational support system (OSS)
operational systems
operations management [See also management]
OPNET
ACE
IT Guru
optimization
architecture
delivery
end-to-end models
latency-bounded applications
SAN traffic delivery
servers
TE
transactions
windows
organizational charts, IT
OSPF (Open Shortest Path First)
OSS (operational support system)
overloading clients/servers
oversight, management


P

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

packet-by-packet traffic
packets
BPDU
flow
latency
loss
SLAs
transaction captures
marking
webtraffic
parallel fault processing
passive agents
passive probes
paths
capture points
conversation
conversational flows
LSP
TE
transmission
patterns, traffic
PDP (policy decision point)
PDUs (protocol data units)
peer support
penalties, SLAs
PEPs (policy enforcement points)
performance [See also optimization]
APM 2nd
benefits of
end-to-end systems
isolation
lifecycles
meeting service needs
monitoring SLAs
overview of
sharing information
baselining
demarcation points
dynamic environments
end-to-end models
goals
increasing
KPI
management 2nd
polling
SLAs
Cisco NetFlow
defining
delivery
management
monitoring
profiling
requirements
troubleshooting 2nd 3rd 4th
permanent virtual circuit (PVC)
phases, capacity planning
PHB (per-hop behavior)
physical connectivity, transactions
pilot deployments
ping
delay
response times
planning
capacity 2nd 3rd
deployment
disaster recovery plans
dynamic environments
end-to-end delivery
infrastructure
networks
performance
pilot deployments
proactive troubleshooting
enabling technologies
validating changes
scenario modeling
SLMs
systems
policies
architecture management
domains
QoS 2nd
suppression
policing
policy decision point (PDP)
policy enforcement points (PEPs)
polling 2nd
data
monitor
OIDs
performance
records
SNMP
threshold
portfolio transactions
ports
allocation
critical switched connections
mirroring
monitoring
POST statements
PQ (priority queuing) 2nd 3rd
PQCBWFQ (priority queue class-based weighted fair queuing)
predeployment testing [See also deployment]
predictability, SLAs
predicted access points, transactions
predictive modeling
predictive performance analysis
priorities, mapping
prioritization
business-critical applications
CLP
priority queue class-based weighted fair queuing (PQCBWFQ)
priority queuing (PQ) 2nd 3rd
priority schemes
proactive fault processing
proactive planning
capacity planning
dynamic environments
end-to-end delivery
performance
scenario modeling
SLMs
troubleshooting
enabling technologies
validating changes
probes
application-aware
capture
external
passive
profiling
software
processes
capture
OSS
timing of transactions
processing
delay
faults
time
production environments
productivity, TCO
profiling 2nd
agents
analyzers
applications 2nd 3rd
bandwidth
conversation
flow
jitter
latency
methodologies
metrics
probes
SAA
SLAs
transactions
profits, TCO
projecting future resource needs
propagation
protocol data units (PDUs)
protocols
analyzers
baselining
delay
discovery
effect of
QoS
routing
SNMP
TCP
provisioning network devices
PSTN (Public Switched Telephone Network)
purchasing, TCO
PVC (permanent virtual circuit)
pyramid model, monitoring delivery


Q

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

QoS (quality of service) 2nd
802.11 networks
allocation
assigning
configuration
core devices
devices
DiffServ
edge devices
end-to-end
FIFO
implementation
IntServ
network resource allocation
policies
provisioning
queuing
resource allocation
RFCs
RSVP
services
standards
ETSI
IETF
ITU
traffic-handling mechanisms
troubleshooting tools
application monitoring
network monitoring
VoIPs
WAN
QPM (QoS Policy Manager) 2nd 3rd
quality of service. [See QoS]
quality, APM
quantity of metrics, selecting
queries
amount of data transferred
analysis
repeating
queuing 2nd 3rd
algorithms
business transactions
delay
FIFO
simulators
types of


R

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

radio frequency (RF) 2nd
RAID (redundant array of inexpensive disks)
random early detection (RED) 2nd
RCA (root-cause analysis) 2nd
re-marking traffic
Real-Time Transport Protocol header (cRTP)
real-time troubleshooting
reallocation
records
capture time
polling
transaction time
UDP
recovery
disaster
ERC
RED (random early detection) 2nd
reduction
of interactions
of window sizes
redundancy
redundant array of inexpensive disks (RAID)
relocation of servers
remote monitoring (RMON)
Remote Switched Port Analyzer (RSPAN)
repairability of MTTR
repeating queries
reporting
CNS delay
deployment
hierarchies
profiling
to control consoles
Request For Comment (RFC)
requests, VC
requirements
business
business-critical applications
delivery
end-to-end delivery systems
QoS
SLAs
Resource Reservation Protocol (RSVP)
resources
allocation
forecasting
monitoring
QoS allocation
requirements
response times 2nd
APM
business-critical application requirements
SLAs
defining
delivery
responsibilities
delivery models
limitations of
results, profiling
retransmissions, TCP
revenue
key areas of business focus
streams
TCO
reviewing
SLAs 2nd
traces
RF (radio frequency) 2nd
RFC (Request For Comment)
RMON (remote monitoring)
rollouts 2nd
root-cause analysis (RCA) 2nd
round-trip time (RTT)
routers, LSR
routes, defining
routing
constraint-based
ns
protocols
RSVP (Resource Reservation Protocol) 2nd
RTT (round-trip time)
rules
capture
SLAs


S

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

SAA (Service Assurance Agent) 2nd 3rd 4th
SAA/RTR (Service Assurance Agent/Response Time Reporter)
SACK (selective acknowledgement)
SANs (storage-area networks)
scaling
infrastructure
TCP windows
scenario modeling
schemes
dropping
queuing
security
management
networks
selection of deployments
selective acknowledgment (SACK)
serialization
server message block (SMB)
servers
CNS delay reporting
conversational maps
optimization
overloading
processing time
relocating
Service Assurance Agent (SAA) 2nd 3rd 4th
Service Assurance Agent/Response Time Reporter (SAA/RTR)
service level agreements. [See SLAs]
service level management 2nd 3rd
service provider (SP) networks
services
APM
BSM
customer
IntServ
management
mapping
meeting needs of
NetFlow
QoS
troubleshooting
validation
sessions, RSPAN
setsockopt
shaping traffic 2nd
sharing information, APM
shortest path first (SPF)
show commands
show processes cpu command
signaled mechanisms
Simple Network Management Protocol (SNMP)
simple transactions
simulation
DES
hybrid
models 2nd
simulators, calculations
single-point-of-failure notification
single-transaction monetary functions
site surveys, WLANs 2nd
sizes, windows 2nd
skills, limitations of
SLAs (service level agreements) 2nd 3rd
Cisco NetFlow
compliance analysis
defining 2nd
delivery
management
monitoring
penalties
profiling
requirements
reviewing
SLM (service level management) 2nd 3rd
slow delivery, TCO
SMB (server message block)
Sniffer
SNMP (Simple Network Management Protocol)
agents
data-collection methods
get process
MIB
polling
snmpwalk tool
soft costs
software
baselining
probes
SP (service provider) networks
SPAN (Switched Port Analyzer)
specialized support
SPF (shortest path first)
staged deployments
standards, QoS
statements
GET
inquiries
POST
statistics
error
merging
traffic
types of
status, applications
storage-area networks (SANs)
strategic connection points, demarcation points
strategies, APM
streaming video/voice
streams, revenue
stress, verifying functions under
sub-branches, transactions
support
demarcation points
future services
peer
specialized
TCO
suppression of events
Switched Port Analyzer (SPAN)
switched port connections
switches, LSR
switching, DLSW
synthetic transactions
syslog protocol
systems
administration
BSM
DBMS
design
EMS
management
multiple
NFS
NMS
polling


T

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

tail drop 2nd 3rd
targets, transactions
tasks 2nd
TCO (total cost of ownership)
TCP (Transmission Control Protocol)
ACKs
retransmissions
SAN
turns 2nd
windows
scaling
sizing
TDM (time-division multiplexing)
TE (traffic engineering)
teams
application-deployment
application-development
technical alignments
technical baselines
technical criteria category (SLAs)
technical requirements
technologies, enabling
telecommunication costs
Telnet 2nd
testing. [See also troubleshooting; verification]
environments
predeployment
threads, applications
thresholds
analyzing
capacity
load
metrics
polling
time
processing
RTT
transactions
time-division multiplexing (TDM)
time-sensitive conditions, thresholds
timing
token buckets
tools
collection
diagnostic
CLI
GUI
disaster recovery
management
configuration
DiffServ
FIFO
IntServ
MPLS
network resource allocation
non-work-conserving queuing
provisioning
queuing
RSVP
TE
traffic-handling mechanisms
work-conserving queuing
pyramid model
snmpwalk
troubleshooting
application monitoring
network monitoring
toolsets, monitoring
top-down (pushed) mechanisms
topologies, building
ToS (Type of Service)
total cost of ownership (TCO)
total traffic (etherStatsOctets)
traces
capturing 2nd
files
reviewing
tracking
metrics
SLAs
traffic
capturing
Cisco IO traffic-conditioning mechanisms
defining
delivery
dropping
emulation
end-to-end models
jitter
network resource allocation
packet-by-packet
patterns
QoS
re-marking
real-time troubleshooting
RSPAN
shaping 2nd
simulation models
simulators
statistics
voice (WLANs)
webtraffic packets
traffic engineering (TE)
traffic-conditioning implementations
traffic-handling mechanisms
traffic-shaping mechanisms
trailing data, deleting
trances, cleaning
transactions 2nd
base
capturing
defining
flow
optimizing
profiling
synthetic
targets
timing
transfers
amount of data
data
transmissions
delay
paths
traps
triggers
implementation
transactions
troubleshooting
delivery
dependency failures
diagnostic tools
CLI
GUI
disaster recovery
end-to-end models
fault-conditions
flowcharts
IT organizational
jitter
methodologies
MTTR
performance 2nd
proactive
real-time
services
tools
transactions
triggers
WLANs
tuning [See also optimization]
tunneling, IP interfaces
turns
application
TCP
Type of Service (ToS)
types
of queuing
of statistics


U

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

UBR (unspecified bit rate)
UDP (User Datagram Protocol)
unified views of networks
UNIX, snmpwalk tool
unspecified bit rate (UBR)
updates, SLAs
upgrades
User Datagram Protocol (UDP)
users
management
TCO
transactions
utilization


V

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

validation
configuration
services
values, business
variable bit rate (non-real time)
variable bit rate (real time) (VBR)
VC (virtual circuit) requests
vendor specifications
verification
delivery
deployment
performance increase
video
interactive
streaming
videoconferencing
viewing
average bytes per frame
profiling results
virtual circuit (VC) requests
virtual environments
creating
scenario modeling
troubleshooting
validating
virtual models, creating
virtual private networks (VPNs) 2nd
virtual-environment modeling methods
VLANs (virtual LANs)
key data collection points
RSPAN
voice
interactive
streaming
WLANs
VoIP (Voice over IP) 2nd 3rd
VPNs (virtual private networks) 2nd


W

Team LiB
Previous Section

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

WAN (wide-area network)
jitter
QoS
SP clouds
web-based applications
webtraffic packets
weighted fair queuing (WFQ)
weighted random early detection (WRED) 2nd 3rd 4th
WFQ (weighted fair queuing)
wide-area network. [See WAN]
windowing, TCP
windows
sizing 2nd
TCP
wireless LANs
WLANs (wireless LANs)
work-conserving queuing
workloads, applications
WRED (weighted random early detection) 2nd 3rd 4th

Team LiB
Previous Section