ISP EssentialsISP Essentials —— Best PracticeBest Practice
Cisco IOS TechniquesCisco IOS Techniques to Scale the Internetto Scale the Internet
Session XXXX
Agenda for the DayAgenda for the DayAgenda for the Day
• General Features
• ISP Security
• Operations Essentials
Operations EssentialsOperations EssentialsOperations Essentials
Things an ISP’s NOC Should Consider 
 
 
 
No amount of magic knobs will save a
sloppily designed network
Ferguson’s Law of Engineering
Ferguson’s Law ofFerguson’s Law of EngineeringEngineering
Holds true
for sloppy
Operational ConsiderationsOperational ConsiderationsOperational Considerations
Why design the world’s best network when you have not thought about what operational good practices should be
implemented?
Maintenance WindowsMaintenance WindowsMaintenance Windows
ü Never work on the live network, no matter how trivial the modification may seem
ü Establish published maintenance window which your customers are aware of 
e.g. Tuesday 4-7am, Thursday 4-7am
ü Never do maintenance on a Friday
unless you want to work all weekend cleaning up
ü Never do maintenance on a Monday
unless you want to work all weekend preparing
ü Plan what you are going to do in your maintenance window.
 
Operational Considerations Support
• Differentiate between Customer Support Center (CSC) and the Network Operations Center (NOC)
ü CSC fixes customer problems – Customers all the CSC
ü NOC deals with and fixes backbone and Internet related problems – NOCs call NOCs
• Network Engineering Team is last resort
ü They design the next generation network, improve the routing design, implement new services, etc
 
 
Traffic AnalysisTraffic AnalysisTraffic Analysis
 Seek F irs t to Unders tand  Seek F irs t to Unders tand 
10Presentation_ID  © 1999, Cisco Systems, Inc. www.cisco.com
 
Traffic AnalysisTraffic AnalysisTraffic Analysis
Available Bandwidth
Customer  Demand
Underutilized Oversubscribed
Traffic AnalysisTraffic AnalysisTraffic Analysis
• Traffic pattern analysis tools to generate reports periodically
• Generate weekly reports on utilization. Watch the core QOS baselines.
• Tools to evaluate customer-usage per month/week, etc.
 
Traffic AnalysisTraffic AnalysisTraffic Analysis
• Such analysis helps in
ü Easing bottle-necks before they are severe
ü Understanding customer behavor 
 
Traffic Analysis – No ExcusesTraffic AnalysisTraffic Analysis – – No ExcusesNo Excuses
• The tools to create a simple network manage system that will give an ISP the basics comprise of the following:
ü PC with LINUX (free UNIX)
ü CMU SNMP (free SNMP)
ü PERL5 (free UNIX script language)
ü GNU Plot (free graphic plot tool)
ü Printer 
1300_05_2000_c2 © 2000, Cisco Systems, Inc.
• Example of what can be done with another Shareware tool - MRTG ….
• This may seem like common sense for some, but you will be surprised at how many ISPs do not do this very basic step of traffic analysis.
Fundamental Tools are Cheap!Fundamental Tools are Cheap!Fundamental Tools are Cheap!
 
• Baseline Quality Levels are critical for any ISP Server.
ü  Average Utilization and Packet Loss need to be monitored on the entire network.
ü QoS Threshold need to be set and acted on to maintain any sort of foundation to build advanced IP services. This is ISP 101 which most new ISPs forget!
ü  All you need is SNMP! It’s not rocket science.
Threshold
Upgrade!
 
 
 
NetworkNetwork  Manag ing Chaos Manag ing Chaos
18Presentation_ID  © 1999, Cisco Systems, Inc. www.cisco.com
 
• Prepare and Train
üMake sure has the in-depth knowledge AND demonstrates troubleshooting skills.
üJust because someone has a CC$%^ Certification, does not mean they can troubleshoot.
üDocument the most common troubleshooting techniques.
üMake sure the tools are installed, ready, and tested.
Troubleshooting the NetworkTroubleshooting the NetworkTroubleshooting the Network
 
Troubleshooting the NetworkTroubleshooting the NetworkTroubleshooting the Network
Troubleshooting Principles are univers al tools to  solve technical problems.
ü MIT
ü RTFM
ü Test the Hypotheses
ü Follow through and document
ü Crisis Mode: Do Not React! Take a step back and think first ….
ü Consultation!
ü Inclus ion/Consultation Points
Troubleshooting the NetworkTroubleshooting the NetworkTroubleshooting the Network
• Trouble Ticket System is Critical!
ü Make sure you have a way capture, track, and manage problem.
ü Make sure the system works with your NOC’s culture. If not, it will not get used and your back to square one.
• Troubleshooting Access:
ü Pro-actively use scripts to run auto-dial from NOC and check modem stats. Look for the dial problems before customers call.
ü Line-status, modems, CPE, address verification, etc.
 
Troubleshooting the NetworkTroubleshooting the NetworkTroubleshooting the Network
• Troubleshooting the Backbone:
ü Use common sense & sanity checks
ü Verify route propagation and convergence
ü Traceroute, to various parts of the network and verify correct operation
ü Dynamic re-routing through weights, if required
ü Computing traffic patterns based on shortest-path algorithms and sanity check them
ü Periodically verify round-trip times within the network
 
 
StaffingStaffingStaffing
One of the hardes t challenges to running a IS P
One of the hardes t challenges to running a IS P
23Presentation_ID  © 1999, Cisco Systems, Inc. www.cisco.com
 
Staffing IssuesStaffing IssuesStaffing Issues
• People —not bandwidth, content, or applications—are THE most critical factor 
• Raise skills
• Provide Tools
Staffing IssuesStaffing IssuesStaffing Issues
• Any self-respecting ISP needs a UNIX hacker 
• Finding competent staff is the first problem
• Keeping them with you, once you find them is the next problem
• Average turn-over is about two years…
• Training is a key issue, most often neglected
• Keep a few of the staff “in the wings”
 
Staffing HintsStaffing HintsStaffing Hints
• Force Everyone in the NOC to use Unix on their Desktop.
ü Unix and specifically PERL skills are extremely important to running an ISP
• Keep Staff focused so they get really good at one area before moving to the next.
 
Staffing HintsStaffing HintsStaffing Hints
• Find ways to get operations folks from other Networks (like other ISPs) together.
ü One of the big values of “OG” meetings are the hallway and barroom exchange and war stories.
ü Could be as simple as meeting at a local hang out (the early days in Singapore it was 88 Boat Quay).
• Get Staff to formerly teach the next generation.
ü Classes, E-mails, and Apprenticeships.
 
 
What Every NOCWhat Every NOC Should HaveShould Have
An incomplete list …..An incomplete list …..
28ISP/IXP Workshops © 1999, Cisco Systems, Inc. www.cisco.com
 
1300_05_2000_c2 © 2000, Cisco Systems, Inc.
What Every NOC Should HaveWhat Every NOC Should HaveWhat Every NOC Should Have
• Anyone who has worked or run a NOC has their own list of what should be in a NOC.
ü Make your own wish list.
ü Talk to colleagues and get their list.
ü Then try to make it happen.
 
1300_05_2000_c2 © 2000, Cisco Systems, Inc.
What Every NOC Should HaveWhat Every NOC Should HaveWhat Every NOC Should Have
• #1 item on the list – ATTITUDE!
ü The Team has to have the attitude to make things happen on behalf of the network.
• Next? Do a key word search on “NOC” on the NANOG Archives. Read it from your colleagues.
http://www.merit.edu/mail.archives/nanog/
 
1300_05_2000_c2 © 2000, Cisco Systems, Inc.
What Every NOC Should HaveWhat Every NOC Should HaveWhat Every NOC Should Have
• Key word search on the Net for “Network Operations Center” and NOC.
Verio’s NOC - http://www.quest-dynamics.com/html/netmap.html
NOC CommunicationsNOC CommunicationsNOC Communications
ü Customers
ü Other ISPs
• E-mail and Web Pages are the most common forms of communication.
• Pagers and Handphones are secondary communication tools.
 
NOC CommunicationsNOC CommunicationsNOC Communications
• Q. Does noc@someisp.net work?
ü Contact information.
ü Network policies (i.e. RFC 1998)
ü Security policies and contact information.
 
 
Out of Band Management Out of BandOut of Band ManagementManagement
34ISP/IXP Workshops © 1999, Cisco Systems, Inc. www.cisco.com
 
Out of Band ManagementOut of Band ManagementOut of Band Management
• Not optional!
• Ensures quality of service to customers
ü minimises downtime
 
Out of Band ManagementOut of Band ManagementOut of Band Management
• OoB Example - Access server:
ü modem attached to allow NOC dial in
ü console ports of all network equipment connected to serial ports
ü LAN and/or WAN link connects to network core, or via separate management link to NOC
• Full remote control access under all circumstances
 
Out of Band NetworkOut of Band NetworkOut of Band Network
Ethernet
Modem - access
band dialin
Out of Band ManagementOut of Band ManagementOut of Band Management
• OoB Example - Statistics gathering:
ü Management data is congestion/failure sensitive
ü Ensures management data integrity in case of failure
• Full remote information under all circumstances
 
 
 
Test LaboratoryTest LaboratoryTest Laboratory
ü operated like a typical PoP
• Used to trial new services or new software under realistic conditions
 
Test LaboratoryTest LaboratoryTest Laboratory
• Some ISPs dedicate equipment to the lab
 
Test LaboratoryTest LaboratoryTest Laboratory
• Can’t afford a test lab?
ü Set aside one spare router and server to trial new services
ü Never ever try out new hardware, software or services on the live network
• Every major ISP in the US and Europe has a test lab
ü It’s a serious consideration
 
 
43ISP/IXP Workshops © 1999, Cisco Systems, Inc. www.cisco.com
 
More InformationMore InformationMore Information
 
 
3302
1300_05_2000_c2
IOS EssentialsIOS Essentials —— Best Practice Cisco IOSBest Practice Cisco IOS Techniques to Scale theTechniques to Scale the
InternetInternet Session 3302Session 3302
 
3302
1300_05_2000_c2
Please Complete YourPlease Complete Your Evaluation FormEvaluation Form
Session 3302Session 3302