COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Steven Cherny (admission pro hac vice pending)[email protected] KIRKLAND & ELLIS LLP 601 Lexington Avenue New York, New York 10022 Telephone: (212) 446-4800 Facsimile: (212) 446-4900 Adam R. Alper (SBN 196834) [email protected] KIRKLAND & ELLIS LLP 555 California Street San Francisco, California 94104 Telephone: (415) 439-1400 Facsimile: (415) 439-1500 Michael W. De Vries (SBN 211001) [email protected] KIRKLAND & ELLIS LLP 333 South Hope Street Los Angeles, California 90071 Telephone: (213) 680-8400 Facsimile: (213) 680-8500 Attorneys for Plaintiff Cisco Systems, Inc.
UNITED STATES DISTRICT COURT
NORTHERN DISTRICT OF CALIFORNIA
CISCO SYSTEMS, INC.,
Plaintiff, v.
ARISTA NETWORKS, INC.,
Defendant.
)))))))))) )
CASE NO. 3:14-cv-5343 COMPLAINT FOR PATENT INFRINGEMENT DEMAND FOR JURY TRIAL
Case3:14-cv-05343 Document1 Filed12/05/14 Page1 of 22
2
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
COMPLAINT FOR PATENT INFRINGEMENT
Plaintiff Cisco Systems, Inc. (“Cisco”), for its complaint against Defendant Arista Networks, Inc.
(“Arista”), hereby demands a jury trial and alleges as follows:
INTRODUCTION
1. Cisco is an information technology (IT) company that was founded in 1984. Cisco is the
worldwide leader in developing and implementing the networking technologies that enable global
interconnectivity and the Internet of Everything. Cisco employs thousands of networking engineers at
its headquarters in San Jose, California, and elsewhere, and invests billions of dollars annually in
research and development focused on creating the future of networking technologies.
2. Decades after Cisco’s founding, Arista was founded by former Cisco employees, many of
whom are named inventors on Cisco’s networking patents. Among others, Arista’s 1) founders, 2)
President and CEO, 3) Chief Development Officer, 4) Chief Technology Officer, 5) Senior Vice
President for Customer Engineering, 6) Vice President of Business Alliances, 7) former Vice President
for Global Operations and Marketing, 8) Vice President of Systems Engineering and Technology
Marketing, 9) Vice President of Hardware Engineering, 10) Vice President of Software Engineering, and
11) Vice President of Manufacturing and Platform Engineering all were employed by Cisco prior to
joining Arista. Moreover, four out of the seven members of Arista’s Board of Directors were previously
employed by Cisco. Arista’s goal is to sell networking products. Rather than building its products and
services based on new technologies developed by Arista, however, and providing legitimate competition
to Cisco, Arista took a shortcut by using innovative networking technologies designed, developed, and
patented by Cisco.
3. Notably, Arista was founded by former Cisco employees who were intimately and
directly familiar with Cisco’s patented networking technologies, including those protected by patents
asserted in this action. Two of Arista’s founders, Andreas Bechtolsheim and David Cheriton, developed
patented technologies while at Cisco. While each has had a long career in the networking and
computing fields, they are each named inventors on a number of the Cisco patents asserted in this case.
Messrs. Bechtolsheim and Cheriton are aware of Cisco patents on which they were named inventors and
that they developed while employed by Cisco. Arista, despite knowing that Cisco’s networking
Case3:14-cv-05343 Document1 Filed12/05/14 Page2 of 22
3
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
technologies are protected by Cisco’s patents, blatantly incorporated those technologies into Arista’s
products.
4. Arista has acknowledged the substantial investment in time and employment that would
have been required to legitimately compete with Cisco. Arista’s President and Chief Executive Officer,
former Cisco employee Jayshree Ullal, has stated:
“Since I helped build the enterprise [at Cisco], I would never compete with Cisco directly in the enterprise in a conventional way. It makes no sense. It would take me 15 years and 15,000 engineers, and that’s not a recipe for success.” (emphasis added)
By simply incorporating numerous patented technologies developed by Cisco into Arista’s
products, covering a variety of critical features, Arista avoided hiring the thousands of engineers
and making the substantial investments that would otherwise have been needed to legitimately
develop its own technologies. Arista took an unfair shortcut to compete with Cisco using
Cisco’s own technologies, while avoiding the investments in employees, money, and time that
would have been needed to develop products based on new technologies. Indeed, Cisco is not
the only party to find itself aggrieved regarding Arista’s alleged misappropriation of intellectual
property. Arista co-founder David Cheriton has himself alleged that Arista misappropriated his
own intellectual property in a complaint filed against Arista by his company, Optumsoft.
5. Arista’s actions have caused harm to Cisco, as alleged below, by incorporating Cisco’s
patented technologies into Arista’s products. The patents asserted in this case were invented by Cisco
personnel, are proprietary, and are implemented by Cisco in its innovative products in order to
successfully compete in the marketplace. Arista’s actions also significantly harm innovation. If Arista’s
use of Cisco technologies allows it to avoid what is needed to develop new technologies, other
companies will be encouraged to simply use others’ proprietary technologies rather than to hire
engineers, invest in innovation, and develop new technologies. Cisco therefore seeks injunctive relief to
stop Arista’s widespread and improper infringement of Cisco’s lawful patent rights.
6. Cisco welcomes legitimate competition in the marketplace. Its executives have written
and spoken in support of employee mobility, and Cisco believes strongly and has stated that allowing
Case3:14-cv-05343 Document1 Filed12/05/14 Page3 of 22
4
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
people to move freely between companies fosters innovation.1 But Arista has unlawfully and
intentionally used technologies developed by Cisco’s personnel, including without limitation
technologies that Arista’s own founders had developed while at Cisco, where Cisco invested the
necessary research and development, funding, personnel, and engineering hours to support these
innovations. Arista’s intellectual property infringement stifles innovation and cannot be condoned.
NATURE OF THE ACTION
7. This is a civil action for patent infringement under the Patent Laws of the United States,
35 U.S.C. §§ 1 et seq., and for such other relief as the Court deems just and proper.
THE PARTIES
8. Plaintiff Cisco is a company duly organized and existing under the laws of California,
having its principal place of business at 170 West Tasman Drive, San Jose, CA 95134.
9. On information and belief, Defendant Arista is a corporation duly organized and existing
under the laws of Delaware, having its principal place of business at 5453 Great America Parkway,
Santa Clara, CA 95054.
JURISDICTION
10. This civil action asserts claims arising under the Patent Laws of the United States, 35
U.S.C. §§ 1 et seq. This Court has subject matter jurisdiction under 28 U.S.C. §§ 1331 and 1338(a).
11. This Court has personal jurisdiction over Arista. Arista has maintained its principal place
of business in the Northern District of California since 2004. Arista also has engaged in substantial and
not isolated business activities in the Northern District of California. Specifically, Arista, directly and/or
through third parties, has made, used, sold, and/or offered for sale within the Northern District of
California and/or imported into the Northern District of California infringing networking products.
VENUE
12. Venue properly lies in this District under 28 U.S.C. §§ 1391 and 1400(b) because
Arista’s principal place of business is in this District, acts of infringement have been committed in this
district, and Arista is subject to personal jurisdiction in this district. In addition, venue is proper because 1 Cisco, Cisco Blog - The Platform, “Employee Mobility,” available at
http://blogs.cisco.com/tag/employee-mobility/.
Case3:14-cv-05343 Document1 Filed12/05/14 Page4 of 22
5
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Cisco has suffered harm in this district.
INTRADISTRICT ASSIGNMENT
13. This Complaint includes an Intellectual Property Action, which is an excepted category
under Civil Local Rule 3-2(c). Consequently, this action is assigned on a district-wide basis.
GENERAL ALLEGATIONS
CISCO IS THE WORLDWIDE LEADER IN NETWORKING INNOVATIONS
14. Founded in 1984, Cisco is the worldwide leader in developing, implementing, and
providing the technologies behind networking products and services. Cisco develops and provides a
broad range of networking products and services that enable seamless communication among
individuals, businesses, public institutions, government agencies, and service providers. Specifically,
the thousands of engineers who work at Cisco develop and provide networking hardware, software, and
services that utilize cutting-edge technologies to transport data, voice, and video within buildings, across
cities and campuses, and around the world.
15. Since its founding, Cisco has pioneered many of the important technologies that created
and enabled global interconnectivity. During the past three decades, Cisco has invested billions of
dollars, and the time and dedication of thousands of its engineers, in the research and development of
networking products and services, culminating in the development of a highly-successful interface and
related technologies that have driven the proliferation of Cisco’s computer networking technologies and
the Internet.
16. Cisco’s networking devices and operating systems (including its Internetwork Operating
System (“IOS”, “IOS XR”, and “IOS XE”) and its Nexus Operating System (“NX-OS”)) are recognized
by customers and the industry generally as very important and unique, contributing tremendously to the
success and widespread acceptance of Cisco’s products. Included in Cisco’s products are features
important to the successful deployment of large and small networks and crucial to meeting the demands
of today’s networking environments, including networking device System Database (“SysDB”), Zero-
Touch Provisioning (“ZTP”), On Board Failure Logging (“OBFL”), Control Plane Policing (“CoPP”),
Spanning Tree Loop Guard, In-Service System Upgrades (“ISSU”), Virtual Port Channels (“vPC”),
Access Control Lists (“ACL”), and Private Virtual Local Area Networks (“Private VLANs”).
Case3:14-cv-05343 Document1 Filed12/05/14 Page5 of 22
6
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
17. As computing technologies evolve and new networking challenges arise, Cisco has
continued to innovate and develop new solutions for its customers. No matter what type of network
environment – whether large scale Internet backbone networks, enterprise-level local area networks, or
networks supporting data centers and today’s “cloud computing” services – Cisco’s technologies have
transformed how people connect, communicate, and collaborate. Cisco remains at the forefront of
developing cutting-edge networking technologies: in its last fiscal year alone (FY 2014), Cisco invested
more than $6 billion in ongoing research and development and employed more than ten thousand
engineers in California and elsewhere.
18. Cisco’s intellectual property rights, including its patent rights, protect the valuable
technologies developed by Cisco. As a result of its innovations, Cisco has developed a substantial
portfolio of U.S. patents, including the 12 patents asserted in this action.
CISCO’S PATENTED TECHNOLOGIES
19. Cisco’s products incorporate numerous patented technologies developed and owned by
Cisco. Twelve examples of Cisco’s patented technologies that are included in Cisco’s products are
described below (collectively, “the Patents-in-Suit”). See Exhibit 1. These patented technologies drive
customer demand for Cisco’s products, and Cisco relies on these technologies to lawfully compete in the
marketplace.
U.S. Patent No. 6,377,577
20. U.S. Patent No. 6,377,577 (“the ’577 patent”) entitled “Access Control List Processing in
Hardware” issued on April 23, 2002 and lists Andreas V. Bechtolsheim and David R. Cheriton as
inventors. A true and correct copy of the ’577 patent is attached hereto as Exhibit 2.
21. Cisco is the owner by assignment of the ’577 patent and has the full right to enforce
and/or license the ’577 patent.
22. The ’577 patent is valid and enforceable.
U.S. Patent No. 7,023,853
23. U.S. Patent No. 7,023,853 (“the ’853 patent”) entitled “Access Control List Processing in
Hardware” issued on April 4, 2006 and lists Andreas V. Bechtolsheim and David R. Cheriton as
inventors. A true and correct copy of the ’853 patent is attached hereto as Exhibit 3.
Case3:14-cv-05343 Document1 Filed12/05/14 Page6 of 22
7
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
24. Cisco is the owner by assignment of the ’853 patent and has the full right to enforce
and/or license the ’853 patent.
25. The ’853 patent is valid and enforceable.
U.S. Patent No. 7,340,597
26. U.S. Patent No. 7,340,597 (“the ’597 patent”) entitled “Method and Apparatus for
Securing a Communications Device Using a Logging Module” issued on March 4, 2008 and lists David
R. Cheriton as inventor. A true and correct copy of the ’597 patent is attached hereto as Exhibit 4.
27. Cisco is the owner by assignment of the ’597 patent and has the full right to enforce
and/or license the ’597 patent.
28. The ’597 patent is valid and enforceable.
U.S. Patent No. 7,162,537
29. U.S. Patent No. 7,162,537 (“the ’537 patent”) entitled “Method and System for
Externally Managing Router Configuration Data in Conjunction With a Centralized Database” issued on
January 9, 2007 and lists Pradeep Kathail as inventor. A true and correct copy of the ’537 patent is
attached hereto as Exhibit 5.
30. Cisco is the owner by assignment of the ’537 patent and has the full right to enforce
and/or license the ’537 patent.
31. The ’537 patent is valid and enforceable.
U.S. Patent No. 8,051,211
32. U.S. Patent No. 8,051,211 (“the ’211 patent”) entitled “Multi-Bridge LAN Aggregation”
issued on November 1, 2011 and lists Norman W. Finn as the inventor. A true and correct copy of the
’211 patent is attached hereto as Exhibit 6.
33. Cisco is the owner by assignment of the ’211 patent and has the full right to enforce
and/or license the ’211 patent.
34. The ’211 patent is valid and enforceable.
U.S. Patent No. 8,356,296
35. U.S. Patent No. 8,356,296 (“the ’296 patent”) entitled “Method and System for Minimal
Disruption During Software Upgrade or Reload of a Network Device” issued on January 15, 2013 and
Case3:14-cv-05343 Document1 Filed12/05/14 Page7 of 22
8
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
lists John Thomas Welder, Ratheesh Krishna Vadhyar, Sudhir Rao, and Thomas W. Uban as inventors.
A true and correct copy of the ’296 patent is attached hereto as Exhibit 7.
36. Cisco is the owner by assignment of the ’296 patent and has the full right to enforce
and/or license the ’296 patent.
37. The ’296 patent is valid and enforceable.
U.S. Patent No. 7,290,164
38. U.S. Patent No. 7,290,164 (“the ’164 patent”) entitled “Method of Reverting to a
Recovery Configuration in Response to Device Faults” issued on October 30, 2007 and lists Andrew G.
Harvey, John Ng, and Gilbert R. Woodman III as inventors. A true and correct copy of the ’164 patent
is attached hereto as Exhibit 8.
39. Cisco is the owner by assignment of the ’164 patent and has the full right to enforce
and/or license the ’164 patent.
40. The ’164 patent is valid and enforceable.
U.S. Patent No. 6,741,592
41. U.S. Patent No. 6,741,592 (“the ’592 patent”) entitled “Private VLANs” issued on May
25, 2004 and lists Thomas J. Edsall, Marco Foschiano, Michael Fine, and Thomas Nosella as inventors.
A true and correct copy of the ’592 patent is attached hereto as Exhibit 9.
42. Cisco is the owner by assignment of the ’592 patent and has the full right to enforce
and/or license the ’592 patent.
43. The ’592 patent is valid and enforceable.
U.S. Patent No. 7,200,145
44. U.S. Patent No. 7,200,145 (“the ’145 patent”) entitled “Private VLANs” issued on April
3, 2007 and lists Thomas J. Edsall, Marco Foschiano, Michael Fine, and Thomas Nosella as inventors.
A true and correct copy of the ’145 patent is attached hereto as Exhibit 10.
45. Cisco is the owner by assignment of the ’145 patent and has the full right to enforce
and/or license the ’145 patent.
46. The ’145 patent is valid and enforceable.
Case3:14-cv-05343 Document1 Filed12/05/14 Page8 of 22
9
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
U.S. Patent No. 7,460,492
47. U.S. Patent No. 7,460,492 (“the ’492 patent”) entitled “Spanning Tree Loop Guard”
issued on December 2, 2008 and lists Maurizio Portolani, Shayamasundar S. Kaluve, and Marco E.
Foschiano as inventors. A true and correct copy of the ’492 patent is attached hereto as Exhibit 11.
48. Cisco is the owner by assignment of the ’492 patent and has the full right to enforce
and/or license the ’492 patent.
49. The ’492 patent is valid and enforceable.
U.S. Patent No. 7,061,875
50. U.S. Patent No. 7,061,875 (“the ’875 patent”) entitled “Spanning Tree Loop Guard”
issued on June 13, 2006 and lists Maurizio Portolani, Shayamasundar S. Kaluve, and Marco E.
Foschiano as inventors. A true and correct copy of the ’875 patent is attached hereto as Exhibit 12.
51. Cisco is the owner by assignment of the ’875 patent and has the full right to enforce
and/or license the ’875 patent.
52. The ’875 patent is valid and enforceable.
U.S. Patent No. 7,224,668
53. U.S. Patent No. 7,224,668 (“the ’668 patent”) entitled “Control Plane Security and
Traffic Flow Management” issued on May 29, 2007 and lists Adrian C. Smethurst, Michael F. Keohane,
and R. Wayne Ogozaly as inventors. A true and correct copy of the ’668 patent is attached hereto as
Exhibit 13.
54. Cisco is the owner by assignment of the ’668 patent and has the full right to enforce
and/or license the ’668 patent.
55. The ’668 patent is valid and enforceable.
ARISTA IS WILLFULLY INFRINGING CISCO’S PATENTS
56. Decades after Cisco’s founding, Arista was founded by former Cisco employees who
were intimately and directly familiar with Cisco’s pioneering networking technologies, including those
protected by patents asserted in this action. Since its founding, numerous additional Cisco employees
have also joined Arista. For example, Arista founder and Chief Development Officer Andreas
Bechtolsheim served as Vice President and General Manager of Cisco’s Gigabit Systems Business Unit;
Case3:14-cv-05343 Document1 Filed12/05/14 Page9 of 22
10
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Arista founder and Chief Scientist David Cheriton served as a Chief Architect on Cisco’s Catalyst
products; Arista founder, Chief Technology Officer, and Senior Vice President Kenneth Duda worked at
Cisco for several years as a software engineer in Cisco’s Gigabit Systems Business Unit; and Arista’s
current President and Chief Executive Officer, Jayshree Ullal, worked at Cisco for more than a decade,
including as Senior Vice President of Cisco’s Data Center, Switching, and Services Group (which is
responsible for some of Cisco’s flagship networking product lines). Cisco strongly believes, and has
repeatedly stated, that mobility of employees between companies fosters innovation.2 But widespread
intellectual property infringement like that engaged in by Arista stifles innovation and cannot be
condoned.
57. Arista knew that Cisco’s pioneering networking technologies drive customer demand for
and are important to the market success of Cisco’s products. Rather than invest in the expensive and
time-consuming effort that would have been necessary to develop its own features for Arista’s products,
and specifically instead of investing the time and expense of developing its own technologies, Arista
instead decided to use Cisco’s pioneering proprietary technologies, and even to explicitly tout these
technologies to the market in attempts to sell Arista products that compete directly with Cisco products.
58. Cisco inventions are important to and drive customer demand for Arista’s products. For
example, Cisco’s patented technology can be found in Arista’s System Database (“SysDB”), Zero-
Touch Provisioning (“ZTP”), Multi-Chassis Link Aggregation (“MLAG”), Control Plane Protection
(“CoPP”), In-Service System Upgrades (“ISSU”), Extensible API (“eAPI”), Access Control Lists
(“ACL”), Spanning Tree Loop Guard, and Private Virtual Local Area Networks (“Private VLANs”).
59. Arista’s misappropriation of Cisco technology has been crucial to Arista’s attempts to
compete with Cisco. Arista claims that the Cisco technologies it has unlawfully used are the “secret
sauce” of its product line, and touts that these features, inter alia, “simplif[y] deployment and
minimize[] errors,” function as the “core” of its operating system, “eliminate bottlenecks and provide
resiliency” to “protect the control plane from potential denial of service attacks,” and “provide[] the
foundation for . . . updates and self-healing resiliency.” By extensively using Cisco’s patented 2 Cisco, Cisco Blog - The Platform, “Employee Mobility,” available at
http://blogs.cisco.com/tag/employee-mobility/.
Case3:14-cv-05343 Document1 Filed12/05/14 Page10 of 22
11
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
technologies, Arista took improper shortcuts, thereby avoiding the investments that would have been
necessary had Arista not used Cisco’s technology.
60. There is no question that Arista personnel – many of whom worked at Cisco at or after
the time the technologies were developed by Cisco – were aware that the pioneering Cisco networking
technologies that Arista appropriated are protected by U.S. patents. For example, two of Arista’s own
founders are named inventors on a number of Cisco patents asserted in this action. By this action, Cisco
seeks to stop Arista’s willful, unauthorized, and improper use of Cisco’s patented technologies, and to
obtain damages for the significant harm caused to Cisco by Arista’s willful infringement of certain
Patents-in-Suit.
COUNT I – INFRINGEMENT OF THE ’577 PATENT
61. Cisco incorporates and realleges Paragraphs 1 through 60 of this Complaint as if fully set
forth herein.
62. The USPTO duly and legally issued the ’577 patent on April 23, 2002.
63. Arista has infringed, and continues to infringe, one or more claims of the ’577 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’577 patent, including but not limited to the
Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, including, without limitation,
those devices’ implementations of access control list functionality.
64. The ’577 patent was issued to Messrs. Bechtolsheim and Cheriton on April 23, 2002,
while they were Cisco employees. The ’577 patent is assigned to Cisco. Messrs. Bechtolsheim and
Cheriton are co-founders of Arista. Accordingly, Arista has had knowledge of the ’577 patent since its
founding in October 2004. In addition to directly infringing the ’577 patent, Arista has indirectly
infringed and continues to indirectly infringe one or more claims of the ’577 patent, including at least
claim 1, by actively inducing others to directly infringe the ’577 patent in violation of 35 U.S.C. §
271(b). Specifically, and in light of the knowledge of its founders, Arista knowingly induced
infringement of the ’577 patent with specific intent to do so by its activities relating to the marketing,
distribution, and/or sale of its networking products to their purchasers, including but not limited to the
Case3:14-cv-05343 Document1 Filed12/05/14 Page11 of 22
12
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, and by instructing and
encouraging purchasers (including through product documentation) to operate and use those products in
an infringing manner with knowledge that these actions would infringe the ’577 patent.
65. Arista has contributed to infringement of the ’577 patent by others by selling and/or
offering for sale to Arista’s purchasers within the United States and/or importing into the United States
networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and
7500E series of switches, that are especially made and/or adapted for infringing the ’577 patent and are
not staple articles of commerce suitable for substantial noninfringing use and that have been sold to
purchasers who infringe the ’577 patent. As alleged in the prior paragraphs, the ’577 patent was issued
to Messrs. Bechtolsheim and Cheriton on April 23, 2002, while they were Cisco employees.
Specifically, and in light of the knowledge of its founders, Arista had knowledge that its networking
products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of
switches, were specifically made and/or adapted for infringement of the ’577 patent and are not staple
articles of commerce suitable for substantial noninfringing use.
66. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
67. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
68. Arista has infringed the ’577 patent as alleged above despite having prior knowledge of
the patent and has acted with willful, intentional, and conscious disregard of the objectively high
likelihood that its acts constitute infringement of the ’577 patent. Arista’s infringement of the ’577
patent has been and continues to be willful, entitling Cisco to enhanced damages under 35 U.S.C. § 284.
COUNT II – INFRINGEMENT OF THE ’853 PATENT
69. Cisco incorporates and realleges Paragraphs 1 through 68 of this Complaint as if fully set
forth herein.
70. The USPTO duly and legally issued the ’853 patent on April 4, 2006.
71. Arista has infringed, and continues to infringe, one or more claims of the ’853 patent,
Case3:14-cv-05343 Document1 Filed12/05/14 Page12 of 22
13
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
including at least claim 63, either literally or under the doctrine of equivalents, by making, using,
selling, and/or offering for sale within the United States and/or importing into the United States
networking products that are covered by one or more claims of the ’853 patent, including but not limited
to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of switches, including, without
limitation, those devices’ implementations of access control list functionality.
72. The ’853 patent is a continuation of the ’577 patent and was issued on April 4, 2006 to
Messrs. Bechtolsheim and Cheriton, and is assigned to Cisco. In addition to directly infringing the ’853
patent, Arista has indirectly infringed and continues to indirectly infringe one or more claims of the ’853
patent, including at least claim 63, including by actively inducing others to directly infringe the ’853
patent in violation of 35 U.S.C. § 271(b). Specifically, and in light of the knowledge of Arista’s
founders of their Cisco patent, Arista knowingly induced infringement of the ’853 patent with specific
intent to do so by its activities relating to the marketing, distribution, and/or sale of its networking
products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and 7500E series of
switches, and by instructing and encouraging purchasers (including through product documentation) to
operate and use those products in an infringing manner with knowledge that these actions would infringe
the ’853 patent.
73. Arista has contributed to infringement of the ’853 patent by others by selling and/or
offering for sale to Arista’s purchasers within the United States and/or importing into the United States
networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X, and
7500E series of switches, that are especially made and/or adapted for infringing the ’853 patent and are
not staple articles of commerce suitable for substantial noninfringing use. As alleged in the prior
paragraphs, the ’853 patent was issued to Messrs. Bechtolsheim and Cheriton on April 6, 2006 and is
assigned to Cisco. Specifically, and in light of the knowledge of its co-founders, Arista had knowledge
that its networking products, including but not limited to the Arista 7048, 7050X, 7250X, 7300, 7300X,
and 7500E series of switches, were specifically made and/or adapted for infringement of the ’853 patent
and are not staple articles of commerce suitable for substantial noninfringing use.
74. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
Case3:14-cv-05343 Document1 Filed12/05/14 Page13 of 22
14
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
infringement is enjoined by this Court.
75. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
76. Arista has infringed the ’853 patent as alleged above despite having prior knowledge of
the patent and has acted with willful, intentional, and conscious disregard of the objectively high
likelihood that its acts constitute infringement of the ’853 patent. Arista’s infringement of the ’853
patent has been and continues to be willful, entitling Cisco to enhanced damages under 35 U.S.C. § 284.
COUNT III – INFRINGEMENT OF THE ’597 PATENT
77. Cisco incorporates and realleges Paragraphs 1 through 76 of this Complaint as if fully set
forth herein.
78. The USPTO duly and legally issued the ’597 patent on March 4, 2008.
79. Arista has infringed, and continues to infringe, one or more claims of the ’597 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’597 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s process manager
functionality.
80. The ’597 patent was issued to Arista co-founder Cheriton on March 4, 2008, and is
assigned to Cisco. In addition to directly infringing the ’597 patent, Arista has indirectly infringed and
continues to indirectly infringe one or more claims of the ’597 patent, including at least claim 1,
including by actively inducing others to directly infringe the ’597 patent in violation of 35 U.S.C. §
271(b). Specifically, and in light of the knowledge of its co-founder, Arista knowingly induced
infringement of the ’597 patent with specific intent to do so by its activities relating to the marketing,
distribution, and/or sale its networking products, including but not limited to the Arista 7010, 7048,
7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of switches, and by
instructing and encouraging purchasers (including through product documentation) to operate and use
those products in an infringing manner with knowledge that these actions would infringe the ’597 patent.
Case3:14-cv-05343 Document1 Filed12/05/14 Page14 of 22
15
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
81. Arista has contributed to infringement of the ’597 patent by others by selling and/or
offering for sale to Arista’s purchasers within the United States and/or importing into the United States
networking products, including but not limited to the Arista 7010, 7048, 7050, 7050X, 7100, 7150,
7250X, 7280E, 7300, 7300X, 7500, and 7500E series of switches, which are especially made and/or
adapted for infringing the ’597 patent and are not staple articles of commerce suitable for substantial
noninfringing use. As alleged in the prior paragraphs, the ’597 patent was issued to Arista co-founder
Cheriton on March 4, 2008, and is assigned to Cisco. Specifically, and in light of the knowledge of its
co-founder, Arista had knowledge that its networking products, including but not limited to the Arista
7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, were specifically made and/or adapted for infringement of the ’597 patent and are not staple
articles of commerce suitable for substantial noninfringing use.
82. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
83. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
84. Arista has infringed the ’597 patent as alleged above despite having prior knowledge of
the patent and has acted with willful, intentional, and conscious disregard of the objectively high
likelihood that its acts constitute infringement of the ’597 patent. Arista’s infringement of the ’597
patent has been and continues to be willful, entitling Cisco to enhanced damages under 35 U.S.C. § 284.
COUNT IV – INFRINGEMENT OF THE ’537 PATENT
85. Cisco incorporates and realleges Paragraphs 1 through 84 of this Complaint as if fully set
forth herein.
86. The USPTO duly and legally issued the ’537 patent on January 9, 2007.
87. Arista has infringed, and continues to infringe, one or more claims of the ’537 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’537 patent, including but not limited to the
Case3:14-cv-05343 Document1 Filed12/05/14 Page15 of 22
16
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s SysDB functionality.
88. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
89. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT V – INFRINGEMENT OF THE ’211 PATENT
90. Cisco incorporates and realleges Paragraphs 1 through 89 of this Complaint as if fully set
forth herein.
91. The USPTO duly and legally issued the ’211 patent on November 1, 2011.
92. Arista has infringed, and continues to infringe, one or more claims of the ’211 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’211 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s multi-chassis link
aggregation, or MLAG, functionality.
93. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
94. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT VI – INFRINGEMENT OF THE ’296 PATENT
95. Cisco incorporates and realleges Paragraphs 1 through 94 of this Complaint as if fully set
forth herein.
96. The USPTO duly and legally issued the ’296 patent on January 15, 2013.
97. Arista has infringed, and continues to infringe, one or more claims of the ’296 patent,
Case3:14-cv-05343 Document1 Filed12/05/14 Page16 of 22
17
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’296 patent, including but not limited to the
Arista 7300, 7300X, 7500, and 7500E series of switches, including, without limitation, those devices’
implementations of Arista’s in-service software upgrade, or ISSU, functionality.
98. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
99. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT VII – INFRINGEMENT OF THE ’164 PATENT
100. Cisco incorporates and realleges Paragraphs 1 through 99 of this Complaint as if fully set
forth herein.
101. The USPTO duly and legally issued the ’164 patent on October 30, 2007.
102. Arista has infringed, and continues to infringe, one or more claims of the ’164 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’164 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s zero touch
provisioning, or ZTP, functionality.
103. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
104. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT VIII – INFRINGEMENT OF THE ’592 PATENT
105. Cisco incorporates and realleges Paragraphs 1 through 104 of this Complaint as if fully
Case3:14-cv-05343 Document1 Filed12/05/14 Page17 of 22
18
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
set forth herein.
106. The USPTO duly and legally issued the ’592 patent on May 25, 2004.
107. Arista has infringed, and continues to infringe, one or more claims of the ’592 patent,
including at least claim 6, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’592 patent, including but not limited to the
Arista 7010, 7050, 7050X, 7100, 7150, 7250X, 7300, and 7300X series of switches, including, without
limitation, those devices’ implementations of Arista’s private VLAN functionality.
108. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
109. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT IX – INFRINGEMENT OF THE ’145 PATENT
110. Cisco incorporates and realleges Paragraphs 1 through 109 of this Complaint as if fully
set forth herein.
111. The USPTO duly and legally issued the ’145 patent on April 3, 2007.
112. Arista has infringed, and continues to infringe, one or more claims of the ’145 patent,
including at least claim 5, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’145 patent, including but not limited to the
Arista 7010, 7050, 7050X, 7100, 7150, 7250X, 7300, and 7300X series of switches, including, without
limitation, those devices’ implementations of Arista’s private VLAN functionality.
113. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
114. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
Case3:14-cv-05343 Document1 Filed12/05/14 Page18 of 22
19
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
COUNT X – INFRINGEMENT OF THE ’492 PATENT
115. Cisco incorporates and realleges Paragraphs 1 through 114 of this Complaint as if fully
set forth herein.
116. The USPTO duly and legally issued the ’492 patent on December 2, 2008.
117. Arista has infringed, and continues to infringe, one or more claims of the ’492 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’492 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s loop guard
functionality.
118. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
119. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT XI – INFRINGEMENT OF THE ’875 PATENT
120. Cisco incorporates and realleges Paragraphs 1 through 119 of this Complaint as if fully
set forth herein.
121. The USPTO duly and legally issued the ’875 patent on June 13, 2006.
122. Arista has infringed, and continues to infringe, one or more claims of the ’875 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’875 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s loop guard
functionality.
123. Arista’s infringement has caused and is continuing to cause damage and irreparable
Case3:14-cv-05343 Document1 Filed12/05/14 Page19 of 22
20
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
124. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
COUNT XII – INFRINGEMENT OF THE ’668 PATENT
125. Cisco incorporates and realleges Paragraphs 1 through 124 of this Complaint as if fully
set forth herein.
126. The USPTO duly and legally issued the ’668 patent on May 29, 2007.
127. Arista has infringed, and continues to infringe, one or more claims of the ’668 patent,
including at least claim 1, either literally or under the doctrine of equivalents, by making, using, selling,
and/or offering for sale within the United States and/or importing into the United States networking
products that are covered by one or more claims of the ’668 patent, including but not limited to the
Arista 7010, 7048, 7050, 7050X, 7100, 7150, 7250X, 7280E, 7300, 7300X, 7500, and 7500E series of
switches, including, without limitation, those devices’ implementations of Arista’s control plane
policing, or CoPP, functionality.
128. Arista’s infringement has caused and is continuing to cause damage and irreparable
injury to Cisco, and Cisco will continue to suffer damage and irreparable injury unless and until that
infringement is enjoined by this Court.
129. Cisco is entitled to injunctive relief and damages in accordance with 35 U.S.C. §§ 271,
281, 283, and 284.
PRAYER FOR RELIEF
WHEREFORE, Cisco prays for relief as follows:
1. For a declaration that Arista has infringed the Patents-in-Suit;
2. For a declaration of a substantial likelihood that Arista will continue to infringe Cisco’s
intellectual property unless enjoined from doing so;
3. That, in accordance with 35 U.S.C. § 283, Arista, and all affiliates, employees, agents,
officers, directors, attorneys, successors, and assigns, and all those acting on behalf of or
in active concert or participation with any of them, be preliminarily and permanently
Case3:14-cv-05343 Document1 Filed12/05/14 Page20 of 22
21
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
enjoined from infringing the Patents-in-Suit;
4. For an award of damages sufficient to compensate Cisco for Arista’s infringement of the
Patents-in-Suit, including lost profits suffered by Cisco as a result of Arista’s
infringement and in an amount not less than a reasonable royalty;
5. For an award of increased damages in an amount not less than three times the damages
assessed for Arista’s infringement of the Patents-in-Suit, in accordance with 35 U.S.C. §
284;
6. For a declaration that this case is “exceptional” under 35 U.S.C. § 285, and an award to
Cisco of its reasonable attorneys’ fees, expenses, and costs incurred in this action;
7. For an award of prejudgment and post-judgment interest; and
8. For such other and further relief as this Court shall deem appropriate.
DEMAND FOR JURY TRIAL
Pursuant to Rule 38(b) of the Federal Rules of Civil Procedure, Cisco demands a trial by jury on
all issues raised by the Complaint.
Case3:14-cv-05343 Document1 Filed12/05/14 Page21 of 22
22
COMPLAINT FOR PATENT INFRINGEMENT Case No. 3:14-cv-5343
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
DATED: December 5, 2014 Respectfully submitted, /s/ Adam R. Alper
Steven [email protected] KIRKLAND & ELLIS LLP 601 Lexington Avenue New York, New York 10022 Telephone: (212) 446-4800 Facsimile: (212) 446-4900 Adam R. Alper (SBN 196834) [email protected] KIRKLAND & ELLIS LLP 555 California Street San Francisco, California 94104 Telephone: (415) 439-1400 Facsimile: (415) 439-1500 Michael W. De Vries (SBN 211001) [email protected] 333 South Hope Street Los Angeles, California 90071 Telephone: (213) 680-8400 Facsimile: (213) 680-8500 Attorneys for Plaintiff Cisco Systems, Inc.
Case3:14-cv-05343 Document1 Filed12/05/14 Page22 of 22
Exhibit 1
Case3:14-cv-05343 Document1-1 Filed12/05/14 Page1 of 2
Patent # Technology Area Description Injunction Available Until
Drives Customer Demand
Used In Cisco Products
7,162,537 Configuration / Management networking device sysDB
1/6/2020 Yes Yes
8,356,296 Configuration / Management ISSU 1/26/2024 Yes Yes
7,290,164 Configuration / Management ZTP 6/7/2025 Yes Yes
7,340,597 Configuration / Management configurationchange detection
1/26/2026 Yes Yes
7,023,853 Access Control (ACL) ACL/TCAM 6/30/2018 Yes Yes
6,377,577 Access Control (ACL) ACL/TCAM 6/30/2018 Yes Yes
7,460,492 Router / Switch Control loop guard 2/12/2022 Yes Yes
7,061,875 Router / Switch Control loop guard 9/17/2024 Yes Yes
7,224,668 Router / Switch Control CoPP 8/23/2025 Yes Yes
8,051,211 Router / Switch Control MLAG 12/20/2028 Yes Yes
6,741,592 VLAN private VLANs 5/22/2020 Yes Yes
7,200,145 VLAN private VLANs 5/22/2020 Yes Yes
Patents in Suit
Case3:14-cv-05343 Document1-1 Filed12/05/14 Page2 of 2
Exhibit 2
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page1 of 13
(12) United States Patent Bechtolsheim et al.
(54) ACCESS CONTROL LIST PROCESSING IN HARDWARE
(75) Inventors: Andreas V. Bechtolsheim, Stanford; David R. Cheriton, Palo Alto, both of CA(US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.
(21) Appl. No.: 09/108,071
(22)
(51) (52) (58)
(56)
Filed: Jun. 30, 1998
Int. Cl? . ... ... .. ... ... ... ... .. ... ... ... ... ... .. ... ... ... . G06F 9/34 U.S. Cl. ................... 370/392; 370/395.32; 370/389 Field of Search ................................. 370/392, 393,
370/394, 396, 397, 398, 399, 400, 389, 395.32; 709/220, 221, 222, 227, 228, 229
References Cited
U.S. PATENT DOCUMENTS
4,131,767 A 12/1978 4,161,719 A 7/1979 4,316,284 A 2/1982 4,397,020 A 8/1983 4,419,728 A 12/1983 4,424,565 A 1!1984 4,437,087 A 3/1984 4,438,511 A 3/1984 4,439,763 A 3/1984 4,445,213 A 4/1984 4,446,555 A 5/1984 4,456,957 A 6/1984 4,464,658 A 8/1984 4,499,576 A 2/1985 4,506,358 A 3/1985 4,507,760 A 3/1985 4,532,626 A 7/1985 4,644,532 A 2/1987 4,646,287 A 2/1987 4,677,423 A 6/1987
Weinstein Parikh eta!. Howson Howson Larson Larson Petr Baran Limb Baugh eta!. Devault et a!. Schieltz Thelen Fraser Montgomery Fraser Flores eta!. George eta!. Larson eta!. Benvenuto et a!.
100"
PACKET INPUT
INTERFACES
111111 1111111111111111111111111111111111111111111111111111111111111 US006377577Bl
(10) Patent No.: US 6,377,577 Bl Apr. 23, 2002 (45) Date of Patent:
4,679,189 A 4,679,227 A 4,723,267 A 4,731,816 A
7/1987 Olson eta!. 7/1987 Hughes-Hartogs 2/1988 Jones et a!. 3/1988 Hughes-Hartogs
(List continued on next page.)
OTHER PUBLICATIONS
Alessandri, Access Control List Processing in Hardware, Diploma Thesis, ETH, pp. 1-85, Oct. 1997.*
Primary Examiner-Wellington Chin Assistant Examiner-Frank Duong (74) Attorney, Agent, or Firm---Skjerven Morrill MacPherson LLP
(57) ABSTRACT
The invention provides for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a rate comparable to wirespeed. The CAM includes an ordered sequence of entries, each of which has an array of ternary elements for matching "0", "1", or any value, and each of which generates a match signal. The ACL entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier. A router including the CAM can also include preprocessing circuits for certain range comparisons which have been found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM, such as comparisons of the port number against known special cases such as "greater than 1023" or "within the range 6000 to 6500".
31 Claims, 3 Drawing Sheets
PACKET OUTPUT
INTERFACES
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page2 of 13
US 6,377,577 Bl Page 2
U.S. PATENT DOCUMENTS 5,309,437 A 5/1994 Perlman et a!. .......... 730/85.13 5,311,509 A 5/1994 Heddes eta!.
4,750,136 A 6/1988 Arpin eta!. 5,313,454 A 5/1994 Bustini et a!. 4,757,495 A 7/1988 Decker eta!. 5,313,582 A 5/1994 Hendel eta!. 4,763,191 A 8/1988 Gordon eta!. 5,317,562 A 5/1994 Nardin eta!. 4,769,810 A 9/1988 Eckberg, Jr. et a!.
5,319,644 A 6/1994 Liang 4,769,811 A 9/1988 Eckberg, Jr. et a!. 4,771,425 A 9/1988 Baran eta!. 5,327,421 A 7/1994 Hiller eta!.
4,819,228 A 4/1989 Baran eta!. 5,331,637 A 7/1994 Francis et a!.
4,827,411 A 5/1989 Arrowood et a!. 5,345,445 A 9/1994 Hiller eta!.
4,833,706 A 5/1989 Hughes-Hartogs 5,345,446 A 9/1994 Hiller eta!.
4,835,737 A 5/1989 Herrig eta!. 5,359,591 A 10/1994 Corbalis et a!.
4,879,551 A 11/1989 Georgiou et a!. 5,361,250 A 11/1994 Nguyen eta!. 4,893,306 A 1!1990 Chao eta!. 5,361,256 A 11/1994 Doeringer et a!. 4,903,261 A 2/1990 Baran eta!. 5,361,259 A 11/1994 Hunt eta!. 4,922,486 A 5/1990 Lidinsky et a!. 5,365,524 A 11/1994 Hiller eta!. 4,933,937 A 6/1990 Konishi 5,367,517 A 11/1994 Cidon eta!. 4,960,310 A 10/1990 Cushing 5,371,852 A 12/1994 Attanasio et a!. 4,962,497 A 10/1990 Ferenc eta!. 5,386,567 A 1!1995 Lien eta!. 4,962,532 A 10/1990 Kasirai et a!. 5,390,170 A 2/1995 Sawant eta!. 4,965,772 A 10/1990 Daniel eta!. 5,390,175 A 2/1995 Hiller eta!. 4,970,678 A 11/1990 Sladowski et a!. 5,394,394 A 2/1995 Crowther et a!. 4,979,118 A 12/1990 Kheradpir ................... 364/436 5,394,402 A 2/1995 Ross 4,980,897 A 12/1990 Decker eta!. 5,400,325 A 3/1995 Chatwani et a!. 4,991,169 A 2/1991 Davis eta!. 5,408,469 A 4/1995 Opher eta!. 5,003,595 A 3/1991 Collins eta!. 5,416,842 A 5/1995 Aziz 5,014,265 A 5/1991 Hahne eta!. 5,422,880 A 6/1995 Heitkamp et a!. 5,020,058 A 5/1991 Holden eta!. 5,422,882 A 6/1995 Hiller eta!. 5,033,076 A 7/1991 Jones eta!. 5,423,002 A 6/1995 Hart 5,054,034 A 10/1991 Hughes-Hartogs 5,426,636 A 6/1995 Hiller eta!. 5,059,925 A 10/1991 Weisbloom 5,428,607 A 6/1995 Hiller eta!. 5,072,449 A 12/1991 Enns eta!. 5,430,715 A 7/1995 Corbalis et a!. 5,088,032 A 2/1992 Bosack 5,430,729 A 7/1995 Rahnema 5,095,480 A 3/1992 Fenner 5,442,457 A 8/1995 Najafi RE33,900 E 4/1992 Howson 5,442,630 A 8/1995 Gagliardi et a!. 5,115,431 A 5/1992 Williams et a!. 5,452,297 A 9/1995 Hiller eta!. 5,128,945 A 7/1992 Enns eta!. 5,473,599 A 12/1995 Li eta!. 5,136,580 A 8/1992 Videlock et a!. 5,473,607 A 12/1995 Hausman et a!. 5,166,930 A 11/1992 Braff eta!. 5,477,541 A 12/1995 White eta!. 5,199,049 A 3/1993 Wilson 5,485,455 A 1!1996 Dobbins et a!. 5,206,886 A 4/1993 Bingham 5,490,140 A 2/1996 Abensour et a!. 5,208,811 A 5/1993 Kashio eta!. 5,490,257 A 2/1996 Fenner 5,212,686 A 5/1993 Joy eta!. 5,491,687 A 2/1996 Christensen et a!. 5,224,099 A 6/1993 Corbalis et a!. 5,491,804 A 2/1996 Heath eta!. 5,226,120 A 7/1993 Brown eta!. 5,497,368 A 3/1996 Reijnierse et a!. 5,228,062 A 7/1993 Bingham 5,504,747 A 4/1996 Sweasey 5,229,994 A 7/1993 Balzano et a!. 5,509,006 A 4/1996 Wilford et a!. 5,237,564 A 8/1993 Lespagnol et a!. 5,517,494 A 5/1996 Green 5,241,682 A 8/1993 Bryant eta!. 5,519,704 A 5/1996 Farinacci et a!. 5,243,342 A 9/1993 Kattemalalavadi et a!. 5,519,858 A 5/1996 Walton eta!. .............. 395/600 5,243,596 A 9/1993 Port eta!. 5,526,489 A 6/1996 Nilakantan et a!. 5,247,516 A 9/1993 Bernstein et a!. 5,530,963 A 6/1996 Moore eta!. 5,249,178 A 9/1993 Kurano eta!. 5,535,195 A 7/1996 Lee 5,253,251 A 10/1993 Aramaki 5,539,734 A 7/1996 Burwell eta!. 5,255,291 A 10/1993 Holden eta!. 5,541,911 A 7/1996 Nilakantan et a!. 5,260,933 A 11/1993 Rouse 5,546,370 A 8/1996 Ishikawa 5,260,978 A 11/1993 Fleischer et a!. 5,555,244 A 9/1996 Gupta eta!. 5,268,592 A 12/1993 Bellamy eta!. 5,561,669 A 10/1996 Lenney eta!. 5,268,900 A 12/1993 Hluchyj et a!. 5,583,862 A 12/1996 Calion 5,271,004 A 12/1993 Proctor et a!. 5,592,470 A 1!1997 Rudrapatna et a!. 5,274,631 A 12/1993 Bhardwaj 5,598,581 A 1!1997 Daines eta!. 5,274,635 A 12/1993 Rahman eta!. 5,600,798 A 2/1997 Chenrukuri et a!. 5,274,643 A 12/1993 Fisk 5,604,868 A 2/1997 Komine eta!. 5,280,470 A 1!1994 Buhrke eta!. 5,608,726 A 3/1997 Virgile 5,280,480 A 1!1994 Pitt eta!. 5,617,417 A 4/1997 Sathe eta!. 5,280,500 A 1!1994 Mazzola eta!. 5,617,421 A 4/1997 Chin eta!. 5,283,783 A 2/1994 Nguyen eta!. 5,630,125 A 5/1997 Zellweger 5,287,103 A 2/1994 Kasprzyk et a!. 5,631,908 A 5/1997 Saxe 5,287,453 A 2/1994 Roberts 5,632,021 A 5/1997 Jennings et a!. 5,291,482 A 3/1994 McHarg eta!. 5,634,010 A 5/1997 Ciscon eta!. 5,305,311 A 4/1994 Lyles 5,638,359 A 6/1997 Peltola et a!. 5,307,343 A 4/1994 Bostica et a!. 5,644,718 A 7/1997 Belove eta!.
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page3 of 13
US 6,377,577 Bl Page 3
5,659,684 A 8/1997 Giovannoni et a!. 5,748,617 A 5/1998 McLain, Jr. 5,666,353 A 9/1997 Klausmeier et a!. 5,754,547 A 5/1998 Nakazawa 5,673,265 A 9/1997 Gupta eta!. 5,802,054 A 9/1998 Bellenger 5,678,006 A 10/1997 Valizadeh et a!. 5,835,710 A 11/1998 Nagami eta!. 5,680,116 A 10/1997 Hashimoto et a!. 5,854,903 A 12/1998 Morrison et a!. 5,684,797 A 11/1997 Aznar eta!. 5,856,981 A 1!1999 Voelker 5,687,324 A 11/1997 Green eta!. 5,892,924 A 4/1999 Lyon eta!. ............ 395/200.75 5,689,506 A 11/1997 Chiussi et a!. 5,898,686 A 4/1999 Virgile 5,694,390 A 12/1997 Yamato eta!. 5,903,559 A 5/1999 Acharya et a!. 5,724,351 A 3/1998 chao eta!. 5,748,186 A 5/1998 Raman * cited by examiner
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page4 of 13
100'
\..._
• • •
PACK
ET
INPU
T IN
TERF
ACES
/
110
ROUT
ING
AC
CESS
EL
EMEN
T CO
NTRO
L EL
EMEN
T
FIG.
1
~
• • •
PACK
ET
OUTP
UT
INTE
RFAC
ES
d • \Jl
• ~
~
......
~ =
...... >
't:l :-: N
~~
N c c N
'JJ.
=- ~ ~ .....
'"""'
0 ......, ~ e rJ
'l 0'
1 ~
""-l
""-l
11.
""-l
""-l ~
1-"
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page5 of 13
U.S. Patent
INPUT PORT 201
Apr. 23, 2002
COMPARE CIRCUIT
230
COMPARE BITS 231
Sheet 2 of 3
213 211 ACCESS • CONTROL • PATTERN SPECIFIER
211
ACCESS CONTROL MEMORY
210
FIG. 2
US 6,377,577 Bl
OlJTPUT PORT 202
PRIORITY ENCODER
220
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page6 of 13
U.S. Patent Apr. 23, 2002 Sheet 3 of 3
300\...,
RECEIVE ~READYTO
321 RECEIVE PACKET
i 322
IDENTIFY HEADER
~ 323
SELECT LABEL
! 324 COUPLE
LABEL TO A.C. ELEM.
i 325 DETERMINE
OUTPUT INTERFACE
I
FIG. 3
US 6,377,577 Bl
l 326 DETERMINE
INPUT PERMISSION
l 327 COUPLE LABEL
& OIFTO OUTPUT A. C.
~ 328 DETERMINE
OlfTPUT PERMISSION
~READY TO MIT TRANS
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page7 of 13
US 6,377,577 Bl 1
ACCESS CONTROL LIST PROCESSING IN HARDWARE
2 these applications is hereby incorporated by reference as if fully set forth herein.
In a computer network for transmitting information, messages can be restricted from being transmitted from 5 selected source devices to selected destination devices. In
While netflow switching achieves the goal of improving the speed of enforcing access control by the router, it still has the drawback that comparing at least some incoming packets against the ACL must be performed using software. Thus,
known computer networks, this form of restriction is known as "access control" and is performed by routers, which route messages (in the form of individual packets of information) from source devices to destination devices. One known technique for access control is for each router to perform access control by reference to one or more ACLs (access control lists); the ACL describes which selected source devices are permitted (and which denied) to send packets to which selected destination devices.
In a known standard for ACL format, each ACL includes a plurality of access control specifiers, each of which selects a range of sender and destination IP address prefix or subnet, and port, and provides that packet transmission from that selected set of senders to that selected set of destinations is either specifically permitted or specifically denied. ACLs are associated with input interfaces and independently with output interfaces for each router. In known routers such as those manufactured by Cisco Systems, Inc., of San Jose, Calif., the router is provided with an ACL using an ACL command language, interpreted by operating system software for the router, such as the lOS operating system.
One problem in the known art is that processing of packets to enforce access control according to the ACL is processor-intensive and can therefore be relatively slow, particularly in comparison with desired rates of speed for routing packets. This problem is exacerbated when access control is enforced for packets using software in the router, because software processing of the ACL can be quite slow relative to hardware processing of the packet for routing.
One known solution is to reduce the number of packets for which access control requires actual access to the ACL. In a technique known as "netflow switching," packets are identified as belonging to selected "flows," and each packet
the relative slowness required by software processing of the ACL is not completely avoided.
A second problem in the known art is that software
10 processing of the ACL takes increased time when the ACL has numerous entries, such as when the requirements for access control are complex. The more entries in the ACL, the more time is expected to be required for software processing of the ACL, and thus the more time is expected to be required for software enforcement of access control. Since
15 known routers require at least some software enforcement of access control, this reduces the routing speed at which the router can operate.
For example, for some large ACLs, routing speed can be reduced to as low as about 10,000 packets per second.
20 However, the wirespeed rate of incoming packets is presently (for relatively short packets) about 1.5 million packets per gigabit per second transmission capacity, or in the range of about tens to hundreds of millions of packets per second for gigabit networks. Since it would be desirable for routers
25 to operate at speeds comparable to the wirespeed, the present limitation on router speed is unacceptably low.
Accordingly, it would be desirable to provide a method and system for hardware processing of ACLs and thus hardware enforcement of access control. This advantage is
30 achieved in an embodiment of the invention in which a sequence of access control specifiers from an ACL are recorded in a CAM (content-addressable memory), and in which matching (or lack of matching) of information from the packet header to specifiers recorded in the CAM are used
35 to enforce access control.
SUMMARY OF THE INVENTION
in a flow is expected to have identical routing and access 40 control characteristics. Therefore, access control only requires reference to the ACL for the first packet in a flow; subsequent packets in the same flow can have access control enforced identically to the first packet, by reference to a routing result cached by the router and used for the entire 45 flow.
The invention provides a method and system for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the
Netflow switching is further described in detail in the following patent applications:
U.S. application Ser. No. 08/581,134, titled "Method For Traffic Management, Traffic Prioritization, Access Control, and Packet Forwarding in a Datagram Computer Network", filed Dec. 29, 1995, in the name of inventors David R.
Cheriton and Andreas V. Bechtolsheim, assigned to Cisco Technology, Inc., attorney docket number CIS-019;
U.S. application Ser. No. 08/655,429, titled "Network Flow Switching and Flow Data Export", filed May 28, 1996, in the name of inventors Darren Kerr and Barry Bruins, and assigned to Cisco Technology, Inc., attorney docket number CIS-016; and
U.S. application Ser. No. 08/771,438, titled "Network Flow Switching and Flow Data Export", filed Dec. 20, 1996, in the name of inventors Darren Kerr and Barry Bruins, assigned to Cisco Technology, Inc., attorney docket number CIS-017.
These patent applications are collectively referred to herein as the "Netflow Switching Disclosures". Each of
sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a
50 rate comparable to wirespeed. In a preferred embodiment, the CAM includes an ordered
sequence of entries, each of which has an array of ternary elements for matching on logical "0", logical "1", or on any value, and each of which generates a match signal. The ACL
55 entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier.
A router including the CAM can also include preprocess-60 ing circuits for certain range comparisons which have been
found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM. For example, comparisons of the port number against known special cases, such as "greater than 1023" and "within the
65 range 6000 to 6500", can be treated by circuitry for performing range comparisons or by reference to one or more auxiliary CAMs.
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page8 of 13
US 6,377,577 Bl 3
The invention can also be used to augment or override routing decisions otherwise made by the router, so as to implement QOS (quality of service), and other administrative policies, using the CAM.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a block diagram of a system for access control list processing.
FIG. 2 shows a block diagram of an access control element.
FIG. 3 shows a flow diagram of a method for access control list processing in hardware.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
In the following description, a preferred embodiment of the invention is described with regard to preferred process steps and data structures. Those skilled in the art would recognize after perusal of this application that embodiments of the invention can be implemented using circuits adapted to particular process steps and data structures described herein, and that implementation of the process steps and data structures described herein would not require undue experimentation or further invention. System Elements
FIG. 1 shows a block diagram of a system for access control list processing.
A system 100 includes a set of packet input interfaces 101, a routing element 10, an access control element 120, and a set of packet output interfaces 102. The system 100 receives packets 130 at the input interfaces 101; each packet 130 indicates a source device 131, from which it was sent, and a destination device 132, to which it is intended to go. The routing element 110 processes each packet 130 to select one or more of the output interfaces 102 to which the packet 130 should be forwarded. The access control element 120 determines if the packet 130 has permission to be forwarded from its source device 131 to its destination device 132. Each packet 130 that has permission to be forwarded is output to its selected output interfaces 102.
In a first set of alternative embodiments, the system 100 may include a plurality of access control elements 120 operating in parallel in place of the single access control element 120.
In a second set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to the input interfaces 101 and operating to determine if packets 130 have permission to be forwarded from their source devices 131 at all. The access control element 120 is shown coupled to the routing element 110 to perform access control after a routing decision has been made. However, the access control element 120 is still capable of denying access to packets 130 responsive to whether they have permission to be forwarded from their source devices 131 at all.
In a third set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to individual input interfaces 101 and operating to make access control determinations for packets 130 arriving
5
4 In a preferred embodiment, the access control element
120 operates on a set of selected elements of a packet header 133 for each packet 130. The system 100 collects the selected elements into a packet label 200.
In a preferred embodiment using netfiow switching, the packet label 200 used for access control at the input interfaces 101 includes a source device 131, the destination device 132, a port identifier for a port at the source device 131, a port identifier for a port at the destination device 132,
10 and a protocol type. In alternative embodiments, the packet label200 may be any collection of information derived from the packet 130 (preferably from the packet header 133) used for access control.
The concept of preprocessing the packet label has wide applicability, including determining other routing inform a-
15 tion in response to data in the packet header. For example, in addition to or instead of comparing data in the packet header against known special cases, such as "greater than 1023" and "within the range 6000 to 6500," preprocessing can include performing logical or arithmetic operations on
20 data in the packet header. Preprocessing can also include data lookup, or substituting new data, in response to data in the packet header.
The access control element 120 includes an input port 201 coupled to the packet label 200, an access control memory
25 210, a priority encoder 220, and an output port 202 coupled to the priority encoder 220.
When the access control element 120 is disposed for controlling access for packets responsive to their input interfaces 101, the packet label200 includes an identifier for
30 the input interface 101. When the access control element 120 is disposed for controlling access for packets responsive to their output interfaces 102, the packet label 200 includes an identifier for the output interface 102.
The access control memory 210 includes a CAM 35 (content-addressable memory) having a sequence of access
control specifiers 211. Each access control specifier 211 includes a label match mask 212 and a label match pattern 213. For each access control specifier 211, each bit of the label match mask 212 determines whether or not a corre-
40 sponding bit of the packet label 200 is tested. If so, the corresponding bit of the label match pattern 213 is compared for equality with the corresponding bit of the packet label 200. If all compared bits are equal, the access control specifier 211 matches the packet label200. Bits that are not
45 compared have no effect on whether the access control specifier 211 is considered to match the packet label 200 or not.
The priority encoder 220 is coupled to all of the access control specifiers 211, and receives an indicator from each
50 one whether or not that access control specifier 211 matched the packet label 200. The priority encoder 220 selects the single access control specifier 211 with the highest priority (in a preferred embodiment, the one with the lowest address in the access control memory 210) and provides an indicator
55 of that single access control specifier 211 to the output port 202.
The indicator provided to the output port 202 specifies
at particular input interfaces 101. Similarly, the system 100 60
may include one or more access control elements 120 coupled to individual output interfaces 102 and operating to make access control determinations for packets 130 forwarded to particular output interfaces 102.
whether or not the packet 130 has permission to be forwarded from its specified source device 131 to its specified destination device 132. In a preferred embodiment, the indicator specifies one of three possibilities: (a) the packet 130 is forwarded to its calculated output interface and on to its specified destination device 132; (b) the packet 130 is dropped; or (c) the packet 130 is forwarded to a "higher-
Access Control Element FIG. 2 shows a block diagram of an access control
element.
65 level" processor for further treatment. When a packet 130 is dropped it is effectively denied access from its specified source device 131 to its specified destination device 132.
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page9 of 13
US 6,377,577 Bl 5 6
number with these known ranges and provides a set of comparison bits 231 indicating whether or not the source port number and the destination port number are within each specified range. The comparison circuit 230 includes a finite
The higher-level processor includes a general-purpose processor, program and data memory, and mass storage, executing operating system and application software for software (rather than hardware) examination of the packet 130. The packet 130 is compared, possibly to the access control specifiers 211 and possibly to other administrative policies or restrictions, by the higher-level processor. The higher-level processor specifies whether the packet 130, after processing by the higher-level processor, is forwarded to a selected output interface or is dropped.
5 state machine 232 (or other element) for storing lower and upper bounds for each specified range. The comparison bits 231 are coupled to the input port 201 of the access control element 120 for treatment as matchable input bits supplemental to the header of the packet 130.
Access Control Lists 10
In various embodiments, the invention can be used to augment or override routing decisions otherwise made by the router, using the access control element 120. In addition to specifying that the packet 130 is to be dropped or forwarded to the higher-level processor, the access control element 120 can alter the output interface, which was
A Cisco access control list includes a sequence of access control entries, which are mapped to a set of access control specifiers 211. Each access control entry has a structure according to the following syntax:
access-list access-list-number [dynamic dynamic-name [timeout minutes]] { denylpermit} protocol source source-wildcard [operator port [port]] destination destination-wildcard [operator port [port]] [established] [precedence precedence] [tos tos] [log]
15 selected by the routing element 110, to another selected output interface. The invention can thus be used to implement QOS (quality of service) policies and other administrative policies.
This syntax, its meaning, and access control entries in 20
general, are further described in documentation for Cisco lOS software, available from Cisco Systems, Inc., in San Jose, Calif., and hereby incorporated by reference as if fully set forth herein.
Method of Operation FIG. 3 shows a flow diagram of a method for access
control list processing in hardware. A method 300 includes a set of flow points to be noted,
and steps to be executed, cooperatively by the elements of the system 100.
At a flow point 310, a packet is received at one of the packet input inter-faces 101.
At a step 321, the routing element 110 receives an input packet 130.
Access control entries can specify that particular actions 25
are permitted, denied, or that they will be recorded in a log. Access control entries are interpreted sequentially. Thus, an earlier more specific access control entry can prohibit particular actions (such as receiving messages from a particular sending device), while a later more general access control entry can permit the same actions for other devices (such as other sending devices in the same network).
At a step 322, the routing element 110 identifies the
30 header for the packet 130.
At a step 323, the routing element 110 selects portions of the header for use as the packet label 200 for access control. In a preferred embodiment, the packet label 200 used for access control at the input interfaces 101 includes the source device 131, the destination device 132, the port identifier at
When an access control list is translated for entry into the access control memory, it is optimized to reduce the number of separate entries that are used. Thus, an access control list with N separate access control entries is translated into a set of access control specifiers 211 that can be smaller or larger than N, depending on the effect of optimization.
35 the source device 131, the port identifier at the destination device 132, and a protocol type.
A first optimization detects separate access control entries that each refer to a special case of a more general access 40 control specifier 211, such as in one of the following cases:
At a step 324, the routing element 110 couples the packet label 200 and an input interface specifier to the input access control element 120.
At a step 325, the routing element 10 determines a selected output inter-face for the packet 130.
A first access control entry provides a selected permission for a selected source device 131 2S, and a second access control entry provides the same permission for a selected source device 1312S+l. The first and second access control entries can be translated into a single more general access control specifier 211 with an unmatched bit in the 2° position.
At a step 326, preferably performed in parallel with the step 325, the input access control element 120 determines the input permission for the packet 130, that is, whether the
45 routing element 110 permits forwarding the packet 130 from the source device 131 for the packet 130.
The step 326 includes matching the packet label 200 against the access control memory 210 for the input access control element 120, determining all of the successful A set of access control entries each provides the same
selected permission for a range of selected source devices 131 S through T, and the rangeS through T can be represented as a smaller number of bit strings with unmatched bits.
50 matches, coupling the successful matches to the priority encoder 220 for the input access control element 120, determining the highest-priority match, and providing an output result from the input access control element 120.
A set of access control entries provides a selected permission for a comparison of source device 131 55
addresses with a test value V. A second optimization detects range comparisons that
have been found to be particularly common. For example, it is common to compare the source or destination port number for being greater than 1023, or for being within the range 60
6000 to 6500. To compare the source or destination port number for being greater than 1023 with matched and unmatched bits would use about six entries for each such comparison (to test each one of the six high-order bits of the port number for being logical "1"). 65
In a preferred embodiment, a comparison circuit 230 compares the source port number and the destination port
If at the step 326, the input access control element 120 determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the input access control element 120.
If at the step 326, the input access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.
At a step 327, the routing element 110 couples the packet label 200 and the output interface specifier to the output access control element 120.
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page10 of 13
US 6,377,577 Bl 7
At a step 328, the output access control element 120 determines the output permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 to the destination device 132 for the packet 130.
The step 326 includes the following actions: matching the packet label 200 against the access control
memory 210 for the out-put access control element 120;
determining all of the successful matches; coupling the successful matches to the priority encoder
220 for the output access control element 120; determining the highest-priority match; and providing an output result from the output access control
element 120. If at the step 328, the output access control element 120
determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the output access control element 120.
If at the step 328, the output access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to
8 each row being associated with a pattern of bits not for
matching, said set of patterns of bits not for matching being fewer than a number of said rows.
7. A method as in claim 1, wherein said associative 5 memory includes a ternary content-associative memory.
8. A method as in claim 1, wherein said packet label includes a source IP address or subnet, a destination IP address or subnet, a source port, a destination port, a protocol specifier, or an input interface.
10 9. A method as in claim 1, wherein said priority informa-
tion for each said access control pattern is responsive to a position of said access control pattern in a memory.
10. A method as in claim 1, wherein said priority infor-15 mation includes a position in said associative memory, and
said step of selecting includes choosing a first one of said matches.
20
11. A method as in claim 1, wherein said routing decision includes a committed access rate decision.
12. A method as in claim 1, wherein said routing decision includes an administrative policy decision regarding treatment of said packet.
the packet 130. 25
At a flow point 330, the packet is ready for transmission
13. A method as in claim 1, wherein said routing decision includes determining an output interface for said packet.
14. A method as in claim 1, wherein said routing decision includes implementing a quality of service policy. to one of the packet output interfaces 102.
Alternative Embodiments Although preferred embodiments are disclosed herein,
many variations are possible which remain within the concept, scope, and spirit of the invention, and these variations would become clear to those skilled in the art after perusal of this application.
What is claimed is: 1. A method, including the steps of maintaining a set of
access control patterns in at least one associative memory; receiving a packet label responsive to a packet, said
packet label being sufficient to perform access control processing for said packet;
matching matchable information, said matchable information being responsive to said packet label, with said set of access control patterns in parallel, and generating a set of matches in response thereto, each said match having priority information associated therewith;
selecting at least one of said matches in response to said priority information, and generating an access result in response to said at least one selected match; and
making a outing-decision in response to said access result. 2. A method as in claim 1, including the step of perform
ing at least two of said steps of receiving, matching, selecting, and making a routing decision, in parallel using a pipeline technique.
3. A method as in claim 1, wherein said access control patterns each include a bit pattern for matching and a mask pattern of bits not for matching.
4. A method as in claim 1, wherein said access control patterns each include a set of ternary elements, each representative of a logical "0," logical "1", or "don't care" value.
15. A method as in claim 1, wherein said routing decision includes permitting or denying access for said packet.
16. A method as in claim 1, wherein said step of gener-30 ating said access result is responsive to a plurality of said at
least one matches. 17. A method as in claim 1, wherein said step of matching
is performed in order of constant time, whereby said step of
35 matching is performed in time not responsive to a number of said access control patterns.
18. A method as in claim 1, wherein said steps of matching and selecting are performed at a rate exceeding 1 megapacket per second.
40 19. A method as in claim 1, including the step of making
a preliminary routing decision for said packet, wherein said packet routing information includes a result of said preliminary routing decision.
20. A method as in claim 19, wherein said preliminary
45 routing decision includes determining at least one output interface for said packet.
21. A method as in claim 19, wherein said packet routing information includes an output interface for said packet.
22. A method as in claim 1, including the step of prepro-50 cessing said packet label to generate said matchable infor
mation.
55
23. A method as in claim 22, wherein said step of preprocessing includes the steps of
performing an arithmetic, logical, or comparison operation on said packet label; and
generating a bit string for said matchable information in response to said arithmetic, logical, or comparison operation.
24. A method as in claim 22, wherein said step of preprocessing includes the step of comparing a field of said packet label with an arithmetic range or mask value.
5. A method as in claim 1, wherein said associative memory includes a hardware content-associative memory 60
having a plurality of rows, each row including one of said access control patterns and one of said access results. 25. A method as in claim 22, wherein said step of
preprocessing includes the step of comparing a source IP port value or a destination IP port value with a selected port
65 value.
6. A method as in claim 1, wherein said associative memory includes a hardware content-associative memory having a plurality of rows,
each row including a bit pattern for matching and one of said access results, and
26. A method as in claim 1, including the step of postprocessing said selected match to generate said access result.
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page11 of 13
US 6,377,577 Bl 9
27. A method as in claim 26, wherein said step of postprocessing includes accessing a memory in response to a bitstring included in said selected match.
10 storing said sequence of access control patterns in said
associative memory.
30. A method as in claim 29, wherein said step of translating includes the step of generating a plurality of said access control patterns in response to one of said access control specifiers.
28. A method as in claim 1, wherein said set of access control patterns is responsive to a sequence of access control s specifiers, each one of said sequence of access control specifiers declaring whether to permit or deny access for a set of packets. 31. A method as in claim 29, wherein said step of
translating includes the step of generating a single one of 10 said access control patterns in response to a plurality of said
29. A method as in claim 28, wherein said step of maintaining includes the steps of
receiving said sequence of access control specifiers;
translating said sequence of access control specifiers into said sequence of access control patterns; and
access control specifiers.
* * * * *
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page12 of 13
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. : 6,377,577 B1 Page 1 of 1 DATED : Apri123, 2002 INVENTOR(S) : Bechtolsheim et al.
It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:
Column 7, Line 48, please delete "outing-decision" and insert therefore-- routing decision--.
Signed and Sealed this
Twelfth Day of August, 2003
JAMES E. ROGAN Director of the United States Patent and Trademark Office
Case3:14-cv-05343 Document1-2 Filed12/05/14 Page13 of 13
Exhibit 3
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page1 of 12
c12) United States Patent Bechtolsheim et al.
(54) ACCESS CONTROL LIST PROCESSING IN HARDWARE
(75) Inventors: Andreas V. Bechtolsheim, Stanford, CA (US); David R. Cheriton, Palo Alto, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 297 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 10/087,342
(22) Filed: Mar. 1, 2002
Related U.S. Application Data
(63) Continuation of application No. 09/108,071, filed on Jun. 30, 1998, now Pat. No. 6,377,577.
(51) Int. Cl. H04L 12128 (2006.01)
(52) U.S. Cl. ...................................................... 370/392 (58) Field of Classification Search ................ 370/392,
(56)
370/393,394,396,397,398,389,400,399, 370/395.32; 709/220, 221, 222, 227, 228,
709/229; 711/108 See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
5,386,413 A * 5,414,704 A * 5,509,006 A * 5,920,886 A * 5,938,736 A *
111995 McAuley eta!. ........... 370/392 5/1995 Spinney ...................... 370/389 4/1996 Wilford eta!. ............. 370/401 7/1999 Feldmeier ................... 7111108 8/1999 Muller et a!. ............... 709/243
100"
PACKET INPUT
INTERFACES
111111 1111111111111111111111111111111111111111111111111111111111111 US007023 853B 1
(10) Patent No.: US 7,023,853 Bl *Apr. 4, 2006 (45) Date of Patent:
OTHER PUBLICATIONS
Alessandri, Access Control List Processing in Hardware, Diploma Thesis, pp. 1-85, Oct. 1997.* Miei et a!, Parallelization of IP-Packet Filter Rules, IEEE, pp. 381-388, 1997.*
(Continued)
Primary Examiner-Frank Duong (74) Attorney, Agent, or Firm--Campbell Stephenson Ascolese LLP
(57) ABSTRACT
The invention provides for hardware processing of ACLs and thus hardware enforcement of access control. A sequence of access control specifiers from an ACL are recorded in a CAM, and information from the packet header is used to attempt to match selected source and destination IP addresses or subnets, ports, and protocols, against all the ACL specifiers at once. Successful matches are input to a priority selector, which selects the match with the highest priority (that is, the match that is first in the sequence of access control specifiers). The specified result of the selected match is used to permit or deny access for the packet without need for software processing, preferably at a rate comparable to wirespeed. The CAM includes an ordered sequence of entries, each of which has an array of ternary-elements for matching "0", "1 ", or any value, and each of which generates a match signal. The ACL entered for recording in the CAM can be optimized to reduce the number of separate entries in the CAM, such as by combining entries which are each special cases of a more general access control specifier. A router including the CAM can also include preprocessing circuits for certain range comparisons which have been found both to be particularly common and to be otherwise inefficiently represented by the ternary nature of the CAM, such as comparisons of the port number against known special cases such as "greater than 1023" or "within the range 6000 to 6500".
63 Claims, 3 Drawing Sheets
PACKET OUTPUT
INTERFACES
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page2 of 12
US 7,023,853 Bl Page 2
OTHER PUBLICATIONS
McAuley et a!, Fast Routing Table Lookup Using CAMs, Bellcore, pp. 1-10, 1993.* Doeringer et a!, Routing on Longest-Matching Prefixes, IEEE, pp. 86-97, 1996.*
Shaffer, Designing Very Large Content-Addressable Memories, University of Pennsylvania, pp. 1-38, 1992. * Molitor, Architecture for Advanced Packet Filtering, USENIX UNIX Security Symposium, pp. 1-13, 1995.*
* cited by examiner
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page3 of 12
100"'
• • •
PACKET INPUT
INTERFACES
/
110
ROUTING ELEMENT
FIG. 1
ACCESS CONTROL ELEMENT '\ •
• •
PACKET OUTPUT
INTERFACES
e • 00 • ~ ~ ~ ~ = ~
~ :-: ~ ... N 0 0 0\
rFJ
=-('D ('D ..... .... 0 ..... (.H
d rJl
"'--...1 = N w Oo u. w
= """"'
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page4 of 12
U.S. Patent Apr. 4, 2006
COMPARE CIRCUIT
230
COMPARE BITS 231
212MASK
ACCESS CONTROL MEMORY
210
Sheet 2 of 3
FIG. 2
US 7,023,853 Bl
OUTPUT PORT 202
PRIORITY ENCODER
220
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page5 of 12
U.S. Patent Apr. 4, 2006
300"'
RECEIVE YREADYTO
321 RECEIVE PACKET
! 322 - IDENTIFY
HEADER
! ~
SELECT LABEL
l 324 COUPLE
LABEL TO A. C. ELEM.
~ 325 DETERMINE
OlffPUT INTERFACE
I
Sheet 3 of 3 US 7,023,853 Bl
FIG. 3
J !l2Q DETERMINE
INPUT PERMISSION
~ 327 COUPLE LABEL
& OIFTO Olf{PUT A. G.
~ 328 DETERMINE
OUTPUT PERMISSION
~~M£s YTO MIT
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page6 of 12
US 7,023,853 Bl 1
ACCESS CONTROL LIST PROCESSING IN HARDWARE
Continuation of prior application Ser. No. 09/108,071 filed on Jun. 30, 1998 now U.S. Pat. No. 6,377,577 entitled:
2 1996, in the name of inventors Darren Kerr and Barry Bruins, and assigned to Cisco Technology, Inc.; and
U.S. application Ser. No. 08/771,438, titled "Network Flow Switching and Flow Data Export", filed Dec. 20, 1996, in the name of inventors Darren Kerr and Barry Bruins, assigned to Cisco Technology, Inc. ACCESS CONTROL LIST PROCESSING IN HARD
WARE.
BACKGROUND OF THE INVENTION
These patent applications are collectively referred to herein as the "Netflow Switching Disclosures". Each of these applications is hereby incorporated by reference as if
10 fully set forth herein. 1. Field of the Invention This invention relates to access control list processing. 2. Related Art
While netflow switching achieves the goal of improving the speed of enforcing access control by the router, it still has the drawback that comparing at least some incoming packets against the ACL must be performed using software. Thus,
15 the relative slowness required by software processing of the ACL is not completely avoided.
In a computer network for transmitting information, messages can be restricted from being transmitted from selected source devices to selected destination devices. In known computer networks, this form of restriction is known as "access control" and is performed by routers, which route messages (in the form of individual packets of information) from source devices to destination devices. One known 20
technique for access control is for each router to perform access control by reference to one or more ACLs (access control lists); the ACL describes which selected source devices are permitted (and which denied) to send packets to which selected destination devices. 25
In a known standard for ACL format, each ACL includes a plurality of access control specifiers, each of which selects a range of sender and destination IP address prefix or subnet, and port, and provides that packet transmission from that
30 selected set of senders to that selected set of destinations is either specifically permitted or specifically denied. ACLs are associated with input interfaces and independently with output interfaces for each router. In known routers such as those manufactured by Cisco Systems, Inc., of San Jose,
35 Calif., the router is provided with an ACL using an ACL command language, interpreted by operating system software for the router, such as the lOS operating system.
One problem in the known art is that processing of packets to enforce access control according to the ACL is
40 processor-intensive and can therefore be relatively slow, particularly in comparison with desired rates of speed for routing packets. This problem is exacerbated when access control is enforced for packets using software in the router, because software processing of the ACL can be quite slow
45 relative to hardware processing of the packet for routing.
One known solution is to reduce the number of packets for which access control requires actual access to the ACL. In a technique known as "netflow switching," packets are identified as belonging to selected "flows," and each packet 50 in a flow is expected to have identical routing and access control characteristics. Therefore, access control only requires reference to the ACL for the first packet in a flow; subsequent packets in the same flow can have access control enforced identically to the first packet, by reference to a 55 routing result cached by the router and used for the entire flow.
Netflow switching is further described in detail in the following patent applications:
SUMMARY OF THE INVENTION
The invention provides a method for processing of access control lists and thus enforcement of access control. A plurality of access control specifiers are configured in an access control element according to the priority of the type of each access control specifier, one or more characteristics of a packet are matched with the access control specifiers, one of the matches is selected according to the access control specifier with the highest associated priority, and the selected packet is processed. Types of access control specifiers correspond to the information in an access control entry. In one aspect of the present invention, an access control element is a content addressable memory. In another aspect of the present invention, the matching and processing are performed in parallel. In a further aspect of the present invention, the characteristics of the packet include one or more of a source address, a destination address, a source port, a destination port, a protocol type, an input interface and an output interface. In another aspect of the present invention, one or more of the access control specifiers are identified based on the matching and the identified access control specifiers are prioritized based on the matching.
The present invention further provides a system for processing a packet. The system includes one or more access control specifiers, an access control element, and a priority encoder. The access control specifiers are have a plurality of types and those types are related to information in an access control entry. The access control element is configured to store the access control specifiers according to the priority of each access control specifier and to match one or more characteristics of a packet with the access control specifiers. The priority encoder is configured to select the highest priority match from among the matches. In one aspect of the present invention, the access control specifier further includes a label match mask for determining whether a first bit of the packet characteristics is tested and a label match pattern for comparing to a second bit of the packet characteristics. In another aspect of the present invention, a pro-cessor is coupled to the access control element to process a packet not otherwise processed by the access control element.
The present invention also provides a system for process-ing a packet that includes a means for configuring a plurality of access control specifiers in an access control element according to a priority of a type of each access control specifier, a means for matching one or more characteristics
U.S. application Ser. No. 08/581,134, titled "Method For 60
Traffic Management, Traffic Prioritization, Access Control, and Packet Forwarding in a Datagram Computer Network", filed Dec. 29, 1995, in the name of inventors David R. Cheriton and Andreas V. Bechtolsheim, assigned to Cisco Technology, Inc.; 65 of the packet with one or more of the access control
specifiers, and a means for processing the packet based on the matching.
U.S. application Ser. No. 08/655,429, titled "Network Flow Switching and Flow Data Export", filed May 28,
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page7 of 12
US 7,023,853 Bl 3
The present invention further provides a system that includes a means for maintaining a set of access control patterns, means for receiving a packet label responsible to a packet, means for matching matchable information responsive to the packet and the access control patterns, means for generating a set of matches in response to the means for matching, means for selecting at least one of the matches in response to priority information in the set of matches and generating an access result in response thereto, and a means for making a routing decision based on the access result.
The present invention also provides a method for processing a packet that includes selecting an output interface
4 access control after a routing decision has been made. However, the access control element 120 is still capable of denying access to packets 130 responsive to whether they have permission to be forwarded from their source devices 131 at all.
In a third set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to individual input interfaces 101 and operating to make access control determinations for packets 0.130 arriv-
10 ing at particular input interfaces 101. Similarly, the system 100 may include one or more access control elements 120 coupled to individual output interfaces 102 and operating to make access control determinations for packets 130 forwarded to particular output interfaces 102.
to which to forward the packet, determining forwarding permission for the packet by matching one or more characteristics of the packet with one or more access control 15
specifiers, and processing the packet based on the forwarding permission, wherein the selecting and the determining are performed in parallel.
Access Control Element FIG. 2 shows a block diagram of an access control
element.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a block diagram of a system for access control list processing.
FIG. 2 shows a block diagram of an access control element.
FIG. 3 shows a flow diagram of a method for access control list processing in hardware.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
In the following description, a preferred embodiment of the invention is described with regard to preferred process steps and data structures. Those skilled in the art would recognize after perusal of this application that embodiments of the invention can be implemented using circuits adapted to particular process steps and data structures described herein, and that implementation of the process steps and data structures described herein would not require undue experimentation or further invention.
System Elements FIG. 1 shows a block diagram of a system for access
control list processing.
In a preferred embodiment, the access control element 20 120 operates on a set of selected elements of a packet header
133 for each packet 130. The system 100 collects the selected elements into a packet label 200.
In a preferred embodiment using netflow switching, the packet label 200 used for access control at the input inter-
25 faces 101 includes a source device 131, the destination device 132, a port identifier for a port at the source device 131, a port identifier for a port at the destination device 132, and a protocol type. In alternative embodiments, the packet label 200 may be any collection of information derived from
30 the packet 130 (preferably from the packet header 133) used for access control.
The concept of preprocessing the packet label has wide applicability, including determining other routing information in response to data in the packet header. For example,
35 in addition to or instead of comparing data in the packet header against known special cases, such as "greater than 1023" and "within the range 6000 to 6500," preprocessing can include performing logical or arithmetic operations on data in the packet header. Preprocessing can also include
40 data lookup, or substituting new data, in response to data in the packet header.
A system 100 includes a set of packet input interfaces 101, 45 a routing element 110, an access control element 120, and a
The access control element 120 includes an input port 201 coupled to the packet label 200, an access control memory 210, a priority encoder 220, and an output port 202 coupled to the priority encoder 220.
set of packet output interfaces 102. The system 100 receives packets 130 at the input interfaces 101; each packet 130 indicates a source device 131, from which it was sent, and a destination device 132, to which it is intended to go. The 50 routing element 110 processes each packet 130 to select one or more of the output interfaces 102 to which the packet 130 should be forwarded. The access control element 120 deter-mines if the packet 130 has permission to be forwarded from its source device 131 to its destination device 132. Each 55 packet 130 that has permission to be forwarded is output to its selected output interfaces 102.
In a first set of alternative embodiments, the system 100 may include a plurality of access control elements 120 operating in parallel in place of the single access control 60
element 120.
When the access control element 120 is disposed for controlling access for packets responsive to their input interfaces 101, the packet label200 includes an identifier for the input interface 101. When the access control element 120 is disposed for controlling access for packets responsive to their output interfaces 102, the packet label 200 includes an identifier for the output interface 102.
The access control memory 210 includes a CAM (content-addressable memory) having a sequence of access control specifiers 211. Each access control specifier 211 includes a label match mask 212 and a label match pattern 213. For each access control specifier 211, each bit of the label match mask 212 determines whether or not a corre-sponding bit of the packet label 200 is tested. If so, the corresponding bit of the label match pattern 213 is compared for equality with the corresponding bit of the packet label 200. If all compared bits are equal, the access control specifier 211 matches the packet label 200. Bits that are not
In a second set of alternative embodiments, the system 100 may include one or more access control elements 120 coupled to the input interfaces 101 and operating to determine if packets 130 have permission to be forwarded from their source devices 131 at all. The access control element 120 is shown coupled to the routing element 110 to perform
65 compared have no effect on whether the access control specifier 211 is considered to match the packet label 200 or not.
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page8 of 12
US 7,023,853 Bl 5
The priority encoder 220 is coupled to all of the access control specifiers 211, and receives an indicator from each one whether or not that access control specifier 211 matched the packet label 200. The priority encoder 220 selects the single access control specifier 211 with the highest priority (in a preferred embodiment, the one with the lowest address in the access control memory 210) and provides an indicator of that single access control specifier 211 to the output port 202.
6 access control entries can be translated into a single more general access control specifier 211 with an Uflllatched bit in the 2° position.
A set of access control entries each provides the same selected permission for a range of selected source devices 131 S through T, and the rangeS through T can be represented as a smaller number of bit strings with Uflllatched bits.
A set of access control entries provides a selected permission for a comparison of source device 131 addresses with a test value V.
A second optimization detects range comparisons that have been found to be particularly common. For example, it is common to compare the source or destination port number for being greater than 1023, or for being within the range 6000 to 6500. To compare the source or destination port number for being greater than 1023 with matched and uumatched bits would use about six entries for each such comparison (to test each one of the six high-order bits of the
The indicator provided to the output port 202 specifies 10
whether or not the packet 130 has permission to be forwarded from its specified source device 131 to its specified destination device 132. In a preferred embodiment, the indicator specifies one of three possibilities: (a) the packet 130 is forwarded to its calculated output interface and on to 15
its specified destination device 132; (b) the packet 130 is dropped; or (c) the packet 130 is forwarded to a "higherlevel" processor for further treatment. When a packet 130 is dropped it is effectively denied access from its specified source device 131 to its specified destination device 132.
The higher-level processor includes a general-purpose processor, program and data memory, and mass storage, executing operating system and application software for software (rather than hardware) examination of the packet 130. The packet 130 is compared, possibly to the access 25
control specifiers 211 and possibly to other administrative policies or restrictions, by the higher-level processor. The higher-level processor specifies whether the packet 130, after processing by the higher-level processor, is forwarded
20 port number for being logical "1"). In a preferred embodiment, a comparison circuit 230
compares the source port number and the destination port number with these known ranges and provides a set of comparison bits 231 indicating whether or not the source port number and the destination port number are within each specified range. The comparison circuit 230 includes a finite
to a selected output interface or is dropped.
Access Control Lists
state machine 232 (or other element) for storing lower and upper bounds for each specified range. The comparison bits 231 are coupled to the input port 201 of the access control
30 element 120 for treatment as matchable input bits supplemental to the header of the packet 130.
In various embodiments, the invention can be used to augment or override routing decisions otherwise made by
A Cisco access control list includes a sequence of access control entries, which are mapped to a set of access control specifiers 211. Each access control entry has a structure according to the following syntax:
access-list access-list-number [dynamic dynamic-name [timeout minutes]] { denylpermit} protocol source source-wildcard [operator port [port]] destination destination-wildcard [operator port [port]] [established] [precedence precedence] [tos tos] [log]
35 the router, using the access control element 120. In addition to specifying that the packet 130 is to be dropped or forwarded to the higher-level processor, the access control element 120 can alter the output interface, which was selected by the routing element 110, to another selected
40 output interface. The invention can thus be used to implement QOS (quality of service) policies and other administrative policies. This syntax, its meaning, and access control entries in
general, are further described in documentation for Cisco lOS software, available from Cisco Systems, Inc., in San Jose, Calif., and hereby incorporated by reference as if fully 45 set forth herein.
Access control entries can specifY that particular actions are permitted, denied, or that they will be recorded in a log. Access control entries are interpreted sequentially. Thus, an earlier more specific access control entry can prohibit par- 50
ticular actions (such as receiving messages from a particular sending device), while a later more general access control entry can permit the same actions for other devices (such as other sending devices in the same network).
When an access control list is translated for entry into the 55
access control memory, it is optimized to reduce the number of separate entries that are used. Thus, an access control list with N separate access control entries is translated into a set
Method of Operation FIG. 3 shows a flow diagram of a method for access
control list processing in hardware. A method 300 includes a set of flow points to be noted,
and steps to be executed, cooperatively by the elements of the system 100.
At a flow point 310, a packet is received at one of the packet input interfaces 101.
At a step 321, the routing element 110 receives an input packet 130.
At a step 322, the routing element 110 identifies the header for the packet 130.
At a step 323, the routing element 110 selects portions of the header for use as the packet label 200 for access control. In a preferred embodiment, the packet label 200 used for access control at the input interfaces 101 includes the source of access control specifiers 211 that can be smaller or larger
than N, depending on the effect of optimization. A first optimization detects separate access control entries
that each refer to a special case of a more general access control specifier 211, such as in one of the following cases:
60 device 131, the destination device 132, the port identifier at the source device 131, the port identifier at the destination device 132, and a protocol type.
A first access control entry provides a selected permission for a selected source device 131 2S, and a second 65
access control entry provides the same permission for a selected source device 131 2S+ 1. The first and second
At a step 324, the routing element 110 couples the packet label 200 and an input interface specifier to the input access control element 120.
At a step 325, the routing element 110 determines a selected output interface for the packet 130.
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page9 of 12
US 7,023,853 Bl 7
At a step 326, preferably performed in parallel with the step 325, the input access control element 120 determines the input permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 from the source device 131 for the packet 130.
The step 326 includes matching the packet label 200 against the access control memory 210 for the input access control element 120, determining all of the successful matches, coupling the successful matches to the priority encoder 220 for the input access control element 120, 10
determining the highest-priority match, and providing an output result from the input access control element 120.
If at the step 326, the input access control element 120 determines that the higher-level processor should process the packet 130, the higher-level processor processes the 15
packet 130. A result from the higher-level processor is substituted for the result from the input access control element 120.
If at the step 326, the input access control element 120 (or the higher-level processor) determines that the packet 130 20
should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.
At a step 327, the routing element 110 couples the packet label 200 and the output interface specifier to the output 25
access control element 120.
8 matching one or more characteristics of said packet with
one or more of the access control specifiers; selecting a match corresponding to an access control
specifier with a highest associated priority; and processing said packet based on said selecting. 2. The method of claim 1, wherein said access control
element is a content addressable memory. 3. The method of claim 1, wherein said matching and said
processing is done in parallel. 4. The method of claim 1, wherein said characteristics of
said packet comprises one or more of a source address, a destination address, a source port, a destination port, a protocol type, an input interface and an output interface.
5. The method of claim 1, wherein said characteristics of said packet comprises data carried by said packet in a packet header.
6. The method of claim 1, further comprising: receiving said packet. 7. The method of claim 1, further comprising: identifYing one or more of said access control specifiers
based on said matching. 8. The method of claim 7, further comprising: prioritizing said one or more of said access control
specifiers identified based on said matching to generate a set of prioritized access control specifiers.
9. The method of claim 8, wherein said prioritizing is done in parallel by a priority encoder. At a step 328, the output access control element 120
determines the output permission for the packet 130, that is, whether the routing element 110 permits forwarding the packet 130 to the destination device 132 for the packet 130.
10. The method of claim 8, wherein said prioritizing is done based on an address of said access control specifiers in
30 said access control element. The step 326 includes the following actions: matching the packet label 200 against the access control
memory 210 for the output access control element 120; determining all of the successful matches; coupling the successful matches to the priority encoder 35
220 for the output access control element 120; determining the highest-priority match; and providing an output result from the output access control
element 120. If at the step 328, the output access control element 120 40
determines that the higher-level processor should process the packet 130, the higher-level processor processes the packet 130. A result from the higher-level processor is substituted for the result from the output access control element 120. 45
If at the step 328, the output access control element 120 (or the higher-level processor) determines that the packet 130 should be dropped, the packet 130 is dropped, and the routing element 110 takes no further action with regard to the packet 130.
At a flow point 330, the packet is ready for transmission to one of the packet output interfaces 102.
ALTERNATIVE EMBODIMENTS
Although preferred embodiments are disclosed herein, many variations are possible which remain within the concept, scope, and spirit of the invention, and these variations would become clear to those skilled in the art after perusal of this application.
What is claimed is: 1. A method of processing a packet comprising: configuring a plurality of access control specifiers in an
access control element according to a priority of a type of each access control specifier, wherein the type of an access control specifier corresponds to
information in an access control entry;
50
55
60
65
11. The method of claim 8, wherein said processing is done based on said set of prioritized access control specifiers.
12. The method of claim 1, wherein said processing further comprising:
if said packet requires processing by a higher-level processor, forwarding said packet to said higher-level processor.
13. The method of claim 1, further comprising: if said packet requires dropping,
dropping said packet. 14. The method of claim 1, further comprising: if said packet requires forwarding,
forwarding said packet. 15. The method of claim 1, wherein said one or more
access control specifiers include a label match mask and a label match pattern.
16. The method of claim 1 further comprising: a plurality of types of access control specifiers, wherein
each type of access control specifier corresponds to a respective packet field.
17. A system for processing a packet comprising: one or more access control specifiers, wherein
said one or more access control specifiers are of one or more types of access control specifiers, and
said one or more types of access control specifiers being related to information in an access control entry;
an access control element, wherein said access control element is configured to
store said one or more access control specifiers according to a priority of the type of each access control specifier, and
match one or more characteristics of said packet with one or more access control specifiers; and
a priority encoder coupled to said access control element, wherein
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page10 of 12
US 7,023,853 Bl 9
said priority encoder is configured to select a highest priority match based on the priority
of the types of access control specifiers. 18. The system of claim 17, wherein said priority encoder
is further configured to prioritize said one or more access control specifiers
according to an address of said one or more access control specifiers in said access control element.
19. The system of claim 17, further comprising:
10 means for matching one or more characteristics of said
packet with one or more of the access control specifiers; means for selecting a match corresponding to an access
control specifier with a highest associated priority; and
means for processing said packet based on said matching. 33. The system of claim 32, wherein said access control
element is a content addressable memory. 34. The system of claim 32, wherein said matching and
said processing is done in parallel. a compare unit coupled to said access control element, 10
wherein said compare unit is configured to compare said one or more characteristics of said packet with one 35. The system of claim 32, wherein said characteristics
of said packet comprises one or more of a source address, a destination address, a source port, a destination port, a
15 protocol type, an input interface and an output interface. 36. The system of claim 32, wherein said characteristics
of said packet comprises data carried by said packet in a packet header.
or more values. 20. The system of claim 19, wherein said one or more
values are predetermined. 21. The system of claim 19, wherein said one or more
values are dynamically determined. 22. The system of claim 19, wherein said compare unit is
further configured to perform arithmetic operation on data carried by said 20
packet in a packet header. 23. The system of claim 19, wherein said compare unit is
further configured to perform logical operation on said data carried by said
25 packet in said packet header.
24. The system of claim 17, wherein said access control element further comprising:
an access control memory. 25. The system of claim 24, wherein said access control 30
memory is a content-addressable memory.
37. The system of claim 32, further comprising: means for receiving said packet. 38. The system of claim 32, further comprising: means for identifYing one or more of said access control
specifiers based on said matching. 39. The system of claim 38, further comprising: means for prioritizing said one or more of said access
control specifiers identified based on said matching to generate a set of prioritized access control specifiers.
40. The system of claim 39, wherein said prioritizing is done in parallel by a priority encoder.
41. The system of claim 39, wherein said prioritizing is done based on an address of said access control specifiers in said access control element. 26. The method of claim 24, wherein said access control
memory stores at least one of said access control specifier. 27. The system of claim 24, wherein said access control
specifier further comprising: a label match mask configured to determine whether a
first corresponding bit of said one or more characteristics of said packet is tested; and
42. The system of claim 39, wherein said processing is done based on said set of prioritized access control speci-
35 tiers.
a label match pattern, wherein said label match pattern is compared with a second corresponding bit of said one 40
or more characteristics of said packet. 28. The system of claim 17, further comprising: a processor, coupled to said access control element, said
processor is configured to process said packet when said packet is not processed by said access control 45
element. 29. The system of claim 17, further comprising: at least one input port coupled to said access control
element, wherein said input port is configured to receive said packet; and
at least one output port coupled to said access control element, wherein said packet is forwarded via said output port.
50
30. The system of claim 17, wherein said one or more access control specifiers include a label match mask and a 55
label match pattern. 31. The system of claim 17 further comprising: a plurality of types of access control specifiers, wherein
each type of access control specifier corresponds to a 60 respective packet field.
32. A system for processing a packet comprising: means for configuring a plurality of access control speci
fiers in an access control element according to a priority of a type of each access control specifier, wherein the type of an access control specifier corresponds to
information in an access control entry;
65
43. The system of claim 32, wherein said processing further comprising:
means for forwarding said packet to said higher-level processor if said packet requires processing by a higher-level processor.
44. The system of claim 32, further comprising: means for dropping said packet if said packet requires
dropping. 45. The system of claim 32, further comprising: means for forwarding said packet if said packet requires
forwarding. 46. A system comprising: means for maintaining a set of access control patterns in
at least one associative memory; means for receiving a packet label responsible to a packet,
said packet label being sufficient to perform access control processing for said packet;
means for matching matchable information, said matchable information being responsive to said packet label, with said set of access control patterns in parallel;
means for generating a set of matches in response thereto, each said match having priority information associated therewith;
means for selecting at least one of said matches in response to said priority information, and generating an access result in response to said at least one selected match; and
means for making a routing decision in response to said access result.
47. The system of claim 46 further comprising: means for choosing a first one of said matches.
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page11 of 12
US 7,023,853 Bl 11
48. The system of claim 46, further comprising: means for determining an output interface for said packet. 49. The system of claim 46, further comprising: means for implementing a quality of service policy. 50. The system of claim 46, further comprising: means for permitting or denying access for said packet. 51. The system of claim 46, further comprising: means for making a preliminary routing decision for said
packet. 52. The method of claim 46, further comprising: means for determining at least one output interface for
said packet. 53. The system of claim 52, further comprising:
10
means for performing one or more of an arithmetic, logical, and comparison operation on said packet label; 15
and means for generating a bit string for said matchable
information in response to said arithmetic, logical, and comparison operation.
54. The system of claim 46, further comprising: means for preprocessing said packet label; and means for generating said matchable information. 55. The system of claim 46, further comprising:
20
means for comparing a field of said packet label with an 25 arithmetic range or mask value.
56. The system of claim 46, further comprising: means for comparing a source IP port value or a destina
tion IP port value with a selected port value. 57. The system of claim 46, further comprising: means for postprocessing said selected match to generate
said access result. 58. The system of claim 46, further comprising: means for accessing a memory in response to a bitstring
included in said selected match.
30
12 59. The system of claim 46, further comprising: means for declaring whether to permit or deny access of
a set of packets. 60. The system of claim 46, further comprising: means for receiving a sequence of access control speci
fiers; means for translating said sequence of access control
specifiers into a sequence of access control patterns; and
means for storing said sequence of access control patterns in said associative memory.
61. The system of claim 46, further comprising: means for generating a single one of said access control
patterns in response to a plurality of said access control specifiers.
62. The system of claim 46, further comprising: means for generating a single one of said access control
patterns in response to a plurality of said access control specifiers.
63. A method of processing a packet comprising: selecting an output interface to which to forward the
packet; determining forwarding permission for the packet,
wherein the determining comprises matching one or more characteristics of said packet
with one or more access control specifiers in at least one access control element;
processing said packet based on said forwarding permission;
wherein, the selecting step is performed in parallel with the
determining step.
* * * * *
Case3:14-cv-05343 Document1-3 Filed12/05/14 Page12 of 12
Exhibit 4
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page1 of 26
c12) United States Patent Cheriton
(54) METHOD AND APPARATUS FOR SECURING A COMMUNICATIONS DEVICE USING A LOGGING MODULE
(75) Inventor: David R. Cheriton, Palo Alto, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 860 days.
(21) Appl. No.: 10/664,551
(22) Filed: Sep. 19, 2003
(51) Int. Cl. G06F 1124 (2006.01)
(52) U.S. Cl. ...................... 713/100; 713/160; 713/164; 713/168; 713/193
(58) Field of Classification Search ................ 713/100, 713/160, 164, 168, 193
See application file for complete search history.
Communications Interface
Processor 260 ~
J Communications Interface Memory
Unit 265
..... Logging Module
Processor 270
Logging Module
Memory Unit
Logging Module 275
220
111111 1111111111111111111111111111111111111111111111111111111111111 US007340597B 1
(10) Patent No.: US 7,340,597 Bl Mar.4,2008 (45) Date of Patent:
(56) References Cited
U.S. PATENT DOCUMENTS
5,018,079 A * 5,197,128 A * 5,293,466 A *
5/1991 Shukunami et al .......... 358/1.6 3/1993 Campbell et a!. . . . . . . . . . . . . . 710/56 3/1994 Bringmann ................ 358/1.15
* cited by examiner
Primary Examiner-Thomas R. Peeso (74) Attorney, Agent, or Firm-Campbell Stephenson LLP
(57) ABSTRACT
A logging module is disclosed. A communications device can include, and so be made secure through the use of, the logging module. The logging module is configured to communicate information regarding a change to a configuration of a subsystem of the communications device.
110 Claims, 12 Drawing Sheets
~ Line Card j.-230
.... H Line Card r-
235 ~
Switching H Line Card r-
240 ~ ~ Architecture
225 ~Lin~ardr- ~
:--+ Hlio~ardr- ~
H Line Card I+ .222 ~
Communications Interface 215
Network Device 210
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page2 of 26
U.S. Patent Mar.4,2008
Logging Module 120
Sheet 1 of 12
Subsystem 115
Fig. 1
US 7,340,597 Bl
Communications Dev1ce
110
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page3 of 26
U.S. Patent
Communications Interface
Processor
Mar.4,2008
260 1+-
Communi cations Interface Memory ~
Unit
Logging Module Processor
270
Logging Module 220
265
Logging Module
Memory Unit 275
Sheet 2 of 12 US 7,340,597 Bl
H Lin;3~ard ~ f+
H lin;3~ard r- I+
H Line Card L ~ Switching w 1 ,..-
~Architecture
225 ~ Lin~ard ~ 1- r+
rl Lin~ard I+ r+
H Lin~ard ~ r+-
Communications Interface 21.5.
Network Device 210
Fig. 2
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page4 of 26
U.S. Patent Mar.4,2008
Network
300~
Sheet 3 of 12
Fig. 3
Security Monitor
345
US 7,340,597 Bl
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page5 of 26
U.S. Patent Mar.4,2008 Sheet 4 of 12
Start
Determine and Save Configuration of
Communications Device 410
No Compare Current to Saved Configuration
415
Yes
Indicate Configuration Changed 430
Fig. 4
US 7,340,597 Bl
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page6 of 26
U.S. Patent Mar.4,2008 Sheet 5 of 12 US 7,340,597 Bl
( Start ) ~
Assign an IP Address to the Network Device 510
l Assign a Multicast Address to the Logging Module
515
l Configure Logging Module to Communicate Using
Communications Protocol 520
l Reset Network Device to "Known State"
525
l Logging Module Logs and Broadcasts
Configuration Change (from Reset Event) to Security Monitors
530
l Security Monitors Receive Broadcast and
Determine Reset Event Occurred 535
l Security Monitors Begin Tracking Configuration of
Network Device 540
~ ( End
Fig. 5
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page7 of 26
U.S. Patent Mar.4,2008 Sheet 6 of 12 US 7,340,597 Bl
Logging Module Monitors Changes to Configuration of Communications Interface 1+---------------.
610
Yes
Create Log Entry Indicating the Changes 630
No
Yes
No
Yes
Prepare Data Packets & Broadcast According to Communications Protocol
645
No
Yes
Broadcast Status Message 615
Fig. 6
No
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page8 of 26
U.S. Patent Mar.4,2008
Start
Arrange Log Report into Packets
710
Generate Sequence Number and Add to Packets
715
Sheet 7 of 12 US 7,340,597 Bl
>------Yes----.
No
Broadcast Packet 730
End
Fig. 7
Reduce Rate to Comply with Requirement
725
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page9 of 26
U.S. Patent Mar.4,2008 Sheet 8 of 12 US 7,340,597 Bl
Start
t1,1onitor Network T raffle of Communications Interface
810 ~-----------------------No
Yes
~~----------Yes------~
Drop Packet and Enter Log Entry of Communications
Interface's Broadcast Attempt 825
Broadcast Communications Interface's Attempt to the
Security Monitors 830
Fig. 8
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page10 of 26
U.S. Patent Mar.4,2008 Sheet 9 of 12 US 7,340,597 Bl
( Start
Receive List of Active Logging Modules and Multicast
Address 910
+ Subscribe to Logging
Module's Multicast Address 915
+ Receive Log Packets from
Logging Module 920
+ Record a Network Device's
Initial Configuration 925
+ Begin Monitoring Changes to
Network Device's Configuration
930
( End )
Fig. 9
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page11 of 26
U.S. Patent Mar.4,2008
No
Yes
Receive Report from Logging Module 1015
Yes
Update Network Device's Configuration and Analyze Updated Configuration
1040
Set Device's Status to "Trustworthy"
1070
Sheet 10 of 12
Set Device's Status to "Untrustworthy" and Notify Administrator
1035
Fig. 10
US 7,340,597 Bl
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page12 of 26
U.S. Patent Mar.4,2008 Sheet 11 of 12
Start
Start Timer 1110
Yes
Read Configuration 1130
Send "Heartbeat" to Security Monitor
1140
Fig. 11
US 7,340,597 Bl
No
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page13 of 26
U.S. Patent Mar.4,2008
No
Yes
Receive Log Report from Network Device's Logging Module
1220
Compare Network Device's Received Configuration to Configuration Stored
by Security Monitor 1230
Set Network Device's Status to "Trustworthy"
1270
Store Received Configuration 1280
Sheet 12 of 12 US 7,340,597 Bl
Yes
Set Network Device's Status to "Untrustworthy" & Notify Administrator
1260
Fig. 12
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page14 of 26
US 7,340,597 Bl 1
METHOD AND APPARATUS FOR SECURING A COMMUNICATIONS DEVICE USING A
LOGGING MODULE
BACKGROUND OF THE INVENTION
1. Field of the Invention
2 at least in the sense of hard-coded state machines, special memories, registers and the like. Moreover, hardware is conventionally designed to be driven from configuration registers and tables that are written by software, so compromised software can effectively compromise hardware operations as well.
Thus, security features are continuously being developed and implemented to restrict access by unauthorized entities, and so protect such network devices, while maintaining ease
This invention relates to the field of information networks, and more particularly to a method and apparatus for securing a communications device using a logging module.
2. Description of the Related Art Today's networks are an efficient and effective platform
for providing communications between large numbers of computing devices. Each device on the network has easy access to the information and services provided by the other networked devices. The convenience of access, however, significantly increases the risk of an outside attack on one or more of these network devices. Network security is therefore
10 of access by authorized entities. Unfortunately, many of the security solutions deployed to secure network devices are configurable via the network. For example, many such solutions are implemented in software or firmware. Though easy access to the configuration of security devices is
of increasing importance.
15 convenient, this access can significantly undermine the effectiveness of the security device. An attacker can, for example, disable a security device by changing its configuration and then proceed to attack the now-defenseless network device. As will be appreciated, any weak link in the
To date, most of the functionality provided by network devices and other such communication systems is implemented in software, particularly in the control plane of such systems, with hardware simply providing for the acceleration of performance-critical functions. While software has some key advantages over hardware (e.g., flexibility, upgradeability and lower cost), software suffers from several disadvantages with regard security, including the ease with which software may be altered and the difficulty encountered in verifying software.
20 security chain can thus put the entire network at risk. What is needed, therefore, is a mechanism to leverage the
security properties of a system's hardware, and in so doing, improve security of the system (e.g., network device), without placing unrealistic demands on the system's hard-
25 ware, either in terms of complexity or restricted configurability. Preferably, such a mechanism is itself inaccessible or (substantially) non-configurable, such that an attacker cannot compromise the security of the system. In addition, the security solution should be simple enough to implement
30 cost-effectively. The ease with which software can be altered can allow an attacker to compromise restrictions on access to the network device (e.g., router), and so modifY the software to defeat the network device's overall operation. This is particularly dangerous because the attacker can then use this compromised network device as an agent to proceed with further 35
compromise of the network's security. Using an initial security hole as a stepping stone to further compromise a network is a common strategy in this regard.
SUMMARY
In one embodiment, a communications device is disclosed. The communications device includes a logging module. The logging module is configured to communicate information regarding a change to a configuration of a subsystem of the communications device. In certain aspects of this embodiment, the communications device further comprises the subsystem, with the logging module being coupled to the subsystem. Moreover, in other aspects of this embodiment, the logging module is further configured to detect the change.
In another embodiment, a method for securing a communications device using a logging module is disclosed. The method includes detecting a change in a configuration of a subsystem of a communications device, and communicating information regarding the change. In certain aspects of this embodiment, the method further includes determining the
Moreover, the flexibility/complexity of software typically results in an overall lower degree of "verification" than 40
hardware. In fact, the term used in the software arts is "testing" (and not "verification"). This inability to fully exercise such systems portends the risk of unknown weaknesses, which can then be exploited by those wishing to subvert such systems' operations. The ease with which 45
software can be altered also typically implies that any weak point in the software that permits the software to be compromised means that even well-tested software components can be modified to defeat security measures intended for their protection. 50
configuration. The foregoing is a summary and thus contains, by neces
sity, simplifications, generalizations and omissions of detail; consequently, those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any
In contrast, functionality implemented in hardware caunot be easily altered by an attacker, save for an attacker having physical access to the system. Even then, it is extremely difficult to make such alterations without disrupting the operation of such equipment. Moreover, hardware is con- 55
ventionally subjected to far more complete and rigorous verification. An equipment manufacturer is strongly incentivized to do so because of the enormous cost associated with the replacement of design-defective hardware in the field (or even late in the production and manufacturing 60
cycle). However, despite these advantages, hardware implementations are also subject to a number of limitations, including being limited to relatively simple functionality and the need for configuration using software.
Hardware is typically limited to relatively simple func- 65
tionality. In particular, it is impractical to implement complex security control mechanisms and protocols in hardware,
way limiting. Other aspects, inventive features, and advantages of the present invention, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below.
DESCRIPTION OF THE DRAWINGS
The present invention may be better understood, and numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings.
FIG. 1 is a block diagram illustrating a communications device that incorporates embodiments of the present invention.
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page15 of 26
US 7,340,597 Bl 3
FIG. 2 is a block diagram illustrating a network device that incorporates embodiments of the present invention.
FIG. 3 is a block diagram illustrating a network of devices that incorporate embodiments of the present invention.
FIG. 4 is a flow diagram illustrating a process according to embodiments of the present invention.
FIG. 5 is a flow diagram illustrating a process for setting up the logging module according to embodiments of the present invention.
4 to a network administration facility, if the now-compromised network device appears to be misconfigured. In fact, the network security monitors can also be designed to disconnect the now-compromised network device from the network.
FIG. 6 is a flow diagram illustrating a process for moni- 10
taring and indicating changes to the configuration of a network device according to embodiments of the present invention.
In one embodiment, a network device configuration logging protocol is provided, as well as an assigned internet protocol (IP) and Ethernet multicast address and protocol type. The protocol specifies a sequence number plus representation of log records describing configuration operations performed on the network device. For example, each packet can consist of a sequence number plus a sequence of log records, with each log record representing a configuration register or table entry, and the information written to that FIG. 7 is a flow diagram illustrating a process for broad
casting changes to the configuration of a network device according to embodiments of the present invention.
FIG. 8 is a flow diagram illustrating a process for restricting access to the configuration of the logging module by the communications interface.
FIG. 9 is a flow diagram illustrating a process for setting up a security monitor according to embodiments of the present invention.
FIG. 10 is a flow diagram illustrating a process for receiving and acting upon changes to the configuration of a network device.
FIG. 11 is a flow diagram illustrating a process for sending a network device's configuration information.
FIG. 12 is a flow diagram illustrating a process for receiving a network device's configuration information.
The use of the same reference symbols in different drawings indicates similar or identical items.
DETAILED DESCRIPTION OF THE INVENTION
The following is intended to provide a detailed description of an example of the invention and should not be taken to be limiting of the invention itself. Rather, any number of variations may fall within the scope of the invention which is defined in the claims following the description.
Introduction
15 table entry. Each network device supporting such a protocol includes
a hardware component (e.g., a logging module) that generates a log record for each write operation to a hardware configuration register or table. This hardware component
20 then stores each such log record into a packet frame labeled with the next sequence number, and transmits the packet once it is full. Alternatively, the hardware component can be designed to transmit the packet within some timeout period. The packet is transmitted with the address and protocol type
25 assigned to this protocol. The hardware component further includes a mechanism that prevents the software running on the network device from transmitting any logging protocol packets (e.g., packets having this protocol type and address). Any such attempts are dealt with by dropping the packet,
30 and generating a log message (and in so doing, treating this action as an attempt to change the network device's configuration). The hardware component can further include a mechanism that prevents the network device from forwarding any logging protocol packet that specifies the given
35 network device's address as the packet's source address. A reset of the hardware should cause the sequence number to be reset (e.g., to zero), and should be logged as a configuration change.
The logging protocol can also be designed to employ a 40 periodic "heartbeat." This can be implemented as standard
procedure, or simply in the absence of reconfiguration activity. Such a heartbeat can be generated by the hardware, or the software can be designed to cause such by periodically The invention provides a method and apparatus for secur
ing a device using a logging module having restricted configurability. The logging module is configured to monitor 45
a configuration of the device and is further configured to indicate a change to the configuration. The device is restricted from changing a configuration of the logging module.
writing to a null configuration register, for example. In one embodiment, a network security monitor maintains
the identity of each network device incorporating the logging mechanism, and is connected directly or indirectly through the network to each such device. The network security monitor receives logging protocol packets by sub-
50 scribing to the associated multicast address. On reset of a given network device, the network security monitor synchronizes with the state of the hardware on that network device (and its sequence number) when the network security monitor receives a log record of a reset operation. It is
In one embodiment, a logging module is implemented (preferably, primarily in hardware) and a packet generation facility is provided in a network device. The logging module and packet generation facility preferably offer restricted software configurability and control. The logging module and packet generation facility transmit packets reporting configuration actions on the hardware. The device is designed such that software operating on the device is unable to compromise the reporting of the device's configuration (e.g., by altering the logging module's configuration). This can be achieved, for example, by restricting (or sub- 60
stantially restricting) the software's access to the logging module.
One or more network security monitors are then able to receive these hardware-generated logging packets, and so, to monitor the network device's configuration, in order to detect any compromise of the network device. The network security monitors can be designed to report the compromise
55 assumed that a hardware reset forces a network device to a known state. Once the network security monitor receives information regarding the reset, the network security monitor can track the configuration actions of the software on the network device. In particular, when the network security monitor subscribes to this logging protocol address, the network security monitor can track configuration actions at the network device. Subsequently, the network security monitor can detect changes to the hardware configuration (and any unaccounted for logging packets) by examining the
65 sequence number of logging packets received. A network device directly connected to a network security
monitor is similarly protected. One or more missing num-
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page16 of 26
US 7,340,597 Bl 5
bers in the sequence number space of the logging packets received by the network security monitor from the network device indicates the possibility of the network device's having been compromised.
Software executing on such a directly-connected network device cannot forge a logging packet because the hardware component prevents the generation of such forgeries. It is assumed that, in order to preclude the forgery of packets on the link between the network device and the network security monitor, the attacker is not physically able to compromise the link. This is a reasonable assumption, as such is commonly the case (e.g., either the attacker is not physically present, the link is physically protected, the link is encrypted (e.g., in the case of wireless or other technologies) or the like). Typically, the network security monitor is connected through an internal wireline link so this is not an issue.
For an indirectly-connected device, the network security monitor first ensures the integrity of directly-connected devices, and then the devices directly connected through these first hop devices, and so on. The network security monitor has the ability to receive log records generated by a network device configured (e.g., by configuring its forwarding tables) to implement the forwarding of logging packets. This allows the network security monitor to ensure that each network device is properly configured to forward logging packets in an appropriate manner, as requested by the network security monitor.
6 Example of an Apparatus for Securing a Communications Device
FIG. 1 is a block diagram illustrating an architecture of communications device 100 according to embodiments of the present invention. Communications device 100 includes a subsystem 115 and a logging module 120, which is coupled to subsystem 115. Logging module 120 determines a configuration of the subsystem 115, detects a change in the configuration of the subsystem 115, and indicates that the
10 change has occurred. Logging module 120 should also be designed to substantially restrict subsystem 115 from changing a configuration of logging module 120 and to monitor any network data sent by subsystem 115.
Regarding the signals described herein, those skilled in 15 the art will recognize that a signal may be directly trans
mitted from a first block to a second block, or a signal may be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered or otherwise modified) between the blocks. Although the signals of the above described
20 embodiment are characterized as transmitted from one block to the next, other embodiments of the present invention may include modified signals in place of such directly transmitted signals as long as the informational and/or functional aspect of the signal is transmitted between blocks. To some extent,
25 a signal input at a second block may be conceptualized as a second signal derived from a first signal output from a first block due to physical limitations of the circuitry involved (e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, a second signal derived from a
In particular, in response to a request to receive the logging traffic, the network security monitor should receive one or more log records from each affected network device, indicating that the given network device has configured its multicast parameters to forward the logging packet traffic towards the network security monitor. Similarly, the network security monitor can observe (via the logging protocol) configuration actions such as a network device setting its 35
source IP and MAC address.
30 first signal includes the first signal or any modifications to the first signal, whether due to circuit limitations or due to passage through other circuit elements which do not change the informational and/or final functional aspect of the first signal.
The foregoing described embodiment wherein the differ-ent components are contained within different other components (e.g., the various elements shown as components of communications device 100). It is to be understood that such depicted architectures are merely examples, and that in fact
The network security monitor can be, for example, a special network device designed to be highly resistant to attack, possibly with a separate command and reporting connection to the network operations center. This can also be implemented as a module that runs as a feature linecard on various network devices (e.g., routers). Alternatively, this monitor can run as a software module within a network devices, simply receiving logging packets from its neighbors. In such a case, the monitoring should be replicated on several network devices, in order to increase the difficulty encountered by an attacker in attempting to compromise a sufficient number of such modules simultaneously, and so compromise the network's security (specifically, attack detection).
40 many other architectures can be implemented which achieve the same functionality. In an abstract, but still definite sense, any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Hence, any two components
45 herein combined to achieve a particular functionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermediate components. Likewise, any two components so associated can also be viewed as being "operably con-
50 nected", or "operably coupled", to each other to achieve the desired functionality. Moreover, a network implementing the present invention
can be designed to defend against compromised nodes reporting false attacks (in order to cause portions of the network to be disconnected) by maintaining a number of network security monitors, and relying on the majority being 55
accurate against a compromised few. In any case, for reasons of fault-tolerance and security, it is preferable to employ multiple network security monitors. One embodiment of the present invention includes a "overlay" network of network security monitors that are redundant and use a separate 60
authentication, control and reporting mechanism, such that it is effectively impossible for an attacker to compromise the monitoring and reporting performed by these network security monitors. It will be appreciated that the simplicity of the monitoring function relative to the full functionality of the 65
typical network device makes such embodiments particularly feasible.
Example of an Apparatus for Securing a Network Device FIG. 2 is a block diagram illustrating a network device
210 that incorporates embodiments of the present invention. Network device 210 includes a communications interface 215 and a logging module 220 coupled to a communications interface 215. Communications interface includes line cards 230-255, a switching architecture 225, a communications interface processor 260, and a communications interface memory unit 265. Logging module 220 includes a logging module processor 270 and logging module memory unit 275.
Line cards 230-265 are configured to connect network device 210 to other network devices through one or more networks. Switching architecture 225 is coupled to line cards 230-265 and receives network data from and transmits
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page17 of 26
US 7,340,597 Bl 7
network data to line cards 230-255. Connnunications interface processor 260 is coupled to switching architecture 225 and controls operations in connnunications interface 215. Connnunications interface processor 260 communicates with connnunications interface memory unit 265 and writes data to and reads data from communications interface memory unit 265. Connnunications interface memory unit 265 is configured to store, among other data, a configuration of connnunications interface 215.
8 As an extension of this embodiment, the hardware com
ponent further includes a fixed rate-limiting mechanism that limits the rate at which these logging packets can be generated. For example, rate may be limited to say 3 percent of the bandwidth on its ports or 3 Mbps on a 100 Mbps Ethernet port. Writing to a configuration register is disabled temporarily until the packet buffers of the hardware component are transmitted at the limited rate. The rate cannot be modified by software running on the device but only by physical
A logging module processor 270 is configured to control operations oflogging module 220. Logging module processor 270 is coupled to a logging module memory unit 275 and writes data to and reads data from the memory unit. Logging module memory unit 275 stores, among other data, a configuration of the logging module 220.
10 modification of the device, such as changing a jumper on the supervisor card. This mechanism ensures that the logging traffic cannot be an high percentage of the overall traffic on the links.
As a further extension, the hardware component includes 15 a digital signature capability as well as a private key
embedded in hardware that is inaccessible to software. Then, each logging packet can be signed using standard digital signature and message integrity techniques (Message
Logging module processor 270 is also coupled to communications interface processor 260 and substantially restricts connnunications interface processor 260 from modifYing a configuration oflogging module 220 by accessing logging module memory unit 275. Logging module 20
processor 270 also monitors a configuration of connnunications interface 215 via its being coupled to connnunications interface memory unit 265. Logging module processor 270 detects a change in the configuration of connnunications interface 215 and indicates such change. This indication can 25
take any of a number of forms, including a simple mechanism (e.g., an indicator lamp, a message to a display, a message to another network device, broadcast message to specially-configured security devices, or other such mechanisms). Moreover, in addition to monitoring the configura- 30
tion of communications interface 215 (e.g., by recording write operations to connnunications interface memory unit 265), logging module processor 270 can be configured to block writes to connnunications interface memory unit 265. This might be the case, for example, in a situation in which 35
such writes were occurring at a rate that exceeded the allowable log packet rate.
In one embodiment, logging module processor 270 is coupled to switching architecture 225 and broadcasts the change in the configuration of connnunications interface 215 40
to one or more security monitors on the network via switching architecture 225, and one or more of line cards 230-255. Logging module processor 240 can broadcast the change using a particular broadcast address and/or connnunications protocol, in order to prevent other modules from broadcast- 45
ing using the same address and/or protocol. In one embodiment, logging module processor 270 monitors network traffic through switching architecture 225. Logging module processor 270 can then cause connnunications interface 215 to drop any outgoing packets of data not generated by the 50
logging module processor that use the broadcasting address and communications protocol used by logging module processor 270.
Authentication Code (MAC)) to allow the monitors to verifY the logging packet is unmodified from that generated by the source host. This can be achieved by the hardware compo-nent computing a cryptographic MAC on the log packet, either relying on the private/public key of the generating device or else on a shared secret key that is exchanged or configured in the hardware component. In the latter case, it is also feasible to encrypt the log packets to provide confi-dentiality between the reporting nodes and the monitors. Note: this confidentiality is relatively weak and not viewed as critical to the security, because the secret key needs to be shared with the multiple monitors. Its provision is considered useful only to alleviate the potential discomfort to network administrators of otherwise effectively broadcasting configuration actions over the network in the clear.
Although a preferred invention is described as a hardware implementation, the configuration logging could be implemented in software with some benefits. For instance, an attacker may gain the password of a router and proceed to change the configuration yet have no ability to modifY the software itself. In this case, even software configuration logging has benefits because the reconfiguration caused by the attacker would be reported. In this case, ideally, the connnand line interface (CLI) should not provide (at least in production mode) the ability to disable the configuration logging.
Example of a Network that Includes Network Devices Incorporating Embodiments of the Present Invention
FIG. 3 is a block diagram illustrating a network of devices (a network 300) that incorporate embodiments of the present invention. Each of network devices 310-340 includes a logging module for monitoring any changes made to the configuration of the network devices. Security monitor 345 is also connected to network 300 and monitors the broadcasts of the logging modules of network devices 310-340.
55 Additional security monitors can be connected to the network as well. One or more of network devices 310-340 can
In the preferred embodiment, an network device of this invention such as a switch or router handles a well-defined vast majority of the data path activity in hardware carefully separated from software, similar to that provided in the Catalyst 4K Supervisor III. Then, a review of the setting of the hardware configuration by the monitor can determine that a well-defined set of the packet traffic will transit this 60
device without software being able to receive, modify or forge the traffic no matter how compromised it is. Conversely, a device that includes software on the packet datapath in all configurations provides relatively little value using this invention. Fortunately, performance demands tend 65
to make the connnon case packet handling entirely in hardware.
be configured to also act as a security monitor. Security monitor 345 analyzes the broadcasts from network devices 310-340 and determines a "trustworthy" status of a given network device. If security monitor 345 determines that a network device is not "trustworthy," security monitor 345 disconnects the device from the network and/or notify a network administrator.
Security monitor 345 can be connected directly to a network device (as is the case for network devices 320, 330, and 340) or can be connected to a network device indirectly (as is the case for network devices 310, 315, and 325). To
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page18 of 26
US 7,340,597 Bl 9
determine a "trustworthy" status of a given network device that is not directly connected to the security monitor, a security monitor can also determine a "trustworthy" status of one or more devices along a path leading to the network device whose status is to be determined.
10 executed by software modules. The functionality of steps referred to herein may correspond to the functionality of modules or portions of modules.
The operations referred to herein may be modules or portions of modules (e.g., software, firmware or hardware modules). For example, although the described embodiment includes software modules and/or includes manually entered user commands, the various example modules may be application specific hardware modules. The software mod-
10 ules discussed herein may include script, batch or other executable files, or combinations and/or portions of such files. The software modules may include a computer program or subroutines thereof encoded on computer-readable media.
Advantages of a network designed in accordance with the present invention include the following. Such a network provides substantially enhanced security. An attacker cannot compromise the software on a network device implementing the present invention without the network device reporting the associated configuration changes, such that a network security monitor alerts the network administrator or the control systems designated to take action. In embodiments that implement the present invention in hardware, an attacker that does not have direct physical (and sophisti- 15
cated) access to the network device is precluded from interfering with the logging/reporting of the configuration changes. Moreover, a network administrator can rely on the configuration logging to report the results of their actions in administering or reconfiguring a router at a CLI level, which ensures that these actions were actually effected.
A network (including the requisite network devices and network security monitor(s )) is relatively easy to implement. The simplicity of logging makes systems according to the present invention relatively inexpensive to implement in hardware. The hardware architectures of many of today's network devices are already well-positioned to generated the requisite logging packets as part of configuration changes, as well as to prevent software from generating these packets. It should be noted, however, that the configuration logging of the present invention can be implemented in software to some advantage, as well.
Moreover, the present invention provides this increase in security in a efficient marmer, both from an implementation and management perspective. The basic scheme does not rely on secret keys that need to be refreshed periodically, but instead depends on making substantially impossible (at least in some practical sense) to hide the compromising of a configuration. The present invention also allows remote monitoring of devices (subject to the integrity of the links), allowing network administrators to monitoring network configuration activity from a central location, either directly or through an overlay of monitoring nodes.
Example of a Process for Securing a Communications Device
FIG. 4 is a flow diagram illustrating a process according
Additionally, those skilled in the art will recognize that the boundaries between modules are merely illustrative and alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed
20 into submodules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, those skilled in the art will recognize that the operations described
25 in example embodiment are for illustration only. Operations may be combined or the functionality of the operations may be distributed in additional operations in accordance with the invention.
Alternatively, such actions may be embodied in the struc-30 ture of circuitry that implements such functionality, such as
the micro-code of a complex instruction set computer (CISC), firmware programmed into programmable or erasable/progrmable devices, the configuration of a fieldprogrammable gate array (FPGA), the design of a gate array
35 or full-custom application-specific integrated circuit (ASIC), or the like.
Each of the blocks of the flow diagrams depicted herein may be executed by a module (e.g., a software module) or a portion of a module or a computer system user. Thus, the
40 above described method, the operations thereof and modules therefore may be executed on a computer system configured to execute the operations of the method and/or may be executed from computer-readable media. The method may be embodied in a machine-readable and/or computer-read-
45 able medium for configuring a computer system to execute the method. Thus, the software modules may be stored within and/or transmitted to a computer system memory to configure the computer system to perform the functions of to embodiments of the present invention. The process begins
with the determination and saving of the configuration of the communications device (step 410). Next, the configuration 50
is determined again and compared to the saved configuration (step 415). A determination is then made as to whether the current configuration has changed as compared to the saved configuration (step 420). If the configuration has not changed (step 420), the configuration is again determined 55
and compared to the saved configuration (step 415). If it is determined that the configuration has changed (step 420), the change in configuration of the communications device is indicated (step 430). Processing subsequently returns to step 410, where the new configuration is determined and saved, 60
and the comparison process repeated.
the module. Such a computer system normally processes information
according to a program (a list of internally stored instructions such as a particular application program and/or an operating system) and produces resultant output information via I/0 devices. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall functionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall function-
As noted, FIG. 4 depicts a flow diagram illustrating a process according to an embodiment of the present invention. It is appreciated that operations discussed herein may consist of directly entered commands by a computer system 65
user or by steps executed by application specific hardware modules, but the preferred embodiment includes steps
ality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.
Such a computer system typically includes multiple computer processes executing "concurrently." Often, a computer system includes a single processing unit which is capable of
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page19 of 26
US 7,340,597 Bl 11
supporting many active processes alternately. Although multiple processes may appear to be executing concurrently, at any given point in time only one process is actually executed by the single processing unit. By rapidly changing the process executing, a computer system gives the appearance of concurrent process execution. The ability of a computer system to multiplex the computer system's resources among multiple processes in various stages of execution is called multitasking. Systems with multiple processing units, which by definition can support true concurrent processing, are 10
called multiprocessing systems. Active processes are often referred to as executing concurrently when such processes are executed in a multitasking and/or a multiprocessing environment.
The software modules described herein may be received 15
by such a computer system, for example, from computer readable media. The computer readable media may be permanently, removably or remotely coupled to the computer system. The computer readable media may non-exclusively include, for example, any number of the following: 20
magnetic storage media including disk and tape storage media. Optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage memory including semiconductor-based memory units such as FLASH memory, 25
EEPROM, EPROM, ROM or application specific integrated circuits; volatile storage media including registers, buffers or caches, main memory, RAM, and the like; and data transmission media including computer network, point-to-point telecommunication, and carrier wave transmission media. In 30
a UNIX-based embodiment, the software modules may be embodied in a file which may be a device, a terminal, a local or remote file, a socket, a network connection, a signal, or other expedient of communication or state change. Other new and various types of computer-readable media may be 35
used to store and/or transmit the software modules discussed herein.
12 the security monitors receive the broadcast and determine that there has been a change in the network device's configuration and that the network device has been reset to a predetermined, known configuration. After the initial setup, at step 540, the security monitors begin tracking configuration changes of the network defines.
Example of a Process for Monitoring and Indicating Changes to the Configuration of a Network Device
FIG. 6 is a flow diagram illustrating a process for moni-toring and indicating changes to the configuration of a network device according to embodiments of the present invention. At step 610, the logging module begins monitoring changes to the configuration of the communications interface. A determination is then made as to whether there has been a change in the configuration (step 620). If there has not been a change in the configuration (step 620), another determination is made as to whether the time since the last broadcasts was greater than the expected broadcast period (step 625). If the time since the last broadcasts is not greater than the broadcast period (step 625), the logging module continues the process of monitoring changes to the configuration of the communications interface (step 610). If the time since the last broadcast is greater than the broadcast period (step 625), a status message indicating that no changes have occurred is broadcast (step 615).
If a change in the configuration of the communications interface has occurred (step 620), a log entry is created by the logging module indicating the changes (step 630). A determination is then made as to whether the change that occurred is above a threshold criticality (step 635). If the change is not above a certain threshold criticality, processing resnmes (step 610), where the logging module continues to monitor for changes in the configuration of the communi-cations interface. If the change that occurred is above a threshold criticality (step 635), another determination is made as to whether the size of the generated log for broadcast is above a threshold amount (step 640). If the size of the generated broadcast log is not above a certain thresh-Example of a Process for Setting Up a Logging Module
FIG. 5 is a flow diagram illustrating a process for setting up the logging module according to embodiments of the present invention. At step 510, an address (e.g., an internet protocol (IP) address) is assigned to the network device. The
40 old (step 640), processing resnmes (step 610), where the logging module continues to monitor for changes to the configuration of the communications interface.
If the amount of the size of the log is above a certain amount (step 640), the log is converted to data packets and
45 is broadcast on the network according to the logging module's communications protocol (step 645). Processing then resnmes (step 610), where the logging module continues to monitor for changes in the configuration of the communications interface.
IP address is used by the network device to communicate with other devices on the network. A multicast address is assigned to the logging module (step 515). The multicast address is used by the logging module to broadcast security information (such as changes in the configuration of the network device) to the one or more security monitors on the network. A security monitor must subscribe to this multicast 50
address in order to monitor the logging module's broadcasts. The logging module is then configured to communicate using a particular communications protocol (step 520). In one embodiment, the logging module is configured to prevent any broadcasts, other than broadcasts by the logging 55
module, using the same communications protocol as the logging module's communications to ensure that no other part of the network device attempts to assume the identity of the logging module.
At step 525, the network device is reset to a predetermined, "known" state. After receiving a reset command, a stored configuration in a memory unit of the networking device is loaded as the new, active configuration of the network device. At step 530, the logging module detects the loading of the known configuration as a change in the configuration of the logging module and logs and broadcasts the change to the one or more security monitors. At step 535,
An Example of a Process for Broadcasting Configuration Changes
FIG. 7 is a flow diagram illustrating a process for broad-casting changes to the configuration of a network device according to embodiments of the present invention. The process begins by arranging the log report to be broadcast into data packets (step 710). The packets are created accord-ing to a communications protocol employed by the logging module. A new sequence nnmber is then generated (step 715). In one embodiment, a sequence number is generated
60 for each broadcast by the logging module. The security monitor can examine these nnmbers to determine whether the security monitor missed any broadcasts between the current and last broadcast the security monitor received.
A determination is then made as to whether a limit of the 65 bandwidth used by the logging module for broadcasting has
been exceeded (step 720). In one embodiment, a limit may exist on the amount of data the logging module can broad-
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page20 of 26
US 7,340,597 Bl 13
cast on the network to ensure that the logging module's broadcasts do not interfere with other network traffic. For example, broadcasting by the logging module may be limited to 5% of the total bandwidth of the network. If the limit on the bandwidth has been exceeded (step 720), the rate is reduced to comply with the bandwidth requirement (step 725). Typically, this will be accomplished by suspending further configuration updates, pending availability of additional memory space for tracking such configuration updates (e.g., after log packets are sent, ostensibly at the maximum rate). Processing subsequently continues (step 730). If the bandwidth limit has not been exceeded, the generated data packets are broadcast on the network (step 730).
An Example of a Process for Restricting Access to the Configuration of a Logging Module
FIG. 8 is a flow diagram illustrating a process for restricting access to the configuration of the logging module by the communications interface. The logging module is configured to monitor the network traffic of the communications interface in order to prevent the communications interface from assuming the identity of the logging module by broadcasting using the logging module's communications (broadcast) address and/or the logging module's communications protocol. Processing begins with the logging module's monitoring of the communications interface's network communications (step 810).
A determination is then made as to whether the commu-
14 monitor then receives a series of log packets from each logging module (step 920). It will be noted that this is only necessary in the case where the security monitor has no information as to a network device's initial configuration (i.e., after reset) or current configuration. The expected case is one in which a monitored device's initial state is known to the security monitor. Next, an initial configuration of the network device is recorded (step 925), and the security monitor begins monitoring for changes to the configuration
10 of the communications device (step 930).
An Example of a Process for Receiving and Acting Upon Changes to the Configuration of a Network Device
FIG. 10 is a flow diagram illustrating a process performed
15 by a security monitor in receiving and acting upon changes to a network device's configuration. The process begins with awaiting broadcast of a log report by a logging module on the given multicast address (step 1010). Once transmitted, the log report is received from the network device's logging
20 module by the security monitor (step 1 015). A determination is then made as to whether the sequence number received with the log report is the correct (expected) value (step 1020). A sequence number is incorrect if the number is not what is expected in view of the sequence number received
25 with the last broadcast. For example, if the sequence of numbers being used for the broadcasts are consecutive positive integers and the last broadcast was number 53, anything other than 54 for the current broadcast is consid-ered an incorrect sequence number.
If the sequence number is incorrect (step 1020), a deter-mination is made as to whether to defer processing of the log report until the earlier log report(s) are received (step 1025). This is the case in which the security monitor, having received a log report (packet(s)) with a later-than-expected
35 sequence number, defers the processing oflog reports (packet(s)) having a later sequence number until the log report (packet(s)) having an earlier sequence number is received. If processing is to be deferred (step 1025), a determination is then made as to whether a timeout for reception of earlier log
nications interface's network communications packets are using the logging module's communications address (step
30 815). If the communications interface has not attempted to broadcast using the logging module's communications address (step 815), another determination is made as to whether the communications interface has attempted to send packets using the logging module's communications protocol (step 820). If the packets are not being sent using the logging module's communications protocol (step 820), processing resumes (step 810), where the logging module continues to monitor the network traffic generated by the communications interface. If the packets are being sent using the logging module's communications protocol (step 820), the packets are dropped and a log entry is entered indicating the communications interface's attempt (step 825). Similarly, if the communications interface has attempted to transmit using the logging module's communications address (step 815), the packets are also dropped and a log entry entered indicating the communications interface's attempt (step 825). The log containing the attempt by the communications interface is then broadcast to the security monitors (step 830). Processing then resumes
50 (step 810), where the logging module continues to monitor the network traffic generated by the communications interface.
40 report has occurred (step 1030). This avoids the security monitor waiting endlessly for the earlier log report. If no timeout has occurred, the process awaits reception of the next log report.
However, if either processing will not be deferred (step
45 1025) or a timeout has occurred (step 1030), the device's status is set to "untrustworthy" and the administrator notified (step 1035). In this case, it is concluded that the device is in an unsafe state, and the security monitor ceases waiting for the reception of the missing log report( s) (packet( s) ).
If the sequence number is correct (step 1020), the con-figuration of the network device is updated, and the updated configuration analyzed (step 1040). A determination is then made as to whether the network device's updated configuration appears anomalous (i.e., compromised) (step 1050). An Example of a Process for Setting Up a Security Monitor
FIG. 9 is a flow diagram illustrating a process for configuring a security monitor according to embodiments of the present invention. At step 910, the security monitor receives a list containing the active logging modules and the multicast address that the security monitor should listen on for the logging modules' transmissions. It will be noted that multiple multicast addresses can be employed (e.g., one for each logging module). The process begins with the security monitor subscribing to a logging module's multicast address in order to receive broadcasts from the logging module (step 915). Alternatively, the logging module's multicast address can be predetermined (e.g., by the manufacturer of such equipment, by a standards body or the like). The security
55 For example, the security monitor can determine whether a change in the configuration was scheduled and/or whether the change in the configuration corresponds to an expected change in the configuration.
If the updated configuration change appears anomalous or 60 compromised, a determination is made as to whether the
severity of the change in the configuration of the network device is above a certain threshold (step 1060). For example, a change may be too minor by itself to cause security concerns. If the severity is above a certain threshold (step
65 1060), the device's status is set to "untrustworthy" and the administrator notified (step 1035). It will be appreciated that the actions on deciding a node is untrustworthy can further
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page21 of 26
US 7,340,597 Bl 15
include automatically commanding the nodes that it connects with to disable reception and discontinue transmission to this node
16 logging module on the given multicast address (step 1210). Once transmitted, the log report is received from the network device's logging module by the security monitor (step 1220). The log report received is compared to a log report for the network device stored by the security monitor (step 1230).
If the comparison of the received log report and stored log report indicate that the configuration appears anomalous or
If the updated configuration state does not appear anomalous, or if the severity is not above a certain threshold (steps 1050 and 1060), the device's status is set to (or maintained at) a "trustworthy" status (step 1070). The security monitor then returns to awaiting the transmission of log reports (step 1010).
An Example of Processes for Sending and Receiving Configuration Information Regarding a Network Device
the network device appears to be compromised, a determi-10 nation is made as to whether the severity of the differences
FIG. 11 is a flow diagram illustrating a process for sending a network device's configuration information on a periodic basis. In this approach, a logging module sends 15 information on a regular basis, as a "heartbeat" of sorts, to
between the received and stored configurations are above a certain threshold (step 1250). For example, a change may be too minor by itself to cause security concerns. If the severity is above a certain threshold (step 1250), the device's status is set to "untrustworthy" and the administrator notified (step 1260).
Alternatively, if the received log report does not appear to be anomalous, or if the severity is not above a certain
a security monitor. This allows the security monitor to easily ascertain whether a given device (and/or its logging module) has become inoperative or otherwise unable to communicate for some reason. 20 threshold (steps 1240 and 1250), the device's status is set to
(or maintained at) a "trustworthy" status (step 1270) and the received log report is stored (and so become the stored log report) (step 1280). The security monitor then returns to
Typically, a heartbeat includes only some minimal amount of information, such as an artificial change (e.g., information such as a "LastHeartBeatTime" can be updated, causing that information to be sent to the security monitor). This is primarily to indicate that updates may have been 25
missed, if nothing is received within some timeout period, while minimizing the amount of information that needs to be conveyed to the security monitor. Thus, complete configuration information is sent periodically only when the con- 30
awaiting the transmission of log reports (step 1210). It will be noted that, if no log report has been received for
a given network device, the first log report can simply be taken as a known state. In that case, the received log report is stored as the stored log report, for use in comparisons to subsequently-received log reports.
While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled in the art that, based upon the teachings herein, changes and modifications may be made without departing from this invention and its broader aspects and, therefore,
figuration information includes a minimal amount of information. This is done in order to avoid using sequence numbers and performing resynchronization operations. However, in other embodiments, all (or a pertinent portion) of the network device's configuration can be sent to the security monitor, regardless of the amount of information being conveyed (e.g., if necessary from a security perspective, or for some other reason).
The security monitor receiving this configuration information compares the configuration information thus received to stored configuration information, and from that comparison makes a determination as to the trustworthiness
35 the appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of this invention. Moreover, while the invention has been particularly shown and described with reference to these specific embodiments, it will be understood by those
40 skilled in the art that the foregoing and other changes in the form and details may be made therein without departing from the spirit or scope of the invention.
of the network device. This approach also has the added benefit of being able to identify a compromised or failed 45 network device by detecting a lack of heartbeat. In this scenario, a network device's existence is identified by the security monitor's receipt of configuration information. If the periodic transmission of configuration information ceases, the security monitor detects the cessation (e.g., 50 through the use of a timer), and changes the status of the compromised/failed network device to "untrustworthy". Should the network device's heartbeat return, a decision can then be made as to whether to return its status to "trustworthy" or leave the network device's status in the "untrust- 55 worthy" state.
The transmission process begins by starting a timer, which is used to decide when to send the current configuration information (step 1110). The process loops until the timer times out (step 1120). Once the timer times out, the 60 logging module reads the network device's current configuration information (step 1130). The logging module then broadcasts this current configuration to the security monitor (step 1140), and restarts the timer (step 1110).
FIG. 12 is a flow diagram illustrating a process for 65
receiving a network device's configuration information. The process begins with awaiting broadcast of a log report by a
What is claimed is: 1. An apparatus comprising: a communications device comprising:
a subsystem; and a logging module, coupled to said subsystem, and
configured to detect a change to a configuration of said subsystem of said communications device, and communicate information regarding said change to
said configuration of said subsystem of said communications device.
2. The communications device of claim 1, wherein the logging module is further configured to broadcast a
data packet using a logging module network address and a logging module communications protocol.
3. The communications device of claim 2, wherein the logging module is further configured to restrict a
change to a configuration of the logging module by the subsystem.
4. The communications device of claim 2, wherein the logging module is further configured to communicate
the change to the configuration of the subsystem by broadcasting the data packet, wherein the data packet indicates the change to the configuration of the subsystem.
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page22 of 26
US 7,340,597 Bl 17
5. The communications device of claim 4, wherein the logging module is configured to broadcast the change
to the configuration of the subsystem to at least one security monitor coupled to the subsystem via a network.
6. The communications device of claim 5, wherein the security monitor is configured to set the communica
tions device to an "untrustworthy" status in response to receiving the change to the configuration of the sub-system. 10
7. The communications device of claim 6, wherein the security monitor is configured to disconnect the com
munications device from the network in response to the communications device being set to the "untrustwor-thy" status. 15
8. The communications device of claim 2, wherein the logging module is further configured to restrict the
subsystem from broadcasting using the logging module network address and the logging module communica-tions protocol. 20
9. The communications device of claim 2, wherein the logging module is configured to broadcast a series of
data packets, each of the data packets comprises an index number, and each of the index numbers is taken from a sequence of 25
numbers. 10. The communications device of claim 2, wherein the logging module is configured to communicate the
change to the configuration of the subsystem when a condition is satisfied. 30
11. The communications device of claim 10, wherein the logging module is configured to communicate the
change to the configuration of the subsystem when an amount of the change is above a certain threshold.
12. The communications device of claim 10, wherein 35
the logging module is configured to communicate the change to the configuration of the subsystem when a criticality of the change is above a certain threshold.
13. The communications device of claim 10, wherein the logging module is configured to communicate the 40
change to the configuration of the subsystem periodically.
14. The communications device of claim 1, wherein the subsystem is a communications interface. 15. The communications device of claim 14, wherein 45
the logging module is further configured to restrict a change to a configuration of the logging module by the communications interface.
16. The communications device of claim 14, wherein the logging module is further configured to broadcast 50
using the communications interface using a logging module network address and a logging module communications protocol.
17. The communications device of claim 16, wherein the logging module is further configured to restrict a 55
change to a configuration of the logging module by the communications interface.
18. The communications device of claim 16, wherein the logging module is configured to communicate the
change to the configuration of the communications 60
interface by broadcasting the change to the configuration of the communications interface.
19. The communications device of claim 18, wherein the logging module is configured to broadcast the change
to the configuration of the communications interface to 65
at least one security monitor coupled to the communications interface via a network.
18 20. The communications device of claim 19, wherein the security monitor is configured to set the communica
tions device to an "untrustworthy" status in response to receiving the change to the configuration of the communications interface.
21. The communications device of claim 20, wherein the security monitor is configured to disconnect the com
munications device from the network in response to the communications device being set to the "untrustworthy" status.
22. The communications device of claim 16, wherein the logging module is further configured to restrict the
communications interface from broadcasting using the logging module network address and the logging module communications protocol.
23. The communications device of claim 22, wherein the logging module is configured to restrict the commu
nications interface from broadcasting a change to the configuration of the communications interface using the logging module network address and the logging module communications protocol.
24. The communications device of claim 16, wherein the logging module is configured to broadcast using a
series of data packets, each of the data packets comprises an index number, and each of the index numbers is taken from a sequence of
numbers. 25. The communications device of claim 16, wherein the logging module is configured to communicate the
change to the configuration of the communications interface when a condition is satisfied.
26. The communications device of claim 25, wherein the logging module is configured to communicate the
change to the configuration of the communications interface when an amount of the change is above a certain threshold.
27. The communications device of claim 25, wherein the logging module is configured to communicate the
change to the configuration of the communications interface when a criticality of the change is above a certain threshold.
28. The communications device of claim 25, wherein the logging module is configured to communicate the
change to the configuration of the communications interface periodically.
29. The communications device of claim 1, wherein the logging module is configured to communicate the
change to the configuration of the subsystem by broadcasting the change to the configuration of the subsystem.
30. The communications device of claim 29, wherein the logging module is configured to broadcast the change
to the configuration of the subsystem to at least one security monitor coupled to the subsystem via a network.
31. The communications device of claim 30, wherein the security monitor is configured to set the communica
tions device to an "untrustworthy" status in response to receiving the change to the configuration of the subsystem.
32. The communications device of claim 31, wherein the security monitor is configured to disconnect the com
munications device from the network in response to the communications device being set to the "untrustworthy" status.
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page23 of 26
US 7,340,597 Bl 19
33. The communications device of claim 1, wherein the logging module is further configured to broadcast the
change to a security monitor. 34. The communications device of claim 33, wherein the logging module is configured to communicate the
change when a condition is satisfied. 35. The communications device of claim 34, wherein the logging module is configured to communicate the
change when an amount of the change is above a certain threshold.
10
36. The communications device of claim 34, wherein the logging module is configured to communicate the
change when a criticality of the change is above a certain threshold.
37. The communications device of claim 34, wherein the logging module is configured to communicate the
change periodically. 38. The communications device of claim 34, wherein the subsystem is a communications interface. 39. A method comprising: detecting a change in a configuration of a subsystem of a
communications device wherein a logging module is coupled to said subsystem and said detecting is performed at the logging module; and
communicating information regarding the change comprises causing said logging module to communicate the change information.
40. The method of claim 39, further comprising: determining the configuration. 41. The method of claim 40, wherein
15
20
25
30
20 50. The method of claim 49, wherein the security moni
toring process comprises: setting the communications device to an "untrustworthy"
status in response to receiving the change to the configuration of the communications interface.
51. The method of claim 50, wherein the security monitoring process comprises:
disconnecting the communications device from the network in response to the communications device being set to the "untrustworthy" status.
52. The method of claim 46, wherein the communicating comprises:
indicating the change to the configuration of the communications interface when a condition is satisfied.
53. The method of claim 52, wherein the communicating comprises:
indicating the change to the configuration of the communications interface when an amount of the change is above a certain threshold.
54. The method of claim 52, wherein the communicating comprises:
indicating the change to the configuration of the communications interface when a criticality of the change is above a certain threshold.
55. The method of claim 52, wherein the communicating is performed periodically.
56. The method of claim 44, further comprising: executing at least one process in the subsystem according
to the configuration of the subsystem. 57. The method of claim 56, wherein the executing the at
least one process in the communications interface com-prises:
executing a communications process. the information comprises an indication of an occurrence of the change.
42. The method of claim 40, wherein 58. The method of claim 57, wherein the executing the
35 logging process further comprises: the information comprises a change made to the configu
ration. 43. The method of claim 40, further comprising: executing a process in said logging module of the com
munications device, wherein the logging process per- 40
forms the detecting the change and the communicating information regarding the change.
44. The method of claim 43, wherein the subsystem is a communications interface, and
45 the executing the process in the logging module com-
prises executing a logging process. 45. The method of claim 44, wherein the executing a
logging process comprises: executing a logging process in the logging module of the 50
communications device according to a configuration of the logging module.
46. The method of claim 44, wherein the communicating comprises:
restricting a change to a configuration of the logging module by the communications process.
59. The method of claim 57, wherein the executing the logging process further comprises:
broadcasting through the communications interface using a logging module network address and a logging module communications protocol.
60. The method of claim 59, wherein the executing the logging process further comprises:
restricting a change to a configuration of the logging module by the communications process.
61. The method of claim 59, wherein the executing the logging process further comprises:
restricting the communications process from broadcasting using the logging module network address and the logging module communications protocol.
62. The method of claim 61, wherein the restricting comprises:
restricting the communications interface from broadcast-broadcasting the information. 47. The method of claim 46, wherein the broadcasting is
performed using the communications interface. 48. The method of claim 47, wherein the broadcasting is
55 ing a change to the configuration of the communications interface using the logging module network address and the logging module communications protocol.
performed using: a logging module network address, and a logging module communications protocol. 49. The method of claim 47, wherein the broadcasting
comprises: broadcasting the information to a security monitoring
process executing on a security monitor coupled to the communications interface via a network.
63. The method of claim 40, wherein the communicating 60 comprises:
broadcasting the information. 64. The method of claim 63, wherein the broadcasting is
performed using the subsystem. 65. The method of claim 64, wherein the broadcasting is
65 performed using: a logging module network address, and a logging module communications protocol.
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page24 of 26
US 7,340,597 Bl 21
66. The method of claim 65, wherein the broadcasting comprises:
broadcasting the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.
67. The method of claim 66, wherein the security monitoring process comprises:
setting the communications device to an "untrustworthy" status in response to receiving the change to the con- 10
figuration of the subsystem. 68. The method of claim 66, wherein the security moni
toring process comprises: disconnecting the communications device from the net
work in response to the communications device being 15
set to the "untrustworthy" status. 69. The method of claim 63, wherein the communicating
comprises: indicating the change to the configuration of the sub
system when a condition is satisfied. 70. The method of claim 69 wherein the communicating
is performed periodically. 71. A communications device comprising: a subsystem;
20
22 the process in the logging module of the communications device is further configured to cause the processor to:
execute a logging process, wherein the subsystem is a communications interface.
79. The communications device of claim 78 wherein the computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:
broadcast the information. 80. The communications device of claim 79, wherein the
computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:
indicate the change to the configuration of the communications interface when a condition is satisfied.
81. The communications device of claim 78, wherein the computer code is further configured to cause the processor to:
execute at least one process in the subsystem according to the configuration of the subsystem.
82. The communications device of claim 81, wherein the computer code configured to cause the processor to execute the at least one process in the subsystem according to the configuration of the subsystem is further configured to cause
a processor, coupled to the subsystem; 25 the processor to: computer readable medium coupled to the processor; and computer code, encoded in the computer readable
medium, configured to cause the processor to:
execute a communications process. 83. The communications device of claim 82, wherein the
computer code configured to cause the processor to execute the logging process is further configured to cause the pro-detect a change in a configuration of the subsystem; and
communicate information regarding the change. 30 cessor to: 72. The communications device of claim 71, wherein the
computer code is further configured to cause the processor to:
determine the configuration. 73. The communications device of claim 72, wherein the 35
computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:
broadcast the information. 74. The communications device of claim 73, wherein the 40
computer code configured to cause the processor to communicate the information regarding the change is further configured to cause the processor to:
indicate the change to the configuration of the subsystem when a condition is satisfied.
75. The communications device of claim 73, wherein the computer code configured to cause the processor to communicate broadcast the information is configured to use:
a logging module network address, and a logging module communications protocol. 76. The communications device of claim 75, wherein the
computer code configured to cause the processor to communicate broadcast the information is further configured to cause the processor to:
45
50
broadcast the change to the configuration of the sub- 55
system to a security monitoring process executing on a security monitor coupled to the communications device via a network.
77. The communications device of claim 72, wherein the computer code is further configured to cause the processor 60
to: execute a process in a logging module of the communi
cations device, wherein the logging process performs the detecting the change and the communicating information regarding the change.
78. The communications device of claim 77, wherein the computer code configured to cause the processor to execute
65
broadcast through the communications interface using a logging module network address and a logging module communications protocol.
84. A computer program product comprising: a first set of instructions, executable on a computer
system, configured to detect a change in a configuration of a subsystem of a communications device;
a second set of instructions, executable on the computer system, configured to communicate information regarding the change; and
computer readable media, wherein the computer program product is encoded in the computer readable media.
85. The computer program product of claim 84, further comprising:
a third set of instructions, executable on the computer system, configured to determine the configuration.
86. The computer program product of claim 85, wherein the second set of instructions comprises:
a first subset of instructions, executable on the computer system, configured to broadcast the information.
87. The computer program product of claim 86, wherein the second set of instructions comprises:
a second subset of instructions, executable on the computer system, configured to indicate the change to the configuration of the subsystem when a condition is satisfied.
88. The computer program product of claim 86, wherein the second set of instructions use:
a logging module network address, and a logging module communications protocol. 89. The computer program product of claim 88, wherein
the second set of instructions comprises: a third subset of instructions, executable on the computer
system, configured to broadcast the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page25 of 26
US 7,340,597 Bl 23
90. The computer program product of claim 88, further comprising:
a fourth set of instructions, executable on the computer system, configured to execute a process in a logging module of the communications device, wherein the logging process performs the detecting the change and the communicating information regarding the change.
24 100. The apparatus of claim 99, wherein the means for
communicating comprises: means for indicating the change to the configuration of the
subsystem when a condition is satisfied. 101. The apparatus of claim 99, wherein the means for
broadcasting is configured to use: a logging module network address, and a logging module communications protocol. 91. The computer program product of claim 90, wherein
the fourth set of instructions comprises: a first subset of instructions, executable on the computer
system, configured to execute a logging process, wherein
102. The apparatus of claim 101, wherein the means for 10 broadcasting comprises:
the subsystem is a communications interface. 92. The computer program product of claim 91, wherein
the second set of instructions comprises: a first subset of instructions, executable on the computer
system, configured to broadcast the information. 93. The computer program product of claim 92, wherein
the second set of instructions comprises:
15
a second subset of instructions, executable on the com- 20
puter system, configured to indicate the change to the configuration of the subsystem when a condition is satisfied.
means for broadcasting the change to the configuration of the subsystem to a security monitoring process executing on a security monitor coupled to the communications device via a network.
103. The apparatus of claim 98, further comprising: means for executing a logging process in a logging
module of the communications device, wherein the logging process performs the detecting the change and the communicating information regarding the change.
104. The apparatus of claim 103, wherein the subsystem is a communications interface, and the means for executing the process in the logging module
comprises means for executing a logging process. 105. The apparatus of claim 104, wherein the means for 94. The computer program product of claim 91, further
comprising: 25 communicating comprises: a fifth set of instructions, executable on the computer
system, configured to execute at least one process in the subsystem according to the configuration of the subsystem.
95. The computer program product of claim 94, wherein 30
the fifth set of instructions comprises:
means for broadcasting the information. 106. The apparatus of claim 105, wherein the means for
communicating comprises: means for indicating the change to the configuration of the
communications interface when a condition is satisfied. 107. The apparatus of claim 104, further comprising: means for executing at least one process in the subsystem
according to the configuration of the subsystem. a first subset of instructions, executable on the computer
system, configured to execute a communications process.
96. The computer program product of claim 95, wherein the first subset of the fourth set of instructions comprises:
108. The apparatus of claim 107, wherein the means for 35 executing the at least one process in the communications
interface comprises: a first sub-subset of instructions, executable on the com
puter system, configured to broadcast through the communications interface using a logging module network address and a logging module communications proto- 40
col. 97. An apparatus comprising: means for detecting a change in a configuration of a
subsystem of a communications device; and means for communicating information regarding the 45
change. 98. The apparatus of claim 97, further comprising: means for determining the configuration. 99. The apparatus of claim 98, wherein the means for
communicating comprises: means for broadcasting the information.
50
means for executing a communications process. 109. The apparatus of claim 108, wherein the means for
executing the logging process further comprises: means for broadcasting through the communications
interface using a logging module network address and a logging module communications protocol.
110. The method of claim 46, wherein the broadcasting comprises:
sending a series of data packets, wherein each of the data packets comprises an index number
and each of the index numbers is taken from a sequence of
numbers.
* * * * *
Case3:14-cv-05343 Document1-4 Filed12/05/14 Page26 of 26
Exhibit 5
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page1 of 19
c12) United States Patent Kathail
(54) METHOD AND SYSTEM FOR EXTERNALLY MANAGING ROUTER CONFIGURATION DATA IN CONJUNCTION WITH A CENTRALIZED DATABASE
(75) Inventor: Pradeep Kathail, Sunnyvale, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.
(21) Appl. No.: 09/479,607
(22) Filed: Jan. 6, 2000
(51) Int. Cl. G06F 151173 (2006.01) G06F 7100 (2006.01) H04L 12128 (2006.01)
(52) U.S. Cl. ........................ 709/238; 370/351; 707/10; 707/102
(58) Field of Classification Search ................ 707/1-2; 709/223,238,239,242-244
See application file for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
5,634,010 A * 5/1997 Ciscon et al ................ 709/223
100
110
140
150
170
180
111111 1111111111111111111111111111111111111111111111111111111111111 US007162537Bl
(10) Patent No.: US 7,162,537 Bl Jan.9,2007 (45) Date of Patent:
5,909,686 A * 6,032,194 A * 6,119,129 A * 6,226,644 B1 *
2002/0021675 A1 *
6/1999 Muller et al ............. 707/104.1 212000 Gai et a!. ................... 709/239 9/2000 Traversat et al ............ 707/202 5/2001 Ciscon eta!. ................. 707/10 212002 Feldmann ................... 370/254
OTHER PUBLICATIONS
Rekhter, Y. and Li, T. "Request for Conunents 1771: A Border Gateway Protocol 4 (BGP-4)", published by Network working Group, Mar. 1995.*
* cited by examiner
Primary Examiner-David Wiley Assistant Examiner--George C. Neurauter, Jr. (74) Attorney, Agent, or Firm-Sierra Patent Group, Ltd.
(57) ABSTRACT
A method and system for externally managing router configuration data in conjunction with a centralized database subsystem in a router device. The centralized database provides external management registration and unregistration for various managing router subsystems associated with said database system. The centralized database and the subsystems registered for external data management engage in transaction request sequences to provide router data requested by other client subsystems. The arrangement of the various client subsystems associated with the database subsystem allows the client subsystems to remain modular and independent of each other.
22 Claims, 8 Drawing Sheets
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page2 of 19
U.S. Patent Jan.9,2007 Sheet 1 of 8 US 7,162,537 Bl
CPU ~1 2
14 v-RAM
10 16 ~
NVRAM
18
FLASH
20 ~·
ROM
INT 1 ~22 a
_-2 2b INT2
INT3 f------2 2c
•••
- 2 2n
INT n
FIG. 1
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page3 of 19
U.S. Patent Jan.9,2007 Sheet 2 of 8 US 7,162,537 Bl
4~ 2\
jD sysDB
f--- sysDB OTHER TREE SUBSYSTEMS
I I I I I p ETHERNET DIALER AAA ppp CON FIG
\ _\, J J J _\_
30 32 34 36 38 28
FIG. 2
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page4 of 19
U.S. Patent Jan.9,2007
42
'\
50
~ADDRESS
__.....-E 1/0 44
Sheet 3 of 8 US 7,162,537 Bl
cfg--43
IPV4 PPP
•••
E 1/1-----46
FIG. 3
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page5 of 19
U.S. Patent Jan.9,2007 Sheet 4 of 8 US 7,162,537 Bl
26 48 7 / SYSDB MANAGING SUBSYSTEM
71 LOCAL
EXTERNAL MANAGING 1--DATA
I STORE MANAGING UNIT UNIT I I
~ /2 sk 5~
I SYSDB TREE I
FIG. 4
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page6 of 19
U.S. Patent Jan.9,2007 Sheet 5 of 8 US 7,162,537 Bl
100
110
140
150
MANAGING REGISTRATION REQUEST TO SysDB
FIND REQUESTED TUPLE
SET THE "TUPLE HAS EXTERNAL MANAGER" FLAG
DEFINE FUNCTION TO ACCESS DATA FROM
MANAGING SUBSYSTEM
170 CREATE TUPLE DATA STORE FOR CACHE VALUE
180~ SET THE CACHE TIMEOUT OR PERIOD UPDATE VALUE
190
130
NO CREATE TUPLE IN "NO DATA" STATE
NO
FIG. 5
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page7 of 19
U.S. Patent Jan.9,2007 Sheet 6 of 8
200
210
MANAGING UNREGISTRATION REQUEST TO SysDB
FIND REQUESTED TUPLE
240 REMOVE THE "TUPLE HAS
250
EXTERNAL MANAGER" FLAG
REMOVE FUNCTION TO ACCESS DATA FROM
MANAGING SUBSYSTEM
270 REMOVE THE CACHE TIMEOUT OR PERIOD
UPDATE VALUE
280 SET THE DATA VALUE FOR TUPLE TO CURRENT VALUE
290-
NO
NO
FIG. 6
US 7,162,537 Bl
230
ERROR
275
CREATE TUPLE DATA STORE FOR DATA VALUE
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page8 of 19
U.S. Patent Jan.9,2007 Sheet 7 of 8 US 7,162,537 Bl
300 TRANSACTION REQUEST
310 FIND REQUESTED TUPLE
3 0
ERROR
NO
370 NO
REQUEST DATA FROM
EXTERNAL MANAGER
390
FIG. 7
NO
DONE
USE TUPLE VALUE
400
350
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page9 of 19
U.S. Patent Jan.9,2007
410 TRANSACTION REQUEST
420 FIND REQUESTED TUPLE
Sheet 8 of 8 US 7,162,537 Bl
ERROR
CALL VERIFICATION HANDLER
460
NO~--------------< YES
REQUEST DATA CHANGE TO
EXTERNAL MANAGER
FIG. 8
480
NO
5 0
SET TUPLE 520 VALUE
UPDATE CACHE VALUE
NOTIFICATION SEQUENCE 540
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page10 of 19
US 7,162,537 Bl 1
METHOD AND SYSTEM FOR EXTERNALLY MANAGING ROUTER CONFIGURATION
DATA IN CONJUNCTION WITH A CENTRALIZED DATABASE
BACKGROUND OF THE INVENTION
1. Field of the Invention This invention pertains generally to internetwork router
operating systems. More particularly, the invention is a method and system for managing data externally in a centralized database system for a router operating system.
2. The Prior Art In a routing device, internetwork operating systems (I OS)
or more commonly, router operating systems (OS), provide the basic command functions for the routing device as well as various subsystem components which provide specific functions or routines provided by the routing device.
In general, routing devices carry out the operation of reliably transferring network messages or packets between a network of coupled devices, or a collection of such networks. A reliable transfer protocol is provided by the lOS for carrying out such operation. Additionally, an interface in communication with a Configuration ( config) subsystem is provided which allows a user of the routing device to configure the operations of the routing device.
The user may configure, for example, the IP address of an Ethernet interface facility or the default route for the routing device. A config command issued by the user is received by the config subsystem and processed therein. The config subsystem determines from the config command issued by the user which client subsystem is affected by configuration information contained in the config command. The config subsystem then carries out a communication exchange with the affected client subsystem to deliver the change in configuration information.
However, router devices typically include a plurality of client subsystems which manage specific functions, requiring multiple dependencies between the config subsystem and such client subsystems. Furthermore, client subsystems often have multiple dependencies with other client subsystem. For example, the PPP subsystem is dependent upon the IP subsystem for Internet address information and the AAA subsystem for user authentication and credential information. These and other subsystem dependencies as is known in the art prevent modularity in subsystem design and implementation within the lOS of the router.
Another drawback with current subsystem implementation schemes arises when temporary configuration changes
2 opment of the various subsystems of the lOS must take into account these multiple dependencies requiring longer design and development time.
Another situation where a temporary change is desired is when a user connects to the router via a "dial-in" connection port. Dial-in connections are provided by a plurality of subsystem of the lOS. Certain default settings may be configured for most users. However, specialized settings may be configured for certain users, such as network admin-
10 istrators who have particular access privileges, for example. Where a user connects via a dial-in connection, a dialer subsystem communicates with an AAA subsystem to provide name and password information. Responsive to this communication, the AAA subsystem determines the access
15 credentials of the dial-in user from the name and password information and communicates with a PPP subsystem. The access credentials provide, among other things, the configurations for the user at the dial-in connection port. The PPP subsystem then sets the port configurations for the user
20 according to the user's access credentials thereby enabling point-to-point communication for the user.
When the user disconnects, the PPP subsystem, the AAA subsystem and the dialer subsystem need to communicate with each other to restore default settings. This situation
25 presents another illustration where multiple dependencies between the various subsystems of the lOS make common transactions cumbersome and unnecessarily complicated.
Yet another disadvantage with current lOS transaction methods arises when a configuration command issued by a
30 remote user of the routing device causes the user to be disconnected. Such configuration commands while not permanent prevent the user from remotely reconnecting to the routing device to correct the configuration. Normally, the user would need to directly interface with the routing device
35 to correct the configuration, or otherwise manually restart or "reboot" the routing device to discard the current configuration and restore the previous configuration. Current transaction methods do not provide a facility or method for sensing such disconnection events and for restoring the
40 previous connection upon the discovery of such disconnection.
Copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CONFIGURATION CHANGE
45 USING A CENTRALIZED DATABASE, filed Oct. 12, 1999, describes a method and system for transacting routing device configurations using a centralized information provider or database system and is incorporated herein by reference. In this copending application, a centralized data-
50 base system (sysDB) is provided within the lOS which manages transactions on router configuration data. The sysDB receives configuration commands from various lOS subsystems. Such commands may include, for example, a request to change configuration data and a request to revert
to a subsystem are to be carried out. A temporary change is desired when, for example, a user of the routing device wishes to test a particular configuration to analyze the efficiency of such configuration, but would like the opportunity to subsequently revert or "back-out" of the change if desired. During such a configuration sequence, multiple transactions will typically need to be carried out between various subsystems. For example, where a user configures the IP address of an Ethernet facility port, the config subsystem will communicate the new IP address to the IP subsystem. In turn, the IP subsystem will communicate to 60
the Ethernet subsystem that Ethernet facility port has new IP address information. When the changes are to be aborted or otherwise reverted, a similar chain of communication is necessary to complete the task of reverting prior changes. Such multiple dependencies between the various subsystems 65
of the lOS make common transactions cumbersome and
55 changes made to the configuration data. Use of the sysDB allows the system to be modular and avoid unnecessary dependencies between subsystem components.
However, the centralized database scheme is somewhat inefficient when the information stored in the database contains a large amount of data or is changing very fast. For example, when the data in the database is constantly changing (such as statistic counters), the sysDB may have to continuously perform transaction routines, notification routines, and verification routines. The verification routine is described in further detail in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGU-
unnecessarily complicated. Furthermore, design and devel- RATION TRANSACTIONS MANAGED BY A CEN-
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page11 of 19
US 7,162,537 Bl 3
TRALIZED DATABASE filed on Oct. 12, 1999 which is incorporated by reference herein. The notification routine is described in further detail in copending application entitled SUBSYSTEM APPLICATION NOTIFICATION METHOD IN A CENTRALIZED ROUTER DATABASE, filed Oct. 12, 1999 and is incorporated herein by reference.
On the other hand, if the constantly changing data is stored outside (externally) of the centralized database, subsystem applications may become aware of the other applications storing the external data, which may cause the system to become dependent upon these other applications, and therefore cause the system to be non-modular.
Accordingly, there is a need for a method and system for externally managing router configuration data in conjunction with a centralized database which allows the various subsystems of the lOS to be modular and independent. The present invention satisfies these needs, as well as others, and generally overcomes the deficiencies found in the background art.
4 ration information stored on the sysDB may include, for example, Internet protocol (IP) addresses, Ethernet configurations, subnet masks, default routes, protocol configuration, name server information, user and password data, access levels, and other router data as is known in the art. As noted above, prior art router implementations have required the individual subsystems to handle storage and retrieval of configuration information related to the corresponding subsystem (i.e., IP subsystems contained IP configuration data,
10 AAA subsystems contained user authentication information). The present invention employs a centralized sysDB which handles storage and retrieval tasks normally assigned to various subsystems. By centralizing such configuration information in a sysDB, multiple dependencies between the
15 other individual subsystem are avoided or greatly reduced. This arrangement allows the subsystem design and implementation to be modular. Subsystems may be added and removed with greater ease due to the lack of multiple and prevalent dependencies.
An object of the invention is to provide a method and 20
system for externally managing router configuration data which overcomes the prior art.
According to another aspect of the sysDB, certain router configuration information (such as fast changing data or large amounts of data, for example) may be managed externally from the sysDB in one of the other client subsystems. For example, network interface statistic counter
Another object of the invention is to provide a method and system for externally managing router configuration data in conjunction with a centralized database system.
Another object of the invention is to provide a method and system for externally managing router configuration data which does not require multiple dependencies between subsystem applications of the router.
25 information may be managed by the interference management subsystem. Modularity and independence is provided by allowing the externally maintained data to be retrieved from the sysDB, rather than through the externally manag-ing subsystem, as described further below.
Another object of the invention is to provide a method and 30
system for externally managing router configuration data which allows the subsystem applications of the router to be modular and independent of each other.
The sysDB subsystem preferably employs a hierarchical name space scheme in a tree format (sysDB tree) for data storage and retrieval of configuration and other information for the router. Each branch or leaf on the tree is treated as a node or a "tuple". In an illustrative example, the sysDB tree employs a naming convention analogous to the UNIX® file system where intermediate nodes of the tree are analogous
Further objects and advantages of the invention will be brought out in the following portions of the specification, 35
wherein the detailed description is for the purpose of fully disclosing the preferred embodiment of the invention without placing limitations thereon.
to UNIX® directories and where leaf nodes are treated as files and data which are associated with the files. In the preferred embodiment, each node or tuple in the sysDB tree
BRIEF DESCRIPTION OF THE INVENTION 40 has a pointer to its parent node, a pointer to its next peer, and a pointer to its first child. With this arrangement, all the children of a tuple can be iterated by using the first child as the head of a link list and traversing through the corresponding peer of each child. While the sysDB described above
The present invention is a method and system for managing data externally in conjunction with a centralized information provider or database system. The method of the invention is provided by operating system software which is run or otherwise executed on the routing device (router). The invention further relates to machine readable media on which are stored embodiments of the present invention. It is contemplated that any media suitable for retrieving instructions is within the scope of the present invention. By way of 50
example, such media may take the form of magnetic, optical,
45 employs a tree structure for data storage and retrieval, other data storage facilities known in the art may be utilized including, for example, a table, b-tree or relational table scheme without deviating from present invention disclosed herein.
The sysDB subsystem is operatively coupled to the other subsystems of the lOS for providing transactional services.
or semiconductor media. The invention also relates to data structures that contain embodiments of the present invention, and to the transmission of data structures containing embodiments of the present invention.
In its most general terms, the method of the invention comprises software routines and algorithms which are generally provided as part of an operating system which is executed in a router device. The operating system software which is also known as internetwork operating system (I OS) comprises a plurality of subsystems, each of which perform functions for the router.
One of the subsystems provided by the lOS is a centralized database system (sysDB). The sysDB executes as a subsystem component in the router and provides a centralized storage and retrieval facility for configuration information required by other subsystems of the lOS. The configu-
An illustrative lOS may include an Internet protocol (IP) subsystem, an Ethernet subsystem, a dialer subsystem, a point-to-point (PPP) subsystem, an authentication (AAA)
55 subsystem, and a config subsystem, each subsystem operatively coupled to the sysDB subsystem, but not coupled to each other. As is known in the art, the IP subsystem manages the Internet addresses for various port facilities on the router; the Ethernet subsystem manages the Ethernet port
60 facilities on the router; the dialer subsystem manages the dial-in communication access; the PPP subsystem manages the point to point protocol functions on the router; the AAA subsystem manages the user authentication process on the router; and the config subsystem provides configuration
65 management for the router. The sysDB further includes an external managing unit
coupled for communication to the sysDB tree and to other
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page12 of 19
US 7,162,537 Bl 5
subsystems which provide router configuration data externally. The external managing unit carries out the operating of registering and unregistering subsystems for external data managing services. The external managing unit further carries out the operation of providing transaction access to such externally managed data to client subsystems requesting such data.
Managing subsystems (which provide configuration data externally from the sysDB) further include a local managing unit coupled for communication to the external managing 10
unit of the sysDB and a local data store. The local managing unit is coupled to the local data store for managing the external router data within the local data store and providing the router data to the external managing unit of the sysDB upon request. The local managing unit also carries out the 15
operation of registering and unregistering with the sysDB for external data managing services.
A managing subsystem registers for external data managing services with the sysDB by transmitting a "managing" registration request. The managing subsystem may register 20
to externally manage router configuration data which would otherwise be maintained within the sysDB tree. Accordingly, the managing subsystem may register to externally manage an individual tuple of the sysDB tree or an entire sub-tree (or namespace) of the sysDB tree. Upon registering, the sub- 25
system indicates whether the externally managed data may be "cached" (locally copied) in the sysDB tree, in which case the managing subsystem may further indicate when the cached data expires (timeout) or may later invalidate the cached data. The managing subsystem further indicates a 30
lookup address (or function) from which the sysDB may obtain the externally managed data from the managing subsystem.
6 FIG. 6 is a flow chart showing generally the actions
involved in unregistering from external data managing services in accordance with the present invention.
FIG. 7 is a flow chart showing generally the actions involved in handling a request transaction in accordance with the present invention.
FIG. 8 is a flow chart showing generally the actions involved in handling a change transaction in accordance with the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Persons of ordinary skill in the art will realize that the following description of the present invention is illustrative only and not in any way limiting. Other embodiments of the invention will readily suggest themselves to such skilled persons having the benefit of this disclosure.
Referring more specifically to the drawings, for illustrative purposes the present invention is embodied in the apparatus shown FIG. 1 through FIG. 4 and the method outlined in FIG. 5 through FIG. 8. It will be appreciated that the apparatus may vary as to configuration and as to details of the parts, and that the method may vary as to details and the order of the actions, without departing from the basic concepts as disclosed herein. The invention is disclosed generally in terms of a method and system for carrying managing external router data in conjunction with a centralized database, although numerous other uses for the invention will suggest themselves to persons of ordinary skill in the art.
Referring first to FIG. 1, there is shown generally a block diagram of a router device 10 suitable for use with the present invention. The router device 10 includes circuitry or like hardware components well known by those in the art and comprises a CPU 12, random access memory (RAM) 14 operatively coupled to the CPU 12, non-volatile memory (NVRAM) 16 operatively coupled to the CPU 12, flash memory (FLASH) 18 operatively coupled to the CPU 12,
When a request is made to the sysDB for data which is externally managed, the sysDB provides the requested data 35
from the cache if the data is resident in the cache and the data has not expired (or otherwise been invalidated). For other data (non-cached or expired data), the sysDB accesses the lookup address (or function) to provide the data value requested by the requesting subsystems.
It is noted that according the present invention, router configuration data is requested from the sysDB by requesting subsystems, thus providing independence between the various subsystems of the lOS. That is, requesting subsystem do not request data from managing subsystems, but 45
rather from the sysDB. However, by providing a method and apparatus for externally managing data as described herein, the disadvantages associated a centralized database (such overhead with fast changing or large data, for example) can
40 and read-only memory (ROM) 20 operatively coupled to the CPU 12.
be avoided or otherwise reduced.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will be more fully understood by reference to the following drawings, which are for illustrative purposes only.
FIG. 1 is a block diagram of a router device suitable for use with the present invention.
FIG. 2 is a block diagram of an internetwork operating system in accordance with the present invention.
FIG. 3 is a block diagram of an exemplary tree structure for data storage suitable of use with the present invention.
FIG. 4 is a block diagram depicting the relationship between a managing subsystem and the sysDB.
The router device 10 further includes a plurality of interface facilities (INT) 22a through 22n, each of which are operatively coupled to the CPU 12. The interface facilities (INT) 22a through 22n comprise typical ports known in the art which connect to external input/output (I/0) devices. For example, INT 22a may comprise a console port, INT 22b may comprise an Ethernet port, INT 22c may comprise an auxiliary port, and INT 22d may comprise a serial port.
50 Various other port configurations as is known in the art may be arranged without deviating from the present invention.
The CPU 12 carries out the computational tasks associated with executing and rum1ing the internetwork operating system (lOS) software of the present invention and com-
55 prises circuitry or other hardware as is known in the art. In one exemplary embodiment, the CPU 12 comprises a MIPS R4000 CPU.
The RAM 14 may comprise random access memory or dynamic random access memory. The RAM 14 provides the
60 main storage component for the router 10. The RAM 14 is also referred to as working storage and contains the running configuration information of the router which is managed by the system database (sysDB) as described in further detail
FIG. 5 is a flow chart showing generally the actions 65
involved in registering for external data managing services
below. RAM 14 is volatile memory as is lost when power is interrupted to the router 10.
The NVRAM 16 normally contains a persistent copy of the configuration information of the router. The configura-in accordance with the present invention.
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page13 of 19
US 7,162,537 Bl 7
tion information includes, among other things, statements about router-specific attributes, protocol functions, and interface addresses. If power is interrupted to the router 10, the persistent copy of the configuration is provided to the router to provide normal routing operation without the need 5
for reprogramming or reconfiguring. The FLASH 18 is an erasable, programmable read-only
memory which contains the internetwork operating system (I OS) software of the router 10. As is known in the art, flash memory has a structure that enables the flash to store 10
multiple copies of the lOS software. Flash memory content is retained when power is interrupted from the router or the router is restarted.
The ROM 20 contains an initializing bootstrap program and is used during initial start up of the router 10. The ROM 15
20 usually carries out a power-on self-test (POST) to check the hardware components of the router 10 as is known in the art.
During start up, the router 10 conducts the POST check routine which is provided by the ROM 20. The POST check 20
includes a diagnostic which verifies the basic operation of the CPU 12, the RAM 14, the NVRAM 16, the FLASH 18, and interface circuitry 22a through 22n. At the conclusion of the POST, the router 10 loads the lOS software from the FLASH 18 into the RAM 14. It will be appreciated that lOS 25
software may be loaded using a variety of methods without deviating from the present invention including, for example, loading the lOS from an external source such as a TFTP server. The router configuration information is then loaded into RAM 14 from the NVRAM 16. More particularly, the 30
configuration information is loaded into the database system in RAM 14. The configuration information for the router may also be loaded into RAM 14 using other means known
8 (FIG. 3). The sysDB tree 42 contains the running router configuration information used by the various subsystems to carry out their respective tasks.
The sysDB tree structure includes a plurality of branches and leaves which stem from the root configuration (cfg) 43, wherein each branch or leaf is treated as a node or "tuple". For example, FIG. 3 shows a portion of a sysDB tree 42 which includes seven (7) tuples for accommodating router configuration data. For example, Ethernet (E) 1/0 tuple 44 contains Internet address information for Ethernet Port 0 (not shown), and Ethernet (E) 1/1 tuple 46 contains Internet address information for Ethernet Port 1 (not shown). Each tuple includes a first "current" field for storing a current or "default" value associated with configuration information related to the tuple and a second "old" field for storing an "old" configuration value for the tuple. As described further below, the "old" field at a tuple will contain a value when a transaction is currently active on that tuple. When the "old" field value is empty or NULL at a tuple, a transaction is not associated with that tuple.
In certain cases, a plurality of values may be stored at a given tuple by providing an array of fields wherein each field of the array may accommodate a certain value. Other data structures for storing data at a tuple may also be implemented at a tuple without deviating from the present inven-tion.
For router configuration data which is managed externally, the related tuple of the sysDB tree 42 may contain a cached (or local copy) of the external data. In such case, additional data may be provided at the tuple including a timeout period, after which the cached data becomes invalid, and must be "refreshed" (i.e., re-obtained from the managing subsystem). By caching the external data, performance of in the art. The CPU 12 then proceeds to carry out the tasks
required by the lOS. 35 the sysDB can be improved at the cost of slightly stale data. Referring next to FIG. 2, there is shown a block diagram Additional data provided at the tuple may also include an
address (or function) from which the sysDB may obtain the external data from the managing subsystem.
In the preferred embodiment, each node or tuple in the sysDB tree has a pointer to its parent node, a pointer to its next peer, and a pointer to its first child. Thus, E 1/0 tuple 44 has a pointer to Address tuple 50 and to E 1/1 tuple 46. With this arrangement, all the children of a tuple can be iterated by using the first child as the head of a link list and
of an internetwork operating system (I OS) 24 in accordance with the present invention. The lOS 24 which is stored in the FLASH 18 provides the software functions and routines executed by the CPU 12 for the router device 10. The 40
method of the present invention is preferably incorporated into the lOS software device and is executed by the CPU 12. FIG. 3 depicts a block diagram of an exemplary tree structure 42 for data storage which is used in conjunction with the lOS 24 as described herein. 45 traversing through the corresponding peer of each child.
The lOS 24 comprises a plurality of subsystem applications which are executed by the CPU 12 and are loaded and resident in RAM 14. The lOS 24 includes a system database (sysDB) 26 subsystem, a config subsystem 28 coupled to the sysDB 26, an Internet Protocol (IP) subsystem 30 coupled to 50
the sysDB 26, an Ethernet subsystem 32 coupled to the sysDB 26, a dialer subsystem 34 coupled to the sysDB 26, an authentication (AAA) subsystem 36 coupled to the sysDB 26, and a point-to-point protocol (PPP) subsystem 38 coupled to the sysDB 26. It will be appreciated that the 55
configuration shown for lOS 24 is only exemplary and various arrangements of subsystems as known in the art may be used with the method of the present invention. Thus, other subsystems 40 may be coupled to the sysDB 26 to provide additional functions. For example, a SONET sub- 60
system may be coupled to the sysDB 26 to provide optical serv1ces.
The config subsystem 28 carries out the operation of receiving configuration commands for a user of the router, executing the configuration command received from the user and providing configuration information to the user of the router upon request from the user, among other things. As described above, this router configuration information is stored and managed by the sysDB 26 in the sysDB tree 42.
The IP subsystem 30 carries out the operation of providing wide-area connectivity using a set of protocols associated with Internet Protocol (IP). As is known in the art, the IP subsystem provides packet filtering and forwarding functions for the IP protocol.
A connector device (not shown) may be provided as one of the interface facilities 22a through 22n to connect Ethernet facilities to the router 10. The Ethernet subsystem 32 carries out the operation of providing packet filtering based on Ethernet MAC (Layer 2) or IP (Layer 3) addresses as is known in the art and packet forwarding as is known in the
The sysDB 26 manages a centralized database coupled therewith which is shown and generally designated as sysDB tree 42. The centralized database (sysDB tree 42) may comprise any data storage structure known in the art, and is preferably structured and configured as a tree format
65 art. The dialer subsystem 34 carries out the operation of
providing dial-in connection services to a user of the router.
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page14 of 19
US 7,162,537 Bl 9
To this end, the dialer subsystem initiates terminal reception of a user's access credentials, normally in the form of a name and a password.
The AAA subsystem 36 carries out the operation of authenticating the access credentials of users of the router. The AAA subsystem 36 verifies the name and password of the user, which is obtained from the dialer subsystem 34 and determines configuration data for the user as well as access privileges. Configuration data may include such information as the user's IP address, for example. The configuration data for the user is stored in the sysDB tree 42 by sysDB 26 via a transaction request from the AAA subsystem 36.
The PPP subsystem 38 carries out the operation of providing Point-to-Point protocol services over a point-to-point link. As an aspect of providing Point-to-Point protocol services, the PPP subsystem 38 provides a method of encapsulating multi-protocol datagrams into an encapsulated protocol, provides a Link Control Protocol (LCP) which establishes, configures and test the point-to-point link, and provides a Network Control Protocol (NCP) using the encapsulated protocol, which is normally IP.
The sysDB 26 further includes an iterating function (not shown) for navigating to a particular tuple within the sysDB tree 42. A tuple iterator is created for traversing the sysDB tree 42 and is destroyed after completion of its traversal operation. Preferably a tuple iterator does not lock any of the tuples over which it traverses.
Referring now to FIG. 4, there is shown a block diagram generally depicting the relationship between a managing subsystem 48 and the sysDB 26 according to the present invention. A managing subsystem 48 is a subsystem of the lOS which carries out the operation of externally managing router configuration data as described herein.
10 Upon registering, the requesting subsystem 48 indicates
whether the externally managed data may be "cached" (locally copied) in the sysDB tree 42, in which case the managing subsystem 48 may further indicate when the cached data expires (timeout). Alternatively, the managing subsystem 48 may, during operation, invalidate the cached data. The managing subsystem 48 further indicates a lookup address (or function) from which the sysDB 26 may obtain the externally managed data from the managing subsystem
10 48.
When a request is made to the sysDB 26 for data which is externally managed, the sysDB 26 provides the requested data from the cache if the data is resident in the cache and
15 the data has not expired (or otherwise been invalidated). For other data (non-cached or expired data), the sysDB 26 accesses the lookup address (or function) to provide the data value requested by the requesting subsystems.
The method and operation of invention will be more fully 20 understood with reference to the flow charts of FIG. 5
through FIG. 8, as well as FIG. 1 through FIG. 4. FIG. 5 is a flow chart showing generally the actions involved in registering for external data managing services in accor-
25 dance with the present invention. FIG. 6 is a flow chart showing generally the actions involved in unregistering from external data managing services in accordance with the present invention. FIG. 7 is a flow chart showing generally the actions involved in handling a request transaction in
30 accordance with the present invention. FIG. 8 is a flow chart showing generally the actions involved in handling a change transaction in accordance with the present invention. The order of actions as shown in FIG. 5 through FIG. 8 and described below is only exemplary, and should not be
As depicted, sysDB 26 further includes an external managing unit 51 coupled for communication to the sysDB tree 42. The external managing unit 51 is further coupled for communication to a managing subsystem, if any. FIG. 4 depicts a single managing subsystem 48 coupled to the external managing unit 51, although one or more other managing subsystems may also be coupled to the external managing unit 51 as described herein for the managing subsystem 48. The external managing unit 51 carries out the operating of registering and unregistering managing subsystems (such as subsystem 48) for external data managing 45 services. The external managing unit 51 further carries out the operation of providing transaction access to such externally managed data to client subsystems (not shown) requesting such data.
35 considered limiting.
Referring now to FIG. 5, as well as FIG. 1 through FIG. 4, there is shown generally the actions associated with registering a subsystem for external data management. Prior to the registration sequence described herein, the managing
40 subsystem 48 creates the data store 54 for storing the externally managed router data. The data store 54 may be any conventional data store (such as table) for storing data as is known in the art.
At box 100, a managing subsystem 48 (via local manag-ing unit 52) issues a management registration request to the sysDB 26 for external management services. This request will indicate, among other things, the configuration data (tuple) for which the managing subsystem 48 is registering
Managing subsystem 48 includes a local managing unit 52 coupled for communication the external managing unit 51 of the sysDB 26 and a local data store 54. The local managing unit 52 is coupled to the local data store 54 for managing the external router data maintained within the local data store 54 and providing the router data to the external managing unit 51 of the sysDB 26 upon request. The local managing unit 52 also carries out the operation of registering and unregistering with the sysDB 26 for external data managing services.
50 external management and whether the subsystem is registering for management of a "name space" which includes the sub-tree data associated with the tuple. Additionally, the managing subsystem 48 indicates an address (or function) from which the sysDB 26 may obtain the external router data
55 maintained by the managing subsystem 48. The managing subsystem 48 may also indicate whether the external data may be cached (or locally copied) to the sysDB tree 42, and a timeout period after which the cached data becomes invalid and must be refreshed by accessing the address (or
60 function) once again. Box 110 is then carried out. In operation, the one or more managing subsystem applications (such as subsystem 48) may register for external data managing services with the sysDB subsystem 26. During the registration, the subsystem identifies the tuple for which the subsystem is registering for external management. The system may also identifY a name space (i.e., the sub-tree of a 65
tuple) for which the subsystem would like to provide external management.
At box 110, the sysDB 26 receives the registration request of step 100. In response to this request, the sysDB 26 calls a tuple iterator function to find the location of the tuple for which registration is requested. The iterator function searches the sysDB tree 42 starting at the root (cfg) 43 to ascertain the location of the requested tuple. Diamond 120 is then carried out.
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page15 of 19
US 7,162,537 Bl 11
At diamond 120, the iterator function determines whether the requested tuple was found during the search of box 110. If the tuple is not found, box 130 is carried out. Otherwise, box 140 is carried out.
At box 130, the iterator function was not able to find the requested tuple in the sysDB tree 42. The absence of a tuple indicates that data for that tuple currently is not available. However, since some of the configuration data maintained in the sysDB 26 is generated dynamically during the operation
12 At box 230, the iterator function was not able to find the
requested tuple in the sysDB tree 42. The absence of a tuple for unregistration is interpreted as an error because unregistration is proper only when a prior registration was made which would have involved the creation of the requested tuple. Since the iterator function did not find the requested tuple, the unregistration request is improper and an error message is displayed to the user to indicate an unregistration error.
At box 240, the sysDB 26 removes the management registration from the requested tuple by deleting the tuple has external manager flag. Once management registration is removed or otherwise deleted, the requesting subsystem of box 200 will no longer manages router data externally from
of the router, the tuple may contain configuration data at 10
some later time during the operation of the router. Thus at box 130, a tuple associated with the present registration request is created in the sysDB tree 42. The value for this newly created tuple is not set since data is maintained externally. Box 140 is then carried out. 15 the sysDB 26. Box 250 is then carried out.
At box 140, the sysDB 26 registers external management for the requested tuple. The sysDB 26 sets the tuple has external manager flag to indicate the requesting managing subsystem from box 100 will carry out external router data management for the requested tuple. Box 150 is then carried 20
out. At box 150, the sysDB 26 defines the address (or func
tion) from which the sysDB 26 retrieves the actual value of the router data which is externally managed. The address (or function) is provided by the registration request of box 100 25
and is set at the requested tuple. Diamond 160 is then carried out.
At box 250, the sysDB 26 removes or otherwise deletes the address (or function) from which the sysDB 26 access the externally managed data from the managing subsystem 48. Diamond 260 is then carried out.
At diamond 260, the sysDB 26 determines whether the externally managed data was cached in the requested tuple. If externally managed data was cached, box 270 is carried out. Otherwise, box 275 is carried out.
At box 270, the sysDB 26 has determined that the externally managed data was cached at the requested tuple. The sysDB 26 removes the cache timeout for the requested tuple. Box 280 is then carried out.
At box 275, the sysDB 26 has determined that the At diamond 160, the sysDB 26 determines whether the
registration request of step 100 included a request to cache the externally managed data within the tuple. If so, box 170 is carried out. Otherwise box 190 is carried out.
At box 170, the sysDB 26 creates a tuple data store for the cache value at the requested tuple. The tuple data store holds the cache value for the externally managed data. Box 180 is then carried out.
30 externally managed data was not cached at the requested tuple. Since router data will now be managed locally at the sysDB tree 42, storage for tuple data will be required. The sysDB 26 creates a tuple data store for the router value at the requested tuple. Box 280 is then carried out.
35 At box 280, the sysDB 26 sets the tuple current value to the most recent value for the tuple. Alternatively, the tuple may be set to the "no data" state. Box 290 is then carried out.
At box 180, the sysDB 26 defines the cache timeout value after which the cached value becomes invalid. Alternatively, the managing subsystem 48 may proactively invalidate the cache value. Once the cache value is invalid, the sysDB 26 must request the actual value from the managing subsystem as described further in conjunction with FIG. 7 below. Box 190 is then carried out.
At box 290, the unregistration is complete. The sysDB 26 will transmit an acknowledgment to the requesting sub-
40 system to indicate that its management unregistration request was successful.
At box 190, the registration is completed. The sysDB 26 will transmit an acknowledgment to the requesting subsystem to indicate that its registration for external manage- 45
ment was successful. Referring next to FIG. 6, as well as FIG. 1 through FIG.
Referring next to FIG. 7, as well as FIG. 1 through FIG. 4, there is generally shown the actions associated with handling transactions in accordance with the present invention. In the following example, the transaction illustrated is a "tuple query request" where a subsystem is requesting the value of router data from the sysDB 26. FIG. 8 describes a change or modifY transaction request further below. 4, there is shown generally the actions associated with
unregistering a managing subsystem from external data management. Once a subsystem is unregistered with the sysDB 26, the subsystem no longer carries out external data management of sysDB tree 42 tuple data.
At box 300, a subsystem issues a tuple query request to 50 the sysDB 26. Box 310 is then carried out.
At box 200, a subsystem issues a management unregistration request to the sysDB 26. This request indicates the router configuration data for which unregistration is 55
requested. Box 210 is then carried out. At box 210, the sysDB 26 receives the unregistration
request of box 200. In response to this request, the sysDB 26 calls a tuple iterator function to find the location of the tuple for which unregistration is requested. The iterator function 60
searches the sysDB tree 42 starting at the root (cfg) 43 to ascertain the location of the requested tuple. Diamond 220 is then carried out.
At diamond 220, the iterator function determines whether the requested tuple was found during the search of box 210. 65
If the tuple is not found, box 230 is carried out. Otherwise, box 240 is carried out.
At box 310, the sysDB 26 receives the tuple query request of box 300 and ascertains the location of the tuple in the sysDB tree 42. The sysDB 26 calls a tuple iterator function to find the location of the tuple for which a change is requested. The iterator function searches the sysDB tree 42 starting at the root ( cfg) 43 to ascertain the location of the requested tuple. Diamond 320 is then carried out.
At diamond 320, the iterator function determines whether the requested tuple was found during the search of box 310. If the tuple is not found, diamond 340 is carried out. Otherwise, box 330 is carried out.
At box 330, the iterator function was not able to find the requested tuple in the sysDB tree 42. The absence of a tuple for change or update is interpreted as an error because a query transaction at a tuple is proper only if the tuple was previously created. Since the iterator function did not find
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page16 of 19
US 7,162,537 Bl 13
the requested tuple, the query request is improper and an error message is displayed to the user to indicate a query request error.
At diamond 340, the sysDB 26 determines whether the requested tuple has its "tuple has external manager" flag set. 5
If the "tuple has external manager" flag is set, then a managing subsystem is registered to manage the requested data externally from the sysDB 26. If the "tuple has external manager" flag is set at the requested tuple, then diamond 360
14 change transaction at a tuple is proper only if the tuple was previously created. Since the iterator function did not find the requested tuple, the change request is improper and an error message is displayed to the user to indicate a change request error.
At diamond 450, the sysDB 26 determines whether the requested tuple found in box 420 has verification registrations. If a tuple has verification registrations, subsystems that are registered for "verification" at the tuple must first
is carried out. Otherwise, box 350 is carried out. 10 authorize changes before such changes are permitted. Verification registrations are described in further detail in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGURATION TRANSACTIONS MANAGED BY A CENTRALIZED DATABASE filed on
At box 350, external management is not provided at the requested tuple and the transaction is carried out using the method described in copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CON- 15
FIGURATION CHANGE USING A CENTRALIZED Oct. 12, 1999 which is incorporated by reference herein. If verification registrations exist at the requested tuple then box 460 is carried out. Otherwise diamond 490 is carried out.
At box 460, the sysDB 26 determines that the requested tuple has verification registrations. The sysDB 26 then calls
DATABASE, filed Oct. 12, 1999, which is incorporated herein by reference. In the present example, where the transaction is a query request, the sysDB 26 provides the data from the tuple. Box 440 is then carried out.
At diamond 360, external management is provided at the requested tuple. The sysDB 26 determines whether cached data is provided at the requested tuple. If external data is cached at the requested tuple, then diamond 380 is carried out. Otherwise box 370 is carried out.
20 the verification handler routine which either authorizes or denies a change request. The verification handle routine is described further in copending application entitled METHOD AND SYSTEM FOR VERIFYING CONFIGURATION TRANSACTIONS MANAGED BY A CEN-
At diamond 380, the sysDB 26 determines whether the cached value for the external data is still valid. If the cache timeout has not yet expired, and the managing subsystem 48 has not otherwise invalidated the cache value, then box 390
25 TRALIZED DATABASE filed on Oct. 12, 1999. Diamond 470 is then carried out.
is carried out to provide the cache value. Otherwise box 370 30
is carried out.
At diamond 470, the sysDB 26 receives a reply from the verification handler routine. The verification handler will return a "success" reply for authorized changes, or an "error" reply for unauthorized changes. If a "success" reply is issued, diamond 490 is carried out to set the tuple value. Otherwise box 480 is carried out to generate an error message.
At box 390, the sysDB 26 has determined that the cache value is still valid and provides this cache value to the requesting subsystem of box 300 in response to the query request. Box 400 is then carried out.
At box 480, the verification handler returned an "error" in 35 response to proposed changes. An error message is gener
ated and is displayed to the user. At box 370, the sysDB 26 has determined that either the cache value is invalid or that cache data is not provided at the requested tuple. In either case, the external managing unit 51 accesses the address (or function) from which data may be retrieved from the managing subsystem 48 regis- 40
tered at the requested tuple. The sysDB 26 then provides the ascertained data value to the requesting subsystem of box 300 in response to the query request. Box 400 is then carried out.
At diamond 490, the sysDB 26 determines whether the requested tuple has its "tuple has external manager" flag set. If the "tuple has external manager" flag is set, then a subsystem is registered to manage the requested data externally from the sysDB 26. If the "tuple has external manager" flag is set at the requested tuple, then diamond 510 is carried out. Otherwise, box 500 is carried out.
At box 500, external management is not provided at the At box 400, the request transaction has been completed.
The process described herein is carried out for each such request transaction made to the sysDB 26.
Referring now to FIG. 8, as well as FIG. 1 through FIG.
45 requested tuple and the transaction is carried out using the method described in copending application entitled METHOD AND SYSTEM FOR EXECUTING, TRACKING AND RESTORING TEMPORARY ROUTER CONFIGURATION CHANGE USING A CENTRALIZED 4, there is generally shown the actions associated with
handling change or modify transactions in accordance with 50
the present invention. At box 410, a subsystem issues a tuple query request to
the sysDB 26. Box 420 is then carried out. At box 420, the sysDB 26 receives the tuple query request
of box 410 and ascertains the location of the tuple in the 55
sysDB tree 42. The sysDB 26 calls a tuple iterator function to find the location of the tuple for which a change is requested. The iterator function searches the sysDB tree 42 starting at the root ( cfg) 43 to ascertain the location of the requested tuple. Diamond 430 is then carried out.
At diamond 430, the iterator function determines whether the requested tuple was found during the search of box 420. If the tuple is not found, diamond 450 is carried out. Otherwise, box 440 is carried out.
60
At box 440, the iterator function was not able to find the 65
requested tuple in the sysDB tree 42. The absence of a tuple for change or update is interpreted as an error because a
DATABASE, filed Oct. 12, 1999, which is incorporated herein by reference. In the present example, where the transaction is a change request, the sysDB 26 sets the value at the requested tuple. Box 540 is then carried out to carry out notification sequence.
At diamond 510, external management is provided at the requested tuple. The sysDB 26 determines whether cached data is provided at the requested tuple. If external data is cached at the requested tuple, then box 520 is carried out. Otherwise box 530 is carried out.
At box 520, the sysDB 26 has determined that external management is provided at the requested tuple. To effect the current change request, the sysDB 26 updates the cached value to the requested changed value. Box 530 is then carried out.
At box 530, the sysDB 26 requests a data change to the external managing subsystem to effectuate the change externally. In response to this requests, the managing subsystem
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page17 of 19
US 7,162,537 Bl 15
48 updates the local data store 54 to reflect the changed value. Box 540 is then carried out to carry out notification.
At box 540, the sysDB 26 executes the notification routine which notifies registered subsystems of changes made to the requested tuple. Copending application entitled SUBSYSTEM APPLICATION NOTIFICATION METHOD IN A CENTRALIZED ROUTER DATABASE, filed Oct. 12, 1999 describes in further detail the method for carrying out router configuration change notifications in conjunction with a centralized database and is incorporated by reference 10
herein. Accordingly, it will be seen that this invention provides a
method for externally managing router data in conjunction with a centralized database system. Although the description above contains many specificities, these should not be 15
construed as limiting the scope of the invention but as merely providing an illustration of the presently preferred embodiment of the invention. Thus the scope of this invention should be determined by the appended claims and their legal equivalents. 20
What is claimed is: 1. A method for reducing computational overhead in a
centralized database system by externally managing router data in conjunction with a centralized database subsystem, said database subsystem operatively coupled for communi- 25
cation with a plurality of router subsystems one of which is a first managing subsystem, comprising:
a) transmitting a management registration request by said first managing subsystem to said database subsystem, said registration request indicating router configuration 30
data for which said first managing subsystem is requesting to provide external management services, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configu- 35
ration subsystem before being stored in said database; b) receiving said management registration request by said
database subsystem; and c) registering said first managing subsystem for external
management by said database subsystem. 40
2. The method of claim 1 further comprising maintaining router configuration data using a tree structure having a plurality of tuples by said database system.
3. The method of claim 2 wherein said registering said 45
managing subsystem for external management further com-prises:
(a) finding a requested tuple for which external management is requested; and
16 6. The method of claim 1 further comprising: (a) transmitting a query request for router data by a
requesting subsystem to said database subsystem; (b) receiving said query request by said database sub
system; (c) determining whether said router data is externally
managed by a second managing subsystem; (d) determining whether said router data is locally cached
when said database subsystem determines said router data is externally managed by a second managing subsystem;
(e) determining whether said cache value is valid when said database subsystem determines said router data is locally cached;
(f) determining cache value of router data when said cache value is valid; and
(g) providing said cache value to said requesting subsystem when said cache value is valid.
7. The method of claim 6 further comprising: a) accessing said router data from said managing sub
system by said database subsystem when said cache value is not valid; and
b) said database subsystem providing said router data to said requesting subsystem when said cache value is not valid.
8. The method of claim 1 further comprising: (a) transmitting a change request for router data by a
requesting subsystem to said database subsystem; (b) receiving said change request by said database sub
system; (c) determining whether said router data is externally
managed by a second managing subsystem; and (d) requesting a data change for said router data to said
second managing subsystem by said database subsystem when said database subsystem determines said router data is externally managed by a second managing subsystem.
9. The method of claim 8 further comprising: a) determining whether said router data is locally cached;
and b) updating the cache value to said router data when said
router data is locally cached. 10. A program storage device readable by a machine,
tangibly embodying a program of instructions executable by the machine to perform a method for reducing computational overhead in a centralized database system by exter-
(b) defining an address to access external router data from said first managing subsystem at said requested tuple.
4. The method of claim 3 wherein said registering said managing subsystem for external management further comprises caching said external router data at said requested tuples.
50 nally managing router data in conjunction with a centralized database subsystem, said database subsystem operatively coupled for communication with a plurality of router subsystems one of which is a first managing subsystem, said method comprising:
5. The method of claim 1 further comprising: a) transmitting a query request for router data by a
requesting subsystem to said database subsystem; b) receiving said query request by said database sub
system; c) determining whether said router data is externally
managed by a second managing subsystem;
55
60
d) accessing said router data from said second managing subsystem by said database subsystem when said database subsystem determines said router data is exter- 65
nally managed by a second managing subsystem; and e) providing said router data to said requesting subsystem.
(a) transmitting a management registration request by said first managing subsystem to said database subsystem, said registration request indicating router configuration data for which said first managing subsystem is requesting to provide external management services, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configuration subsystem before being stored in said database;
(b) receiving said management registration request by said database subsystem; and
(c) registering said first managing subsystem for external management by said managing subsystem.
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page18 of 19
US 7,162,537 Bl 17
11. The program storage device of claim 10, said method further comprising maintaining router configuration data using a tree structure having a plurality of tuples by said database system.
12. The program storage device of claim 11, wherein said registering said managing subsystem for external management further comprises:
(a) finding a requested tuple for which external management is requested; and
(b) defining an address to access external router data from 10
said first managing subsystem at said requested tuple.
18 17. The program storage device of claim 10, said method
further comprising: (a) transmitting a change request for router data by a
requesting subsystem to said database subsystem; (b) receiving said change request by said database sub
system; (c) determining whether said router data is externally
managed by a second managing subsystem; and (d) requesting a data change for said router data to said
second managing subsystem by said database subsystem when said database subsystem determines said router data is externally managed by a second managing subsystem.
13. The program storage device of claim 12, wherein said registering said managing subsystem for external management further comprises caching said external router data at said requested tuples. 15
18. The program storage device of claim 17, said method further comprising: 14. The program storage device of claim 10, said method
further comprising: (a) transmitting a query request for router data by a
requesting subsystem to said database subsystem; (b) receiving said query request by said database sub- 20
system; (c) determining whether said router data is externally
managed by a second managing subsystem; (d) accessing said router data from said second managing
subsystem by said database subsystem when said data- 25
base subsystem determines said router data is externally managed by a second managing subsystem; and
(e) providing said router data to said requesting subsystem.
15. The program storage device of claim 10, said method 30
further comprising: (a) transmitting a query request for router data by a
requesting subsystem to said database subsystem; (b) receiving said query request by said database sub
system; (c) determining whether said router data is externally
managed by a second managing subsystem;
35
(d) determining whether said router data is locally cached when said database subsystem determines said router data is externally managed by a second managing 40
subsystem; (e) determining whether said cache value is valid when
said database subsystem determines said router data is locally cached;
(f) determining cache value of router data when said 45
cache value is valid; and (g) providing said cache value to said requesting sub
system when said cache value is valid. 16. The program storage device of claim 15, said method
further comprising: (a) accessing said router data from said managing sub
system by said database subsystem when said cache value is not valid; and
50
(b) said database subsystem providing said router data to said requesting subsystem when said cache value is not 55
valid.
(a) determining whether said router data is locally cached; and
(b) updating the cache value to said router data when said router data is locally cached.
19. In a router device having a processor and memory, a router operating system executing within said memory comprising:
(a) a database subsystem; (b) a plurality of client subsystems, each operatively
coupled for communication to said database subsystem, one of said client subsystems configured as a managing subsystem to externally manage router data upon issuing a management request to said database subsystem; and
(c) a database operatively coupled to said database subsystem, said database configured to store router configuration data and delegate management of router configuration data to a management subsystem that requests to manage router configuration data, said router configuration data managed by said database system and derived from configuration commands supplied by a user and executed by a router configuration subsystem before being stored in said database.
20. The router operating system of claim 19 wherein said database subsystem is structured and configured as a tree structure having a plurality of tuples.
21. The router operating system of claim 19 wherein said database subsystem further comprises an external managing unit configured to access external router data from said managing subsystem.
22. The router operating system of claim 19 wherein said managing subsystem further comprises:
(a) a local managing unit; and (b) a data store unit structured to store external router data
for said managing subsystem, said local managing unit configured to provide external router data from said data store to said database subsystem.
* * * * *
Case3:14-cv-05343 Document1-5 Filed12/05/14 Page19 of 19
Exhibit 6
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page1 of 19
(12) United States Patent Finn
(54) MULTI-BRIDGE LAN AGGREGATION
(75) Inventor: Norman W. Finn, Livermore, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 2244 days.
(21) Appl. No.: 10/282,438
(22) Filed: Oct. 29, 2002
(65) Prior Publication Data
US 2004/0098501 Al May 20,2004
(51) Int. Cl. G06F 15116 (2006.01)
(52) U.S. Cl . ........................ 709/249; 709/238: 714/4.12 (58) Field of Classification Search .................. 709/245,
(56)
709/246,249,238, 220; 714/4.11,4.12 See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
6,657,951 B1 * 12/2003 Carroll eta!. ........... ..... 370/222 6,978,308 B2 * 12/2005 Boden et al ............. ..... 709/229 6,996,628 B2 * 212006 Keane eta!. ........... 709/238 7,027,412 B2 * 4/2006 Miyamoto eta!. ............ 370/255 7,028,333 B2 * 412006 Tuomenoksa eta!. ......... .. 726/3 7,028,334 B2 * 4/2006 Tuomenoksa ....... ... 726/3 7,028,337 B2 * 4/2006 Murakawa ....... 726/15 7,209,435 B1 * 4/2007 Kuo et al. ..... 370/219
2003/0048746 A1 * 3/2003 Guess eta!. ...... 370/219
OTHER PUBLICATIONS
Hewlett Packard, LAN Aggregation through Switch Meshing, Jun. 1998, pp. 1-11*
.,.,-300
111111 1111111111111111111111111111111111111111111111111111111111111 US008051211B2
(IO) Patent No.: (45) Date of Patent:
US 8,051,211 B2 Nov.l, 2011
Amendment to Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications, IEEE Computer Society, Mar. 30, 2000, pp. 1-173.* Lan Aggregation Through Switch Meshing, Hewlett Packard, Jun. 1998, pp. 1-12, http://www.hp.com/rnd!library/pdf/techlib_meshing.pdf. Dynamic LACP Trunking, Hewlett Packard, 2000, pp. 1-6, http:// www.hp.com/rnd/support/config_examples/2524_1acp.pdf. Software Configuration Guide-Release 5.2, Configuring Fast EtherChannel and Gigabit EtherChannel, Software Configuration Guide, Cisco Systems, Jun. 27, 2002, pp. 7-1-7-16, http://www.cisco. com/univercd! cc/td! doc/product/ian/ catS 000 I reL 5 _2/ config/ channel.pdf. Sofhvare Cof!figuration Guide-Release 5.2, Configuring VLAN Trunks on Fast Ethernet and Gigabit Ethernet Ports, Cisco Systems, Jun. 27,2002, pp. 12-1-12-28, http://www.cisco.com/univercd!cc/td! doc/product/lan/cat5000/reL5_2/config/e_trunk.pdf. Amendment to Carrier Sense Multiple Access with Collision Detection (CSMAICD) Access Method and Physical Layer Specifications-Aggregation of Multiple Link Segments, LAN MAN Standards Committee of the IEEE Computer Society, Mar. 30, 2000, pp. i-ix and 1-173, IEEE Std 802Jad-2000. International Search Report as mailed from the PCT on Aug. 2, 2004 for cotmterpart WO Application (PCT!US03/34423; Filed Oct. 29, 2003), 3 pages).
* cited by examiner
Primary Examiner- Kevin Bates (74) Attorney, Agent, or Finn Campbell Stephenson LLP
(57) ABSTRACT
A method and system for multi-bridge LAN aggregation is disclosed. The method includes aggregating a plurality of LANs coupling a host to a first and a second intermediate network device. The system includes an intermediate network device. The intem1ediate network device includes a multi-bridge engine. The multi-bridge engine includes a tunnel engine coupled to a bridge interconnect port and a first physical port.
32 Claims, 9 Drawing Sheets
LAN310
r344
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page2 of 19
U.S. Patent Nov.l, 2011
z :)
J- w Cl (.')
~ 0 ~
Cl ro
0 H ('I
<{
Cl Q_
Sheet 1 of9
z <{ _!
N 0
f-(/)
0 I
US 8,051,211 B2
~
e " "' u:
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page3 of 19
U.S. Patent
1-Ci. ~ Ci. 0 H Ci. 0...
'<t 0 N
z :)
Nov.l, 2011
CD
;:;
00 0 N
\
0
"' N
;,(
ro w (9 0 ~ co
co 0 N
\
<t: w (9 0 ~ ro
Sheet 2 of9 US 8,051,211 B2
00
;:;
~ N 0 N
l '<t ;:;
~
0 N
z <{ ..J
1-UJ
~ 0 N :r
~] 0
N 0) z N <t
..J
~
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page4 of 19
340
I 354-------- i [r;~ 352
338 ~
~ 346
( d r- 332
LAN 302
340
346 352 ~ 322 I
( ~ r 0
342
360
~ 352 324~
~ ~ ~~ 346
320
Figure 3A
340
-300
352
340 340
LAN 310
344
352
358
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ :--" N 0 ~ ~
ILl =('D ('D -(.H
0 -I..C
d rJJ QO = Ul ,........
'N ,........ ,........
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page5 of 19
352
342
366
B99.0 364
BRIDGE 1 353---......._
366
352
LAN 312
-~··;6·2 356
HOST
Figure 3B
LAN 310
;- 352
344 BRIDGE 2
~352
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ :--" N 0 ~ ~
ILl =('D ('D -~ 0 -I..C
d rJJ QO
= Ul ,........
'N ,........ ,........
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page6 of 19
1-400
BRIDGE ~404
i. :----------------------------------------------------------------------------------------------- _, __ ;;, 412
: MULTI-BRIDGE BRIDGE-: ~ 406 ENGINE INTERCONNECT _
416 : ~n
: Pl SUB-PORT ,r- 414 : ~ ~ 0
: HAGGREGATEDri----------------------------------------------------H : P2 BRIDGE PORT I SUB-PORT 1
' l~ 408 1 : BRIDGE - I AGGREGATED r suB-PORT I I PB FORWARDING P3 I BRIDGE PORT ,~--------------------+--1 2
I ENGINE
I 1~ 408
' AGGREGATED !" SUB-PORT P7 P4 1----+--------l-------11 BRIDGE PORT ,~--------------1---j 3 I 418
----------~ SUB-PORT' MUX I 1+-'---/ __ _ p6 p5 4 DEMUX ~~;;;:E
: LINK : SUB-PORT I 5 I
' I I
' ! : , I I
j lsuseo) I 1 I· N ! ll l r 4to ~ 410 1
i I TUNNEL l [ TUNNEL l l • ENGINE ENGINE • ' ' ! '
l ---------- ----------- --------- --------------------- ---------- --------------------------~ ~-------~~ 402
I PORT 0 I PORT 1 I PORT 2 ) PORT 3 ! PORT 4 I PORT 5 i PORT 6 I . . I PORT N I F1gure 4
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ ~
N 0 ~ ~
ILl =('D ('D -lh 0 -I..C
d rJJ QO = Ul ,........
'N ,........ ,........
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page7 of 19
r------6 BYTES •I• 6 BYTES ·,J· 4 BYTES •• ."[... 4 BYTES t ;-506
ETHER TYPE PRIORITY CFI VL.AN OX8100 ID
506(a) 506{b) 506(c) 506(d)
Figure 5
r5oo
DATA
2 BYTES N BYTES t I
4BYTEs~
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ ~
N 0 ~ ~
ILl =('D ('D -0\ 0 -I..C
d rJJ QO = Ul ,...... 'N ,...... ,......
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page8 of 19
618 618
610
620 624 620
BRIDGE 1 BRIDGE 2
624
620
LAN 606
614
''C''~l·~· HOST
Figure 6
LAN 604
612
618
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ ~
N 0 ~ ~
ILl =('D ('D --l 0 -I..C
d rJJ QO
= Ul ,........
'N ,........ ,........
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page9 of 19
U.S. Patent Nov.l, 2011 Sheet 8 of9 US 8,051,211 B2
CONFIGURE TUNNELING BRIDGE /702
704 CONFIGURE VIRTUAL. PORTS
ON TUNNELING BRIDGE
CONFIGURE PASS·THOUGH PATH r 706
MAP VIRTUAL PORT DIRECTLY TO PHYSICAL
PORT
CONFIGURE HOST
AGGREGATE NETWORK INTERFACE PORTS OF HOST
v708
/710
v 712
GENERATE IP ADDRESS FOR v- 714
VIRTUAL NETWORK INTERFACE
CONFIGURE HOME BRIDGE
CONFIGURE VIRTUAL PORTS ON TUNNELING BRIDGE
AGGREGATE PORTS OF HOME BRIDGE
Figure 7 A
r 716
718
v-720
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page10 of 19
ENCAPSULATE FRAME /722
GENERATE TAG /724
[ SET CFI BIT " 0 ~726
--------- 728 SET USER PRIORITY TO
VIRTUALPORTPRIORITY
SET VLAN-ID TO VIRTUAL 730 PORT NUMBER
( r~m l ADD TAG TO FRAME
~ 734 TRANSMIT FRAME TO
BRIDGE 1 )
~ DE-ENCAPSULATE FRAME r 736
( DETERMINE VIRTUAL PORT I ~ 738 NUMBER
FRAME RECEIVED BY VIRTUAL PORT r740
CORRESPONDING TO VIRTUAL PORT NUMBER
742
744
, TRANSMIT FRAME TO HOST Figure 7B
TRANSMIT FRAME DIRECnY TO VIRTUAL PORT
CORRESPONDING TO PHYSICAL PORT
ENCAPSULATE FRAME
746
748
/750
GENERATE TAG ~ 752
( SET CFI BIT= 0 J~ 754
SET USER PRIORITY TO 1-756 VIRTUAL PORT PRIORITY
SET VLAN-ID TO VIRTUAL PORT NUMBER
758
;-760
ADD TAG TO FRAME
I ' 762 DE-ENCAPSULATE FRAME /
,---- '~ 764 DETERMINE VIRTUAL PORT
NUMBER
FRAME RECEIVED BY VIRTUAL PORT
CORRESPONDING TO VIRTUAL PORT NUMBER
766
F1gure 7C
~ rJl . ~ ~ ~ ~
= ~
z 0 ~ ~
N 0 ~ ~
ILl =('D ('D -1,0
0 -1,0
d rJJ QO = Ul ,........
'N ,........ ,........
= N
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page11 of 19
US 8,051,211 B2 1
MULTI-BRIDGE LAN AGGREGATION
BACKGROUND OF THE INVENTION
1. Field of the Invention The present invention relates generally to computer net
works and, more specifically, to a multi-bridge LAN aggregated method and system for use in a computer network.
2. Description of the Related Art Computer Networks Generally, a computer network is a group of computers (or
hosts) coupled to each other in a way that allows information to be exchanged between the computers. A local area network
2 improve availability. Unfortnnately. some current methods of providing redundancy are inefficient, limited in use, and create undesirable consequences.
Link Aggregation FIG. 1 illustrates a host 102 coupled to a LAN 104 via a
bridge 106 and LANs 108 and 109. Host 102 includes network interfaces 110 (e.g., SO and S1), both of the same medium (e.g., Ethernet). Bridge 106 includes ports 114 (e.g., portsAO-A2).As seeninFIG.1, two LANs are provided from
10 host 102 to bridge 106, LAN 108 from network interface SO to port AO, and LAN 109 from network interface S1 to port Al. In configuring host 102 and budge 106 in this matmer, should one of the LANs 108-109 fail, another LAN is avail-
is a common example of a computer network. As its name implies, a LAN is a computer network which is organized 15
within a given geographic area or locale, such as a college campus, a site of a corporation, a single building, etc. Various types ofLANs include Ethemet, FDDI, and Token Ring. The LAN type refers to the physical medium and connections over which traffic (i.e., data) is carried using hardware specific to 20
the LAN type. Data on LANs are carried in frames. As used herein, a frame refers to information which is transferred between a host and a bridge and/or between multiple bridges. Each frame includes, at least, a destination link layer address,
able to transport data. Unfortunately, a consequence of utilizing multiple net
work interfaces 110 of host 102 to connect to LAN 104 is an increase in the number of IP addresses which host 102 is associated. This increase results from an intemet protocol (IP) address being associated with each network interface of a host. Having multiple IP addresses associated with a single host is disadvantageous for a number of reasons. Initially, confusion results as to which IP addresses should be used to communicate with the host and management of the host is made more difficult. Further, use of multiple IP addresses for a host creates inefficiency due to the time it takes to determine the IP addresses associated with the host, and the space con-
a source link layer address, a frame type indication, and data. 25
Each frame transmitted to a given LAN can be observed and/or received by every other computer or intermediate network device attached to that LAN.
sumed in storing the multiple IP addresses. Link aggregation (also known as tmnking) alleviates some
of the problems associated with multiple IP addresses by A number of individual LANs may be coupled together with bridges to create a Bridged LAN. Bridges are intermediate network devices which can be used to interconnect LANs at the link layer to enable computers on one LAN to commtmicate with the computers of another LAN. Bridges forward frames, as necessary, from one LAN to another, using the destination link layer address of the frames. Bridges learn which LAN Segments to forward the frames on based on the source link layer addresses of the fran1es. In general, a Bridged LAN can com1ect many more computers together, and cover a much wider geographical range, than a single LAN. The term "LAN Segment" is often used to refer to a non-bridged LAN (a LAN that includes no bridges). Although a LAN may refer to a computer network organized in a given locale, as used herein, the term "LAN" is used to refer to a physical connection between one or more hosts (e.g., a LAN Segment). Also, as used herein, the term "host" refers to an end-station which is the source o±~ or destination of, frames transmitted over a network.
A router is an intermediate network device which also interconnects a number ofLANs and/or other types of transmission media. For example, a router may be used to connect one Ethemet LAN (or Bridged LAN) to another, or to cmmect a FDDI LAN to a digital satellite link. Routers generally forward packets, which are essentially data coupled with header information which describes properties of the packet such as a source network layer address, a destination network layer address. and a packet length. The router forwards pack
30 grouping LANs of the same medium type and speed together to form a link aggregation group, which is treated as a single link with the capacity of all the links combined. Commonly known protocols and software may be used to enable link aggregation on bridge 106 and host 102. For example, Link
35 Aggregation Control Protocol (LACP) is a connnon link aggregation protocol defined in IEEE Std.802.3-2000, clause 43. Similarly, Port Aggregation Protocol (PAgP) is a well known protocol developed by Cisco Systems. Inc. and useful for dynamically aggregating redundant links cmmecting two
40 or more devices. With bridge 106 and host 102 configured for link aggrega
tion, the multiple links between them are seen as one link, and consequently host 102 may be seen from a network as having one IP address, even though multiple interfaces of host 102
45 may be cmmected to the network. Additionally, aggregating the multiple links through link aggregation has the advantage of increasing the bandwidth between host 102 and bridge 106 to twice that of what the bandwidth would be without link aggregation. Further if one of the links between host 102 and
50 bridge 106 were to fail, communication would resume on the remaining links. Unforttmately, although link aggregation provides a redundant coupling to bridge 106, no redundancy is provided between LAN 104 and bridge 106. Thus, ifbridge 106 were to ±ail, host 102 would lose connnunication with
55 LAN 104.
ets from one LAN to another, adding or removing frame information such as link layer addresses, as needed. A router knows which LAN to send the packets on based on infonnation within the packet itself and a configuration table acces- 60
sible by the router which correlates address information to LAN information. The term "switch" is sometimes used for
Another method of providing redundant com1ections between a host and a LAN is to couple a host to multiple bridges with multiple wires. FIG. 2 includes a host 202 coupled to LAN 204 via bridges 206-208 and LANs 210-212. Host 202 includes network interfaces 214 (e.g., SO and S1). Bridges 206 and 208 includes ports 218 (e.g., ports AO-A1 and BO-B1, respectively). As can be seen from FIG. 2, redundancy is provided from host 202 to LAN 204, accomplished in part by bridges 206 and 208.
an intermediate network device that combines some or all of the functions of both a router and a bridge.
Often, it is desirable to have redundant physical cmmec- 65
tions to a computer network (e.g., redundant physical connections to multiple intem1ediate network devices) to
The advantages provided by the configuration illustrated in FIG. 2 include the utilization of redundant bridges. If bridge 206 where to fail, host 202 would still be able to communicate
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page12 of 19
US 8,051,211 B2 3 4
FIG. 2, labeled as prior art, is a block diagram of an exemplary network connection in the absence oflink aggregation.
FIGS. 3(A-B) are block diagrams of a computer network including an embodiment of multi-bridge LAN aggregation in accordance with the present invention.
FIG. 4 is a block diagram of a bridge for use in multi-bridge LAN aggregation in accordance with the present invention.
with LAN 204, and vice versa if bridge 208 were to fail. However, two IP addresses are associated with host 202, one for each network interface SO and Sl, which introduces the aforementioned disadvantages associated with managing hosts with multiple IP addresses. Link aggregation is not available in such a scenario because link aggregation does not support multi-bridge configurations with a single host. Link aggregation is traditionally only available between two devices (e.g., one bridge and one host). In addition, although two LANs are shown coupled to host 202, the bandwidth of host 202 is not automatically doubled. The use of LANs 210-212 is determined by the host IP address chosen, and thus may be under the control of neither host 102 nor bridges 206-208.
FIG. 5 is a block diagram of an encapsulated unit transmitted in multi-bridge LAN aggregation in accordance with the
10 present invention.
Another traditional implementation of providing redun- 15
dant physical connections to a network involves stacking multiple bridges together and configuring the multiple bridges to appear as one bridge. In order to accomplish this, however, the bridges must be configured to connnunicate with each other for sharing and leaming network and frame 20
information. Although this configuration may be a reasonable solution for providing redundancy if the bridges are in close proximity to each other (e.g., stacked on top of or next to each other) and the necessary cabling and configuration is provided, it has many pitfalls. If one of the bridges and or cables 25
connecting the bridges were to fail, the network com1ection would be lost, since the bridges could no longer "learn" from each other (e.g., learning link layer addresses from each other). Additionally, such a configuration is not desirable if the bridges are to be at arms length from each other and 30
independent of each other (i.e., not have to depend on other bridges for learning network and address information).
FIG. 6 is a block diagram of another embodiment of a bridge for use in multi-bridge LAN aggregation in accordance with the present invention.
FIGS. 7(A-C) are flow charts illustrating a process of aggregating a plurality of LAN s coupling a host to a first and second intermediate network device, according to one embodiment of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
Introduction The present invention generally enables link aggregation to
be utilized on redundant physical connections between a host and multiple network devices, thus improving reliability and availability of data transmitted to and from the host. As used herein, link aggregation is used to refer to implementations of link aggregation such as IEEE Standard 802.3-2000, clause 43 and Ethercham1el. The following is intended to provide a detailed description of an example of the invention and should not be taken to be limiting of the invention itself. Rather, any number of variations may fall within the scope of the inven-tion which is defined in the claims following the description.
Exemplary Network Architecture SUMMARY OF THE INVENTION FIG. 3 is a block diagram of a computer network 300
35 including an embodiment of multi-bridge LAN aggregation In one embodiment of the present invention, a method of in accordance with the present invention. Network 300
multi-bridge LAN aggregation is disclosed. The method includes LANs 302-338 interconnected by a number of inter-includes aggregating a plurality ofLANs coupling a host to a mediate network devices, such as bridges 340-344, and rout-first and a second intermediate network device. ers 346. The intem1ediate network devices include ports 352
Inanotherembodimentofthepresentinvention, a system is 40 which allow the intermediate network devices to physically disclosed. The system includes an intermediate network connect to one another and to other network devices. As used device. The intermediate network device includes a multi- herein, port refers to a point of attachment to a LAN through bridge engine. The multi-bridge engine includes a tu1mel which an intennediate network device (e.g., a bridge) trans-engine coupled to a bridge interconnect port and a first physi- mits and receives data (e.g., media access control frames). cal port. 45 Network 300 also includes hosts 354-358 which, in the
The foregoing is a summary and thus contains, by neces- described embodiment of the present invention, are server sity, simplifications, generalizations and omissions of detail; computer systems configured to send and receive information consequently, those skilled in the art will appreciate that the on network 300. surnn1ary is illustrative only and is not intended to be in any It is desirable that a host, host 356 for example, be coupled way limiting. As will also be apparent to one of skill in the art, 50 to network 300 such that if one path connecting host 356 to the operations disclosed herein may be implemented in a network 300 were to fail, including the failure of an intenne-number of ways, and such changes and modifications may be diate device (e.g., bridge 342), an altemate path would be made without departing from this invention and its broader available. Additionally, it is also desirable that such a con-aspects. Other aspects, inventive features, and advantages of figuration provide host 356 on network 300 with a single IP the present invention, as defined solely by the claims, will 55 address, and also utilize any redundant links coupled to host become apparent in the non-limiting detailed description set 356 to increase the bandwidth ofinfonnation sent to and from forth below. host 356. Accordingly, host 356 is coupled to network 300 in
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the figures designates a like or similar element.
FIG.l, labeled as prior art, is a block diagran1 of an implementation of link aggregation.
a multi-bridge LAN aggregation configuration in accordance with one embodiment of the present invention. Similarly, host
60 358 is also coupled to network 300 in a multi-bridge LAN aggregation configuration in accordance with one embodiment of the present invention.
As seen from FIG. 3A, host 356 is coupled to LAN 310 via, at least, LANs 312-314 and bridges 342-344. If bridge 342
65 ancl/or LAN 312 were to fail, host 356 would be able to communicate with LAN 310 via LAN 314 and bridge 344. A similar situation would result if bridge 344 ancl/or LAN 314
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page13 of 19
US 8,051,211 B2 5
were to fail. Additionally, bridge 344 and host 356 are configured for link aggregation, allowing both LAN 312-314 to be simultaneously utilized for transmitting information to and from host 356 and allowing host 356 to be seen from LAN 310 as having a single IP address.
6 such examination. Thus, port AO on bridge 320 is slaved through sub-port A99.0 to sub-port B99.0 of bridge 344. Bridge 342 is essentially transparent to bridge 344 and host 356.
In similar fashion, host 358 is coupled to LAN 310 via, at least, LANs 316-318 and bridges 342-344. If bridge 344 and/or LAN 318 were to faiL host 358 would still be able to communicate with LAN 310 via LAN 316 and bridge 342. A similar situation would result if bridge 342 and/or LAN 316 10
were to fail. Additionally, bridge 342 and host 358 are configured for link aggregation, allowing both LANs 316-318 to
From the perspective of bridge 344, sub-port B99.0 is treated as a separate interface, and link aggregation is configured on bridge 344 for port BO and sub-port B99.0. Additionally, from the perspective of host 356, LANs 312 and 314 are "seen" as being directly connected to bridge 344, by virtue of the pas-through path 353. Thus, in the described embodiment, there are two paths between host 356 and LAN 310, with one of the paths transparently traversing bridge 342 (e.g., via pas-through path 353). be simultaneously utilized for transmitting information to and
from host 358 and allowing host 358 to be seen from LAN 310 as having a single IP address.
It should be understood that network 300 is provided as an exemplary network in which multi-bridge LAN aggregation
Redundancy and Failure Recovery 15
in accordance with the present invention can be implemented. Other embodiments of the present invention may be implemented in networks other than network 300, and conse- 20
quently the present invention should not be limited to network 300.
With the benefits of Link Aggregation, one IP address, represented by virtual network interface 362, is associated with host 356, even though both network interfaces SO and S1 are coupled to LAN 310. The present invention thus provides the benefit of bridge redundancy and link redundancy between host 356 and LAN 310 with host 356 seen by LAN 310 as having a single IP address via Link Aggregation. If LAN 312 and/or bridge 342 were to fail, link redundancy on host 356 would detect the failure and "shift" all commtmications to LAN 314 and bridge 344.
Exemplary Multi-Bridge LAN Aggregation Configuration FIG. 3 B illustrates a portion of network 3 0 0 in closer detail.
In response to the loss of LAN 314 and/or bridge 344, however, bridge 342 would reconfigure portAO as a "normal" bridge port, discard the use of sub-portA99 .0, and also begin executing Link Aggregation (i.e., port AO would no longer be slaved through sub-port A99.0 to sub-port B99.0 of bridge
30 344 ). With the reconfiguration of portAO as a bridge port, and without pas-through path 353, host 356 (i.e., the link aggregationonhost356) no longer sees LANs 312 and314 as being directly coupled to bridge 344. Rather, LAN 312 would be seen as coupled to bridge 342, and LAN 314 would be seen as
The portion illustrated is provided only for aiding in the 25
description of the present invention. As illustrated in FIG. 3B, LAN 310 is coupled to intermediate network devices 342 and 344. In one embodiment of the present invention, intennediate network devices 342 and 344 are Layer 2 (or "L2") Ethernet switches. A L2 switch, as is conunonly known, is a switch which operates on the data-link layer of the OSI reference model. In the described embodiment, intermediate network devices 342 and 344 are bridges. Bridges 342 and 344 include ports 352 (e.g., ports AO-A1 and ports BO-B1, respectively) and bridge-interconnect ports 366. In the described embodiment, bridge-interconnect port 366 represents a logical port. Many logical ports may be multiplexed over a single physical port (or multiple physical ports, if Link Aggregation is employed). Included in bridge-intercom1ect ports 366 is sub-portA99.0 which is logically coupled to port AO. Ports 352 allow bridges 342 and 344 to be physically coupled to LANs 310-314. Bridge 342 and 344 are coupled to each other via inter-bridge link 3 64. In one embodiment of the present invention, inter-bridge link 364 represents one LAN
35 coupled to bridge 344, or not seen at all, depending on the extent of the failure. Host 356 (via a link redundancy protocol such as LACP or PagP) would see this as a re-cmmection of network interface SO to another device, and abort the aggregation. Host 356 would then choose either LAN 312 or LAN
40 314 for its virtual IP interface, according to the usage of the aggregation protocol. A similar reconfiguration would occur if inter-bridge link 364 connecting bridges 342 and 344 were to fail. Because bridges 342 and 344 are able to operate independently from each other (e.g., no "learning" is required
or many LANs Link Aggregated together. Host 356 is cmmected to both bridges 342 and 344 through
network interfaces 360 (e.g., SO and S1). Together, network interfaces 360 are configured as one virtual network interface 362 (having one IP address for virtual network interface 362) in accordance with the present invention. In the described embodiment, network interfaces 360 are Ethernet interfaces. In accordance with the present invention, bridge 342 is configured with a pass-through path 353 between port AO and sub-port A99.0 (a description of how this configuration is accomplished is described in greater detail below). Any frames transmitted to port AO are sent directly to sub-port A99.0 without any examination by bridge 320. Similarly, any frames transmitted to sub-portA99.0 are sent directly to port AO without any exan1ination by bridge 320. As used herein, tunneling is used to refer to transmitting a frame without examination. For example, when a bridge receives a frame, it generally examines the frame to determine the corresponding LAN Segment to forward the frame to. Additionally, a bridge will process the frame according to a number of protocols. However, in accordance with the present invention, bridge 342 is configured to internally transmit a frame between bridge inter-com1ect port 366 and port AO directly, without
45 between bridges 342 and 344 ), host 356 can preferably remain in communication with network 310 even if interbridge link 364 were to fail.
In addition to the benefits of bridge redundancy and link redundancy between host 356 and LAN 310, the present
50 invention also provides for the advantages of! ink redundancy in a multi-bridge configuration. In other words, even though host 356 is configured to utilize multiple network interfaces 360 in coll1111unicating with LAN 310, host 356 is preferably able to interface with LAN 310 with a single IP address as a
55 result of virtual network interface 362 configured by link aggregation. Being associated with a single IP address provides for easier management of host 356 (and network 300), and eliminates any confusion as to which address IP address should be in conllllunicating with host 356. Further, with link
60 redundancy, the bandwidth between host 356 and LAN 310 is increased (e.g., doubled) since LANs 312 and 314 are able to share the load of the traffic.
Once any failures ofbridge 342, bridge 344, LAN s 312 and 314, and/or inter-bridge link 364 are corrected, LinkAggre-
65 gation protocol would resume utilizing both bridges 342 and 344 in accordance with the present invention (e.g., bridge 342 is able to automatically reconfigure portAO as a pass-through
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page14 of 19
US 8,051,211 B2 7
port once a failure on bridge 344, for example, has been corrected.) Although the present embodiment is described as including LANs 312 and 314 for use in link aggregation, it is recognized that any number of LAN s (and ports and network interfaces) may be used. Additionally, the LANs may pass through any number of bridges, and in order to increase bandwidth without a corresponding increase in reliability, multiple paths may be tmmeled through one bridge.
Exemplary Bridge Architecture FIG. 4 is a block diagram of a bridge 400 in accordance 10
with one embodiment of the present invention. Bridge 400 includes ports 402 (e.g., port 0-port N) and a multi-bridge engine 404. Multi-bridge engine 404 provides forwarding and pass-through capabilities to bridge 400 which allow data to either be forwarded by bridge 400 (e.g., processing a frame 15
received by bridge 400 according to a number of different protocols) or passed through bridge 400 (e.g., passing a frame through bridge 400 with little or no processing of the frame) in accordance with the present invention.
Multi-bridge engine 404 includes a bridge-forwarding 20
engine 406, aggregated-bridge ports 408, turmel engines 410, and a bridge-intercom1ect port 412. It will be recognized that a nmnber of different types of mux/demux technologies may be used in the present invention. For example, Layer 3 tunnels, a number of dedicated Ethernets, and/or an array ofATM 25
emulated LANS. Bridge-forwarding engine 406 receives data on one or more ports (e.g., P1-PN), examines the data according to a number of protocols, and forwards the data out the one or more ports P1-PN. Aggregated bridge ports 408 provide link aggregation support for bridge 400 in accordance with 30
the present invention. Tunnel engines 410 are utilized for transparently passing data through bridge 400 (e.g., tmmel engine 410 may strip of an encapsulated header on a frame passed through bridge 400). It will be recognized that bridge forwarding engine 406, aggregated bridge ports 408, and 35
tutmel engines 410 may be implemented in hardware or software.
Bridge-interconnect port 412 intemally routes data to bridge forwarding engine 406, aggregated bridge ports 408, and/or tunnel engine 410 depending on, for example, infor- 40
mation associated with the data (e.g., an encapsulation tag 506 such as that shown in FIG. 5). Additionally, bridgeinterconnect port 412 receives data from bridge forwarding engine 406, aggregated bridge ports 408 and/or turmel engine 410 and adds information to the data (e.g., encapsulates data 45
with an encapsulation tag 506 such as that shown in FIG. 5). Bridge-interconnect port 412 includes sub-ports 414 (e.g., sub-port 0-sub-port N) and multiplexer/de-multiplexer 416. Sub-ports 414 allow data to be transmitted between interbridge link 418 and various internal objects of bridge 400 50
(e.g., bridge forwarding engine 406, aggregated bridge ports 408, tunnel engine 410, and/or ports 402). Multiplexer/demultiplexer 416 provides encapsulation and de-encapsulation to frames received by bridge-inter-connect port 412. Bridge 400 also includes one or more buffers (not shown) and or 55
memory hierarchies (not shown) to temporarily store frames received on, for example, ports 402 and/or bridge-interconnect port 412.
Bridge 400 is coupled, via inter-bridge link 418, to a bridge 420 (similar to bridge 400, and not shown for simplicity of 60
illustration). Within bridge 400, sub-port 0 is coupled to P1 of bridge forwarding engine 406. P1 represents a LAN linking bridge 400 to bridge 420. A similar configuration is provided in bridge 420. In addition, P1 is utilized to transmit a frame between bridge 400 and 420 at a periodic frequency in order 65
for bridge 400 and 420 to detect a failure in conmmnication (e.g., the loss of inter-bridge link 418) between bridge 400
8 and bridge 420. For example, in one embodiment of the present invention, bridge 400 is configured, via P1 of bridge forwarding engine 406, to transmit a keep-alive frame to bridge 420. Similarly, bridge 420 is configured to receive the keep-alive frame, and transmit the keep alive frame back to bridge 400. This process is repeated with a frequency of preferably 1 second or less. The loss of three keep-alive frames (either consecutively or within a defined time) results in a reconfiguration of bridges 400 and 420. The home bridge (which is the bridge not configured as the tmn1eling bridge) will consider the encapsulated link over inter-bridge link 418 as failed. The tunneling bridge will cease pass-through capabilities (but only temporarily, since automatic configuration of pass-through is possible once commtmication be re-established) and operate with forwarding capabilities.
Each tunneling sub-port of bridge 400 corresponds (is remoted to) an aggregated sub-port on bridge 420. Similarly, each aggregated sub-port on bride 400 corresponds (is remoted to) a tutmeling sub-port on bridge 420. For example, sub-ports 1 and 3 of bridge 400 are configured to be remoted by sub-ports of bridge 420. In other words, sub-port 1 (or sub-port 2, or sub-port 3) of bridge 400 is coupled to a subport of bridge 420 that is configured to turmel the frames through bridge 420. Similarly, sub-ports 4 and 5, which are configured on bridge 400 as pass-through ports via tunnel engines 410, are remoted to sub-ports of bridge 420, which, on bridge 420, are configured as aggregated bridge ports.
It will be noted that the variable identifier "N" is used in several instances in the figures described herein to more simply designate the final element of a series of related or similar elements. The repeated use of such variable identifiers is not meant to necessarily imply a correlation between the sizes of such series of elements, although such correlation may exist. The use of such variable identifiers does not require that each series of elements has the same munber of elements as another series delimited by the same variable identifier. Rather, in each instance of use, the variable identified by "N" (or any other such identifier) may hold the same or a different value than other instances of the same variable identifier.
Moreover, regarding the signals described herein, those skilled in the art will recognize that a signal may be directly transmitted from a first block to a second block, or a signal may be modified (e.g., amplified, attenuated, delayed, latched, buffered, inverted, filtered or otherwise modified) between the blocks. Although the signals of the above described embodiment are characterized as transmitted from one block to the next, other embodiments of the present invention may include modified signals in place of such directly transmitted signals as long as the informational and/ or functional aspect of the signal is transmitted between blocks. To some extent, a signal input at a second block may be conceptualized as a second signal derived from a first signal output from a first block due to physical limitations of the circuitry involved (e.g., there will inevitably be some attenuation and delay). Therefore, as used herein, a second signal derived from a first signal includes the first signal or any modifications to the first signal, whether due to circuit limitations or due to passage through other circuit elements which do not change the informational and/or final functional aspect of the first signal.
The foregoing described embodiment wherein the different components are contained within different other components (e.g., the various elements of bridge 400). It is to be understood that such depicted architectures are merely examples, and that in fact many other architectures can be implemented which achieve the same functionality. In an abstract, but still definite sense, any arrangement of compo-
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page15 of 19
US 8,051,211 B2 9
nents to achieve the same fnnctionality is effectively "associated" such that the desired fimctionality is achieved. Hence, any two components herein combined to achieve a particular fnnctionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermediate components. Likewise, any two components so associated can also be viewed as being "operably com1ected". or "operably coupled". to each other to achieve the desired fimctionality.
Encapsulation 10
10 between bridge 612 and host 602. with bridge 610 being transparent to both bridge 612 and 602.
As can be seen from the presently described embodiment of FIG. 6, rather than utilizing, inter alia, one inter-bridge LAN between multiple bridges and a bridge inter-cmmect port, the embodiment of FIG. 6 provides one physical connection between each port A2, A3 and B2, B3 respectively, and no encapsulation! de-encapsulation is necessary. Such a configuration may be used in a network with few hosts, mak-ing it easier to physically connect multiple ports via a number of inter-bridge LANs 624. However, for larger configurations, it is preferable to utilize the embodiment illustrated in FIG. 5, which allows for multiple bridges to be coupled with,
15 at a minimum, one inter-bridge link (although more may be used).
FIG. 5 is a block diagram of encapsulated data (e.g., Ethernet frame 500) in accordance with one embodiment of the present invention. In one embodiment of the present invention bridge-interconnect port 412 encapsulates (e.g., includes a tag within, or appended to, frame 500) and de-encapsulates (e.g., examines fields in frame 500 and/or removes tags from frame 500) frame 500 on input buffer logic (IBL) with a multi-bridgetag. It will be recognized that subtle variations of the encapsulation methods described herein may be incorporated with objects other than Ethernet frames. Further, it will 20
be recognized that fields other than those shown in FIG. 5may be included in, or removed from, frame 500.
In the present embodiment, frame 500 includes destination media access control (mac) address 502 and source mac address 504, representing the address of a destination host 25
and the address of a source host offrame 500. Frame 500 also includes tags 506 and 508, length!Ethertype field 510, data field 512 and cyclical rednndancy check (CRC) field 514.
Configuration and Operation FIGS. 7(A-C) are flow charts illustrating a process of
aggregating a plurality of LAN s coupling a host to a first and second intermediate network device, according to one embodiment of the present invention. For clarity of descrip-tion, the flow charts of FIG. 7 are described with reference to FIGS. 3 and 4.
Tag 506 is a multi-bridge tag preferably used to indicate the sub-port number to which frame 500 should be sent. In one embodiment of the present invention, tag 506 is a 4-byte virtual local area network (VLAN) tag as defined in IEEE std 802.1Q-1998. In one embodiment of the present invention, tag 506 includes 2-byte ethertype field 506(a), 3 bit priority field 506(b), 1 bit canonical format indicator (CFI) field 506 (c), and 12-bit VLAN-ID field. It is preferable that etherytype field 506( a) be set to Ox81 00 and priority field 506( b) be set according to the relative priority of the corresponding subport. Additionally, CFI field 506( c) is preferably set to 0 and VLAN-ID field 506(d) field is preferably used to indicate the sub-port of a bridge (e.g., bridge 400) which should transmit/ receive frame 500. Although tag 506 is a VLAN tag, it is not necessary for a bridge or host to be VLAN -aware in order to process tag 506. A tag 508, optionally included in frame 500,
Initially, a tunneling bridge (e.g., a first intermediate device, bridge 342) is configured (step 702). The tunneling bridge is a bridge which provides a pass-through path. A tunneling bridge may also be a home bridge, which is a bridge that is configured for link aggregation with respect to a given host. To configure the tunneling bridge, the virtual ports (e.g.,
30 sub-ports 414) are configured (step 704) and a pass-through path (e.g., pass-through pass 353) is configured (step 706). In configuring the virtual ports, one or more physical ports ofthe tunneling bridge are mapped to a number of virtual ports.
35 Each virtual port may then be mapped to a number of internal destinations with the tum1eling bridge (e.g., bridge forwarding engine 406, aggregated bridge ports 408, or ports 402). To configure the pass-through path, a virtual port is mapped directly to another physical port of the tunneling bridge via a
40 tunnel engine (step 708). For example, referring to FIG. 4 sub-port 4 is mapped directly to port 5 via tunnel engine 410 to provide a pass-through pass from sub-port 4 to port 5. In this configuration, a frame received by sub-port 4 will pref-erably be immediately transmitted directly to port 5.
is also preferably a VLAN tag, however the settings of tag 508 45
are dependent upon, inter alia, the topology of the virtual local area network.
A host (e.g., host 356) is configured in step 710. Initially, to configure the host, a number of network interfaces (e.g., SO and S1) are aggregated (i.e., collectively joined or mapped) to provide a virtual network interface (e.g., network interface 362) (step 712). In one embodiment of the present invention, Alternative Embodiments
50 Link Aggregation is used for the aggregation. Next, an inter-FIG. 6 illustrates another embodiment of the present inven- net protocol (IP) address is generated for the virtual network
tion. A host 602 is coupled to a LAN 604 via LANs 606 and interface (step 714). With the generated IP address, data 608 and bridges 610 and 612. Host 602 includes a number of addressed to the same IP address may be received on either (e.g., 2) network interfaces 614. Together, network interfaces network interface SO or Sl. 614 are represented as one virtual network interface 616 55 In step 716, the home bridge (e.g. a second intermediate (having one IP address for virtual network interface 616). device. bridge 344), is configured (as mentioned above. a Bridge 610 includes ports 618 (e.g., portsAO-A4). Bridge 612 home bridge can also serve as a tmmeling bridge). To config-includes ports 618 (e.g., ports BO-B4). A mm1ber of (e.g., 2) ure the home bridge, the virtual ports (e.g., sub-ports 414) are inter-bridge LANs 624 couple bridge 610 and 612 to each configured (step 718) and the ports (e.g., ports BO and bridge-otherviabridgeportsA2-B2andA3-B3,respectively. 60 interconnect port 366) are aggregated (step 720). In one
PortAO ofbridge 610 is tum1eled to portA3, and portAl of embodiment of the present invention, Link Aggregation is bridge 610 is tunneled to port A2. In other words, bridge 610 used for the aggregation. configures ports AO and AI to be slaved to bridge-inter- As a result of the configuration of the tunneling bridge, cmmect ports A2 and A3 and ports B2 and B3, respectively. Link Aggregation on bridge 344 sees bridge-intercmmect Bridge 612 and host 602 are to aggregate LANs 606 and 608 65 port 366 as directly coupled to host 356, even though bridge and inter-bridge LANs 624. In other words, LANs 606 and 342 is coupled between bridge-intercom1ectport 366 and host 608 and inter-bridge LANS 624 appear as a logical LAN 356. Similarly, Link Aggregation on host 356 sees virtual
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page16 of 19
US 8,051,211 B2 11
network interface 362 as directly coupled to bridge 344, even though bridge 342 is coupled between virtual network interface 362 and host 356.
FIG. 7B is a flow chart illustrating tunneling a frame from
10
12 mabie gate array (FPGA). the design of a gate array or fullcustom application-specific integrated circuit (ASIC), or the like.
Each of the blocks of the flow diagram may be executed by a module (e.g., a software module) or a portion of a module or a computer system user. Thus, the above described method, the operations thereof and modules therefor may be executed on a computer system configured to execute the operations of the method and/or may be executed from computer-readable media. The method may be embodied in a machine-readable and/or computer-readable medium for configuring a computer system to execute the method. Thus, the software modules may be stored within and/or transmitted to a computer
a home bridge (e.g., bridge 344) to a tunneling bridge (e.g., bridge 342), in accordance with one embodiment of the present invention. It is assumed that the destination address of the frame is the IP address corresponding to the virtual network interface of the host. Initially, when a frame is sent from the home bridge to the tum1eling bridge, the frame is encapsulated (step 722). In encapsulating a frame, a tag (e.g., tag 506 of FIG. 5) is generated (step 724) and added to the frame (step 732). In the present embodiment, the tag is generated by using a IEEE Std. 802.1Q-1998. The CFI bit is set to 0, the user priority field is set to the priority of the virtual port, and the VLAN-ID field is set to the virtual port number (steps 726, 728, and 730, respectively). Once the tag is encapsulated, it is transmitted to the tmmeling bridge (step 734).
15 system memory to configure the computer system to perform the functions of the module.
The tnm1eling bridge de-encapsulates the frame (step 736). 20
In de-encapsulating the frame, the virtual port munber is determined (step 738) and the frame is received by the virtual port corresponding to the determined virtual port nnmber (step 740). Next, the frame is transmitted directly to the physical port to which the virtual port is mapped (step 742) 25
(e.g., port 5, which is mapped to sub-port 4) and the frame is then transmitted to the host (step 744). FIG. 7C is a flow chart similar to 7B, but from the perspective oftransmitting a frame from a host to a home bridge (e.g., bridge 344 ).
As noted, FIG. 3 depicts a flow diagran1 illustrating a 30
process according to one embodiment of the present invention. It is appreciated that operations discussed herein may consist of directly entered coll1lllands by a computer system user or by steps executed by application specific hardware modules, but the preferred embodiment includes steps 35
executed by software modules. The functionality of steps referred to herein may correspond to the fimctionality of modules or portions of modules.
The operations referred to herein may be modules or portions of modules (e.g., software, firmware or hardware mod- 40
ules). For example, although the described embodiment includes software modules and/or includes manually entered user commands, the various example modules may be application specific hardware modules. The software modules discussed herein may include script, batch or other executable 45
files, or combinations and/or portions of such files. The software modules may include a computer program or subroutines thereof encoded on computer-readable media.
Additionally, those skilled in the art will recognize that the bmmdaries between modules are merely illustrative and alter- 50
native embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into submodules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, 55
alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, those skilled in the art will recognize that the operations described in exan1ple embodiment are for illustration only. Operations may be combined or the functionality of the operations may 60
be distributed in additional operations in accordance with the invention.
Such a computer system normally processes information according to a program (a list of intemally stored instructions such as a particular application program and/or an operating system) and produces resultant output information via I/0 devices. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall fm1ctionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall functionality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.
Such a computer system typically includes multiple com-puter processes executing "concurrently." Often, a computer system includes a single processing unit which is capable of supporting many active processes altemately. Although multiple processes may appear to be executing concurrently, at any given point in time only one process is actually executed by the single processing unit. By rapidly changing the process executing, a computer system gives the appearance of concurrent process execution. The ability of a computer system to multiplex the computer system's resources among multiple processes in various stages of execution is called multitasking. Systems with multiple processing units, which by definition can support true concurrent processing, are called multiprocessing systems. Active processes are often referred to as executing concurrently when such processes are executed in a multitasking and/or a multiprocessing environn1ent.
The software modules described herein may be received by such a computer system, for example, from computer readable media. The computer readable media may be permanently, removably or remotely coupled to the computer sys-tem. The computer readable media may non-exclusively include, for example, any number of the following: magnetic storage media including disk and tape storage media optical storage media such as compact disk media (e.g., CD-ROM, CD-R. etc.) and digital video disk storage media nonvolatile memory storage memory including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM or application specific integrated circuits volatile storage media including registers, buffers or caches, main memory, RAM, and the like and data transmission media including computer network, point-to-point telecommtmication, and carrier wave transmission media. In a UNIX-based embodiment, the software modules may be embodied in a file
Alternatively, such actions may be embodied in the structure of circuitry that implements such functionality, such as the micro-code of a complex instruction set computer (CISC), finnware progranm1ed into progrannnable or erasable/progrannnable devices, the configuration of a field-program-
65 which may be a device, a terminal, a local or remote file, a socket, a network connection, a signal, or other expedient of conn1mnication or state change. Other new and various types
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page17 of 19
US 8,051,211 B2 13
of computer-readable media may be used to store and/or transmit the software modules discussed herein.
While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled in the art that, based upon the teachings herein, changes and modifications may be made without departing from this invention and its broader aspects and, therefore, the appended claims are to encompass within their scope all such changes and modifications as are within the true spirit and scope of this invention. Furthermore, it is to be understood 1 o that the invention is solely defined by the appended claims.
What is claimed is: 1. A method comprising: aggregating a plurality ofLANs, wherein 15
said aggregating comprises aggregating a first LAN and a second LAN.
said first LAN couples a host to a first intermediate device and said second LAN couples said host to a second intermediate network device, said plurality of 20
LANs comprising said first LAN and said second LAN, and
subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit information from said host to said second inter- 25
mediate network device. 2. The method of claim 1, fi.1rther comprising: tunneling said first LAN with a third LAN through said first
intermediate network device, said plurality of LANs comprising said third LAN. 30
3. The method of claim 2, wherein said aggregating comprises:
aggregating a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter- 35
mediate network device over said first LAN, and said second network interface is coupled to said second
intermediate network device over said second LAN. 4. The method of claim 3, fi.1rther comprising: generating an interne protocol address for said virtual net- 40
work interface; transmitting a first frame addressed to said internet proto
col address from said second intermediate network device via said first LAN; and
transmitting a second frame addressed to said internet pro- 45
taco! address from said second intermediate network device via said second LAN.
5. The method of claim 4, further comprising: transmitting said first and said second frame simulta-
neously. 50
6. The method of claim 2 wherein said aggregating comprises:
aggregating a first and a second port of said second intermediate network device, wherein said first port is coupled to said first intermediate net- 55
work device over said third LAN, and said second port is coupled to said host over said second
LAN. 7. The method of claim 2, wherein said tunneling com-
prises: 60
configuring a pass-through path on said first intermediate network device.
8. The method of claim 7, wherein said tunneling further comprises:
coupling a first port of said first intermediate network 65
device to a second port of said first intermediate network device.
14 9. The method of claim 8, wherein said tum1eling further
comprises: transmitting a frame directly from said first port of said first
intern1ediate network device to said second port of said internwdiate network device.
10. The method of claim 7, fhrther comprising: configuring a plurality of virtual ports on said first inter
mediate network device; and mapping a virtual port of said plurality directly to a physi
cal port of said first intermediate network device. 11. The method of claim 10, wherein said tunneling com
prises: encapsulating a frame with a virtual port number; transmitting said frame to said first intermediate network
device; de-encapsulating said frame to determine said virtual port
number; and directly transmitting said frame from said virtual port to
said physical port. 12. A computer program product, said computer program
product comprising: a non-transitory computer readable media, wherein the
non-transitory computer readable medium stores a first set of instructions, executable on an intermediate network device, configured to aggregate a plurality of LAN s, wherein said first set of instructions is configured to aggregate a first LAN and a
second LAN, said first LAN couples a host to a first intermediate device
and said second LAN couples said host to a second intern1ediate network device, said plurality of LANs comprising said first LAN and said second LAN, and
subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit information from said host to said second intermediate network device.
13. The computer program product of claim 12, further comprising:
a second set of instructions, executable on said intermediate network device, configured to h1m1el a first LAN with a third LAN through said first intern1ediate network device, said plurality of LANs comprising said third LAN.
14. The computer program product of claim 13, wherein said first set of instructions comprises:
a first sub-set of instructions, executable on said intermediate network device, configured to aggregate a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter
mediate network device over said first LAN, and said second network interface is coupled to said second
intermediate network device over said second LAN. 15. The computer program product of claim 14. further
comprising: a third set of instructions, executable on said intermediate
network device, configured to generate an internet protocol address for said virhwl network interface;
a fourth set of instructions. executable on said intermediate network device, configured to transmit a first frame addressed to said internet protocol address from said second intermediate network device via said first LAN; and
a fifth set of instructions, executable on said intermediate network device, configured to transmit a second frame
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page18 of 19
US 8,051,211 B2 15
addressed to said intemet protocol address from said second intermediate network device via said second LAN.
16. The computer program product of claim 15, further comprising:
a sixth set of instructions, executable on said intermediate network device, configured to transmit said first and said second frame simultaneously.
17. The computer program product of claim 13, wherein said first set of instructions comprises: 10
a first sub-set of instructions, executable on said intermediate network device, configured to aggregate a first and a second port of said second intermediate network device, wherein said first port is coupled to said first intermediate net-
work device over said third LAN, and 15
said second port is coupled to said host over said second LAN.
18. The computer program product of claim 13, wherein said second set of instructions comprises:
a first sub-set of instructions, executable on said intem1e- 20
diate network device, configured to provide a passthrough path on said first intennediate network device.
19. The computer program product of claim 18, wherein said second set of instructions further comprises:
a second sub-set of instructions, executable on said inter- 25
mediate network device, configured to couple a first port of said first intermediate network device to a second port of said first intennediate network device.
20. The computer program product of claim 19, wherein said second set of instructions further comprises: 30
a third sub-set of instructions, executable on said intermediate network device, configured to transmit a frame directly from said first port of said first intermediate network device to said second port of said intermediate network device.
21. The computer program product of claim 18, further 35
comprising: a third set of instructions, executable on said intermediate
network device, configured to configure a plurality of virtual ports on said first intem1ediate network device; ~ ~
a fourth set of instructions, executable on said intermediate network device, configured to map a virtual port of said plurality directly to a physical port of said first intennediate network device.
22. The computer program product of claim 21, wherein 45 said second set of instructions comprises:
a first sub-set of instructions, executable on said intermediate network device, configured to encapsulate a frame with a virtual port number;
a second sub-set of instructions, executable on said inter-50
mediate network device, configured to transmit said frame to said first intermediate network device;
a third sub-set of instructions, executable on said intermediate network device, configured to de-encapsulate said frame to determine said virtual port number; and
a fourth sub-set of instructions, executable on said inter- 55
mediate network device, configured to directly transmit said frame from said virtual port to said physical port.
23. A system comprising: means for aggregating a plurality ofLANs; and means for tmmeling a first LAN with a second LAN 60
through said first intermediate network device, said plurality of LANs comprising said first and said second LAN, wherein said aggregating comprises aggregating said first LAN
and a third LAN,
16 said first LAN couples a host to a first intermediate
device and said third LAN couples said host to a second intennediate network device, and
subsequent to said aggregating, said first LAN and said second LAN are both usable to simultaneously transmit infonnation from said host to said second intermediate network device.
24. The system ofclaim23, wherein said aggregating comprises:
aggregating a first and a second network interface of said host to provide a virtual network interface, wherein said first network interface is coupled to said first inter
mediate network device over said first LAN, and said second network interface is coupled to said second
intermediate network device over said third LAN. 25. The system of claim 24, further comprising: means for generating an intemet protocol address for said
virtual network interface; means for transmitting a first frame addressed to said inter
net protocol address from said second intermediate network device via said first LAN; and
means for transmitting a second frame addressed to said intemet protocol address from said second intermediate network device via said third LAN.
26. The system of claim 25, further comprising: means for transmitting said first and said second frame
simultaneously. 27. The system of claim23, wherein said aggregating com
prises: means for aggregating a first and a second port of said
second intermediate network device, wherein said first port is coupled to said first intennediate net
work device over said second LAN, and said second port is coupled to said host over said third
LAN. 28. The system of claim 23, wherein said tunneling com
prises: means for configuring a pass-through path on said first
intermediate network device. 29. The system of claim28, wherein said tmmeling further
comprises: means for coupling a first port of said first intermediate
network device to a second port of said first intermediate network device.
30. The system of claim29, wherein said tunneling further comprises:
means for transmitting a frame directly from said first port of said first intermediate network device to said second port of said intennediate network device.
31. The system of claim 30, further comprising: means for configuring a plurality of virtual ports on said
first intermediate network device; and means for mapping a virtual port of said plurality directly
to a physical port of said first intermediate network device.
32. The system of claim 31, wherein said tunneling comprises:
means for encapsulating a frame with a virtual port number;
means for transmitting said frame to said first intermediate network device;
means for de-encapsulating said frame to detennine said virtual port number; and
means for directly transmitting said frame from said virtual port to said physical port.
* * * * *
Case3:14-cv-05343 Document1-6 Filed12/05/14 Page19 of 19
Exhibit 7
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page1 of 15
c12) United States Patent Welder et al.
(54)
(75)
METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE UPGRADE OR RELOAD OF A NETWORK DEVICE
Inventors: John Thomas Welder, San Jose, CA (US); Ratheesh Krishna Vadhyar, Irinjalakuda (IN); Sudhir Rao, San Jose, CA (US); Thomas W. Uban, Valpraiso, IN (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 158 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 12/852,265
(22) Filed: Aug. 6, 2010
Related U.S. Application Data
(63) Continuation of application No. 10/646,453, filed on Aug. 21, 2003, now Pat. No. 7,814,481.
(51) Int. Cl. G06F 91445 (2006.01) G06F 151177 (2006.01)
(52) U.S. Cl. ............................................ 717/176; 713/2 (58) Field of Classification Search ................... 717/176
See application file for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
5,263,168 A 6,154,849 A 6,314,532 B1 * 6,393,582 B1 * 6,397,385 B1
1111993 Toms et a!. 1112000 Xia 1112001 Daudelin et al ............ 714/38.14 5/2002 Klecka eta!. ................... 714/11 5/2002 Kravitz
111111 1111111111111111111111111111111111111111111111111111111111111 US008356296Bl
(10) Patent No.: US 8,356,296 Bl (45) Date of Patent: *Jan. 15, 2013
6,535,924 B1 6,601,186 B1 6,639,910 B1 6,658,659 B2 6,718,461 B1 * 6,880,086 B2 6,983,362 B1 7,062,642 B1 7,222,147 B1 7,225,244 B2 7,359,377 B1 7,525,973 B1
3/2003 Kwok et al. 7/2003 Fox et a!.
10/2003 Provencher eta!. 12/2003 Hiller eta!. 4/2004 Ewertz .............................. 713/1 4/2005 Kidder et a!. 112006 Kidder et a!. 6/2006 Langrind et al. 5/2007 Black eta!. 5/2007 Reynolds et a!. 4/2008 Kompella et al. 4/2009 McRae
(Continued)
OTHER PUBLICATIONS
Li, V.O.K.; Zaichen Zhang, "Internet multicast routing and transport
control protocols," Proceedings of the IEEE, vol. 90, No. 3, pp. 360-391, Mar. 2002, URL: http://ieeexplore.ieee.org/stamp/stamp. jsp? arnumbeF993404&isnumbeF21426.
(Continued)
Primary Examiner- James D Rutten (74) Attorney, Agent, or Firm- Edwards Wildman Palmer LLP; James M. Behmke; Stephen D. LeBarron
(57) ABSTRACT
A method and system for resetting a network device. Specifically, in one embodiment, a method is disclosed for upgrading and/or reloading software for a network device with minimal disruption. The method begins by separating operations associated with layer two of an International Standardization Organization Open Systems Interconnect (ISO/OSI) reference model from other layers in the ISO/OS I reference model in a network device. Then, the software operations in layer two of the network device are reset. The software operations are reset while maintaining continuity for a communication session between the network device and other network devices coupled together through a network. Thereafter, for minimal disruption, execution of the software operations is recovered at layer two before continuity of the communication session s terminated.
18 Claims, 6 Drawing Sheets
300
Separating Operations Associated with Layer Two of an Open Systems Interconnect (OSI) Seven-Layer Model from Other Layers in the OSI Seven-Layer Model in a Network Device
310
Resetting Software Operations in the Layer Two of the Network Device
Maintaining Continuity for a Communication Session Between the Network Device and Other Network
Devices Coupled Together through a Network
Recovering Execution of the Software Operations at the Layer Two Before the Continuity of the
Communication Session is Terminated
320
330
340
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page2 of 15
U.S. PATENT DOCUMENTS
2003/0084440 A1 2003/0157899 A1 2006/0233182 A1 2006/0242402 A1 2007/0162565 A1
5/2003 Lownes 8/2003 Trossen et a!.
10/2006 Appanna eta!. 10/2006 Swift et a!. 7/2007 Hanselmann
OTHER PUBLICATIONS
US 8,356,296 Bl Page 2
Baumann, Andrew; "Improving Operating System Availability With Dynamic Update", Sep. 14, 2004, 7 pgs. Riverstone Networks Whitepapers, "MPLSNPLS Evolution: A Riverstone Perspective", Nov. 5, 2004, 9 pgs. Bowen et a!., "Availability in parallel systems: Automatic process restart", Feb. 1997, 17 pgs. Roch, Benjamin; "Monolithic kernel vs. Microkernel", Jul. 7, 2004. Stolowitz Ford Cowger LLP, Listing of Related Cases, Sep. 24, 2010.
Gallo et a!. "Networking Explained, Second Edition" Dec. 15, 2001, Digital Press, pp. 42-44. * cited by examiner
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page3 of 15
U.S. Patent Jan.15,2013
w <.9 ~w OO..;tl f->O r--U)w..-<(0 f-<( 0
w .....1
!;;: ~ .....1 C'? I ooo -0::::>..-
I z 0 z
w .....1
~ f- Nl <(<(0 -0:::.....1..-
0 >
0:::: 0 U)
r--~ U) ..-I wo (_)...-
0 0 0:::: N 0....
..,.....
Sheet 1 of 6
~
(I) L.. ::J C) ·-LL
:::s::::W o::::O oiiool so::::o f-W..-Wf-:z z
US 8,356,296 Bl
:::s:::: 0:::: 0 s fw z
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page4 of 15
U.S. Patent Jan.15,2013 Sheet 2 of 6
200
f----. 210 Route
Processor
f----. 220
Line Card • • •
225 -I-- I ~227
235---- r----- 237
Figure 2
US 8,356,296 Bl
Layers 3-7
Layer2
Layer1
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page5 of 15
U.S. Patent Jan. 15,2013 Sheet 3 of 6 US 8,356,296 Bl
Start 300 -
+ Separating Operations Associated with Layer Two of an Open Systems Interconnect (OSI) Seven-Layer Model from Other f.------ 310 Layers in the OSI Seven-Layer Model in a Network Device
+ Resetting Software Operations in the
---------Layer Two of the Network Device 320
+ Maintaining Continuity for a Communication Session
Between the Network Device and Other Network _______... 330 Devices Coupled Together through a Network
+ Recovering Execution of the Software Operations
at the Layer Two Before the Continuity of the f.------ 340 Communication Session is Terminated
+ End
Figure 3
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page6 of 15
U.S. Patent Jan. 15,2013 Sheet 4 of 6 US 8,356,296 Bl
400
Pre-Loading New Software Implementing the Software Operations to a Memory Location, as Indicated by 410
Pointer 525 of Figure 5, of the Network Device
Loading a Bootstrap Operation to another Memory Location, as Indicated by the Pointer 535 of Figure 5, of the Network Device, Wherein the Bootstrap Code is for Loading the New Software to a Predetermined Location, 420
as Indicated by Pointer 515 of Figure 5, the Predetermined Location Storing Existing Software
Implementing the Software Operations
Performing a Minimal Reset of Hardware Components Associated with Layer Two to Minimize Interruptions to an 430 Operating System of the Network Device During Recovery
of the Execution of the Software Operations in 440
Figure 4
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page7 of 15
U.S. Patent Jan.15,2013 Sheet 5 of 6 US 8,356,296 Bl
500 515
l------~------------------------1 I I I I
Program I I ~~ ---.; .. :
1 Current OS Software Image ~~ 510 Counter . _ ) l _______________________________ _
540 525
r------~------------------------1 I
j I New OS Software Image ~ 520 I I
L--------------------------------1
r--------------------------------535 : : "--.J: I I l_ . Bootstrap Code 1: -530
I ~----------------~ I
r--------------------------------1
Figure 5
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page8 of 15
U.S. Patent Jan.15,2013 Sheet 6 of 6 US 8,356,296 Bl
600
Executing the Bootstrap Operation by Moving a Program Counter of the Network Device to a First Beginning Instruction of the Bootstrap Operation to Overwrite Existing Software at
the Predetermined Location with the New Software
Executing the New Software by Moving the Program Counter to a Second Beginning Instruction of the New
Software to Initialize the New Software
Resume Operations of Hardware Components That Were Reset or Suspended in 530
Figure 6
610
620
630
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page9 of 15
US 8,356,296 B 1 1
METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE
UPGRADE OR RELOAD OF A NETWORK DEVICE
CROSS REFERENCE TO RELATED APPLICATION
This application is a continuation of U.S. patent application Ser. No. 10/646,453 filed Aug. 21, 2003, entitled METHOD AND SYSTEM FOR MINIMAL DISRUPTION DURING SOFTWARE UPGRADE OR RELOAD OF A NETWORK DEVICE, the disclosure of which is incorporated herein by reference in its entirety.
BACKGROUND OF THE INVENTION
1. Field of the Invention Embodiments of the present invention relate to the field of
network communications. More particularly, embodiments of the present invention relate generally to reloading and upgrading software in network devices.
2. Related Art A network provides an infrastructure that facilitates com
munication between electronic systems, or computer systems, associated with end users. Internetworking protocols allow for the transfer of communication between two or more individualized networks. As such, smaller, individualized networks, such as, local area networks (LAN), are combined into a larger internetwork cable of providing communication services to an ever increasing number of end users. For example, the Internet is a global interconnection of individual networks.
2 particular network device impacts the capability of an associated network to pass communication.
In particular, delays occurring during the installation of upgrades, or during a reset of software, can be attributed to the loading of the software image code and the resetting of hardware components associated with the network device. Because of these issues, the network device with software to be upgraded is cutoff from an associated network. This leads to termination of current communication sessions held
10 between the network device that is being upgraded and other network devices.
Specifically, prior art methods brought the network device down from the network when installing the software image.
15 The software image comprises the programming code for the upgrade or reset of software that runs a network device. In some cases, the software image is transferred from a secondary device, such as, a hard disk, floppy disk, CD ROM, etc. In other cases, the software image is downloaded from a server
20 through the network. For example, the software images can be loaded from flash cards, disks, or from a trivial file transfer protocol (TFTP) server via a network.
Also, prior art methods required the blanket reset of many hardware components associated with the network device. As
25 such, hardware components are reset during an upgrade to the software, and the network device is rebooted from scratch in order to initialize the upgraded software images.
In both cases, these may lead to the physical layer of a network going down. This effect would spread to upper layers
30 in a network or communication session, resulting in route flaps, traffic loss, etc. These other network devices may also undergo their own software reset as a result of the termination of the communication session, thereby proliferating further delays throughout a network due to network device down-Network devices are hardware devices that facilitate com
munication through a network. For example, the heart of any network includes hubs, switches, routers, access servers, and firewalls. The hub acts as a central point to host computers associated with end users in a local network. As such, the host computers and the hub make up a LAN segment. A switch connects the host computers of a particular LAN to an inter- 40
network of a collection of LANs. The switch provides for a single switched virtual circuit to facilitate communication between two host devices through a larger internetwork across two or more LANs. Access servers connect remote users that are not part of any network to one or more networks. Firewalls act as checkpoints that check message traffic for harmful and unwanted data before routing the traffic through. Routers are intelligent devices that forward message traffic based on the Internet protocol (IP) addresses provided.
35 times. As such, an entire network may be affected because of a software upgrade on a single network device. Since the availability of a network device is critical and software upgrades are necessary, it is important to reduce the downtime of a network device during a software upgrade.
SUMMARY OF THE INVENTION
Accordingly, various embodiments of the present invention disclose a method and system for minimal disruption of
45 communication during the upgrading and resetting of network devices. As a result, the upgrading and reloading of system software of a network device is performed with minimal delay to reduce the downtime of the network device. Moreover, the upgrading and reloading of the system soft-
Upgrades to software that are used for implementing specific features or services as provided by a network device are necessary to capture new features, enhancements, and fixes to programming errors. For example, software upgrades are implemented when customers want or need new and additional features added to their existing software applications. Also, solutions to specific programming errors require an upgrade to their existing software applications. As a result, these new software upgrades are provided for in software images.
However, a significant impact on the availability of a network device occurs when upgrading associated software. In general, hardware associated with the network device needs resetting and the network device requires rebooting to initial-ize upgrades to software associated with a network device. This network device downtime problem occurs also when resetting the network device after a general failure that does not require any software upgrades. As a result, downtime of a
so ware is performed without resetting all of the hardware components of the network device, thereby limiting the impact on the control plane when communicating through a network.
Specifically, in one embodiment, a method is disclosed for resetting a network device that provides for minimal disrup-
55 tion of communication. The method is implemented when upgrading system software associated with the network device, or when reloading the system software upon system failure.
The method begins by separating operations associated 60 with layer two of an International Standardization Organiza
tion Open Systems Interconnect (ISO/OSI) reference model from other layers in the ISO/OSI reference model in a network device. In essence, the data plane is separated from the control plane in the network device. As a result, current com-
65 munication sessions involving the network device upon which a software upgrade or reload at layer two is being performed will continue to be controlled at the control plane
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page10 of 15
US 8,356,296 B 1 3
without being terminated, despite having data forwarded through layer two being suspended.
The method continues by resetting the software operations at layer two of the network device. The software operations are reset/suspend while maintaining continuity for a communication session between the network device and other network devices coupled together through a network. Resetting is implemented by loading the software image that implements the software operations to main memory of the network device. This is accomplished before initiating the actual 10
upgrading process. In addition, a minimum reset of hardware components that
are associated with the network device is performed. Reset/ suspend of hardware components is accomplished to remove interruptions to the operating code of the network device. As 15
such, these interruptions are removed before transferring to the software image that previously was loaded during the upgrading process. In this way, interruptions from those hardware components will not occur when the operating system of the network device is down during the upgrading process. 20
Thereafter, for minimal disruption, execution of the software operations is recovered at layer two before continuity of the communication session is terminated. This is accomplished by executing a bootstrap code. Bootstrap code is loaded to the main memory before initiating the upgrade/ 25
reload process. The bootstrap code performs a raw copy of the software image to a predetermined location in memory. The predetermined location comprises the code for implementing the software operations. During the raw copy, the software image overwrites any code, e.g., the old software image, at the 30
predetermined location in memory.
BRIEF DESCRIPTION OF THE DRAWINGS
4 device with minimal disruption of communication, examples of which are illustrated in the accompanying drawings.
Accordingly, various embodiments of the present invention provide for the upgrading and reloading of system software of a network device that is performed with minimal delay to reduce the downtime of the network device. Moreover, the upgrading and reloading of the system software is performed without resetting all of the hardware components of the network device, thereby limiting the impact on the control plane when communicating through a network. As an advantage, embodiments of the present invention are able to upgrade and reload system software on a network device without terminating the continuity of communication sessions between the network device and other network devices in a network. Furthermore, the system software is reloaded or upgraded without dropping any communication sess10ns between end users within a network.
NOTATION AND NOMENCLATURE
Referring now to FIG. 1, portions of the present invention are comprised of computer-readable and computer-executable instructions which reside, for example, in computerreadable media of an electronic system that are networked devices, such as, a server computer, mainframe, networked computer, workstation, hub, router, switch, firewall, access server, and the like. FIG. 1 is a block diagram of interior components of an exemplary electronic system 100, upon which embodiments of the present invention may be implemented.
While embodiments of the present invention are described within the context of networked devices, other embodiments of the present invention are well suited to implementations within any electronic device. More specifically, other
FIG. 1 is a block diagram of an electronic device that is capable of upgrading and/or reloading system software with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.
35 embodiments of the present invention are well suited for methods and systems of upgrading and/or reloading system software on any electronic device.
Exemplary electronic system 100 includes an address/data bus 120 for communicating information, a central processor
FIG. 2 is a block diagram of a network device that is capable of upgrading and/or reloading system software with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.
40 101 coupled with the bus 120 for processing information and instructions, a volatile memory 102 (e.g., random access memory (RAM), static RAM dynamic RAM, etc.) coupled with the bus 120 for storing information and instructions for the central processor 101, and a non-volatile memory 103
FIG. 3 is flow chart illustrating steps in a computer implemented method for upgrading and/or reloading software for a network device with minimal disruption to communication sessions involving the network device, in accordance with one embodiment of the present invention.
45 (e.g., read only memory (ROM), programmable ROM, flash memory, EPROM, EEPROM, etc.) coupled to the bus 120 for storing static information and instructions for the processor 101.
FIG. 4 is a flow chart illustrating steps in a computer 50
implemented method for resetting software operations in a network device, in accordance with one embodiment of the present invention.
FIG. 5 is a diagram of memory illustrating the storage of existing software, new software, and a bootstrap code for 55
replacing the existing software with the new software. FIG. 6 is a flow chart illustrating steps in a computer
implemented method for initializing and executing the new software with minimal disruption to communication sessions, in accordance with one embodiment of the present 60
invention.
DETAILED DESCRIPTION OF THE INVENTION
Exemplary electronic system 100 also includes an optional data storage device 104 (e.g., memory card, hard drive, etc.) coupled with the bus 120 for storing information and instructions. Data storage device 104 can be removable. With reference still to FIG. 1, a network interface 108 (e.g., signal input/output device) is provided which is coupled to bus 120 for providing a communication link between electronic system 100 and a network enviroument. As such network inter-face 108 enables the central processor unit 101 to communicate with or monitor other electronic systems or coupled to a communication network.
Some portions of the detailed descriptions which follow are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by
Reference will now be made in detail to the preferred embodiments of the present invention, a method and system of upgrading and/or reloading system software on a network
65 those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer executed step, logic block, process,
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page11 of 15
US 8,356,296 B 1 5 6
Returning back to FIG. 2, the network device 200 also comprises a plurality of layer two devices. In the configuration of FIG. 2, the plurality oflayer two devices comprises a plurality of add-on modules that are plugged into the network device 200. These add-on modules facilitate networking within a particular networking environment, and as such, are defined as network interface modules, or line cards. For example, in network device 200, the line card 220 is a layer two device. Line card 220, for instance, is an Ethernet card for
etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physi-
10 facilitating the Ethernet networking protocol. In addition, the network device 200 comprises one or more layer two devices in various embodiments. cal quantities and are merely convenient labels applied to
these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that 15
throughout the present invention, discussions utilizing terms such as "performing," "separating," "resetting," "maintaining," "recovering," and "loading," and "executing," or the like, refer to the action and processes of a computer system, or similar electronic computing device, including an embedded system, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories
Communicatively coupled to the line card 220 is a plurality of port adapters. For example, line card 200 comprises port adapter 225 and port adapter 227. Each of the port adapters 225 and 227 provide an interface through which message traffic is transferred between the network device 200 and other devices on the network in a communication session.
20 That is, the port adapters 225 and 227 provide an interface for connection to external network connections. While FIG. 2 discloses two port adapters associated with line card 220, other embodiments of the present invention are well suited to line cards comprising one or more port adapters.
or registers or other such information storage, transmission or 25
display devices. The network device 200 is capable of upgrading and/or
reloading software operations associated with the layer two devices while minimally disrupting the communication occurring at layers other than layer two in the ISO/OS I reference model. That is, the software operations at layer two of the network device is upgraded and/or reloaded without terminating the communication session at layers other than layer
Method and System for Minimal Disruptive Restart Referring now to FIG. 2, a block diagram of a network
device 200 that is capable of upgrading and/or reloading hardware with minimal disruption to communication ses- 30
sions is disclosed, in accordance with one embodiment of the present invention. FIG. 2 is an exemplary diagram of the implementation of the International Organization for Standardization Open Systems Interconnection (ISO/OSI) reference model within the network device 200. The ISO/OSI 35
reference model provides for a seven-layered architecture within network devices that standardizes interconnections between communicating network devices through a network.
two of the ISO/OSI reference model. Lines 235 and 237 comprise the physical layer, or layer
one, of the ISO/OSI reference model for the network device 200. The physical layer, layer one, defines the communication medium by which message traffic is transferred through the port adapters of the network device 200. For example, the communication medium may be defined by electrical, mechanical, or optical characteristics. As such, the commu-As described previously, the network device 200 is any
electronic system that facilitates communication between end users in a network. For example, in embodiments of the present invention, the network device 200 is a hub, or a switch, or a router, or a server, or an access server, or a firewall, etc.
40 nication medium is comprised of, for example, twisted-pair cabling, fiber-optic cabling, coaxial cabling, etc.
For purposes of illustrating the methods and systems of the present Application, the network device 200 is divided into three parts. Each of the parts of the network device 200 is associated with one or more layers in the ISO/OS I reference model.
In the network device 200, the route processor 210 provides control plane functionality during communication sessions involving the network device 200. In relation to the ISO/OSI reference model, the route processor 210 provides functionality for layers three, four, five, six, and seven of the ISO/OS I reference model, or the network, transport, session, presentation, and application layers, respectively in the network device 200.
Now referring to FIG. 3, a flow chart 300 is disclosed illustrating steps in a computer implemented method for resetting a network device with minimal disruption to com-
45 munication sessions involving the network device, in one embodiment of the present invention. The method illustrated in flow chart 300 is implemented when upgrading software associated with layer two components in the network device, in one embodiment. In another embodiment, the method
50 illustrated in flow chart 300 is implemented when reloading software associated with layer two components in the network device, as when correcting a system failure.
At 310, the present embodiment begins by separating operations associated with layer two of the ISO/OS I refer-
55 ence model from other layers in the ISO/OSI reference model. That is, the data plane associated with layer two is separated from the control plane in the network device. The data plane is associated with layer two of the ISO/OSI reference model and controls the forwarding of traffic across the
More specifically, the control plane of the network device 200 provides for keeping the network device 200 in the network. For example, in a network device 200 that is a router, the control plane provides the routing protocol layer to facilitate the routing of message traffic through the router. In addition, the control plane establishes and terminates connections for communication sessions between computers associated with end users. For example, the control plane will terminate 65
a communication session after a predetermined time-out period.
60 network device through the network device. The control plane is associated with layers 3-7 of the ISO/OSI reference model. The control plane controls and maintains communication sessions with other network devices when transmitting message traffic.
In addition, in another embodiment, once the data plane is separated from the control plane, the data plane is also effectively separated from the physical layer of the network
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page12 of 15
US 8,356,296 B 1 7
device. That is layer two, the data plane, of the network device is separated from layer one, the physical layer, of the network device.
By separating the data plane from the control plane in the network device, the present embodiment creates an environment in which changes to the data plane are implemented with minimal impact on the control plane. Most importantly, software located at the data plane can be upgraded and/or reloaded without affecting the continuity of communication sessions managed by the control plane.
At 320, the present embodiment continues by resetting software operations in layer two of the network device. During the reset, the actual upgrade or reload of the software that implements the software operations is postponed, as will be fully described later in relation to FIG. 4. In this manner, minimal downtime is achieved when upgrading or reloading the new software.
At 330, the present embodiment continues by maintaining continuity for a communication session between said network device and other network devices coupled together through a network. As stated previously, by separating the data plane from the control plane, communication sessions can be maintained effectively even though the data plane is down due to upgrading or reloading of software. That is, the upgrading and/or reloading of new software at layer two of the network device is transparent to end users transmitting and receiving message traffic through communication sessions involving the network device. As such, the continuity of the communication session is maintained.
At 340, the present embodiment recovers execution of the software operations at layer two of the network device. Recovery is achieved before continuity of the communication sessions is terminated. That is, the new software that implements the software operations at layer two is initialized and operational before a time-out period occurs at the control plane and the physical layer.
In this way, an upgrading and/or reloading of the software
8 At 420, the present embodiment continues by loading a
bootstrap code (e.g. bootstrap code 530) to another memory location, as indicated by pointer 535 ofFIG. 5, of the network device. Execution of the bootstrap code occurs during the upgrade process implemented in 430. As such, execution of the bootstrap code comprises executing instructions loading the new software to a predetermined location, as indicated by pointer 515, in the memory of the network device. More specifically, the execution of the bootstrap code executes
10 instructions that performs a raw copy of the new software from the memory location, as indicated by pointer 525, to a designated and predetermined location, as indicated by pointer 515, as per design of the system.
The predetermined location, as indicated by pointer 515, 15 stores the instructions in the memory of the network device
for implementing the software operations associated with the network device. At present, an existing software for implementing software operations associated with the network device is located at the predetermined location. This boot-
20 strap code replaces the existing software with the new software to implement the software operations associate with the network device. The new software contains upgraded features to be implemented in the software operations, in one embodiment. The new software is a working copy of the existing
25 software, in another embodiment, and is reloaded when the existing software or network device fails.
FIG. 5 illustrates a memory 500 of the network device after implementing blocks 410 and 420 of flow chart 400. As a result, the memory 500 comprises storage of the current oper-
30 ating system (OS) software image in a predetermined location 510. In addition, the new OS image is located at a first location 520 in the memory 500. Also, the bootstrap code is located at a second location 530 in the memory 500.
FIG. 5 also illustrates the position of the program counter 35 540. The program counter comprises a memory register that
defines the address of the next instruction for execution of a
at layer two of the network device is achieved without disrupting the communication sessions at other layers of the ISO/OSI reference model, namely the control plane (layers 40
3-7) and the physical layer (layer 1 ).As such, since continuity
program sequence. The program counter is pointed at the current, or existing, OS software image, since the update process or reload process of the software associated with the network device has not been installed and initiated. As such, the network device is still operating under the existing OS
at the control plane and the physical layer is maintained, i.e., continuity of communication sessions, the control plane and the physical layer need not undergo a reset process, even though software at layer two of the network device is 45
upgraded or reloaded. Referring now to FIG. 4, a flow chart 400 is described
illustrating steps in a computer implemented method for upgrading and/or reloading software at layer two of a network device, in accordance with one embodiment of the present 50
invention. Specifically, the flow chart 400 discloses the resetting of software operations in layer two of the ISO/OS I reference model in the network device. In one embodiment, the flow chart 400 is an extension of block 320 in FIG. 3.
software image. Returning back to FIG. 4, at 430, the embodiment illustrat
ing flow chart 400 continues by performing a minimal reset of hardware components associated with layer two of the OSII ISO reference model in the network device. The minimal reset is performed to minimize interruptions to the operating system of the network device during recovery of the execution of software operations performed in block 340 of flow chart 300.
The reset of hardware components is unavoidable forcertain reasons. Many hardware components generate interrupts during operation. These interrupts access certain portions of the software operations that is currently being upgraded or
At 410, the present embodiment begins by pre-loading new software (e.g., software 520 of FIG. 5) to a memory location, as indicated by pointer 525 of FIG. 5 of the network device. The new software, after loading and initialization, implements the software operations associated with the network device. In one embodiment, the new software comprises a software image that implements the software operations. The new software is preloaded to the main memory of the network device before initializing the actual process for upgrading or reloading the software at layer two of the network device. In this way, the delay due to downtime of the network device when transferring the new software to the network device from a secondary source or device is eliminated.
55 reloaded. However, during the recovery process of the software performed in block 340, the software operations are not available. This may result in unexpected behavior, e.g., further failures. The present embodiment is able to avoid these unexpected behaviors by addressing the interrupts before the
60 actual upgrade or reload during the recovery of the execution of the software operations is performed in block 340.
Different hardware components will have different modes of reset. For example, a hardware component can have a soft (warm) reset, a hard (cold), or even a suspend operation, etc.
65 Determining the types of resets performed and which of the hardware components need resetting is accomplished to remove interruptions to the software operations, as previ-
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page13 of 15
US 8,356,296 B 1 9
ously described. For minimal impact, the minimum number of reset to hardware components is performed to reduce the downtime associated when updating the software at layer two of a network device.
10 structures from the known regions of memory. In this way, the data and the table information is reacquired.
In another embodiment of the present invention, the operating code of the network device is reloaded with minimal delay when the new operating code does not include any software upgrades. The present embodiment maintains a copy of the operating code in memory. In the case of software failure, the operating code remains unchanged. In that case, the operating code does not need copying. As such, the
The hardware components comprise the devices supporting layer two of the ISO/OS I reference module of the network device. For example, in one embodiment, the hardware components are line cards. In other embodiments, the hardware components are contained within a specific line card, upon which the software is being upgraded or reloaded. 10 present embodiment re-initializes the data, as previously
described, and then restarts the control code. Turning now to FIG. 6, a flow chart 600 is described illus
trating steps in a computer implemented method for upgrading and/or reloading software at layer two of a network device, in accordance with one embodiment of the present invention. Specifically, the method of flow chart 600 further describes the recovery process of the new software that implements the software operations of the network device. In one embodiment, the method of flow chart 600 is an extension of block 430 of flow chart 400. As such, the new software has 20
been preloaded, and the bootstrap code has also been preloaded.
Accordingly, embodiments of the present invention are capable of upgrading system software (e.g., software operations for a network device) with minimal delay. This reduces
15 the resultant downtime of a network device. Other embodi-
At 610, the present embodiment begins by moving a program counter of the network device to start execution of the bootstrap code. That is, the present embodiment moves the 25
program counter to the beginning instruction of the bootstrap code. The bootstrap code overwrites the existing software with the new software that was previously pre-loaded in block 410 of flow chart 400. The existing software is located at the predetermined location of the memory, as previously dis- 30
cussed. The new software was previously pre-loaded and is located at the first memory location.
ments are capable of reloading system software (without loading any software upgrades) with minimal delay, in the case of system failure. As a result, embodiments of the present invention are capable of bringing back up to operation a network device without resetting all the hardware components in the network device. In addition, embodiments of the present invention are capable of upgrading and/or reloading system software while limiting the impact on the control plane of the network device.
While the methods of embodiments illustrated in flow charts 300, 400, and 600 show specific sequences and quantity of steps, the present invention is suitable to alternative embodiments. For example, not all the steps provided for in the method are required for the present invention. Furthermore, additional steps can be added to the steps presented in the present embodiment. Likewise, the sequences of steps can be modified depending upon the application.
Embodiments of the present invention, a method and system for upgrading and/or reloading software at layer two of
35 the ISO/OSI reference model of a network device with mini-
At 620, the present embodiment continues by executing the new software. Execution of the new software initializes the new software, and implements the software operations of the network device. The present embodiment executes the new software by moving the program counter to the beginning instruction of the new software. By initializing and executing the new software, the software operations can be upgraded, or reloaded upon failure, with minimal disruption to communi- 40
cation sessions through the network.
mal disruption to communication is described. While the invention is described in conjunction with the preferred embodiments, it is understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the detailed description of the present invention, numerous specific details are set forth in order to provide
At 630, the present embodiment continues by resuming operations of the hardware components. The hardware components were previously reset or suspended when performing the minimal reset of block 430 of flow chart 400. 45 a thorough understanding of the present invention. However,
it will be recognized by one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as
In one embodiment, data is recovered after an upgrading or reloading of the software located at layer two of the ISO/OS I reference model. In the data portion of the software operations, there are two types of data: data that requires re-initialization after an upgrade or reload, and data that does not require re-initialization after an upgrade or reload. For example, certain counters comprise data that requires reinitialization. Also, routing tables comprise data that does not need re-initialization. Data of the first type or the second type typically is lost during an upgrade of the software associated 55
with layer two of the ISO/OS I reference model. In that case, the data would need to be regenerated.
50 not to urmecessarily obscure aspects of the present invention.
However, in the present embodiment, versioning for data and tables that require carry over overcome the limitation of losing data during when upgrading or reloading software at 60
layer two of the ISO/OS I reference model. Specifically, the present embodiment includes a mechanism to copy the relevant tables and data structures to known regions of the memory of the network device before upgrading or reloading software of the network device. Later, after the upgrading or 65
reloading of the software of the network device, are-initialization routine is performed to read the saved tables and data
What is claimed is: 1. A method comprising: identifYing data for re-initialization after a software reset,
wherein the data is different from software to be run for the software reset;
copying the data to a predetermined region of memory to store during the software reset;
initiating the software reset of a network device during a communication session including the network device and one or more other network devices;
temporarily suspending software operations associated with one or more data plane components of the network device during the software reset;
maintaining the continuity of the communication session between the network device and the one or more other network devices during the software reset;
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page14 of 15
US 8,356,296 B 1 11
recovering execution of the software operations before the communication session is terminated; and
running a reinitializing routine to reacquire the data. 2. The method of claim 1, wherein the data plane corre
sponds to International Standardization Organization Open Systems Interconnect (ISO/OS I) reference model layer two.
3. The method of claim 2, further comprising separating the data plane from a control plane of the network device, wherein the control plane is associated with ISO/OSI layer three.
4. The method of claim 1, wherein temporarily suspending the software operations comprises temporarily suspending a transfer of data through the one or more data plane components.
5. The method of claim 1, further comprising:
10
15
12 identifYing data for re-initialization after a software reset,
wherein the data is different from software to be run for the software reset;
copying the data to a predetermined region of memory to store during the software reset;
initiating the software reset of a network device during a communication session including the network device and one or more other network devices;
temporarily disabling software operations associated with one or more data plane components of the network device during the software reset;
maintaining the continuity of the commnnication session between the network device and the one or more other network devices during the software reset;
recovering execution of the software operations before the communication session is terminated; and
running a reinitializing routine to reacquire the data. 13. The memory device of claim 12, further comprising
preloading a bootstrap program that implements the software operations prior to initiating the software reset; and
executing the bootstrap program during the software reset. 6. The method of claim 1, wherein the software reset com
prises an upgrade process of the software operations. 7. A network device comprising: one or more interfaces configured to transfer messages
between the network device and one or more other net-
20 separating the data plane from a control plane of the network device, wherein the data plane corresponds to International Standardization Organization Open Systems Interconnect (ISO/OSI) reference model layer two, and wherein the con-
work devices during a communication session; one or more data plane components commnnicatively 25
coupled to the one or more interfaces; and a processor configured to:
identify data for re-initialization after a software reset, wherein the data is different from software to be run for the software reset;
copy the data to a predetermined region of memory to store during the software reset;
initiate the software reset during the communication session;
30
temporarily suspend software operations associated 35
with the one or more data plane components during the software reset;
maintain the continuity of the communication session between the network device and the one or more other network devices during the software reset;
recover execution of the software operations before the communication session is completed; and
40
trol plane is associated with ISO/OS I layer three. 14. The memory device of claim 13, wherein the operations
further comprise: maintaining continuity in the commnnication session at
ISO/OS I layer one during the software reset, wherein the data plane is logically separated from ISO/OSI layer one in the network device.
15. The memory device of claim 13, wherein the operations further comprise:
maintaining continuity in the communication session at the control plane during the software reset, wherein the data plane is logically separated from the control plane in the network device.
16. The memory device of claim 12, wherein the operations further comprise:
loading the software operations in the data plane, wherein data plane functionality associated with the software operations is temporarily disabled during the loading of the software operations. run a reinitializing routine to reacquire the data.
8. The network device of claim 7, wherein the data plane corresponds with International Standardization Organization Open Systems Interconnect (ISO/OS I) reference model layer two.
17. The memory device of claim 16, wherein the operations 45 further comprise:
9. The network device of claim 8, wherein the data plane is logically separated from a control plane of the network device, and wherein the control plane is associated with ISO/ 50
OSI layers three, four, five, six, and seven. 10. The network device of claim 7, wherein the processor is
further configured to execute a bootstrap program to implement the software operations during the software reset.
11. The network device of claim 7, wherein the processor is 55
further configured to initiate the software reset in response to a failure of the network device.
12. A memory device having stored thereon computerexecutable instructions that, in response to execution by a processor, cause the processor to perform operations com- 60
prising:
maintaining the commnnication session between the network device and the other network device while the data plane functionality associated with the software operations is temporarily disabled.
18. The memory device of claim 12, wherein the operations further comprise:
executing a pre-loaded bootstrap program to implement the software operations during the software reset, wherein the software reset is initiated in response to a failure of the network device; and
loading the software operations stored in memory of the network device to reinitialize the data plane components.
* * * * *
Case3:14-cv-05343 Document1-7 Filed12/05/14 Page15 of 15
Exhibit 8
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page1 of 19
(12) United States Patent Harvey et al.
(54) METHOD OF REVERTING TO A RECOVERY CONFIGURATION IN RESPONSE TO DEVICE FAULTS
(75) Inventors: Andrew G. Harvey, Pleasanton, CA (US); John Ng, Fremont, CA (US); Gilbert R. Woodman, III, San Jose, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the tenn of this patent is extended or adjusted under 35 U.S.C. 154(b) by 461 days.
(21) Appl. No.: 10/792,946
(22) Filed: Mar. 3, 2004
(51) Int. Cl. G06F 11100 (2006.01)
(52) U.S. Cl. .......................................................... 714/2 (58) Field of Classification Search .................... 714/2,
(56)
714/19, 4; 709/220, 227 See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
5,050,070 A 9/1991 Chastain et al. 5,150,464 A 9/1992 Sidhu eta!. 5,274,771 A 12/1993 Hamilton et a!. 5,351,040 A * 9/1994 Matsuura et a!. ........... 370/223 5,778,323 A * 7/1998 Dorenbosch et a!. ....... 455/561 5,948,077 A 9/1999 Choi eta!. 6,363,499 B1 * 3/2002 Delo eta!. .................... 714/15 6,430,541 B1 8/2002 Brown eta!. 6,438,749 B1 * 8/2002 Chamberlain ............... 717 I 174 6,523,070 B1 2/2003 Stapleton et a!. 6,587,827 B1 7/2003 Hennig eta!. 6,647,510 B1 * 1112003 Ganesh eta!. ................ 714/16 6,654,726 Bl 1112003 Hanzek 6,769,074 B2 * 7/2004 Vaitzblit ...................... 714/16
111111 1111111111111111111111111111111111111111111 111111111111 US007290164Bl
(IO) Patent No.: US 7,290,164 Bl Oct. 30, 2007 (45) Date of Patent:
6,907,602 B2 * 6/2005 Tsai eta!. ................... 717/168 7,017,155 B2 * 3/2006 Peev eta!. .................. 717/176
200210001307 A1 112002 Nguyen eta!. 2002/0133573 A1 9/2002 Matsuda et a1. 2003/00097 52 A1 * 1/2003 Gupta ........................ 717/171 2003/0061315 A1 3/2003 Jin 2003/0061320 AI 3/2003 Grover eta!. 2003/0108039 A1 * 6/2003 Shell eta!. ................. 370/389 2003/0121033 A1 * 6/2003 Peev eta!. .................. 717/175
FOREIGN PATENT DOCUMENTS
GB 2006/010952 * 212006
OTHER PUBLICATIONS
Netility Corporation, "Netility Makes Patent-Pending Auto-Configuration and Auto-Update Technology Available for Licensing," http:/ /www.netility.com/technology-licensing~pr.html, printed Sep. 2, 2003.
(Continued)
Primary Examiner~Bryce P. Bonzo Assistant Examiner~Amine Riad (7 4) Attorney, Agent, or Firm~ Hickman Palenno Truong & Becker LLP
(57) ABSTRACT
A method is disclosed for reverting to a recovery configuration in response to device faults. A change to the configuration is received. The change may be in the fonn of configuration instructions that comprise input from a user identifying changes to be made to the configuration information reflecting the configuration of cards or interface devices in the device. A user, an IT administrator or the like can provide configuration instructions. The device may change its current configuration to a new configuration based upon the configuration instructions. If a loss of connectivity resulting from the configuration change is detected, the device will recover from the loss of cmmectivity by reverting to a recovery configuration.
38 Claims, 8 Drawing Sheets
m I
I
I
I
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page2 of 19
US 7,290,164 Bl 2
OTHER PUBLICATIONS
Cisco Systems, Inc., "Programmable Network Solutions, Business and Operational Challenges," White Paper, 1992-2002, pp. 1-7. Cisco Systems, Inc., "Sigma and Cisco Systems Service Fulfillment Offering for Advanced IP Services," 1992-2001, pp. 1-6. Cisco Systems, Inc., "Cisco DSL Manager, Maintaining System Performance," 1992-2003, http://www.cisco.com/en/US/products/ sw/netmgtsw/ps826/product_user_guide_ chapter09186ao0, data retrieved Oct. 22, 2003, pp. 1-8. Cisco Systems, Inc., "Simple Network-Enabled Auto-Provisioning for Cisco IAD2420 Series lADs," undated, pp. 1-26. Norte! Networks, ''Passport 6400 Express Manager," Product Brief, 2000, pp. 1-4. Netility Corporation, "Products and Technology Overview," http:// \VWw.netility.cOJn!pasover.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "3400 Business Gateway Product Overview," http://www.netility.com/3400_overview.html, printed Sep. 2, 2003, pp. 1-4.
Netility Corporation, "Netility Annouuces First Auto-Configuring G.SHDSLCPE," Dec. 11, 2001, http//www.netility.com/3400_pr. html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "Frequently Asked Questions," http://www. netility.cornifaq.html, printed Sep. 9, 2003, pp. 1-3. Netility Corporation, "Products/Solutions Overview," http://www. netility.corn/3100_overview.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "3100 Product Overview," undated, 3 pages. Netility Corporation, "Network Access Device User's Guide, Model 3100," Jun. 5, 2001, pp. 1-64. Netility Corporation, "Netility Redefines SDSL Deployment with First Auto-Configuring CPE," Oct. 1, 2001, http://www.netility. corn/3100_pr.html, printed Sep. 2, 2003, pp. 1-2. Netility Corporation, "Netility Technology Overview," http://www. netility.cornitechnology_overview.html, printed Sep. 2, 2003, 1 page, copy not provided. Netility Corporation, "Netility Configuration System," http:/ /wwrw. netility.corninc.html, printed Sep. 2, 2003, 1 page.
* cited by examiner
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page3 of 19
CPE A .l.ill
l.l.l.QA INITIAL/BOOTSTRAP CONFJG I
g I m RUNNING coNFIG j
l.l.!lC. RECOVERY CONFIG I
Fig.l
~
AGGREGATOR l..W_
I \
g
~- - \
NETWORK C .lQ5.
e . rJl . ~ ~ ~ ~
= ~
0 (")
:-+-(.H
~0 N 0 0 -l
ILl =('D ('D --0 -QO
d rJJ -.l 'N '-C ... = ,......... 0', .. = ,.........
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page4 of 19
U.S. Patent Oct. 30, 2007 Sheet 2 of 8
Fig. 2A
START
.2.Q2
RECEIVE CONFIGURATION INSTRUCTIONS
2.04 CHANGE CURRENT CONFIGURATION TO NEW
CONFIGURATION BASED UPON CONFIGURATION INSTRUCTIONS
(FIG. 2B)
2.QQ DETECT LOSS OF CONNECTIVITY RESULTING FROM
CONFIGURATION CHANGE (FIG. 2C)
21Q RECOVER FROM LOSS OF CONNECTIVITY BY
REVERTING TO A RECOVERY CONFIGURATION (FIG. 20)
US 7,290,164 Bl
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page5 of 19
U.S. Patent Oct. 30, 2007 Sheet 3 of 8
Fig. 2B 2.12
TEST IF CONFIGURATION INSTRUCTIONS REQUIRE CHANGE TO CURRENT CONFIGURATION
2.1fi
US 7,290,164 Bl
NO
CHANGE CURRENT CONFIGURATION TO NEW CONFIGURATION BASED UPON CONFIGURATION INSTRUCTIONS
ill SET STATE TO "PENDING COMMIT"
RETURN
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page6 of 19
U.S. Patent Oct. 30, 2007 Sheet 4 of 8 US 7,290,164 Bl
Fig. 2C
222. SEND A TEST MESSAGE USING THE NETWORK
m START TIMER
RETURN .EAJ.L
ElG.._2A
YES
23.Q CLEAR TIMER
231 CHANGE STATE TO CONFIRM COMMIT
RETURN SUCCESS
ElG.._2A
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page7 of 19
U.S. Patent Oct. 30, 2007 Sheet 5 of 8 US 7,290,164 Bl
Fig. 2D 23.2
RETRIEVE RECOVERY CONFIGURATION FROM PERSISTENT STORAGE
,, m
CLEAR OLD CONFIGURATION STATE FROM THE DEVICE
,, 2.3.4.
MAKE RECOVERY CONFIGURATION THE CURRENT CONFIGURATION FOR THE DEVICE
, 23.Q
ESTABLISH CONNECTIVITY TO A CONFIGURATION MANAGER
,, 2.3.8.
RECEIVE NETWORK LEVEL CONFIGURATION FROM CONFIGURATION MANAGER
, 24.Q
MAKE NETWORK LEVEL CONFIGURATION CURRENT CONFIGURATION FOR THE DEVICE
,
RETURN
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page8 of 19
U.S. Patent Oct. 30,2007 Sheet 6 of 8 US 7,290,164 Bl
Fig. 3A
START
30.2.a RECEIVE CONFIGURATION REQUEST
3Ma. SEARCH DATABASE FOR CONFIGURATION
INFORMATION CORRESPONDING TO DEVICE MAKING THE REQUEST
31Qa PROVIDE NETWORK LEVEL CONFIGURATION FOR
DEVICE
RETURN
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page9 of 19
U.S. Patent
YES
llQb.
Oct. 30, 2007 Sheet 7 of 8
Fig. 3B
~ RECEIVE CONFIGURATION REQUEST
3Q4l2
SEARCH DATABASE FOR CONFIGURATION INFORMATION CORRESPONDING TO DEVICE
MAKING THE REQUEST
.3.0..6.b. DETERMINE WHETHER DEVICE IS A NEW DEVICE OR
A DEVICE RECOVERING FROM A DEVICE FAULT
US 7,290,164 Bl
NO
.3.12.b. PROVIDE NETWORK LEVEL
CONFIGURATION FOR RECOVERING DEVICE
PROVIDE NETWORK LEVEL CONFIGURATION FOR NEW DEVICE
RETURN
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page10 of 19
FIG. 4 ------------------------1
MAIN MEMORY
4QQ
PROCESSOR
4.0!
TERMINAL ill
ROM
40a
BUS
COMMUNICATION INTERFACE
ill
STORAGE DEVICE
4.1.0
4.02
SWITCHING SYSTEM lli
400_
SERVER 4.30.
414
HOST
ill
e . rJl . ~ ~ ~ ~
= ~
0 (")
:-+-(.H
~0 N 0 0 -l
ILl =('D ('D -QO
0 -QO
d rJJ -.l 'N '-C ... = ,......... 0', .. = ,.........
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page11 of 19
US 7,290,164 Bl 1
METHOD OF REVERTING TO A RECOVERY CONFIGURATION IN RESPONSE TO
DEVICE FAULTS
FIELD OF THE INVENTION
2 to recognize security related parameters do not track the history of these parameters. Such systems will not recognize a previous password or an outdated certificate as an attempt to reestablish connectivity after a device fault or corruption of a configuration file.
Based on the foregoing, there is a clear need for improved recovery capabilities for remotely installed devices to recover from a device fault or a corrupted device configuration.
The present invention generally relates to deploying network devices. The invention relates more specifically to a method of reverting to a recovery configuration in response to device faults.
10 BRIEF DESCRIPTION OF THE DRAWINGS
BACKGROUND
The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in tins section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer
15 to similar elements and in which: FIG. 1 is a block diagram depicting an example network
in which reverting to a recovery configuration may be implemented in one embodiment of the invention;
Network service providers desire to provide for deployment and maintenance of customer premises equipment (CPE) devices, such as broadband routers and the like, which may be used for residences and small businesses. Automatic network configuration provisioning approaches may provide for generating and downloading sets of configuration instructions, or configuration files, for network 25 devices that are deployed in the field to subscribers of services provided by service providers. It is desirable to be able to perfom1 such provisioning automatically, however, without requiring a subscriber to manually enter configuration commands, and without requiring a technician associated with a network service provider to visit the subscriber and configure the device.
FIG. 2A is a flow diagram that illustrates a high level overview of one embodiment of processing for reverting to
20 a recovery configuration;
FIG. 2B is a flow diagram that illustrates a high level overview of receiving and processing a change to a configuration operable with the processing depicted by FIG. 2A in one embodiment;
FIG. 2C is a flo~ diagram that illustrates a high level overview of detecting a loss of connectivity resulting from a configuration change operable with the processing depicted by FIG. 2A in one embodiment;
FIG. 2D is a flow diagram that illustrates a high level
30 overview of recovering from a loss of connectivity using a recovery configuration in one embodiment;
In one example approach, a vendor manufactures customer premises equipment network devices, and "dropships" the CPEs to the premises of subscribers of a network service provider. The CPEs are shipped with a generic bootstrap or minimal configuration that is copied from or generated at the vendor based on a standard template or format specified by the service provider. W11en a subscriber installs and powers-up a CPE, under control of the bootstrap configuration the CPE uses an interface specified in the bootstrap configuration to contact a configuration manager associated with the service provider. The configuration manager downloads a pem1anent, application-specific configuration to the CPE, which executes the received configuration and begins normal operation.
FIG. 3A is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device cormecting with a recovery configuration;
35 FIG. 3B is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device cormecting with a recovery configuration; and
FIG. 4 is a block diagram that illustrates a computer 40 system upon which an embodiment may be implemented.
In the process described above, the startup configuration 45
is typically overwritten in memory and there is no current provision for persistent storage of the initial minimal configuration. As a result, if the permanent configuration is lost or modified in a way that prevents the CPE and the network management system from conmmnicating, there is no way 50
to recover that conununications in an automatic manner. Typically, a technician or other skilled service person must travel to the customer's premises to manually reconfigure the device.
One approach that addresses these issues uses a rollback 55
mechanism that saves a current configuration or configurations at periodic intervals, and enables the user to rollback to a previous configuration. While at first glance this approach seems to address lost or corrupted configuration issues, in reality such rollback approaches are fraught with
60 difficulty. One disadvantage to rollback approaches is that there is no certainty that because a prior configuration worked in establishing connectivity that the prior configuration will still be workable in a current environment. Network environments are fluid and accordingly, what worked yesterday may not work tomorrow. In particular, 65
security related network parameters, such as certificates, passwords and the like, change with time. Systems designed
DETAILED DESCRIPTION
A method and apparatus for reverting to a recovery configuration in response to device faults in various embodiments is described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough nnderstanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram fonn in order to avoid um1ecessarily obscuring the present invention.
Embodiments are described herein according to the fol-lowing outline:
1.0 General Overview 2.0 Structural and Functional Overview 3.0 Method for Reverting to a Recovery Configuration in
Response to Device Faults 3.1 Overview 3.2 Process of Changing a Configuration 3.3 Process of Detecting a Device Fault 3.4 Process of Reverting to a Recovery Configuration 3.5 Process of Providing a Network Configuration to a
Device Cormecting with a Recovery Configuration 4.0 Implementation Mechanisms-Hardware Overview 5.0 Extensions and Alternatives
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page12 of 19
US 7,290,164 Bl 3
1.0 General Overview
The needs identified in the foregoing Background, and other needs and objects that will become apparent for the following description, are achieved in the present invention, which comprises, in one aspect, a method for reverting to a recovery configuration in response to device faults. Network devices such as routers and switches and the like maintain a configuration state using a text format file also known as a "configuration file," or just "configuration" as used herein.
10 The configuration reflects the various services, functions, parameters and interface devices with which the device may be equipped. A change to the configuration is received. The change may be in the form of configuration instmctions that comprise input from a user identifYing changes to be made 15
to the configuration information reflecting the configuration of cards or interface devices in the device. A user, such as an IT administrator or the like can provide configuration instmctions. The device may change its current configuration to a new configuration based upon the configuration 20
instmctions. If a loss of connectivity resulting from the configuration change is detected, the device will recover from the loss of connectivity by reverting to a recovery configuration. A recovering device may be revert to a recovery configuration that is identical to a new device 25
joining the network, or may revert to a recovery configuration that is different from that of a new device.
Changing the current configuration to a new configuration based upon the configuration instmctions includes a variety
4 establish connectivity with the configuration manager as a device seeking reconfiguration after recovering from an error or device fault.
In one embodiment, retrieving the recovery configuration includes obtaining security credentials. The security credentials enable the device to establish cmmectivity to the configuration manager. Security credentials may include public or private cipher keys, certificates and the like in various embodiments.
In one embodiment, retrieving the recovery configuration includes obtaining a configuration for a state that enables the device to establish cmmectivity to a configuration manager in order to obtain a network level configuration even if the device's configuration has been corrupted or is in an unknown state.
In one embodiment, the method additionally includes receiving an updated recovery configuration. The device can then replace the recovery configuration with the updated recovery configuration. In this way, the recovery configuration can be kept current as the network enviromnent changes.
In one embodiment, receiving an updated recovery configuration includes com1ecting to a configuration manager using a maintenance network connection and receiving the updated configuration from the configuration manager.
In one embodiment, the method further includes blocking changes to the recovery configuration. In such embodiments, the recovery configuration is made read only to the customer.
In another aspect, the present invention provides in one embodiment, a method for providing a network configuration to a device connecting with a recovery configuration. According to the method, the configuration manager receives a request from a device for a device configuration.
of steps. The device detects whether the new configuration 30
will require a change to the current configuration of the device. If a change to the configuration is needed, the device makes the proposed change and uses the result as the current configuration. An indication that the configuration is in a state of pending connnit is set. In one embodiment, a flag is 35 The configuration manager searches in a database for a
device configuration corresponding to an identifier of the device associated with the request. The configuration server may provide a network level configuration for the device to
set to indicate the pending conn11it state.
In one embodiment, the step of detecting a loss of connectivity resulting from the configuration change further comprises starting a timer. The device listens for a connection to be established. If a timeout is detected before an 40
indication that a cmmection has been established, a recovery routine is invoked. If a connection is established, the configuration is moved from the "pending commit" state to the "conn11it" state and the processing for configuration change completes.
In one embodiment, the step of recovering from the loss of com1ectivity by reverting to a recovery configuration further comprises making a recovery configuration the current configuration. The recovery configuration is stored in a persistent storage of the device in association with manufacturing the device, in one embodiment. Connectivity to a configuration manager is established using the recovery configuration. A network level configuration may be received from the configuration manager once cmmectivity is established. The current configuration is replaced with the network level configuration received from the configuration manager.
In one embodiment, the recovery configuration is the same as a boot configuration that is loaded onto the device for initial boot up of the device at the customer's site. In embodiments that use identical boot and recovery configurations, the device can establish connectivity with the configuration manager as a new device.
the device making the request. In one embodiment, the method further includes verifying
security credentials of the device provided with the request. If the security credentials are invalid, the configuration manager denies the request.
In one embodiment, the method further includes receiving 45 an update to a recovery configuration for network devices.
The configuration manager will send the updated recovery configuration to the device as appropriate.
In one embodiment, the configuration manager may determine based upon infom1ation in the database whether the
50 device is either a new device or a device requesting a reconfiguration after reverting to a recovery configuration. The configuration manager may change its behavior accordingly. For example, in one embodiment, a different configuration is provided to the requesting device if the device is
55 seeking to be reconfigured after reverting to a recovery configuration as compared with a newly installed device.
60
In other aspects, the invention encompasses a computer apparatus and a computer-readable medium configured to carry out the foregoing steps.
2.0 Stmctural and Functional Overview
In an alternative embodiment, the recovery configuration 65
differs from a boot configuration. In embodiments having a unique recovery and boot configurations, the device can
FIG. 1 is a block diagram depicting an example network in which reverting to a recovery configuration may be implemented in one embodiment of the invention. While the invention is illustrated generally with reference to an example of a device deployed in a service provider envirom11ent, the present invention does not require such an
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page13 of 19
US 7,290,164 Bl 5
environment, and in some embodiments, techniques according to the invention may be implemented in devices deployed in enterprise or home environments.
As shown in FIG. 1, a device CPE A 110 is installed at a customer's premises. In the example configuration depicted
6 which, when executed by CPE A 110, enable the device to determine which of several line cards or interfaces in the device can communicate through network 103 to configuration manager 152. In some embodiments. configuration manager 152 may periodically refresh the recovery configuration110C with an updated version, e.g., as changes occur in network 103.
According to one embodiment, the configuration manager 152 creates and transmits a configuration based on a con-
by FIG. 1, CPE A 110 has been installed by an IT administrator of a Network A 101 in order to connect Network A 101 to another network, such as network 103. Networks 101 and 103 may any type of network. In other example configurations, CPE A 110 may be installed by the customer in order to provide an interface for a computer to an external network such as Network 103. Accordingly, in various example configurations, CPE A 110 may be a DSL modem, a cable modem, a router, a wireless access point or various combinations thereof.
10 figuration template to the CPE A 110. According to one embodiment, configuration manager 152 provides the network level configuration based upon a configuration template provided by the CPE A device 110 upon connecting to the configuration manager 152 to request a recovery con-
15 figuration. According to one embodiment, the configuration manager 152 is a Cisco CNS 2100 Series Intelligence Engine ("IE 2100"), from Cisco Systems, Inc.
When the CPE A 110 is installed, it is communicatively coupled to a switch 102 of network 103 to establish a physical connection. Once properly configured, the CPE A device 110 is capable of connecting to an aggregator 150 through the network 103. Aggregator 150 is coupled to a network C 105. In the embodiment illustrated by FIG. 1, a configuration manager 152 is communicatively coupled to network B 107. In specific embodiments, configuration manager 152 may be on the san1e network as aggregator 150, for example. Configuration manager 152 comprises 25
configuration infonnation, which may be exchanged with CPE A 110 in order to reconfigure the CPE A 110 in the event that the CPE A 110 device suffers a configuration error or a device fault. The ability to automatically configure a remotely installed device, such as CPE A 110 in the event 30
that the CPE A 110 loses its configuration information is provided by one embodiment that will be discussed in further detail below.
3.0 Method for Reverting to a Recovery Configuration in Response to Device Faults
20 3.1 Overview According to one embodiment, reverting to a recovery
configuration in response to device faults is facilitated by a storable recovery configuration that enables a device to establish connectivity to a configuration manager. The configuration manager provides a network level configuration to the device, enabling the device to re-establish connectivity to other devices in the network substantially independent of human intervention.
FIG. 2A is a flow diagram that illustrates a high level overview of one embodiment of a method for reverting to a recovery configuration. In block 202, the configuration instructions are received. Configuration instructions may include directions to change the current configuration of the
35 device, which may be made by an IT administrator or the customer in response to a perceived need or change in the network enviromnent.
As can be seen from FIG. 1, a path may be established from CPE A 110 to configuration manager 152 via switch 102 and switch 106 of network 103 as indicated by a solid line in FIG. 1. Accordingly, a recovery configuration for CPE A 110 need only provide sufficient infonnation for the CPE A 110 to establish the com1ection with the configuration manager 152 in order to obtain complete and current con- 40
figuration information (herein referred to as a network level configuration) in order to then com1ect with other devices, such as aggregator 150, for example using the network 103.
Once a customer installs CPEA110 innetworkA101, the CPE A 110 establishes connectivity with the configuration 45
manager 152 using a generic or "boot strap" configuration loaded into the CPE A 110 prior to shipment to the customer.
According to one embodiment, a generic or "boot" configuration is 110A loaded as the current configuration of CPE A 110 during manufacture. This "boot" configuration is 50
used to bootstrap the CPE A 110 upon installation and startup in order to establish connectivity with the configuration manager 152. A persistent or running configuration 110B is then received from the configuration manager 152. In one embodiment, a recovery configuration 110C. may 55
also be loaded onto CPE A 110 during the time of manufacture. In the event that later changes to the current configuration information for the CPE A 110 overwrite the boot configuration, the CPE A 110, upon detecting a loss of connectivity resulting from changes to the configuration, can 60
revert to the recovery configuration. The CPE A 110 uses the recovery configuration to re-establish connectivity with the configuration manager 150 in order to obtain a network level configuration which may be used by the CPE A 110 to regain connectivity to other devices in the network 103. 65
According to one embodiment, the recovery configuration comprises a minimal configuration that contains instructions
In block 204, the current configuration of the device is changed to a new configuration based upon the configuration instructions as discussed below in connection with FIG. 2B.
In block 206. the device tests whether connectivity has been lost resulting ±rom the configuration change as discussed below in connection with FIG. 2C.
In block 208, a determination is made whether a cmmection has been established using the changed configuration. If connectivity is established with the new configuration, then the processing of FIG. 2A is completed.
Otherwise, if the connection is not established, then in block 210, an automated recovery process is invoked in order to establish connectivity by reverting to a recovery configuration, as is described in further detail with reference to FIG. 2D.
3.2 Process of Changing a Configuration FIG. 2B is a flow diagram that illustrates a high level
overview of receiving and processing a change to a configuration operable with the processing depicted by FIG. 2A in one embodiment. In block 212, the configuration instructions are tested to determine whether a change to the current configuration is required.
In block 214, a test is performed to determine if the configuration is to be changed. If a configuration change is to be made, then control passes to block 216. Otherwise processing for this task completes and control returns back to block 204 of FIG. 2A.
In block 216, the current device configuration is changed to the new configuration based upon configuration instructions. Depending upon the implementation and the nature of
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page14 of 19
US 7,290,164 Bl 7
the change, the current device configuration may be completely replaced with a new configuration that reflects the change, or changes may be made to a portion of the current device configuration to reflect the change.
8 storage such as USB memory ("jump drives") or the like. In various embodiments, the recovery configuration may be an XML fonnatted file, or a file formatted in either a human or a machine-only readable format, or may be encrypted using an encryption technique or the like. In one embodiment, however, the recovery configuration is a text file.
In block 233, the old configuration state is cleared. In block 234, the recovery configuration is made the current
In block 218, the state for the device configuration is set to "pending commit" state. The pending commit state indicates that a change to the device configuration has been made, but the new device configuration has not been tested for connectivity.
3.3 Process of Detecting a Device Fault FIG. 2C is a flow diagram that illustrates a high level
overview of detecting a loss of connectivity resulting from a configuration change in one embodiment. In block 222 a test message is sent using the network. In one embodiment,
10 configuration for the device. When the recovery configuration is made the current configuration, a process in the CPE A 110 device copies the recovery configuration file to the current configuration file and reboots the CPE A 110 device.
a ping or Internet Control Message Protocol (ICMP) mes- 15
sage is used to test connectivity with a remote device via the network connection under test. In one embodiment, the CPE A 110 device builds a ping message. The CPE A 110 device copies a known network address of a known device on the network into the destination address field of the ping mes- 20
sage header and copies its own network address in the sender address field of the ping message header. Then, the CPE A 110 device sends the ping message to the destination along the network. The CPE A 110 device may send the test message to any destination specified by the recovery con- 25
figuration, however, in one embodiment, the configuration server is used as the destination device.
In block 224, a timer is started. In block 226. the device tests to see if the ping result is
successful by checking to see if a response to the ping 30
message was received. If a response is received then the ping message is determined to be successful.
If the ping is successful, then in block 230, the timer is cleared. In block 231, the configuration state is changed from "pending commit" to "confinn connnit." At this point 35
cmmectivity in the network is established using the newly changed configuration and control returns back to block 208 of FIG. 2A with an indication that cmmectivity has been successfully established.
If no response to the ping message was received in block 40
226, then in block 228, another test is perforn1ed to determine if the timer has expired. If the timer has not expired, then processing continues back with block 226 to again check if a response to the ping message has been received.
When the device boots, the recovery configuration, now also the current configuration, is loaded into memory for the CPE A 110 device to use.
In block 236, cmmectivity to a configuration manager is established using the recovery configuration. In one embodiment, the recovery configuration emulates a boot configuration shipped with the CPE A 110 device, so that when the device uses a recovery configuration the device will appear to be a new machine joining the network to the configuration manager. In an alternative embodiment, the recovery configuration is sufi!ciently difierent from the boot configuration shipped with the CPE A 110 device, so that the device using the recovery configuration will not appear to be a new machine joining the network to the configuration manager. In this embodiment, the configuration manager will identifY the CPE A 110 device as a device seeking reconfiguration.
In block 238, a network level configuration is received from the configuration manager. Based upon an identifier provided by the CPE A 110 device, the configuration manager obtains a network configuration for the CPE A 110 device by searching a database accessible to the configuration manager using the identifier, as is described in further detail with reference to FIGS. 3A and 3B below. The network configuration, in one embodiment, is configured to the interfaces shipped with the CPE A 110 device, in order to enable the device to establish connectivity using the interfaces installed in the CPE A 110 device. The configuration manager sends the network level configuration to the CPE A 110 device.
In block 240, the network level configuration is made the current configuration for the device. The CPE A 110 device copies the network level configuration received from the configuration manager into the current configuration for the device. Then, the CPE A 110 device reboots, loads the
If the timer has expired, then no response has been received 45
within the time allotted, then a failure of connectivity is established. In this case, control returns to block 208 of FIG. 2A with an indication that cmmectivity has not been established. network level configuration into memory and uses the
50 configuration to regain functionality as a network element substantially free of operator intervention.
3.4 Process of Reverting to a Recovery Configuration FIG. 2D is a flow diagram that illustrates a high level
overview of recovering from a loss of connectivity using a recovery configuration in one embodiment. In block 232, a recovery configuration is retrieved. The recovery configuration is a configuration for a state that enables the CPE A 55
110 device to establish connectivity to the configuration manager. In one embodiment, retrieving the recovery configuration also includes obtaining security credentials to enable the CPE A 110 device to access the configuration manager. Security credentials include, without limitation 60
encryption keys, certificates, or the like. In one embodiment, the recovery configuration is stored
in persistent storage. For example, the recovery configuration may be stored in a programmable read-only memory (PROM), a flash memory, an electrically alterable program- 65
mabie read-only memory (EAPROM) or a direct access storage device (DASD), CD-ROM, DVD, tape, removable
In one embodiment, the configuration manager may make updates to the recovery configuration. When the CPE A 110 device receives an updated recovery configuration from the configuration manager, the CPE A 110 device replaces the present recovery configuration with the updated recovery configuration. In this way, the recovery configurations of many devices on the network can be kept current in order to accommodate for changes in network envirmm1ents, the configuration manager or security credentials.
In one embodiment, the owner or other user of the CPE A 110 device is blocked from changing the recovery configuration. In this way, the recovery configuration is protected from unauthorized modifications that could potentially render the CPE A 110 device incapable of automatically reestablishing com1ectivity to the configuration manager.
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page15 of 19
US 7,290,164 Bl 9
3.5 Process of Providing a Network Configuration to a Device Cmmecting with a Recovery Configuration
FIG. 3A is a flow diagram that illustrates a high level overview of one embodiment of processing for providing a network configuration to a device connecting with a recovery configuration. In block 302a, the configuration manager receives a request for a network level configuration file.
10
10 memory, or other dynamic storage device, coupled to bus 402 for storing information and instmctions to be executed by processor 404. Main memory 406 also may be used for storing temporary variables or other intermediate information during execution of instmctions to be executed by processor 404. Computer system 400 further includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instmctions for processor 404. A storage device 410, such as a magnetic disk, flash memory or optical disk, is provided and coupled to bus 402 for storing information and instructions.
In block 304a, the configuration manager searches a database for configuration information corresponding to the device making the request. In one embodiment, the database includes configurations for known devices on the network. The configurations for these devices may be searched using an identifier associated with each device. In one embodiment, a manufacturer using an automated provisioning service provides an appropriate configuration file for a device
A communication interface 418 may be coupled to bus 402 for communicating infom1ation and conu11and selec-
15 tions to processor 404. Interface 418 is a conventional serial interface such as an RS-232 or RS-422 interface. An extemal at the time the device is manufactured and stores the
configuration file with the configuration manager. In embodiments using such automated provisioning techniques, once the customer installs the device on the customer site, the device boots using a boot configuration and estab- 20
lishes cmmectivity with the configuration manager in order to obtain a network level configuration.
A device seeking to recover its configuration might be trusted less than a new device. Generally, a system administrator knows if a new device installation is expected. The 25
system administrator might not expect a device to need recovery however. Since a recovering device might have otherwise been compromised, in one embodiment, additional state infom1ation might be required from the recovering device prior to permitting the device to receive a 30
recovery configuration. For example, the configuration manager may inquire whether anyone is logged into the recovering device or the like.
In block 310a, the configuration manager provides a network level configuration for the device over the network. 35
FIG. 3B is a flow diagram that illustrates a high level overview of another embodiment of processing for providing a network configuration to a device connecting with a recovery configuration. In block 302b, the configuration manager receives a request for a network level configuration 40
file.
terminal 412 or other computer system connects to the computer system 400 and provides connnands to it using the interface 414. Firmware or software mnning in the computer system 400 provides a terminal interface or character-based command interface so that extemal commands can be given to the computer system.
A switching system 416 is coupled to bus 402 and has an input interface 414 and an output interface 419 to one or more external network elements. The extemal network elements may include a local network 422 coupled to one or more hosts 424, or a global network such as Intemet 428 having one or more servers 430. The switching system 416 switches information traffic arriving on input interface 414 to output interface 419 according to pre-detennined protocols and conventions that are well known. For example, switching system 416. in cooperation with processor 404, can determine a destination of a packet of data arriving on input interface 414 and send it to the correct destination using output interface 419. The destinations may include host 424, server 430, other end stations, or other routing and switching devices in local network 422 or Internet 428.
The invention is related to the use of computer system 400 for reverting to a recovery configuration in response to device faults. According to one embodiment of the invention, reverting to a recovery configuration in response to device faults may be provided by computer system 400 in response to processor 404 executing one or more sequences
In block 304b, the configuration manager searches a database for configuration information corresponding to the device making the request.
In 306b, the configuration manager determines whether the device is a new device seeking a first network level configuration or a device using a recovery configuration to request a network level configuration. In block 308b, a test is performed to determine whether device seeking a network level configuration is a device recovering from a fault.
45 of one or more instmctions contained in main memory 406. Such instmctions may be read into main memory 406 from another computer-readable medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the
In block 310b, the configuration manager provides a network level configuration for the recovering device. Otherwise, in block 312b, the configuration manager provides a network level configuration for a new device.
50 process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instmctions contained in main memory 406. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with
4.0 Implementation Mechanisms-Hardware Overview 55 software instmctions to implement the invention. Thus,
embodiments of the invention are not limited to any specific combination of hardware circuitry and software. FIG. 4 is a block diagram that illustrates a computer
system 400 upon which an embodiment of the invention may be implemented. The preferred embodiment is implemented using one or more computer programs mnning on a 60
network element such as a router device. Thus, in this embodiment, the computer system 400 is a router.
Computer system 400 includes a bus 402 or other communication mechanism for conmmnicating infonnation, and a processor 404 coupled with bus 402 for processing infor- 65
mation. Computer system 400 also includes a main memory 406, such as a random access memory (RAM), flash
The term "computer-readable medium" as used herein refers to any medium that participates in providing instmctions to processor 404 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 410. Volatile media includes dynamic memory, such as main memory 406. Transmission media includes coaxial cables, copper wire and fiber optics, includ-ing the wires that comprise bus 402. Transmission media can
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page16 of 19
US 7,290,164 Bl 11
also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any 10
other medium from which a computer can read. Various forms of computer readable media may be
involved in carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a magnetic disk of a 15
remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared 20
signal. An infrared detector coupled to bus 402 can receive the data carried in the infrared signal and place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received by main memory 406 may option- 25
ally be stored on storage device 410 either before or after execution by processor 404.
Communication interface 418 also provides a two-way data communication coupling to a network link 420 that is
30 cmmected to a local network 422. For example, communi-cation interface 418 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, conmmnication interface 418 may
35 be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, commU11ication interface 418 sends and receives electrical, electromagnetic or optical signals that carry digital data
40 streams representing various types of information. Network link 420 typically provides data connnunication
through one or more networks to other data devices. For example, network link 420 may provide a connection through local network 422 to a host computer 424 or to data
45 equipment operated by an Internet Service Provider (ISP) 426. ISP 426 in turn provides data cmlUllunication services through the world wide packet data comnnmication network now commonly referred to as the "Intemet" 428. Local network 422 and Internet 428 both use electrical, electro-
50 magnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 420 and through conununication interface 418, which carry the digital data to and from computer system 400, are exemplary forms of carrier waves transporting the
55 information.
Computer system 400 can send messages and receive data, including program code, through the network(s), network link 420 and cmlUllunication interface 418. In the Internet exan1ple, a server 430 might transmit a requested 60 code for an application program through Internet 428, ISP 426, local network 422 and cotlUllunication interface 418. In accordance with the invention, one such downloaded application provides for reverting to a recovery configuration in response to device faults as described herein. 65
The received code may be executed by processor 404 as it is received, and/or stored in storage device 410, or other
12 non-volatile storage for later execution. In this mmmer, computer system 400 may obtain application code in the form of a carrier wave.
5.0 Extensions and Altematives In the foregoing specification, the invention has been
described with reference to specific embodiments thereof. In particular, while the invention has been described generally with reference to one exm11ple embodiment, in which confignration instmctions are provided that lead to a loss in connectivity, it will be appreciated by those skilled in the art that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
What is claimed is: 1. A method of reverting to a recovery configuration in
response to faults of a network device, the method comprising the computer-implemented steps of:
receiving configuration instructions; chm1ging a current configuration to a new configuration
based upon the confignration instmctions; detecting a loss of counectivity between the device and a
network resulting from the configuration change; and recovering from the loss of connectivity by reverting to a
recovery configuration, wherein the recovery configuration is stored in a persistent storage of the device in association with manufacturing the device, wherein the recovering step further comprises: retrieving a recovery configuration; making the recovery confignration the current configu
ration; and establishing counectivity to a configuration manager
using the recovery configuration. 2. A method as recited in claim 1, wherein changing a
current confignration to a new configuration based upon the confignration instmctions comprises:
detecting whether the new configuration will require a change to the current configuration of the device; and if so:
making the proposed change as the current configuration and setting a flag indicating a pending commit state.
3. A method as recited in claim 1, wherein the step of detecting a loss of connectivity resulting from the configuration change further comprises the steps of:
sending a test message; determining whether a connection is established; and invoking a recovery routine if a timeout occurs. 4. A method as recited in claim 3, wherein sending a test
message comprises sending a ping message. 5. A method as recited in claim 1, wherein the step of
recovering from the loss of connectivity by reverting to a recovery configuration further comprises the steps of:
receiving from the configuration mm1ager a network level configuration; and
replacing the current configuration with the network level configuration.
6. A method as recited in claim 1, wherein the recovery confignration is a boot configuration and wherein establishing connectivity to a configuration manager using the recovery configuration comprises:
establishing com1ectivity with the configuration manager as a new device.
7. A method as recited in claim 1, wherein the recovery confignration differs from a boot configuration and wherein
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page17 of 19
US 7,290,164 Bl 13
establishing connectivity to a configuration manager using the recovery configuration comprises:
establishing com1ectivity with the configuration manager as a device seeking reconfiguration.
8. A method as recited in claim 1, wherein retrieving the recovery configuration comprises:
obtaining security credentials enabling the device to establish cmmectivity to the configuration manager.
9. A method as recited in claim 1, wherein retrieving the recovery configuration comprises:
obtaining a configuration for a state enabling the device to establish cmmectivity to the configuration manager.
10. A method as recited in claim1, further comprising the steps of:
10
14 replacing the current configuration with the network
level configuration. 18. A computer-readable medium carrying one or more
sequences of instructions for reverting to a recovery configuration in response to faults of a network device, which instructions, when executed by one or more processors, cause the one or more processors to carry out the steps of:
receiving configuration instructions; changing the current configuration to a new configuration
based upon the configuration instructions; detecting a loss of connectivity between the device and a
network resulting from the configuration change; recovering from the loss of connectivity by reverting to a
recovery configuration establishing connectivity to a network using the network 15
level configuration as the current configuration. wherein the recovery configuration is stored in a persis
tent storage of the device in association with manufacturing the device, wherein the recovering step further 11. A method as recited in claim 1, fi1rther comprising the
step of: receiving an updated recovery configuration; and
comprises: retrieving the recovery configuration;
replacing the recovery configuration with the updated 20
recovery configuration. making the recovery configuration the current configu
ration; and 12. A method as recited in claim 11, wherein receiving an
updated recovery configuration comprises: connecting to a configuration manager using a mainte
nance network connection; receiving from the configuration manager an updated
configuration. 13. A method as recited in claim 1, further comprising: blocking changes to the recovery configuration.
establishing com1ectivity to a configuration manager using the recovery configuration.
19. A computer-readable medium as recited in claim 18, 25 further comprising instructions which, when executed by the
one or more processors, cause the one or more processors to carry out the steps of:
14. A method as recited in claim 1, further comprising: 30
committing the configuration if a connection is estab-
storing the recovery configuration into a persistent storage of the device in association with manufacturing the device.
20. A computer-readable medium as recited in claim 19, wherein the instructions for changing the current configuration to a new configuration based upon the configuration instmctions further comprise instructions for carrying out
lished and no timeout occurs. 15. A method as recited in claim 1, wherein establishing
connectivity to a configuration manager using the recovery configuration comprises:
copying the recovery configuration from a persistent storage to the current configuration for the device; and
rebooting the device.
35 the steps of: detecting whether the new configuration will require a
change to the current configuration of the device; and if so:
16. A method as recited in claim 15, wherein rebooting the device includes loading the recovery configuration into the 40
memory of the device to detennine the configuration of interfaces of the device.
making the proposed change as the current configuration and setting a flag indicating pending commit state.
21. A computer-readable medium as recited in claim 19, wherein the instructions for detecting a loss of connectivity resulting from the configuration change further comprise instructions for carrying out the steps of:
17. A method of reverting to a recovery configuration in response to device faults, the method comprising the computer-implemented steps of:
receiving configuration instructions; changing a current configuration to a new configuration
based upon the configuration instructions by detecting whether the new configuration will require a
change to the current configuration of the device; and if so:
making the proposed change as the current configuration and setting a flag indicating pending commit;
detecting a loss of connectivity resulting from the configuration change by sending a test message; determining whether a connection is established; and invoking a recovery routine if a timeout occurs; and
recovering from the loss of connectivity by reverting to a recovery configuration by making a recovery configuration stored in a persistent
storage of the device in association with manufacturing the device the current configuration;
establishing connectivity to a configuration manager using the recovery conllguration;
receiving from the configuration manager a network level configuration; and
45 sending a test message; detennining whether a connection is established; and invoking a recovery routine if a timeout occurs. 22. A computer-readable medium as recited in claim 21,
50 wherein the instmctions for sending a test message further comprising instructions for carrying out the step of:
sending a ping message. 23. A computer-readable medium as recited in claim 19,
wherein the instmctions for recovering from the loss of
55 connectivity by reverting to a recovery configuration further comprise instructions for carrying out the steps of:
retrieving a recovery configuration;
60
making a recovery configuration the current configuration; and
establishing connectivity to a configuration manager using the recovery configuration.
24. A computer readable medium as recited in claim 23, wherein the instructions for recovering from the loss of connectivity by reverting to a recovery configuration further
65 comprise instructions for carrying out the steps of: receiving from the configuration manager a network level
configuration; and
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page18 of 19
US 7,290,164 Bl 15
replacing the current configuration with the network level configuration.
25. A computer-readable medium as recited in claim 23, wherein the recovery configuration is a boot configuration and wherein the instructions for establishing connectivity to a configuration manager using the recovery configuration further comprise instructions for carrying out the step of:
establishing connectivity with the configuration manager as a new device.
26. A computer-readable medium as recited in claim 23, 10
wherein the recovery configuration differs from a boot configuration and wherein the instructions for establishing connectivity to a configuration manager using the recovery configuration further comprise instructions for carrying out the step of: 15
establishing connectivity with the configuration manager as a device seeking reconfiguration.
27. A computer-readable medium as recited in claim 23, wherein the instmctions for retrieving the recovery configuration further comprise instructions for carrying out the step 20
of: obtaining security credentials enabling the device to
establish connectivity to the configuration manager. 28. A computer-readable medium as recited in claim 23,
wherein the instmctions for retrieving the recovery configu- 25
ration further comprise instructions for carrying out the step of:
obtaining a configuration for a state enabling the device to establish connectivity to the configuration manager.
29. A computer-readable medium as recited in claim 23, 30
further comprises the steps of: establishing cmmectivity to a network using the network
level configuration as the current configuration. 30. A computer readable medium as recited in claim 23,
wherein the instructions for establishing cmmectivity to a 35
configuration manager using the recovery configuration further comprise instructions carrying out the steps of:
copying the recovery configuration from a persistent storage to the current configuration for the device; and
rebooting the device. 40
31. A computer readable medium as recited in claim 30, wherein rebooting the device includes:
loading the recovery configuration into the memory of the device to determine the configuration of interfaces of the device. 45
32. A computer-readable medium as recited in claim 19, further comprising instructions for carrying out the step of:
receiving an updated recovery configuration; and replacing the recovery configuration with the updated
recovery configuration. 50
33. A computer-readable medium as recited in claim 32, wherein the instructions for receiving an updated recovery configuration fl.Jrther comprising instructions for carrying out the step of:
connecting to a configuration manager using a mainte- 55
nance network connection; receiving from the configuration manager an updated
configuration. 34. A computer-readable medium as recited in claim 19,
further comprising instmctions for carrying out the step of: 60
blocking changes to the recovery configuration. 35. A computer-readable medium as recited in claim 19,
further comprising instructions for carrying out the step of:
16 connnitting the configuration if a connection is estab
lished and no timeout occurs. 36. A computer-readable medium as recited in claim 19,
wherein the instructions for recovering from the loss of connectivity by reverting to a recovery configuration further comprise instructions for carrying out the steps of:
retrieving a recovery configuration; and making a recovery configuration the current configura
tion. 37. A.n apparatus for reverting to a recovery configuration
in response to device faults, comprising: means for storing a proposed change as the current
configuration and setting a flag indicating pending commit if the new configuration will require a change to the current configuration of the device;
means for sending a test message and detennining whether a connection is established;
means for detecting a timeout; means for invoking a recovery routine if a timeout occurs; means for making a recovery configuration the current
configuration; means for establishing cmmectivity to a configuration
manager using the recovery configuration; means for receiving from the configuration manager a
network level configuration; and means for replacing the current configuration with the
network level configuration. 38. An apparatus for reverting to a recovery configuration
in response to device faults, comprising: a network interface that is coupled to the data network for
receiving one or more packet flows therefrom; a processor; one or more stored sequences of instructions which, when
executed by the processor, cause the processor to carry out the steps of: receiving configuration instructions; changing a current configuration to a new configuration
based upon the configuration instructions by detecting whether the new configuration will require
a change to the current configuration of the device; and if so:
making the proposed change as the current configuration and setting a flag indicating pending commit;
detecting a loss of cmmectivity resulting from the configuration change by sending a test message; detem1ining whether a connection is established; detecting a timeout; and invoking a recovery routine if a timeout occurs; and
recovering from the loss of connectivity by reverting to a recovery configuration by making a recovery configuration stored in a persis
tent storage of the device in association with manufacturing the device the current configuration;
establishing connectivity to a configuration manager using the recovery configuration;
receiving from the configuration manager a network level configuration; and
replacing the current configuration with the network level configuration.
* * * * *
Case3:14-cv-05343 Document1-8 Filed12/05/14 Page19 of 19
Exhibit 9
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page1 of 17
(12) United States Patent Edsall et al.
(54) PRIVATE VLANS
(75) Inventors: Thomas J. Edsall, Cupertino, CA (US); Marco Foschiano, San Jose, CA (US); Michael Fine, San Francisco, CA (US); Thomas Nosella, Sunnyvale, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.
(21) Appl. No.: 09/575,774
(22) Filed: May 22, 2000
(51) Int. Cl? ........................ G06F 15/173; H04L 12/56 (52) U.S. Cl. ........................ 370/389; 370/401; 709/225 (58) Field of Search ................................. 370/389, 392,
370/401, 464, 465; 709/218, 221, 225
(56) References Cited
U.S. PATENT DOCUMENTS
5,959,989 A * 9/1999 Gleeson eta!. ............. 370/390 6,058,429 A * 5!2000 Ames et a!. ................ 709/242 6,111,876 A * 8/2000 Frantz et a!. ............... 370/392 6,147,995 A * 11/2000 Dobbins eta!. ............ 370/392 6,208,649 B1 3/2001 Kloth 6,304,901 B1 10/2001 McCloghrie et a!. 6,560,236 B1 * 5!2003 Varghese et a!. ............ 370/401
OTHER PUBLICATIONS
Andrew Tanenbaum, "Computer Netwroks, Third Edition," Pretice Hall, 1996, pp. 417-419.
* cited by examiner
USER#1
120
111111 1111111111111111111111111111111111111111111111111111111111111
1QQ
US006741592Bl
(10) Patent No.: US 6,741,592 Bl May 25,2004 (45) Date of Patent:
Primary Examiner---Alpus H. Hsu
Assistant Examiner-Toan Nguyen
(74) Attorney, Agent, or Firm---Cesari and McKenna, LLP
(57) ABSTRACT
The invention uses a layer 2 switch (L2 switch), or bridge,
to separate user's message traffic by use of Virtual Local
Area Networks (VLANs) defined within the switch. Three
new types of ports are defined, "promiscuous" ports "isolated" ports, and "community" ports. Three types of VLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs. The promiscuous ports are connected to layer 3 or layer 4 devices. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic for each user separate from other users. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports. An isolated VLAN connects to all promiscuous ports and to all isolated ports. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.
26 Claims, 8 Drawing Sheets
COMMUNITY OR ISOlATED PORTS
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page2 of 17
U.S. Patent May 25,2004 Sheet 1 of 8 US 6,741,592 Bl
140
1
122
l31l4 DEVICE
152
NETWORK CLOUD
r----'---..... ,---A----.. ......---'-----.
143
L3/l4 DEVICE
150
L3/L4 DEVICE
146
108 104 _!. NJ PROMISCUOUS PORTS
102
114 USER#1
SERVER USER#1
r"""""-----~~---"'"...,
126
L2SWITCH
SERVER USER#2
FIG. 1
COMMUNITY OR ISOLATED PORTS
USER#M
SERVER USER#M
132
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page3 of 17
U.S. Patent
102 -
TED ISOLA PORT s
May 25,2004 Sheet 2 of 8
PROMISCUOUS PORTS
#A #B 232
- ,A, ---r .....
220- f--222 . • .
~230
PRIMARY VLAN
L2 SWITCH
204- 206- 208-' 210- 212-.
US 6, 741,592 Bl
#N 224-
240-ISOLATED
VLAN
I'
~214
. . USER# 1 2 3 4 5 "-v--' N
230
FIG. 2
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page4 of 17
U.S. Patent
102 -
COMMU PORTS
NITY
May 25,2004 Sheet 3 of 8
PROMISCUOUS PORTS
#A #B 232
r ' 320- f---322 . . .
-330 PRIMARY VLAN
, L2SWITCH
COM. VLAN#3 COM. VLAN#2
COM. VLAN#1
304-' 306- 308-' 310- 312-.
US 6,741,592 Bl
#N
324-
r-
354-
I
352--r350
--
r-314 . .
USER# 1 2 3 4 5 '-y--/ N 230
FIG. 3
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page5 of 17
U.S. Patent May 25,2004
402 '"'\ 404 '"'\
PREAMBLE L2HEADER
502 ., 504 '"'\
Sheet 4 of 8
400
406 '"'\
L3HEADER
PACKET
FIG. 4
410 '".
DATA
US 6,741,592 Bl
412 ~""'\
TRAILING FIELDS
506
' l3LAYER PRlMARY ISOLA TED OR COMMUNITY INTERFACE
NUMBER
510-< I
512/
VLAN VLAN NUMBER NUMBER
l
514./
PROMISCUOUS PORT ASSIGNMENT TABLE
FOR OUTGOING TRAFFIC
FIG. 5
. . .
......
,.._ .....
516
518 520
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page6 of 17
U.S. Patent
560
570
580
May 25,2004 Sheet 5 of 8
552 ., 550 554 '.
PRIMARY SECONDARY VLAN VlANs
2 20
2 21
2 22
2 23
3 30 3 31
3 32
. .
. . • .
TRUNK TYPE PROMISCUOUS PORTVLAN MAPPING TABLE
FIG. SA
US 6,741,592 Bl
-------
560A
5608
560C
5600
570A
5708
570C
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page7 of 17
U.S. Patent May 25,2004 Sheet 6 of 8 US 6,741,592 Bl
602 ........ 604 605 606 '"'\ ,......_
'"'\ 608 '"'\ 610 '"'\ 612 ........
VLAN PORT TRAILING DESIGNATION NO OTHER l2HEADER L3HEADER DATA FIELDS (COLOR)
PACKETS INTERNAL TO l2 SWITCH
FIG. 6
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page8 of 17
U.S. Patent May 25, 2004 Sheet 7 of 8
700 702 ., 706 ,,
PORT No. ISOLATED OR COMMUNITY VLAN
\ I ) \.
712 716
ISOLATED OR COMMUNITY PORT ASSIGNMENT TABLE
FIG. 7
US 6,741,592 Bl
>710
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page9 of 17
U.S. Patent
830
806
802
810
May 25,2004
DISTRIBUTION SWITCH
L2
864
ACCESS SWITCH
L2
812 814
Sheet 8 of 8
BACKBONE TO
INTERNET
FIG. 8
816
US 6,741,592 Bl
848
DISTRIBUTION SWITCH 808
L2
866
ACCESS SWITCH
l2
818 820
TRUNK CONNECTION
804
ISOLATED PORTS
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page10 of 17
US 6,741,592 Bl 1
PRIVATE VLANS
BACKGROUND OF THE INVENTION
1. Field of the Invention
The invention relates to Virtual Local Area Networks (VLANs), and more particularly to the use of VLANs to establish separation between different users of a shared switch.
2. Background Information
It is today a common computer network engineering practice to separate packet traffic belonging to different users
5
2 A better way to keep the message traffic of different users
separate in a computer network is needed, particularly a method which can scale to a large number of users.
SUMMARY OF THE INVENTION
The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three new types of ports are defined, "promiscuous" ports, "isolated" ports, and "community" ports. Three types of VLANs
10 internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs.
The promiscuous ports are connected to layer 3 or layer 4 devices, for example routers which may in turn connect to the worldwide Internet, load balancers which also may
15 connect to the worldwide Internet, administrative work stations such as used by network administrators, back up devices, etc. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic
by use of a router, a Layer 3 (L3) device. Separation of users' traffic is accomplished by assigning each user to a different subnetwork (subnet). A subnet is identified by a unique L3 address. The router then transmits a particular user's packets out through a port assigned to that subnet. However, only a limited number of bits in the L3 address (for example IP address) are assigned to the subnet, and so 20
only a limited number of subnets may be addressed by a particular router. Subnet design is described by Andrew Tanenbaum in his book Computer Networks, Third Edition, published by Prentice Hall, Copyright date 1996, all disclosures of which are incorporated herein by reference, particularly at pages 417-419. For example, if 6 bits are assigned to a subnet mask, then only 62 different subnets may be addressed (0 and 64 are reserved). Further, for every subnet assigned two addresses are wasted, for example the multicast and broadcast addresses.
for each user separate from other users. Isolated ports and community ports exchange packets
with the promiscuous ports by use of the VLANs internal to the switch. The difference between isolated and community ports is that an isolated port cannot transfer packets to another isolated port, however a community port has a
25 designated number of community ports to which it can transfer packets.
A primary VLAN internal to the switch is defined as follows. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The
30 primary VLAN receives packets from outside of the switch arriving at any of the promiscuous ports, and transfers the packets to the isolated or community ports. However, an isolated or community port cannot receive traffic from the external LAN connected to it, and transfer the packets to the
As an example of many users of a switch who require that their message traffic be kept separate, an Internet service provider (ISP) may have many customers who want to connect to a server farm. Access to the ISP is through a router connected to a common external computer network, for example the worldwide Internet. The router must route each customer's traffic to that customer's local area network in such a manner as to maintain protection and privacy between the data of different customers. It is desirable for an ISP to prevent traffic originating from one customer's server from being received by another customer's server.
A second example of many users of a computer network who must have their traffic separated in order to guarantee privacy and protection is the use of a television cable Internet distribution system. Each home is assigned a separate subnet so that routers may route only a particular customer's message traffic to that customer. This subnet routing prevents, for example, one customer looking at another customer's message traffic by use of, for example, a network snifter.
A third example is a server farm, for example a multiclient backup service. Each client's message traffic arrives at a router. The router uses a subnet mask to keep the traffic of each client separate from the traffic of another client, as it routes the traffic to the client's backup server.
A limitation in the use of subnets, and subnet masks, in a multiclient environment is that there is only a limited number of subnets which can be defined from standard Layer 3 addresses. In modem computer network systems, this numerical limitation severely restricts the number of individual users who can be serviced, and also have their message traffic maintained separate. Further, the management of a large number of subnets by a network manager becomes burdensome, especially in the event that the network has thousands of customers whose packet traffic must be kept separate.
35 primary VLAN. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports.
An isolated VLAN is defined as connecting to all promiscuous ports and connecting to all isolated ports. An isolated VLAN receives packets arriving from outside of the
40 switch at an isolated port, and transfers the packets to the promiscuous ports. An isolated VLAN does not carry packets received by a promiscuous port from outside of the switch. Also, an isolated VLAN does not deliver any packets to another isolated port. The isolated VLAN is a one way
45 connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group
of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. The
50 community VLAN transfers a packet received from outside the switch at a community port to all of the promiscuous ports, and also transfers the packet to the other community ports attached to that community VLAN. A plurality of "communities" of community ports may be defined, and
55 each community of ports has its own assigned community VLAN. A community VLAN cannot transfer packets received from outside of the switch at a promiscuous port. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a
60 packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.
These new types of VLANs and ports are implemented, in part, by particular settings of the Color Blocking Logic
65 (CBL) logic circuits used by normal ports of an L2 switch which supports VLANs, and also by use of assignment tables.
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page11 of 17
US 6,741,592 Bl 3
Traffic generated by different user's servers is kept separate from other user's servers, by each user having his own isolated port or community of community ports.
4 least many, of the occurrences on the network. Promiscuous Port A 104 connects to layer 3 or layer 4 device (L3/L4 device) A 140, and L3/L4 device 140 connects to network
The VLANs defined in a first L2 switch chassis can be trunked to other L2 switch chassises using ordinary trunking 5
technology, in order to increase the number of ports.
cloud 142. Promiscuous port B 106 connects to L3!L4 device 143, and L3/L4 device 143 connects to network cloud 144. Promiscuous port N 108 connects to L3/L4 device 146. L3!L4 device 146 connects to network cloud 148. Alternatively, a single L2 switch, or a network of trunked
L2 switches, may have its promiscuous ports divided into subsets. Each subset of promiscuous ports is then associated with its subset of isolated ports and community ports, along 10
with the necessary VLANs.
Three dots 150 indicate that L2 switch 102 may have a plurality of promiscuous ports, etc. Three dots 152 indicate a plurality of L3/L4 devices, connected to the promiscuous ports, portA104, port B 106, port N 108, etc. Three dots 149 indicate that the L3/L4 devices may connect to a plurality of network clouds 142, 144, 148, etc.
Other and further aspects of the present invention will become apparent during the course of the following description and by reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
Referring now to the drawings, in which like numerals represent like parts in the several views:
FIG. 1 is a block diagram of a computer network in accordance with the invention;
FIG. 2 is a block diagram of a L2 switch in accordance with the invention;
FIG. 3 is a block diagram of a L2 switch in accordance with the invention;
FIG. 4 is a field diagram of a layer 3 packet; FIG. 5 is an assignment table for a promiscuous port for
outgoing traffic, in accordance with the invention;
FIG. SA is a Trunk Type Promiscuous Port VLAN Mapping Table, in accordance with the invention;
FIG. 6 is a field diagram of a VLAN packet internal to a L2 switch;
FIG. 7 is a port assignment table for an isolated or community port;
FIG. 8 is a block diagram of a two level layer 2-switch network in accordance with the invention.
DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT
Turning now to FIG. 1, computer network 100 is shown.
L2 switch 102 has promiscuous ports, port A 104, port B 106, port N 108, etc. Promiscuous port 108 is indicated as "N", indicating that an arbitrary number of promiscuous ports may be employed by L2 switch 102.
L2 switch 102 also has isolated or community ports, port
Network clouds 142, 144, 148 may be different network 15 clouds, for example each may comprise a backup device for
a particular user server. Alternatively, each network cloud 142, 144, 148 may represent the worldwide Internet. Further, each network cloud 142, 144, 148, may represent a particular device, may represent several particular devices, and may
20 also represent the worldwide Internet, etc. Turning now to FIG. 2, the interior of L2 switch 102 is
shown. Isolated port 204,206,208,210,212,214 are labeled progressively as user #1, 2, 3, 4, 5, N, etc., and each may connect to a different user. Isolated ports 204, 206, 208, etc.
25 correspond to isolated ports 114, 116, 118, etc. shown in FIG. 1. Promiscuous ports, port A 220, port B 222, port N 224, connect to various L3!L4 devices (not shown in FIG. 2), such as devices 140, 143, 146, etc. Promiscuous ports 220, 222, 224, etc. correspond to promiscuous ports 104,
30 106, and 108, etc. as shown in FIG. 1. Three dots 230 indicate a plurality of users, each con
nected to an isolated port 204, 206, 208, 210, 212, 214, etc. Three dots 232 indicate a plurality of promiscuous ports,
35 220, 222, 224, etc. and indicate that L2 switch 102 may have a plurality of promiscuous ports.
The VLANs utilized by L2 switch 102 are described below.
VLAN 230 is a primary VLAN, and connects to promis-40 cuous ports 220, 222, 224, etc., and also connects to each of
the isolated ports 204,206,208, ... 214, etc. Primary VLAN 230 carries packet traffic from the promiscuous ports to isolated ports. Primary VLAN 230 is configured to reject any packets arriving at an isolated port from the external
45 local area network connected to the isolated port.
#1 114 is connected to user #1 VLAN 120, and user #1 VLAN 120 connects to user #1 server 122. Isolated or 50
During ordinary operation, any packet received by a promiscuous port from the L3!L4 device is transmitted on the primary VLAN, and may be received by any isolated port or community port having a destination for that packet on an external LAN connected thereto.
community port #2 116 connects to user #2 VLAN 124, and user #2 VLAN 124 connects to user #2 server 126. Isolated or community port #M 118 is labeled "M" to indicate that L2 switch 102 may have an arbitrary number of isolated or community ports. Isolated or community port #M 118 connects to user #M VLAN 130, and user #M VLAN 130 connects to user #M server 132. "Three dots" 134 indicate that L2 switch 102 may have a plurality of isolated or community ports, etc. "Three dots" 136 indicate that a plurality of user servers, each connected to a different isolated or community port, etc.
The promiscuous ports 104, 106, 108, etc. connect to layer 3 or layer 4 devices 140,143,146. Examples of layer 3
Isolated VLAN 240 connects to isolated ports 204, 206, 208, ... 214, etc., and also connects to each of the promiscuous ports 224, 222, ... 220, etc. Isolated VLAN 240 carries packet traffic from isolated ports to the promis-
ss cuous ports. Isolated VLAN 240 is configured to reject any traffic arriving from a promiscuous port. Also, isolated VLAN 240 is configured so that it cannot deliver any packets to an isolated port. That is, packets transferred onto isolated VLAN 240 by an isolated port cannot be received
60 by another isolated port. Packets transferred onto isolated VLAN 240 from an isolated port are received by promiscuous ports 220, 222, 224, etc., and from the promiscuous ports may be transferred to network clouds, for example,
or layer 4 devices comprise routers, load balancers, administrative work stations, back-up devices, etc. An administra- 65
tive work station is a work station utilized by a network administrator and permits the administrator to view all, or at
network cloud 142, 144, 148. Mechanisms, for example, color blocking logic (CBL)
and assignment tables, may be used to permit primary VLAN 230 to transfer packets from promiscuous ports to
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page12 of 17
US 6,741,592 Bl 5
isolated ports, and prohibit an isolated port from transmitting onto primary VLAN 230. Also, mechanisms within L2 switch 102 such as CBL and assignment tables may be used to permit isolated VLAN 240 to transfer packet traffic from an isolated port to a promiscuous port, and prevent isolated 5
VLAN 240 from transferring a packet to an isolated port. Community VLANs implemented in L2 switch 102 are
described next.
6 104, etc. to network cloud 142, or any of the other network clouds from one of the other L3/L4 devices.
Turning now to FIG. 5, "Promiscuous Port Assignment Table for Outgoing Traffic" 500 is shown with three columns. Table 500 is a conceptual table which is an aid to understanding the invention. Data shown in table 500 may be held, in a particular implementation, in a variety of places. For example some data is in the header of a received packet, some data may be held in hardware such as memory A community VLAN connects to a designated group of
community ports, and to all of the promiscuous ports. A community port receives a packet from outside of switch 102 and transfers the packet to the community VLAN. A packet transferred to the community VLAN from a community port is received by all of the community ports connected to the community VLAN, and also all of the promiscuous ports receive the packet from the community VLAN. The promiscuous ports then transfer the packet out
10 in an ASIC chip in the interface, or further, some of the data may be held in a software lookup table in the memory for a processor of the router. As a further example, an implementation may use a table such as Table 500 in main memory for a processor of the router. Column 502 contains a layer 3
15 interface number. Column 504 contains a primary VLAN assignment number. Column 506 contains an isolated or community VLAN assignment number.
of the L2 switch. A Community VLAN is configured to reject any traffic arriving from a promiscuous port.
Turning now to FIG. 3, community VLAN #1 350, 20
community VLAN #2 352, and, community VLAN #3 354 are shown. Community VLAN #1 350 is shown connected to community ports 306, and 308. Community VLAN #1 350 permits community ports connected thereto to exchange packets. For example, a packet entering L2 switch 102 from 25
user #2 at community port 306 is transferred by community VLAN #2 350 to the other community ports, for example community ports 308, etc., connected to community VLAN #1 350, and is also transferred to all of the promiscuous ports, ports 320, 322, 324, . . . 30
Community VLAN #2 352 is shown connected to community port 310 and 312. A packet originating from user #4 or user #5 will enter L2 switch 102 at either community port 310,312, respectively, and will be transferred by community
35 VLAN #2 352 to the other isolated port, and to all of the promiscuous ports 320, 322, ... 324, etc.
As a further example of a community VLAN, community VLAN #3 354 is shown. Community VLAN 354 is shown connected to community port 304 and community port 314.
40 Community VLAN #3 354 also connects to all of the promiscuous ports 320, 322, ... 324, etc.
In the present description, the isolated ports are shown in FIG. 2, and the community ports are shown in FIG. 3. Switch 102 may have, for example, both isolated ports and 45 community ports. In this case, both of the port arrangements of FIG. 2 and of FIG. 3 are implemented within L2 switch 102. In a second exemplary embodiment, L2 switch 102 may have only isolated ports as shown in FIG. 2. In a third exemplary embodiment, L2 switch 102 may have only 50 community ports as shown in FIG. 3.
A terminology which can be used is to refer to the isolated VLAN and the community VLAN as a "secondary" VLAN. Using this terminology, a primary VLAN takes packets from the promiscuous ports to either the isolated ports or the 55 community ports. In contrast, the secondary VLAN takes packets from either the isolated ports or community ports to the promiscuous ports.
Turning now to FIG. 4, a field diagram 400 of a typical L2 packet which reaches an L2 switch from a network cloud is 60
shown. Field 402 is the preamble. Field 404 contains the L2 header. Field 406 contains the L3 header. Data carried by packet 400 is in field 410. Trailing fields 412 contain fields typically trailing the data fields of a typical data packet, and normally include a cyclical redundancy check (CRC) field. 65
The field diagram of a packet shown in FIG. 4 also represents the fields in a packet departing from L3!L4 device
A primary VLAN Assignment Number, as held in column 504, is a designation which is written into a field of a packet transferred from layer 2 switch 102 to L3!L4 device 140, or, for example, any of the other L3!L4 devices 143, 146, etc., using standard L2 switch to L3/L4 device protocol. For example, the Primary VLAN Number may be written into the L3 data field 410 as part of a Layer 4 (L4) header using a standard VLAN protocol. The receiving network device reads the primary VLAN number from the header, writes it into column 504, and makes a one-to-one correspondence with a layer 3 interface number (L3 Interface Number) which is written into column 502. Table 500 then may have multiple entries in column 506 for a many to one correspondence. That is, there may be many entries in column 506, one for the isolated VLAN, and one entry for each community VLAN associated with that primary VLAN.
Rows, for example, row 510 of promiscuous port assignment table 500 for outgoing traffic, contain an entry for each Layer 3 Interface Number. A Layer 3 Interface Number corresponds to a L3 destination address to which a Layer 3/Layer 4 (L3/L4) device 140, etc., transfers data packets in computer network 100.
In operation, a packet arrives at a promiscuous port on an isolated VLAN or a community VLAN for transmission out ofL2 switch 102. A process enters Promiscuous Port Assignment Table for Outgoing Traffic 500 through either the isolated VLAN number or the community VLAN number, thereby obtaining the corresponding L3 Interface Number from column 502 of the entry. The Primary VLAN directs the packet from the L2 switch 102 to the proper L3/L4 device 140, etc., using a protocol for transfer of packets from a L2 switch to a L3!L4 device. The L3/L4 device then interprets the Primary VLAN and directs the packet to the appropriate destination address in Network Cloud 142, etc.
Alternatively, the Primary VLAN of the destination computer could be held in Column 504 of Promiscuous Port Assignment Table for Outgoing Traffic 500, and the packet transferred, for example by TCP/IP, from L2 switch 102 to the L3!L4 device.
In the conceptual table "Promiscuous Port Assignment Table for Outgoing Traffic", Table 500 there is a one-to-one correspondence between a Primary VLAN number and a L3 Interface number. An L3 Interface, designated by L3 Interface Number, is usually associated to a subnet, that is to a whole group of addresses. Once the packets reach an L3 Interface, then thy are normally routed by the router without any remaining knowledge of the Private VLANs. At the L3 Interface there is no distinction between normal traffic, and traffic coming from a private VLAN.
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page13 of 17
US 6,741,592 Bl 7
During operation, a packet such as network packet 400 shown in FIG. 4, is received by an L3!L4 device, for example, L3/L4 device 140, etc. from a network cloud, for example, network cloud 142. The received packet has the field structure as shown in fields 400 of FIG. 4. The network 5
packet is transferred by the receiving L3!L4 device to L2 switch 102. L2 switch 102 receives the packet on a promiscuous port, for example, port 104, 106, ... , 108. Upon receipt by a promiscuous port, the packet is transferred to primary VLAN 230, 330 as shown in FIG. 2 or FIG. 3 10
respectively. The packet then is transferred to each of the isolated ports 204, 206, 208 ... 214, etc and community ports 304, ... 314, etc. The packet is transmitted out of the appropriate isolated port or community port by the L2 switch 102 using standard forwarding mechanisms, for 15
example by TCP/IP. A typical entry for a Primary VLAN is shown at entry
510. Entry 510 shows the one-to-one correspondence between the L3 Interface Number held in field 512 and the Primary VLAN Number held in field 514. Associated with 20
entry 510 are a plurality of entries for isolated or community VLANs, as shown in fields 516, 518, 520, and a possible extension to further "many" entries shown by "three dots"
8 ignations are sometimes referred to as a "color", as is indicated in field 602. Field 604 contains the port number of the port designated to receive that particular packet. Field 605 contains any other fields carried by the packet as it travels through the internals of L2 switch 102.
When packet 600 represents a packet received at a promiscuous port, then field 604 contains the port number of the isolated port 204,206,208, ... 214, etc., or community port 304, 306, ... 314, etc., to which the packet is directed. The port circuitry reads field 604 and the correct port then receives the packet.
When packet 600 represents a packet received from an isolated or community port, the isolated or community VLAN number is written into field 602, and the port number of the promiscuous port designated to receive the packet is written into field 604.
When the receiving port is a community port, a distinction is made between unicast packets and broadcast packets. When the packet is a unicast packet, and in the rare event that the hardware has not yet learned the packet destination address, the packet is broadcast on the community ports so that the hardware can learn the address-port-association. Subsequent unicast packets to this particular community
522. 25
address are then forwarded out through the appropriate port. As an example, primary VLANs and secondary VLANs
(that is Isolated or Community VLANs) are programmed in the router using Color Blocking Logic (CBL). A special value is programmed for all primary and secondary VLANs. For example, a value of "forwarding" as defined in the Spanning Tree Protocol Standard IEEE 802.1D may be used. 30
This exemplary assignment allows the hardware to let all the traffic from those VLANs out of the port, and also to accept the ingress traffic for the primary VLANs.
In the alternative event that the incoming packet is a broadcast packet, the packet is replicated and forwarded out through each of the community ports of the community of designated ports.
Field 608 contains the L3 header of the underlying packet. Field 610 contains the data which is/was transmitted through the Internet. Field 612 contains the trailing fields of the underlying packet.
In the event that the port needs to be able to map many-secondaries-to-one-primary only, this exemplary mapping method is sufficient to define the promiscuous port.
Turning now to FIG. 7, isolated or community port 35 assignment table 700 is shown.
A port having mapping of many-secondaries-to-one-primary only port is referred to as a "non-trunk" promiscuous port.
Alternatively, in the event that the port needs to be able to 40
map many-secondaries-to-different-primaries, then an explicit table such as "Trunk Type Promiscuous Port VLAN Mapping Table" 550 as given in FIG. SA may be employed to provide the required mapping. A port which maps manysecondaries-to-different-primaries is referred to as a "trunk"
45 type promiscuous port. Turning now to FIG. SA, column 552 holds an indicia of the Primary VLAN. Column 554 contains an indicia of the Secondary VLANs (either Isolated or Community VLANs) corresponding to the Primary VLAN.
For example, entries 560 refer to Primary VLAN number 50 "2". Entries 570 refer to Primary VLAN number "3", etc.
Isolated or community port assignment table 700 contains entries for directing a packet received from outside of switch 102 by an isolated or community port. Column 702 contains the isolated or community port number from which a packet is received, for example from a user LAN. Column 706 contains the designation of the isolated or community VLAN associated with that isolated or community port.
A typical entry 710 of isolated or community port assignment table 700 is shown. The isolated or community port number is found in field 712. The designation of the isolated or community VLAN associated with that isolated or community port is found in field 716.
During typical operation, a packet is received at an isolated or community port, port 114, 116, ... 118, etc. from an external LAN connected to the port. A process in L2 switch 102 uses the port number at which the packet was received as an entry into table 700 at column 702, and finds the receiving port number in field 712. The process then
Primary VLAN "2" is shown associated with: Secondary VLAN "20" at entry 560A; Secondary VLAN "21" at entry 560B; Secondary VLAN "22" at entry 560C; Secondary VLAN "23" at entry 560D, etc.
Further, Primary VLAN "3" is shown associated with: 55 reads the isolated or community VLAN to which the packet
is to be transferred from field 716.
Secondary VLAN "30" at entry 570A; Secondary VLAN "31" at entry 570B; with Secondary VLAN "32" at entry 570C, etc. Entries 580, represented by "three dots" in both column 552 and 554, indicate that a further plurality of 60
Primary VLANs may each be associated with its particular plurality of secondary VLANs by use of "Trunk Type Promiscuous Port VLAN Mapping Table" 550.
Turning now to FIG. 6, packet 600 is shown. Packet 600 is the VLAN packet travelling inside L2 switch 102. Fields 65
of packet 600 are shown. Field 602 contains the VLAN designation to which the packet is transferred. VLAN des-
Turning now to FIG. 8, computer network 800 is shown in an alternative embodiment of the invention. Access switch 802 and access switch 804 are typically L2 switches. Distribution switch 806 and distribution switch 808 are also both typically L2 switches. The access switches and the distribution switches are trunked together so as to share VLANs.
Computer network 800 has two layers of Layer 2 switching, the lower layer comprises access switch 802, and access switch 804. The higher, or second, level of Layer 2 switching in network 800 comprises distribution switch 806
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page14 of 17
US 6,741,592 Bl 9
and distribution switch 808. Typical Layer 2 switch trunk connections 860, 862, 864, and 866 are shown. Trunk connection 860 connects access switch 802 with distribution switch 808; trunk 862 connects access switch 804 with distribution switch 806; trunk connection 864 connects 5
access switch 802 with distribution switch 806; and, trunk connection 866 connects access switch 804 with distribution switch 808. The trunk connections, 860, 862, 864, 866 are typical standard engineering practice trunk connections between Layer 2 switches. The trunk connections carry all of 10
the VLANs interconnecting the access switches 802, 804 with the distribution switches 806, 808.
The two layers of Layer 2 switching, represented by the lower layer of access switches 802, 804 and the upper layer represented by distribution switches 806, 808, are a gener- 15
alization of L2 switch 102. The two layer switching arrangement in network 800 at Layer 2 permits the implementation of more ports in the network so that a greater number of server users, for example, 122, 126, 132, etc. may be serviced by the system. 20
Access switch 802 has isolated or community ports 810, 812, ... 814, etc., and these isolated or community ports are analogous to isolated or community ports 114, 116, ... 118 , etc. of L2 switch 102. Access switch 804 also has similar isolated or community ports 816, 818, . . . 820, etc. The 25
isolated or community ports are connected to external LANs which in turn connect to customer's servers, customer's other equipment, etc., as shown in FIG. 1.
Distribution switch 806 and distribution switch 808 have promiscuous ports connected to Layer 3 routers. Distribu- 30
tion switch 806 has promiscuous port 830, promiscuous port 832, ... promiscuous port 834, etc. Distribution switch 808 has promiscuous port 844, 846, ... 848, etc.
Trunk connections 860, 862, 864, 866, etc. carry the 35
primary VLANs, the isolated VLANs, and the community VLANs interconnecting the promiscuous ports, the isolated ports, and the community ports.
Network 800 is analogous to L2 switch 102, in that the access switches 802, 804 provide the isolated or community 40 ports, the distribution switches 806, 808 provide the promiscuous ports, and the trunk lines 860, 862, 864, 866 carry the necessary VLANs. Also, a further plurality of L2 switches may be trunked together as access switches to provide a desired number of ports for customer's equipment. 45 Also, a further plurality of L2 switches may be trunked together as distribution switches to provide more connections to routers connecting to the Internet.
As an example, promiscuous port 830 of distribution switch 806 is shown connected to router 850. In turn, router 50
850 connects to network cloud 852, which is labeled "backbone to Internet". Network cloud 852 is typically a connection to the Internet, and alternatively represent the world wide Internet itself. Also, distribution switch 808 has port 848 shown connected, for example, to router 854. Router 55
854 also connects to network cloud 852. In operation, router 850 and router 854 connect the distribution layer switches 806, 808 to the Internet.
In the exemplary embodiment of the invention described above, the primary VLAN 230, 330 connects to all of the 60
promiscuous ports, however in an alternative exemplary embodiments of the invention, a single primary VLAN may connect to only a subset of promiscuous ports. In such an alternative embodiment of the invention, there may be a plurality of primary VLANs, each with its associated pro- 65
miscuous ports and associated isolated or community ports. Implementing a plurality of primary VLANs gives a system
10 designer flexibility in arranging connections to L3/L4 devices through promiscuous ports, and to user equipment connected at isolated ports or community ports.
It is to be understood that the above described embodiments are simply illustrative of the principles of the invention. Various other modifications and changes may be made by those skilled in the art which embody the principles of the invention and fall within the spirit and scope thereof.
What is claimed is: 1. A method of implementing virtual local area networks
(VLANs) in a switch in a computer network, comprising: establishing at least one promiscuous port, said promis
cuous port receiving packets from an external circuit connected to said promiscuous port;
transferring a packet by a first VLAN within said switch from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and
transferring packets by a second VLAN from said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.
2. The method as in claim 1 further comprising: directing a first user's packets, said first user's packets
received by a first isolated port assigned to said first user, from an external circuit connected to said first isolated port, to a selected promiscuous port by said isolated VLAN, and preventing any other isolated port from receiving said first user's packets from said isolated VLAN.
3. The method as in claim 1 further comprising: transferring a packet by a community VLAN from a
community port to both said plurality of promiscuous ports, and to all other community ports connected to said community VLAN, but not to any other isolated ports or community ports.
4. A method of implementing virtual local area networks (VLANs) in a switch in a computer network, comprising:
transferring first packets by a first VLAN from at least one promiscuous port to a plurality of isolated ports, said at least one promiscuous port receiving said first packets from an external circuit connected to said at least one promiscuous port, each of said isolated ports detecting a packet addressed to a selected isolated port, and only said selected isolated port receiving said packet; and
transferring second packets by a second VLAN from said plurality of isolated ports to said at least one promiscuous port, a selected isolated port of said plurality of isolated ports receiving said second packets from an external circuit connected to said selected isolated port, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.
5. The method as in claim 4 further comprising: directing a first user's packets, said first user's packets
received by a first isolated port assigned to said first user from an external circuit connected to said first isolated port, to a selected promiscuous port by said second VLAN, and preventing any other isolated port from receiving said first user's packets from said second VLAN.
6. A switch, comprising: a promiscuous port for receiving incoming packets from
an external network, and for transmitting outgoing packets to said external network; and
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page15 of 17
US 6,741,592 Bl 11
a plurality of isolated ports, a selected isolated port of said plurality of isolated ports connected to a selected private network, said selected isolated port receiving packets from said selected private network and transmitting packets onto said selected private network, said 5
selected isolated port exchanging packets with said promiscuous port through a path inside said switch, and said isolated port not exchanging packets with another isolated port.
7. The switch of claim 6 further comprising: 10
12 first user, from an external circuit connected to said first isolated port, to a selected promiscuous port by said isolated VLAN, and preventing any other isolated port from receiving said first user's packets from said isolated VLAN.
13. The switch of claim 11, further comprising: means for transferring a packet by a community VLAN
from a community port to both said plurality of promiscuous ports, and to all other community ports connected to said community VLAN, but not to any other port of said switch. a plurality of community ports, each of said community
ports of said plurality of community ports receiving packets from a selected external network and transmitting packets onto said selected external network, each port of said community of ports exchanging packets through a path internal to said switch with said promiscuous port, and said each port of said community of ports exchanging packets with all ports of said plurality
14. A computer readable media containing instructions for a computer for executing the method of claim 1, or claim 4.
15. Signals on a computer network, said signals readable 15 by a computer, said signals carrying instructions for execut
ing the method of claim 1, or claim 4.
of community ports through a path within said switch, and said each port of said community of ports not 20
exchanging packets with any other port of said switch through a path within said switch.
8. The switch of claim 6 further comprising:
a plurality of virtual local area networks (VLANS) imple-mented inside said switch; 25
a first VLAN of said plurality of VLANs designated as a primary VLAN, said primary VLAN carrying packets from said promiscuous port to said plurality of isolated ports, and only a designated isolated port receiving said
30 packet; and
a second VLAN of said plurality of VLANs designated as an isolated VLAN, said isolated VLAN carrying packets from said plurality of isolated ports to said promiscuous port.
9. The switch as in claim 8 further comprising:
a plurality of community ports; and a third VLAN of said plurality of VLANs designated as
35
16. A method of implementing virtual local area networks (VLANs) in a switch in a computer network, comprising:
receiving, by at least one promiscuous port, packets from an external circuit connected to said promiscuous port;
transferring packets by a first VLAN from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and
transferring packets by a second VLAN from said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.
17. A method of implementing virtual local area networks (VLANs) in a switch in a computer network, comprising:
receiving a user's packet, by a first isolated port assigned to said user, said packet received from an external circuit connected to said first isolated port; and
transferring said packet by an isolated VLAN to a selected promiscuous port to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connection from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports.
a community VLAN, said community VLAN carrying packets received from each community port of said 40
plurality of community ports to said promiscuous port, and said community VLAN carrying said community packets to each community port of said plurality of community ports, but not carrying said community packets to any other port of said switch. 45
18. A method of implementing virtual local area networks (VLANs) in a switch in a computer network, comprising:
10. The switch of claim 6 further comprising: said switch is a layer 2 switch.
11. A switch in a computer network, comprising
means for implementing a plurality of virtual local area networks (VLANs) in said switch, comprising: 50
means for establishing at least one promiscuous port, said promiscuous port receiving packets from an external circuit connected to said promiscuous port; p1 means for transferring a packet by a first VLAN of said
55 plurality of VLANs from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and
means for transferring packets by a second VLAN of said plurality of VLANs from said plurality of isolated ports 60 to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.
12. The switch of claim 11, further comprising: means for directing a first user's packets, said first user's
packets received by a first isolated port assigned to said
65
receiving a user's packet, by a first community port assigned to said user, said packet received from an external circuit connected to said first community port; and
transferring said packet by a community VLAN to both a plurality of promiscuous ports connected to external circuits, and to all other community ports connected to said community VLAN, but not to any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports said all promiscuous ports also connected via a one way primary VLAN to all community ports.
19. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:
at least one first promiscuous port to receive packets from an external circuit connected to said promiscuous port;
a plurality of isolated ports to receive packets through a first VLAN from said at least one promiscuous port, and only a selected isolated port receiving said packet; and
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page16 of 17
US 6,741,592 Bl 13
at least one second promiscuous port to receive packets through a second VLAN from said plurality of isolated ports, said second VLAN configured so that a packet received through said second VLAN by a first isolated port cannot be received and retransmitted by a second 5
isolated port. 20. A switch implementing virtual local area networks
(VLANs) in a computer network, comprising:
a first isolated port assigned to a user to receive said user's packet from an external circuit connected to said first 10
isolated port; and
a selected promiscuous port to receive said packet through an isolated VLAN, said packet to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connec- 15
tion from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports. 20
21. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:
a plurality of community ports, including a first community port assigned to a user to receive said user's packet
25 from an external circuit connected to said first community port; and
a plurality of promiscuous ports connected to external circuits to receive said packet through a community VLAN, all other community ports connected to said 30 community VLAN also receiving said packet, but not any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports, said all promiscuous ports also connected via a 35 one way primary VLAN to all community ports.
22. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:
means for receiving, by at least one promiscuous port, packets from an external circuit connected to said 40
promiscuous port;
means for transferring packets by a first VLAN from said at least one promiscuous port to a plurality of isolated ports, and only a selected isolated port receiving said packet; and
14 means for transferring packets by a second VLAN from
said plurality of isolated ports to said plurality of promiscuous ports, said second VLAN configured so that a packet transferred to said second VLAN by a first isolated port cannot be received and retransmitted by a second isolated port.
23. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:
means for receiving a user's packet, by a first isolated port assigned to said user, said packet received from an external circuit connected to said first isolated port; and
means for transferring said packet by an isolated VLAN to a selected promiscuous port to be transferred to an external circuit connected to said promiscuous port, said isolated VLAN configured as a one way connection from all isolated ports to all promiscuous ports and also configured to prevent any other isolated port from receiving said user's packets from said isolated VLAN, said all promiscuous ports also connected via a one way primary VLAN to said all isolated ports.
24. A switch implementing virtual local area networks (VLANs) in a computer network, comprising:
means for receiving a user's packet, by a first community port assigned to said user, said packet received from an external circuit connected to said first community port; and
means for transferring said packet by a community VLAN to both a plurality of promiscuous ports connected to external circuits, and to all other community ports connected to said community VLAN, but not to any other ports of said switch, said community VLAN configured as a one way connection from all community ports in said community VLAN to all promiscuous ports, said all promiscuous ports also connected via a one way primary VLAN to all community ports.
25. A computer-readable media, comprising: said computer-readable media containing instructions for execution in a processor for the practice of the method of claim 16, or claim 17, or claim 18.
26. Electromagnetic signals propagating on a computer network, comprising: said electromagnetic signals carrying instructions for execution on a processor for the practice of the method of claim 16, or claim 17, or claim 18.
* * * * *
Case3:14-cv-05343 Document1-9 Filed12/05/14 Page17 of 17
Exhibit 10
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page1 of 22
c12) United States Patent Edsall et al.
(54) PRIVATE VLANS
(75) Inventors: Thomas J. Edsall, Cupertino, CA (US); Marco Foschiano, San Jose, CA (US); Michael Fine, San Francisco, CA (US); Thomas Nosella, Sunnyvale, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 302 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 10/840,212
(22) Filed: May 5, 2004
(51) Int. Cl. H04L 12156 (2006.01)
(52) U.S. Cl. ....................... 370/389; 370/401; 709/225 (58) Field of Classification Search ................ 370/389,
(56)
370/392, 401, 464, 465; 709/218, 221, 225 See application file for complete search history.
References Cited
U.S. PATENT DOCUMENTS
5,959,989 A 9/1999 Gleeson et a!. 6,058,429 A 5/2000 Ames eta!. 6,111,876 A 8/2000 Frantz et a!. 6,147,995 A 1112000 Dobbins et a!. 6,208,649 B1 3/2001 Kloth 6,304,901 B1 10/2001 McCloghrie et a!. 6,560,236 B1 5/2003 Varghese et al.
120
122 126
111111 1111111111111111111111111111111111111111111111111111111111111 US007200145Bl
(10) Patent No.: US 7,200,145 Bl *Apr. 3, 2007 (45) Date of Patent:
OTHER PUBLICATIONS
Tanenbaum, Andrew, Computer Networks, Third Edition, Prentice Hall, 1996, pp. 417-419.
Primary Examiner-Buy D. Vu Assistant Examiner-Toan Nguyen (74) Attorney, Agent, or Firm-Cesari and McKenna LLP
(57) ABSTRACT
The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three new types of ports are defined, "promiscuous" ports "isolated" ports, and "community" ports. Three types ofVLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs. The promiscuous ports are connected to layer 3 or layer 4 devices. Isolated ports and community ports are connected to individual user's servers, etc., and maintain traffic for each user separate from other users. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports. An isolated VLAN connects to all promiscuous ports and to all isolated ports. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports. A community VLAN is defined as connecting to a group of community ports, and also connecting to all of the promiscuous ports. The group of community ports is referred to as a "community" of community ports. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected to that community VLAN.
46 Claims, 8 Drawing Sheets
COMMUNITY OR ISOlATED PORTS
USER#M
132
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page2 of 22
U.S. Patent Apr. 3, 2007 Sheet 1 of 8 US 7,200,145 Bl
140
122
L3/L4 DEVICE
100
NETWORK CLOUD
152
NETVVORK CLOUD
...------'-----..., ~ ...------'-----...,
L3/L4 DEVICE
L3/L4 DEVICE
146
108 1043- NJ PROMISCUOUS PORTS
r-'"'"----'"""--....LC£a..,.
120
102
114 USER #1
SERVER USER#1
126
L2 SWITCH
124
SERVER USER#2
FIG. 1
COMMUNITY OR ISOLATED PORTS
USER#M
'----y--/
136
130----
SERVER USER #M
132
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page3 of 22
U.S. Patent
102 ~
TED ISOLA PORTS
Apr. 3, 2007 Sheet 2 of 8
PROMISCUOUS PORTS
#A #B 232 .A ..
/ '"'\
220- ~222 . . .
---230 PRIMARY VLAN
L2 SWITCH
204- 206~ 2oa~ 210~ 212~
.
US 7,200,145 Bl
#N
224-
240-ISOLATED
VLAN
~214
. . USER# 1 2 3 4 s"~N
230
FIG. 2
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page4 of 22
U.S. Patent
102 -
COMMU PORTS
NITY
Apr. 3, 2007 Sheet 3 of 8
PROMISCUOUS PORTS
#A #B 232
.A, / ' 320- ~--"-322 . . .
~--"-330
PRIMARY VLAN
,, L2 SWITCH
COM. VLAN#3 COM. VLAN#2
COM. VLAN#1
304~ 306- 308~ 310~ 312-.
USER# 1 2 3 4
FIG. 3
US 7,200,145 Bl
#N
324-
r-
354-
352- --..
~350
~-------
'-----
~314
. .
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page5 of 22
U.S. Patent Apr. 3, 2007
402 "\ 404 "\
Sheet 4 of 8
400
406 )"\ 410 '"\
US 7,200,145 Bl
412 ""\
PREAMBLE L2 HEADER L3 HEADER DATA TRAILING
502 "\ L3 LAYER
INTERFACE NUMBER
510-< I
512)
FIELDS
PACKET
FIG. 4
500 504 \ 506 \ PRIMARY ISOLATED OR COMMUNITY
VLAN VLAN NUMBER NUMBER
I
514/
PROMISCUOUS PORT ASSIGNMENT TABLE
FOR OUTGOING TRAFFIC
FIG. 5
. . .
I--
I--
~-
516
518 520
~22
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page6 of 22
U.S. Patent
552 '\
560
570
580
Apr. 3, 2007 Sheet 5 of 8
550 554 \
PRIMARY SECONDARY VLAN VLANs
2 20
2 21
2 22
2 23
3 30
3 31
3 32
• . . . • .
TRUNK TYPE PROMISCUOUS PORT VLAN MAPPING TABLE
FIG. 5A
US 7,200,145 Bl
--..--..
--!-
r-
560A
5608
560C
5600
570A
5708
570C
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page7 of 22
U.S. Patent Apr. 3, 2007 Sheet 6 of 8 US 7,200,145 Bl
602 >"\ 604 605 606 t""'\ '"""'\ '\ 608 ''\ 610 '"""'\ 612 :"""'\
VLAN PORT TRAILING DESIGNATION NO OTHER L2 HEADER L3 HEADER DATA
FIELDS (COLOR)
PACKETS INTERNAL TO L2 SWITCH
FIG. 6
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page8 of 22
U.S. Patent Apr. 3, 2007 Sheet 7 of 8 US 7,200,145 Bl
700 702 ., 706
'-~ PORT No. ISOLATED OR COMMUNITY
VLAN
\ ( ) \
712 716
ISOLATED OR COMMUNITY PORT ASSIGNMENT TABLE
FIG. 7
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page9 of 22
U.S. Patent Apr. 3, 2007
850
834
830
806 DISTRIBUTION
SWITCH L2
864
ACCESS 802 SWITCH
L2
810 814
Sheet 8 of 8
BACKBONE TO
INTERNET
860 862
FIG. 8
846 844
816
US 7,200,145 Bl
800
'\
848
DISTRIBUTION SWITCH 808
L2
866 TRUNK CONNECTION
ACCESS SWITCH 804
L2
l ISOLATED 818 820 PORTS
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page10 of 22
US 7,200,145 Bl 1
PRIVATE VLANS
BACKGROUND OF THE INVENTION
1. Field of the Invention
The invention relates to Virtual Local Area Networks (VLANs), and more particularly to the use of VLANs to establish separation between different users of a shared switch.
2. Background Information
It is today a common computer network engineering practice to separate packet traffic belonging to different users
2 A better way to keep the message traffic of different users
separate in a computer network is needed, particularly a method which can scale to a large number of users.
SUMMARY OF THE INVENTION
The invention uses a layer 2 switch (L2 switch), or bridge, to separate user's message traffic by use of Virtual Local Area Networks (VLANs) defined within the switch. Three
10 new types of ports are defined, "promiscuous" ports, "isolated" ports, and "community" ports. Three types ofVLANs internal to the switch are defined, "primary" VLANs, "isolated" VLANs and "community" VLANs.
The promiscuous ports are connected to layer 3 or layer 4 devices, for example routers which may in turn connect to the worldwide Internet, load balancers which also may connect to the worldwide Internet, administrative work stations such as used by network administrators, back up devices, etc. Isolated ports and community ports are con-nected to individual user's servers, etc., and maintain traffic for each user separate from other users.
Isolated ports and community ports exchange packets with the promiscuous ports by use of the VLANs internal to the switch. The difference between isolated and community ports is that an isolated port cannot transfer packets to another isolated port, however a community port has a designated number of community ports to which it can transfer packets.
by use of a router, a Layer 3 (L3) device. Separation of users' traffic is accomplished by assigning each user to a 15
different subnetwork (subnet). A subnet is identified by a unique L3 address. The router then transmits a particular user's packets out through a port assigned to that subnet. However, only a limited number of bits in the L3 address (for example IP address) are assigned to the subnet, and so 20
only a limited number of subnets may be addressed by a particular router. Subnet design is described by Andrew Tanenbaum in his book Computer Networks, Third Edition, published by Prentice Hall, Copyright date 1996, all disclosures of which are incorporated herein by reference, par- 25
ticularly at pages 417-419. For example, if 6 bits are assigned to a subnet mask, then only 62 different subnets may be addressed (0 and 64 are reserved). Further, for every subnet assigned two addresses are wasted, for example the multicast and broadcast addresses. 30
A primary VLAN internal to the switch is defined as follows. The primary VLAN connects to all promiscuous ports, to all isolated ports, and to all community ports. The primary VLAN receives packets from outside of the switch arriving at any of the promiscuous ports, and transfers the
As an exan1ple of many users of a switch who require that their message traffic be kept separate, an Internet service provider (ISP) may have many customers who want to connect to a server farm. Access to the ISP is through a router connected to a common external computer network, for exan1ple the worldwide Internet. The router must route each customer's traffic to that customer's local area network in such a man11er as to maintain protection and privacy between the data of different customers. It is desirable for an
35 packets to the isolated or community ports. However, an isolated or community port cannot receive traffic from the external LAN connected to it, and transfer the packets to the primary VLAN. The primary VLAN is a one way connection from promiscuous ports to isolated or community ports.
ISP to prevent traffic originating from one customer's server 40
from being received by another customer's server.
An isolated VLAN is defined as connecting to all pro-miscuous ports and connecting to all isolated ports. An isolated VLAN receives packets arriving from outside of the switch at an isolated port, and transfers the packets to the promiscuous ports. An isolated VLAN does not carry pack-
A second example of many users of a computer network who must have their traffic separated in order to guarantee privacy and protection is the use of a television cable Internet distribution system. Each home is assigned a separate subnet so that routers may route only a particular customer's message traffic to that customer. This subnet routing prevents, for example, one customer looking at another customer's message traffic by use of, for exan1ple, a network snifter.
A third exan1ple is a server farm, for example a multiclient backup service. Each client's message traffic arrives at a router. The router uses a subnet mask to keep the traffic of each client separate from the traffic of another client, as it routes the traffic to the client's backup server.
A limitation in the use of subnets, and subnet masks, in a multiclient environment is that there is only a limited number of subnets which can be defined from standard Layer 3 addresses. In modern computer network systems, this numerical limitation severely restricts the number of individual users who can be serviced, and also have their message traffic maintained separate. Further, the management of a large number of subnets by a network manager becomes burdensome, especially in the event that the network has thousands of customers whose packet traffic must be kept separate.
45 ets received by a promiscuous port from outside of the switch. Also, an isolated VLAN does not deliver any packets to another isolated port. The isolated VLAN is a one way connection from an isolated port to the promiscuous ports.
A community VLAN is defined as connecting to a group 50 of community ports, and also connecting to all of the
promiscuous ports. The group of community ports is referred to as a "community" of community ports. The community VLAN transfers a packet received from outside the switch at a community port to all of the promiscuous
55 ports, and also transfers the packet to the other community ports attached to that community VLAN. A plurality of "communities" of community ports may be defined, and each community of ports has its own assigned community VLAN. A community VLAN cannot transfer packets
60 received from outside of the switch at a promiscuous port. A community VLAN is a one way connection from a community of ports to the promiscuous ports, but allows a packet received by one community port to be transmitted out of the switch, through the other community ports connected
65 to that community VLAN. These new types ofVLANs and ports are implemented, in
part, by particular settings of the Color Blocking Logic
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page11 of 22
US 7,200,145 Bl 3
(CBL) logic circuits used by normal ports of an L2 switch which supports VLANs, and also by use of assignment tables.
4 administrator and permits the administrator to view all, or at least many, of the occurrences on the network. Promiscuous Port A 104 connects to layer 3 or layer 4 device (L3/L4
Traffic generated by different user's servers is kept separate from other user's servers, by each user having his own 5
isolated port or community of community ports.
device) A 140, and L3/L4 device 140 connects to network cloud 142. Promiscuous port B 106 connects to L3/L4 device 143, and L3/L4 device 143 connects to network cloud
The VLANs defined in a first L2 switch chassis can be trunked to other L2 switch chassis using ordinary trunking technology, in order to increase the number of ports.
144. Promiscuous port N 108 connects to L3/L4 device 146. L3/L4 device 146 connects to network cloud 148.
Alternatively, a single L2 switch, or a network oftrunked 10
L2 switches, may have its promiscuous ports divided into subsets. Each subset of promiscuous ports is then associated with its subset of isolated ports and community ports, along with the necessary VLANs.
Three dots 150 indicate that L2 switch 102 may have a plurality of promiscuous ports, etc. Three dots 152 indicate a plurality of L3/L4 devices, connected to the promiscuous ports, portA 104, port B 106, port N 108, etc. Three dots 149 indicate that the L3/L4 devices may connect to a plurality of network clouds 142, 144, 148, etc.
Other and further aspects of the present invention will 15
become apparent during the course of the following description and by reference to the accompanying drawings.
Network clouds 142, 144, 148 may be different network clouds, for example each may comprise a backup device for a particular user server. Alternatively, each network cloud 142, 144, 148 may represent the worldwide Internet. Further, each network cloud 142, 144, 148, may represent a particu-BRIEF DESCRIPTION OF THE DRAWINGS
Referring now to the drawings, in which like numerals represent like parts in the several views:
FIG. 1 is a block diagram of a computer network in accordance with the invention;
FIG. 2 is a block diagram of a L2 switch in accordance with the invention;
FIG. 3 is a block diagram of a L2 switch in accordance with the invention;
FIG. 4 is a field diagram of a layer 3 packet; FIG. 5 is an assignment table for a promiscuous port for
outgoing traffic, in accordance with the invention; FIG. SA is a Trunk Type Promiscuous Port VLAN Map
ping Table, in accordance with the invention; FIG. 6 is a field diagram of a VLAN packet internal to a
L2 switch; FIG. 7 is a port assignment table for an isolated or
community port; FIG. 8 is a block diagram of a two level layer 2-switch
network in accordance with the invention.
DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT
Turning now to FIG. 1, computer network 100 is shown. L2 switch 102 has promiscuous ports, port A 104, port B
106, port N 108, etc. Promiscuous port 108 is indicated as "N", indicating that an arbitrary number of promiscuous ports may be employed by L2 switch 102.
L2 switch 102 also has isolated or community ports, port #1 114 is connected to user #1 VLAN 120, and user #1 VLAN 120 connects to user #1 server 122. Isolated or community port #2 116 connects to user #2 VLAN 124, and user #2 VLAN 124 connects to user #2 server 126. Isolated or community port #M 118 is labeled "M" to indicate that L2 switch 102 may have an arbitrary number of isolated or community ports. Isolated or community port #M 118 connects to user #M VLAN 130, and user #M VLAN 130 connects to user #M server 132. "Three dots" 134 indicate that L2 switch 102 may have a plurality of isolated or community ports, etc. "Three dots" 136 indicate that a plurality of user servers, each connected to a different isolated or community port, etc.
The promiscuous ports 104, 106, 108, etc. connect to layer 3 or layer 4 devices 140,143,146. Examples of layer 3
20 lar device, may represent several particular devices, and may also represent the worldwide Internet, etc.
Turning now to FIG. 2, the interior of L2 switch 102 is shown. Isolated port 204, 206, 208, 210, 212, 214 are labeled progressively as user #1, 2, 3, 4, 5, N, etc., and each may
25 connect to a different user. Isolated ports 204, 206, 208, etc. correspond to isolated ports 114, 116, 118, etc. shown in FIG. 1. Promiscuous ports, port A 220, port B 222, port N 224, connect to various L3/L4 devices (not shown in FIG. 2), such as devices 140, 143, 146, etc. Promiscuous ports
30 220, 222, 224, etc. correspond to promiscuous ports 104, 106, and 108, etc. as shown in FIG. 1.
Three dots 230 indicate a plurality of users, each connected to an isolated port 204, 206, 208, 210, 212, 214, etc. Three dots 232 indicate a plurality of promiscuous ports,
35 220, 222, 224, etc. and indicate that L2 switch 102 may have a plurality of promiscuous ports.
The VLANs utilized by L2 switch 102 are described below.
VLAN 230 is a primary VLAN, and connects to promis-40 cuous ports 220, 222, 224, etc., and also connects to each of
the isolated ports 204, 206, 208, ... 214, etc. Primary VLAN 230 carries packet traffic from the promiscuous ports to isolated ports. Primary VLAN 230 is configured to reject any packets arriving at an isolated port from the external
45 local area network connected to the isolated port. During ordinary operation, any packet received by a
promiscuous port from the L3/L4 device is transmitted on the primary VLAN, and may be received by any isolated ort or community port having a destination for that packet on an
so external LAN connected thereto. Isolated VLAN 240 connects to isolated ports 204, 206,
208, . . . 214, etc., and also connects to each of the promiscuous ports 224, 222, ... 220, etc. Isolated VLAN 240 carries packet traffic from isolated ports to the promis-
55 cuous ports. Isolated VLAN 240 is configured to reject any traffic arriving from a promiscuous port. Also, isolated VLAN 240 is configured so that it carmot deliver any packets to an isolated port. That is, packets transferred onto isolated VLAN 240 by an isolated port cannot be received
60 by another isolated port. Packets transferred onto isolated VLAN 240 from an isolated port are received by promiscuous ports 220, 222, 224, etc., and from the promiscuous ports may be transferred to network clouds, for example, network cloud 142, 144, 148.
or layer 4 devices comprise routers, load balancers, admin- 65
istrative work stations, back-up devices, etc. An administrative work station is a work station utilized by a network
Mechanisms, for example, color blocking logic (CBL) and assignment tables, may be used to permit primary VLAN 230 to transfer packets from promiscuous ports to
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page12 of 22
US 7,200,145 Bl 5
isolated ports, and prohibit an isolated port from transmitting onto primary VLAN 230. Also, mechanisms within L2 switch 102 such as CBL and assignment tables may be used to permit isolated VLAN 240 to transfer packet traffic from an isolated port to a promiscuous port, and prevent isolated 5
VLAN 240 from transferring a packet to an isolated port. Community VLANs implemented in L2 switch 102 are
described next.
6 104, etc. to network cloud 142, or any of the other network clouds from one of the other L3/L4 devices.
Turning now to FIG. 5, "Promiscuous Port Assignment Table for Outgoing Traffic" 500 is shown with three colunms. Table 500 is a conceptual table which is an aid to understanding the invention. Data shown in table 500 may be held, in a particular implementation, in a variety of places. For example some data is in the header of a received packet, some data may be held in hardware such as memory A community VLAN connects to a designated group of
community ports, and to all of the promiscuous ports. A community port receives a packet from outside of switch 102 and transfers the packet to the community VLAN. A packet transferred to the community VLAN from a community port is received by all of the community ports connected to the community VLAN, and also all of the promiscuous ports receive the packet from the community VLAN. The promiscuous ports then transfer the packet out
10 in an ASIC chip in the interface, or further, some of the data may be held in a software lookup table in the memory for a processor of the router. As a further example, an implementation may use a table such as Table 500 in main memory for a processor of the router. Colunm 502 contains a layer 3
15 interface number. Column 504 contains a primary VLAN assignment number. Colunm 506 contains an isolated or community VLAN assigrillent number.
of the L2 switch. A Community VLAN is configured to reject any traffic arriving from a promiscuous port.
A primary VLAN Assigrillent Number, as held in column 504, is a designation which is written into a field of a packet
Turning now to FIG. 3, community VLAN #1 350, community VLAN #2 352, and, community VLAN #3 354 are shown. Community VLAN #1 350 is shown connected to community ports 306, and 308. Community VLAN #1 350 permits community ports connected thereto to exchange packets. For example, a packet entering L2 switch 102 from user #2 at community port 306 is transferred by community VLAN #1 350 to the other community ports, for example community ports 308, etc., connected to community VLAN
20 transferred from layer 2 switch 102 to L3/L4 device 140, or, for example, any of the other L3/L4 devices 143, 146, etc., using standard L2 switch to L3/L4 device protocol. For example, the Primary VLAN Number may be written into the L3 data field 410 as part of a Layer 4 (L4) header using
#1 350, and is also transferred to all of the promiscuous ports, ports 320, 322, 324 ....
25 a standard VLAN protocol. The receiving network device reads the primary VLAN number from the header, writes it into colunm 504, and makes a one-to-one correspondence with a layer 3 interface number (L3 Interface Number) which is written into colunm 502. Table 500 then may have
Community VLAN #2 352 is shown connected to community port 310 and 312. A packet originating from user #4
30 multiple entries in colunm 506 for a many to one correspondence. That is, there may be many entries in column 506, one for the isolated VLAN, and one entry for each community VLAN associated with that primary VLAN. or user #5 will enter L2 switch 102 at either community port
310, 312, respectively, and will be transferred by community VLAN #2 352 to the other isolated port, and to all of the 35
promiscuous ports 320, 322, ... 324, etc. As a further example of a community VLAN, community
VLAN #3 354 is shown. Community VLAN 354 is shown connected to community port 304 and community port 314.
40 Community VLAN #3 354 also connects to all of the promiscuous ports 320, 322, ... 324, etc.
In the present description, the isolated ports are shown in FIG. 2, and the community ports are shown in FIG. 3. Switch 102 may have, for example, both isolated ports and
45 community ports. In this case, both of the port arrangements of FIG. 2 and of FIG. 3 are implemented within L2 switch 102. In a second exemplary embodiment, L2 switch 102 may have only isolated ports as shown in FIG. 2. In a third exemplary embodiment, L2 switch 102 may have only
50 community ports as shown in FIG. 3.
A terminology which can be used is to refer to the isolated VLAN and the community VLAN as a "secondary" VLAN. Using this terminology, a primary VLAN takes packets from the promiscuous ports to either the isolated ports or the 55 community ports. In contrast, the secondary VLAN takes packets from either the isolated ports or community ports to the promiscuous ports.
Turning now to FIG. 4, a field diagram 400 of a typical L2 packet which reaches an L2 switch from a network cloud is 60 shown. Field 402 is the preamble. Field 404 contains the L2 header. Field 406 contains the L3 header. Data carried by packet 400 is in field 410. Trailing fields 412 contain fields typically trailing the data fields of a typical data packet, and normally include a cyclical redundancy check (CRC) field. 65
The field diagram of a packet shown in FIG. 4 also represents the fields in a packet departing from L3/L4 device
Rows, for example, row 510 of promiscuous port assignment table 500 for to outgoing traffic, contain an entry for each Layer 3 Interface Number. A Layer 3 Interface Number corresponds to a L3 destination address to which a Layer 3/Layer 4 (L3/L4) device 140, etc., transfers data packets in computer network 100.
In operation, a packet arrives at a promiscuous port on an isolated VLAN or a community VLAN for transmission out ofL2 switch 102. A process enters Promiscuous Port Assignment Table for Outgoing Traffic 500 through either the isolated VLAN number or the community VLAN number, thereby obtaining the corresponding L3 Interface Number from colunm 502 of the entry. The Primary VLAN directs the packet from the L2 switch 102 to the proper L3/L4 device 140, etc., using a protocol for transfer of packets from a L2 switch to a L3/L4 device. The L3/L4 device then interprets the Primary VLAN and directs the packet to the appropriate destination address in Network Cloud 142, etc.
Alternatively, the Primary VLAN of the destination computer could be held in Colunm 504 of Promiscuous Port Assignment Table for Outgoing Traffic 500, and the packet transferred, for example by TCP/IP, from L2 switch 102 to the L3/L4 device.
In the conceptual table "Promiscuous Port Assignment Table for Outgoing Traffic", Table 500 there is a one-to-one correspondence between a Primary VLAN number and a L3 Interface number. An L3 Interface, designated by L3 Interface Number, is usually associated to a subnet, that is to a whole group of addresses. Once the packets reach an L3 Interface, then are normally routed by the router without any remaining knowledge of the Private VLANs. At the L3 Interface there is no distinction between normal traffic, and traffic coming from a private VLAN.
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page13 of 22
US 7,200,145 Bl 7 8
During operation, a packet such as network packet 400 shown in FIG. 4, is received by an L3/L4 device, for example, L3/L4 device 140, etc. from a network cloud, for example, network cloud 142. The received packet has the field structure as shown in fields 400 of FIG. 4. The network 5
indicated in field 602. Field 604 contains the port number of the port designated to receive that particular packet. Field 605 contains any other fields carried by the packet as it travels through the internals of L2 switch 102.
When packet 600 represents a packet received at a pro-packet is transferred by the receiving L3/L4 device to L2 switch 102. L2 switch 102 receives the packet on a promiscuous port, for example, port 104, 106, ... , 108. Upon receipt by a promiscuous port, the packet is transferred to primary VLAN 230, 330 as shown in FIG. 2 or FIG. 3 respectively. The packet then is transferred to each of the isolated ports 204, 206, 208 ... 214, etc and community ports 304, ... 314, etc. The packet is transmitted out of the appropriate isolated port or community port by the L2 switch 102 using standard forwarding mechanisms, for example by TCP/IP.
A typical entry for a Primary VLAN is shown at entry 510. Entry 510 shows the one-to-one correspondence between the L3 Interface Number held in field 512 and the Primary VLAN Number held in field 514. Associated with entry 510 are a plurality of entries for isolated or community VLANs, as shown in fields 516, 518, 520, and a possible extension to further "many" entries shown by "three dots" 522.
As an example, primary VLANs and secondary VLANs (that is Isolated or Community VLANs) are programmed in the router using Color Blocking Logic (CBL). A special value is programmed for all primary and secondary VLANs. For example, a value of "forwarding" as defined in the Spanning Tree Protocol Standard IEEE 802. ID may be used. This exemplary assigmnent allows the hardware to let all the traffic from those VLANs out of the port, and also to accept the ingress traffic for the primary VLANs.
In the event that the port needs to be able to map many-secondaries-to-one-primary only, this exemplary mapping method is sufficient to define the promiscuous port. A port having mapping of many-secondaries-to-one-primary only port is referred to as a "non-trunk" promiscuous port.
Alternatively, in the event that the port needs to be able to map many-secondaries-to-different-primaries, then an explicit table such as "Trunk Type Promiscuous Port VLAN Mapping Table" 550 as given in FIG. SA may be employed to provide the required mapping. A port which maps manysecondaries-to-different-primaries is referred to as a "trunk" type promiscuous port. Turning now to FIG. SA, colunm 552 holds an indicia of the Primary VLAN. Column 554 contains an indicia of the Secondary VLANs (either Isolated or Community VLANs) corresponding to the Primary VLAN.
For example, entries 560 refer to Primary VLAN number "2". Entries 570 refer to Primary VLAN number "3", etc.
Primary VLAN "2" is shown associated with: Secondary VLAN "20" at entry 560A; Secondary VLAN "21" at entry 560B; Secondary VLAN "22" at entry 560C; Secondary VLAN "23" at entry 560D, etc.
miscuous port, then field 604 contains the port number of the isolated port 204, 206, 208, ... 214, etc., or community port 304, 306, ... 314, etc., to which the packet is directed. The port circuitry reads field 604 and the correct port then
10 receives the packet. When packet 600 represents a packet received from an
isolated or community port, the isolated or community VLAN number is written into field 602, and the port number of the promiscuous port designated to receive the packet is
15 written into field 604. When the receiving port is a community port, a distinction
is made between unicast packets and broadcast packets. When the packet is a unicast packet, and in the rare event that the hardware has not yet learned the packet destination
20 address, the packet is broadcast on the community ports so that the hardware can learn the address-port-association. Subsequent unicast packets to this particular community address are then forwarded out through the appropriate port.
In the alternative event that the incoming packet is a 25 broadcast packet, the packet is replicated and forwarded out
through each of the community ports of the community of designated ports.
Field 608 contains the L3 header of the underlying packet Field 610 contains the data which is/was transmitted through
30 the Internet. Field 612 contains the trailing fields of the underlying packet.
Turning now to FIG. 7, isolated or community port assigument table 700 is shown.
Isolated or community port assignment table 700 contains 35 entries for directing a packet received from outside of switch
102 by an isolated or community port. Colunm 702 contains the isolated or community port number from which a packet is received, for example from a user LAN. Column 706 contains the designation of the isolated or community
40 VLAN associated with that isolated or community port. A typical entry 710 of isolated or community port assign
ment table 700 is shown. The isolated or community port number is found in field 712. The designation of the isolated or community VLAN associated with that isolated or com-
45 munity port is found in field 716. During typical operation, a packet is received at an
isolated or community port, port 114, 116, ... 118, etc. from an external LAN connected to the port. A process in L2 switch 102 uses the port number at which the packet was
50 received as an entry into table 700 at colunm 702, and finds the receiving port number in field 712. The process then reads the isolated or community VLAN to which the packet is to be transferred from field 716.
Further, Primary VLAN "3" is shown associated with: 55
Secondary VLAN "30" at entry 570A; Secondary VLAN "31" at entry 570B; with Secondary VLAN "32" at entry 570C, etc. Entries 580, represented by "three dots" in both colunm 552 and 554, indicate that a further plurality of Primary VLANs may each be associated with its particular 60
plurality of secondary VLANs by use of "Trunk Type Promiscuous Port VLAN Mapping Table" 550.
Turning now to FIG. 8, computer network 800 is shown in an alternative embodiment of the invention. Access switch 802 and access switch 804 are typically L2 switches. Distribution switch 806 and distribution switch 808 are also both typically L2 switches. The access switches and the distribution switches are trunked together so as to share VLANs.
Computer network 800 has two layers of Layer 2 switching, the lower layer comprises access switch 802, and access switch 804. The higher, or second, level of Layer 2 switching in network 800 comprises distribution switch 806 and distribution switch 808. Typical Layer 2 switch trunk connections 860, 862, 864, and 866 are shown. Trunk connec-
Turning now to FIG. 6, packet 600 is shown. Packet 600 is the VLAN packet travelling inside L2 switch 102. Fields of packet 600 are shown. Field 602 contains the VLAN 65
designation to which the packet is transferred. VLAN designations are sometimes referred to as a "color", as is tion 860 connects access switch 802 with distribution switch
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page14 of 22
US 7,200,145 Bl 9
808; trunk 862 connects access switch 804 with distribution switch 806; trunk connection 864 connects access switch 802 with distribution switch 806; and, trunk connection 866 connects access switch 804 with distribution switch 808. The trunk connections, 860, 862, 864, 866 are typical standard engineering practice trunk connections between Layer 2 switches. The trunk connections carry all of the VLANs interconnecting the access switches 802, 804 with the distribution switches 806, 808.
The two layers of Layer 2 switching, represented by the 10
lower layer of access switches 802, 804 and the upper layer represented by distribution switches 806, 808, are a generalization ofL2 switch 102. The two layer switching arrangement in network 800 at Layer 2 permits the implementation of more ports in the network so that a greater number of 15
server users, for example, 122, 126, 132, etc. may be serviced by the system.
Access switch 802 has isolated or community ports 810, 812, ... 814, etc., and these isolated or community ports are analogous to isolated or community ports 114, 116, ... 118, 20
etc. of L2 switch 102. Access switch 804 also has similar isolated or community ports 816, 818, ... 820, etc. The isolated or community ports are connected to external LAN s which in turn connect to customer's servers, customer's other equipment, etc., as shown in FIG. 1. 25
Distribution switch 806 and distribution switch 808 have promiscuous ports connected to Layer 3 routers. Distribution switch 806 has promiscuous port 830, promiscuous port 832, ... promiscuous port 834, etc. Distribution switch 808 has promiscuous port 844, 846, ... 848, etc. 30
Trunk connections 860, 862, 864, 866, etc. carry the primary VLANs, the isolated VLANs, and the community VLANs interconnecting the promiscuous ports, the isolated ports, and the community ports.
35 Network 800 is analogous to L2 switch 102, in that the
access switches 802, 804 provide the isolated or community ports, the distribution switches 806, 808 provide the promiscuous ports, and the trunk lines 860, 862, 864, 866 carry the necessary VLANs. Also, a further plurality of L2 40 switches may be trunked together as access switches to provide a desired number of ports for customer's equipment. Also, a further plurality of L2 switches may be trunked together as distribution switches to provide more connections to routers connecting to the Internet. 45
As an example, promiscuous port 830 of distribution switch 806 is shown connected to router 850. In turn, router 850 connects to network cloud 852, which is labeled "backbone to Internet". Network cloud 852 is typically a connection to the Internet, and alternatively represent the world 50
wide Internet itself. Also, distribution switch 808 has port 848 shown connected, for example, to router 854. Router 854 also connects to network cloud 852. In operation, router 850 and router 854 connect the distribution layer switches 806, 808 to the Internet. 55
In the exemplary embodiment of the invention described above, the primary VLAN 230, 330 connects to all of the promiscuous ports, however in an alternative exemplary embodiments of the invention, a single primary VLAN may connect to only a subset of promiscuous ports. In such an 60
alternative embodiment of the invention, there may be a plurality of primary VLANs, each with its associated promiscuous ports and associated isolated or community ports. Implementing a plurality of primary VLAN s gives a system designer flexibility in arranging connections to L3/L4 65
devices through promiscuous ports, and to user equipment connected at isolated ports or community ports.
10 It is to be understood that the above described embodi
ments are simply illustrative of the principles of the invention. Various other modifications and changes may be made by those skilled in the art which embody the principles of the invention and fall within the spirit and scope thereof.
What is claimed is: 1. A method for operating a router, comprising: establishing a first VLAN from a port connected to a
shared network to a plurality of user ports, the first VLAN receiving packets from the shared network and transferring them to a designated user port, the first VLAN rejecting packets from the user ports;
establishing a second VLAN from the plurality of user ports, the second VLAN receiving packets from the user ports and transferring them to the port connected to the shared network, the second VLAN preventing transfer of packets from one of the user ports to other user ports, and the second VLAN also rejecting packets from the shared network, in order to separate packet traffic of different users.
2. The method as in claim 1, further comprising: dividing the plurality of user ports into groups of user
ports, each group of user ports assigned to a single designated user; and
establishing a third VLAN from a first group of user ports, the third VLAN receiving packets from the first group of user ports and transferring them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN preventing transfer of packets to other user ports belonging to a group different from the first group of user ports, the third VLAN rejecting packets from the shared network.
3. A router, comprising: means for establishing a first VLAN from a port con
nected to a shared network to a plurality of user ports, the first VLAN receiving packets from the shared network and transferring them to a designated user port, the first VLAN rejecting packets from the user ports;
means for establishing a second VLAN from the plurality of user ports, the second VLAN receiving packets from the user ports and transferring them to the port connected to the shared network, the second VLAN preventing transfer packets from one of the user ports to other user ports, and the second VLAN also rejecting packets from the shared network, in order to separate packet traffic of different users.
4. The router as in claim 3, further comprising: means for dividing the plurality of user ports into groups
of user ports, each group of user ports assigned to a single designated user; and
means for establishing a third VLAN from a first group of user ports, the third VLAN receiving packets from the first group of user ports and transferring them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN preventing transfer of packets to other user ports belonging to a group different from the first group of user ports, the third VLAN rejecting packets from the shared network.
5. A router, comprising: a port connected to a shared network; a plurality of user ports; a first VLAN from the port connected to the shared
network to the plurality of user ports, the first VLAN to receive packets from the shared network and transfer-
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page15 of 22
US 7,200,145 Bl 11
ring them to a designated user port, the first VLAN to reject packets from the user ports;
a second VLAN from the plurality of user ports, the second VLAN to receive packets from the user ports and transferring them to the port connected to the shared network, the second VLAN to prevent transfer of packets from one of the user ports to other user ports, and the second VLAN also to reject packets from the shared network, in order to separate packet traffic of
10 different users.
6. The router as in claim 5, further comprising:
a third VLAN from a first group user ports of a plurality of groups of user ports, each group of user ports assigned to a single designated user, the third VLAN to 15
receive packets from the user ports and transfer them to the port connected to the shared network, the third VLAN able to transfer packets among user ports belonging to the first group of user ports, the third VLAN to prevent transfer of packets to other user ports 20
belonging to a group different from the first group of user ports, the third VLAN to reject packets from the shared network.
7. A router, comprising:
one or more promiscuous ports;
one or more isolated ports;
one or more community ports;
25
a primary VLAN, the primary VLAN to receive packets from outside of the router through the one or more 30
promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports and to transfer the packets to the one or more community ports, the primary VLAN to reject packets from the one or more isolated ports and to reject packets from the 35
one or more community ports;
an isolated VLAN, the isolated VLAN to receive packets from outside of the router through an isolated port of the one or more isolated ports and to transfer the packets to the one or more promiscuous ports, the 40
isolated VLAN to prevent transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN to prevent transfer of the packets from the isolated port to the one or more community ports, and the isolated VLAN to 45
reject packets from the one or more promiscuous ports; and
a community VLAN, the community VLAN to receive packets from outside of the router at a community port of the one or more community ports and to transfer the 50
packets to the one or more promiscuous ports and to transfer the packets to any other community ports, the community VLAN to prevent transfer of packets to the one or more isolated ports, the community VLAN to reject packets from the one or more promiscuous ports. 55
8. The router as in claim 7, further comprising:
a promiscuous port of the one or more promiscuous ports connected to a shared network, the shared network carrying packet traffic of a first user and packet traffic 60 of a second user; and
a first isolated port of the one or more isolated ports connected to a local area network (LAN) assigned to the first user, and a second isolated port of the one or more isolated ports connected to a LAN assigned to the 65
second user, in order to separate The packet traffic of the first user and the second user.
12 9. The router as in claim 7, further comprising: a plurality of sets of community ports; and a community VLAN for each set of community ports, the
community VLAN for a particular set of community ports to prevent transfer packets to another set of community ports.
10. The router as in claim 7, further comprising: a promiscuous port of the one or more promiscuous ports
connected to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and
a first set of community ports of the one or more community ports connected to a local area network (LAN) assigned to the first user, and a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.
11. A router, comprising: one or more promiscuous ports; one or more isolated ports; a primary VLAN, the primary VLAN to receive packets
from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports; and
an isolated VLAN, the isolated VLAN to receive packets from outside of the router through an isolated port of the one or more isolated ports and to transfer the packets to the one or more promiscuous ports, the isolated VLAN to prevent transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN to reject packets from the one or more promiscuous ports.
12. A router, comprising: one or more promiscuous ports; one or more community ports; a primary VLAN, the primary VLAN to receive packets
from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports; and
a community VLAN, the community VLAN to receive packets from outside the router at a community port of the one or more community ports and to transfer the packets to the one or more promiscuous ports and to transfer the packets to any other community ports of the one or more community ports, the community VLAN to reject packets from the one or more promiscuous ports.
13. A router, comprising: one or more promiscuous ports; one or more other ports; a primary VLAN, the primary VLAN to receive packets
from outside of the router through the one or more promiscuous ports and to transfer the packets to a selected one of the one or more other ports, the primary VLAN to reject packets from the one or more other ports; and
a second VLAN, the second VLAN to receive packets from outside the router at the one or more other ports and to transfer the packets to the one or more promiscuous ports, the second VLAN to reject packets from the one or more promiscuous ports.
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page16 of 22
US 7,200,145 Bl 13
14. The router as in claim 13, further comprising: said second VLAN to transfer the packets to the one or
more other ports. 15. A router, comprising: one or more promiscuous ports; one or more other ports; a primary VLAN, the primary VLAN to receive packets
from outside of the router through the one or more promiscuous ports and to transfer the packets to the one
10 or more other ports, the primary VLAN to reject packets from the one or more other ports; and
a second VLAN, the second VLAN to receive packets from outside the router at a one or more other ports and to transfer the packets to a promiscuous port of the one
15 or more promiscuous ports, the second VLAN to reject packets from the one or more promiscuous ports.
16. The router as in claim 15, further comprising: the one or more other ports are one or more an isolated
ports. 20
17. The router as in claim 15, further comprising: the one or more other ports are one or more community
ports. 18. A method for using a router, comprising: establishing one or more promiscuous ports; 25
establishing one or more isolated ports; establishing one or more community ports; receiving a-packets by a primary VLAN, the primary
VLAN receiving the packets from outside of the router 30
through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more isolated ports and reject-
35 ing packets from the one or more community ports;
receiving packets by an isolated VLAN, the isolated VLAN receiving the packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more
40 promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN preventing transfer of the packets from the isolated port to the one or more community ports,
45 and the isolated VLAN rejecting packets from the one or more promiscuous ports; and
receiving packets by a community VLAN, the community VLAN receiving packets from outside of the router at a community port of the one or more community ports 50 and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports, the community VLAN preventing transfer of packets to the one or more isolated ports, and the community VLAN rejecting packets from the 55 one or more promiscuous ports.
19. The method of claim 18, further comprising: connecting a promiscuous port of the one or more pro
miscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic 60
of a second user; and connecting a first isolated port of the one or more isolated
ports to a local area network (LAN) assigned to the first user, and connecting a second isolated port of the one or more isolated ports to a LAN assigned to the second 65
user, in order to separate the packet traffic of the first user and the second user.
14 20. The method of claim 18, further comprising: establishing a plurality of sets of community ports; and connecting a community VLAN for each set of commu-
nity ports, the community VLAN for a particular set of community ports preventing transfer of packets to another set of community ports.
21. The method of claim 18, further comprising: connecting a promiscuous port of the one or more pro
miscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and
connecting a first set of community ports of the one or more community ports to a local area network (LAN) assigned to the first user, and connecting a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.
22. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more isolated ports; receiving packets by a primary VLAN, the primary
VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports, the primary VLAN rejecting receive packets from the one or more isolated ports; and
receiving packets by an isolated VLAN, the isolated VLAN receiving packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports.
23. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more community ports; receiving packets by a primary VLAN, the primary
VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more community ports; and
receiving packets by a community VLAN, the community VLAN receiving packets from outside the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets from the one or more promiscuous ports.
24. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more other ports; receiving packets by a primary VLAN, the primary
VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more other ports, the primary VLAN rejecting packets from the one or more other ports; and
receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at the one or more other ports and transferring the packets to the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page17 of 22
US 7,200,145 Bl 15
25. The method as in claim 24, further comprising: transferring the packets by the second VLAN to the one
or more other ports. 26. A method for using a router, comprising: establishing one or more promiscuous ports; establishing one or more other ports; receiving packets by a primary VLAN, the primary
VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more other ports, the 10
primary VLAN rejecting packets from the one or more other ports; and
receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at a one or more other ports and transferring the packets to a 15
promiscuous port of the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.
27. The method as in claim 26, further comprising: establishing the one or more other ports as one or more 20
isolated ports. 28. The method as in claim 26, further comprising: establishing the one or more other ports as one or more
community ports. 29. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more isolated ports;
25
means for establishing one or more community ports; means for receiving packets by a primary VLAN, the 30
primary VLAN receiving the packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports and transferring the packets to the one or more community ports, the primary VLAN 35 rejecting packets from the one or more isolated ports and rejecting packets from the one or more community ports;
means for receiving packets by an isolated VLAN, the isolated VLAN receiving the packets from outside of 40 the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the 45 isolated VLAN preventing transfer of the packets from the isolated port to the one or more community ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports; and
means for receiving packets by a community VLAN, the 50
community VLAN receiving packets from outside of the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports, the community VLAN 55
preventing transfer of packets to the one or more isolated ports, and the community VLAN rejecting packets from the one or more promiscuous ports.
30. The router as in claim 29 further comprising: means for connecting a promiscuous port of the one or 60
more promiscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and
means for connecting a first isolated port of the one or more isolated ports to a local area network (LAN) 65
assigned to the first user, and connecting a second isolated port of the one or more isolated ports to a LAN
16 assigned to the second user, in order to separate the packet traffic of the first user and the second user.
31. The router as in claim 29 further comprising: means for establishing a plurality of sets of community
ports; and means for connecting a community VLAN for each set of
community ports, the community VLAN for a particular set of community ports preventing transfer of packets to another set of community ports.
32. The router as in claim 29 further comprising: means for connecting a promiscuous port of the one or
more promiscuous ports to a shared network, the shared network carrying packet traffic of a first user and packet traffic of a second user; and
means for connecting a first set of community ports of the one or more community ports to a local area network (LAN) assigned to the first user, and connecting a second set of community ports of the one or more community ports connected to a LAN assigned to the second user, in order to separate the packet traffic of the first user and the second user.
33. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more isolated ports; means for receiving packets by a primary VLAN, the
primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more isolated ports, the primary VLAN rejecting receive packets from the one or more isolated ports; and
means for receiving packets by an isolated VLAN, the isolated VLAN receiving packets from outside of the router through an isolated port of the one or more isolated ports and transferring the packets to the one or more promiscuous ports, the isolated VLAN preventing transfer of the packets from the isolated port to another isolated port of the one or more isolated ports, and the isolated VLAN rejecting packets from the one or more promiscuous ports.
34. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more community ports; means for receiving packets by a primary VLAN, the
primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to the one or more community ports, the primary VLAN rejecting packets from the one or more community ports; and
means for receiving packets by a community VLAN, the community VLAN receiving packets from outside the router at a community port of the one or more community ports and transferring the packets to the one or more promiscuous ports and transferring the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets from the one or more promiscuous ports.
35. A router, comprising: means for establishing one or more promiscuous ports; means for establishing one or more other ports; means for receiving packets by a primary VLAN, the
primary VLAN receiving packets from outside of the router through the one or more promiscuous ports and transferring the packets to a selected one of the one or more other ports, the primary VLAN rejecting packets from the one or more other ports; and
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page18 of 22
US 7,200,145 Bl 17
means for receiving packets by a second VLAN, the second VLAN receiving packets from outside the router at the one or more other ports and transferring the packets to the one or more promiscuous ports, the second VLAN rejecting packets from the one or more promiscuous ports.
36. The router as in claim 35, further comprising: means for transferring the packets by the second VLAN
to the one or more other ports. 37. The router as in claim 35, further comprising: 10
means for establishing the one or more other ports as one or more isolated ports.
38. The router as in claim 35, further comprising: means for establishing the one or more other ports one or
more a community ports. 15
39. A router to separate packet traffic traveling on a shared network, comprising:
a primary VLAN within the router, the primary VLAN to receive packets from the shared network through a promiscuous port and to transfer the packets to a 20
selected one of a one or more isolated ports, the primary VLAN to reject packets from the one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user 25
connected to a second isolated port of the one or more isolated ports; and
a first isolated VLAN within the router, the first isolated VLAN to receive packets through an isolated port connected to the first LAN and to transfer the packets 30
to the promiscuous port, the first isolated VLAN to prevent transfer of the packets from the isolated port connected to the first LAN to another isolated port, and the isolated VLAN rejecting packets from the one or more promiscuous ports; and 35
a second isolated VLAN within the router, the second isolated VLAN to receive packets through an isolated port connected to the second LAN and to transfer the packets to the promiscuous port, the second isolated VLAN to prevent transfer of the packets from the 40
isolated port connected to the second LAN to another isolated port, and the second isolated VLAN to reject packets from the promiscuous port.
40. A method in a router for separating packet traffic traveling on a shared network, comprising: 45
receiving packets by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more isolated ports, the primary VLAN rejecting packets from the 50
one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user connected to a second isolated port of the one or more isolated ports; and 55
receiving packets by a first isolated VLAN within the router, the first isolated VLAN receiving packets through an isolated port connected to the first LAN and transferring the packets to the promiscuous port, the first isolated VLAN preventing transfer of the packets 60
from the isolated port connected to the first LAN to another isolated port, and the first isolated VLAN rejecting packets from the one or more promiscuous ports; and
receiving packets by a second isolated VLAN within the 65
router, the second isolated VLAN receiving packets through an isolated port connected to the second LAN
18 and transferring the packets to the promiscuous port, the second isolated VLAN preventing-transfer of the packets from the isolated port connected to the second LAN to another isolated port, and the second isolated VLAN rejecting-packets from the promiscuous port.
41. A router to separate packet traffic traveling on a shared network, comprising:
means for receiving packets by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more isolated ports, the primary VLAN rejecting packets from the one or more isolated ports, a first local area network (LAN) of a first user connected to a first isolated port of the one or more isolated ports, and a second LAN of a second user connected to a second isolated port of the one or more isolated ports; and
means for receiving packets by a first isolated VLAN within the router, the first isolated VLAN receiving packets through an isolated port connected to the first LAN and transferring the packets to the promiscuous port, the first isolated VLAN preventing transfer of the packets from the isolated port connected to the first LAN to another isolated port, and the first isolated VLAN rejecting packets from the one or more promiscuous ports; and
means for receiving packets by a second isolated VLAN within the router, the second isolated VLAN receiving packets through an isolated port connected to the second LAN and transferring the packets to the promiscuous port, the second isolated VLAN preventingtransfer of the packets from the isolated port connected to the second LAN to another isolated port, and the second isolated VLAN rejecting-packets from the promiscuous port.
42. A router to separate packet traffic traveling on a shared network, comprising:
a primary VLAN within the router, the primary VLAN to receive packets from the shared network through a promiscuous port and to transfer the packets to a selected one of a one or more community ports, the primary VLAN to reject packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports of the one or more community ports, and a second group of LANs of a second user connected to a second group of community ports of the one or more community ports;
a first community VLAN within the router, the first community VLAN to receive packets through the first group of community ports connected to the first group of LAN s and to transfer the packets to the promiscuous port, the first community VLAN prevent transfer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN to reject packets from the one or more promiscuous ports; and
a second community VLAN within the router, the second community VLAN to receive packets through the second group of community ports connected to the second group of LANs and to transfer the packets to the promiscuous port, the second community VLAN to prevent transfer of the packets to the first group of community ports, and the second community VLAN to reject packets from the promiscuous port.
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page19 of 22
US 7,200,145 Bl 19
43. A method for separating packet traffic traveling on a shared network, comprising:
receiving packets from the shared network by a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a selected one of a one or more community ports, the primary VLAN rejecting packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports 10
of the one or more community ports, and a second group of LAN s of a second user connected to a second group of community ports of the one or more community ports;
receiving packets from a first group of community ports 15
by a first community VLAN within the router, the first community VLAN receiving packets through the first group of community ports connected to the first group of LANs and transferring the packets to the promiscuous port, the first community VLAN preventing trans- 20
fer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN rejecting packets from the one or more promiscuous ports; and
receiving packets from a second group of community 25
ports by a second community VLAN within the router, the second community VLAN receiving packets through the second group of community ports connected to the second group of LANs and transferring the packets to the promiscuous port, the second com- 30
munity VLAN preventing transfer of the packets to the first group of community ports, and the second community VLAN rejecting packets from the promiscuous port.
44. A router to separate packet traffic traveling on a shared 35
network, comprising: means for receiving packets from the shared network by
a primary VLAN within the router, the primary VLAN receiving packets from the shared network through a promiscuous port and transferring the packets to a 40
selected one of a one or more community ports, the primary VLAN rejecting packets from the one or more community ports, a first group of local area network (LAN) of a first user connected to a first group of community ports of the one or more community ports, 45
and a second group of LANs of a second user connected to a second group of community ports of the one or more community ports;
means for receiving packets from a first group of community ports by a first community VLAN within the so router, the first community VLAN receiving packets through the first group of community ports connected to the first group of LAN s and transferring the packets
20 to the promiscuous port, the first community VLAN preventing transfer of the packets from the first group of community ports to the second group of community ports, and the first community VLAN rejecting packets from the one or more promiscuous ports; and
means for receiving packets from a second group of community ports by a second community VLAN within the router, the second community VLAN receiving packets through the second group of community ports connected to the second group of LANs and transferring the packets to the promiscuous port, the second community VLAN preventing transfer of the packets to the first group of community ports, and the second community VLAN rejecting packets from the promiscuous port.
45. A computer readable medium containing executable program instructions for operating a router, the executable program instructions comprising program instructions configured to:
establish a first VLAN from a port connected to a shared network to a plurality of user ports, the first VLAN to receive packets from the shared network and to transfer them to one or more of the user ports, the first VLAN to reject any packets received from the user ports;
establish a second VLAN from the plurality of user ports, the second VLAN to receive packets from the user ports and to transfer them to the port connected to the shared network, the second VLAN to prevent transfer of packets from one of the user ports to other user ports, and the second VLAN also to reject packets from the shared network, to thereby separate packet traffic of different users.
46. A computer readable medium containing executable program instructions for operating a router, the executable program instructions comprising program instructions configured to:
establish a primary VLAN, the primary VLAN to receive packets from outside of the router through the one or more promiscuous ports and to transfer the packets to one or more community ports, the primary VLAN to reject packets received from the one or more community ports; and
establish a community VLAN, the community VLAN to receive packets from outside the router on a community port of the one or more community ports and to transfer the packets to the one or more promiscuous ports and to transfer the packets to any other community ports of the one or more community ports, the community VLAN rejecting packets received from the one or more promiscuous ports.
* * * * *
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page20 of 22
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
PATENT NO. : 7,200,145 B1 Page 1 of 1 APPLICATIONNO. : 10/840212 DATED : Apri13, 2007 INVENTOR(S) : Thomas Edsall et al.
It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:
Title Page, Item [63], Related U.S. Application Data: should be added and read-- Continuation of Application No. 09/575,774, filed on May 22, 2000, now Pat. No. 6,741,592. --
Signed and Sealed this
Eighteenth Day of March, 2008
JONW.DUDAS Director of the United States Patent and Trademark Office
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page21 of 22
PATENT NO. APPLICATION NO. DATED INVENTOR(S)
UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION
: 7,200,145 B1 : 10/840212 : Apri13, 2007 : Thomas Edsall et al.
Page 1 of 1
It is certified that error appears in the above-identified patent and that said Letters Patent is hereby corrected as shown below:
At Column 1, line 2, after the heading "PRIVATE VLANS" text should be added and read-- This Application is a continuation of Application Serial No. 09/575,774, filed on May 22, 2000, now issued
as Patent No. 6,741,592. --
Signed and Sealed this Twenty-eighth Day of December, 2010
f)w:J J:•k~ David J. Kappos
Director of the United States Patent and Trademark Office
Case3:14-cv-05343 Document1-10 Filed12/05/14 Page22 of 22
Exhibit 11
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page1 of 17
c12) United States Patent Portolani et al.
(54) SPANNING TREE LOOP GUARD
(75) Inventors: Maurizio Portolani, Milpitas, CA (US); Shyamasundar S. Kaluve, Santa Clara, CA (US); Marco E. Foschiano, San Jose, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 67 days.
This patent is subject to a terminal disclaimer.
(21) Appl. No.: 11/451,888
(22) Filed: Jun.12,2006
(65) Prior Publication Data
US 2006/0233186 AI Oct. 19, 2006
Related U.S. Application Data
(63) Continuation of application No. 10/020,667, filed on Dec. 7, 2001, now Pat. No. 7,061,875.
(51) Int. Cl. H04L 12156 (2006.01) H04L 12128 (2006.01) H04J 3124 (2006.01) G06F 15116 (2006.01)
(52) U.S. Cl. ....................... 370/256; 370/402; 370/408; 709/238
(58) Field of Classification Search . ... ... ... ... .. .. 3 70/238, 370/252,254-256,389,392,397,419,428,
370/429; 709/220-226, 238-242 See application file for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
5,790,808 A 8/1998 Seaman
111111 1111111111111111111111111111111111111111111111111111111111111 US007460492B2
(10) Patent No.: US 7,460,492 B2 (45) Date of Patent: *Dec. 2, 2008
5,959,968 A 9/1999 Chin eta!.
6,032,194 A 212000 Gai et al.
6,188,694 B1* 2/2001 Fine et al. ................... 370/402
6,202,114 B1 3/2001 Dutt et al.
6,219,739 B1* 4/2001 Dutt et al. ................... 710/311
6,304,575 B1 10/2001 Carroll eta!.
6,628,624 B1 9/2003 Mahajan et a!.
(Continued)
OTHER PUBLICATIONS
Perlman, Radia, "Interconnections Second Edition: Bridges, Routers, Switches, and Internetworking Protocols," pp. 54-75, Addison Wesley Longman, Inc., 2000.
Primary Examiner-Hong Sol Cho (74) Attorney, Agent, or Firm---Cesari and McKenna LLP
(57) ABSTRACT
A system and method are provided to prevent the formation of loops in a network. The network device includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol engine. The spanning tree protocol engine, in one embodiment, implements the Rapid Spanning Tree Protocol (RSTP) to transitions the ports among a plurality port states, including a discarding state, a learning state and a forwarding state. The network device further includes a loop guard engine that is in a communicating relationship with the spanning tree protocol engine and the ports. The loop guard engine monitors the receipt of bridge protocol data units (BPDUs) by the ports. If a given port stops receiving BPDUs, the loop guard engine prevents the spanning tree protocol engine from transitioning the given port to the forwarding state. Instead, the loop guard engine causes the port to transition to loop inconsistent state.
24 Claims, 6 Drawing Sheets
307
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page2 of 17
US 7,460,492 B2 Page 2
U.S. PATENT DOCUMENTS 6,891,808 B2 * 6,898,189 B1 6,934,263 B1 * 6,985,449 B2
6,697,339 B1 * 6,765,877 B1 * 6,801,506 B1 6,813,250 B1 6,831,898 B1 6,882,630 B1 *
2/2004 Jain ........................... 370/256 7/2004 Foschiano eta!. ........... 370/250
10/2004 Dey 1112004 Fine eta!. 12/2004 Edsall eta!. 4/2005 Seaman ...................... 370/256
200110025318 A1 * 2003/0016624 A1 *
* cited by examiner
5/2005 Ishii ........................... 370/256 5/2005 Di Benedetto et a!. 8/2005 Seaman ...................... 370/256 112006 Higashiyama 9/2001 Higashiyama .............. 709/238 112003 Bare .......................... 370/217
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page3 of 17
U.S. Patent Dec. 2, 2008 Sheet 1 of 6 US 7,460,492 B2
~100
110 111
119
113
105
123 109
FIG. 1
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page4 of 17
112
' 1204
en 1-
202a 0 .. w """':)
CD 0 z
202b 0 .. -1-D... w 0 w
202c 0::: ·o
z <( _...._
z 202d 0
(/) (f)
~
202e X 206-w ~
~ LL
208'\ PROTOCOL ENTITY
SPANNING TREE PROTOCOL ENGINE
PORT ROLE PORT SELECTION TRANSITION BPDU
MESSAGE STATE STATE
MACHINE MACHINE GENERATOR
212 1 2141 216 1
I
218 ~ LOOP GUARD ENGINE
FORWARDING ENGINE ~210
MEMORY
,.., -.... ~ _....,.
2201
~222 FILTERING DATABASE
"'-.. """""
FIG. 2
~ 00 • ~ ~ ~ ~ = ~
c ('D
~ N ~
N 0 0 QO
rFJ
=('D ('D ..... N 0 ..... 0\
d rJl -....l ~ 0'1 = ~ \C N
= N
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page5 of 17
U.S. Patent Dec. 2, 2008 Sheet 3 of 6 US 7,460,492 B2
GIVEN PORT OF THE SWITCH STOPS RECEIVING BPDU 302 MESSAGES
SPANNING TREE PROTOCOL ENTITY DISCARDS BPDU 304 INFORMATION STORED FOR THE GIVEN PORT
YES
NO
TRANSITION PORT IN ACCORDANCE WITH 324 CONVENTIONAL SPANNING TREE PROTOCOL OPERATION
307
TRANSITION GIVEN PORT 308
TO THE ~~LOOP INCONSISTENr STATE
BLOCK GIVEN PORT FROM SENDING OR RECEIVING 310 NETWORK MESSAGES, OTHER THAN BPDU MESSAGES
RECALCULATE PORT ROLES AND STATES USING NEXT BEST BPDU INFORMATION, AND SUPPRESS THE SENDING 312
OF BPDU MESSAGES OUT OF THE GIVEN PORT
TO FIG. 38
FIG. 3A
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page6 of 17
U.S. Patent
315
NO
Dec. 2, 2008 Sheet 4 of 6 US 7,460,492 B2
FROM FIG. 3A
RELEASE THE GIVEN PORT 318 FROM THE LOOP INCONSISTENT STATE
TRANSITION PORT IN ACCORDANCE WITH CONVENTIONAL SPANNING 320
TREE PROTOCOL OPERATION
FIG. 38
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page7 of 17
U.S. Patent Dec. 2, 2008 Sheet 5 of 6 US 7,460,492 B2
,----- 400
FIG. 4
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page8 of 17
U.S. Patent Dec. 2, 2008 Sheet 6 of 6 US 7,460,492 B2
/'500
EVENT TABLE
EVENT DESCRIPTION
E1 PORT HAVING LOOP GUARD ENABLED STOPS RECEIVING BPDU MESSAGES
E2 PORT RECEIVES A BPDU MESSAGE
E3 FORWARD DELAY TIMER EXPIRES FOR A DESIGNATED PORT OR THE ROOT PORT WHILE IN THE DISCARDING STATE
E4 FORWARD DELAY TIME EXPIRES FOR PORT IN LEARNING STATE WITHOUT THE RECEIPT OF "BETTER" BPDU INFORMATION
E5 PORT IN LEARNING OR FORWARDING STATE RECEIVES "BETTER" BPDU INFORMATION
E6 BLOCKED PORT BECOMES A DESIGNATED PORT OR THE ROOT PORT, AND CAN RAPIDLY TRANSITION TO FORWARDING
FIG. 5
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page9 of 17
US 7,460,492 B2 1
SPANNING TREE LOOP GUARD
RELATED APPLICATION
This Application for United States patent is a continuation ofU.S. patent application Ser. No. 10/020,667, filed on Dec. 7, 2001, entitled Spanning Tree Loop Guard now issued as U.S. Pat. No. 7,061,875.
BACKGROUND OF THE INVENTION
1. Field of the Invention The present invention relates generally to computer net
works, and more specifically, to a method and apparatus for preventing the formation ofloops.
2. Background Information A computer network typically comprises a plurality of
interconnected entities. An entity may consist of any device, such as a computer or end station, that "sources" (i.e., transmits) or "sinks" (i.e., receives) data frames. A common type of computer network is a local area network ("LAN") which typically refers to a privately owned network within a single building or campus. LANs typically employ a data communication protocol (LAN standard), such as Ethernet, FDDI or token ring, that defines the functions performed by the data link and physical layers of a communications architecture (i.e., a protocol stack). In many instances, several LANs may be interconnected by point-to-point links, microwave transceivers, satellite hook-ups, etc. to form a wide area network ("WAN") or intranet that may span an entire country or continent.
One or more intermediate network devices are often used to couple LAN s together and allow the corresponding entities
2 loops may cause a proliferation of data frames so large that the network becomes overwhelmed.
Spanning Tree Protocol
To avoid the formation of! oops, most bridges and switches execute a spanning tree protocol or algorithm which allows them to calculate an active network topology that is loop-free (i.e., a tree) and yet connects every pair ofLANs within the network (i.e., the tree is spanning). The Institute of Electrical
10 and Electronics Engineers (IEEE) has promulgated a standard (the 802.1D standard) that defines a spanning tree protocol to be executed by 802.1D compatible devices. In general, by executing the 802.1D spanning tree protocol, bridges elect a single bridge within the bridged network to be the
15 "root" bridge. The 802.1D standard takes advantage of the fact that each bridge has a unique numerical identifier (bridge ID) by specifying that the root is the bridge with the lowest bridge ID. In addition, for each LAN coupled to more than one bridge, only one (the "designated bridge") is elected to
20 forward frames to and from the respective LAN. The designated bridge is typically the one closest to the root. Each bridge also selects one port (its "root port") which gives the lowest cost path to the root. The root ports and designated bridge ports are selected for inclusion in the active topology
25 and are placed in a forwarding state so that data frames may be forwarded to and from these ports and thus onto the corresponding paths or links of the network. Ports not included within the active topology are placed in a blocking state. When a port is in the blocking state, data frames will not be
30 forwarded to or received from the port. A network administrator may also exclude a port from the spanning tree by placing it in a disabled state.
to exchange information. For example, a bridge may be used 35 to provide a "bridging" function between two or more LAN s. Alternatively, a switch may be utilized to provide a "switching" function for transferring information between a plurality ofLANs or end stations. Typically, the bridge or switch is a computer and includes a plurality of ports that couple the 40 device to the LANs or end stations. The switching function includes receiving data from a sending entity at a source port and transferring that data to at least one destination port for forwarding to the receiving entity.
To obtain the information necessary to rnn the spanning tree protocol, bridges exchanges special messages called configuration bridge protocol data nnit (BPDU) messages. More specifically, upon start-up, each bridge initially assumes that it is the root and transmits BPDU messages accordingly. Upon receipt of a BPDU message from a neighboring device, its contents are examined and compared with similar information (e.g., assumed root and lowest root path cost) stored by the receiving bridge. If the information from the received BPDU is "better" than the stored information, the bridge adopts the better information and uses it in the BPDU s that it sends (adding the cost associated with the receiving port to the root path cost) from its ports, other than the port on which the "better" information was received. Although BPDU mes-
Switches and bridges typically learn which destination port 45
to use in order to reach a particular entity by noting on which source port the last message originating from that entity was received. This information is then stored by the bridge in a block of memory referred to as a filtering database. Thereafter, when a message addressed to a given entity is received on 50
a source port, the bridge looks up the entity in its filtering database and identifies the appropriate destination port to reach that entity. If no destination port is identified in the filtering database, the bridge floods the message out all ports, except the port on which the message was received. Messages 55
addressed to broadcast or multicast addresses are also flooded.
sages are not forwarded by bridges, the identifier of the root is eventually propagated to and adopted by all bridges as described above, allowing them to select their root port and any designated port(s).
In order to adapt the active topology to changes and failures, the root periodically (e.g., every hello time) transmits BPDU messages. The default hello time is two seconds. In response to receiving BPDUs on their root ports, bridges transmit their own BPDU s from their designated ports, if any. Thus, every two seconds BPDU s are propagated throughout the bridged network, confirming the active topology. If a bridge stops receiving BPDU messages on a given port (indieating a possible link or device failure), it will continue to
Additionally, most computer networks are either partially or fully meshed. That is, they include redundant communications paths so that a failure of any given link or device does not isolate any portion of the network. The existence of redundant links, however, may cause the formation of circuitous paths or "loops" within the network. Loops are highly undesirable because data frames may traverse the loops indefinitely. Furthermore, because switches and bridges replicate (i.e., flood) frames whose destination port is unknown or which are directed to broadcast or multicast addresses, the existence of
60 increment a respective message age value until it reaches a maximum age (max age) threshold. The bridge will then age out, i.e., discard, the stored BPDU information and proceed to re-calculate the root, root path cost and root port by transmitting BPDU messages utilizing the next best information it
65 has. Themaximumagevalueused within the bridged network is typically set by the root, which enters the appropriate value in its BPDU messages. Normally, each bridge replaces its
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page10 of 17
US 7,460,492 B2 3
stored BPDU information every hello time, thereby preventing it from being discarded and maintaining the current active topology.
When BPDU information is updated and/or timed-out and the active topology is re-calculated, ports may transition from the blocking state to the forwarding state and vice versa. That is, as a result of new BPDU information, a previously blocked port may learn that it should be in the forwarding state (e.g., it is now the root port or a designated port). Rather than transition directly from the blocking state to the forwarding 10
state, the 802.1 D standard calls for ports to transition through two intermediate states: a listening state and a learning state. In the listening state, a port waits for information indicating that it should return to the blocking state. If, by the end of a preset time, no such information is received, the port transi- 15
tions to the learning state. In the learning state, a port still blocks the receiving and forwarding of frames, but received frames are examined and the corresponding location information is stored in the bridge's filtering database. At the end of a second preset time, the port transitions from the learning 20
state to the forwarding state, thereby allowing frames to be forwarded to and from the port. The time spent in each of the listening and the learning states is referred to as the forwarding delay.
Although the spanning tree protocol provided in the 25
802.1D standard is able to maintain a loop-free topology despite network changes and failures, re-calculation of the active topology can be a time consuming and processor intensive task. For example, recalculation of the spanning tree following an intermediate device crash or failure can take 30
approximately thirty seconds. During this time, message delivery is often delayed as ports transition between states. Such delays can have serious consequences on timesensitive traffic flows, such as voice or video traffic streams.
4 the ports of the downstream bridge are consistent with this port being transitioned to forwarding. The RSTP provides an explicit handshake to be used by neighboring bridges to confirm that a new designated port can rapidly transition to the forwarding state.
Like the STP described in the 802.1D specification standard, bridges running the RSTP also exchange BPDU messages in order to determine which roles to assign to the bridge's ports. The BPDU messages are also utilized in the handshake employed to rapidly transition designated ports to the forwarding state. RSTP also uses timers, including a received information while (rcvdinfo While) timer, which is similar to STP's max age timer. The rcvdinfo While timer is a count down (to zero) timer, while the max age timer is a count up timer.
Loops Undetectable by Spanning Tree Protocols In some cases, a single, duplex link coupling two neigh-
boring bridges (which are also indirectly coupled through other bridges or devices) may physically comprise two simplex, i.e., unidirectional, transmission lines, such as two fiber optic lines, operating in opposite directions. Certain failures associated with such lines can result in the formation ofloops that are undetectable by the STP. For example, suppose two bridges, designated A and B, are connected by a single trunk link formed from two unidirectional transmission lines, and that the respective port at Bridge B is assigned the designated port role, while the peer port at Bridge A is assigned the alternate port role. In this case, the port at Bridge B is placed in the forwarding state and the port at bridge A is placed in the discarding state. As long as the port at Bridge A continues to receive "superior" BPDU messages from Bridge B, it will remain in the blocking state. Suppose, however, that the trunk link becomes unidirectional. That is, bridge B continues to send BPDU messages to Bridge A, but these BPDU messages
Rapid Spanning Tree Protocol 35 are never received, and yet the trunk line is not considered to be "down". Accordingly, the BPDU information stored for the port at Bridge A eventually ages out and the S TP running at Bridge A transitions the port to the forwarding state.
Recently, the IEEE promulgated a new standard (the 802.1 w standard) that defines a rapid spanning tree protocol (RSTP) to be executed by otherwise 802.1D compatible devices. The RSTP similarly selects one bridge of a bridged network to be the root bridge and defines an active topology 40
that provides complete connectivity among the LANs while severing any loops. Each individual port of each bridge is assigned a port role according to whether the port is to be part of the active topology. The port roles defined by the 802.1w standard include Root, Designated, Alternate and Backup. 45
The bridge port offering the best, e.g., lowest cost, path to the root is assigned the Root Port Role. Each bridge port offering an alternative, e.g., higher cost, path to the root is assigned the Alternate Port Role. Each bridge port providing the lowest cost path from a given LAN is assigned the Designated Port 50
Role, while all other ports coupled to the given LAN in loop-back fashion are assigned the Backup Port Role.
Those ports that have been assigned the Root Port and Designated Port Roles are placed in the forwarding state, while ports assigned the Alternate and Backup Roles are 55
placed in a discarding or blocking state. A port assigned the Root Port Role can be rapidly transitioned to the forwarding state provided that all of the ports assigned the Alternate Port Role are placed in the discarding or blocking state. Similarly, if a failure occurs on the port currently assigned the Root Port 60
Role, a port assigned the Alternate Port Role can be reassigned to the Root Port Role and rapidly transitioned to the forwarding state, provided that the previous root port has been transitioned to the discarding or blocking state. A port assigned the Designated Port Role or a Backup Port Role that 65
is to be reassigned to the Designated Port Role can be rapidly transitioned to the forwarding state, provided that the roles of
Because Bridge B is unaware of the link failure, the port at Bridge B remains in the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state a loop is created. As described above, the creation of such a loop causes network messages to be replicated, wasting substantial network bandwidth and potentially causing a network outage.
A loop may also be created as a result of an error or failure in the operation of the STP at Bridge B, such as a software error. Specifically, the STP running at Bridge B may determine that the port of Bridge B that is coupled to Bridge A should be assigned the Designated Port Role and be transitioned to the forwarding state. Yet, the STP running at Bridge B may fail for some reason to have BPDU messages sent from the port. In this case, the STP running at Bridge A concludes that its port should now be assigned the designated port role and that it should be transitioned to the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state, a loop is created. Certain hardware failures can simi-larly result in the creation of loops. For example, the STP running at Bridge B may generate BPDU messages for transmission from the port coupled to Bride A, but those BPDU messages may never get sent due to a hardware problem at Bridge B.
In summary, unidirectional failures resulting in the forma-tion ofloop s may occur as a result of malfunctioning or faulty network interface cards (NICs) and/or transceivers, a switch's central processing nnit (CPU) being too busy with other processes to send BPDU messages for a relatively long time, a software bug in the STP running at the switch, or
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page11 of 17
US 7,460,492 B2 5
congestion algorithms that end up dropping BPDU messages. In addition, if a link up/down detection and/or autonegotiation protocol is disabled, e.g., by network administrator action, unidirectional failures may go undetected, resulting in loops. Accordingly, a need exists to prevent the formation of loops that are undetectable by the STP.
SUMMARY OF THE INVENTION
6 frames to one another over the network 100. Each switch 110-115, moreover, preferably includes a plurality of ports 202 such that each LAN 104-109 is coupled to at least one port of switches 110-115.
At least some of the switches 110-115 may be interconnected by a series oflinks, such as point-to-point duplex links 119-123. Links 119-123 similarly carry messages, such as data frames, between respective switches. Each switch 110-115, moreover, preferably identifies its own ports 202, e.g., by
10 port numbers, such as zero, one, two, three, etc. Switches 110-115 are thus able to associate specific ports with the entities, LANs and/or switches coupled thereto.
Briefly, the present invention is directed to a system and method for preventing the formation of loops that are not detected by spanning tree protocols or algorithms. An intermediate network device operating in accordance with the present invention preferably includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol (STP) engine in communicating relationship with the ports. The STP engine includes a port transition state machine for transitioning the ports among a plurality of STP states, such as a discarding or blocking state, a learning state and a forwarding state. The STP engine may also include a 20
port role selection state machine for assigning STP roles to the ports or for recognizing the association of roles to the ports, including a Root Port Role, an Alternate Port Role, a Designated Port Role and a Backup Port Role. In accordance with the present invention, the device further includes a loop 25
guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given port stops receiving BPDU messages, the loop guard engine prevents the STP engine 30
from allowing the given port to become a designated port, thereby preventing the given port from being transitioned to the forwarding state. Instead, the loop guard engine preferably causes the port to be transitioned to a new state, e.g., the loop inconsistent state. A port in the loop inconsistent state is 35
precluded from forwarding or receiving network messages. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the loop inconsistent state, thereby allowing the port to be transitioned to one of the RSTP states. In the preferred embodiment, the loop guard 40
engine operates on ports assigned to the Root and Alternate Port Roles.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention description below refers to the accompanying drawings, of which:
FIG. 1 is a highly schematic representation of a computer network;
FIG. 2 is a highly schematic, partial block diagram of an intermediate network device in accordance with the present invention;
FIGS. 3A-B is a flow diagram of a preferred method of the present invention; and
FIGS. 4 and 5 are a state diagram and an event table in accordance with the present invention.
DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT
FIG. 1 illustrates a partially meshed, bridged network 100 in accordance with the present invention. The network 100 preferably comprises a plurality of local area networks (LANs) 104-109 that are interconnected by a plurality of intermediate devices, such as switches 110-115. One or more entities or hosts (not shown) are preferably coupled to each LAN 104-109 so that the entities may source or sink data
It should be understood that the network 100 of FIG. 1 is meant for illustrative purposes only and that the present
15 invention will operate with any network having redundant connections.
As shown, network 100 includes redundant paths intercon-necting switches 110-115. For example, switch 112 is connected to switch 113 along at least two different paths; first, via switch 111 and second, via switch 115. The existence of such redundant paths prevents portions of the network 100 from becoming isolated should any constituent link or device fail. Such redundancy, however, also results in the creation of loops, which, as described above, are highly undesirable.
Execution of a spanning tree protocol or algorithm prevents loops by defining a loop-free network topology (i.e., an active topology). However, as set forth above, in some situations, conventional spanning tree protocols or algorithms may not detect the existence or formation of all loops. To avoid the problems created by loops that are not detected by spanning tree protocols or algorithms, among other reasons, at least some of the intermediate network devices (e.g., the switches, bridges, etc.) of network 100 utilize a "loop guard mechanism" in accordance with the present invention.
FIG. 2 is a partial block diagram of an intermediate network device in accordance with the present invention, such as switch 112. Switch 112 includes a plurality of ports 202a-202e each of which is preferably identified by a number (e.g., PO-P4). One or more frame transmission and reception objects, designated generally 204, are associated with the ports 202a-e such that network messages, including data packets and frames, received at a given port, e.g., P3, may be captured, and frames to be transmitted by switch 112 may be delivered to a given port, e.g., Pl. Frame reception and trans-
45 mission objects 204 are preferably message storage structures, such as queues. In the illustrated embodiment, switch 112 includes transmitting and receiving circuitry, including one or more line cards and/or network interface cards (NICs) establishing ports for the exchange of network messages, one
50 or more or central processing units (CPU s) and/or microprocessors and associated memory devices for performing calculations and one or more bus structures.
Switch 112 further includes at least one protocol entity 206 comprising a plurality of components. In particular, the pro-
55 tocol entity 206 includes at least one spanning tree protocol (S TP) engine 208 and at least one forwarding engine 210. The STP engine 208 preferably comprises a plurality of subcomponents, including a port role selection state machine 212, a port transition state machine 214, a bridge protocol data unit
60 (BPDU) message generator 216 and a loop guard engine 218. Except as described herein, the STP engine 208 preferably operates substantially in compliance with a known spanning tree protocol or algorithm, such as the Spanning Tree Protocol (STP) defined in the IEEE 802.1D specification standard, the
65 Rapid Spanning Tree Protocol (RSTP) defined in the IEEE 802.1 w supplement to the 802.1D specification standard, or the Multiple Spanning Trees (MST) protocol defined in the
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page12 of 17
US 7,460,492 B2 7
IEEE 802.1s supplement (Draft 10, Jun. 16, 2001)to the IEEE 802.1 Q specification standard, among others, all of which are hereby incorporated by reference in their entirety. The STP engine 208 includes or is in communicating relationship with a memory 220, which may be a volatile or non-volatile random access memory (RAM) or some other memory structure
8
or device. Memory 220 is preferably organized to include a plurality of records or cells (not shown) for storing spanning tree related information or parameters, such as the switch's numeric bridge identifier (ID), the assigned path cost for each port 202a-e, the current or "best" spanning tree information for each port PO-P4, etc.
The forwarding engine 210 is in communicating relationship with the frame transmission and reception objects 204 and is coupled to at least one filtering database 222 that stores 15
address information corresponding to at least some of the entities of network 100 (FIG. 1). Specifically, filtering database 222 has a plurality of records (not shown) each containing a plurality of cells, including a destination address cell, a destination port cell and a corresponding timer cell. Each 20
record in the filtering database 222 preferably corresponds to
unidirectional. Switch 110 may continue to send BPDU messages on its port coupled to trunk 120, but these BPDU messages are not received by switch 112 as trunk 120 has become unidirectional. As described above, in a stable topology, a non-root bridge, such as switch 112, periodically receives BPDU messages that originate from the root on its root port as well as on its blocked ports. In response, the bridge transmits its own BPDU messages from its designated ports. If the bridge stops receiving BPDU messages on a
10 given port, it will continue to increment the message age value until it reaches the maximum age threshold. At that point, the spanning tree protocol engine 208 discards the BPDU information stored for the respective port, as indicated
a particular network entity. The forwarding engine 210 is configured to switch or
bridge network messages, such as packets and/or frames, from a source port 202 to one or more destinations ports 202 25
depending on information contained in the forwarding database 222 and also on the spanning tree port states of the respective ports 202 as managed by STP engine 208. The forwarding engine 212 is also in communicating relationship with the STP engine 208 and relays RSTP-related messages, 30
such as BPDU messages, received at ports 202 thereto. STP engine 208 may also be directly coupled to the frame transmission and reception objects 204.
It will be understood by those skilled in the art that STP engine 208 and forwarding engine 210 may each comprise 35
registers and combinational logic configured and arranged to produce sequential logic circuits. In the illustrated embodiment, engines 208 and 210 are preferably software modules or libraries containing program instructions pertaining to the methods described herein and executable by one or more 40
processing elements (not shown) of switch 112. Other computer readable media may also be used to store and execute these program instructions. Nonetheless, those skilled in the art will recognize that various combinations of software and hardware, including firmware, may be utilized to implement 45
the present invention.
at block 304.
If switch 112 were following a conventional STP or algorithm, it would then determine the spanning tree port state to which the respective port should be transitioned. In this case, switch 112 would conclude that port PO should become a designated port, and that it should therefore be transitioned either directly or through the learning state to the forwarding state. Transitioning port PO to the forwarding state, however, which would occur with conventional STPs or algorithms, results in the formation of a loop in the bridged network 100. As described above, the existence of a loop may result in a proliferation of network messages, overwhelming the message transport capacity of the bridged network 100.
Utilization of the present invention prevents the formation of such a loop. More specifically, in accordance with the present invention, when the message age timer for port PO expires and the current BPDU information is discarded, the loop guard engine 218 steps in and determines whether "loop guard" has been enabled for port PO, as indicated at decision block 306. If it is, the loop guard engine 218 prevents the port from becoming a designated port. In particular, engine 218 preferably directs the port transition state machine engine 214 to transition port PO to a new spanning tree state, preferably called the "loop-inconsistent" state, as indicated by Yes arrow 307 and block 308. The spanning tree protocol engine 208, moreover, is preferably configured such that network messages are neither forwarded from nor received on a port that is in the loop inconsistent state, as indicated at block 310. For example, the spanning tree protocol engine 208 may instruct the forwarding engine to drop any and all network messages, e.g., data packets or frames, that are received on port PO, and to discard any network messages that would otherwise be forwarded from port PO, other than BPDU messages. The STP engine 208 also recalculates the role and spanning tree port state of each port 202, and suppresses the transmission or
Suitable intermediate network device platforms for use with the present invention include, but are not limited to, the commercially available Catalyst 4000 and 6000 series of switches from Cisco Systems, Inc. of San Jose, Calif. 50 sending of BPDU messages from port PO, as indicated at
block 312. Execution of the STP by the switches 110-115 (FIG. 1) of the bridged network 100 results in the convergence to an active topology with one device, e.g., switch 110, being elected the root. Suppose that port PO of switch 112 is assigned the Root Port Role and is transitioned to the forward- 55
ing state, and that port P1 is assigned the Alternate Port Role as it represents an alternate path to root 110. Port P1 is transitioned to the blocking or discarding state. The terms blocking and discarding are used interchangeably herein. In addition, suppose that ports P2-P4 of switch 112 are assigned 60
the Designated Port Role and that each port is transitioned to the forwarding state.
FIGS. 3A-B is a flow diagram of a preferred embodiment of the method of the present invention. Suppose switch 112 stops receiving BPDU messages on a given port, e.g., port PO 65
which is connected to switch 110 via trunk 120, as indicated at block 302 (FIG. 3A). That is, suppose trunk 120 becomes
While port PO is in the loop inconsistent state, the spanning tree protocol engine 208 preferably checks for the receipt of any BPDU messages on port PO, as indicated by decision block 314 (FIG. 3B). As indicated by No arrow 315 which loops back onto decision block 314, the spanning tree protocol engine 208 keeps checking for the receipt of any BPDU messages. If a BPDU message is received on port PO, the loop guard engine 218 preferably releases port PO from the loop inconsistent state, as indicated by Yes arrow 316 and block 318. Once released from the loop inconsistent state, the port transition state machine 214 preferably transitions port PO to one of the conventional spanning tree port states, e.g., discarding, listening, learning, forwarding, etc., as indicated at block 320. In a conventional mauner, the particular spanning tree port state to which port PO transitions depends on the information contained in the received BPDU message.
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page13 of 17
US 7,460,492 B2 9
Referring again to decision block 306 (FIG. 3A), if loop guard is not enabled on the port which stopped receiving BPDU messages, then the port is preferably transitioned in accordance with the conventional STP or algorithm being executed at switch 112, as indicated by No arrow 322leading to block 324. That is, the port moves to a conventional spanning tree port state.
10 guard. A network administrator working either locally or remotely from switch 112 preferably sets or loads the configuration information. It should be understood that the loop guard mechanism of the present invention may be implemented in such a way that it is implicitly effective on all point-to-point duplex links on any given bridge. Also, determination of a point-to-point link may depend on the configuration items as described in the RSTP specification standard.
In the preferred embodiment, loop guard is designed for FIG. 4 is a highly schematic state diagram 400 in accor
dance with the present invention. The state diagram 400, which is utilized by the spanning tree port state transition machine 214, illustrates the spanning tree port states through which each port 202 may be transitioned. FIG. 5 is an event table 500 describing at least some of the events that result in a transition among the spanning tree port states shown in FIG. 4. In general, the port state transition state machine 214 preferably transitions the ports 202 of switch 112 among the following spanning tree port states: discarding or blocking 402, learning 404, forwarding 406 and loop inconsistent 408. State machine 214 may also transition ports 202 through other spanning tree port states, such as a listening state, among others.
10 use only on ports that are and/or are likely to be assigned the Alternate Port Role or the Root Port Role for all possible combinations of active topologies. In deciding whether or not to enable loop guard on a given port, the network administrator preferably takes into account all possible fail over sce-
15 narios. Ports that are and/or are likely to be assigned to either the Designated Port Role or the Backup Port Role preferably have loop guard disabled. In other words, loop guard is not enabled on ports coupled to shared media, such as ports P2, P3 andP4 ofswitch112whicharecoupledto LANs 107,106,
20 and 105, respectively. By default, loop guard is preferably disabled. That is, loop guard is only enabled in response to explicit or overt action by the network administrator, such as the entering of a specific command during configuration of the switch.
Event E1 occurs when a port, for which loop guard is enabled, stops receiving BPDU messages. As described above and as illustrated in FIG. 4, event E1 results in the port transitioning to the loop inconsistent state 408. This may 25
occur, moreover, from any other state 402-406. That is, an alternate port or root port, which are typically in the discarding and forwarding states 402, 406, respectively, may stop receiving BPDU messages. Furthermore, the root port may stop receiving BPDU messages while it is still in the learning 30
state 404. This also causes a transition to the loop inconsistent state 408.
Ports for which loop guard should be enabled include ports coupled to the uplinks of an access switch, the root port of a secondary root switch, and a designated port of a root switch that could become the root port if some other switch is elected the root, among others. An access switch is an intermediate network device to which end stations, e.g., workstations, servers, etc., are directly coupled and which is typically located at an edge of a computer network. The uplinks refer to the trunk links that couple the access switch to the network backbone.
In the preferred embodiment, when a port is transitioned to the loop inconsistent state, the loop guard engine 218 preferably logs a first message reflecting that occurrence. Similarly, when a port moves out of the loop inconsistent state, the loop guard engine 218logs a second message reflecting that occur-
Event E2 corresponds to the receipt of a BPDU message while in the loop inconsistent state 408. As indicated above, the receipt of a BPDU message causes the port to transition 35
out of the loop inconsistent state 408. The port typically transitions to the discarding state 402, and subsequently, the STP engine 208 determines the proper port role and state, depending on the information contained in the received BPDU message. 40 renee. These messages, which may be accessed and reviewed
by a network administrator as a diagnostic check, are preferably stored in a log file at the switch 112, such as a syslog file. Alternatively or additionally, the messages may be sent to a
Event E3 corresponds to a designated port or the root port in the blocking state 402 transitioning to the learning state 404 due to the expiration of the forward delay time. Event E4 corresponds to the expiration of the forward delay time without receipt of a BPDU message containing "better" informa- 45
tion, while event E5 corresponds to the reciept of "better" BPDU information by a port in the learning state 404 or in the forwarding state 406. Event E6 corresponds to a port in the blocking state 402 becoming a designated port or the root port, and being able to transition directly to the forwarding 50
state 406 as provided by the 802.1w or 802.1s specification standards.
It should be understood that the port transition state machine 214 may employ other spanning tree port states, such as the disabled state, the listening state (which is 55
described in the 802.1D specification standard), and the forgetting state as described in U.S. Pat. No. 5,790,808, titled Active Topology Maintenance in Reconfiguring Bridged Local Area Networks with State Transition with Forgetting Interval to Michael Seaman, which is also hereby incorpo- 60
rated by reference in its entirety, among other spanning tree port states.
As shown, loop guard is preferably enabled or disabled on a port-by-port basis. More specifically, the loop guard engine 218 may have access to configuration information stored for 65
switch 112. This configuration information, moreover, preferably specifies which ports are and are not enabled for loop
network administration console via the well-known Simple Network Management Protocol (SNMP) or by some other method.
As indicated above, it should be understood that the present invention may be used with any spanning tree protocol or algorithm, which, in addition to those previously mentioned, includes the algorithm described at pp. 54-75 ofR. Perlman Interconnections: Bridges and Routers (Addison-Wesley 1992), among others.
It should be further understood that rather than transitioning a port to the loop inconsistent state 408, the loop guard engine 218 could be configured to transition the port to or keep the port in the discarding state 402, as the case may be. Once the port is in the discarding state 402, the loop guard engine 218 keeps it there until a BPDU message is received. In other words, the loop guard engine 218 overrides the port role selection state machine 212 and the port transition state machine 214, which might otherwise try to transistion the port to the learning and/or forwarding states 404, 406 when no BPDU messages are received.
Interoperation With Other Switching Functions The loop guard mechanism of the present invention may
also be configured to operate with other features employed by the switch.
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page14 of 17
US 7,460,492 B2 11
Multiple Spanning Tree Instances 12
rality of spanning trees are defined and shared by a number of VLAN designations. If switch 112 employed a shared spanning tree protocol and stopped receiving BPDU messages for a particular active topology, then it would preferably transition the spanning tree port state associated with the particular active topology to the loop inconsistent state 408. Network messages associated with each of the VLAN designations assigned to the particular spanning tree would be blocked from the affected port, while network messages associated
Those skilled in the art understand that the bridged network 100 may be segregated into a series of logical network segments. U.S. Pat. No. 5,394,402, issued Feb. 28, 1995 (the "'402 Patent"), for example, discloses an arrangement for associating any port of a switch with any particular segregated network group. Specifically, according to the '402 Patent, any number of physical ports of a particular switch may be associated with any number of groups within the switch by using a virtual local area network (VLAN) arrangement that virtually associates the port with a particular VLAN designation. These VLAN designations are also associated with the messages that are received on these ports. In particular, every time
10 with VLAN designations assigned to other spanning trees could continue to be forwarded and received.
Port Aggregation Protocol (PAgP) Multiple physical ports, e.g., ports 202, may also be logi
cally aggregated into a virtual port or a channel. U.S. Pat. No. a message is received on one of these ports, the VLAN designation for that port, as stored in a memory portion of the bridge, is associated with the message. For convenience, each VLAN designation is often associated with a different color, such as red, blue, green, etc.
In addition to the '402 Patent, the IEEE has promulgated the 802.1Q specification standard for Virtual Bridged Local Area Networks. The IEEE's 802.1Q standard supports VLAN sand defines a specific VLAN -tagged message format for transmission on trunks.
15 5,959,968, titled Port Aggregation Protocol to Hon Wah Chin et a!., for example, describes a system for aggregating a plurality of physical ports into a single, logical aggregation port. Alternatively, a network administrator can manually group two or more physical ports into a corresponding chan-
With the development ofVLAN s, several "solutions" have been developed for overlaying spanning trees on these virtually segregated network groups. The IEEE 802.1 Q standards committee, for example, has proposed defining a single spanning tree for all VLAN designations in the computer network. Thus, either all VLAN tagged frames may be forwarded and received through a given port or none may be. An alternative
20 nel. Typically, the spanning tree protocol runs"above" the port aggregation protocol. That is, physical ports are aggregated and then the spanning tree protocol only considers the logical aggregation ports or channels, and not the underlying physical ports, in computing the active topology. If BPDU
25 messages stop being receiving on a logical aggregation port or channel, the loop guard engine 218 preferably causes that logical aggregation port or channel to be transitioned to the loop inconsistent state 408. In other words, network messages are blocked from being forwarded on or received from each of
30 the underlying physical ports that comprise the affected logical aggregation port or channel. to the 802.1Q single spanning tree approach is to define a
separate spanning tree for each VLAN designation within the network. This alternative is currently being implemented by certain networking equipment from Cisco Systems, Inc., as described in the Cisco lOS VLAN Services document. With this approach, BPDU s are preferably tagged with each of the VLAN designations defined within the bridged network. Upon receipt, these tagged BPDU s are then processed by the switches so as to define a separate spanning tree or active topology for each VLAN designation within the bridged net- 40
work. Thus, for a given port, messages associated with one VLAN designation, e.g., blue, may be forwarded and received while messages associated with a second VLAN designation, e.g., green, may be blocked.
Unidirectional Link Detection Protocol (UDLD) The Unidirectional Link Detection Protocol (UDLD) from
Cisco Systems, Inc. is a layer 2 (L2) protocol for determining 35 the physical status of a link. In particular, UDLD detects the
identities of neighbors and shuts down misconnected ports. Together with autonegotiation, which operates at layer 1 (Ll ), UDLD can prevent physical and logical unidirectional connections.
Suppose switch 112 (FIG. 1) is employing the multiple 45
spanning tree solution and 5 that it stops receiving BPDU messages associated with one particular VLAN, e.g.,"red",
The loop guard mechanism of the present invention can work in a complementary fashion with UDLD. That is, both may be implemented on a given port. Depending on the configuration or setting of various STP timers, such as forward delay, UDLD or loop guard will be the first to detect a unidirectional failure.
Uplink/Backbone Fast Those skilled in the art will recognize that other mecha-
nisms exist besides RSTP to transition ports from the blocking spanning tree port state directly and rapidly to the forwarding state. U.S. Pat. No. 6,032,194, titled Method and Apparatus for Rapidly Reconfiguring Computer Networks to Silvana Gai, et a!. describes such a method. U.S. Pat. No. 6,202,114, titled Spanning Tree with Fast Link-Failure Detection to Dinesh Dutt et a!. describes another such method. Many of these other mechanisms transition the affected port before the corresponding maximum age timer expires. As the loop guard mechanism of the present invention preferably waits until the maximum age timer expires before transitioning a port to the loop inconsistent state 408,
on port P1, but that it continues to receive BPDU messages associated with other VLANs, e.g., blue and green, on the port. In this case, the loop guard engine 218 preferably causes 50
the port transition state machine 214 to transition the port PO's spanning tree port state to the loop inconsistent state 408 but only for the red VLAN. That is, the loop guard engine 218 preferably allows the spanning tree port states associated with the blue and green VLANs to remain as they are, as BPDU 55
messages for these VLANs continue to be received on port PO. Accordingly, network messages tagged with the"red" VLAN designation are blocked from port PO, while network messages tagged with either the blue or the green VLAN designations may continue to be forwarded and received.
Rather than providing a separate active topology for each VLAN designation within the bridged network 100, it is also possible to define more than one active topology but some number less than the total number ofVLAN designations. In addition to the MST protocol mentioned above, U.S. Pat. No. 65
6,188,694, titled Shared Spanning Tree Protocol to Michael Fine et a!., for example, describes a system in which a plu-
60 these other mechanisms will operate transparently to loop guard. In other words, the port will transition to forwarding before loop guard is triggered.
The foregoing description has been directed to specific embodiments of this invention. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. Therefore, it is an object of the appended
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page15 of 17
US 7,460,492 B2 13
claims to cover all such variations and modifications as come within the true spirit and scope of the invention. modifications as come within the true spirit and scope of the invention.
What is claimed is: 1. An intermediate network device configured to receive
and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:
one or more processing elements configured to execute a 10
spanning tree protocol engine, the spanning tree protocol engine to implement a Rapid Spanning Tree Protocol (RSTP) to transition at least some of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, 15
and a forwarding port state; and the one or more processing elements further configured to
execute a loop guard engine, the loop guard engine to cooperate with the spanning tree protocol engine, the loop guard engine configured to, in response to a stop- 20
page of a periodic receipt of bridge protocol data units (BPDUs) on a given port, prevent the given port from transitioning to the forwarding port state, if the given port is in a port state other than the forwarding port state.
2. The intermediate network device of claim 1 wherein the 25
loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDU s) on the given port, prevent the given port from forwarding or receiving network messages, if the given port is in the forwarding port state. 30
3. The intermediate network device of claim 1 wherein the plurality of port states further include a loop inconsistent port state, and the loop guard engine is further configured to cause the given port to be transitioned to the loop inconsistent port state. 35
4. The intermediate network device of claim 3 wherein the loop guard engine is further configured to cause the given port to be released from the loop inconsistent port state in response to receipt of a BPDU on the given port.
5. The intermediate network device of claim 4 wherein the 40
spanning tree protocol engine is further configured to, in response to release of the given port from the loop inconsistent port state, transition the given port to another port state according to the RSTP.
6. The intermediate network device of claim 1 further 45
comprising: a timer associated with the given port, the timer configured
to be reset upon receipt of each BPDU message at the given port; and
wherein the loop guard engine is further configured to, in 50
response to the timer reaching zero, detect stoppage of periodic receipt ofBPDUs.
7. The intermediate network device of claim 1 wherein the loop guard engine is further configured to prevent the given port from transitioning to the forwarding port state only if the 55
given port is assigned a RSTP Root Port role or an RSTP Alternate Port role.
8. The intermediate network device of claim 1 wherein the loop guard engine is further configured to prevent the given port from transitioning to the forwarding port state only if the 60
given port is one of a root port and a port that may potentially become a root port if another network device is elected to be a root device.
9. A method for preventing the formation of loops by an intermediate network device having a plurality of ports for 65
receiving and forwarding network messages to one or more other devices, the method comprising the steps of:
14 executing a Rapid Spanning Tree Protocol (RSTP) at the
intermediate network device to transition at least one of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state;
detecting a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on a given port of the intermediate network device's ports; and
in response to the stoppage, preventing the given port from transitioning to the forwarding port state, if the given port is in a port state other than the forwarding port state.
10. The method of claim 9 further comprising the step of: in response to the stoppage, preventing the given port from
forwarding or receiving network messages, if the given port is in the forwarding port state.
11. The method of claim 10 wherein the port states further include a loop inconsistent port state, and the method further comprises the step of:
placing the given port in the loop inconsistent port state. 12. The method of claim 11 further comprising: precluding all ports in the loop inconsistent port state from
transitioning to another port state and from forwarding or receiving network messages.
13. The method of claim 11 further comprising the steps of: releasing the given port from the loop inconsistent port
state, in response to receipt of a BPDU on the given port; and
transitioning the given port to another port state according to the RSTP.
14. The method of claim 11 wherein the placing of ports in the loop inconsistent state is performed on a port-by-port basis.
15. The method of claim 9 further comprising the steps of: resetting a timer associated with the given port upon receipt
of a BPDU message at the given port; and in response to the timer reaching zero, detecting stoppage
of periodic receipt of BPDU s. 16. The method of claim 9 wherein the step of preventing is
performed only if the given port is assigned a RSTP Root Port role or an RSTP Alternate Port role.
17. An intermediate network device configured to receive and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:
means for executing a Rapid Spanning Tree Protocol (RSTP) at the intermediate network device to transition at least one of the intermediate network device's ports among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state;
means for detecting a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on a given port of the intermediate network device's ports; and
means for preventing the given port from transitioning to the forwarding port state, in response to the stoppage, if the given port is in a port state other than the forwarding port state.
18. The intermediate network device of claim 17 further comprising:
means preventing the given port from forwarding or receiving network messages in response to the stoppage, if the given port is in the forwarding port state.
19. An intermediate network device configured to receive and forward network messages, the intermediate network
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page16 of 17
US 7,460,492 B2 15
device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:
a given port, of the plurality of ports, associated with one or more virtual local area networks (VLANs);
a spanning tree protocol engine configured to implement a Spanning Tree Protocol (STP) to transition the given port among a plurality of port states for each VLAN of the given port, such that the given port has a separate port state for each VLAN, the port states including a discard- 10
ing port state, a listening port state, and a forwarding port state; and
a loop guard engine configured to, in response to a stoppage of a periodic receipt of bridge protocol data units (BPDUs) associated with a particular VLAN of the 15
given port, prevent the given port from transitioning to the forwarding port state for the particular VLAN, if the given port is in a port state other than the forwarding port state for the particular VLAN.
20. The intermediate network device of claim 19 wherein 20
the loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDUs) on the given port for the particular VLAN, prevent the given port from forwarding or receiving network messages for the particular VLAN, if the given port is in the
25
forwarding port state for the particular VLAN. 21. The intermediate network device of claim 19 wherein
the STP is a Rapid Spanning Tree Protocol (RSTP).
16 22. An intermediate network device configured to receive
and forward network messages, the intermediate network device having a plurality of ports to connect the intermediate network device to one or more other devices, the intermediate network device comprising:
a virtual port including two or more ports aggregated to function as a single port;
a spanning tree protocol engine configured to implement a Spanning Tree Protocol (STP) to transition the virtual port among a plurality of port states, including a discarding port state, a listening port state, and a forwarding port state; and
a loop guard engine configured to cooperate with the spanning tree protocol engine, the loop guard engine configured to, in response to a stoppage of a periodic receipt of bridge protocol data units (BPDUs) on the virtual port, prevent the virtual port from transitioning to the forwarding port state, if the virtual port is in a port state other than the forwarding port state.
23. The intermediate network device of claim 22 wherein the loop guard engine is further configured to, in response to the stoppage of the periodic receipt of bridge protocol data units (BPDUs) on the virtual port, prevent the virtual port from forwarding or receiving network, if the virtual port is in the forwarding port state.
24. The intermediate network device of claim 22 wherein the STP is a Rapid Spanning Tree Protocol (RSTP).
* * * * *
Case3:14-cv-05343 Document1-11 Filed12/05/14 Page17 of 17
Exhibit 12
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page1 of 15
c12) United States Patent Portolani et al.
(54) SPANNING TREE LOOP GUARD
(75) Inventors: Maurizio Portolani, Milpitas, CA (US); Shyamasundar S. Kaluve, Santa Clara, CA (US); Marco E. Foschiano, San Jose, CA (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 1015 days.
(21) Appl. No.: 10/020,667
(22) Filed: Dec. 7, 2001
(51) Int. Cl. H04L 12156 (2006.01) H04L 12128 (2006.01) H04J 3124 (2006.01) G06F 15116 (2006.01)
(52) U.S. Cl. ...................... 370/256; 370/402; 370/408; 709/238
(58) Field of Classification Search ................ 370/256; 709/238
See application file for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
5,790,808 A 8/1998 Seaman 5,959,968 A 9/1999 Chin eta!. 6,032,194 A 212000 Gai eta!. 6,188,694 B1 * 2/2001 Fine eta!. .................. 370/402 6,202,114 B1 3/2001 Dutt eta!. 6,219,739 B1 * 4/2001 Dutt eta!. .................. 710/311 6,304,575 B1 10/2001 Carroll eta!. 6,628,624 B1 9/2003 Mahajan et al. 6,697,339 B1 * 2/2004 Jain ........................... 370/256 6,765,877 B1 * 7/2004 F oschiano et a!. .......... 370/250 6,801,506 B1 10/2004 Dey 6,813,250 B1 1112004 Fine eta!.
112---..
208
111111 1111111111111111111111111111111111111111111111111111111111111 US007061875Bl
(10) Patent No.: US 7,061,875 Bl Jun.13,2006 (45) Date of Patent:
6,831,898 B1 6,882,630 B1 * 6,891,808 B1 * 6,898,189 B1 6,934,263 B1 *
2001/0025318 A1 * 2003/0016624 A1 *
12/2004 Edsall et a!. 4/2005 Seaman ...................... 370/256 5/2005 Ishii ........................... 370/256 5/2005 Di Benedetto et a!. 8/2005 Seaman ...................... 370/256 9/2001 Higashiyama .............. 709/238 112003 Bare .......................... 370/217
OTHER PUBLICATIONS
Radia Perlman, "Interconnetions Second Edition: Bridges, Routers, Switches, and Internetworking Protocols" pp. 54-75, Addison Wesley Longman, Inc., 2000.
* cited by examiner
Primary Examiner-Hassan Kizou Assistant Examiner-Hong Sol Cho (74) Attorney, Agent, or Firm---Cesari and McKenna, LLP
(57) ABSTRACT
A system and method prevents the formation of loops that are not detected by the Spanning Tree Protocol (STP). An intermediate network device preferably includes a plurality of ports for receiving and forwarding network messages and a STP engine in communicating relationship with the ports. The STP engine transitions the ports among a plurality of spanning tree port states, including a discarding state, a learning state and a forwarding state. The device further includes a loop guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given port stops receiving BPDU messages, the loop guard engine prevents the STP engine from transitioning the given port to the forwarding state. Instead, the loop guard engine preferably causes the port to transition to a new state in which networks messages are explicitly blocked from being forwarded or received. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the new state, thereby allowing it to transition to some other spanning tree port state.
15 Claims, 6 Drawing Sheets
PROTOCOL ENTITY
SPANNING TREE PROTOCOL ENGINE
202a
212 202b
202c
210
202d
202e 206
BPDU MESSAGE
GENERATOR
216 1
MEMORY
220
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page2 of 15
U.S. Patent Jun.l3,2006 Sheet 1 of 6 US 7,061,875 Bl
,_.-100
110 111
119
113
105
123 109
FIG. 1
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page3 of 15
112~
202a
202c
206
208 PROTOCOL ENTITY
SPANNING TREE PROTOCOL ENGINE
PORT ROLE SELECTION
STATE MACHINE
PORT TRANSITION
STATE MACHINE
BPDU MESSAGE
GENERATOR
212 2161
LOOP GUARD ENGINE .
FORWARDING ENGINE 210 MEMORY
222 220
FIG. 2
e • 00 • ~ ~ ~ ~ = ~
2' := .... (.H ~
N 0 0 0\
rFJ
=('D ('D ..... N
0 ..... 0\
d rJl
"'--...1 = 0'1
""""' Oo -.....1 u.
= """"'
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page4 of 15
U.S. Patent Jun. 13,2006 Sheet 3 of 6 US 7,061,875 Bl
GIVEN PORT OF THE SWITCH STOPS RECEIVING BPDU 302 MESSAGES
SPANNING TREE PROTOCOL ENTITY DISCARDS BPDU 304 INFORMATION STORED FOR THE GIVEN PORT
YES
TRANSITION PORT IN ACCORDANCE WITH 324 CONVENTIONAL SPANNING TREE PROTOCOL OPERATION
307
TRANSITION GIVEN PORT 308
TO THE"LOOP INCONSISTENT" STATE
BLOCK GIVEN PORT FROM SENDING OR RECEIVING 310 NETWORK MESSAGES, OTHER THAN BPDU MESSAGES
RECALCULATE PORT ROLES AND STATES USING NEXT BEST BPDU INFORMATION, AND SUPPRESS THE SENDING 312
OF BPDU MESSAGES OUT OF THE GIVEN PORT
TO FIG. 3B
FIG. 3A
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page5 of 15
U.S. Patent
315
NO
Jun.l3,2006 Sheet 4 of 6 US 7,061,875 Bl
FROM FIG. 3A
RELEASE THE GIVEN PORT 318 FROM THE LOOP INCONSISTENT STATE
TRANSITION PORT IN ACCORDANCE 320 WITH CONVENTIONAL SPANNING
TREE PROTOCOL OPERATION
FIG. 38
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page6 of 15
U.S. Patent Jun.13,2006 Sheet 5 of 6 US 7,061,875 Bl
,---400
FIG. 4
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page7 of 15
U.S. Patent Jun.13,2006 Sheet 6 of 6 US 7,061,875 Bl
~500
EVENT TABLE
EVENT DESCRIPTION
E1 PORT HAVING LOOP GUARD ENABLED STOPS RECEIVING BPDU MESSAGES
E2 PORT RECEIVES A BPDU MESSAGE
E3 FORWARD DELAY TIMER EXPIRES FOR A DESIGNATED PORT OR THE ROOT PORT WHILE IN THE DISCARDING STATE
E4 FORWARD DELAY TIME EXPIRES FOR PORT IN LEARNING STATE WITHOUT THE RECEIPT OF "BETTER" BPDU INFORMATION
E5 PORT IN LEARNING OR FORWARDING STATE RECEIVES "BETTER"rBPDU INFORMATION
E6 BLOCKED PORT BECOMES A DESIGNATED PORT OR THE ROOT PORT, AND CAN RAPIDLY TRANSITION TO FORWARDING
FIG. 5
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page8 of 15
US 7,061,875 Bl 1
SPANNING TREE LOOP GUARD
BACKGROUND OF THE INVENTION
1. Field of the Invention The present invention relates generally to computer net
works, and more specifically, to a method and apparatus for preventing the formation of loops.
2. Background Information A computer network typically comprises a plurality of
interconnected entities. An entity may consist of any device, such as a computer or end station, that "sources" (i.e., transmits) or "sinks" (i.e., receives) data frames. A common type of computer network is a local area network ("LAN") which typically refers to a privately owned network within a single building or campus. LANs typically employ a data communication protocol (LAN standard), such as Ethernet, FDDI or token ring, that defines the functions performed by the data link and physical layers of a communications architecture (i.e., a protocol stack). In many instances, several LANs may be interconnected by point-to-point is links, microwave transceivers, satellite hook-ups, etc. to form a wide area network ("WAN") or intranet that may span an entire country or continent.
2 LANs within the network (i.e., the tree is spanning). The Institute of Electrical and Electronics Engineers (IEEE) has promulgated a standard (the 802.1D standard) that defines a spanning tree protocol to be executed by 802.1D compatible devices. In general, by executing the 802.1D spanning tree protocol, bridges elect a single bridge within the bridged network to be the "root" bridge. The 802.1D standard takes advantage of the fact that each bridge has a unique numerical identifier (bridge ID) by specifYing that the root is the bridge
10 with the lowest bridge ID. In addition, for each LAN coupled to more than one bridge, only one (the "designated bridge") is elected to forward frames to and from the respective LAN. The designated bridge is typically the one closest to the root. Each bridge also selects one port (its "root
15 port") which gives the lowest cost path to the root. The root ports and designated bridge ports are selected for inclusion in the active topology and are placed in a forwarding state so that data frames may be forwarded to and from these ports and thus onto the corresponding paths or links of the
20 network. Ports not included within the active topology are placed in a blocking state. When a port is in the blocking state, data frames will not be forwarded to or received from the port. A network administrator may also exclude a port from the spanning tree by placing it in a disabled state.
To obtain the information necessary to run the spanning tree protocol, bridges exchange special messages called configuration bridge protocol data unit (BPDU) messages. More specifically, upon start-up, each bridge initially assumes that it is the root and transmits BPDU messages
One or more intermediate network devices are often used 25
to couple LANs together and allow the corresponding entities to exchange information. For example, a bridge may be used to provide a "bridging" function between two or more LANs. Alternatively, a switch may be utilized to provide a "switching" function for transferring information between a plurality of LANs or end stations. Typically, the bridge or switch is a computer and includes a plurality of ports that couple the device to the LANs or end stations. The switching function includes receiving data from a sending entity at a source port and transferring that data to at least one desti- 35
nation port for forwarding to the receiving entity.
30 accordingly. Upon receipt of a BPDU message from a neighboring device, its contents are examined and compared with similar information (e.g., assumed root and lowest root path cost) stored by the receiving bridge. If the information from the received BPDU is "better" than the stored infor-mation, the bridge adopts the better information and uses it in the BPDUs that it sends (adding the cost associated with the receiving port to the root path cost) from its ports, other than the port on which the "better" information was received. Although BPDU messages are not forwarded by
Switches and bridges typically learn which destination port to use in order to reach a particular entity by noting on which source port the last message originating from that entity was received. This information is then stored by the 40
bridge in a block of memory referred to as a filtering database. Thereafter, when a message addressed to a given entity is received on a source port, the bridge looks up the entity in its filtering database and identifies the appropriate destination port to reach that entity. If no destination port is 45
identified in the filtering database, the bridge floods the message out all ports, except the port on which the message was received. Messages addressed to broadcast or multicast addresses are also flooded.
bridges, the identifier of the root is eventually propagated to and adopted by all bridges as described above, allowing them to select their root port and any designated port(s).
In order to adapt the active topology to changes and failures, the root periodically (e.g., every hello time) transmits BPDU messages. The default hello time is two seconds. In response to receiving BPDUs on their root ports, bridges transmit their own BPDUs from their designated ports, if any. Thus, every two seconds BPDUs are propagated throughout the bridged network, confirming the active topol-
Additionally, most computer networks are either partially or fully meshed. That is, they include redundant communications paths so that a failure of any given link or device does not isolate any portion of the network. The existence of redundant links, however, may cause the formation of circuitous paths or "loops" within the network. Loops are highly undesirable because data frames may traverse the loops indefinitely. Furthermore, because switches and bridges replicate (i.e., flood) frames whose destination port
50 ogy. If a bridge stops receiving BPDU messages on a given port (indicating a possible link or device failure), it will continue to increment a respective message age value until it reaches a maximum age (max age) threshold. The bridge will then age out, i.e., discard, the stored BPDU information
is unknown or which are directed to broadcast or multicast
55 and proceed to re-calculate the root, root path cost and root port by transmitting BPDU messages utilizing the next best information it has. The maximum age value used within the bridged network is typically set by the root, which enters the
addresses, the existence of loops may cause a proliferation 60
of data frames so large that the network becomes overwhelmed.
appropriate value in its BPDU messages. Normally, each bridge replaces its stored BPDU information every hello time, thereby preventing it from being discarded and main-taining the current active topology.
Spanning Tree Protocol To avoid the formation of loops, most bridges and
switches execute a spanning tree protocol or algorithm which allows them to calculate an active network topology that is loop-free (i.e., a tree) and yet connects every pair of
When BPDU information is updated and/or timed-out and the active topology is re-calculated, ports may transition
65 from the blocking state to the forwarding state and vice versa. That is, as a result of new BPDU information, a previously blocked port may learn that it should be in the
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page9 of 15
US 7,061,875 Bl 3
forwarding state (e.g., it is now the root port or a designated port). Rather than transition directly from the blocking state to the forwarding state, the 802.1D standard calls for ports to transition through two intermediate states: a listening state and a learning state. In the listening state, a port waits for information indicating that it should return to the blocking state. If, by the end of a preset time, no such information is received, the port transitions to the learning state. In the learning state, a port still blocks the receiving and forwarding of frames, but received frames are examined and the 10
corresponding location information is stored in the bridge's filtering database. At the end of a second preset time, the port transitions from the learning state to the forwarding state, thereby allowing frames to be forwarded to and from the port. The time spent in each of the listening and the learning 15
states is referred to as the forwarding delay. Although the sparming tree protocol provided in the
802.1D standard is able to maintain a loop-free topology despite network changes and failures, re-calculation of the active topology can be a time consuming and processor 20
intensive task. For example, re-calculation of the spanning tree following an intermediate device crash or failure can take approximately thirty seconds. During this time, message delivery is often delayed as ports transition between states. Such delays can have serious consequences on time- 25
sensitive traffic flows, such as voice or video traffic streams. Rapid Sparming Tree Protocol
4 sages in order to determine which roles to assign to the bridge's ports. The BPDU messages are also utilized in the handshake employed to rapidly transition designated ports to the forwarding state. RSTP also uses timers, including a received information while (rcvdinfoWhile) timer, which is similar to STP's max age timer. The rcvdinfoWhile timer is a count down (to zero) timer, while the max age timer is a count up timer.
Loops Undetectable by Sparming Tree Protocols
In some cases, a single, duplex link coupling two neighboring bridges (which are also indirectly coupled through other bridges or devices) may physically comprise two simplex, i.e., unidirectional, transmission lines, such as two fiber optic lines, operating in opposite directions. Certain failures associated with such lines can result in the formation of loops that are undetectable by the STP. For example, suppose two bridges, designated A and B, are connected by a single trunk link formed from two unidirectional trans-mission lines, and that the respective port at Bridge B is assigned the designated port role, while the peer port at Bridge A is assigned the alternate port role. In this case, the port at Bridge B is placed in the forwarding state and the port at bridge A is placed in the discarding state. As long as the port at Bridge A continues to receive "superior" BPDU messages from Bridge B, it will remain in the blocking state. Suppose, however, that the trunk link becomes unidirectional. That is, bridge B continues to send BPDU messages to Bridge A, but these BPDU messages are never received, and yet the trunk line is not considered to be "down". Accordingly, the BPDU information stored for the port at Bridge A eventually ages out and the STP running at Bridge A transitions the port to the forwarding state. Because Bridge B is unaware of the link failure, the port at Bridge B remains in the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state a loop is created. As described above, the creation of such a loop causes network messages to be replicated, wasting substantial network bandwidth and potentially causing a network outage.
A loop may also be created as a result of an error or failure in the operation of the STP at Bridge B, such as a software error. Specifically, the STP running at Bridge B may determine that the port of Bridge B that is coupled to Bridge A
Recently, the IEEE promulgated a new standard (the 802.1 w standard) that defines a rapid sparming tree protocol (RSTP) to be executed by otherwise 802.1D compatible 30
devices. The RSTP similarly selects one bridge of a bridged network to be the root bridge and defines an active topology that provides complete connectivity among the LANs while severing any loops. Each individual port of each bridge is assigned a port role according to whether the port is to be 35
part of the active topology. The port roles defined by the 802.1 w standard include Root, Designated, Alternate and Backup. The bridge port offering the best, e.g., lowest cost, path to the root is assigned the Root Port Role. Each bridge port offering an alternative, e.g., higher cost, path to the root 40
is assigned the Alternate Port Role. Each bridge port providing the lowest cost path from a given LAN is assigned the Designated Port Role, while all other ports coupled to the given LAN in loop-back fashion are assigned the Backup Port Role. 45 should be assigned the Designated Port Role and be transi
tioned to the forwarding state. Yet, the STP rum1ing at Bridge B may fail for some reason to have BPDU messages sent from the port. In this case, the STP rum1ing at Bridge A concludes that its port should now be assigned the
Those ports that have been assigned the Root Port and Designated Port Roles are placed in the forwarding state, while ports assigned the Alternate and Backup Roles are placed in a discarding or blocking state. A port assigned the Root Port Role can be rapidly transitioned to the forwarding state provided that all of the ports assigned the Alternate Port Role are placed in the discarding or blocking state. Similarly,
50 designated port role and that it should be transitioned to the forwarding state. With the ports at both Bridge A and Bridge B in the forwarding state, a loop is created. Certain hardware failures can similarly result in the creation of loops. For example, the STP running at Bridge B may generate BPDU
if a failure occurs on the port currently assigned the Root Port Role, a port assigned the Alternate Port Role can be reassigned to the Root Port Role and rapidly transitioned to the forwarding state, provided that the previous root port has been transitioned to the discarding or blocking state. A port assigned the Designated Port Role or a Backup Port Role that is to be reassigned to the Designated Port Role can be rapidly transitioned to the forwarding state, provided that the 60
roles of the ports of the downstream bridge are consistent with this port being transitioned to forwarding. The RSTP provides an explicit handshake to be used by neighboring bridges to confirm that a new designated port can rapidly transition to the forwarding state.
Like the STP described in the 802.1D specification standard, bridges rum1ing the RSTP also exchange BPDU mes-
55 messages for transmission from the port coupled to Bride A, but those BPDU messages may never get sent due to a hardware problem at Bridge B.
In summary, unidirectional failures resulting in the formation of loops may occur as a result of malfunctioning or faulty network interface cards (NICs) and/or transceivers, a switch's central processing unit (CPU) being too busy with other processes to send BPDU messages for a relatively long time, a software bug in the STP rum1ing at the switch, or congestion algorithms that end up dropping BPDU mes-
65 sages. In addition, if a link up/down detection and/or autonegotiation protocol is disabled, e.g., by network administrator action, unidirectional failures may go undetected,
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page10 of 15
US 7,061,875 Bl 5
resulting in loops. Accordingly, a need exists to prevent the formation of loops that are undetectable by the STP.
SUMMARY OF THE INVENTION
6 ports 202 such that each LAN 104-109 is coupled to at least one port of switches 110-115.
At least some of the switches 110-115 may be interconnected by a series of links, such as point-to-point duplex links 119-123. Links 119-123 similarly carry messages, such as data frames, between respective switches. Each switch 110-115, moreover, preferably identifies its own ports 202, e.g., by port numbers, such as zero, one, two, three, etc. Switches 110-115 are thus able to associate
10 specific ports with the entities, LANs and/or switches coupled thereto.
Briefly, the present invention is directed to a system and method for preventing the formation of loops that are not detected by spanning tree protocols or algorithms. An intermediate network device operating in accordance with the present invention preferably includes a plurality of ports for receiving and forwarding network messages and a spanning tree protocol (STP) engine in communicating relationship with the ports. The STP engine includes a port transition state machine for transitioning the ports among a plurality of STP states, such as a discarding or blocking state, a learning 15
state and a forwarding state. The STP engine may also include a port role selection state machine for assigning STP roles to the ports or for recognizing the association of roles
It should be understood that the network 100 of FIG. 1 is meant for illustrative purposes only and that the present invention will operate with any network having redundant connections.
to the ports, including a Root Port Role, an Alternate Port Role, a Designated Port Role and a Backup Port Role. In 20
accordance with the present invention, the device further includes a loop guard engine that is in communicating relationship with the STP engine and the ports. The loop guard engine monitors the receipt of configuration bridge protocol data unit (BPDU) messages by the ports. If a given 25
port stops receiving BPDU messages, the loop guard engine prevents the STP engine from allowing the given port to become a designated port, thereby preventing the given port from being transitioned to the forwarding state. Instead, the loop guard engine preferably causes the port to be transi- 30
tioned to a new state, e.g., the loop inconsistent state. A port
As shown, network 100 includes redundant paths interconnecting switches 110-115. For example, switch 112 is connected to switch 113 along at least two different paths; first, via switch 111 and second, via switch 115. The existence of such redundant paths prevents portions of the network 100 from becoming isolated should any constituent link or device fail. Such redundancy, however, also results in the creation of loops, which, as described above, are highly undesirable.
Execution of a spanning tree protocol or algorithm prevents loops by defining a loop-free network topology (i.e., an active topology). However, as set forth above, in some situations, conventional spanning tree protocols or algorithms may not detect the existence or formation of all loops. To avoid the problems created by loops that are not detected by spanning tree protocols or algorithms, among other
in the loop inconsistent state is precluded from forwarding or receiving network messages. If the given port subsequently receives a BPDU message, the loop guard engine releases the port from the loop inconsistent state, thereby 35
allowing the port to be transitioned to one of the RSTP states. In the preferred embodiment, the loop guard engine operates on ports assigned to the Root and Alternate Port Roles.
reasons, at least some of the intermediate network devices (e.g., the switches, bridges, etc.) of network 100 utilize a "loop guard mechanism" in accordance with the present invention.
FIG. 2 is a partial block diagram of an intermediate network device in accordance with the present invention, such as switch 112. Switch 112 includes a plurality of ports 202a-202e each of which is preferably identified by a
BRIEF DESCRIPTION OF THE DRAWINGS
The invention description below refers to the accompanying drawings, of which:
FIG. 1 is a highly schematic representation of a computer network;
FIG. 2 is a highly schematic, partial block diagram of an intermediate network device in accordance with the present invention;
FIGS. 3A-B is a flow diagram of a preferred method of the present invention; and
FIGS. 4 and 5 are a state diagram and an event table in accordance with the present invention.
DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT
FIG. 1 illustrates a partially meshed, bridged network 100 in accordance with the present invention. The network 100 preferably comprises a plurality of local area networks (LANs) 104-109 that are interconnected by a plurality of intermediate devices, such as switches 110-115. One or more entities or hosts (not shown) are preferably coupled to each LAN 104-109 so that the entities may source or sink data frames to one another over the network 100. Each switch 110-115, moreover, preferably includes a plurality of
40 number (e.g., PO-P4). One or more frame transmission and reception objects, designated generally 204, are associated with the ports 202a-e such that network messages, including data packets and frames, received at a given port, e.g., P3, may be captured, and frames to be transmitted by switch 112
45 may be delivered to a given port, e.g., Pl. Frame reception and transmission objects 204 are preferably message storage structures, such as queues. In the illustrated embodiment, switch 112 includes transmitting and receiving circuitry, including one or more line cards and/or network interface
50 cards (NICs) establishing ports for the exchange of network messages, one or more or central processing nnits (CPUs) and/or microprocessors and associated memory devices for performing calculations and one or more bus structures.
Switch 112 further includes at least one protocol entity 55 206 comprising a plurality of components. In particular, the
protocol entity 206 includes at least one spanning tree protocol (STP) engine 208 and at least one forwarding engine 210. The STP engine 208 preferably comprises a plurality of subcomponents, including a port role selection
60 state machine 212, a port transition state machine 214, a bridge protocol data unit (BPDU) message generator 216 and a loop guard engine 218. Except as described herein, the STP engine 208 preferably operates substantially in compliance with a known spanning tree protocol or algorithm,
65 such as the Spanning Tree Protocol (STP) defined in the IEEE 802.1D specification standard, the Rapid Spanning Tree Protocol (RSTP) defined in the IEEE 802.1 w supple-
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page11 of 15
US 7,061,875 Bl 7
ment to the 802.1D specification standard, or the Multiple Spanning Trees (MST) protocol defined in the IEEE 802.1s supplement (Draft 10, Jun. 16, 2001) to the IEEE 802.1Q specification standard, among others, all of which are hereby incorporated by reference in their entirety. The STP engine 208 includes or is in communicating relationship with a memory 220, which may be a volatile or non-volatile random access memory (RAM) or some other memory structure or device. Memory 220 is preferably organized to include a plurality of records or cells (not shown) for storing 10
spanning tree related information or parameters, such as the switch's numeric bridge identifier (ID), the assigned path cost for each port 202a-e, the current or "best" spanning tree information for each port PO-P4, etc.
The forwarding engine 210 is in communicating relation- 15
ship with the frame transmission and reception objects 204 and is coupled to at least one filtering database 222 that stores address information corresponding to at least some of the entities of network 100 (FIG. 1). Specifically, filtering database 222 has a plurality of records (not shown) each 20
containing a plurality of cells, including a destination address cell, a destination port cell and a corresponding timer cell. Each record in the filtering database 222 preferably corresponds to a particular network entity.
8 PO which is connected to switch 110 via trunk 120, as indicated at block 302 (FIG. 3A). That is, suppose trunk 120 becomes unidirectional. Switch 110 may continue to send BPDU messages on its port coupled to trunk 120, but these BPDU messages are not received by switch 112 as trunk 120 has become unidirectional. As described above, in a stable topology, a non-root bridge, such as switch 112, periodically receives BPDU messages that originate from the root on its root port as well as on its blocked ports. In response, the bridge transmits its own BPDU messages from its designated ports. If the bridge stops receiving BPDU messages on a given port, it will continue to increment the message age value until it reaches the maximum age threshold. At that point, the spanning tree protocol engine 208 discards the BPDU information stored for the respective port, as indicated at block 304.
If switch 112 were following a conventional STP or algorithm, it would then determine the spanning tree port state to which the respective port should be transitioned. In this case, switch 112 would conclude that port PO should become a designated port, and that it should therefore be transitioned either directly or through the learning state to the forwarding state. Transitioning port PO to the forwarding state, however, which is would occur with conventional STPs or algorithms, results in the formation of a loop in the bridged network 100. As described above, the existence of a loop may result in a proliferation of network messages, overwhelming the message transport capacity of the bridged network 100.
Utilization of the present invention prevents the formation of such a loop. More specifically, in accordance with the present invention, when the message age timer for port PO expires and the current BPDU information is discarded, the loop guard engine 218 steps in and determines whether
The forwarding engine 210 is configured to switch or 25
bridge network messages, such as packets and/or frames, from a source port 202 to one or more destinations ports 202 depending on information contained in the forwarding database 222 and also on the spanning tree port states of the respective ports 202 as managed by STP engine 208. The 30
forwarding engine 212 is also in communicating relationship with the STP engine 208 and relays RSTP-related messages, such as BPDU messages, received at ports 202 thereto. STP engine 208 may also be directly coupled to the frame transmission and reception objects 204.
It will be understood by those skilled in the art that STP engine 208 and forwarding engine 210 may each comprise registers and combinational logic configured and arranged to produce sequential logic circuits. In the illustrated embodiment, engines 208 and 210 are preferably software modules 40
or libraries containing program instructions pertaining to the methods described herein and executable by one or more processing elements (not shown) of switch 112. Other computer readable media may also be used to store and execute these program instructions. Nonetheless, those skilled in the 45
art will recognize that various combinations of software and hardware, including firmware, may be utilized to implement the present invention.
35 "loop guard" has been enabled for port PO, as indicated at decision block 306. If it is, the loop guard engine 218 prevents the port from becoming a designated port. In particular, engine 218 preferably directs the port transition
Suitable intermediate network device platforms for use with the present invention include, but are not limited to, the 50
commercially available Catalyst 4000 and 6000 series of switches from Cisco Systems, Inc. of San Jose, Calif.
Execution of the STP by the switches 110-115 (FIG. 1) of the bridged network 100 results in the convergence to an active topology with one device, e.g., switch 110, being 55
elected the root. Suppose that port PO of switch 112 is assigned the Root Port Role and is transitioned to the forwarding state, and that port P1 is assigned the Alternate Port Role as it represents an alternate path to root 110. Port P1 is transitioned to the blocking or discarding state. The 60
terms blocking and discarding are used interchangeably herein. In addition, suppose that ports P2-P4 of switch 112 are assigned the Designated Port Role and that each port is transitioned to the forwarding state.
FIGS. 3A-B is a flow diagram of a preferred embodiment 65
of the method of the present invention. Suppose switch 112 stops receiving BPDU messages on a given port, e.g., port
state machine engine 214 to transition port PO to a new spanning tree state, preferably called the "loop-inconsistent" state, as indicated by Yes arrow 307 and block 308. The spanning tree protocol engine 208, moreover, is preferably configured such that network messages are neither forwarded from nor received on a port that is in the loop inconsistent state, as indicated at block 310. For example, the spanning tree protocol engine 208 may instruct the forwarding engine to drop any and all network messages, e.g., data packets or frames, that are received on port PO, and to discard any network messages that would otherwise be forwarded from port PO, other than BPDU messages. The STP engine 208 also recalculates the role and spanning tree port state of each port 202, and suppresses the transmission or sending of BPDU messages from port PO, as indicated at block 312.
While port PO is in the loop inconsistent state, the spanning tree protocol engine 208 preferably checks for the receipt of any BPDU messages on port PO, as indicated by decision block 314 (FIG. 3B). As indicated by No arrow 315 which loops back onto decision block 314, the spanning tree protocol engine 208 keeps checking for the receipt of any BPDU messages. If a BPDU message is received on port PO, the loop guard engine 218 preferably releases port PO from the loop inconsistent state, as indicated by Yes arrow 316 and block 318. Once released from the loop inconsistent state, the port transition state machine 214 preferably transitions port PO to one of the conventional spanning tree port states, e.g., discarding, listening, learning, forwarding, etc.,
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page12 of 15
US 7,061,875 Bl 9
as indicated at block 320. In a conventional manner, the particular spanning tree port state to which port PO transitions depends on the information contained in the received BPDU message.
Referring again to decision block 306 (FIG. 3A), if loop guard is not enabled on the port which stopped receiving BPDU messages, then the port is preferably transitioned in accordance with the conventional STP or algorithm being executed at switch 112, as indicated by No arrow 322 leading to block 324. That is, the port moves to a conventional spanning tree port state.
10
FIG. 4 is a highly schematic state diagram 400 in accordance with the present invention. The state diagram 400, which is utilized by the spanning tree port state transition machine 214, illustrates the spanning tree port states through
15 which each port 202 may be transitioned. FIG. 5 is an event table 500 describing at least some of the events that result in a transition among the spanning tree port states shown in FIG. 4. In general, the port state transition state machine 214 preferably transitions the ports 202 of switch 112 among the following spanning tree port states: discarding or blocking 20
402, learning 404, forwarding 406 and loop inconsistent 408. State machine 214 may also transition ports 202 through other spanning tree port states, such as a listening state, among others.
10 over, preferably specifies which ports are and are not enabled for loop guard. A network administrator working either locally or remotely from switch 112 preferably sets or loads the configuration information. It should be understood that the loop guard mechanism of the present invention may be implemented in such a way that it is implicitly effective on all point-to-point duplex links on any given bridge. Also, determination of a point-to-point link may depend on the configuration items as described in the RSTP specification standard.
In the preferred embodiment, loop guard is designed for use only on ports that are and/or are likely to be assigned the Alternate Port Role or the Root Port Role for all possible combinations of active topologies. In deciding whether or not to enable loop guard on a given port, the network administrator preferably takes into account all possible fail over scenarios. Ports that are and/or are likely to be assigned to either the Designated Port Role or the Backup Port Role preferably have loop guard disabled. In other words, loop guard is not enabled on ports coupled to shared media, such as ports P2, P3 and P4 of switch 112 which are coupled to LANs 107, 106, and 105, respectively. By default, loop guard is preferably disabled. That is, loop guard is only enabled in response to explicit or overt action by the network administrator, such as the entering of a specific command during configuration of the switch.
Ports for which loop guard should be enabled include ports coupled to the uplinks of an access switch, the root port of a secondary root switch, and a designated port of a root switch that could become the root port if some other switch is elected the root, among others. An access switch is an intermediate network device to which end stations, e.g., workstations, servers, etc., are directly coupled and which is typically located at an edge of a computer network. The uplinks refer to the trunk links that couple the access switch
Event E1 occurs when a port, for which loop guard is 25
enabled, stops receiving BPDU messages. As described above and as illustrated in FIG. 4, event E1 results in the port transitioning to the loop inconsistent state 408. This may occur, moreover, from any other state 402-406. That is, an alternate port or root port, which are typically in the dis- 30 carding and forwarding states 402, 406, respectively, may stop receiving BPDU messages. Furthermore, the root port may stop receiving BPDU messages while it is still in the learning state 404. This also causes a transition to the loop inconsistent state 408.
Event E2 corresponds to the receipt of a BPDU message while in the loop inconsistent state 408. As indicated above, the receipt of a BPDU message causes the port to transition out of the loop inconsistent state 408. The port typically transitions to the discarding state 402, and subsequently, the STP engine 208 determines the proper port role and state, 40
depending on the information contained in the received BPDU message.
35 to the network backbone.
Event E3 corresponds to a designated port or the root port in the blocking state 402 transitioning to the learning state 404 due to the expiration of the forward delay time. Event 45
E4 corresponds to the expiration of the forward delay time without receipt of a BPDU message containing "better" information, while event E5 corresponds to the receipt of "better" BPDU information by a port in the learning state 404 or in the forwarding state 406. Event E6 corresponds to 50 a port in the blocking state 402 becoming a designated port or the root port, and being able to transition directly to the forwarding state 406 as provided by the 802.1w or 802.1s specification standards.
It should be understood that the port transition state 55
machine 214 may employ other spanning tree port states, such as the disabled state, the listening state (which is described in the 802.1D specification standard), and the forgetting state as described in U.S. Pat. No. 5,790,808, titled Active Topology Maintenance in Reconfiguring Bridged Local Area Networks with State Transition with 60
Forgetting Interval to Michael Seaman, which is also hereby incorporated by reference in its entirety, among other spanning tree port states.
As shown, loop guard is preferably enabled or disabled on a port-by-port basis. More specifically, the loop guard 65
engine 218 may have access to configuration information stored for switch 112. This configuration information, more-
In the preferred embodiment, when a port is transitioned to the loop inconsistent state, the loop guard engine 218 preferably logs a first message reflecting that occurrence. Similarly, when a port moves out of the loop inconsistent state, the loop guard engine 218 logs a second message reflecting that occurrence. These messages, which may be accessed and reviewed by a network administrator as a diagnostic check, are preferably stored in a log file at the switch 112, such as a syslog file. Alternatively or additionally, the messages may be sent to a network administration console via the well-known Simple Network Management Protocol (SNMP) or by some other method.
As indicated above, it should be understood that the present invention may be used with any spanning tree protocol or algorithm, which, in addition to those previously mentioned, includes the algorithm described at pp. 54-75 of R. Perlman Interconnections: Bridges and Routers (Addison-Wesley 1992), among others.
It should be further understood that rather than transitioning a port to the loop inconsistent state 408, the loop guard engine 218 could be configured to transition the port to or keep the port in the discarding state 402, as the case may be. Once the port is in the discarding state 402, the loop guard engine 218 keeps it there until a BPDU message is received. In other words, the loop guard engine 218 overrides the port role selection state machine 212 and the port transition state machine 214, which might otherwise try to transition the port to the learning and/or forwarding states 404, 406 when no BPDU messages are received.
Interoperation With Other Switching Functions. The loop guard mechanism of the present invention may
also be configured to operate with other features employed by the switch.
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page13 of 15
US 7,061,875 Bl 11
Multiple Spanning Tree Instances Those skilled in the art understand that the bridged
network 100 may be segregated into a series of logical network segments. U.S. Pat. No. 5,394,402, issued Feb. 28, 1995 (the "'402 patent"), for example, discloses an arrangement for associating any port of a switch with any particular segregated network group. Specifically, according to the '402 patent, any number of physical ports of a particular switch may be associated with any number of groups within the switch by using a virtual local area network (VLAN) arrangement that virtually associates the port with a particular VLAN designation. These VLAN designations are also associated with the messages that are received on these ports. In particular, every time a message is received on one of these ports, the VLAN designation for that port, as stored in a memory portion of the bridge, is associated with the message. For convenience, each VLAN designation is often associated with a different color, such as red, blue, green, etc.
In addition to the '402 patent, the IEEE has promulgated the 802.1 Q specification standard for Virtual Bridged Local Area Networks. The IEEE's 802.1Q standard supports VLANs and defines a specific VLAN-tagged message format for transmission on trunks.
With the development of VLANs, several "solutions" have been developed for overlaying spanning trees on these virtually segregated network groups. The IEEE 802.1Q standards committee, for example, has proposed defining a single spanning tree for all VLAN designations in the computer network. Thus, either all VLAN tagged frames may be forwarded and received through a given port or none may be. An alternative to the 802.1Q single spanning tree approach is to define a separate spanning tree for each VLAN designation within the network. This alternative is currently being implemented by certain networking equipment from Cisco Systems, Inc., as described in the Cisco lOS VLAN Services document. With this approach, BPDUs are preferably tagged with each of the VLAN designations defined within the bridged network. Upon receipt, these tagged BPDUs are then processed by the switches so as to define a separate spanning tree or active topology for each VLAN designation within the bridged network. Thus, for a given port, messages associated with one VLAN designation, e.g., blue, may be forwarded and received while messages associated with a second VLAN designation, e.g., green, may be blocked.
Suppose switch 112 (FIG. 1) is employing the multiple spanning tree solution and that it stops receiving BPDU messages associated with one particular VLAN, e.g., "red",
12 a plurality of spanning trees are defined and shared by a number of VLAN designations. If switch 112 employed a shared spanning tree protocol and stopped receiving BPDU messages for a particular active topology, then it would preferably transition the spanning tree port state associated with the particular active topology to the loop inconsistent state 408. Network messages associated with each of the VLAN designations assigned to the particular spanning tree would be blocked from the affected port, while network
10 messages associated with VLAN designations assigned to other spanning trees could continue to be forwarded and received.
Port Aggregation Protocol (PAgP) Multiple physical ports, e.g., ports 202, may also be
15 logically aggregated into a virtual port or a channel. U.S. Pat. No. 5,959,968, titled Port Aggregation Protocol to Hon Wah Chin et a!., for example, describes a system for aggregating a plurality of physical ports into a single, logical aggregation port. Alternatively, a network administrator can manually group two or more physical ports into a corre-
20 sponding channel. Typically, the spanning tree protocol runs "above" the port aggregation protocol. That is, physical ports are aggregated and then the spanning tree protocol only considers the logical aggregation ports or channels, and not the underlying physical ports, in computing the active
25 topology. If BPDU messages stop being receiving on a logical aggregation port or channel, the loop guard engine 218 preferably causes that logical aggregation port or channel to be transitioned to the loop inconsistent state 408. In other words, network messages are blocked from being
30 forwarded on or received from each of the underlying physical ports that comprise the affected logical aggregation port or channel.
Unidirectional Link Detection Protocol (UDLD) The Unidirectional Link Detection Protocol (UDLD)
35 from Cisco Systems, Inc. is a layer 2 (L2) protocol for determining the physical status of a link. In particular, UDLD detects the identities of neighbors and shuts down misconnected ports. Together with autonegotiation, which operates at layer 1 (Ll), UDLD can prevent physical and
40 logical unidirectional connections.
The loop guard mechanism of the present invention can work in a complementary fashion with UDLD. That is, both may be implemented on a given port. Depending on the configuration or setting of various STP timers, such as
45 forward delay, UDLD or loop guard will be the first to detect a unidirectional failure.
Uplink/Backbone Fast Those skilled in the art will recognize that other mecha
nisms exist besides RSTP to transition ports from the blocking spanning tree port state directly and rapidly to the forwarding state. U.S. Pat. No. 6,032,194, titled Method and Apparatus for Rapidly Reconfiguring Computer Networks to Silvana Gai, eta!. describes such a method. U.S. Pat. No. 6,202,114, titled Spanning Tree with Fast Link-Failure Detection to Dinesh Dutt et a!. describes another such method. Many of these other mechanisms transition the affected port before the corresponding maximum age timer expires. As the loop guard mechanism of the present invention preferably waits until the maximum age timer expires before transitioning a port to the loop inconsistent state 408,
on port P1, but that it continues to receive BPDU messages associated with other VLANs, e.g., blue and green, on the port. In this case, the loop guard engine 218 preferably 50 causes the port transition state machine 214 to transition the port PO's spanning tree port state to the loop inconsistent state 408 but only for the red VLAN. That is, the loop guard engine 218 preferably allows the spanning tree port states associated with the blue and green VLANs to remain as they
55 are, as BPDU messages for these VLANs continue to be received on port PO. Accordingly, network messages tagged with the "red" VLAN designation are blocked from port PO, while network messages tagged with either the blue or the green VLAN designations may continue to be forwarded and received. 60 these other mechanisms will operate transparently to loop
guard. In other words, the port will transition to forwarding before loop guard is triggered.
Rather than providing a separate active topology for each VLAN designation within the bridged network 100, it is also possible to define more than one active topology but some number less than the total number ofVLAN designations. In addition to the MST protocol mentioned above, U.S. Pat. 65
No. 6,188,694, titled Shared Spanning Tree Protocol to Michael Fine eta!., for example, describes a system in which
The foregoing description has been directed to specific embodiments of this invention. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. Therefore, it is an object of the
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page14 of 15
US 7,061,875 Bl 13
appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.
What is claimed is: 1. In an intermediate network device having a plurality of
ports for forwarding network messages within a bridged network, a method for preventing the formation of loops within the bridged network, the method comprising the steps of:
executing a spanning tree protocol (STP) at the interme- 10
diate network device so as to elect a root of the bridged network and to transition at least one of the device's ports among a plurality of sparming tree port states, including a discarding state, a listening state and a forwarding state; 15
periodically receiving configuration bridge protocol data unit (BPDU) messages at one or more of the device's ports;
in response to the periodic receipt of BPDU messages being stopped on a given port, (1) preventing the given 20 port from transitioning to the forwarding sparming tree port state, if the given port is in a sparming tree port state other than the forwarding spanning tree port state, or (2) precluding the given port from forwarding or receiving network messages, if the given port is in the 25 forwarding spanning tree port state.
2. The method of claim 1 wherein the sparming tree port states further include a loop inconsistent sparming tree port state, and the method further comprises the step of placing the given port that stopped receiving BPDU messages in the 30
loop inconsistent sparming tree port state. 3. The method of claim 2 wherein a port in the loop
inconsistent state is precluded from transitioning to another spanning tree port state and from forwarding or receiving network messages. 35
4. The method of claim 2 further comprising the steps of: releasing the given port from the loop inconsistent span
ning tree port state, in response to a BPDU message once again being received on the given port; and
transitioning the given port from the loop inconsistent 40
spanning tree port state to another spanning tree port state.
5. The method of claim 4 further comprising the steps of: storing BPDU information from BPDU messages peri
odically received on a first port; resetting a message age timer upon receipt of each BPDU
message at the first port; and if the message age timer reaches a maximum age value
before another BPDU message is received on the first port, discarding the stored BPDU information.
6. The method of claim 5 wherein the given port is considered to have stopped receiving BPDU messages when its message age timer reaches the maximum age value and/or its received information while timer reaches zero.
45
50
7. The method of claim 6 wherein the ability to place ports 55
in the loop inconsistent state is enabled and disabled on a port-by-port basis.
8. The method of claim 6 further comprising the steps of assigning or more ports to a role, the roles including one or more of a Root Port Role, an Alternate Port Role, a Desig- 60 nated Port Role and a Backup Port Role.
9. The method of claim 5 wherein the STP substantially complies with at least one ofthe IEEE 802.1D, 802.1w and 802.1 s specification standards.
14 10. An intermediate network device configured to receive
and forward network messages within a bridged network, the device having a plurality of ports for connecting the device to one or more network entities or other devices, the intermediate network device comprising:
a spanning tree protocol (STP) engine configured and arranged to elect a root of the bridged network and to transition at least some of the device's ports among a plurality of sparming tree port states, including a discarding or blocking state, a listening state and a forwarding state; and
a loop guard engine cooperating with the STP engine, wherein
configuration bridge protocol data unit (BPDU) messages are periodically received at one or more of the device's ports, and
in response to the periodic receipt of BPDU messages being stopped on a given port, the loop guard engine (1) prevents the given port from transitioning to the forwarding spanning tree port state, if the given port is in a spanning tree port state other than the forwarding spanning tree port state, or (2) precludes the given port from forwarding or receiving network messages.
11. The intermediate network device of claim 10 wherein the spanning tree port states further include a loop inconsistent spanning tree port state, and the loop guard engine causes the given port that stopped receiving BPDU messages to be transitioned to the loop inconsistent spanning tree port state.
12. The intermediate network device of claim 11 wherein the spanning tree port states further include a loop incon
sistent spanning tree port state, and the loop guard engine causes the given port that stopped
receiving BPDU messages to be transitioned to the loop inconsistent spanning tree port state.
13. The intermediate network device of claim 12 wherein the loop guard engine causes the given port to be released
from the loop inconsistent spanning tree port state, in response to a BPDU message once again being received on the given port, and
upon being released from the loop inconsistent spanning tree port state, the STP engine transitions the given port to another spanning tree port state.
14. The intermediate network device of claim 13 further comprising a message age time associated with a first port, wherein
the STP engine stores BPDU information from BPDU messages periodically received on the first port,
restarts the message age timer upon receipt of each BPDU message at the first port,
if the message age timer reaches a maximum age value before another BPDU message is received on the first port, the STP engine discards the stored BPDU information, and
the first port is considered to have stopped receiving BPDU messages and is transitioned to the loop inconsistent state when its message age timer reaches the maximum age value.
15. The intermediate network device of claim 10 wherein the given port is kept in a blocking state to preclude the given port from forwarding or receiving network messages.
* * * * *
Case3:14-cv-05343 Document1-12 Filed12/05/14 Page15 of 15
Exhibit 13
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page1 of 15
c12) United States Patent Smethurst et al.
(54) CONTROL PLANE SECURITY AND TRAFFIC FLOW MANAGEMENT
(75) Inventors: Adrian C. Smethurst, Groton, MA (US); Michael F. Keohane, Shrewsbury, MA (US); R. Wayne Ogozaly, Hollis, NH (US)
(73) Assignee: Cisco Technology, Inc., San Jose, CA (US)
( *) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 1000 days.
(21) Appl. No.: 10/307,154
(22) Filed: Nov. 27, 2002
(51) Int. Cl. G06F 11100 (2006.01) H04L 12150 (2006.01) H04L 12128 (2006.01)
(52) U.S. Cl. ...................... 370/229; 370/352; 370/360; 370/401; 370/402
(58) Field of Classification Search ................ 370/229, 370/360, 387, 352, 357, 401, 402; 379/88.22,
379/207.02, 221.08; 709/224, 238 See application file for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
6,304,568 B1 2001/0026612 A1 2002/0097672 A1
10/2001 Kim 10/2001 Duspiva et a!. 7/2002 Barbas et al.
OTHER PUBLICATIONS
Park, K. and Lee, H., "On the Effectiveness of Route-Based Packet Filtering for Distributed DoS Attack Prevention in Power-Law Internets," SIGCOMM' 01:1-12 (2001).
111111 1111111111111111111111111111111111111111111111111111111111111 US007224668B 1
(10) Patent No.: US 7,224,668 Bl May 29,2007 (45) Date of Patent:
Re: [RPSEC] Draft Status, from a protocol developer's angle. [online], Jul. 26, 2002 [retrieved on Sep. 18, 2002]. Retreived from the Internet <URL:https:/ /www1.ietf.org/mailman-archive/working-groups/rpsec/ currentlmsgOO 167 .htrnl>.
Durham, D., et al., "Elimination of Distributed Denial of Service Attacks using Programmable Network Processors," Intel Research and Development: 1-4 (2002).
Flexible Firewalls for Network Processors. [online] [retrieved on Sep. 18, 2002]. Retrieved from the Internet <URL:rnhtrnl:file:// C:\tibnet\dad\clients\cisco\utexas.rnht>.
Primary Examiner-Afsar Qureshi (74) Attorney, Agent, or Firm-Hamilton, Brook, Smith & Reynolds, P.C.
(57) ABSTRACT
An internetworking device that provides improved immunity to Denial of Service attacks, and in general, improved Quality of Service (QoS). An internetworking element or other route processor is composed of two main parts, including a data forwarding plane and a control plane; the control plane runs routing, signaling and control protocols that are responsible for determining the packet forwarding behavior by the data plane. Independent control plane processes may be provided; however, they are considered to be a single network entity that is a uniquely addressable port. Packets thus intended for the control plane always pass through a designated point. As a result, a set of port services unique to the control plane may be applied to the control plane port. These control plane port services thus can be utilized to control all packet traffic entering and exiting the control plane processes as a whole.
72 Claims, 6 Drawing Sheets
r==~~~~~ FORWARDING INTAKES 100
DATA PLANE
135
110
.ACCESS CONTROL LIST~ PER FLOW OoS, etc . .1.§1
ROUTE PROCESSOR
ALL PACKETS
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page2 of 15
100
CONTROL CP PLANE PROCESSOR
150 155
CP 140 __r---j I PACKETS
125-......_ J AGGREGATE CP SERVICES 145
ALL I PACKETS " 1
CENTRAn SWITCH
ENGINE 130
r DATA j PLANE
ALL
135 PACKETS
110 LINECARD
110 LINE CARD
FIG. 1
FORWARDING INTAKES 160 .ACCESS CONTROL LISTS 162 PER FLOW QoS, etc.1§1
n ROUTE
PROCESSOR
ALL ... PACKETS
110 LINE CARD
e • 7J). • ~ ~ ~ ~ = ~
~ ~ N ~'-CI
N 0 0 -....l
rFJ
=('D ('D ..... .... 0 ..... 0\
d rJl -....l 'N N ~ 0.., 0'1 00
= """"'
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page3 of 15
ALL PACKETS ~
r-----v
LEGACY LINECARD
110
CONTROL v--150 PLANE
z ~ CP PACKETS
~140
AGGREGATE CP SERVICES 145
CP CENTRAL SWITCH
ENGINE 130 ~ PACKETS
z~ CP
PACKETS
DISTRIBUTED CP SERVICES 11§
DISTRIBUTED H-131
SWITCH ENGINE
C111
FIG. 2
DISTRIBUTED CP SERVICES 146
DISTRIBUTED ~131 SWITCH ENGINE
'-111
110
e • 00 • ~ ~ ~ ~ = ~
~ ~ N ~'-CI
N 0 0 -....l
rFJ
=('D ('D ..... N
0 ..... 0\
d rJl -....l 'N N ~ 0.., 0'1 00
= """"'
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page4 of 15
PORT6
PORT5 PORT2
PORT4
FIG. 3
e • 00 • ~ ~ ~ ~ = ~
~ ~ N ~'-CI
N 0 0 -....l
rFJ
=('D ('D ..... (.H
0 ..... 0\
d rJl -....l 'N N ~ 0.., 0'1 00
= """"'
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page5 of 15
U.S. Patent May 29,2007 Sheet 4 of 6 US 7,224,668 Bl
LINE CARD RECEIVES THE PACKET AND DELIVERS IT TO CENTRAL SWITCH ENGINE ~ 400
THE CENTRAL SWITCH ENGINE PERFORMS I'-NORMAL INPUT PORT SERVICES AND QOS 402
CENTRAL SWITCH ENGINE PERFORMS L2/L3 SWITCHING OR ROUTING DECISION AND I'-DETERMINES IF THE PACKET IS DESTINED TO THE CP
403
FOR CP PACKETS, THE CENTRAL SWITCH ENGINE PERFORMS AGGREGATE CP SERVICES (TREATED
AS OUTPUT SERVICES FOR THE CP I'- 405 DESTINATION PORT)
BASED ON THE RESULT OF AGGREGATE CP SERVICES, DROP THE PACKET, MARK THE PACKET AND
~ POTENTIALLY DELIVER THE PACKET TO 406 THE CP FOR FINAL PROCESSING
FIG. 4
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page6 of 15
CONFIGURATION
500 ~class: telnet-class match access-group 140
502 .-...__.... policy: control-plane-policy
503 .............__ class tel net-class police 80000 800 conform transmit exceed drop
505 '""'--control-plane service-policy control-plane-policy
506 .-.__access-list 140 deny tcp host 3.3.3.3 any eq telnet 507.....--__ access-list 140 deny tcp host 4.4.4.4 any eq telnet 51Q....,........,_access-list 140 permit tcp any any eq telnet
FIG. 5
COMMENT
Telnet. class defined in telnet-acl
A policy map named control-plane-policy which provides a rate limit of 80,000 bps of telnet traffic: exceeding the limit call be dropped
Attach the control policy to the control plane port
Allow 3.3.3.3 trusted host traffic Allow 4.4.4.4 trusted host traffic Rate limit all other telnet traffic
e • 00 • ~ ~ ~ ~ = ~
~ ~ N \0 ~
N 0 0 -....l
rFJ
=('D
a Ul
0 ..... 0\
d rJl -....l 'N N ~ 0.., 0'1 00
= """"'
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page7 of 15
U.S. Patent May 29,2007 Sheet 6 of 6 US 7,224,668 Bl
DISTRIBUTED LINECARD RECEIVES THE PACKET "-AND DELIVERS IT TO THE DISTRIBUTED SWITCH ENGINE 600
THE DISTRIBUTED SWITCH ENGINE PERFORMS "-NORMAL INPUT PORT SERVICES AND QOS 602
DISTRIBUTED SWITCH ENGINE PERFORMS L2/L3 SWITCHING OR ROUTING DECISION AND I'-DETERMINES IF THE PACKET IS DESTINED 604
TO THE CP
FOR CP PACKETS, THE DISTRIBUTED SWITCH ENGINE PERFORMS DISTRIBUTED CP SERVICES (TREATED !'----AS OUTPUT SERVICES FOR THE CP DESTINATION PORT) 606
BASED ON THE RESULT OF DISTRIBUTED CP SERVICES: DROP THE PACKET, MARK THE PACKET, AND POTENTIALLY I'-DELIVER THE PACKET TO THE CENTRAL SWITCH ENGINE 608
FOR AGGREGATE PROCESSING
THE CENTRAL SWITCH ENGINE PERFORMS AGGREGATE CP SERVICRES AND POTENTIALLY DELIVERS THE ~ 610
PACKET TO THE CP
FIG. 6
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page8 of 15
US 7,224,668 Bl 1
CONTROL PLANE SECURITY AND TRAFFIC FLOW MANAGEMENT
BACKGROUND OF THE INVENTION
2 execution of such processes is critical to the operation of the route processor, when such attacks are directed at such functions, they can be devastating.
DoS attacks that target infrastructure devices they may cause a number of problems, including:
Computer networks, and in particular the web servers, routers, switches, firewalls and end hosts which make up the Internet as well as private intranets have become critical to the operation of many organizations. Recently there have been a number of highly publicized attacks on network 10
equipment belonging to commercial enterprises, government institutions and major internet service providers. This can clearly affect the ability of these institutions to access information, or even to conduct business as usual. Given the widespread adoption of the Internet, in the not to distant 15
future it is foreseeable that such an attack might actually disrupt the conduct of society in general.
loss of line protocol keep alive functions, which causes a network connection to drop, leading to route flaps and major network transitions;
loss of routing protocol updates which can also lead route flaps and network transitions;
causing the control plane to utilize more central processing resources than planned;
causing route processors to lock up, or preventing them from completing higher priority tasks;
reduced response time at user command line prompts, preventing a human administrator from taking corrective action to respond to an attack;
It has been estimated that even a three hour outage of the popular Yahoo servers could cause a loss of commerce and advertising revenue of about $500,000.
Tests have determined that over twelve thousand attacks on five thousand distinct Internet hosts belonging to more than two thousand distinct organizations occurred during a three week period early in 2002.
And, during one week in October 2002, a powerful coordinated electronic attack was directed to the central thirteen servers that manage global Internet traffic. While the attack only lasted for one hour, it caused seven of the thirteen servers to fail to respond.
These attacks, often taking the form of so-called Denial of Service (DoS) attacks, impede the efficient functioning and provisioning of services by network infrastructure elements according to their intended purpose. The impact of such attacks is more pronounced than network congestion itself, due to the concentrated and targeted ability of such attacks to not only deplete specific resources but also clog traffic. In the extreme case, when such attacks are coordinated and distributed over many internetworking devices, they may result in multiple compromised Internet hosts that can disrupt the operation of the Internet itself.
20
consumption of route processor resources such as memory, buffers, data structures, causing negative side effects in being able to process other traffic;
back-up of packet queues leading to indiscriminant drops; ultimately, crashing of the device. Attempts to solve such problems in the past have included
25 such approaches as Reverse Pass Forwarding (RPF) verification and Selective Packet Discard (SPD). Selectively based distributed packet filtering techniques apply filters to packets arriving from specific known mischievous Internet Protocol (IP) addresses. More sophisticated techniques can
30 detect forged source IP addresses by determining the routes from which such disruptive packets originate.
A second technique is to apply an access list configured on an input interface to explicitly deny or limit specific problematic packet types. Hardware based rate limits can then be
35 implemented as a throttling mechanism for the specific packet types so identified. For example, packets of the type SYN can be specifically rate limited on a particular port or other hardware, at least preventing the rate at which such packets are sent to the control plane. A hardware based rate
40 limit may be applied to RPF failures, Internet Control Message Protocol (ICMP) unreachables, Time To Live (TTL) failures, Maximum Transmission Unit (MTU) failures, Internet Protocol Version 4 (IPv4) option bit packets, or similar packets.
Susceptibility to DoS attacks is an intrinsic problem for any service provisioning system where the occurrence of a potentially valid event (such as the request to make a connection, e.g., a Transmission Central Protocol (TCP SYN packet)) must first be processed to ascertain its validity. This 45
is true even though the processing resources needed to handle a single event may be negligible. While such attacks can take on many forms, they typically generate traffic streams at very high data rates. The devices attempt to service even the simplest of commands being thrown at it an 50
extraordinary rate can therefore cause the device to fail.
While these methods all provide some level of control plane protection, specific features and implementations vary from platform to platform. There also remain packet types and scenarios in which a stated feature sets do not provide adequate control plane protection.
For example, based on current day capabilities, a system administrator could construct class maps and policy maps that are specific for control plane packets of known type. Once created, however, these policies would need to be included in the access policy for every interface in a net-
More particularly an internetworking device such as a router typically separates its functionality into control plane functions and data plane functions. The data plane is principally responsible for accepting transit packets at input ports and routing or switching them to output ports. On the other hand, the control plane is responsible for higher layer functions of the device, such as establishing routing tables and entering quality service policies. DoS attacks are thus commonly directed at control plane service functions that reside on route processors such as routers, switches, firewalls and the like, since they are the most likely to cause widespread disruption when they fail. These control plane service functions may include the execution of certain protocols such as a Border Gateway Protocol (BGP), Simple Network Management Protocol (SNMP), route table management, memory management and the like. Because the
55 work. Since there may typically be hundreds, if not thousands of ports in even a modest network, it is not typically feasible for network administrators to deploy and maintain such features everywhere.
Also, when control plane policies are defined within input 60 port features, a significant performance impact typically
results for transit (that is non-control plane) traffic. Because additional control plane classes and policies that need to be executed for transit packets as well as control plane destined packets, overall transit traffic performance is markedly
65 reduced. An interface which previously had no configuration, would be forced to execute control plane policies for every packet it receives. This performance impact, rather
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page9 of 15
US 7,224,668 Bl 3
than help, could thus actually hinder proper operation of currently deployed infrastructure.
For certain packet types that are destined for both the transit and control planes (i.e. special broadcasts, IPv4 option bits, etc.) it is also not possible to set different yet compatible service policies for packets within such a single class. There is for example nothing inherent in such a packet to help understand whether it is destined to the control plane or should be forwarded as a transit packet.
Thus, it is also not typically possible in all cases to configure specific classes to identify all control plane destined packet types, since these packet types cannot be readily identified, and current interface policies cannot be configured to control them efficiently.
SUMMARY OF THE INVENTION
The present invention is a technique for improved immunity to denial of service attacks, and in general, to provide improved Quality of Service (QoS) for network infrastructure equipment.
In one embodiment, an internetworking device or other route processor is composed of two main parts. These include a forwarding path that operates as a data forwarding plane responsible for per packet processing (e.g., forwarding). Above the data plane is a network operating system that is responsible for operations in a control plane. In the case of a device such as a router or switch, the control plane runs routing, signaling and control protocols that are responsible for determining the packet forwarding behavior by the data plane. The control plane in a router, for example, executes routing or switching protocols, manipulates forwarding tables, per flow quality of service tables, access control lists, and the like.
In embodiments of the invention, based upon information acquired through its control plane processes, packet forwarding behavior of the data plane elements is thus dictated. Data planes thus typically otherwise include a plurality of ports that define physical connection points to the network. Port services are then typically applied to operate on packets entering into or exiting from each individual physical port.
4 Embodiments of the invention provide the ability for a
network administrator to prevent denial of service attacks and provide quality of service for control plane packets. A class of packets to be controlled are defined (such as Telnet SYN) and policies are attached to such class. For example, one policy may be to rate limit Telnet SYN packets to a specific rate that is a tolerable rate determined through a specific hardware configuration. The administrator can then apply this limit to the single control plane port, which would
10 limit packets from all ports in the device. A limit command could be applied to the single control plane port rather than modifYing the configuration on all ports.
15
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in
20 which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
FIG. 1 is a block diagram overview of an internetworking 25 device having an aggregate control plane services function.
FIG. 2 is a block diagram of a distributed control plane services implementation.
FIG. 3 is a diagram illustrating how distributed control plane services may affect specific ports, and an aggregate
30 control plane services function may provide top level control.
FIG. 4 is a sequence of steps that may be performed in routing control plane packets from the data plane to the
35 control plane in one environment.
FIG. 5 is an example of a set of rules that may be used to configure aggregate control plane services to rate limit Telnet traffic.
FIG. 6 is a flow chart of the process in a distributed switch
40 engine environment.
In accordance with embodiments of the present invention, the control plane processes are implemented as independently executing processes. However, the control place processes are collectively arranged as a single addressable 45
entity, to provide the ability to better manage control plane traffic.
DETAILED DESCRIPTION OF THE INVENTION
A description of preferred embodiments of the invention follows.
FIG. 1 is a block diagram of a typical internetworking device 100 such as a router, bridge, switch, server or the like in which the invention may be implemented. Such an
More specifically, in embodiments of the invention, a collection of control plane processes are considered to be a single entity that is a uniquely addressable device port. Packets, which are destined to specific control plane processes, are now destined through that specific control plane port, such that such packets intended for the control plane always pass through this designated port. As a result, a set of port services unique to the control plane may be applied to the aggregate control plane port. These control plane port services thus can be ensured to operate on packets entering and exiting each of the control plane processes.
In one embodiment, packets destined to the control plane port can be identified in a number of ways, such as by using information implicit to specific packets (i.e., the recognition
50 internetworking device 100 consists of a nnmber of functional entities. These include line cards 110 that are responsible for physically attaching to network connections such as ports 120. Each of the line cards 110, typically provide a nnmber of ports 120, such as through Network Interface
55 Adaptors. Packets received from the ports 120 are fed to a route processor 125. In the case where the device 100 is a router or switch, the processor 125 includes a central switch engine 130. A control plane 150 associated with the device 100 is defined as a collection of processes, typically running
60 on the route processor 125. These control plane 150 processes collectively provide high level control for most router/switch Input/Output Services (lOS) functions. These control plane 150 processes could be implemented as soft-
of a control plane process address), the result of a routing or switching decision, or by considering other control or configuration information. This allows a route processor to identify candidate packets destined to the control plane port 65
enabling those packets to be processed by the aggregate control plane port services.
ware at any level of a system, or as hardware. As will be understood shortly, the invention herein con
cerns a control plane port 140, defined as a single access path between the switch engine 130 and the control plane 150.
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page10 of 15
US 7,224,668 Bl 5
The control plane port 140 may or may not be a single physical port. For example, it may be a virtual address through which packets travel or are routed from the data plane 135 to the control plane 150.
More specifically, the line cards 110 and central switch engine 130 operate to accept packets received on a given port 120 and route them through to another output port 120. These forwarding or data plane 135 components are thus responsible for forwarding network transit packets.
The control plane 150 on the other hand, functions largely 10
independently of the data plane 135. The control plane 150 is responsible for processing routing, signaling and control protocols that dictate the packet forwarding behavior of the data plane 135. Such protocols typically manipulate forwarding tables 160, per flow Quality of Service (QoS) tables 15
161, access control lists 162, and the like are utilized by the device 100 to make packet forwarding decisions. For example, the control plane 150 might manipulate the forwarding table 160 in the switch engine 130 or change the state of one of the port interfaces 120 in a line card 110 to 20
effect a route change. The control plane 150 is typically not a single process or processor but rather a collection of processes.
The primary goal of Denial of Service (DoS) protection, or otherwise maintaining a specific Quality of Service (QoS) 25
at the control plane 150 is to maintain packet forwarding and protocol states while the device 100 is either under attack or experiencing normal to heavy traffic load. Under these conditions, device 100 should continue to process important packets destined to control plane 150 functions, including 30
protocol control packets, Layer 2 (L2) or Layer 3 (L3) keep alive packets, and the like while at the same time maintaining critical Input/Output Service (lOS) functions.
The central switch engine 130 typically performs high speed Input and Output Services (lOS) for port interfaces 35
such as the line cards 110. An important aspect of the central switch engine 130 is that all packets destined to the control plane 150 must pass through the central switch engine 130 prior to being routed to the functions 155 in the control plane 150. In this instance the central switch engine 130 can be 40
utilized to implement aggregate control plane protection, for all such processes 155 as will be described below.
An alternate arrangement shown in FIG. 2, uses a distributed switching engine architecture. This approach provides high speed switching of packets among specialized 45
distributed line cards 111 typically without utilizing any central switch engine 130 resources. In this architecture, a distributed switch engine 131 may perform input and output services for each respective line card 111. In this instance distributed control plane port services will be utilized to 50
implement the specific aspects of the invention described herein.
Regardless of whether the control plane port services are implemented as aggregate port services 145 or as distributed control plane services 146, they perform certain basic func- 55
tions. Control plane port services most importantly determine if a given packet is destined to a control plane process 150. Such determination can be made through a route look-up mechanism or in other ways. For example, an L2 destination address look-up mechanism may be used for L2 60
port addresses. Alternatively for an L3 port, L3 destination address lookup functions such as Cisco Express Forwarding (CEF) may be used to identify packets destined to control plane processes 155. Both of the look-up mechanisms are able to identifY packets destined for the control plane 150. 65
With processes 155 in the control plane 150 being treated in this way, the control plane port 140 can be treated as a
6 traditional hardware port. As a result, a full range of traditional port control features can be applied to help protect the control plane 150 from a DoS attack, or to provide other QoS. Such control features can, for example, be implemented as a set of progrmed rules that determine whether or not packets arriving at the control plane port 140 qualifY for delivery to the control plane and at what level of QoS.
While this will be described in a detailed specific example below, assume as one example that a system administrator would like to limit packets of type TCP/SYN that are destined to the control plane 150 to a maximum rate of one megabit per second. With the control plane 150 being treated as an addressable single port 140, rules can be established to enforce this rate limit, after port input services are applied to the port 120, and after a switching decision is made in the data plane 130. The rules are applied if and only if a packet has been first determined to have a destination of the control plane 150. The specific control plane feature (i.e., rate limit with access list) can then be applied by the control plane services 145 or 146, thus preventing even correctly addressed packets from progressing up to any of the control plane processes 155 if the specific rate limit has been exceeded.
In some instances, an administrator might employ a more complex set of rules. For example, such rules might also be put in place to allow only a system administrator to access the router through a trusted host address. This allows the administrator to connect to the router, even while it is under attack, since the rate limit and access list would permit the session connected from the trusted host while rate limiting other connectors.
In a similar fashion, important packets such as routing protocol control packets can be placed in appropriate hierarchical queues based on priority as determined by the user. This potentially can improve routing convergence rates. Thus, the user is afforded significant control over the flow of traffic destined to the control plane 150 just as if the control plane 150 were a hardware interface. Since control plane 150 destined packets will invoke only control plane services, transit traffic and system performance is minimally impacted. That is, transit packets will not invoke control plane port services, but will continue to invoke normal input and output port services.
FIG. 3 illustrates how the aggregate control plane services 145 and distributed control plane services 146 can be thought of as providing a hierarchical approach (rings of security) to access control. The central, aggregate control plane services 145 provide a level of service (or control) for all packets received from any port on the device 100. The distributed control plane services 146 provide a level of service (or control) only for those parts with which they are associated, which may be a single port 120 or multiple ports 120. A different level of service may therefore result for ports 1 and 2, serviced by distributed services module 146-1, than for a port 5, which is serviced by a different distributed services module 146-2.
In an implementation such as that shown in FIG. 1, the central switch engine 130 can provide an aggregate level of control planeservice 145, which is applied to all control plane packets received from all interfaces. Central switch engine 130 executes the input port services for the control plane port 140 making routing decisions for packets designated for the control plane 150.
One example is shown in the flow chart in FIG. 4. In a first state 400, a line card detects a packet and delivers it to the central switch engine 130. In a next step 402, the central
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page11 of 15
US 7,224,668 Bl 7
switch engine 130 performs normal input port services and Quality of Service (QoS) processing on the received packet. In a next state 403, the central switch engine 130 performs its normal Layer 2 and Layer 3 switching/routing decision. In the case of a normal transit packet, the packet would be routed to a destination port 120 on an associated line card 110, using for example, the forwarding table information 160. If, however, the packet is destined for a known control plane 150 address, or to an address not on a forwarding table 160, the packet is tagged being destined to as a control plane 10
port. The packet is then routed through the aggregate control plane port 140.
In state 405 the control plane port 140 then performs the aggregate control plane port services on the packet.
In a state 410, based on the results of the aggregate control 15
plane services function 145, the control plane port function will either drop the packet, or mark the packet and potentially deliver it to the control plane 150 for processing.
Class maps and policy maps may be used for both DoS protection and packet quality of service. For the single 20
aggregate port 140 these classifications and policies can be applied to in a known fashion. Consider the control plane services pseudo-example described in FIG. 5. Configuration commands are shown on the left hand side with comments on the right hand side. These types of commands are typical 25
rate limit commands familiar to network administrators. This example is for illustrative purposes only; it should be understood that a whole range of techniques could be used to implement such features.
The particular example limits aggregate control plane 30
services for Telnet type traffic. In a first construction 500, a class map is defined as "telnet-class". These packets are for example identified by matching the telnet access group 140. Telnet access group 140 matches packets with "TCP field" equal to "telnet". In the next definitional statement 502, a 35
policy map is associated with the "control-plane-policy". The next instructions 503 define the policy assigned to the "telnet-class" as allowing 80,000 bits per second of traffic, with excess traffic being dropped. This rate limit definition
8 through the distributed control plane services, and thus through the control plane port 140.
FIG. 6 is a sequence of steps that may be performed to implement the invention in such a distributed control plane environment. In a first state 600 a distributed line card receives a packet delivering it to its associated distributed switch engine. In a next state 602 the distributed switch engine performs normal input port services and quality of service processing.
In state 604 the distributed switch engine performs a Layer 2 and Layer 3 switching routing decision, determining if the packet is destined to the control plane 150. For control plane packets in state 606, the distributed switch engine then performs the distributed control plane services (such as the commands of FIG. 5). In state 608, depending upon the result of those distributed control plane services, the packet is either dropped or marked and potentially delivered to the central switch engine 130.
In a state 610 a central switch engine then performs an aggregate control plane service, for example rate limiting telnet packets and then potentially delivering the packet to the control plane should it pass the aggregate control plane services functions.
In general, it can be determined through the use of route look-up mechanisms for L3 ports such as a Cisco Express Forwarding CEF decision, or a Media Access Control (MAC) layer look-up mechanism for L2 ports, if a given packet or packet stream has the destination of the control plane.
However, candidate packets for control planes services 145 may involve a variety of control packet types that are destined to the control plane 150 even if they do not specifically address the control plane. Most of these control plane destined packets fit into one of three categories. These include:
is then attached to the control plane port by the following statement 505, which assigns the service policy of"controlplane-policy" to the control plane port. Statement 505 represents a control plane port which could be either aggregate 145 or distributed 146. All other commands specified are common and familiar to system administrators.
40 L2 control: These packets include keep alive and control packets for protocols such as HDLC, PPP, FRLMI, ATM control ILMI, x.25 and ISDN call set-up and SDP bpdu.
L3 control: Miscellaneous:
45
Additional attributes of the port services may be defined
May include routing protocol control packets. May include packets destined to an Internet Protocol (IP) address local to a specific processor 100 or miscellaneous packets such as IP options, or special multi-cast broadcast packets, ICMP packets, unroutable packets and so forth.
as access control lists. For example, in statement 506 a trusted address 3.3 .3 .3 is considered and allowed to have any amount of Telnet traffic. Similarly, in statement 507 another address of 4.4.4.4 is defined as trusted. However all other 50
Given that determination, a set of rules is then programmed by a system administrator to determine which packets actually qualifY for delivery to the control plane 150 and at what rates. With the invention the control plane 150 now considered as a uniquely addressable destination port 145, and being forced to be so. The system administrator can
Telnet traffic is rate limited by the final access list command 510.
The above configuration allows trusted host with source addresses 3.3.3.3 or 4.4.4.4 to forward telnet packets to the control plane without rate limit constraints, and all remaining Telnet packets will be policed to the specified rate.
Specifically, only these packets that match the access control list (ACL) are policed. The last ACL statement 512 includes a match for any packet equal to Telnet. The deny ACL statements allow those packet types to skip the policer and therefore would always be forwarded.
In an alternate scenario, a distributed switch engine is used to provide a distributed level of service as per FIG. 2. The distributed switch engine is such that portions may execute on line cards 111, and other portions may execute in a central location to make the routing decision. But all control plane 150 traffic from all ports 120 still passes
55 now access a full range of traditional port based features. These may include access control lists and quality of service features.
The full range of traditional port based features applied to the control plane thus replace specialized control plane
60 protection mechanisms. Examples of such supplanted protection mechanisms include SPD or RPF traditional port services, and other specialized control functions. The control plane can now also utilize the same features to not only maintain security but also guarantee quality of service.
65 Although they have been described herein in connection with L2 and L3 packet processing, these features can span the entire ISO seven layer model.
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page12 of 15
US 7,224,668 Bl 9
With the control plane being treated as a traditional port, rules can be established using the method according to the invention that is enforced after port input services and the switching decision has been made. These rules are supplied if and only if the packet has been first determined to have a destination of a control plane. As a result transit packet throughput performance is minimally affected because control plane port services are applied if and only if a packet is first determined to have a control plane destination.
While this invention has been particularly shown and 10
described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.
What is claimed is: 1. An internetworking device comprising:
15
10 10. A device as in claim 1 wherein the control plane port
services are implemented as distributed control plane port services, and wherein the distributed control plane port services are applied only to the packets received from the specific, pre-determined physical ports.
11. A device as in claim 10 wherein the distributed control plane port services provide additional levels of port services beyond those provided by an aggregate port services function.
12. A device as in claim 10 wherein one or more distributed switch engines provide the distributed control plane port services.
13. A device as in claim 10 wherein one or more distributed switch engines deliver packets to the control plane port.
14. A device as in claim 10 where multiple levels of service are provided through a combination of aggregate and distributed control plane port services, for packets destined to the control plane port. a. a plurality of physical network interface ports, each for
providing a physical connection point to a network for the internetworking device, the ports being configurable by control plane processes;
20 15. A device as in claim 1 wherein a central switch engine
delivers control plane packets to the control plane port. 16. A device as in claim 1 wherein a central switch engine
provides the control plane port services. 17. A device as in claim 1 wherein the services applied to
b. port services, for operating on packets entering and exiting the physical network interface ports, the port services providing an ability to control and monitor packet flows, as defined by control plane configurations;
c. a control plane, comprising a plurality of internetworking control plane processes, the control plane processes for providing high-level control and configuration of
25 the control plane port are selected from the group consisting of Quality of Service (QoS) functions, packet classification, packet marking, packet queuing, packet rate-limiting flow, control, or other access policies for packets destined to the control plane port.
the ports and the port services; 30
d. wherein: 1. a control plane port entity provides access to the
collection of control plane processes, so that a set of control plane port services can be applied thereto;
35 and
ii. the control plane port services operate on packets received from specific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the physical
40 port interfaces and services applied thereto.
2. A device as in claim 1 wherein the control plane processes are accessible through a control plane port on the internetworking device, such that control plane packets originating at a plurality of physical ports and destined to 45 one of a plurality of control plane processes are first processed through the control plane port, rather than to individual control plane processes.
3. A device as in claim 2 wherein packets destined to the control plane port are identified using information implicit to 50 the packets, or information specified in configuration of the internetworking device.
4. A device as in claim 3 wherein the control plane port services are applied after a transit packet forwarding decision is made.
5. A device as in claim 3 wherein Layer 2 control packets are identified and forwarded to the control plane port.
6. A device as in claim 3 wherein Layer 3 control packets are identified and forwarded to the control plane port.
55
7. A device as in claim 1 wherein the control plane 60
processes are distributed across multiple processors. 8. A device as in claim 1 wherein the control plane port
services are implemented as an aggregate control plane function applied to packets received from multiple physical ports on the internetworking device.
9. A device as in claim 8 wherein a central switch engine performs the aggregate control plane port services.
65
18. A device as in claim 1 where in control plane port services are controlled and configured as unique entity, separate from physical port services.
19. A method for processing packets in an internetworking device comprising the steps of:
a. configuring a plurality of physical network interface ports, each port for providing a physical connection point into a network, and the ports being configurable by control plane processes;
b. executing port services on packets entering and exiting the physical network interface ports, the port services for controlling and monitoring packet flows as defined by control plane configurations;
c. executing a plurality of control plane processes, the control plane processes providing high level control and configuration of the ports and port services, and additionally comprising the steps of: i. accessing the collection of control plane processes as
a control plane port entity, so that a set of control plane port services are applied thereto as a set; and
ii. operating on packets received from specific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the individual physical port interface con-figuration and port services applied thereto.
20. A method as in claim 19 wherein the control plane port processes packets originating at a plurality of physical ports, and additionally comprising the step of:
passing packets through the control plane port, rather than directly from the physical ports to individual control plane processes.
21. A method as in claim 20 additionally comprising the step of:
identifYing packets destined to the control plane port using information implicit to the packet or using information specified in configuration of the internetworking device.
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page13 of 15
US 7,224,668 Bl 11
22. A method as in claim 21 additionally comprising the step of:
applying control plane ports services after a transit packet forwarding decision is made.
23. A method as in claim 19 wherein the control plane processes execute as distributed processing across multiple processors.
24. A method as in claim 19 wherein the control plane port services execute as an aggregate control plane function applied to packets received from multiple physical ports.
25. A method as in claim 24 wherein a central switch engine provides aggregate control plane port services.
10
12 control and configuration of the ports and port services, and additionally comprising: i. means for accessing the collection of control plane
processes as a control plane port entity, so that a set of control plane port services are applied thereto as a set; and
ii. means for operating on packets received from spe-cific, predetermined physical ports and destined to the collection of control plane processes in a way that is independent of the individual physical port interface configuration and port services applied thereto.
26. A method as in claim 25 wherein Layer 2 control packets are identified and forwarded to the control plane port.
38. A device as in claim 37 wherein the control plane port additionally comprises means for processing packets origi-
15 nating at a plurality of physical ports, said means further 27. A method as in claim 25 wherein Layer 3 control
packets are identified and forwarded to the control plane port.
28. A method as in claim 19 additionally comprising the step of applying distributed control plane port services only 20
to the packets received from the specific, pre-determined physical ports.
29. A method as in claim 28 additionally comprising the step of:
providing additional levels of port services beyond those 25
provided by an aggregate port services function. 30. A method as in claim 28 wherein one or more
distributed switch engines provide the distributed control plane port services.
31. A method as in claim 28 wherein one or more 30
distributed switch engines deliver packets to the control plane port.
32. A method as in claim 28 additionally comprising the step of:
providing multiple levels of service through a combina- 35
tion of aggregate and distributed control plane port serv1ces.
33. A method as in claim 19 wherein a central switch engine delivers control plane packets to the control plane ~~ ~
34. A method as in claim 19 additionally comprising the step of:
providing control plane port services in a central switch engine.
comprising: means for passing packets through the control plane port,
rather than directly from the physical ports to individual control plane processes.
39. A device as in claim 37 additionally comprising: identifYing packets destined to the control plane port
using information implicit to the packet or using information specified in configuration of the internetworking device.
40. A device as in claim 39 additionally comprising the step of:
applying control plane ports services after a transit packet forwarding decision is made.
41. A device as in claim 37 wherein the control plane processes execute as distributed processing across multiple processor means.
42. A device as in claim 37 wherein the control plane port services execute as an aggregate control plane means applied to packets received from multiple physical ports.
43. A device as in claim 37 additionally comprising: means for applying distributed control plane port services
only to the packets received from the specific, pre-determined physical ports.
44. A device as in claim 43 additionally comprising: means for providing additional levels of port services
beyond those provided by an aggregate port services function.
45. A device as in claim 43 wherein a central switch engine means provides aggregate control plane port ser-
35. A method as in claim 19 wherein the step of applying port services to the control plane port additionally comprises a step of applying services selected from a group consisting
45 VICeS.
of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control plane port.
36. A method as in claim 19 additionally comprising the step of:
configuring the control plane port services as an entity separate from physical port services.
37. A device for processing packets in an internetworking device comprising:
46. A device as in claim 45 wherein Layer 2 control packets are identified and forwarded to the control plane port.
47. A device as in claim 45 wherein Layer 3 control 50 packets are identified and forwarded to the control plane
port.
55
48. A device as in claim 43 wherein one or more distributed switch engines provide the distributed control plane port services.
a. means for configuring a plurality of physical network interface ports, each port for providing a physical connection point into a network, and the ports being 60
configurable by control plane processes;
49. A device as in claim 43 wherein one or more distributed switch engines deliver packets to the control plane port.
50. A device as in claim 43 additionally comprising: means for providing multiple levels of service through a
combination of aggregate and distributed control plane port services.
51. A device as in claim 37 wherein a central switch engine means delivers control plane packets to the control plane port.
b. means for executing port services on packets entering and exiting the physical network interface ports, the port services for controlling and monitoring packet flows as defined by control plane configurations;
c. means for executing a plurality of control plane processes, the control plane processes providing high level
52. A device as in claim 37 additionally comprising the 65 step of:
providing control plane port services in a central switch engine.
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page14 of 15
US 7,224,668 Bl 13
53. A device as in claim 37 wherein the means for applying port services to the control plane port additionally comprises means for applying services selected from a group consisting of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control plane port.
54. A device as in claim 37 additionally comprising: means for configuring the control plane port services as an
entity separate from physical port services. 55. A computer readable storage medium containing
instructions readable by a computer to configure the computer to perform a method for processing packets in an internetworking device comprising:
10
a. configuring a plurality of physical network interface 15
ports, each port for providing a physical connection point into a network, and the ports being configurable by control plane processes;
b. executing port services on packets entering and exiting the physical network interface ports, the port services 20
for controlling and monitoring packet flows as defined by control plane configurations;
c. executing a plurality of control plane processes, the control plane processes providing high level control and configuration of the ports and port services, and 25
additionally comprising the steps of: i. accessing the collection of control plane processes as
a control plane port entity, so that a set of control plane port services are applied thereto as a set; and
ii. operating on packets received from specific, prede- 30
termined physical ports and destined to the collection
14 60. A medium as in claim 55 wherein the control plane
port services execute as an aggregate control plane function applied to packets received from multiple physical ports.
61. A medium as in claim 60 wherein a central switch engine provides aggregate control plane port services.
62. A medium as in claim 61 wherein Layer 2 control packets are identified and forwarded to the control plane port.
63. A medium as in claim 61 wherein Layer 3 control packets are identified and forwarded to the control plane port.
64. A medium as in claim 55 additionally comprising:
applying distributed control plane port services only to the packets received from the specific, pre-determined physical ports.
65. A medium as in claim 64 additionally comprising:
providing additional levels of port services beyond those provided by an aggregate port services function.
66. A medium as in claim 64 wherein one or more distributed switch engines provide the distributed control plane port services.
67. A medium as in claim 64 wherein one or more distributed switch engines deliver packets to the control plane port.
68. A medium as in claim 64 additionally comprising:
providing multiple levels of service through a combination of aggregate and distributed control plane port serv1ces.
69. A medium as in claim 55 wherein a central switch of control plane processes in a way that is independent of the individual physical port interface configuration and port services applied thereto.
56. A medium as in claim 55 wherein the control plane port processes packets originating at a plurality of physical ports, the method additionally comprising:
engine delivers control plane packets to the control plane 35 port.
passing packets through the control plane port, rather than directly from the physical ports to individual control plane processes.
57. A medium as in claim 56 additionally comprising: identifYing packets destined to the control plane port
using information implicit to the packet or using information specified in configuration of the internetworking device.
58. A medium as in claim 57 additionally comprising: applying control plane ports services after a transit packet
forwarding decision is made.
70. A medium as in claim 55 additionally comprising:
providing control plane port services in a central switch engine.
71. A medium as in claim 55 wherein the step of applying 40 port services to the control plane port additionally comprises
applying services selected from a group consisting of Quality of Service functions, packet classification, packet marking, packet queuing, packet rate limiting, flow control, and other access policies for packets destined to the control
45 plane port.
59. A medium as in claim 55 wherein the control plane processes execute as distributed processing across multiple so
72. A medium as in claim 55 additionally comprising:
configuring the control plane port services as an entity separate from physical port services.
processors. * * * * *
Case3:14-cv-05343 Document1-13 Filed12/05/14 Page15 of 15