Under the FCC’s rules, communications service providers, including wireline, wireless, cable, satellite VoIP, and others are required to electronically report through the FCC’s Network Outage Reporting System (“NORS”) significant disruptions to their communications systems that meet specified thresholds based on the area or amount of consumers impacted. The Bureau recommendations show that the FCC analyzes this data, particularly in light of the recent sunny day outages that have garnered publicity.

The Bureau outlined seven best practices to help prevent such outages:

1. Awareness Training: Service providers should make all personnel involved in the operation, maintenance, security, and support of their networks aware of outage risks and the impact of network failures;
2. Required Experience and Training: Service providers should establish a minimum set of work experience and training courses that must be completed before personnel may be assigned to perform maintenance on their networks, especially when the maintenance involves upgrades to new technologies;
3. Access Privileges: Service providers should adopt policies regarding who has access to their networks and procedures for changing and removing access privileges;
4. Network Change Verification: Service providers should adopt procedures for verifying any changes to the operations of their networks before implementation;
5. Network Reconfiguration 911 Assessment: Service providers should assess the impact of any network reconfiguration on 911 call routing before carrying out any changes;
6. Diversity Audits: Service providers should periodically audit the physical and logical “diversity” (e., redundancy) of their networks and take action to ensure continued service in response to uncovered risks; and
7. Network Monitoring: Service providers should actively monitor their networks to enable quick responses to outages and other issues.

1. Access Control: Service providers should limit personnel access to network management support systems that that control a large number of switches, soft switches, or routers;
2. Validation and Authentication: Service providers should implement validation and authentication procedures for any changes that affect call routing, not just changes impacting 911 calls;
3. Software-based Alarming: Service providers should implement software that warns them when a network change is being made that could potentially affect a large number of calls;
4. Enhanced Outage Detection: Service providers should implement traffic measurements or other mechanisms to enable them to detect “silent failures,” where calls are lost but associated equipment continues to operate; and
5. Automatic Re-routing: Service providers should consider implementing automatic re-routing of calls in the event of outages.

The inquiry is focused on what steps the industry has taken voluntarily to combat certain cyber threats. However, the FCC acknowledged that the vulnerabilities addressed by these recommendations remain active threats and sought comment on how to address these concerns and create cyber assurances across the industry. As Chairman Wheeler noted in his June 12 speech, the FCC is open to considering other options if a voluntary, market-driven approach fails to yield measurable, accountable results. The existing best practices were adopted March 2012 by the FCC’s CSRIC III, predecessor of CSRIC IV, to address critical cybersecurity threats, specifically botnets, attacks on the Domain Name Systems (DNS) and Internet route hijacking. CSRIC III also recommended that ISPs implement source-address filtering to prevent attackers from spoofing IP addresses to launch distributed denial of service (DDoS) attacks. In connection with the adoption of the best practices in 2012, several of the largest ISPs participating in CSRIC III committed to voluntarily implementing the recommendations.

Two and a half years later, the FCC’s Public Safety and Homeland Security Bureau is looking to the Internet community, ISPs, consumer organizations and the broader public community for feedback on implementation of the best practices and their overall effectiveness. Stakeholders are encouraged to weigh in on the progress of and any barriers to implementation, discuss any success stories or breakthroughs, evaluate how effective the current recommendations are at mitigating cyber risk, and identify any new alternatives or technologies that could be more effective going forward.

Comments must be submitted to the FCC’s Public Safety and Homeland Security Bureau by September 26, 2014.

The Public Notice stemmed from a review by the Bureau of network outage reports that carriers are required to file. The Bureau stated that it has observed a "significant number" of 911/E911 outages caused by a lack of diversity. Moreover, it notes that these outages "could have been avoided at little expense to the service provider" (emphasis mine). The clear implication is that FCC tolerance for these types of outages will diminish over time.

Follow the link for a discussion of the diversity mistakes highlighted by the Bureau.

The Public Notice identifies the following examples of lack of diversity causing outages:

• Placement of all E911 trunks or ALI links on the same Digital Cross-connect System;
• Placement of all E911 trunks or ALI links on the same DS1 transport facility (which then fails due to a line cut, failure of a control processor or failure of a power supply);
• Failure of E911 due to failure of a single fuse;
• Failure of E911 due to problems with a single Remote Terminal serving a PSAP;
• Failure of E911 due to simultaneous failure of redundant timing cards.

The Bureau cautions carriers to "avoid placing an entire group of 911/E911 trunks or ALI links on one piece of transmission equipment." It also cites the Network Reliabilty and Interoperability Council (NRIC) best practices as a model in several instances in the Public Notice. These are not binding rules, but the Bureau is looking toward them for guidance.

Final note: The Public Notice mistakenly retained a draft date of May 28, 2009 in the footer. Clearly, the Bureau has been concerned with 911 outages for some time.