|
Log
Notes:
2018-12-28 17:11:41 GMT - It has been confirmed that visibility has been restored to the majority of the nodes across the network. Field Operations have been dispatched to assist with recovering
visibility to the few remaining nodes. Engineering is working to systematically review the network alarms on the other nodes and are then performing remote manual resets to individual cards that
remain in alarm. Reinstate times for each card may vary significantly, as such an estimated completion time is not yet available. If cards do not automatically reinstate after remote resets complete,
Field Operations are standing by to dispatch as needed. The Equipment Vendor驴s Tier IV team continues to assist with the resolution
efforts. 2018-12-28 16:00:14 GMT - The Transport NOC has advised that visibility has been restored to the majority of the impacted devices.驴驴The necessary additional
actions are being taken to fully restore the devices at this time including resetting line cards and providing Field Operation dispatches locally where needed to complete the restoration. 2018-12-28
14:26:38 GMT - On December 27, 2018 at 08:40 GMT, CenturyLink identified a service impact on the network. The NOC was engaged and commenced with investigations to isolate the root cause, and Field
Operations were dispatched for additional investigations. Additional investigations confirmed trouble with the equipment vendor. Tier IV Equipment Vendor Support was engaged, and cooperative
troubleshooting on a device in San Antonio, TX commenced. It appeared the device was broadcasting traffic thus consuming capacity, and was therefore isolated from the network, which seemed to
alleviate some impact. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to troubleshoot. Field Operations were
dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL to assist the Equipment Vendor with re-establishing node visibility. Node visibility restoral continued to
progress as multiple CenturyLink teams in cooperation with Tier IV Equipment Vendor Support continued to remove secondary communication channels and apply filters. Additionally, as node visibility
restored, work commenced to review equipment logs and packet captures from the sites to begin restoring client services. On December 28, 2018 at approximately 00:58 GMT, a suspected card issue in
Denver, CO was identified as the root cause of the issue. Filed Operations were dispatched to address the issue, however upon removal of the card it was determined there was no significant
improvements. As restoration efforts continued it was determined some local actions were necessary to assist with node recovery, which delayed the restoral process. These actions included either line
card resets or Field Operations dispatches for local equipment login. Currently, CenturyLink NOC Management has confirmed repair efforts are steadily progressing, and services are incrementally
restoring. Multiple CenturyLink teams and the Equipment Vendor continue to work remotely and locally to recover impacted nodes. 2018-12-28 13:23:07 GMT - The Equipment Vendor and CenturyLink
engineers continue work to apply the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Services
continue to restore in a steady manner as troubleshooting progresses following the recovery of nodes. CenturyLink NOC management remains in contact with the equipment vendor to obtain updates as
restoration efforts continue. 2018-12-28 12:03:18 GMT - CenturyLink NOC Management has advised that repair efforts are steadily progressing, and services are incrementally restoring. Multiple
CenturyLink teams and the Equipment Vendor continue to work remotely and locally to recover impacted nodes. 2018-12-28 10:59:53 GMT - CenturyLink continues to work with the Equipment Vendor to apply
the filters and remove the secondary communication channels. Field Operations and Equipment Vendor dispatches to recover nodes locally remain underway. Client services continue to restore in a steady
manner as troubleshooting progresses following the recovery of nodes. 2018-12-28 10:02:00 GMT - CenturyLink NOC Management reports steady progression of node recovery and restoral of client services.
In addition to the remote node recovery process, Field Operations continue to dispatch and assist the Equipment Vendor with local equipment login. 2018-12-28 08:48:47 GMT - CenturyLink NOC Management
has advised that repair efforts are steadily progressing, and services are incrementally restoring. The Equipment Vendor and CenturyLink engineers continue work to apply the filters and remove the
secondary communication channels at this time. There have been additional restoration steps identified for certain nodes, which includes either line card resets or Field Operations dispatches for
local equipment login, that have impeded the restoration process. Various repair teams are working in tandem on these actions to ensure that services are restored in the most expeditious method
available. Restoration efforts are ongoing. 2018-12-28 07:10:39 GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in
the network continue. Additional information on repair progress will be available from the Equipment Vendor by 07:30 GMT. Information will be relayed as soon as it is obtained. 2018-12-28 05:59:40
GMT - Efforts by the Equipment Vendor and CenturyLink engineers to apply the filters and remove the secondary communication channels in the network continue. The previously provided ETR of 09:00 GMT
remains. 2018-12-28 04:53:20 GMT - CenturyLink engineers in conjunction with the Equipment Vendor驴s Tier IV Technical Support team have identified the
elements causing the impact to customer services. Through the filters being applied and the removal of the secondary communication channels, it is anticipated services will be fully restored within
four hours.驴We apologize for any inconvenience this caused our customers. Additional details regarding details of the underlying cause will be relayed
as available. 2018-12-28 04:08:59 GMT - The Equipment Vendor驴s Tier IV Technical Support team in conjunction with
CenturyLink Tier III Technical Support continues to remotely work to remove the secondary communication channel tunnels across the network until full visibility can be restored, as well as applying
the necessary polling filter to each of the reachable nodes. 2018-12-28 02:53:45 GMT - The Transport NOC has confirmed that cooperative efforts remain ongoing to remove the secondary communication
channel tunnel across the network until full visibility can be restored, as well as applying the necessary filter to each of the reachable nodes. It has been confirmed that both of these actions are
being performed remotely, but an estimated time to complete the activities is not available at this time. 2018-12-28 01:59:36 GMT - Once the card was removed in Denver, CO it was confirmed that there
was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work
with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified
a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes. 2018-12-28 01:00:51 GMT -
Following the review of the logs and packet captures, the Equipment Vendor驴s Tier IV Support team has identified a suspected card issue
in Denver, CO. Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card. 2018-12-27 23:57:16 GMT - The Equipment Vendor is currently reviewing
the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary
communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root
cause. 2018-12-27 22:52:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause
analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility
can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment. 2018-12-27 21:35:55
GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain
additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause. 2018-12-27 21:01:26 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified
a service impact in New Orleans, LA. The NOC was engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Tier IV Equipment
Vendor Support was later engaged. During cooperative troubleshooting a device in San Antonio, TX was isolated from the network as it was seeming to broadcast traffic consuming capacity, which seemed
to alleviate some impact. Investigations remained ongoing. Following the isolation of the San Antonio, TX device troubleshooting efforts focused on additional sites that teams were remotely unable to
troubleshoot. Field Operations were dispatched to sites in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV Equipment Vendor Support continued to investigate the equipment logs
to further assist with isolation. Once visibility was restored to the site in Kansas City, MO and New Orleans, LA a filter was applied to the equipment to further alleviate the impact observed. All
of the necessary troubleshooting teams in cooperation with Tier IV Equipment Vendor Support are working to restore remote visibility to the remaining sites at this time. Tier IV Equipment Vendor
Technical Support continues to review equipment logs from the sites where visibility was previously restored. We understand how important these services are to our clients and the issue has been
escalated to the highest levels within CenturyLink Service Assurance Leadership. 2018-12-27 20:59:24 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans,
LA. The NOC is engaged and investigating in order to isolate the cause. Field Operations were engaged and dispatched for additional investigations. Please be advised that updates for this event will
be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service
resolution, ticket closure, or post incident review, details may evolve.
|
Print 