Technology Procedure Categories
WSC Incident Response Plan
This document documents proactive and reactive procedures relating to major system incidents.
When Was This Document Updated?
Originally written April 28, 2017
Updated May 2, 2017 by Jeremy Nelson
Updated June 28, 2017 by John Dunning
Who Should Read This Document?
Incident: a disruption of the information system environment, including but not limited to:
- Information system failures and loss of service;
- Denial of service;
- Breaches of confidentiality
Chief Information Security Officer (CISO): The Chief Information Officer
Incident Response Team (IRT): NATS staff and functional office employees as may be appropriate for any particular incident. The IRT shall in all cases consist of at least the CISO (CIO), the lead network technician, and the security member of the Administrative Systems team. These standing members of the IRT have 24/7 incident response responsibilities.
Service Catalog: A catalog of services, listed by priority and with service owner and (if applicable), vendor contact information
System Office Information Security Officer (SOISO): The Vice Chancellor for Facilities and Information Technology
Card Brands: The credit card service providers
- Credit Card Brand Incident Contacts:
- Elavon (Processor): https://www.elavon.com/about/contact-us.html
- MasterCard https://www.mastercard.us/en-us/merchants/get-support.html
- Visa https://usa.visa.com/dam/VCOM/download/merchants/cisp-what-to-do-if-compromised.pd
- American Express https://www209.americanexpress.com/merchant/services/en_US/data-securit
- Discover https://www.discovernetwork.com/en-us/business-resources/fraud-security/
Report to WSC CISO
Activate Agency IRP
Localized, minor incidents. Non-critical systems or non-privileged or single level 1 or 2 user accounts.
Incidents affecting critical systems, level 2 privileged user accounts, or non-FERPA protected information; or affecting a quantity of tier 1 or 2 entities, accounts or systems deemed enough to warrant tier 2 escalation.
•Coordinated, distributed attacks
•Any attack which causes Denial of Service
•Internet abuses violating
•Federal/ State law
•Theft of IT devices with storage capabilities
•Service provider outage
Verbally report immediately.
Incidents affecting multiple critical systems, level 3 privileged user accounts, or FERPA protected or personally identifiable information; or a connected set of Tier 1 or Tier 2 incidents, or deemed serious enough to warrant tier 3 escalation.
•Core network outage
•Theft of proprietary information
•Theft or attempted theft of personally identifiable information
•Theft or attempted theft of FERPA protected information
•Unauthorized activity involving a server or host with confidential data or level 3 privileged user account.
Report verbally immediately. WSC CISO shall determine if and when notification of SOISO is warranted.
Incidents impacting or suspected to impact any portion of the PCI scope for campus, including credit card holder data or the cardholder data environment
Immediately escalate to the SOISO as well as State Treasurer and State CISO to coordinate notification of Card Brands
In the event that a Tier 2 or higher incident is deemed to be a cyber-security incident, the IRT shall securely maintain any information collected, generated, or assessed in the course of determining whether an incident is a potential cyber security incident warranting prosecution. Data collection shall focus on identifying who, what, when, where, and the how of an incident. Collected information shall be properly documented and safeguarded. Evidence such as system and network log files, user files, system administrator logs and notes, backup-up tapes, and intrusion detection system logs, alarms or alerts shall be securely maintained and the chain of custody preserved by:
- Ensuring the evidence has not been altered;
- Ensuring the evidence is accounted for at all times;
- Verifying the passage of evidence from one party to another is fully documented; and
- Verifying the passage of evidence from one location to another is fully documented.
If an incident is determined not to be a cyber-security incident, agencies are still required to maintain any evidence and its chain of custody because future incidents may require the previously captured evidence.
Security Incident Evidence File
An evidence file shall be created to record and maintain an inventory of all actions taken, action timestamps and correspondence associated with a security incident.
Notification of FERPA or Personal Information Security Breach
The IRT shall determine if the incident resulted in a breach to a system containing personal information and then notify affected individual as required by the Financial Data Protection and Consumer Notification of Data Security Breach Act of 2006 (Neb. Rev. Stat. §§ 87-801 to 87-807) or other State or Federal regulatory guidelines.
Security Incident Confidentiality
Communication shall be on a need-to-know basis and shall be considered confidential during a security incident investigation. Incident responders are not to share any details with anyone other than the IRT, WSC administration, or the SOISO.
Documentation of information is critical in situations that may eventually involve authorities, as well as provides a historical event of the actions taken to resolve the event. Manually written incident logs are preferable since electronic logs can be altered or deleted. The minimum information that should be recorded is:
- When (date and time) and how the incident was reported, discovered or occurred;
- Who reported or discovered the incident;
- Description of the incident;
- Incident-related tasks and who performed each, and the amount of time spent on each task;
- Individuals contacted regarding the incident; and
- Information system(s), program(s) or network(s) affected.
Reporting and Escalation
WSC employees shall report incident information through the trouble ticket system or, in the event of a suspected breach or serious incident, directly to the CISO. Service owners for impacted services, working with the CISO are empowered and expected to activate this plan for Tier 3 or higher incidents, including the creation of an incident log for the incident.
NATS employees also monitor system state, logs, and events through e-mail and/or SMS notifications provided via the Zenoss event monitoring system. Critical, Alert, and Severe error messages will be responded to on a 24/7 basis.
Agencies should periodically review the incident conditions and determine if escalation to a higher tier is appropriate. An incident may be escalated in any of the following ways:
- Determination by the Chief Information Officer or System Office Information Security Officer;
- Additional related events (i.e. emergence of a distributed, coordinated attack, etc.)
- Requested by WSC administration.
Service owners should consider escalating an incident when certain conditions are met. The following thresholds of incident actions are examples of when to consider incident escalation:
- Multiple machines per LAN segment showing Intrusion Prevention System signature;
- Multiple machines showing multiple Intrusion Prevention System signatures;
- One or more critical infrastructure/application showing Intrusion Prevention system signatures;
- Significant impact on bandwidth;
- When a concerted effort is shown to be attacking the network, either internally or externally;
- Any known or reported compromise of Personal Identifiable Information (PII) or FERPA protected information.;
- Any website defacement or unapproved changes to website deemed detrimental to WSC image or mission.
- Abnormal increases in any of the above
Response to Incidents
Priority in incident response is given to preventing further damage to WSC information systems. Therefore, NATS reserves the right to quarantine any potentially threatening user account or system.
The IRT shall identify containment strategies to control an incident's impact to compromised systems, limit the extent of the incident, prevent further damage and regain normal operations of affected systems. Containment measures should take into consideration available resources, the classification of an incident, the service priority as indicated by the Service Catalog. Containment measures shall also be evaluated against the potential loss or corruption of security incident evidence. Containment methods shall include as a minimum:
- Ensuring redundant systems and data have not been compromised;
- Monitoring system and network activity;
- Disabling access to compromised shared file systems;
- Disabling specific system services;
- Changing passwords or disabling accounts;
- Temporarily shutting down the compromised or at risk system; and
- Disconnecting compromised or at risk systems from the network.
The IRT shall develop and employ mitigation strategies prior to returning compromised systems to service to protect against like or similar types of incidents in the future. Mitigation strategies may include, but are not limited to:
- Changing passwords on compromised systems;
- Disabling compromised accounts;
- Identifying and removing an intruder's access method;
- Installing system patches for known weaknesses or vulnerabilities;
- Adjusting or deploying firewall or intrusion detection system technologies to detect access and intrusion methods; and
- Code changes to internal applications.
The IRT shall evaluate and determine when to return compromised systems to normal operations.
In the event of a failure impacting multiple systems, the IRT shall use the WSC Service Catalog and the inherent priority levels to prioritize system restoration work. Data backups available to the recovery process are governed by the Disaster Recovery Data Retention Procedures document, located here.
Access to compromised systems shall be limited to authorized personnel until the security incident has been contained and root cause mitigated. Analysis and mitigation procedures shall be completed as soon as possible, recognizing systems are vulnerable to other occurrences of the same type. Recovery procedures shall address:
- Recovery Requirements. The IRT shall define and prioritize the requirements to be met before returning an affected or compromised system to normal operations. Recovery strategies may include, but are not limited to:
- Reinstalling compromised systems from trusted backup-ups; and
- Reinstalling system user files, startup routines, or settings from trusted versions or sources.
- Validate Restored Systems. The IRT shall validate the restored systems through system or application regression tests, user verification, penetration tests, and vulnerability testing and test result comparisons.
- Increased Security Monitoring. The IRT shall heighten awareness and monitoring for a recurrence of the incident.
After an incident has been fully handled and all systems are restored to a normal mode of operation, a follow-up analysis should be performed within three to five days of recovering from the incident to discuss actions that were taken and lessons learned. Extended delays may reduce the effectiveness of relating critical information. Follow-up analysis shall include a review of the chronological events, identifying all containment and eradication actions taken, identification of mitigation strategies, examining the lessons learned, and assessing the incident costs. Questions to be addressed may include, but are not limited to:
- Did detection and response systems work as intended? If not, what methods would have prevented the incident?
- Are there additional procedures that would have improved the ability to detect the incident?
- What improvements to existing procedures and tools would have aided in the response process?
- What improvements would have enhanced the ability to contain the incident?
- What correction procedures would have improved the effectiveness of the recovery process?
- What updates to agency policies and procedures would have allowed the response and recovery processes to operate more smoothly?
- How could user and system administrator preparedness be improved?
- How could communication throughout the detection and response processes be improved?
- Was the incident previously identified as a potential threat?
- What was the impact in terms of financial loss, loss of public or customer trust, legal liability, or harm to public health and welfare?
Results of these questions should be documented and incorporated into existing procedures, if necessary.
Incident Response Training
Agencies should provide education and awareness programs for users in incident response procedures and reporting methods. The programs shall address:
- What types of events are incidents;
- Agency notification procedures; and
- Existing and emerging threats.
Agency IT Staff
NATS staff responding to incidents are encouraged to obtain the following training, according to their roles and responsibilities:
- State and Federal security and privacy laws and procedures pertinent to PCI compliance will be provided through the State of Nebraska Treasurer’s office
- SANS Secure the Human training for IT professionals
- Technical training on all platforms, operating systems and applications they may be responding to.
Incident Response Testing
Testing should be conducted at least annually, either in response to an identified incident or as part of a formal readiness test, using defined tests, simulated events, and exercises to determine the effectiveness of the incident response capability.
Release of Information
Control of information during the course of an incident or investigation of a possible incident is very important. Only the affected agency can authorize the release of all incident information. Specific information concerning the incident, such as accounts involved, programs or system names, are not to be provided to any callers regardless of who they claim to be.
WSC reserves the right to amend this document at any time with or without notice.
Last Updated: 6/28/2017