Motivation
From time to time computers served by VT's network are 0wn3d or have copyrighted content. These computer incidences need response by a combination of the IT security office and IT network operations. This process has evolved over many years during the development of global networking. Often times the response takes on a character of the wild west lawman who shoots from the hip.
Unfortunately this process of response has several shortcomings.
- It requires operators or staff that have developed an appropriate sensitivity to and judgement about the right level of response to computer incidences.
- It has the potential to not serve our customer's best interests.
- Depending on who responds, the response may vary.
- The business cost to this type of response is two fold.
- Management can be costly due to inefficient communications channels between the multiple operators who necessarily must handle the incident.
- There is a risk that the incident is not handled quickly enough due to some single person dependencies that currently exist.
- The worst case is we, as the network operators, become the agents of someone else's denial of service attack (as we are denying service when we take down a port as part of incident mitigation.)
A New Response Process
An improved process seems to be indicated. Some qualities of the process are:
- A single well defined communications channel for the management of the incident response. (Our current trouble ticketing system may provide a good basis for this communication channel.)
- No regular dependence on either expert judgement or a single person.
- A clear process that identifies and tracks a well defined mitigation process and timeline.
- Appropriate followup with all parties which ends when the affected computer is cleaned up and all mitigation actions have been un-applied. (I.e. The network has been restored to normal state.)
Details
Each incident should have the following steps or data collection processes:
- Complainants: The complainants and their complaints should be registered. The complainants may also offer logs and other information as evidence. This evidence should be filed separately in the evidence section. Each complainant should be marked with their preference for receiving updates about the response, also some information about the incident may be sensitive and it may not be appropriate to share with all complainants.
- Subject System Operator(s): The administrator or party responsible for the subject system must be identified. In some cases, multiple administrators or parties may be identified. For RESNET, exact information about the operator may not be known (i.e. all roommates in a particular residence hall room.) For non-resnet cases, the initial point of contact is the network liaisons.
- Identity: The identity of the subject system should be verified. Often the complainant will only identify a subject system by IP address or domain name. We need to verify that this address isn't being spoofed (i.e. ARP mapping shouldn't be flapping over recent history.) We cannot have perfect knowledge here.
- Evidence: Evidence may be offered by the complainant as logs. Evidence should also be collected independently by VT IT staff. Evidence should be independently verified. Evidence may take any form including the following:
- logs
- netflow statistics
- vulnerability scans
- connection to suspect system to verify status or verify disclosure of information
- Threat checklist: A determination of the type of threat should be made by VT IT staff. The type of threat AND the type of system (university owned vs. privately owned) will determine the mitigation process and timeline to be followed. NOTE: any mitigation timeline starts from when we are notified of a problem, not from when the determination of type of threat is made.
- Response Status: This includes all the mitigation actions that have been performed and that are currently active. The response to an incident ends when the machine in cleaned up (verification by the subject system operator with perhaps independent verification) and all mitigation actions have been un-applied.
Mitigation
Various actions that can be performed in response to an incident
- Attempt contact of subject system operator
- Resource limiting of subject system
- Network limitation of subject system (as opposed to Resource limiting: Network limitations restrict the reachable end systems by the affected system. E.g. placing someone in a remediation network.)
- Disconnection of subject system
The timeline for mitigation response and the mitigation actions to perform need to be defined for each threat type.
Threats
of sensitive information on subject system. Sensitive information is defined as private information for which the University can be assumed to be the custodian on the subject computer and for which the University has liability for disclosure.
: suspected or potential disclosure of sensitive information on subject system: in this case the system is known to be 0wn3d, but knowledge of whether there is sensitive information on the system is unknown. In some cases sensitive information may be assumed to be on the computer.
complaint
complaint
: in this case, the subject system seems to be taking advantage of a vulnerability for which there exists no patch yet.
: the subject system is operating in a manner which indicates some sort of compromise has happened and probably out of the bounds expected by the system operator. In this case, automated systems may begin mitigation actions in response to the system.
: E.g. Resnet Top Uploaders (chumps)
Leftovers
Some stuff that needs to be moved to other sections.
- When to notify the CIRT team:
- You are contacted by law enforcement.
- Multiple systems appear to have the same compromise.
- Something about the incident may lead to publicity.
- Incidents that may require action from student judicial, personnel, provost, or legal.
- Network liaisons
- Can request to have their own ports disabled.
- Disabled ports may be re-enabled at the request of the network liaison. In many cases, we require that the request to re-enable come from the network liaison. Ticket should have some way to track this.
- Shoot on sight
- Hosts subject to complaint that are using unregistered IP addresses and have no other registrations that would lead to an appropriate contact. Keep off until you hear from a NL.
- Turn off quickly (1hr or less)
- Verified exposure of confidential information, e.g. violation of FERPA.
- Attempt to contact NL first. Follow up and disable if necessary.
- Some tools needed
- MAC address "hot list." Generate some kind of notice if a particular MAC shows up on the network. This could be used in cases where a port has been disabled but the host has been used. Also handy for locating stolen/lost hosts.
- Find a better place to forward CIRT list.
7 Comments
Randy Marchany
Jul 02, 2006The recent brouhaha over SSN disclosures and network disconnections emphasizes the need to get this checklist up asap for the NOC.
Another thing to consider with www exposures is that you may be accessing a file that is on Google Cache and NOT on the VT site. We need to ensure that we don't do a DOS on an entire dept as what happened with ECE last week.
I'll work on a draft checklist tomorrow but would like to propose the following timetable for determining when to disconnect.
1. Once verification is completed successfully
a. offending site admins are notified and given:
i. 8 hours to remove if exposed file is > 4 years old. Offending site is restricted to VT addresses only immediately.
ii. 2 hours to remove if exposed file is < 4 years old. Offending site is restricted to VT addresses only immediately.
2. The ISO was advised by Legal Counsel on this on 6/23/06. These timeframes are acceptable to them.
3. The ISO is drafting a letter to be sent to all DDH advising them to ensure that no exposure leaks exist on their systems.
Unknown User (eholohan)
Jul 31, 2006Interesting wrinkle we encountered today:
Fear of an incident. Employee is fired, NL is afraid $employee will attempt to cause damage. What is our response?
Randy Marchany
Sep 18, 2006Automatic disconnection if complaint is received from REN-ISAC?
We had a discusstion about autmatically disconnecting a system if the complaint came from REN-ISAC. I asked for this exception because:
1. REN-ISAC complaints usually say the host is showing signs of being part of a botnet.
2. Leaving said host on the net for any amount of time AFTER receiving notification endangers the VT net and the Internet.
3. We should contact the system admin/owner asap to tell them of the problem. This is the usual procedure.
Has this procedure been adopted by the NOC?
Randy Marchany
Feb 08, 2007I've uploaded the latest VT Incident Response Checklist doc to this wiki. PLease take a look at it and update it as appropriate. Let me know if you have any questions.
-r.
Randy Marchany
Aug 19, 2008New Trouble Ticket Submission Process - Comments Please
The ITSO is testing a new procedure to reduce the elapsed time between discovery of a compromised host and its resolution. This is our big weakness and we are attempting to document how long it takes to fix a compromised host. We are testing the following procedure:
1. Extract a daily report of compromised VT hosts from the IPS. A compromised host is defined to be a VT system that originates an attack outside of VT. These daily reports are currently at http://candi4.cirt.vt.edu/iss_to_remedy.
2. Look up the NL information and plug it as well as the attack information (IP, Date, #attacks detected, target IP's, attack type) into the online trouble ticket submission system. A couple of points:
a. This is NOT an automated submission to Remedy. A human has final say on the submission. However, there is a high degree of certainty
that the VT host is indeed compromised.
b. DHCP NL's are Phil B. and possibly others. They may not be able to identify the admin or owner of the compromised host. Because the
extracts are done daily, we hope the DHCP info is fresh.
c. We insert an attachment to the trouble ticket that informs the help desk that the ITSO submitted the ticket but the NL or system owner should
be contacted.
3. This trouble ticket is generated ONLY for network based attacks. We remember the sensitivity issues involving sensitive data or criminal investigations. That type of incident is still handled as before.
The BOV is very interested in how long our response capabilities are. Submitting a trouble ticket to Remedy puts the "event" in the system. We've noticed between 8/1/8 - 8/16/8, over 200 VT hosts are the sources of net attacks directed at external sites.
Comments?
Randy
Brad Tilley
Aug 26, 2008IDS triggers from 198.82.0.0/16 can be seen here:
http://candi4.cirt.vt.edu/iss_to_remedy/198index.html
Please note that these are subject to interpretation and should not be deemed 'an attack' on face value.
Randy Marchany
Oct 23, 2008the latest revision of the Incident Response Checklist is here and the appendix contains instructions on how to do Data breach notification. This is timely since VA passed a data breach notification law in 2008.