Modern accelerator and experiment control systems have, over the last few years, been based increasingly on commercial off-the-shelf products such as VME crates, programmable logic controllers (PLCs), supervisory control and data acquisition (SCADA) systems, on Windows or Linux PCs, and on communication infrastructures using Ethernet and TCP/IP.

Despite the benefits that come with this (r)evolution, new vulnerabilities are inherited too: worms and viruses can spread within seconds via the Ethernet cable, and attackers are becoming interested in control systems. Unfortunately control PCs cannot be patched as quickly as office PCs. Even worse, vulnerability scans at CERN using standard IT tools have shown that commercial automation systems often lack even fundamental security precautions: some systems crashed during the scan, while others could easily be stopped or have their process data altered [1].

Several recent incidents in industryhave unfortunately proved that the risks coming from security breaches are no longer fiction, and the resulting consequences can be severe:

• In 2005 "a round of internet worm infections knocked 13 of DaimlerChrysler's US auto manufacturing plants offline for almost an hour, stranding some 50,000 auto workers as infected Microsoft Windows systems were patched" [2].

• In 2006 a nuclear power plant in the US had to be shut down after a "data storm" on its internal control system network stopped the control of two cooling water pumps. The storm was apparently caused by a malfunctioning PLC [3].

• A report in 2007 stated that "US critical infrastructure [is] in serious jeopardy", and that the US "electrical service, transportation, refineries and drinking water are at serious risk from verysimple hacker attacks" [4].

• "Researchers [at the US Sandia National Laboratories] who launched an experimental cyber attack caused a [power] generator to self-destruct…" [5].

Protecting the critical infrastructure

Partially driven by these incidents, but more by the fear of terrorism after 9/11, industry and government authorities have begun to review the consequences of a security incident on the so-called "critical infrastructure". The critical infrastructure is those industrial sectors upon which everyday living depends, such as electricity providers, oil and gas companies, water and waste plants, chemical and pharmaceutical factories, and the transport sector. The demand for critical infrastructure protection has led to a consolidation of worldwide efforts with respect to "control systems cyber security" (CS2) and produced many initiatives, standards, guidelines and regulations [6].

Being ahead of industry, CERN had already developed a thorough security policy for a Computing and Network Infrastructure for Controls (CNIC) [7].Over the last two years the CNIC working group, the IT department's Internet Services group (IT/IS), Communication Systems group (IT/CS), and Fabric Infrastructure and Operations group (IT/FIO), as well as experts from the Accelerators and Beams and Technical Support departments have worked to implement this CNIC Security Policy for Controls [8]. During summer 2007 the CNIC working group took the opportunity to review the security policy and its implementation [9].

This article gives an overview of the implementation, and of tools that will enable system experts, and eventually you, to easily protect control systems against cyber threats.

Step 1: Follow the CERN computing rules [10] to prevent viruses, worms, trojans or attackers creeping onto the CERN site and pursuing their malicious activity.

Security policy

The CNIC Security Policy for Controls gives directives on how to secure CERN control systems. Basically, to obtain maximum security it is necessary to reduce the overall risk to a minimum, where "risk"is defined as:

risk = threat × vulnerability × consequence.

The major part of the "threat" factor originates from outside CERN and cannot be reduced significantly. Thus, protective measures have to be implemented to prevent external threats like viruses and worms or attackers penetrating CERN control systems. These measures should also prevent insiders from unauthorized access – deliberate or accidental.

The "consequences" are inherent to the design of CERN's accelerators and the affiliated experiments. All run a variety of control systems: some of them are complex, some of them deal with personnel safety, and some of them control or protect expensive or irreplaceable equipment. Thus, CERN's assets and their proper operation are at stake. A security incident can lead to loss of beam time and physics data, or, even worse, damage to or destruction of unique equipment and hardware.

The "vulnerability" factor can be minimized (i) by guaranteeing to promptly correct published vulnerabilities; and/or (ii) by adding proactive measures to secure unknown and potential vulnerabilities and those that cannot be fixed. To protect such vulnerabilities against exploitation, either because no patch is available or a patch cannot be applied, the CNIC security policy follows the recommendations of the UK Centre for the Protection of National Infrastructure [11]. Its implementation [9] is based on a "defence-in-depth" approach.

Step 2: Read the CNIC Security Policy for Controls document [8] to get a full view of control systems cyber security at CERN.

Network segregation

To contain and control network traffic, the CERN network has been separated into defined network domains. For example, each of the LHC experiments now uses a dedicated experiment network. CERN's accelerator complex and the control systems that monitor and control the technical infrastructure (e.g. electricity distribution, cooling and ventilation, facility management, and safety and access control systems) use another: the Technical Network. Additional network segregation enables a system expert to further protect vulnerable devices like PLCs [1].

Administrators have been assigned to take full responsibility for each domain. In particular, they supervise the stringent rules for connecting devices to it (all devices must be registered in the CERN network database, see http://network.cern.ch). Several campaigns have been conducted to ensure that the information stored for the Technical Network is correct.

Each domain is interconnected with CERN's General Purpose Network, used for campus and office computing, and with other domains via dedicated network routers. The traffic crossing any two domains is restricted to a minimum by the use of routing tables, and only mandatory traffic can pass such boundaries. While more than 30,000 devices could communicate with the Technical Network before the CNIC implementation, today slightly less than 1000 can do so. Much of this traffic is either dedicated data exchange with CERN's Computer Centre, or is inevitable at the moment due to the ongoing commissioning of the LHC accelerator. This will have to cease in the future, however, if CERN does not want to risk incidents similar to those that have already happened in the industrial sector.

Windows Terminal Servers (WTS), and to a lesser extent Linux-based application gateways, have become the only means for remote user access.

Step 3: Ensure that your control systems are entirely connected to the Technical Network or an experiment network if they pose a risk to CERN's operation or assets when they fail. Such a domain offers them basic protection against threats on the campus network.

Step 4: Use "control sets" to protect your PLCs. These provide additional protection from malicious traffic on a domain.

Step 5: Make sure that the information in the network database (http://network.cern.ch) is correct and up to date. This will make it easier for the domain administrator and for CERN's Computer Security team to handle security incidents; for example, to contact you if necessary.

Step 6: If you need access from your office to your control systems on a domain, avoid taking a direct route and instead use the application gateways. This prevents threats spreading from your office computer to the domain.

Central installation schemes

Experience at CERN has shown that office PCs that are centrally managed and patched are much less likely to be infected or compromised than office PCs that are maintained by individual users. However, central installation methods were at that time not suited for control PCs, as control PCs cannot be rebooted without coordination with the corresponding system expert. Control PCs cannot be switched off or rebooted without proper scheduling by the corresponding system expert. Also, patches cannot be applied easily as controls applications may either not be compliant with a particular patch, or software licences to run controls applications may become invalid.

Two separate PC management schemes for Windows and Linux have been implemented by the IT/FIO and IT/IS groups: Linux For Controls (L4C) for CERN Scientific Linux 3 and 4, and the Computer Management Framework (CMF, see http://cern.ch/cmf) for Windows XP and Windows Server 2003 [12].

Both schemes give control system experts the flexibility to manage their control PCs in a centralized manner, i.e. using a centralized installation scheme. While the standard operating systems, applications and their patches are provided centrally by the corresponding IT service providers, it is up to the control system expert to configure a PC, schedule external interventions (e.g. upgrades or patches), and validate changes before they are applied to PCs that control sensitive equipment. Such a scheme can also make it easier to re-establish and recover a system after a security incident. By passing the flexibility to the experts they take responsibility for securing their own PCs too.

Today CMF is used widely for control PCs, such as for the Windows consoles in the CERN Control Centre, and has been accepted by the controls community at CERN. It has become the standard way to install any Windows PC at CERN, and can additionally be used to configure Windows Terminal Servers. L4C, on the other hand, is used extensively to set up local Linux computing farms for the LHC experiments. For example, at the moment CMS is configuring about 1000 servers through L4C. LHCb uses L4C to install their diskless systems.

Step 7: Make sure that all your PCs arefully patched, the local firewall is enabled, and that the anti-virus program is kept up to date.

Step 8: Use CMF and/or L4C to manage your control PCs. Both provide flexibility and offer you the mandatory security.

Authentication and authorization

Although authentication and authorization is an important pillar in the "defence-in-depth" approach, CERN has previously lacked a coherent solution for control systems, office PCs and web applications. The recent single sign-on project addresses these issues (see "Single sign-on facilitates authentication at CERN").

The Accelerators and Beams department's Controls group has implemented role-based access control [13] for the accelerator controls and operations; user credentials are needed for authentication (i.e. to identify the user), while role assignments define the authorization level (e.g. which actions that user can take). ATLAS and CMS are following a similar route. Furthermore, ALICE has implemented a solution based on Windows credentials using SmartCard technology [14], a direction that has been tested by the IT/IS group. In a broader approach, IT/IS is deploying a CERN-wide Single Sign On portal with a CERN certification authority (http://login.cern.ch).

Step 9: Use passwords with sufficient complexity; avoid using obvious words.

Step 10: Never tell anybody your password, not even the Computer Security team, and do not write it down.

User training

A series of awareness campaigns and training sessions have been held forusers, operators and system experts at CERN. Monthly CNIC meetings provide a forum for questions and discussions (http://indico.cern.ch/categoryDisplay.py?categId=691). Additional dedicated discussions have been held with many system experts (see the CNIC TWiki for details: https://twiki.cern.ch/twiki/bin/
viewauth/CNIC/WebHome
). The TWiki also provides minutes of discussions with users outside CERN, such as other high energy physics laboratories, and with major players in industry. There are links to major standards and guidelines.

Furthermore, CERN has raised aspects of control system cyber security at several conferences and workshops, such as the CS2/HEP workshop [15]; interacted with major vendors of control systems; and is now leading the European Information Exchange on SCADA Security (EuroSCSIE) with members from European governments, industry and research institutions that depend on and/or whose responsibility it is to improve the security of SCADA and control systems.

Step 11: If you have any questions, join the CNIC users exchange meetings held every last Thursday of the month, or contact CNIC-Coordination@cern.ch.

Incident response and system recovery

Even with a stringent security policy, incidents can never be prevented completely. Therefore incidents on a domain have been and will be handled jointly by CERN's Computer Security team and the appropriate domain administrator. If the domain administrator cannot be reached, the acting computer security officer, after receiving approval from the acting IT department head, has the right to take appropriate action in a justified case of emergency.

After an incident has been analysed, the CMF and L4C central installation schemes enable the appropriate system expert to recover the system rapidly.

Step 12: If you follow steps 1–11 the domain administrator or Computer Security team may never need to contact you.

Personal responsibility

The continuing integration of common IT technology into control systems means that IT security vulnerabilities and cyber attackers end up threatening control systems and, therefore, CERN's operation and assets. However, control systems require a different approach to security to that which is appropriate for office systems.

The CNIC Security Policy for Controls document presents a thorough set of rules to secure CERN's control systems. The implementation uses a "defence-in-depth" approach that is based on network segregation, central installation schemes, authentication and authorization, user training, incident response and system recovery, and security auditing. However, each user plays a major role in protecting CERN's assets and operations, so please make sure that you have reviewed all the steps mentioned above.

References

1 S Lüders 2005 "Control systems under attack?" ICALEPCS.
2 "Zotob, PnP worms slam 13 DaimlerChrysler plants" 2005 www.eweek.com/article2/0,1759,1849914,00.asp.
3 "'Data storm' blamed for nuclear-plant shutdown" 2006 www.securityfocus.com/news/11465.
4 "US critical infrastructure in serious jeopardy" 2007 www2.csoonline.com/exclusives/column.html?CID=32893.
5 "Sources: Staged cyber attack reveals vulnerability in power Grid" 2007 http://edition.cnn.com/2007/US/09/26/
power.at.risk/index.html
.
6 S Lüders 2007 "Control system cyber-security in industry" CS2/HEP Workshop, Knoxville, TN, US 14 October.
7 U Epting et al. 2005 "Computing and Network Infrastructure for Controls" ICALEPCS.
8 CNIC WG, CNIC Security Policy for Controls (revised version) EDMS No. 584092.
9 S Lüders et al. 2007 "Update on the CERN Computing and Networking Infrastructure for Controls (CNIC)" ICALEPCS.
10 CERN Human Resources 2000 "CERN operational circular no 5: use of CERN computing facilities" October http://computingrules.web.cern.ch/ComputingRules/OC5_english.pdf.
11 CPNI 2006 "Good practice guidelines 1–7" www.cpni.gov.uk/Products/guidelines.aspx.
12 I Deloose 2006 "The evolution of managing Windows computers at CERN" HEPix.
13 S Gysin et al. 2007 "Role-based access control for the accelerator control system at CERN" ICALEPCS; K Kostro et al. 2007 "Role-based authorization in equipment access at CERN" ICALEPCS; and A Petrov et al. 2007 "User authentication for role-based access control" ICALEPCS.
14 P Chocula 2007 "Cyber security in ALICE" ICALEPCS.
15 Control System Cyber Security Workshop CS2/HEP, 14 October 2007, chaired by S Lüders, Knoxville, TN, US http://indico.cern.ch/conferenceDisplay.py?confId=13367.