Run IX

General

This tree will contain information pertaining to run 9.

Run preparation meetings are held at the usual time i.e. on Friday between 3-5 PM (room reserved , will try to keep to one hour weekly). The following groups are invited to join:

• The S&C core support as appropriate
• The online "Run Time System" representatives
  • DAQ - Jeff Landgraf
  • Slow Control - Yury Gorbunov
  • Trigger - Jon (Jack) Engelage
• All software coordinators as listed on the Organization page

The goal of the meetings are to discuss any issues with the infrastructure, networking, code readiness, resource and associated needs as well as any other computing related issues relevant to the smooth running of online operations. The forum and meeting also serves as a vehicule for passing information on time constraints and requirements to the diverse groups in a structured and cohesive manner.

Related documents

None so far.

Related meetings

• 08W40 You do not have access to view this node
• 08W41 You do not have access to view this node
• 08W43 You do not have access to view this node
• 08W45 You do not have access to view this node
• 08W46 You do not have access to view this node
• 08W48 You do not have access to view this node
• 09W01 You do not have access to view this node
• 09W02 You do not have access to view this node 
• 09W04 You do not have access to view this node

Run VIII

General

This tree will contain information pertaining to run 8.

Run preparation meetings are held on Friday between 3-4 PM (room reserved up to 5 PM). The following groups are invited to join:

• The S&C core support as appropriate
• The online "Run Time System" people
  • DAQ - Jeff Landgraf
  • Slow Control - Will Waggoner
  • Trigger - Jon Engelage
• All software coordinators as listed on the Organization page

The goal of the meetings are to discuss any issues with the infrastructure, networking, code readiness, resource and needs or any other computing and related issues relevant to the smooth running of online operations. The forum and meeting also serves as a vehicule for passing information on time constraints and requirements to and through the diverse group in a structured and cohesive manner.

In Run VII, the forum was used to discuss the security plan and several key reshape of the online computing structure to achieve minimum cyber-security accreditation.

Related documents

• SSH Key Management

Related meetings

• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node

Run VII

Background

Facing a new paradigm of introducing CyberSecurity DOE regulations into our infrastructure, several action items were presented at the 2006 run critique meeting. The presentation is attached below as STAR-Critique-06.pdf (see below). The urgent and immediate items, some of which requiring deep restructuring, were:

• We MUST establish an internal controlled perimeter to the unroutable network. This network will be accessible via a gatekeeper model. Vulnerable devices should be isolated to the internal network layer
• All network and communication layers must be documented
• Physical access to console were describe as part of the Shift procedure and shit alternance. Access to the online computing infrastructure MUST be controlled
• All systems MUST be re mediated and brought up to the proper level of OS version and safety 
  • shall exceptions be needed, the device should have the proper control and monitoring
  • isolation in the private network of node we cannot upgrade due to operational-need is the other solution
• OS flavor reduction – We propose to reduce the OS flavors to enhance and optimize support and maintenance
• Group account access should be regulated via keys (ssh keys) and tight to indivdiuals (no a floating password without a clear understanding of who has it)
• root access shall be restricted
  • A list of users having root access MUST exists at any point in time. In other words, only a few (documented) users should have root access privileges.
  • We must provide best effort to implement a configuration management strategy i.e. how changes occurs in our infrastructure shall follow a procedure and lead to an updated documentation.
• Maintenance of computing equipment will be the responsibility of the S&C, DAQ and Slow Control groups as appropriate under general guidance of the S&C group.

The run preparation will be established within the following guidelines

• General
  • Assess hardware replacement and cost (display, printer, UPS, switches, ...)
  • Assess sub-system needs for resources (disk space, bandwidth, database access, ...)
• Networking 
  • Understand and reshape the current online Network spaghetti to a two layer model, with a gatekeeper model
  • Isolate vulnerable devices on a private network
  • Provide easer a routing or gatekeeper model ; reduce dual or tri-NIC connections
  • Patch all vulnerable machine and bring all equipment to appropriate level
• Organizational needs – root access and password 
  • Establish a in-principle layer of responsibility and accountability
  • Determine root access and generic account access and usage
  • Provide infrastructure to manage keys as a function of nodes machine
  • Document procedure and equipment, establish principles for configuration management
  • Require for new equipment to comply with baseline control
    • New equipment shall not be brought randomly but integrated as part of the online infrastructure documentation
• Software
  • Deploy a new Web server
  • Revisit all online common tools and needs – RunLog, ShiftLog, Web interfaces ...
  • Introduce technology and paradigm change for HTML-refresh poor-man's job approach
    • technique has spread and creates heavy load
  • Review Pplots needs and coverage
  • Introduce Scaler monitoring tool
  • Revisit Ganglia monitoring with special care on broadcast/multi-cast
• Establish a first testbed of database consolidation for high-luminosity regime 
  • With help from Slow Control – IRMIS project

Understanding our online Network

The following table is a first cut to understanding the inter-connections between online hardware.

• ch2connect.xls shows the NFS mounts between machines
• Network-top level.pdf is a rough first cut of the network schematic

Patching and OS version-ing

• July 28^th 2006 
  • The matrix Old_Linux.pdf displays the list of nodes requiring attention
  • Two Windows machines (Alexei's Lebedev responsibility) require immediate attention.

Related meeting

• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node

Run 19

Feedback from software coordinators

Active feedback

Other software coordinators

Run 20

Status of calibration timeline initialization

In RUN: EEMC, EMC, EPD, ETOF, GMT, TPC, MTD, TOF
Test: FST, FCS, STGC (no tables)
Desired init dates where announced to all software coordinators:

- Geometry tag has a timestamp of 20191120
- Simulation timeline [20191115,20191120[
- DB initialization for real data [20191125,...]

     Please initialize your table content appropriate yi.e.
sim flavor initial values are entered at 20191115 up to 20191119
(please exclude the edge),  ofl initial values at 20191125
(run starting on the 1st of December, even tomorrow's cosmic
and commissioning would pick the proper values).

Status - 2019/12/10

EMC  = ready
ETOF = ready - initialized at 2019-11-25, no sim (confirming)
TPC  = NOT ready [look at year 19 for comparison]
MTD  = ready
TOF  = Partially ready? INL correction, T0, TDC, status and alignement tables initialized
EPD  = gain initialized at 2019-12-15 (!?), status not initialized, no sim

EEMC = ready? (*last init at 2017-12-20)
GMT  = ready (*no db tables)

Status - 2019/12/09

EMC  = ready
ETOF = ready? initialized at 2019-11-25, no sim
TPC  = NOT ready
MTD  = ready
TOF  = NOT ready
EPD  = gain initialized at 2019-12-15 (!?), status not initialized, no sim

EEMC = ready? (*last init at 2017-12-20)
GMT  = ready (*no db tables)

Software coordinator feedback for Run 20 - Point of Contacts

Run 21

Status of calibration timeline initialization

Status - 2020/12/10

Software coordinator feedback for Run 21 - Point of Contacts

---

Run 22

Status of calibration timeline initialization

Software coordinator feedback for Run 22 - Point of Contacts (TBC)

---

Run X

Below are the related meetings:

• You do not have access to view this node
• You do not have access to view this node

Run XI

• You do not have access to view this node

Run XIII

• Meeting minutes
• Database initialization check-list
• Calibration requests POC
• Online Monitoring POC

Preparation meeting minutes

• Preparation meeting #1 (this included a summary of the You do not have access to view this node)
• Preparation meeting #2 (this meeting was informal, meeting #3 bundled two weeks progress)
• Preparation meeting #3 (in meeting #4 announcement)
• Preparation meeting #4  (minutes were not sent)
• Preparation meeting #5
• Preparation meeting #6
• Preparation meeting #7
• Preparation meeting #8
• Preparation meeting #9
• Preparation meeting #10 (informally met with a few team members, no formal meeting)
• Run support meeting, #11
• Run support meeting, #12
• Run support meeting #13 (minutes were not sent - all action items followed-up)
• Run support meeting #14
• Run support meeting #15
• Run support meeting #16

Database initialization check list

TPC Software  – Richard Witt          NO
GMT Software  – Richard Witt          NO
EMC2 Software - Alice Ohlson          Yes
FGT Software  - Anselm Vossen         Yes
FMS Software  - Thomas Burton         Yes
TOF Software  - Frank Geurts          Yes
Trigger Detectors  - Akio Ogawa       ??
HFT Software  - Spyridon Margetis     NO (no DB interface, hard-coded values in preview codes)

Calibration Point of Contacts per sub-system

                Coordinator           Possible POC
                ------------          ---------------
TPC Software  – Richard Witt          
GMT Software  – Richard Witt          
EMC2 Software - Alice Ohlson          Alice Ohlson  
FGT Software  - Anselm Vossen         
FMS Software  - Thomas Burton         Thomas Burton    
TOF Software  - Frank Geurts          
Trigger Detectors  - Akio Ogawa       
HFT Software  - Spyridon Margetis     Hao Qiu

Online Monitoring POC

Run XIV

• Meeting minutes
• Pre run-preparation notes (basic checks, annoucements)
• Database initialization check list
• Calibration Point of Contacts per sub-system
• Online Monitoring POC

Preparation meeting meetings, links

• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node
• You do not have access to view this node

Notes

• 2013/11/15
  • Info gathering begins (directories/areas and Point of Contacts)
    Status:
    2013/11/22, directory structure, 2 people provided feedback, Renee coordinated the rest
    2013/11/25, calibration POC, 3 coordinators provided feedback - Closed 2013/12/04
    2013/12/04, geometry for Run 14,
     
  • Basic check: CERT for online is old if coming from the Wireless
    Status: fixed at ITD level, 2013/11/18 - the reverse proxy did not have the proper CERT
• 2013/1125
  • Send the request to software sub-system cooridnators for calibration POC for Run 14, deadline December 2nd.

Database initialization check list

Calibration Point of Contacts per sub-system

Online Monitoring POC

Run XV

Run 15 was preapred essentiallydiscussing with indviduals and a comprehensive page not maintained.

Run XVI

Notes

Online Monitoring POC

Calibration Point of Contacts per sub-system

Database initialization check list

Background as of fall 2009

The network layout at the STAR experiment has grown from a base laid over ten years ago, with a number of people working on it and adding devices over time with little coordination or standardization.  As a result, we have, to put it bluntly, a huge mess of a network, with a mix of hardware vendors and media, cables going all over the place, many of which are unlabelled and now buried to the point of untraceability.  We have SOHO switches all over the place, of various brands, ages and capabilities.  (It was only about one year ago all hubs were at least replaced with switches, or so I think – I haven’t found any hubs since then.)  There are a handful of “managed” switches, but they are generally lower-end switches and we have not taken advantage of even their limited monitoring capabilities.  (In the case of the LinkSys switches purchased one year ago, I found their management web interface poor – slow, buggy and not very helpful.)

In addition to the general messiness, a big (and growing) concern has been that during each of the past several years, there have been a handful of periods of instability in the starp network, typically lasting from a few minutes to hours (or even possibly indefinitely in the most recent cases which were resolved hastily with switch hardware replacements in the middle of RHIC runs).   The cause(s) of these instabilities has never been understood.  The instabilities have typically manifested as slow communications or complete lack of communication with devices on the South Platform (historically, most often VME processors).  Speculation has tended to focus on ITD security scanning.  While this has been shown to be potentially disruptive to some individual devices and services, broad effects on whole segments of the network have never been conclusively demonstrated, nor has there been a testable, plausible explanation for the mechanism of such instability. 

The past year included the two most significant episodes of instability yet on starp, in which LinkSys SLM 2048 switches (after weeks or months of stability) developed problems that appeared to be similar to prior issues, only more severe.  The two had been purchased as a replacement (plus spare) for a Catalyst 1900 on the South Platform.  When the first started showing signs of trouble, it was replaced by the second, which failed spectacularly later in the run, becoming completely unresponsive through its web interface and pings, and was only occasionally transmitting any packets at all, it seemed.   (After all devices were removed, and the switch rebooted, it returned to normal on the lab bench, but has not been put back into service.)

At this point, all devices were removed from the LinkSys switch and sent through a pair of unmanaged SOHO switches, which themselves each link to an old 3Com switch on the first floor.  Since then, no more instabilities have been noted, but it has left a physical cabling mess and a network layout that is quite awkward.  (And further adding to the trouble, at least one of the SOHO switches has a history of sensitivity to power fluctuations, every once in a while needing to be power-cycled after power dips or outages. 

In addition, there have been superficially similar episodes of problems on the DAQ/TRG network, which shares no networking hardware with the starp network.  As far as I know, these episodes spontaneously resolved themselves.  (Is this true?)  Speculation has been on “odd” networked devices (such as oscilloscopes) generating unusual traffic, but here too there is no conclusive evidence of the cause.  Having no explanation, it seems likely this behavior will be encountered again.
 

Core components

There are several “core” pieces currently.  Core is defined somewhat vaguely as connecting lots of devices or requiring relatively high performance: 

1.    ITD’s main switch in the DAQ room
2.    DAQ’s event builder switch in the DAQ room
3.    the starp switch on the South Platform
4.    the DAQ/TRG switch on the South Platform
5.    the Force 10 switches for the HPSS network in the DAQ room

It seems likely that any reshape will have to include those same core components, though perhaps some combinations are possible at the hardware level using VLANs or other technologies.  (combining starp and DAQ/TRG on the platform on to a single large switch, for instance)
 

ITD's Catalyst chassis in the DAQ room (subnets 60, 162, wireless and possibly others in 1006)

This switch chassis is in the networking rack in the northwest corner of the DAQ room.  It is managed by ITD.  STAR has no way to interact with this switch at the software/configuration level.  

Slot 1:  WS-X4013 (Supervisor II Engine, fiber uplink to 515 and local management port)

Slot 2:  WS-X4548-GB-RJ45 (48 1Gb/s copper ports @8:1 oversubscription)  port 43 is 162 subnet, rest are subnet 60.

Slot 3: WS-X4232-RJ-XX (32 copper 100 Mb/s) plus a WS-U5404-FX-MT daughter card with 4 MTRJ fiber ports at 100Mb/s)

Slot 4: WS-4148-RJ (48 copper 100Mb/s) - mix of subnets 60 and 162?

Slot 5: WS-4148-RJ (48 copper 100Mb/s)  - all subnet 60?

Slot 6: WS-X4306-GB (6 GBIC (not mini!) ports, 3 of which have 1000-SX modules with SC connectors)

Images and miscellaneous files

Here we can keep miscellaneous files documenting the state of the network.

First, I have attached an image showing the current (late 2009/early 2010) switch layout and links in the WAH. ("WAH_switches.pdf")

Then there is an "after" picture with a rough idea of the patch panel placement to replace most of the unmanaged switches. ("WAH_patch_panels.pdf")

For the South Platform, a more refined patch panel plan was put together in June 2010 ("Network Plan for South Platforms.doc")

There is an attachment with general guidelines for installing UTP ("Cat5e_Network_cable.ppt")

Locations needing network access

WAH: (starp and DAQ/TRG devices are scattered throughout these locations.  I am going to use the term “satellite racks” to include all locations within the C-AD PASS system that are NOT on the South Platform.  Also, note that the satellite racks are semi-mobile, and the entire detector platform (North and South) can move into the Assembly Building.):

-    PMD racks: ~3 devices on starp and ~3 on DAQ/TRG

-    FMS/FPD east side:  Handful of devices on DAQ/TRG and on star

-    Southwest corner work area: rarely more than two systems here, but might want starp, “trailers” and DAQ/TRG networks here for use as needed

-    EEMC racks, west side:  Handful of devices on DAQ/TRG and on starp

-    FPD/FMS west racks:  Handful of devices on DAQ/TRG and on starp

-    PP2PP east and west:  at least one VME processor on DAQ/TRG on each side - these are in the RHIC tunnel, technically not in the WAH.

-    South platform – (IMPORTANT NOTE:  The south platform must remain electrically isolated from the rest of the facility – there can be no conducting cables running from the South Platform to other locations)
o    First floor:  Three rows of 8-9 racks each (volatile, in that subsystems and components are installed or removed each year)
o    Second floor:  Three rows of 8-9 racks each (volatile)

-    North platform:  currently unoccupied, but has had devices in the past and a switch on the starp network is still present there, with a fiber link back to the South Platform (somewhere!)

Control Room:
-    Perimeter (~3 dozen PCs), almost all on starp, but
o    2-3 on DAQ/TRG
o    4-5 on C-AD 108
o    1-2 on C-AD 90 network?.
o    Numerous small unmanaged switches in this room currently

DAQ Room:  (Highest performance of the entire facility is needed in rack row DA, including a minimum 56-port switch with non-blocking/line rate 1Gb inter-links on the DAQ/TRG network)

-    three “rows” plus two networking racks:
o    the “old” network rack and the “new network rack” near the northwest corner
o    rack row “DA” on west side (nearest the Control Room)
o    shelf row in middle with a racks at each end.

• Northern-most rack is ~20 nodes on “starp” – current rack has at least three unmanaged 8-port switches.
• Remainder of row is primarily DAQ/TRG with 3-4 starp nodes - both netwokrs go through two unmanaged switches in the rack immediately to the south of the shelves.

o    East row:  ~6 stand-alone starp servers (one of which has a DAQ/TRG connection as well), along with a handful of VME devices on starp.  DAQ or trigger might have a device or two here.  The rack space is primarily occupied by devices on a C-AD network.

GMR:
- 3 PCs – generally stable area.

Clean room:
-    several jacks needed, network use may vary between starp, daq/trg and the 130.199.162 subnet depending on the active use at any time

1006C and 1006D (trailers):
    - typically only subnet 130.199.162 is needed here.


 

Meeting notes for week of Oct. 19, 2009

Online network reshape notes from the week of Oct. 18, 2009

During this week, three meetings were held to discuss the STAR online networking reshape plans.

The first meeting included Jeff Landgraf, Wayne Betts, Dan Orsatti (ITD) and Frank Burstein (ITD).  At this meeting the ITD network engineers presented two proposals for core network components based on information previously provided to them by STAR.  The two options were Force-10 based and Cisco-based, with costs of approximately $150,000 and $100,000 respectively.  They included a shared infrastructure for the DAQ/TRG and STARP networks, including a switch redundancy in the DAQ room to handle the two networks and meet DAQ’s relatively high performance needs in the DAQ room.  These ITD options are generally smart, expandable, highly configurable and well-supported by ITD, and meet the initial requirements.

However, in informal discussions since then, Bill Christie suggested that we should consider the possibility of radiation damage and/or errors in any electronic equipment in the WAH.  While this had been mentioned as a possibility in the past, it was not generally taken seriously by those of us in STAR looking after the networks.  Nor is there any way for us to test this to a standard of “beyond reasonable doubt” (or any other standard really).  At Bill’s suggestion, we (Jeff L., Wayne B., Jack E., Yuri G. and Bill C.) met with three members of  C-AD’s networking group, who stated they were certain that radiation could impair switches and strongly suggested that ITD’s suggested equipment was inappropriate for a radiation area.  They also provided some feedback from individuals at two other laboratories that networking equipment in radiation areas are subject to upsets, with one explanation for effects on metal-oxide semiconductors, which at face value would suggest that newer (thus generally smaller) electronic components would be less susceptible, however my intuition is that smaller electronics are denser, and more easily upset by smaller deposited charge, and thus might be more susceptible. 

Here are excerpts from the other labs:

From JLab:  "The flash memory loses its ability to hold data, making it
useless. We have worked around the problem by pulling cable or fiber
back to lower radiation areas wherever we can. Because we made these
cabling changes when we were only using cisco fixed-configuration
100Mbit switches ( 29XX models), I have no data for Gigabit switches.
Since our experience is that it's the flash memory that fails, I'd
expect no better performance from any other switches. All of our
switches that use modular supervisor modules are outside of radiation
areas."

From FermiLab:  "The typical devices used employ metal oxide
semiconductors and the lock up happens when ionizing radiation is
trapped in the gate region of the devices. We see this happen at our two
detectors (CDF and DZero) when losses go up and power supplies circuits
latch up. The other thing working in the positive direction is that when
IC feature sizes go down, there is less likelihood for the charge to get
trapped so they are more radiation tolerant. Having said all that I
can't answer your specific question because we don't put switches or
routers in the tunnel at all."

All this said, the general consensus was that we should move as much “intelligence” as far away from the beam line as reasonably possible.  (Until now, the “big” switches on the platform have actually been about as close to the beam line as possible!)  This means putting any switches in rack rows 1C.  Given both the cost and the radiation concern, we (the STAR personnel) agreed to investigate less expensive switches than ITD’s suggestion, while trying to provide some level of intelligence for monitoring.  We also have a consensus that the DAQ/TRG and STARP networks should try to use common hardware whenever possible, and that we should work to remove as many SOHO-type unmanaged switches as possible as time permits (replacing them with well-documented and labelled patch panels feeding back to core switches).  The C-AD personnel also recommended Cisco’s 2950, 2960 and 3750 switches and Garrett products in general.  One more miscellaneous tidbit from Jack we should avoid LanCast media convertors.

The final meeting of the week included Jerome, Wayne and Matt Ahrenstein, in which Jerome was briefed on the two prior meetings and he generally agreed with the direction we are taking.  At this meeting, we selected an additional area to try to clean-up before the run, specifically the racks on the west side, where there are at least four 8-port unmanaged switches (3 on DAQ/TRG and one on STARP).  He also suggested we consult with Shigeki from the RACF about the whole affair, and is trying to arrange such a meeting as soon as possible.

In addition to this, Jeff has also stated that while either ITD solution would meet DAQ’s needs for several years, he believes he can obtain adequate performance for far less money with lower end equipment.  Here is Jeff's latest on the DAQ needs for the network:

"My target is 20Gb/sec network capability across switches.   In likely 
scenarios, the network capability would be significantly higher than 
this because hi bandwidth nodes would all be on the same switch 
(ironically, the cheaper switches mostly seem to be line-speed switches 
internally, unlike the big cisco switches...)    However, in the current 
year, I'll have a hard limit of 12 gigabit ethernet cards incoming on 
EVBs for a hard max of 12Gb/sec.    The projected desired data, 
according to the trigger board is around 6Gb/sec (600MB/sec).   I don't 
expect much more than a factor of two through the EVBs above this 
600MB/sec in the lifetime of STAR (meaning current TPC + HFT + FGT), 
although there are big uncertainties particularly for the HFT.     The 
one lump in the planning involves potential L3 farms - and I don't know 
how this will play out.   There are many scenarios some of which would 
not impact the network (ie... specialized hardware plugged into the TPX 
machines...),  but my current approach is that the network needs will 
have to be incorporated in the L3 farm design plan..." 

Where does this leave us?  We need to quickly evaluate options for the “big” switches for the DAQ room and the South Platform.  The DAQ and Trigger groups have 3(?) similar managed switches that might be adequate for the South platform (including a spare), and we should look into the Cisco models suggested by C-AD.  We also should let ITD make another round of suggestions based on our discussions to date, and especially focus with them on what to do with the large ITD switch in the DAQ room that currently has the link to the rest of the campus “public” network.  And we need to do this rather hastily.

Open Questions

Do we support multiple networks on single switches with VLANs, switch port segmentation or other means?  For instance, at remote spots, like PMD’s racks, can we put in a single switch and have it handle both starp and DAQ/TRG?  Daniel Orsatti's most recent advice was leaning towards having a few large switches in four or five core places with VLANs and installing patch panels at or near the various locations needing network connections.
 
Is there a single brand/line of switch equipment that meets most or all of our goals?  Can we get a line of switch products that includes a range from small (~8 port) switches up to the large switches required for DAQ’s event builders or ITD’s main switch, such that they can interoperate and be part of shared monitoring?  (If we go with a patch-panels-to-big-switches approach, then the small switches would not be necessary.)

What kind of monitoring can we expect and how much effort will it take for it to be useful?  SNMP-based?  Nagios?  Etc…

Can we setup a shared but “private” monitoring network for the managed switches, such that starp and DAQ/TRG monitoring share the same infrastructure?  (Most likely, yes.)

Can fiber connectors be easily changed/replaced/repaired?  STAR apparently does not have the tools to terminate fibers at this point.  Do we want to acquire the tools and know-how to do this, or continue to rely on ITD and/or folks like Frank Naase (C-AD) who have done most of our fiber termination to date? 

Overview of the reshape started in 2009

The goal of the online networking reshape is to provide a stable and well-understood networking environment with the possibility of future expansion to meet STAR’s foreseeable needs over time.  The physical layout needs to be well understood, with elements of redundancy and/or easily swapped parts on hand as much as possible.  The devices on the network should be known, including their location, what other systems they are expected to interact with and traffic volumes.  Significant networking errors should be detected at the switch level and allow for troubleshooting without significant disruption to large parts of the network. 

Along the way, it will be very useful to increase the availability of knowledge and sources of assistance related to the network.  Naturally this calls for a well documented network in any case.  Consolidating networking hardware into a common brand or line for the multiple online networks (which are currently a hodgepodge) may reduce the number of errors encountered, improve the ability of STAR's personnel to understand more fascets of the networking environment and allow for better monitoring of the network performance.  Our network should mesh well with existing ITD infrastructure so that their expertise can be brought to bear as needed.  However, ITD expertise cannot be the sole source of support for the online networks – at least two individuals in STAR (but not much more than that) should have broad access to realtime network data and configuration.  STAR’s 24-hour on-call experts (DAQ and online computing in particular) need to be able to respond quickly to incidents and gather clues and information from all sources.

Plan of action / critical path items

I think we need to start from the core and work outwards.  This will allow us to finish as much as possible before the run starts and start to see the most benefits as early as possible.  The two big pieces at the core (in order of importance) are:

1. DAQ’s event builder switch, which calls for 56 (let’s say 64) non-blocking/line speed 1Gb/s ports.  No matter what, this piece needs to be put in place before the run starts.  We can probably limp by with everything else as it exists now if we have to, but this has to be a new piece of hardware in place before December 1 (is this a reasonable deadline?).

2. Whatever ITD wants to replace the current Catalyst 4000-series chassis and blades in the DAQ room.

After this, the next items for consideration/replacement are the starp and DAQ/TRG switches on the South Platform.

Then it is on to the satellite racks in the WAH with their relatively small number of devices.

Then the DAQ room, cleaning up the handful of unmanaged switches that exist for both starp and DAQ/TRG.

Control Room clean-up.  The available wall jacks in the Control Room are insufficient for the number of devices, and many of the jacks are inaccessible behind the west side console, but at least this area is always accessible and has had few problems, so it isn’t a high priority.

Remote power cycling network switches in the WAH

This documents the Network Power Switch plugs used to remotely power cycle STAR's network switches in the Wide Angle Hall.

Updated February 8, 2019  (Ideally, STAR's RackTables would be the definitive source for this information, but it is far from complete.) 

Reshape design goals

A.  Only use managed switches and have each networked device plug directly into a managed switch port.
   
-    Eliminate all “dumb” consumer/SOHO/desktop switches – they are not robust,  add to confusion when troubleshooting and prevent isolation of individual devices
-    allow the blocking of any single device at any time through its nearest  switch’s management interface
-    block the addition of any new, unknown nodes and/or be informed of anything showing up unexpectedly
-    ability to monitor individual ports for traffic volumes, link settings, errors, major links going down, preferably with some history/logging.
-    allow real-time monitoring and alerts for unusual event (capabilities will be hardware/vendor dependent and subject to available time to develop monitoring tools and become familiar with capabilities)


B.  All devices should be within 10-15 feet of a “core” patch panel or network switch.
-    Individuals working on detector subsystems should not have to install network cables that cross rack rows, go from one floor (or room) to another, etc.
-    Piecemeal additions of network segments by subsystems should not be done – that is to say, no one should be adding switches to the network other than core personnel using “approved” devices consistent with the rest of the network components.
-    This calls for cabled and labeled patch panels and/or switches liberally placed throughout the WAH, the Control Room and the DAQ Room.   



C.    Some degree of “commonality” between the infrastructures of the starp and DAQ/TRG networks.  Same line of hardware, media convertors (when needed), switches, monitoring tools, possibly even shared switches with VLANs.  This is a big question – are VLAN’s viable to share switch hardware amongst starp and DAQ/TRG?  A shared “private” management network for the switches is likely a good idea. 

D.    An easily extensible network, such that new locations can be added easily, and existing locations can have additional capacity added and subtracted in accord with the other goals.

E.    Redundant links (fibers or copper, as appropriate) available between all linked core components (preferably with automatic failover).

F.    Spares on hand for just about everything – a good reason to use as few models of hardware as possible.  If we develop a plan with 10 small 8-port switches in various locations, ideally all 10 will be identical and we will have one or two spares on the shelf at all times.

G.    All network components should be on UPS power so that short and/or localized power outages do not bring down portions of the network.  This is not terribly important, but should be kept in mind and allowed for when feasible.

H.  (Added after the initial items above)  Move IC-based devices (switches) away from beam line and attempt to reduce radiation load.  Our working hypothesis, based on anecdotal evidence, is that at least some of the networking problems last year were caused by errors caused by radiation.  The two "big" switches on the South Platform have historically always been in just about the WORST place for radiation load, so these need to be moved away from the beam line.

Rules to Live By in Online Networking at STAR

Document everything!

All hardware with an IP address should be labelled.

All installed cables should have a label on each end that is adequate to quickly locate the other end.

All patch panel ports with cables connected should be labelled appropriately to identify the other end.

All network equipment (switches, patch panels, cable runs, etc.) need to be documented, preferably in appropriate documents in Drupal.

Copper connections:

Use Cat5e or higher graded cables.

Use yellow cables for devices connected to the STARP network (130.199.60-61.x IP addresses).

Use green cables for devices connected to the DAQ/TRG network (172.16.x.x IP addresses).

Use colors other than yellow and green for any other network connections.

Use T568A termination when adding connectors to bare cable.

Fiber Connections:

Use 50 micron multi-mode fiber.

Use 1000Base-SX fiber transcievers where possible.

STAR networks in 1006 and their nicknames

“starp”:  130.199.60.0/23

“DAQ/TRG”: 172.16.0.0/16  (non-routed)

“HPSS”:  RCF network for DAQ → HPSS transfers

“Alexei”: Alexei’s video camera and laser network (currently consists of a switch on the South Platform and a switch in the DAQ room connected by a fiber pair?).  This includes 3-4 PCs including obsolete Windows OSes (e.g. Win 98).  No devices on this network are dual-homed, so it is very isolated from everything else and is mentioned here for completeness.

“trailers”:  130.199.162.  - includes wired connections for printers, vistors’ laptops and workstations not directly involved in operations and may exist outside of the trailers, such as the Control Room for visitors’ laptops while on shift.

“Wireless”: Not really relevant conceptually, but there are also three ITD wireless access points in the area.

“C-AD 108” and “C-AD 90”:  C-AD has at least two networks operating in the DAQ and Control Rooms, which are left well enough alone in their hands, but are mentioned here for the sake of completeness.
 

Admin interface access

Run 16 shift dues

• Dues
  • Past shortfalls, shift coverage by institution
• Shift sign-up - Run 16
  
  • Important dates
  • Shift Layout, Period Coordinators and special arrangements and requests

Dues

Requests to serve additional shifts should be made PRIOR to the final calculation of the dues to the shift committee (D. Keane and D. Smirnov). Please, refer to the important dates section in this document for the "until when" you could make such request.

Past shortfalls, shift coverage by institution

The below table shows the percentage of missed shifts over the past 4 years. This information can be used to exclude authors from the author list in Run 16.

Note: If your institution is in this table and fails again to fulfill its dues and the 4 years average is below the threshold defined by our author exclusion policy (see STAR Note 0545), author would be excluded.

Shift sign-up - Run 16

Important dates

• 2015/12/04 - initial shift dues calculated - council feedback requested.
• 2015/12/04 - Shift sign-up opens for TESTING purposes only - you may exercise the interface by emulating a sign-up
• 2015/12/13 - The shift sign-up committee needs all council feedback by 12/11. Final shift dues will then be re-computed and provided (they should not change much and will account for all reported changes by that date)
• 2015/12/16 - The testing interface will be turned OFF that day and all test records flushed/removed. The countdown will begin.
• 2015/12/17 - Opening will occur at 10 AM BNL time - please, remember to log prior and wait for the countdown to open the signing

Shift Layout, Period Coordinators and special arrangements

Shift layout

STAR shifts begin January 12, 2016 with cosmic data taking shifts.

Period coordinators

As usual, period coordinators are pre-assigned / pre-signed as selected by the Spokesperson office.

Special arrangements and requests

• UTA has requested for Lanny Ray (QA coordinator) to have the first QA shift.
  Status: the shift sign-up coordinators have had flexibility for such arrangements.
• FIAS requested 10 shifts to catch-up for unfilled dues in past years.
  

• 2) We've agreed to pre-assign the following QA shifts under the new family-related policy:
  Sevil Salur (LBNL)      FEB 16
  Richard Witt (Yale)     FEB 23
  Juan Romero (UC Davis)  MAY 31
  
  3) Bob Tribble (TAMU) is pre-assigned to a shift during  APR 12-19.
  
  4) To correct an unusual rounding anomaly, we've agreed to subtract one week from Valparaiso U dues.  



• Dec 19, 2015. Run 16 will be shorter by two week than originally planed: 20 weeks total instead of 22
  

  Dear STAR Collaborators:
  
  We have just received the guidance from DOE (to BNL) that there will be
  20 cryo-week of RHIC run instead of the originally planned 22 weeks.
  
  Our shift sign-up was designed for 22 weeks. For those who have already
  signed up for the last two weeks, please try to un-sign and help to fill other
  open slots. By now we have 8 open slots and need to un-sign 24.
  
  For those who are not able to re-sign to other spots, we will credit your dues,
  but may ask for help if slots open due to unexpected events (visa etc.).
  
  I am looking forward to a successful run 16 and exciting physics from it.
  
  
  Happy Holidays!
  
  Zhangbu
  

  
  Below is a screen shot of the last two weeks of original shift schedule Run 16 as of Jan 5, 2016.
  Only those who signed up for shift before Dec 19, 2015 will be eligible for a credit:
  
  Anju Bhasin, University of Jammu
  Evan Finch, Brookhaven National Laboratory
  Abhinav Sharma, University of Jammu
  Yuri Panebratsev, Joint Institute for Nuclear Research
  Madan Aggarwal, Panjab University
  Isaac Upsal, Ohio State University
  Yang Wu, Kent State University
  Grazyna Odyniec, Lawrence Berkeley National Laboratory
  Kunsu Oh, Pusan National University
  Liwen Wen, University of California - Los Angeles
  Saskia Mioduszewski, Texas A&M University
  Maowu Nie, Shanghai Institute of Applied Physics
  Abhinav Sharma, University of Jammu
  Renee Fatemi, University of Kentucky
  Madan Aggarwal, Panjab University
  Subhash Singha, Kent State University
  Liang He, Purdue University
  Declan Keane, Kent State University
  Sonya Kabana, Kent State University (offline QA)
  
  The following shifters signed up after the announcement:
  
  Devika Gunarathne, Temple University
  Amani Kraishan, Temple University
  
  Before:
  

  
  After:
  

Run 17 shift dues

• Dues
  • Past shortfalls, shift coverage by institution
• Shift sign-up - Run 17
  
  • Important dates
  • Shift Layout, Period Coordinators and special arrangements and requests

Dues

Requests to serve additional shifts should be made PRIOR to the final calculation of the dues to the shift committee (D. Keane and D. Smirnov). Please, refer to the important dates section in this document for the "until when" you could make such request.

Past shortfalls, shift coverage by institution

The below table shows the percentage of missed shifts over the past 4 years. This information can be used to exclude authors from the author list in Run 17.

Note: If your institution is in this table and fails again to fulfill its dues and the 4 years average is below the threshold defined by our author exclusion policy (see STAR Note 0545), author would be excluded.

Shift sign-up - Run 17

Important dates

• 2016/11/29 - initial shift dues calculated - council feedback requested.
• 2016/11/29 - Shift sign-up opens for TESTING purposes only - you may exercise the interface by emulating a sign-up
• 2016/12/13 - The shift sign-up committee needs all council feedback by 12/11. Final shift dues will then be re-computed and provided (they should not change much and will account for all reported changes by that date)
• 2016/12/19 - The testing interface will be turned OFF that day and all test records flushed/removed. The countdown will begin.
• 2016/12/20 - Opening will occur at 10 AM BNL time - please, remember to log prior and wait for the countdown to open the signing

Shift Layout, Period Coordinators and special arrangements

Shift layout

STAR shifts begin January XX, 2017 with cosmic data taking shifts.

Period coordinators

As usual, period coordinators are pre-assigned / pre-signed as selected by the Spokesperson office.
 

Feb. 7-March 7  Oleg Eyser (BNL)
March 7- April 4 Sal Fazio (BNL)
April 4 – April 28 Shuai Yang (BNL)
April 28-May 23 Xiaofeng Luo (CCNU)
May 23 – June 20 Jinlong Zhang (LBL)
June 20 – July 11 Nihar Sahoo (TAMU)

Special arrangements and requests

0) Bob Tribble: SL, evening, beginning Mar 21
1) Pavla + Pavol: 5 shifts as below.
2) Juan Romero wants QA for 1 week, beginning May 02.
3) Sevil Salur wants QA for 1 week, beginning Mar 07.
4) Richard Witt wants QA for 1 week, beginning Mar 21.
5) Lanny Ray, as always, is pre-assigned the first QA shift.
6) Jan Rusnak wants QA for 1 week, beginning Apr 04.
7) FIAS wants pre-assigned shifts like last year:
Day, beginning Apr 4: Belousov, 2 weeks of shift crew;
Evening, beginning Apr 4: Pugash, 2 weeks of shift crew;
Day, beginning Apr 4: Vassiliev, 1 week DO trainee + 1 week DO;
Evening, beginning Apr 4: Zyzak, 1 week DO trainee + 1 week DO

Run 18 shift dues

• Dues

Run 18 Shift Dues & Notes

1. Under the family-related policy, the following 6 weeks of offline QA shifts were pre-assigned:
   MAR 27 Kevin Adkins (Kentucky)
   APR 03 Kevin Adkins
   APR 10 Sevil Salur (Rutgers)
   APR 17 Richard Witt (USNA/Yale)
   MAY 22 Juan Romero (UC Davis)
   JUN 12 Terry Tarnowsky (Michigan State)
    
2. Lanny Ray (UT Austin), as QA coordinator, always is pre-assigned the first QA week.
    
3. FIAS remains in “catch-up mode” and is taking extra shifts above their dues. Pre-assigned shifts can be requested in this scenario. FIAS has been pre-assigned 4 Detector Op shifts.
    
4. Bob Tribble (TAMU) requests the evening Shift leader slot during Apr 10-17.

Run 19 special requests

    9 WEEKS PRE-ASSIGNED QA AS FOLLOWS
    ==================================
    Lanny Ray (UT Austin) QA Mar 5
    Richard Witt (USNA/Yale) QA Mar 19
    Sevil Salur (Rutgers) QA Apr 16
    Wei Li (Rice) QA Apr 23
    Kevin Adkins (Kentucky) QA May 14
    Juan Romero (UC Davis) QA May 21
    Jana Bielcikova (NPI, Czech Acad of Sci) QA May 28  
    Yanfang Liu (TAMU) QA June 25 
    Yanfang Liu (TAMU) QA July 02
    
    8 WEEKS PRE-ASSIGNED REGULAR SHIFTS AS FOLLOWS
    ==================================
    Bob Tribble (BNL) Feb 05 SL evening 
    Daniel Kincses (Eotvos) Mar 12  DO Trainee Day
    Daniel Kincses (Eotvos) Mar 19  DO Day
    Mate Csanad (Eotvos) Mar 12 SC Day
    Ronald Pinter (Eotvos) Mar 19 SC Day
    Carl Gagliardi (TAMU)  May 14  SL day
    Carl Gagliardi (TAMU)  May 21 SL day 
    Grazyna Odyniec (LBNL) July 02 SL evening
    
    

Shift Dues and Special Requests Run 20

For the calculation of shift dues, there are two considerations.
1) The length of time of the various shift configurations (2 person, 4 person no trainees, 4 person with trainees, plus period coordinators/QA shifts)
2) The percent occupancy of the training shifts

For many years, 2) has hovered about 45%, which is what we used to calculate the dues.  Since STAR gives credit for training shifts (as we should) this needs to be factored in or we would not have enough shifts.

The sum total of shifts needed are then divided by the total number of authors minus authors from Russian institutions who can not come to BNL.

date                  weeks           crew           training           PC           OFFLINE          
11/26-12/10    2                  2                      0                  0           0           
12/10-12/24    2                  4                      2                 1            0   
12/24-6/30      27                4                      2                 1            1   
7/02-7/16        2                  4                      0                 1            1   

Adding these together (3x a shift for crew, 3x45% for training, plus pc plus offline) gives a total of 522 shifts.
The total number of shifters is 303 - 30 Russian collaborators = 273 people
Giving a total due of 1.9 per author.

For a given institution, their load is calculated as # of authors - # of expert credits x due -> Set to an integer value as cutting collaborators into pieces is non-collegial behavior.

However, this year, this should have been:
date                  weeks           crew           training           PC           OFFLINE          
11/26-12/10    2                  2                      0                  0           0           
12/10-12/24    2                  4                      2                 1            0   
12/24-6/02      23                4                      2                 1            1   
6/02-6/16        2                  4                      0                 1            1   

Adding these together (3x a shift for crew, 3x45% for training, plus pc plus offline) gives a total of 456 shifts for a total due of 1.7 per author.

We allowed some people to pre-sign up, due to a couple different reasons.

Family reasons so offline QA:
James Kevin Adkins
Jana Bielčíková
Sevil Selur
Md. Nasim
Yanfang Liu

Additionally, Lanny Ray is given the first QA shift of the year as our experience QA shifter.

This year, to add an incentive to train for shift leader, we allowed people who were doing shift leader training to sign up for both their training shift and their "real" shift early:
Justin Ewigleben
Hanna Zbroszczyk
Jan Vanek
Maria Zurek
Mathew Kelsey
Kun Jiang
Yue-Hang Leung

Both Bob Tribble and Grazyna Odyniec sign up early for a shift leader position in recognition of their schedules and contributions

This year because of the date of Quark Matter and the STAR pre-QM meeting, several people were traveling on Tuesday during the sign up.  These people I signed up early as I did not want to punish some of our most active colleagues for the QM timing:
James Daniel  Brandenburg
Sooraj Radhakrishnan

3 other cases that were allowed to pre-sign up:
Panjab University had a single person who had the visa to enter the US, and had to take all of their shifts prior to the end of their contract in March.  So that the shifter could have some spaces in his shifts for sanity, I signed up:
Jagbir Singh
Eotvos Lorand University stated that travel is complicated for their group, and so it would be good if they could insure that they were all on shift at the same time.  Given that they are coming from Europe I signed up:
Mate Csanad
Daniel Kincses
Roland Pinter
Srikanta Tripathy
Frankfurt Institute for Advanced Studies (FIAS) wanted to be able to bring Masters students to do shift, but given the training requirements and timing with school and travel for Europe, this leaves little availability for shift.  So I signed up:
Iouri Vassiliev
Artemiy Belousov
Grigory Kozlov

EVO

This section contains information about using EVO for STAR meetings.

If you would like to be able to use EVO in the 1006 trailer, there is a conference PC setup for use.  There is a generic account on the computer for everyone to share.

The account credentials are:
Username: rhicstar
Password: (See below)
Log On To: Conference (This computer)

I will not post the password anywhere that is not encrypted for security purposes, so please come see me in my office (Building 510 Room 1-179) or send me an e-mail containing your GPG public key.  If you do not have a GPG public key, please bring your laptop, (for desktop users, call me, and I'll come to see you) and I'll help you set it up.  It is quite useful.

FUSE & SSHFS - Overview and example in STAR online environment

FUSE (Filesystem in Userspace)

FUSE is a kernel module that acts as a bridge between the kernel’s built-in filesystem functions and user-space code that “understands” the (arbitrary) structure of the mounted content.  It allows non-root users to add filesystems to a running system.

Typically, FUSE-mounted filesystems are (nearly) indistinguishable from any other mounted filesystem to the user.

Some examples of FUSE in action:

• WikipediaFS - viewing and editing Wikipedia articles as if they are local files.
• Archive access - accessing and in some cases manipulating files in tarballs, zip archives, cpio archives, etc.
• Encrypted filesystems
• Union of filesystems (as is done in many live Linux boot disks and Linux installation routines to merge the read-only CD-rom filesystem with read-write space on disk)
• Event Triggering - FUSE implementations can have triggered events.  Some possible uses might be:
  
  • automatically restarting a service if its configuration file is altered
  • automatically re-compiling code whenever a source file is changed
  • making a back-up after a file is changed
• Arbitrary hardware interface
• ... and the one we will focus on here:  SSHFS

The Fuse project FileSystems page has a more complete list and links to individual software projects that use FUSE.

 
SSHFS (Secure Shell Filesystem)

SSHFS allows a user (not necessarily root) on host A (the "client") to mount a directory on host B (the "server") using the (almost) ubiquitous SSH client-server communication protocols.  Generally, no configuration changes or software installations are required on host B.

The directory on host B then looks like a local directory on host A, at a location in host A's directory structure chosen by the user (in a location where user A has adequate privileges of course).

Unlike NFS, the user on host A must authenticate as a known user on host B, and the operations performed on the mounted filesystem are performed as known user on host B.  This avoids the "classic" NFS problem of UID/GID clashes between the client and server.

Here is a sample session with some explanatory comments:

In this example, host A is "stargw1" and host B is "staruser01".  The user name is wbetts on both hosts, but the user on host B could be any account that the user can access via SSH.
 
First, create a directory that will serve as the mountpoint:

[wbetts@stargw1 ~]$ mkdir /tmp/wbssh
[wbetts@stargw1 ~]$ ls -ld /tmp/wbssh
drwxrwxr-x  2 wbetts wbetts 4096 Oct 13 10:52 /tmp/wbssh

Second, mount the remote directory using the sshfs command:

[wbetts@stargw1 ~]$ sshfs staruser01.star.bnl.gov: /tmp/wbssh

In this example, no remote username or directory is specified, so the remote username is assumed to match the local username and the user’s home directory is selected by default.  So the command above is equivalent to:

% sshfs wbetts@staruser01.star.bnl.gov:/home/wbetts /tmp/wbssh

That’s it!  (No password or passphrase is required in this case, because wbetts uses SSH key agent forwarding) 

Now use the remote files just like local files:

[wbetts@stargw1 ~]$ ls -l /tmp/wbssh |head -n 3
total 16000
-rw-rw-r--  1 1003 1003    6412 Oct 19  2005 2005_Performance_Self_Appraisal.sxw
-rw-rw-r--  1 1003 1003   10880 Oct 19  2005 60_subnet_PLUS_SUBSYS.sxc
[wbetts@stargw1 ~]$ ls -ld /tmp/wbssh
drwx------  1 1003 1003 4096 Oct 11 15:56 /tmp/wbssh

The permissions on our mount point have been altered -- now the remote UID is shown (a source of possible confusion) and the permissions have morphed to the permissions on the remote side, but this is potentially misleading too…

[root@stargw1 ~]# ls /tmp/wbssh
ls: /tmp/wbssh: Permission denied

Even root on the local host can’t access this mount point, though root can see it in the list of mounts.
 
In addition to the ACL confusion, there can be some quirks in behaviour, where sshfs doesn't translate perfectly:

[wbetts@stargw1 ~]$ df /tmp/wbssh
Filesystem                                       1K-blocks       Used     Available        Use%     Mounted on
sshfs#staruser01.star.bnl.gov:    1048576000         0     1048576000       0%     /tmp/wbssh

Ideally the user unmounts it once finished, else it sits there indefinitely (it is probably subject to the same timeouts (TCP, firewall conduit, SSH config, etc.) as an ordinary ssh connection, but in limited testing so far, the connection has been long term)  Here is the unmount command:

[wbetts@stargw1 ~]$ fusermount -u /tmp/wbssh/
[wbetts@stargw1 ~]$ ls /tmp/wbssh
[wbetts@stargw1 ~]$

Some additional details:

By default, users other than the user who initiated the mount are not permitted access to the local mountpoint (not even root), but that can be changed by the user, IF it is permitted by the FUSE configuration (as decided by the admin of the client node).  The options though are not very granular.  The three possible options are:

1. access for the user who mounted it (and no one else)
2. the mounter plus root
3. everybody

In any case, whoever accesses the mount point will act as (and have the permissions of) the user on host B specified by the mounter.  This requires careful evaluation of the options permitted and user education on the possibilities of allowing inappropriate or unnecessary access to other users.

The mount is not tied to the specific shell it is started in.  It lasts indefinitely it seems – the user can log out of host A, kill remote agents, etc. and the mount remains accessible on future logins.  (Interpretation: an agent of some sort is maintained on the client (host A) on the user’s behalf.  (If multiple users have access to the user account on A, this could be worrisome, in the same manner as the allowance of others to access the mount point mentioned above.)) 

Here are some potential advantages and benefits of using SSHFS, some of which are mentioned above:

• User-initiated
• Encrypted communications over the network
• Authenticated (at first order) – somewhat better user tracing than NFS
• SSH keys/forwarding can make it relatively painless (no pass{words,phrases} required for mounting)
• Networking/firewalling is simple – if ssh works between the two nodes, then so will sshfs (unlike NFS, where port configuration and firewalls are a pain)
• “Passthrough” mounting works -- an sshfs mount point can be mounted from another node (if host B mounts a directory on C, then A can mount B's mountpoint and have access to C's filesystem.  In this case, B acts as both a client (to C) and a server (to A).)
• No server-side configuration is needed.
• These mounts can be automounted by the user somewhat like autofs using afuser ( http://afuse.sourceforge.net/ ), though this is primarily for interactive use based on SSH agents.

And some drawbacks:

• User initiated (they are unlikely to clean up after themselves)
• Access controls are either very strict (by default), or very lax in the hands of users (-o allow_other or -o allow_root) -- nothing else
• Cross-system UID overlap and ACLs can be confusing
• Availability of FUSE for RHEL/SL 3 and other clients?
• Use of SSHFS in scripts could entice users to create SSH keys without passphrases -- a real no-no!

And some final details about the configuration of the online gatekeepers that presumably are prime candidates for the use of SSHFS:

The standard installation of FUSE for Scientific Linux 4 seems to not be quite complete.  A little help is required to make it work:

In /etc/rc.d/rc.local:

/etc/init.d/fuse start
/bin/chown root.fuse /dev/fuse
/bin/chmod 660 /dev/fuse

“fuse” group created – each user who will use SSHFS needs to be a member of this group (must be kept in mind if we use NIS or LDAP for user management on the gateways)

Server Logging

The default openssh packages  from Scientific Linux 3, 4 and 5 (~openssh 3.6, 3.9 and 4.3 respectively) do not support sftp-subsystem logging.  Later versions of openssh do (starting at version ~4.4).  This provides the ability to log file accesses and trace them to individual (authenticated) users. 

I grabbed the latest openssh source (version 5.1) and built it on an SL4 machine with no trouble:

% ./configure --prefix=/opt/openssh5.1p1 --without-zlib-version-check --with-tcp-wrappers
% make
% make install

Then in the sshd_config file, append "-f AUTHPRIV -l INFO" to sftp-subsystem line.  This activates the logging level (INFO) and causes the logs to be sent to /var/log/secure.  (To be tried: VERBOSE log level).

Even at the INFO level, the logs are fairly detailed.  Shown below is a sample session, with the client commands on the left and the resulting log entries from the server (carradine, using port 2222 for testing) on the right.  For brevity, the time stamps from the log have been removed after the first entry.

From these logs, we would appear to have a good record of the who/what/when of sshfs usage.  But the need to build our own openssh packages puts a burden on us to track and install updated openssh versions in a timely fashion, rather than relying on the distribution maintainer and the OS's native update manager(s).  The log files on a heavily utilised server may also become unwieldy and cause a performance degredation, but I've not made any estimates or tests of these issues.

Here are the specific relevant packages installed on the client test nodes (stargw1 and stargw2):

fuse-2.7.3-1.SL
fuse-libs-2.7.3-1.SL
fuse-devel-2.7.3-1.SL
fuse-sshfs-2.1-1.SL
kernel-module-fuse-2.6.9-78.0.1.ELsmp-2.7.3-1.SL

(Exact versions should not be terribly important, but it appears that fuse-2.5.3 included up to SL4.6 requires more tweaking after installation than fuse 2.7.3 included in SL4.7).

 

Implementing SSL (https) in Tomcat using CA generated certificates

/etc/pki/tls/certs/wildcard.star.bnl.gov.Nov.2012.cert – host cert.
/etc/pki/tls/private/wildcard.star.bnl.gov.Nov.2012.key – host key (don’t give this one out)
/etc/pki/tls/certs/GlobalSignIntermediate.crt – intermediate cert.
/etc/pki/tls/certs/GlobalSignRootCA_ExtendedSSL.crt –root cert.
/etc/pki/tls/certs/ca-bundle.crt – a big list of many cert.

cat /etc/pki/tls/certs/GlobalSignIntermediate.crt > Global_plus_Intermediate.crt
cat /etc/pki/tls/certs/GlobalSignRootCA_ExtendedSSL.crt >> Global_plus_Intermediate.crt
cat /etc/pki/tls/certs/ca-bundle.crt >> Global_plus_Intermediate.crt

 openssl pkcs12 -export -in wildcard.star.bnl.gov.Nov.2012.cert -inkey wildcard.star.bnl.gov.Nov.2012.key -out mycert.p12 -name tomcat -CAfile Global_plus_Intermediate.crt -caname root -chain

keytool -list -v -storetype pkcs12 -keystore mycert.p12

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
           maxThreads="150" scheme="https" secure="true"
           keystoreFile="/home/lbhajdu/certs/mycert.p12" keystorePass="changeit"
           keystoreType="PKCS12" clientAuth="false" sslProtocol="TLS"/>

Online Linux pool

March 15, 2012:

THIS PAGE IS OBSOLETE!  It was written as a guide in 2008 for documenting improvements in the online Linux pool, but has not been updated to reflect additional changes to the state of the pool, so not all details are up to date. 

One particular detail to be aware of:  the name of the pool nodes is now onlNN.starp.bnl.gov, where 01<=NN<=14.  The "onllinuxN" names were retired several years ago.

Historical page (circa 2008/9):

Online Linux pool for general experiment support needs

GOAL: 

Provide a Linux environment for general computing needs in support of the experiemental operations.

HISTORY (as of approximately June 2008):

A pool of 14 nodes, consisting of four different hardware classes (all circa 2001) has been in existence for several years.  For the last three (or more?) years, they have had Scientific Linux 3.x with support for the STAR software environment, along with access to various DAQ and Trigger data sources.  The number of significant users has probably been less than 20, with the heaviest usage related to L2.  User authentication was originally based on an antique NIS server, to which we had imported the RCF accounts and passwords.  Though still alive, we have not kept this NIS information maintained over time.  Over time, local accounts on each node became the norm, though of course this is rather tedious.  Home directories come in three categories:  AFS, NFS on onllinux5, and local home directories on individual nodes.  Again, this gets rather tedious to maintain over time.

There are several "special" nodes to be aware of:

1. Three of the nodes (onllinux1, 2 and 3) are in the Control Room for direct console login as needed.  (The rest are in the DAQ room.)
2. onllinux5 has the NFS shared home directories (in /online/users).  (NB.  /online/users is being backed up by the ITD Networker backup system.)
3. onllinux6 is (was?) used for many online database maintenance scripts (check with Mike DePhillps about this -- we had planned to move these scripts to onldb).
4. onllinux1 was configured as an NIS slave server, in case the NIS master (starnis01) fails.

PLAN:

For the run starting in 2008 (2009?), we are replacing all of these nodes with newer hardware.

The basic hardware specs for the replacement nodes are:

Dual 2.4 GHZ Intel Xeon processors

1GB RAM

2 x 120 GB IDE disks

These nodes should be configured with Scientific Linux 4.5 (or 4.6 if we can ensure compatibility with STAR software) and support the STAR software environment.

They should have access to various DAQ and Trigger NFS shares.  Here is a starter list of mounts:

WISHLIST Items with good progress:

• <Uniform and easy to maintain user authentication system to replace the current NIS and local account mess.  Either a local LDAP, or a glom onto RCF LDAP seems most feasible> -- An ldap server (onlldap.starp.bnl.gov) has been set-up and the 15 onllinux nodes are authenticating to it *BUT* it is using NIS!

• <Shared home directories across the nodes with backups> -- onlldap is also hosting the home directories and sharing them via NFS.  EMC Networker is backing up the home directories and Matt A. is recieving the email notifications.

• <Integration into SSH key management system (mechanism depends upon user authentication method(s) selected).> --  The ldap server has been added to the STAR SSH key management system, and users are able to login to the new onlXX nodes with keys now.

• <Common configuration management system> -- Webmin is in use.

• <Ganglia monitoring of the nodes> -- I think this is done...

• <Osiris monitoring of the nodes> -- I think this is done - Matt A. and Wayne B. are receiveing the notices...

WISHLIST Items still needing significant work:

• None?

SSH Key Management

Overview 

An SSH public key management system has been developed for STAR (see D. Arkhipkin et al 2008 J. Phys.: Conf. Ser. 119 072005), with two primary goals stemming from the heightened cyber-security scrutiny at BNL:

• Use of two-factor authentication for remote logins

• Identification and management of remote users accessing our nodes (in particular, the users of "group" accounts which are not tied to one individual) and achieve accountability

A benefit for users also can be seen in the reduction in the number of passwords to remember and type.

In purpose, this system is similar to the RCF's key management system, but is somewhat more powerful because of its flexibility in the association of hosts (client systems), user accounts on those clients, and self-service key installation requests.

Here is a typical scenario of the system usage: 

1. A sysadmin of a machine named FOO creates a user account named "JDOE" and, if not done already, installs the key_services client.
2. A user account 'JDOE' on host 'FOO' is configured in the Key Management system by a key management administrator.
3. John Doe uploads (via the web) his or her public ssh key (in openssh format).
4. John Doe requests (via the web) that his key be added to JDOE's authorized_keys file on FOO.
5. A key management administrator approves the request, and the key_services client places the key in ~JDOE/.ssh/authorized_keys.

At this point, John Doe has key-based access to JDOE@FOO.  Simple enough?  But wait, there's more!  Now John Doe realizes that he also needs access to the group account named "operator" on host BAR.  Since his key is already in the key management system he has only to request that his key be added to operator@BAR, and voila (subject to administrator approval), he can now login with his key to both JDOE@FOO and operator@BAR.  And if Mr. Doe should leave STAR, then an administrator simply removes him from the system and his keys are removed from both hosts.

Slightly Deeper...

There are three things to keep track of here -- people (and their SSH keys of course), host (client) systems, and user accounts on those hosts:

People want access to specific user accounts at specific hosts.

So the system maintains a list of user accounts for each host system, and a list of people associated with each user account at each host.
(To be clear -- the system does not have any automatic user account detection mechanism at this time -- each desired "user account@host" association has to be added "by hand" by an administrator.)

This Key Management system, as seen by the users (and admins), consists simply of users' web browsers (with https for encryption) and some PHP code on a web server (which we'll call "starkeyw") which inserts uploaded keys and user requests (and administrator's commands) to a backend database (which could be on a different node from the web server if desired). 

Behind the scenes, each host that is participating in the system has a keyservices client installed that runs as a system service.  The keyservices_client periodically (at five minute intervals by default) interacts a different web server (serving different PHP code that we'll call starkeyd).  The backend database is consulted for the list of approved associations and the appropriate keys are downloaded by the client and added to the authorized_keys files accordingly.

In our case, our primary web server at www.star.bnl.gov hosts all the STAR Key Manager (SKM) services (starkeyw and starkeyd via Apache, and a MySQL database), but they could each be on separate servers if desired.

Perhaps a picture will help.  See below for a link to an image labelled "SKMS in pictures".

Deployment Status and Future Plans

We have begun using the Key Management system with several nodes and are seeking to add more (currently on a voluntary basis).  Only RHEL 3/4/5 and Scientific Linux 3/4/5 with i386 and x86_64 kernels have been tested, but there is no reason to believe that the client couldn't be built on other Linux distributions or even Solaris.  We do not anticipate "forcing" this tool onto any detector sub-systems during the 2007 RHIC run, but we do expect it (or something similar) to become mandatory before any future runs.  Please contact one of the admins (Wayne Betts, Jerome Lauret or Mike Dephillips) if you'd like to volunteer or have any questions.

User access is currently based on RCF Kerberos authentication, but may be extended to additional authentication methods (eg., BNL LDAP) if the need arises.

Client RPMs (for some configurations) and SRPM's are available, and some installation details are available here: 

http://www.star.bnl.gov/~dmitry/skd_setup/

An additional related project is the possible implementation of a STAR ssh gateway system (while disallowing direct login to any of our nodes online) - in effect acting much like the current ssh gateway systems role in the SDCC.  Though we have an intended gateway node online (stargw1.starp.bnl.gov, with a spare on hand as well), it's use is not currently required.

Anxious to get started? 

Here you go: https://www.star.bnl.gov/starkeyw/ 

You can use your RCF username and Kerberos password to enter.

When uploading keys, use your SSH public keys - they need to be in OpenSSH format. If not, please consult SSH Keys and login to the SDCC.

STAR Electronic Shiftlog (ESL) Administrator Manual

STAR Electronic Shiftlog (ESL) Administration guide

The STAR (ESL) Electronic Shiftlog is written in JSP (Java server pages) and requires a web server that can render JSP content. Unlike php JSP is compiled into JAVA classes using a method call “Just in Time” this means the page is compiled the first time the page is accessed, then it does not have to be compiled again for the life of the page or until the page is modified. The forbearer of JSP is serverlets these are also used in the shiftlog mostly to stream images. The technology differs in that serverlets need to be compiled in advance of being deployed.

Our JSP server is Apache Tomcat. Documentation and newer versions can be downloaded from http://tomcat.apache.org/. Although tomcat is a fully functional web server unto its self we prefer to allow the Apache web server to serve the HTML content and only require Tomcat to serve the JSP pages that Apache can not. This is accomplished by way of the mod_jk Apache Tomcat Connector using the ajp13 protocol. Tomcat hosts on port 8080. This is blocked from the outside but can be seen on a browser started up on the online web server its self.

The Tomcat server hosting the shiftlog is deployed on the online web server online.star.bnl.gov and run under the tomcat account. In order to log on to the online web server to administrate Tomcat and the ESL you will need keys mapped to the Tomcat user account. Please see Wayne Betts or Jérôme Lauret about getting your keys mapped. There are multiple version of Tomcat residing in /opt.

Conventions relating to install of newer versions of Tomcat on the online web server

All versions of tomcat are placed in the /opt folder, in a sub folder clearly demoting the version number. (When you unzip Tomcat this is usually how it comes.) Examples are:

/opt/apache-tomcat-5.5.20/
/opt/apache-tomcat-6.0.18/

The currently used version of Tomcat is link to /opt/tomcat/. Below is an ls of the tomcat folder:

-bash-3.00$ ls -l /opt/tomcat
lrwxrwxrwx 1 root root 22 Nov 17 11:11 /opt/tomcat -> ./apache-tomcat-6.0.18

Note that this folder is the tomcat’s users home directory. It contains the .ssh folder which holds your keys, so relinking this may cause you to become locked out if you do not transfer this folder in advance.

Configuring Tomcat & The Tomcat Directory Structure

After you install a new version of Tomcat you will want to configure it.

There are some environment variables whose existences you will want to verify, and if they don’t exist you will want to set them, preferably in a start-up script so they will survive a server restart.

$CATALINA_HOME: /opt/tomcat
$JAVA_HOME: /usr/java/default

Inside the Tomcat folder you will find these directories (and some others):

$CATALINA_HOME/bin/
$CATALINA_HOME/logs/
$CATALINA_HOME/webapps/
$CATALINA_HOME/conf/

$CATALINA_HOME/bin/ holds the executables (for linux and windows).

To startup the Tomcat server use:

% $CATALINA_HOME/bin/startup.sh

To shut it down use:

% $CATALINA_HOME/bin/shutdown.sh

You will want to modify the $CATALINA_HOME/bin/catalina.sh this is a script called by startup.sh its function is to invoke the java process which is the Tomcat server.

Directly under the header these lines are added:

# added by Levente Hajdu ##################################### "
export JAVA_OPTS=$JAVA_OPTS" -Xmx512M -Djava.library.path=/usr/lib64 -Djava.awt.headless=true"
############################################################# 

A description of the options used follows

• -Xmx512M sets the memory ceiling on the JAVA VM which runs the server to 512MB this should be sufficient for our needs. Any more consumption over this limit will lead to the Tomcat process being terminated.

• -Djava.library.path this sets the library path for an optional set of native (non-JAVA) libraries which Tomcat can utilize for improved performance. If this is not present you will see suggestions to set it in the tomcat log.

• Djava.awt.headless=true this line prevents a particular type of crash. This server also hosts the SUMS statistics pages. These use libraries (jFreeChart) to render images for display which have a relation to x-server libraries. If Tomcat is started by a user that has X-forwarding enabled but no server running, Tomcat would crash as it tries to execute the JSP without this line present.

You will be spending a lot of time in $CATALINA_HOME/conf/. The file that controls the Tomcat context paths is $CATALINA_HOME/conf/server.xml. This file requires editing when ever software is deployed at a new context path. Before you edit this file always make a backup. Each year of the shiftlog resides on a different context path. Here is the list:

http://online.star.bnl.gov/apps/shiftLog2003/
http://online.star.bnl.gov/apps/shiftLog2004/
http://online.star.bnl.gov/apps/shiftLog2005/
http://online.star.bnl.gov/apps/shiftLog2006/
http://online.star.bnl.gov/apps/shiftLog2007/
http://online.star.bnl.gov/apps/shiftLog2008/
http://online.star.bnl.gov/apps/shiftLog2009/

The current year is always at:

http://online.star.bnl.gov/apps/shiftLog/

If we look inside the $CATALINA_HOME/conf/server.xml file we will see an entry for each one of these paths:

<!--Shiftlog 2007-->
<Context className="org.apache.catalina.core.StandardContext" cachingAllowed="true" 
 charsetMapperClass="org.apache.catalina.util.CharsetMapper" cookies="true" crossContext="false" debug="0" 
 docBase="/var/tomcat/webapps/shiftLog2007.war" mapperClass="org.apache.catalina.core.StandardContextMapper" 
 path="/apps/shiftLog2007" privileged="false" reloadable="true" swallowOutput="false" useNaming="true" 
 wrapperClass="org.apache.catalina.core.StandardWrapper">
<Environment description="" name="year" override="false" type="java.lang.Integer" value="2007"/>
<Environment description="" name="isEditable" override="false" type="java.lang.Boolean" value="false"/>
<Environment description="" name="runLogLink" override="false" type="java.lang.String" 
 value="http://online.star.bnl.gov/RunLog/Summary.php?run="/>
<Environment description="" name="runNumber" override="false" type="java.lang.Integer" value="7"/>
</Context>

This is the block of XML for the shiftlog for 2007. With different versions of Tomcat the syntax of this file can change, however it usually doesn’t change too much. Lets go over the important properties in this block:

docBase – Tomcat supports web archive files (.war). This is basically a zip file with a special internal structure. The explanation of the preparation of one of these files would take a whole Drupal page unto its self.

Path – This is the context path at which the site will appear when you look at it over your web browser. It is the part of the url after the server name.

Environment – The environment sub-tag makes information available to the program. The format if fairly simple, However you have to be careful to set the override="false" or else the .war files ./WEB-INF/web.xml will over write these values with its own values.

The environment properties for the shiftlog are:

year – this is the shiftlog year. Example: “2007”

isEditable – this is a boolean value after the run has completed access to the editor is turned off by setting this to false.

runLogLink – This is the url for the run log. The shiftlog uses this to build links to the run log.

runNumber – this is almost the same as the year it’s just the number. Examples:

run 8 = 2008

run 9 = 2009

run 10 = 2010

The $CATALINA_HOME/webapps/ web apps folder holds the default pages that come pre-packaged with the Tomcat server. This is also the location where Tomcat unpacks the war files. The folder naming conventions can change from Tomcat version to Tomcat version.

The $CATALINA_HOME/logs/ directory, as you may have guessed, holds log files. You will want to look over all files in here even if Tomcat would seem to be functioning correctly. The logs can point out errors you many not be aware of. The file $CATALINA_HOME/webapps/catalina.out holds the stander output stream of your JSPs (not to be confused with the HTML output stream) along with Tomcats own stander output stream, making this a handy file for debugging.

Deploying new war files

To deploy a war file the procedure is as follows:

1. Stop Tomcat:

   $CATALINA_HOME/bin/shutdown.sh

   NOTE: If you deploy the tomcat administrative web interface shutting down the whole server is not strictly required because you could just shut down the context path, but I prefer to shut down the whole server as a matter of habit because time required is so short no one really notices.
    

2. If this is an upgrade of an existing .war file (else move to step 3), back up the old .war file. All war files are located in /var/tomcat/webapps/ here is the listing of the directory, note the convention for the naming of the web archive files:

   -bash-3.00$ ls -1 /var/tomcat/webapps/shiftLog*.war
   /var/tomcat/webapps/shiftLog2003.war
   /var/tomcat/webapps/shiftLog2004.war
   /var/tomcat/webapps/shiftLog2005.war
   /var/tomcat/webapps/shiftLog2006.war
   /var/tomcat/webapps/shiftLog2007.war
   /var/tomcat/webapps/shiftLog2008t.war
   /var/tomcat/webapps/shiftLog2008.war
   /var/tomcat/webapps/shiftLog2009.war

   When removing one of these files I move it to the /var/tomcat/webapps/old/ directory and rename it following the convention here:

   shiftLog2007.Apr03.965628000.war
   shiftLog2007.Apr04.288184000.war
   shiftLog2007.Apr09.200079000.war
   shiftLog2007.Dec03.805483000.war
   shiftLog2007.Feb07.785336000.war
   ...
   shiftLog2007.Mar27.875569000.war
   shiftLog2007.Nov09.134343000.war
   shiftLog2007.Nov28.320967000.war
   shiftLog2007.Nov28.657299000.war

   It is important to retain the backup in case there is something wrong with the new .war file, keeping the old one will allow you to roll back whilst the problem is being corrected.

3. Next copy over the new .war file from the node on which it resides. Scp is the method I use for this. The syntax is:

   % scp [username]@[nodeName]:[Path&File]/var/tomcat/webapps/shiftLog[year].war

4. If this is a new deploy and not an upgrade of an existing .war file you will have to configure a context path in $CATALINA_HOME/conf/server.xml (else move to step 6)
    

5. If this is an upgrade you will have to dump (delete) the expanded .war file in $CATALINA_HOME/webapps/ it should be a directory having a name similar to that of the name of the .war file. You do not have to back this up because you already have the .war file backed up.

6. Startup Tomcat

   % $CATALINA_HOME/bin/startup.sh

7. Open up a web browser and check that the page displays correctly

8. Run the shift log Java web start application to confirm that the developer has signed his or her jar files within the .war file, if not you will need to have the .war file rebuilt.

Tips

Because upgrades are done fairly frequently mostly for request for new features and some bug fixes I keep a script to do the upgrade process listed above, however the script requires modification before running it. The name of the script is $CATALINA_HOME/bin/deploy_year .

If you have done the upgrade but do not notice any change:

1. checked that you dumped $CATALINA_HOME/webapps/ (step 5)

2. also dump your web browsers cache

If you get the “page unavailable” message, check that the tomcat process is running. Use the command

ps –ef | grep tomcat | grep java 

Even if it is running shut it down and try and restart it again, like an old car Tomcat may not start the first time you try to crank it over.

   ssh tomcat@online.star.bnl.gov

<tomcat-users>
  <role rolename="manager"/>
  <role rolename="logEditor"/>
  <user username="jfaustus" roles="logEditor"/>
  <user username="mephistophilis" roles="logEditor"/>
</tomcat-users>

$CATALINA_HOME/bin/shutdown.sh
$CATALINA_HOME/bin/startup.sh

FAQs and Tips

Software Site Licenses

The answer is (almost) always: No!
Neither STAR nor BNL have site licences for any Microsoft product, Hummingbird Exceed, WinZIP, ssh.com's software or much of anything intended to run on individual users' desktops. Furthermore, for most purposes BNL-owned computers do not qualify for academic software licenses, though exceptions do exist.

FAQ: PDF creation

Without Adobe Acrobat (an expensive bit of software), this can be a daunting question. I am researching answers, some of which are available in my Windows software tips. Here is the gist of it in a nutshell as I write this -- there are online conversion services and OpenOffice is capable of exporting PDF documents.

FAQ: X Servers

I recommend trying the X Server that is available freely with Cygwin, for which I have created some documentation here: Cygwin Tips. If you can't make that work for you, then I next recommend a commercial product called Xmanager, available from http://www.netsarang.com. Last time I checked, you could still download a fully functional version for a time-limited evaluation period.

TIP: Windows Hibernation trick

• "Standby" puts the machine in a low power state from which it can be woken up nearly instantly with some stimulus, such as a keystroke or mouse movement (much like a screensaver) but the state requires a continuous power source.  The power required is quite small compared to normal running, but it can eventually deplete the battery (or crash hard if the power is lost in the case of a desktop).
• "Hibernate" actually dumps everything in memory to disk and turns off the computer, then upon restarting it reloads the saved memory and basically is back to where it was.  While hibernating, no power source is required.  It can't wake up quickly (it takes about as long as a normal bootup), but when it does wake up, (almost) everything is just the way you left it.  One caveat about networking is in order here:  Stateful connections (eg. ssh logins) are not likely to survive a hibernation mode (though you may be able to enable such a feature if you control both the client and server configurations), but most web browsing activity and email clients, which don't maintain an active connection, can happily resume where they left off.

Imagine:  the lightning is starting, and you've got 50 windows open on your desktop that would take an hour to restore from scratch.  You want to hibernate now!  Here's how to enable hibernating if it isn't showing up in the shutdown box: 
Open the Control Panels and open "Power Options".  Go to the "Hibernate" tab and make sure the the box to enable Hibernation is checked.  When you hit "Turn Off Computer" in the Start menu, if you still only see a Standby button, then try holding down a Shift key -- the Standby button should change to a Hibernate button.  Obvious, huh?

For the curious:
There are actually six (or seven depending on what you call "official") ACPI power states, but most motherboards/BIOSes only support a subset of these.  To learn more, try Googling "acpi power state", or you can start here as long as this link works.  (Note there is an error in the main post -- the S5 state is actually "Shutdown" in Microsoft's terminology). 
From the command line, you can play around with these things with such straightforward commands as:

%windir%\System32\rundll32.exe powrprof.dll,SetSuspendState 1 

Even more obvious, right?  If you like that, then try this on for size.

TIP: My new computer is broken!:

ITD backups of STAR computers

The following Linux systems are being backed up by the Avamar system: 

 
Avamar backup summary reports are being sent to starsupport@bnl.gov.

STAR has a group (/clients/servers/linux/star) in which we can initiate restore requests using a Windows application available here (note, this is updated somewhat frequently, this is the current version as of July 10, 2013):

https://avamar1.b459.bnl.gov/DPNInstalls/downloads/WIN32/AvamarConsoleMultiple-windows-x86-6.1.1-87.exe

Note, with ITD's help, it is also possible to restore files to a system other than the one they were taken from - very useful for recovering from complete system failures.

Spring 2013: Growth rate of onlldap's storage (passing 600GB during the 2013 run) prompted a request from ITD to remove it from Avamar and return to the Networker system for its backups.  This serves as the NIS and NFS server for users of the online Linux pool and the STAR SSH gateways.
 

For Windows workstations, until mid-2010, ITD provided a Retrospect backup client, which was capable of making bare-bones recoveries, but had a very short retention policy and no way to restore files without ITD's help.  They are now using Avamar for Windows workstation backups, which provides a longer retention policy, but eliminates the bare-bones recovery and excludes various multimedia files.  An improvement in Avamar is the ability to restore files directly from the client, without need for ITD assistance.  The Shift Leader computer, and the workstations used by Liz M., Bill C. and the STSG file server (amongst others) are all backed up using this system.

Printers

STAR's publicly available printers are listed below. 

There are additional printing resources available at BNL, such as large format paper, plotters, lamination and such.  Email us at starsupport 'at' bnl.gov and we might be able to help you locate such a resource.

Printing from the wireless (Corus) network

The "standard" way of printing from the wireless network is to go through ITD's CUPS server on the wireless network.  How to do this varies from OS to OS, but here is a Windows walkthrough.  The key thing is getting the URI for the printer into the right place:
 

• Open the Printers Control Panel and click "Add a Printer". 
• Select the option to add a network printer.  (Ignore the list of printers that it generates automatically).
• Click on the button or option for "the printer that I want isn't listed". 
• Select the option for a shared printer and enter the green URL from the list above for the printer you want.
  eg. http://cups.bnl.gov:631/printers/HP_Color_LaserJet_4700_2
• On the next window, select the hardware manufacturer and model (if not listed, let Windows search for additional models).
• Print a test page and cross your fingers... 
• If your test print does not come out, it doesn't necessarily mean your configuration is wrong - sometimes a problem occurs on the the CUPS server that prevents printing - it isn't always easy to tell where the fault lies.

Since printing through ITD's CUPS servers at BNL has not been very reliable, here are some less convenient alternatives to using the printers that you may find handy.  (Note that with these, you can even print on our printers while you are offsite - probably not something to do often, but might come in handy sometimes.)
 

1.  Use VPN.  But if you are avoiding the internal network altogether for some reason, or can't use the VPN client, then keep reading...

2.  Get your files to rcf2.rhic.bnl.gov and print from there.  Most of printers listed above have rcf print queues (hence the column "rcf2 queue name").  But if you want to use a printer for which there is no queue on rcf2, or you have a format or file type that you can't figure out how to print from rcf2, then the next tip might be what you need.

3.  SSH tunnels can provide a way to talk directly (sort-of) to almost any printer on the campus wired network.  At least as far as your laptop's print subsystem is concerned, you will be talking directly to the printer.  (This is especially nice if you want to make various configuration changes to the print job through a locally installed driver.)  But if you don't understand SSH tunnels, this is gonna look like gibberish:

Here is the basic idea, using the printer in the Control Room.
It assumes you have access to both the RSSH and STAR SSH gateways.

The ITD SSH gateways might also work in place of rssh (I haven't
tried them yet).  If they can talk directly to our printers,
then it would eliminate step C below.

A.  From your laptop:

ssh -A -L 9100:127.0.0.1:9100 <username>@rssh.rhic.bnl.gov

(Note 1:  -A is only useful if you are running an ssh-agent with a
loaded key, which I highly recommend)

(Note 2:   Unfortunately, the rssh gateways cannot talk directly to our
printers, so we have to create another tunnel to a node that can...  If the
ITD SSH gateways can communicate directly with the printers, then the
next hop would be unnecessary...)

B.  From the rssh session:

ssh -L 9100:130.199.162.165:9100 <username>@stargw1.starp.bnl.gov

(Note 1: 130.199.162.165 is the IP address of onlprinter1.star.bnl.gov -
it could be replaced with any printer's IP address on the wired network.)
(Note 2:  port 9100 is the HP JetDirect default port - non-HP printers
might not use this, and there are other ways of communicating with HP
network printers, so ymmv - but the general idea will work with most TCP 
communications, if you know the port number in use. 

C.  On your laptop, set up a local print queue as if you were going to
print directly to the printer over the network (with no intermediate
server), but instead of supplying the printer's IP address, use
127.0.0.1 instead.

D. Start printing...


If you close either of the ssh sessions above, you will have to
re-establish them before you can print again. 

The two ssh commands can be combined into one and you can create an alias to
save typing the whole thing each time.  (Or use PuTTY or some other GUI SSH client
wrapper to save these details for reuse.)

You could set up multiple printers this way, but to use them
simultaneously, you would need to use unique port numbers for each one
(though the port number at the end of the printer IP would stay 9100).

Quick (?) start guide for visitors with laptops

So you brought a laptop to BNL… and the first thing you want to do is get online, right?
Ok, here's a quick (?) guide to getting what you want without breaking too many rules.

Wired Options:

• Visitors' network: Dark purple jacks (usually labeled VNxx) are on a visitors' network and are effectively outside of the BNL firewall. They support DHCP and do not require any sort of registration to use. Being outside the firewall can be advantageous, but will prevent you from
  using some network services within BNL (printing, for instance). (The rest of this page is largely irrelevant if you are using the visitors' network.)

• BNL network: If it isn't dark purple (and it isn't a phone jack) then it is on the BNL network, which supports DHCP on most subnets. (NB. The 60/61 subnet (available in parts of 1006, including the WAH) has a locally managed DHCP server -- contact Wayne Betts to be added to the access list). All devices on the BNL networks are required to be registered based on the MAC address that is unique to each network interface. To help enforce this policy, if you request a DHCP
  address from an unregistered node, you will be assigned a restricted address. With a restricted IP address, your web browser will be automatically redirected to the BNL registration page, and you will be unable to surf anywhere else until you are registered.

  When registering a laptop, fill in "varies" for the location fields. For the computer name field, I recommend using "DHCP Client" (unless you have a static IP address of course).

  Previously registered users are encouraged to verify and update their registration information by going to http://register.bnl.gov from the machine to be updated.

  There you can also find out more about the registration system and find links to some useful information for network users.

   

Using the 1-189 Conference PC

If you'd like to take advantage of the PC or the projector located in Building 510 Room 1-189 (STAR Conference Room) then you will need to know the following:

• You must turn on the projector to see the login panel
  The default screen is the projector, the terminal will not display anything until the primary terminal is on.
• The conference user password is: talkmeet
• Upon logging in EVO will automatically start up and login to a special user account for this room. 
• Use the phone bridge for the audio - it is much clearer and more stable (BNL has an EVO reflector at x6100) - be sure EVO is on mute (default)
• You can then use the wireless keyboard and mouse to present on the projector, which is permanently attached to the conference PC. 
  Please,  do not disconnect the projector from the conference PC.
• Please, do not move the camera by hand - the camera control is placed as a shortcut on the desktop
• You are expected to upload your talks to a Drupal-like agenda and display from the main computer

Firefox, Microsoft Office, and Open Office are also installed for presentation use.

To know:

• THIS COMPUTER HAS SOFTWARE INSTALLED THAT PREVENTS LONG TERM FILE STORAGE.
  IF THE MACHINE IS REBOOTED, YOUR FILES WILL BE DELETED FROM THE HARD DRIVE!
  If you intend to keep files arround for a while, please keep it on drive T:
• The above applies to any software you may attempt to install - if you need a software, please contact the STAR user support team.
• Credentials are wiped out upon termination log-off. If you intend to carry on a login to a Web site, you need to stay logged on.

Windows

This area is intended to provide information for STAR members to assist in configuring and using typical desktop/laptop PCs at BNL.

  Windows 2000/XP and Scientific Linux/Redhat Enterprise Linux are the preferred Operating Systems within STAR at BNL for desktop computing, though there is no formal requirement to use any particular OS.

  These pages are intended to be dynamic, subject to the constantly changing software world and user input.   Feedback from users -- what you find indispensable; what is misleading, confusing or flat-out wrong; and what is missing that you wish was here -- can help to significantly increase the value of these pages.

  Additional pages that are under consideration for creation:

• Windows installation checklist (the basic software and configuration that should probably be on every Windows PC)
• Linux installation checklist
• Common Linux details and useful links, such as Linux equivalents to software for Windows.
• Resources specific to the experiment operations (eg. common DAQ NFS mounts)
• Publically useable terminals

Getting a computer account in STAR

1. Introduction
2. Getting an account and performing work at BNL
   • New users in STAR
   • Previously a STAR user
   • Additional steps for everyone

Introduction

First of all, if you are a new user, WELCOME to the RHIC/STAR collaboration and experiment. STAR is located at Brookhaven National Laboratory and is one of the premier particle detectors in the world.

As a (new) STAR user, you will need to be granted access to our BNL Tier0 computing facility in order to have access to the offline and online infrastructure and resources. This includes accessing BNL from remote or directly while visiting us on site. Access includes access to data, experiment, mailing lists, desktop computer for visitors to name only those. As a National Facility under the Department of Energy (DOE) regulations, a few steps are required for this to happen. Please, follow them precisely and make sure you understand their relevance.

Note:

The DOE requires proper credentials for anyone accessing a computing "resource" and expect such individual to keep credentials up-to-date i.e. in good standing. It is YOUR responsibility to keep valid credentials with Brookhaven National Laboratory's offices. Credentials include: being a valid and active STAR member, having a valid and active guest/user ID and appointment, having and keeping proper trainings. Any missing component would cause an immediate closure of access to computing resources.

In many cases, we rely on account name matching the one created at the RCF (for example, Hypernews or Drupal accounts need exact match to be approved) - this is enforced so we can accurately rely on the work already done by the RCF personnel and only base our automation on "RCF account exist and is active". The RCF personnel work with the user's office and other agencies to verify your credentials.


If you were a STAR user before and seek to re-activate your account, this page also has information for you.

 

Getting an account and performing work at BNL

ALL USERS - Ensure/Verify you are affiliated to STAR in our records

• Ask your council representative that he/she sends your information to the collaboration's record keeping person (at this point in time, this person is Liz Mogavero).
  Note: Your council representative IS the one responsible for keeping the list of authors and active members at all times. We will not (and cannot) consider requests coming from other STAR members.
   
• Pro-actively check the presence of your name and record in our Phone Book.
  Note: If you are not in our Phone Book, you are simply NOT a STAR user as far as we know as our PhoneBook is the central repository of active STAR members as defined by the STAR council representatives. 
   

New users in STAR

1. Request a Guest appointment
   You must be sure you have a valid guest appointment with the BNL User Office.
   
   Note1: Requesting a Guest ID requires a procedure called “Foreign Visit and Assignment”. This procedure involves steps such as background checks with Counter Intelligence and approval from the Department of State. The procedure could take up to 60 days from the time it is started (sensitive countries may take 90 days).
   Note2 : If you have done this already and are a valid Guest, please go to to this section.
   
   
   • Go to the Guest Registration Form and complete the registration as instructed.
     • Purpose of Visit: likely "Research" but if you come for other purposes, chose as appropriate ("CRADA" or "Interview" may apply for example) 
     • Experiment/Facility:  "Physics Dept (RHIC/AGS)"
     • Facility Code: "RHIC"
     • Type of Research: "STAR"
     • Type of Access Requested: likely "Open Research" if you stated your visit purpose as "Research"
     • Subject Code for this Visit/Assignment: likely "General Physics"
        
2. Be patient and wait for further instructions and the approval.
   • ONLY AFTER THE FIRST STEPS will you be able to proceed with the rest of the instructions below.
   • We will assume that you, from now on, have a valid Guest appointment and hold a Guest/BNL ID.
      
3. Ensure you have the required and mandatory training
   You MUST take the Cyber Security training and course GE-CYBERSEC. This training is mandatory and access to the facility computing resources will NOT be granted without it.
   You are also requested to read the Personal User Agreement which describes your responsibilities, the reasonable use and scope of personal use of computing equipments. In recent years, the BNL User Office have requested for the form to be signed and returned for their records. Please, do not skip any of those steps.
    
4. Request an RCF account
   To request a new RCF account, start here. The fields are explained on this instruction page.
   Note: There is a "Contact information" field which is aimed to be filled using an existing RHIC member (holder of a valid account and appointment) who can vouch for you. Put your council representative or team lead name there OR (in case of interview / CRADA etc...) the name of your contact and host at BNL. DO NOT use your own name for this field. DO NOT use the name a person who is NOT yet a STAR Member.
    
5. Additional steps are described below.

Previously a STAR user

1. you cannot remember how to login and need access but you are in good standing (all training valid, your BNL appointment is valid)
2. you have let your training expire but you are a valid BNL guest (your appointment with BNL has NOT expired)
3. your BNL appointment is about to expire or has expired not long ago
4. you are a RHIC user (from another experiment), and now coming to STAR

• First of all, please make sure you are in the STAR PhoneBook as indicated here.
  If you were a member of another experiment before, you will be joining STAR as either a member of an existing institution or joining as a new institution. All membership handling are the responsibilities of the STAR council (approval of new institutions) or your council representative. In both cases, we MUST find your name in our PhoneBook records.
   
• Instructions for the several use cases above
  
  1. If you are in good standing but cannot remember your login information at the RCF facility, please see Account re-activation
      
  2. You have let your BNL training expire - likely, you have not renewed or taken GE-CYBERSEC training available from the training page (please locate in the list at the bottom the course named GE-CYBERSEC). Within 24 hours of the training being taken/renewed again, the privilege to access the BNL computing resources using your RCF account will be re-established (the process will be automatic).
      
  3. Your appointment is about to expire, or has expired not long ago - you will need to go to the Extension requests .
     The Guest Central interface will help identifying your status and appointment expiration. This form could be used by users already having a BNL Guest ID. If you have let your appointment expire for a long time however, this form may let you know (or not show your old BNL badge/guest ID at all). In such a case, you should consider yourself as a "New user" and follow the first set of instructions above.
     For an appointment renewal, the starting point will be the Guest Extension Form.
      
  4. If you were a user before and now coming to STAR, you will need to follow
     • The general principles of being recorded in our PhoneBook as described above.
     • Request a NEW account at the RCF (using a different username than the one you previously had in another experiment)
     • Keep your training up-to-date as before (GE-CYBERSEC) ...
        
  5. Additional steps are described below.

Additional steps for everyone

1. Generate and upload your SSH keys to ensure secure login
   You may now read SSH Keys and login to the SDCC and following information in this section.
    
2. Drupal access
   
   1. Log in to RCF node to verify your account username/password working
   2. Download 2-Factor Authentication app to your mobile device (application ranges from Google or Microsoft Authenticator, Duo Mobile, Authy, FreeOTP, Aegis, ...)
   3. Contact Dmitry Arkhipkin or Jerome Lauret on MatterMost (https://chat.sdcc.bnl.gov, choose "BNL Login" with your RCF username/password) to obtain the 2-Factor Authentication QR code.
   4. Use your RCF username/password + 2-Factor Authentication code (read from the app on your mobile device) to log in to drupal.

• Web Access
• Accessing The STAR Protected Network your ssh keys will need to be uploaded to a different interface if you have a need to login to the online setup - this is needed mostly by experts (not by all users in STAR).
• Software Infrastructure for general information
• Video Conferencing ... and the related comments at the bottom of some of those pages (viewable only if you are authenticated to Drupal).

Account re-activation

The instructions here are for users who have an account at the RCF but have unfortunately let their BNL appointment expire or do not know how to access their (old) account.

• Account expired or is disabled
• Forgot your Kerberos password but need to upload your SSH keys to log-in

Account expired or is disabled

First of all, please be sure you understand the requirements and rationals explained in Getting a computer account in STAR.
As soon as your appointment with BNL ends or expires, all access to BNL computing resources are closed / suspended and before re-establishing it, you MUST renew your appointment first. In such case, we will not provide you with any access which may include access to Drupal (personal account) and mailing lists.

The simplest way to proceed is to

• Check that you do have GE-CYBERSEC training. You can do this by checking your training records.
  • If you do not, please take this training NOW as any future request will be denied until this training is complete
• Send an Email to RT-RACF-UserAccounts@bnl.gov requesting re-activation of your account. Specify the account name (Unix account name, not your name) if you remember it. If you don't your full name may do. The RCF team will check your status (Cyber training, appointment status) and
  • if any is not valid, you will be notified and further actions will be needed.
  • If all is fine, they will re-activate your account after verifying with us that you are a valid STAR user. Please consult Getting a computer account in STAR for what this means ...

If your appointment has expired, you will need to renew it. Please, follow the instructions available here.

Chicken and Egg issue? Forgot your password but did not upload SSH keys

If your account is valid, so is your appointment  but you have not logged in the facility for a while and hence, are unable to upload your SSH keys (as described in SSH Keys and login to the SDCC and related documents) this may be for you.

You cannot access the upload page unless you have a valid password as the access to the RCF requires a double authentication scheme (Kerberos password + SSH key). In case you have forgotten your password, you have first to send an Email to the RCF at RT-RACF-UserAccounts@bnl.gov asking to reset your password, then thereafter go to the SSH key upload interface and proceed.

Drupal access

This page describes how you can obtain the access to the STAR drupal pages. Please understand that your Drupal access is now tight to a valid SDCC login - no SDCC account, no access to Drupal. This is because we integrated Drupal login to the common infrastructure (the login is Kerberos based). Here are the steps to gain access then:

1. Get a computer account in STAR, make sure you have a valid guest appointment and valid cyber training, then request a RCF account  

   https://drupal.star.bnl.gov/STAR/comp/sofi/facility-access/general-access
   
   
   

2. Generate SSH keys and upload them to the SDCC - this has formally little to do with access to Drupal but will allow you to log to the facility and verify you know your Kerberos password (it should have been sent to you via EMail during the account creating with instructions on how to change it)
   https://drupal.star.bnl.gov/STAR/comp/sofi/facility-access/ssh-keys
   Now Log in to RCF node- again, this is to ensure you have the proper
   1. ssh xxx@ssh.rhic.bnl.gov    (xxx is your username on RCF, enter the passphrase for your SSH key)
   2. kinit (enter your SDCC kerberos password - this is the one you will use for accessing Drupal)
       
3. Download 2-Factor Authentication app to your mobile device (application ranges from Google or Microsoft Authenticator, Duo Mobile, Authy, FreeOTP, Aegis, ...)
   
   
   
4. The FIRST time you log to Drupal, you will actually NOT need a second factor (leave the "code" box blank) but MUST generate it right away (the second login will require it).
   1. To generate it, use the "(re)Create 2FA login" in the left hand-side menu, leave all default option (you need a time based OTP), import the QRCode displayed.
   2. Once imported, a 6 Digits code should appear in your app - this is he "code" you will need to enter in future in the third field named "code". Note the code changes every 30 seconds.
5. IF you have logged out  without generating a QRCode or forgot to import and have no "code", then you will need to contact Dmitry Arkhipkin or Jerome Lauret on MatterMost to obtain your 2FA QR-code (either can generate it for you after the fact or start a chat with both). SDCC chat is available at https://chat.sdcc.bnl.gov (please use a Direct Message, not a channel). Use "BNL Login" and your SDCC account user/Kerberos password to login. Do as discussed above (import the QRCode, make sure an entry appears, test your login right away, ...)
    
6. You are set. Drupal login will now ask for your SDCC username and Kerberos password and a 6-digit code you read from the 2-Factor Authentication app.

SSH Keys and login to the SDCC

How to generate keys for about every platform ... and actually be able to log to the SDCC

• General / Introduction
• Instructions for Windows, Linux/Windows interoperability
• Uploading keys to the SDCC
• I have done all of this, what's next?

General

What you find below is especially useful for those of you that work on several machines and platforms in and out of BNL and need to use ssh key pairs to get into SDCC.

• If you use Linux only or Windows only everywhere, all you need is follow the instructions on the SDCC web site and you are all set (see especially their Unix SSH Key generation page).
• Otherwise this page is for you.

The findings on this web page are a combined effort of Jérôme Lauret, Jim Thomas, and Thomas Ullrich. All typos and mistakes on this page are my doing. I am also not going to discuss the wisdom of having to move private keys around - all I want to do is get things done.

The whole problem arises from the fact that there are 3 different formats to store ssh key-pairs and all are not compatible:

• ssh.com: Secure Shell is the company that invented the (now public) ssh protocol. They provide the (so far) best ssh version for Windows which is far nicer than PuTTY. Especially the File Browser provided is so much nicer than the scp command interface. It is free for academic/university sites.
• PuTTY: a free ssh tool for Windows.
• OpenSSH: runs on all Linux boxes and via cygwin on Windows.

Despite all claims, OpenSSH cannot export private keys into ssh.com format, nor can it import ssh.com private keys. Public keys seem to work but this is not what we want. So here is how it goes:

[A] Windows: follow one of the instructions below

1. PuTTY (Windows)
   1. Download puttygen.exe from the PuTTY download page. You only need it once, but it might be good to keep it in case you need to regenerate your keys.
   2. Start the program puttygen.exe
      • Under parameters pick SSH-2 (RSA) and 1024 for the size of the key in bits.
      • Then press the Generate button. You will be asked to move your mouse over the blank area.
      • Enter a passphrase in the referring fields. The passphrase is needed as it will correspond to a password. Make a mental note of it as keys will not be usable without it.
      • I recommend to save the "key fingerprint" too since you will need it at the SDCC web site when uploading your public key. Just save it in a plain text file.
        Note: You can always generate it later from Linux with ssh-keygen -l -f <key_file> but since you will need access to a Linux system to do this, it is important you keep a copy of this now so you could proceed with the rest of the instructions. The picture below shows where the important fields are
        
   3. Saving keys
      • Press Save Public Key. To not confuse all the keys you are going to generate I strongly recommend to call it rsa_putty.pub.
      • Next press Save Private Key. Type rsa_putty as a name when prompted. PuTTY will automatically name it rsa_putty.ppk. That's your private key.
        Don't quit puttygen yet. Now comes the important stuff.
      • In the menu bar pick Conversions->Export OpenSSH key. When prompted give a name that indicated that this is the private key for OpenSSH (Linux). I used rsa_openssh. There is no public key stored only the private. We will generate the public one from the private one later.
      • In the menu bar pick Conversions->Export ssh.com key. When prompted give a name that indicated that this is the private key for ssh.com. I used rsa_sshcom. Again, there is no public key stored only the private. We will generate the public one from the private one later.
   4. All done. Now you have essentially 4 files: public and private keys for putty and private keys for ssh.com and OpenSSH.
2. Getting ssh.com to work (Windows):
   1. Here I assume that you have SSHSecureShell (client) installed, that is the ssh.com version. Open a DOS (or cygwin) shell. We now need to generate a public key from the private key we got from puttygen. Best is to change into the directory where your private key is stored and type: ssh-keygen2 -D rsa_sshcom . Note that the command has a '2' at the end. This will generate a file called rsa_sshcom.pub containing the public key. Now you have your key pair.
   2. Launch SSH and pick from the menu bar Edit->Settings.
      Click on GlobalSettings/UserAuthentication/Keys and press the Import button. Point to your public key rsa_sshcom.pub. The private key will be automatically loaded too. That's it. Press OK and quit SSH. We are not quite ready yet. We still have to generate and upload the OpenSSH key to SDCC.
3. Getting keys to work with OpenSSH/Linux:
   1. Copy the private key rsa_openssh to a Linux box (cygwin on Windows works of course too).
   2. Set the permissions such that only you can read the private key file:

      % chomod 600 rsa_openssh

   3. Generate the public key with:
      

      % ssh-keygen -y -f rsa_openssh > rsa_openssh.pub

   4. Now you have the key pair.
   5. To install the key pair on a Linux box copy rsa_openssh and rsa_openssh.pub to your ~/.ssh directory.
      Important: the keys ideally will be named id_rsa and id_rsa.pub, otherwise extra steps/options will be required to work with them.  So, you are recommended to also do

      % mv rsa_openssh ~/.ssh/id_rsa  
      % mv rsa_openssh.pub ~/.ssh/id_rsa.pub

      All done.  Note that there is no need to put your key files on every machine to which you are going to connect.  In fact, you should keep your private key file in as few places as possible -- just the source machine(s) from which you will initiate SSH connections.  Your public key file is indeed safe to share with the public, so you need not be so careful with it and in fact will have to provide it to remote systems (such in the next section) in order to use your keys at all.
       

[B] Uploading the public key to SDCC:

1. https://web.racf.bnl.gov/Facility/SshKeys/UploadSshKey.php
2. Make sure you upload the OpenSSH public key. Everything else won't work.
   You need to provide the key fingerprint which you hopefully saved from the instructions above.  In case of OpenSSH based keys, you can re-generate the fingerprint with
   % ssh-keygen -Emd5 -l -f <key_file>

Note that forcing MD5 hash is important (default hash is SHA256 the RACF interface will not take). All done.
If you followed all instructions you now have 3 key pairs (files). This covers essentially all SSH implementations there are. Where ever you go, whatever machine and system you deal with, one key pair will work. Keep them all in a very save place.

[C] Done.What's next?

Uploading your keys to the SDCC and STAR SSH-key management interfaces

You need to upload your SSH keys only once. But after your first upload, please wait a while (30 mnts) before connecting to the SDCC SSH Gatekeepers. Basic connection instructions, use:

% ssh -AX xxx@sssh.sdcc.bnl.gov
% rterm

The rterm command will open an X-terminal on a valid STAR interactive node. If you do NOT have an X11 server running on your computer, you could use the -i options of rterm for interactive (non X-term based) session.

If you intend to logon to our online enclave, please check the instructions on Accessing The STAR Protected Network to request an account on the STAR SSH gateways and Linux pool (and upload your keys to the STAR Key SSH Management system).  Note that you cannot upload your keys anywhere without a Kerberos password (both the SDCC and STAR's interface will require a real account kerberos password to log in). Logging in to the Online enclave involves the following ssh connection:

% ssh -AX xxx@cssh.sdcc.bnl.gov
% ssh -AX xxx@stargw.starp.bnl.gov

A first thing to see is that SDCC gatekeeper is here "cssh" as the network is spearated into a "campus" side (cssh) and a ScienceZone side (sssh). For convenience, we have asked Cyber security to allow connections from "sssh" to our online enclave as well (so if you use sssh all the time, it will work).

For the requested an account online ... note that users do not request access to the individual stargw machines directly.  Instead, a shared user database is kept on onlcs.starp.bnl.gov - approval for access to onlcs grants access to the stargw machines and the Online Linux Pool.  Such access is typically requested on the user's behalf when the user requests access to the online resources following the instructions at Accessing The STAR Protected Network, though users may also initiate the request themselves. 

Logging in to the stargw machines is most conveniently done Using the SSH Agent, and is generally done through the SDCC's SSSH gateways.  This additional step of starting an agent would be removed whenever we will be able to directly access the STAR SSH GW (as per 2009, this is not yet possible due to technical details).

See also

To learn more, see:

• SSH connection stability
• Caveats, issues, special cases and possible problems
• SSH Key Management (an overview of the STAR SSH key management system)

Caveats, issues, special cases and possible problems

Shortcut links

• SSH side effects (login in using different user names than on the remote client)
• X11 Forwarding, -X or -Y?? (and all of what you want to know and what you may not want to know)
  • When to use trusted (-Y not-secure) and untrusted (-X secure)
  • Customized setups
• Kerberos hand-shake and the RCF
• Generic group accounts - how does this work?
• Special nodes
• Other caveats related to Kerberos

SSH side effects

Please note that if you remote account name is different from your RCF account name, you will need to use

% ssh -X username@rssh.rhic.bnl.gov

specifying explicitly username rather as the form

% ssh -X rssh.rhic.bnl.gov

will assume a username defaulting to your local machine (remote from the BNL ssh-daemon stand point) user name where you issue the ssh command. This has been a source of confusion for a few users. The first form by the way is preferred as always work and removes all ambiguities.

X11 Forwarding: -X or -Y ??

-X is used to automatically set the display environment to a secure channel (also called untrusted X11 forwarding) . In other words, it enables X11 forwarding without having to grant remote applications the right to manipulate your Xserver parameters. If you want ssh client to always act like with X11 forwarding, have the following line added in your /etc/ssh/ssh_config (or any /etc/ssh*/ssh*_config ).

ForwardX11 yes

-Y enables trusted X11 forwarding. So, what does trusted mean? It means that the X-client will be allowed to gain full access to your Xserver, including changing X11 properties (i.e. attributes and values which alters the look and feel of opened X windows or things such as mouse controls and position info, keyboard input reading and so on).  Starting with OpenSSH 3.8, you will need to set

ForwardX11Trusted yes 

in the client configuration  to allow remote nodes full access to your Xserver as it is NOT enabled by default.

When to use trusted, when to use untrusted

Recent OpenSSH version supports both untrusted (-X) and trusted (-Y) X11 Forwarding. As hinted above, the difference is what level of permissions the client application has on the Xserver running on the client machine.  Untrusted (-X) X11 Forwarding is more secure, but unfortunately several applications (especially older X-based applications) do not support running with less privileges and will eventually die and/or crash your entire Xserver session.

Dilema? A rule of thumb is that while using trusted (-Y) X11 Forwarding will have less applications problems for the near future, try first the most secured untrusted (-X) way and see what happens. If remote X applications fail with a errorssimilar to the below:

X Error of failed request: BadAtom (invalid Atom parameter)
  Major opcode of failed request: 18 (X_ChangeProperty)
  Atom id in failed request: 0x114
  Serial number of failed request: 370
  Current serial number in output stream: 372

you will have to use the trusted (-Y) connection.

Per client / server setup?

Instead of a system global configuration which will require your system administrator's assistance, you may create a config file in your user’s home directory (client side) under the .ssh directory with the following line $HOME/.ssh/config

ForwardX11Trusted yes 

But it gets better as the config file allows per host or per-domain configuration. For example, the below is valid

Host *.edu
	ForwardX11 no
	User jlauret

Host *.starp.bnl.gov
	ForwardX11 yes
    	Cipher blowfish
	User jeromel

Host orion.star.bnl.gov
     ForwardAgent yes
     Cipher 3des
     ForwardX11Trusted yes

Host what.is.this
    User exampleoptions
    ServerAliveInternal=900
    Port 666
    Compression yes
    PasswordAuthentication no
    KeepAlive yes
    ForwardAgent yes
    ForwardX11 yes
    RhostsAuthentication no
    RhostsRSAAuthentication no
    RSAAuthentication yes
    TISAuthentication no
    PasswordAuthentication no
    FallBackToRsh no
    UseRsh no

As a side note, 3des is more secure thank blowfish but also 3x slower. If speed and security is important, use at least aes cypher.

Kerberos hand-shake, How to.

OK, now you are logged to the facility gatekeeper but any sub-sequent login would ask for your password again (and this would defeat security). But you can cure this problem by, on the gatekeeper, issue the following command (we assume $user is your user name)

% kinit -5 -d -l 7d $user

-l 7d is used to provide a long life K5 ticket (7 days long credentials). Note that you should afterward be granted an AFS token automatically upon login to the worker nodes on the facility. From the gatekeeper, the command

% rterm

would open a terminal from the least loaded node on the cluster where you are allowed to log.

Generic (group) accounts

Due to policy regulations, group or generic accounts login cannot be allowed at the facility unless the login is traceable to an individual. The way to log in is therefore to

• Log to the gatekeeper using SSH keys under your PERSONAL account as described at SSH Keys and login to the SDCC
• kinit -5 -4 -l 7d $gaccount
• In case of wide use generic account, one more jump to a "special" node will be necessary. For starreco and starlib for example, this additional gatekeeper node is rcas6003. From there, login to the rest of the facility could be done using rterm as usual (at least in STAR)

Special nodes

This section is about standing on one foot, tapping on to of your head and chanting a mantra unless the moon is full (in such case, the procedure involves parsley and sacrificial offerings). OK, we are in the realm of the very very special tricks for very very special nodes:

• The rmine nodes CANNOT be connected to anymore. However, one can use rsec00.rhic.bnl.gov has a gatekeeper, using your desktop keys and then jump from there to the rmine nodes.
  Scope: Subject to special authorization.
• The test node aplay1.usatlas.bnl.gov cannot be accessed using a Kerberos trick. Since there are two HOPs from your machine to aplay1, you need to use the ssh-agent. See instructions on the Using the SSH Agent help page.

K5 Caveats

• If you log to gatekeeper GK1 for your personal account, you will need to chose another gatekeeepr GK2 for your group account login. This will allow not interference of Kerberos credentials.
• Whenever you log in a gatekeeper and you know you had previously obtained Kerberos credentials on this gatekeeper, you should ensure the destruction of previous credential to avoid premature lifetime expiration. In other words, -l 7d will NOT give you a 7 days lifetime K5 ticket on a gatekeeper where previous credentials exists. To destroy previous credentials, be sure
  1. you do not have (still) opened windows using the credential. Check this by issuing a klist and observe the listing. Valid credentials used in opened session would look like this

     Valid starting     Expires            Service principal
     12/26/06 10:59:28  12/31/06 10:59:28  krbtgt/RHIC.BNL.GOV@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 10:59:30  12/31/06 10:59:28  host/rcas6005.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 11:11:48  12/31/06 10:59:28  host/rplay43.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 17:51:05  12/31/06 10:59:28  host/stargrid02.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 18:34:03  12/31/06 10:59:28  host/stargrid01.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/26/06 18:34:22  12/31/06 10:59:28  host/stargrid03.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25
     12/28/06 17:53:29  12/31/06 10:59:28  host/rcas6011.rcf.bnl.gov@RHIC.BNL.GOV
            renew until 01/02/07 10:59:25 
     

  2. If nothing appears to be relevant or existing, it is safe to issue the kdestroy command to wipe out all old credentials and then re-initiate a kinit.

Using the SSH Agent

General

The ssh-agent is a program you may use together with OpenSSH or similar ssh programs. The ssh-agent provides a secure way of storing the passphrase of the private key.

One advantage and common use of the agent is to use the agent forwarding. Agent forwarding allows you to open ssh sessions without having to repeatedly type your passphrase as you make multiple SSH hops. Below, we provide instructions on starting the agent, loading your keys and how to use key forwarding.

Instructions

Starting the agent

The ssh-agent is started as follow.

% ssh-agent

Note however that the agent will immediately display information such as the one below

% ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-fxDmNwelBA/agent.5884; export SSH_AUTH_SOCK;
SSH_AGENT_PID=3520; export SSH_AGENT_PID;
echo Agent pid 3520;

 
It may not be immediately obvious to you but you actually MUST type those commands on the command line for the next steps to be effective.

Here is what I usually do: redirect the message to a file and source it from the shell like this:

% ssh-agent >agent.sh 
% source agent.sh

The commands above will create a script containing the necessary shell commands, then the source command will load the information into your shell.  This assumes you are using sh. For csh, you need use the setenv shell command to define both SSH_AUTH_SOCK and SSH_AGENT_PID. A simpler approach may however be to use

% ssh-agent csh

The command above will start a new shell, in which the necessary environment variables will be defined in the newly started shell (no sourcing needed). 

Yet another method to start an agent and set the environment variables in tcsh or bash (and probably other shells) is this:
 

% eval `ssh-agent`

Now that you've started an agent and set the environment variables to use it, the next step is to load your SSH key.

 

Loading a key

The agent alone is not very useful until you've actually put keys into it. All your agent key management is handled by the ssh-add command. If you run it without arguments, it will add any of the 'standard' keys $HOME/.ssh/identity, $HOME/.ssh/id_rsa, and $HOME/.ssh/id_dsa.

To be sure the agent has not loaded any id yet, you may use the -l option with ssh-add.  Here's what you should see if you have not loaded a key:

% ssh-add -l
The agent has no identities.

To load your key, simply type

% ssh-add
Enter passphrase for /home/jlauret/.ssh/id_rsa:
Identity added: /home/jlauret/.ssh/id_rsa (/home/jlauret/.ssh/id_rsa)

To very if all is fine, you may use again the ssh-add command with the -l option. The result should be different now and similar to the below (if not, something went wrong).

% ssh-add -l
1024 34:a0:3f:56:6d:a2:02:d1:c5:23:2e:a0:27:16:3d:e5 /home/jlauret/.ssh/id_rsa (RSA)

Is so, all is fine.

Agent forwarding

Two conditions need to be present for agent forwarding to function:

• The server need to be set to accept forwards (enabled by default)
• You need to use the ssh client with the -A option

Usage is simply

% ssh -A user@remotehost

And that is all. For every hop, you need to use the -A option to have the key forwarded throughout the chain of ssh logins. Ideally, you may want to use -AX (where "X" enabled X11 agent forwarding).

Agent security concern

The ssh-agent creates a unix domain socket, and then listens for connections from /usr/bin/ssh on this socket. It relies on simple unix permissions to prevent access to this socket, which means that any keys you put into your agent are available to anyone who can connect to this socket. BE AWARE that root especially has acess to any file hence any sockets and as a consequence, may acquire access to your remote system whenever you use an agent.

Manpages indicates you may use the -c of ssh-add and this indeed adds one more level of safety to the agent mechanism (the agent will aks for the passphrase confirmation at each new session). However, if root has its mind on stealing a session, you are set for a lost battle from the start so do not feel over-confident of this option.

Addittional information

Help pages below links to the OpenSSH implementation of the ssh client/server and other ssh related documentation from our site.

• ssh-agent manpage
• ssh-add manpage
• ssh manpage
• How to safely start the ssh-agent from .bashrc
• ssh-add / ssh-agent issue
• SSH connection stability

SSH connection stability

• Your SSH connections are closed from home
• You get disconnected from any nodes without any reasons?
• ... and you are a PuTTY user
• ... or an Uglix SSH client user

PuTTY users

PuTTY to connect to gateway (from a home connection), you have to

• set a session, be sure to enable SSH

• go to the 'Connection' menu and have the following options box checked

• Disable Nagle's algorithm (TCP_NODELAY option)

• Enable TCP keepalives (SO_KEEPALIVE option)

• Furthermore, in 'Connection' -> 'SSH' -> 'Tunnels' enable the option

• Enable X11 forwarding

• Enable MIT-Magic-Cookie-1

• Save the session

Documentation on those features (explanation for the interested) are added at the end of this document.

SSH Users

SSH users and owner of their system could first of all be sure to manipulate the SSH client configuration file and be sure settings are turned on by default. The client configuration is likely located as /etc/ssh_config or /usr/local/etc/ssh_config depending on where you have ssh installed.

But if you do NOT have access to the configuration file, the client can nonetheless pass on options from the command line. Those options would have the same name as they would appear in the config file.

Especially, KEEP_ALIVE is controlled via the SSH configuration option TCPKeepAlive.

% ssh -o TCPKeepAlive=yes

You will note in the next section that a spoofing issue exists with keep alive (I know it works well, but please consider the ServerAliveCountMax mechanism) so, you may use instead

% ssh -o TCPKeepAlive=no -o ServerAliveInterval=15

Note that the value 15 in our example is purely empirical. There are NO magic values and you need to test your connection and detect when (after what time) you get kicked out and disconnected and set the parameters from your client accordingly. Let's explain the default first and come back to this and a rule of thumb.

There are two relevant parameters (in addition of TCPKeepAlive):

ServerAliveInterval

Sets a timeout interval in seconds after which if no data has been received from the server, ssh will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.

This option applies to protocol version 2 only.

ServerAliveCountMax

Sets the number of server alive messages (see above) which may be sent without ssh receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

The default value is 3. If, for example, ServerAliveInterval (above) is set to 15, and ServerAliveCountMax is left at the default, if the server becomes unresponsive ssh will disconnect after approximately 45 seconds.

In our example

% ssh -o TCPKeepAlive=no -o ServerAliveInterval=15

The recipe should be: if you get disconnected after N seconds, play with the above and be sure to set a

time of ServerAliveInterval*ServerAliveCountMax <= 0.8*N, N being the timeout. Since ServerAliveCountMax is typically not modified, in our example we assume the default value of 3 and therefore a a 3x15 = 45 seconds (and we guessed a disconnect every minute or so). If you set the value too low, the client will send to much "chatting" to the server and there will be a traffic impact.

Appendix

Nagle's algorithm

This was written based on this article.

RPC implementations on TCP should disable Nagle. This reduces average RPC request latency on TCP, and makes network trace tools work a little nicer.

Determines whether Nagle's algorithm is to be used. The Nagle's algorithm tries to conserve bandwidth by minimizing the number of segments that are sent. When applications wish to decrease network latency and increase performance, they can disable Nagle's algorithm (that is enable TCP_NODELAY). Data will be sent earlier, at the cost of an increase in bandwidth consumption.

KeepAlive

The KEEPALIVE option of the TCP/IP Protocol ensures that connections are kept alive even while they are idle. When a connection to a client is inactive for a period of time (the timeout period), the operating system sends KEEPALIVE packets at regular intervals. On most systems, the default timeout period is two hours (7,200,000 ms).

If the network hardware or software drops connections that have been idle for less than the two hour default, the Windows Client session will fail. KEEPALIVE timeouts are configured at the operating system level for all connections that have KEEPALIVE enabled.

If the network hardware or software (including firewalls) have a idle limit of one hour, then the KEEPALIVE timeout must be less than one hour. To rectify this situation TCP/IP KEEPALIVE settings can be lowered to fit inside the firewall limits. The implementation of TCP KEEPALIVE may vary from vendor to vendor. The original definition is quite old and described in RFC 1122.

MIT Magic cookie

To avoid unauthorized connections to your X display, the command xauth for encrypted X connections is widely used. When you login, a .Xauthority file is created in your home directory ($HOME). Even SSH initiate the creation of a magic cookie and without it, no display could be opened. Note that since the .Xauthority file IS the file containing the MIT Magic cookie, if you ever run out of disk quota or the file system is full, this file CANNOT be created or updated (even from the sshd impersonating the user) and consequently, no X connections can be opened.

The .Xauthority file sometimes contains information from older sessions, but this is not important, as a new key is created at every login session. The Xauthority is simple and powerful, and eliminates many of the security problems with X.

Examples and other documentation

A few examples will be left here to guide users and installer.