CTIO DECam Support Pages

Contents of this page:

-------------------------------------------------------------------------------
SISPI GUIs INTERFACE

Here: http://system1.ctio.noao.edu:7001/apps/

Only from within the CTIO network
 

-------------------------------------------------------------------------------
EXPERT CHECK LIST

-- These instructions are meant to be done before the observer takes flats.  Verify that telops has done this.

-------------------------------------------------------------------------------
RESTARTING AND SHUTTING DOWN A SISPI INSTANCE (Experts only, call CTIO support astronomer)

 

--  These instructions detail how to restart and shutdown a SISPI instance

--  The current *ini file and current SISPI configuration is also posted here.
 

-------------------------------------------------------------------------------
INCREASING THE GUIDER EXPOSURE TIME

1.  Go to the Architect console:   http://system1.ctio.noao.edu:7001/apps/architect_console/
2.  Find the System Control Box.  Replace the boxes that say:

Component         Device           Command: Press Enter When Complete

with:

GCS     GCS          set guideexptime nnnn

nnnn is the  time in ms. The default is 600 ms.

NOTE: this can now be done directly from the menu in the guider GUI

 

-------------------------------------------------------------------------------
HELPING THE GUIDER FIND GUIDE STARS IN CROWDED FIELDS

You can change the frame search and isolation parameters (these are arcsec) from the architect console

to:

GUIDER set search_range 5
GUIDER set isolation_range 7

the values below are the default at the moment:

GUIDER set search_range 6
GUIDER set isolation_range 12
 

 

-------------------------------------------------------------------------------
RESTARTING THE COMFORT DISPLAY

1.  Follow instructions here (bottom of page):  Instance Start-up For Experts
2.  If there is no "comfort1" node and no "DISPIB" role to restart, first go to the architect console and replace

Component         Device           Command: Press Enter When Complete

with:

ARCHITECT     MANAGER          start_node comfort1

2a.  Connect to the VNC server on system1, display 2. Find the OCS Console and the Manager Console:

OCS Console: select ARCHITECT
MANAGER Console: start_role DISPIB comfort1
MANAGER Console: start_role DISPIS comfort1
MANAGER Console: select OCS

2b.   Go to the comfort display (comfort1) and make sure a desktop session is running. If not, start one using the DECamObserver account.  Open a terminal and enter the command

xhost +localhost

-------------------------------------------------------------------------------
RESTARTING THE DECAL COMPUTER

If SISPI loses contact with DECal (for example, error message "DECal not ready"), the DECal computer must be re-initialized:

  1. Go to the DECal room (floor MZ) and power-cycle the computer. There is no password in this computer and a straight-forward power-cycle will start automatically all three DECal servers. You may need to do a SISPI "Restart" (and maybe a "Configure").

        Alternatively, this procedure can be performed remotely by VNC connection into the DECalS machine:                                              
        
vncviewer -Shared 139.229.13.61   (or vncviewer -Shared DECalS.ctio.noao.edu)

 
           (password available in the Blanco 4m control room)

If this fails (it shouldn't) there are two other options you can try which involve unlinking DECal from SISPI. Notice that these options will not allow to take calibration frames. 

  1. Go to the architecture console and replace 

Component         Device           Command: Press Enter When Complete

with:

OCS                      OCS          remove_component DECAL

to add it back in:

OCS                       OCS         add_component DECAL,DECAL,DECAL

      2. A more drastic option is to re-start SISPI with no DECal role by using the following initialization file:

 

architect -c DES_nodecal.ini

Additional instructions on the DECal computer are found here: DECal Server Emergency Restart Procedure

 

-------------------------------------------------------------------------------
RESTARTING THE GUIDER GUI

Open a terminal on observer 1 and start a vnc viewer that connects to readout2.ctio.noao.edu:15

> vncviewer readout2.ctio.noao.edu:15

(password is on Andreas updated list, come to office 14 to pick up a copy)

If you vnc into the guider fui from home, please use this only when needed and then disconnect - sending the image data across the network uses resources.

 

-------------------------------------------------------------------------------
RESTARTING CHECKING THE STATE OF VSUB (IE, IF VSUB BUTTON IS YELLOW)

Typically vsub is either on or off for the entire system (all 6 Monsoon crates, counting each split backplane as a crate). Sometimes the crates are in a mixed state – some on, some off. In this case the vsub indicator on the ObserverConsole GUI turns yellow. To find out which crate and which module (slot) in each crate has vsub on, use the command get vsub details.

Go to the Architect Console, select the OCS role and enter the command in the control section. This returns a string like this:
<pre>
FPANS:
SL4_CCD12_VSUB_ENBL 0.000000
SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkpf
SL4_CCD12_VSUB_ENBL 0.000000
SL3_CCD12_VSUB_ENBL 0.000000 << _decam_bkp4
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp5
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp1
SL4_CCD12_VSUB_ENBL 0.000000
SL5_CCD12_VSUB_ENBL 0.000000
SL6_CCD12_VSUB_ENBL 0.000000 << _decam_bkp3 GPAN: 0.000000 </pre> GPAN is the guider backplane (bkpg).

A 0.000000 indicates that vsub is off for this module.

A 1.000000 indicates that vsub is on.

-------------------------------------------------------------------------------
 ACTIVATING A MASTER CONSOLE

Sometimes the GUIs get slow and we restart them, or we've restarted the SISPI instance, logged in to the GUIs, then remember to restart them because that's easier than refreshing each one. When this happens, the new GUIs have a new session, but the old session still has Master Console. To reset this deadlock and to restart the GUIServer:

(a) open up an xterm, and log into system1.ctio.noao.edu as the "sispi" user.

xterm> ssh sispi@system1

(b) setup a SISPI-specific set of commands

[sisp@system1 ~]$ setup SISPI

(c) execute the releaseMasterConsole command

[sisp@system1 ~]$ releaseMasterConsole DECam_YYYYMMDD

(where YYYY = year, MM = numerical month, DD = numerical day)

(d) log out-of the GUIs (or open them-up again)

 

-------------------------------------------------------------------------------
STOPPING DTS

To observe without DTS use this command in the Arquitecture Console:

OCS OCS set dtsqueue None

Alternatively, Tell the OCS “ib set call_dts false” to disconnect the IBs from the DTS

The dtsqueue and all other exposure parameters are set when the request is entered in the queue and not when the exposure is started. If using this command, the change will be effective for every new request that is loaded. To inmediately stop DTS use the second parameter. To resume DTS, use

OCS OCS ib set call_dts true

Always send a message to sdmops@noao.edu when this happens so they can transfer the images later.
 

-------------------------------------------------------------------------------
MONITORING DTS

you can always check

http://dts.noao.edu/status

to check on DTS at all nodes.  The "Pend" column is the number of pending transfers out of that node and is usually near 1, but can be much higher during fast cadence programs or calibration sequences.  

-------------------------------------------------------------------------------
TURNING AOS ON/OFF

 AOS (Adaptive Optics System) is on by default. The active optics system corrects the focus and alignment of DECam automatically using information obtained from the eight focus and alignment sensors. Currently SISPI will initialize DECam to control all 5 terms in the hexapod (we call this AOS5). In some ocassions, the observer may want to turn this feature off. For example, recently we had a case in which the observer wanted to do consecutive images, with no dithering, hoping their objects will lie in the same pixel (or about) from image to image. This usually does not happen due to tip/tilt movements of the hexapod. To turn this off, AOS1 should be used. This allows movements of the hexapod only in the z direction (focus). To turn AOS off:

In the arquitecture console  write the following in the three command boxes (or scroll down to find AOS then it will fill in the first two for you):

FOCUS AOS init_pid aos1

To turn AOS on (default):

FOCUS AOS init_pid aos5

The system will get turned back to AOS5 by a "Configure" or by a SISPI restart.

In case the focus needs to be turn off as well, untick the "telecsope" box on the hexapod panel in the ICS display.

 

-------------------------------------------------------------------------------
QUICK REDUCE

The Reset button can be used to recover from unexpected errors.  If this does not work, restart the server as follows:

ssh quickreduce@quick1.ctio.noao.edu
(the password is on Tim's list)
[quickreduce@quick1 ~]$ bash
[quickreduce@quick1 ~]$ ctio_portal_server.sh restart

 

If you are missing the browser display,  open the follwoing page in observer2:
 

http://quick1.ctio.noao.edu:8080/# 

and login as DECamObserver (password in Tim's list). Then, hit QuickReduce and Observing History

If the problem remains send email to:  helpdesk@linea.gov.br

See https://cdcvs.fnal.gov/redmine/projects/des-sci-verification/wiki/Qr and reference manual  DES-doc 6705 

-------------------------------------------------------------------------------
GOING THROUGH THE DECam LOGS

DECam writes logs for every action of every component, for every instance, for ever.   It is all on disk.  Some useful tips to look for information in the logs:

ssh into system1.ctio.noao.edu as sispi   (as always when logging as sispi, BE CAREFUL, tread lightly)
cd decam/instances

Every instance since September 2012 is there as a directory.  Change to the one you want.  e.g.:
cd DECam_20140322

then go to the logs:
cd logs
ls   (terrifying, isn’t it!)

What you want usually are the OCS logs:
ls -l *OCS*

If you have restarted SISPI more than once in the 24 hour period there will be multiple logs.  The one in use has a link to it, which is convenient (this is a general feature). The others you can identify by the time stamp on last access, which is why you do “ls -l”. Now you can use your favorite unix tools to parse the log file.    Go to the end, see why it stopped. I usually just search for “ERROR”. If you identify a component giving trouble you can go and look at the corresponding log for that component, but often just looking at the OCS logs can tell you what is causing a problem.

-------------------------------------------------------------------------------
FROZEN OBSERVER2

If observer2 gets frozen, the first thing to determine is if it is really hanged. Try to login from another machine (via ssh for example) and check if images are still transferred. If that is the case, maybe the best bet for the night will be to continue observing and use observer2's tools remotely. This option has zero downtime. On the other hand, should observer2 completely stop, SISPI would hang trying to copy an image to it. At that point you have no choice but to restart and to reboot. 

If rebooting is necessary, this is the correct order: (1) Stop SISPI, (2) reboot observer2, (3) restart SISPI

 If observer2 is rebooted while SISPI is taking data, SISPI will hang, because it copies images over to observer2. Don't want to hang SISPI. 

 

-------------------------------------------------------------------------------
COPYING IMAGES MANUALLY TO OBSERVER2

If for any reason images are not transferred automatically to observer2, there is an option to transfer them manually (one by one or in blocks). The procedure is the following:

1) login in observer2 as sispi

2) cd to the diretory where the images should appear (for community observers this is /home4/images/fits/XXXX  where XXXX is the Prop-ID)

3) setup DDS

4) setup Scripts

5) get_images --help

to obtain the options for getting images. A particularly useful one is: get_images -v '[id>408065]'

 

-------------------------------------------------------------------------------
OBSERVER1 DEAD

If there is a real failure in observer1, observer2 can be used as a backup following the following instructions:

1. open a vnc session from observer2 into system1, as sispi
2.  cd ~/decam/architectures/ctio
3. setup SISPI
4. shutdown SISPI  (if it was still running, or just in case)
5. architect -c   DES-observer2.ini

Notice that: (i) this has not been tested yet!, and (ii) when observer3 (spare) machine is installed, that one should be used as backup for observer1.

 

-------------------------------------------------------------------------------
TELEMETRY VIEWER

For debugging and finding problems is useful to make your own plots and find correlations among the many quantities of the Telemetry database.

http://system1.ctio.noao.edu:8080/TV/app/index

Works from within the CTIO network.

-------------------------------------------------------------------------------
RESTART Chrome session (if pulldown buttons on desktop don't work):

[observer3 ~]$ ssh -l sispi system1.ctio.noao.edu
Password: [the system1/sispi password, ask TelOps]
[system1 ~]$ setup SISPI
[system1 ~]$ releaseMasterConsole DECam_YYYYMMDD

-------------------------------------------------------------------------------

CHECK_HARDWARE

a command that tests the following DECam hardware for basic functionality:  hexapod, filter unit, shutter, DECal.    For safety, only the filters and the hexapod are actually moved - not the shutter.

Note that this command does not necessarily FIX a problem, it is meant to be used for instance in situations  like we have had in the last month with the power fails, to check that the hardware is still OK  (or not) after the power is back on, but when we had the DECam crates off.

Prequisites

SISPI has to be running. It can be configured but this is not required (for example when the CCD crates are off).
The components to be tested have to be included in the configuration, i.e. in the .ini file.
The hardware controllers have to be on and connected to the network (of course)

Usage

From the either the text console or the Architect Console enter the command (OCS) check_hardware

Tests

The routines checks the hardware components in turn. It configures them individually and then performs a set of predefined operations to check if the device is functioning properly.
Here is a description of the individual commands used:

HEXAPOD -- configure, check for errors (check_ready function), move relative in z +20/-20  
FCM -- configure, get current filter, if not block, move in block else move in and out z filter  
SHUTTER -- configure, call status function and check if connected  
DECAL -- configure, get led

-------------------------------------------------------------------------------

How to turn off Vsub  after a sispi crash

There is in fact a low level script that can turn on/off the vsubs with or
without sispi being present. Now, the VSUB voltage is handled exclusively
through the monsoon controllers, so the ICS has no way of knowing or
handling it, which in turn means that the panviews should be running for
this script to work.
So if sispi is not running, but the panviews are, you can type this on a
terminal on server1 or any of the readout machines:

<i>vsub_all off</i>

or if you run it without argument it should return the status of vsub on
each of the backplanes. Note that you can run this script even is sispi is
running -it goes trough another instance of the socket server.

If something major happened -like a server1 crash, so nothing is running-,
unfortunatelly there is nothing to do about the vsubs other than indeed
turning the crates off.

-------------------------------------------------------------------------------
USEFUL LINKS

1.  SISPI NEWS

2.  Instance Start-up For Experts

3.  Get-Ready Check List

4.  Ann Elliot's Debugging throug the GUIs page

5.  DECam Operations

6.  Current SISPI Configuration