Updates on the f/8 recovery are now in inverse chronological order.
The mirror was packed up and shipped back to Chile April 12, arriving the following week. The first few pictures show it in the crate and being loading on the freight company's truck by Orion Wiest:
Here it is the basement of the CTIO 1.5-m telescope:
The final picture on the right is a good view of the center plug.
Work on rebuilding the cell and associated mechanisms is underway in La Serena, with integration starting this week using the dummy mirror. The first tests at the telescope will occur at the end of June; these involve verification of all the electronic and mechanical interfaces and preliminary alignment (still using the dummy mirror). The mirror itself will be installed afterward, and the first on-sky run, with ISPI, will take place in August.
The mirror has been successfully repaired. Below, we describe the work carried out after the last post, and outline the next steps required to get the f/8 focus back to doing science.
Completion of the Mirror Repair
The mirror was successfully coated on Kitt Peak and brought back for repeat optical testing. The picture below shows the mirror being brought back into the optics shop basement for testing. The bondline between the plug and the rest of the mirror is just visible.
The movie below shows the handling process required to prepare for one test cycle (removing the secondary from the basement, rotating 60 degrees, and returning to the basement).
Testing proceeded as in September (see 2012/9/27 post). The new measurement shows much less astigmatism than before. In fact, the test results are now essentially the same as the 1993 measurements after the re-figuring carried out that year. This implies that the astigmatism deduced on the telescope is not intrinsic to the secondary. (Note that the secondary properties on the telescope are deduced by comparing optical properties at the cassegrain focus with those from the prime focus; the primary mirror dominates both measurements. In any case, it is clear the performance in the future will be the same as it was before the accident.)
More important, this also implies that the astigmatism seen in the September tests was a result either of stresses induced by the accident or by the stabilization process. The removal of the damaged area has evidently also removed the stress as well, leaving the mirror as good as before.
The 1993 and 2013 measurements are shown below (left and right respectively):
The inner circle in the right-hand figure corresponds roughly to the plug diameter, while the outer circle is the full diameter of the mirror; the clear aperture is approximately the size of the actual surface map, for on-axis observations. Note the good match between the features in both maps; the overall figure is well under 0.1 wave rms in both cases.
Mirror Cell Repairs
Work on the mechanisms and other mechanical parts has been continuing in Chile. The new mirror cart has been completed (see partial assembly below, with mirror cell installed). The new cart is much safer; it has been designed to preclude improper installation of the mirror.
Mechanical fabrication of the new actuator assemblies has been completed (below) and bench testing has just begun.
Nearly all of the work over the next 3 months will take place in Chile. The mechanisms need to be tested and installed, and the complete secondary assembly then needs to be integrated and tested with the dummy secondary mirror. At that point, the secondary assembly will be ready for testing and initial alignment on the telescope itself, for which a block of time is scheduled starting June 23. The work on the telescope represents a first use of the new handling equipment and interfaces associated with DECam.
After this first block of telescope time, any required re-work will be completed and the real mirror will be installed, after re-aluminization. First light with the mirror is planned for the second half of August. The commissioning instrument will be ISPI. Our schedule allows for an additional block of time during the October bright run, as contingency.
There has been considerable progress during the past few weeks!
The plug for the center of the mirror was completed (slightly oversized) and then lapped to fit exactly into the hole in the secondary (with allowance for the glue layer). The plug was then carefully epoxied in place. After the epoxy cured, the fiducial corresponding to the optical vertex was marked on the plug and the remaining silver coating was stripped. The mirror was loaded in its shipping crate today to go to Kitt Peak, where it will be aluminized and then returned for what may well be the final round of optical testing.
It will take about 3 weeks from today until we are certain about the figure, but by comparing initial measurements with the measurement from September (see below), we should have a good idea right away whether the final repair has changed anything.
In Chile, fabrication work for the new mirror cart and related parts for the mirror cell mechanisms is well underway.
Lapping the plug against the hole in the mirror
Checking the fit with a spherometer
After gluing - this close-up shows the edge of the plug, which is barely visible in the center of the picture. The glue bond is about 65 microns thick. The left-over silver coating had not been stripped at this point.
Loading the crate with the mirror onto the Kitt Peak cargo truck.
The work to remove the damaged material from the center of the secondary mirror is now complete:
The first picture shows the mirror when the cutting was just starting. The blue surface of the mirror is a sprayed-on protective coating.
The next photo in the sequence shows the cut at about 40% of its nominal depth; the fractured material in the center is clearly visible, as well as the undamaged material near the edge of the cut. Cutting needed to proceed cautiously at this point, in order to stop and remove or re-stabilize loose material. The hole visible slightly below and left of the center is where material had broken away on the back of the mirror in the accident.
The next photo shows the hole near final depth. The damaged material has been almost completely removed. The opening at the bottom is the top of the central light-weighting pocket in the mirror.
The hole is shown here with cutting complete; in addition the plug (shown upside down in the following photo) has been lapped together with the mirror so that the radii match exactly.
What's next? The plug needs to go back to the vendor (TORC) to have the front surface finished. This involves carefully matching the plug diameter to the diameter of the hole in the secondary, while grinding and polishing the front surface to the nominal radius of the secondary. One this work is complete, there will be careful fit checks and perhaps some more lapping prior to gluing the two parts together. Even though the center plug is inside the central obscuration due to the prime focus cage, we want the surfaces to match well enough to simplify telescope alignment, and to avoid crevices or other irregularities that would complicate aluminization.
Once the plug is glued in place, the mirror will be sent up to Kitt Peak for aluminization; at a minimum we need a coat uniform enough for reliable optical tests, but we hope to get a coat good enough to go on the telescope (though our plan allows for re-aluminization on Tololo if needed). Then we will do another round of optical tests, which we hope will show that the figure is still undistorted, so the mirror can then be shipped to Chile. If not, we will have the figure "adjusted", which obviously adds to the overall schedule.
A variety of tasks are underway in Chile to prepare the mirror cell. Long-lead-time components have been orders, and some fabrication of new or replacement parts has begin in the instrument shop.
Although no updates have been posted in the last two months, a lot of work has gone on during that interval. In particular:
Photos of the mirror cell being inspected for deformation (none found, above) and the plug about halfway through fabrication below (bottom):
Work on bringing the generating engine is now largely complete, and we have also defined and verified the procedures for handling the mirror. (The verification work was done with dummy weights, not the mirror, obviously.) Work is now underway to complete the details of the design of the fixturing for this process, and to complete documentation of the handling procedure.
The picture below shows the generating engine ready for use:
Preparation for cutting the hole for the plug in the center of the secondary mirror are now underway. This work involves preparation of the generating engine that will cut the hole, tests with the engine on a suitable piece of glass, and design, review, and fabrication of the necessary fixturing for handling and cutting the secondary mirror.
2012.09.27 (revised 09.28)
Current Status: The initial optical testing of the secondary mirror figure is now complete. The testing entailed making measurements with the secondary mirror in 6 positions, rotated about its optical axis in 60o increments. By rotating the interferograms by the same amount, and then averaging, the secondary figure can be measured with the wavefront errors in the test setup largely subtracted out. The results are shown below, where the figure on the left shows the raw data, and the figure on the right shows a fit to the data.
The data were trimmed to include only the clear aperture of the secondary, and also to exclude a bad spot in the silver coating (top of mirror in lefthand figure).
The dominant error in the mirror figure is astigmatism. Comparison with data taken on the telescope stretching back ~18 years indicates that the astigmatism has not changed significantly after the accident. Comparison of data for higher-order errors does not appear to show a change, either. This strongly suggests that we will not need to re-figure the secondary after it is repaired, though we will not be certain until we complete the repair and re-test the mirror.
The final testing after the repair will be done after aluminizing the mirror; this will provide a more uniform coating. We have also made a few improvements in our measurement technique that should improve the accuracy of the overall determination.
A few photos from the optics testing:
Test setup with all mirrors aligned. The interferometer is out of the picture, just beyond the lower right corner.
Next Steps: The next steps in the return to service comprise two main activities: repair of the mirror (and subsequent re-test), and repair of the secondary mirror cell.
The mirror repair involves grinding out the damaged center portion of the mirror, and gluing in a Cervit plug. The plug is required to ensure proper operation of the vacuum support system; it needs to be smooth and flush with the rest of the mirror to facilitate washing and alumizining, but it does not require a very good optical figure as it is within the central obscuration due to the prime focus cage.
Considerable preparation is required before we start grinding on the mirror, including tests of the grinding procedure and fabrication of the plug.
The mirror cell repair work should be starting soon. Various electro-mechanical components were damaged in the accident, and these need to be repaired or replaced. In some cases, engineering work is required because the original components are no longer available.
Today, the f/8 secondary mirror was removed from its crate, installed in its test cell, and its mercury-filled support band was filled to its optimal level. The next steps will be to clean the f/8 secondary mirror, apply silver coating, and transport it to the optical test setup.
Progress is more or less on schedule, and preliminary optical tests of the 100" Hindle sphere and 36" fold flat mirror have been conducted. The optical figures of both of these test mirrors must be well characterized in order to measure the optical figure of the Blanco f/8 secondary. The optical figures must also be maintained in a stable condition, which requires monitoring of temperature and control of air flow.
In July, the mounting fixture for the 36" fold flat was completed and the flat was installed into it. The flat mirror was repositioned to the north end of the basement of the optics lab at NOAO North, and the 100" Hindle sphere was moved to its position near the middle of the basement (the optical setup is a Ritchey-Common test at 20 degree angle). On 6 August, the 36" fold flat mirror was carefully aligned and placed 156" in front of the Hindle sphere at 20 degrees and the interferometer with the f/1.5 transmission sphere (TS) was positioned 82" off to the side where it was well out of the beam. The interferometer was used to measure the optical surface of both the Hindle sphere alone and the Hindle sphere + flat mirror (the difference of both provides a map of the optical figure of the 36" flat mirror). The f/8 fixture for the optical setup has been fabricated. Insertion of the f/8 secondary mirror into the optical test setup is anticipated to take place soon, but only after the handling procedures have been finalized and testing of the flat mirror and Hindle sphere is complete.
Preparations are underway for the f/8 secondary mirror to undergo optical testing at the optics lab at NOAO-North in Tucson. An optical testing setup consisting of the f/8 mirror, a 36" diameter flat mirror, and a 100" diameter Hindle sphere (and associated fixtures) is being assembled. The optical set up is shown below.
Fabrication work on the fixtures for the flat mirror is nearing completion. Procedures are being completed for the safe handling of the f/8 and flat mirrors, and preparing the rubber support band for f/8. The week of July 16th, NOAO optical engineers plan to complete the installation of the flat mirror into its test fixture, and move the Hindle sphere to its test position. Later that week, and going on into the week of July 23rd, work will begin on the f/8 (removal from its container, cleaning, and installation into its test cell). Following that, a series of optical tests on the Hindle sphere and flat mirror will be conducted. Near the end of July, there will be a review of the progress on the optical setup, and further planning will be made for the optical testing in August.
The current state of the f/8 mirror's surface figure is unknown. The optical test setup shown above provides a measurement of the errors in all optical surfaces in the set-up; the errors in the flat and Hindle sphere (which are significant) must be subtracted to determine the errors in the secondary itself. This is accomplished by rotating the secondary about its optical axis by a modest amount (~45 degrees each time), re-aligning, and repeating the measurements. If the secondary figure is terrible, this may be apparent on the initial tests, but if (as hoped) the figure remains good, it will not be confirmed until the full test sequence is complete.
Based on the optical test sequence, an initial decision will be made on whether it is necessary to re-figure the mirror, and if so, what method is suitable (e.g., ion beam polishing vs. generating a new surface).
The next step (barring some really adverse result from testing) is to repair the center section. This involves grinding out the center section, leaving a spherical interface, and then cementing a matching plug into the central hole. The plug should be polished smooth and be flush to the rest of the mirror, in order to facilitate cleaning and coating, but it does not require a good optical figure because it is inside the central obscuration of the telescope (due to the prime focus
Unless the initial testing led to a decision to generate a new surface, the mirror would be re-tested at this point, in order to verify the figure after repair. The re-measurement would either (a) confirm that the mirror can be sent back to Chile or (b) provide final guidance for the re-figuring (ion beam polishing or similar). Note that if a completely new surface has to be generated, no re-test is required, but of course the generation of the new figure will take longer than correction via ion-beam polishing (or analogous process).
The accident to the secondary also caused damage to the mirror cell. While this is in principle fairly straightforward to repair, some engineering is required to modernize mechanisms and sensors for which direct replacements are not available any more. The secondary mirror cart must also be replaced with a safer design. The overall effort comprises something like a person-year of effort, but the duration should be less if sufficient resources are available. The work should begin as soon as engineering staff are freed up from DECam installation, and should probably be completed around the same time that the repaired secondary mirror arrives in Chile (in the best-case scenario). Once both the repaired mirror and mirror cell are available, they can be integrated, installed in the telescope, and tested.
By early September, we should have a good idea whether refiguring of the f/8 will be needed. By mid-October, we should have a better estimate of when the mirror can be shipped to Chile. By January (approximately), we should have a clear idea of the scope and duration of remaining tasks, and a status review will be held before either shipping the mirror (either to be refigured, if needed, or to Chile, if not).
Last week, the f/8 crate was opened at NOAO North. The shipping cell was removed by crane and spreader bar, and the subfloor of the crate was removed, revealing the f/8 mirror (covered with tissues and foam pieces). The mirror appears to have arrived in the same condition that it departed Tololo. For now, the mirror will remain in the crate (protected) until the optical fixtures in the optics lab are ready to support the f/8. The next steps will then be to tighten the cinch around the mirror (for handling), setting the mirror upon a 3-point support, cleaning the optical surface (a wash of KOH, CaCO3, and HNO3, followed by a deionized water rinse), spray silvering, and blown dry. At that point the mirror can be placed in its cell, installed in its fixture, and it will be ready to start optical testing.
The f/8 in its opened crate after arriving at NOAO North in Tucson on Tuesday 12 June 2012 (Photo: Gary Poczulp, NOAO/AURA/NSF).
Test fixtures are being machined for supporting the f/8 secondary mirror and a large flat test mirror at the NOAO Optics Shop in Tucson. The complete setup for testing the f/8 secondary mirror should be ready by July 16th.
The mirror was delivered to NOAO North in Tucson late Monday. The crate appears to be in good condition and none of the external shock sensors were tripped.
The f/8 mirror arrived at LAX at 7:56am PDT.
The f/8 mirror crate arrived safely at a DHL warehouse at Santiago airport at mid-day. The box was in perfect condition and
none of the external shock sensors appeared to have been activated. The crate will be transfered to a plane this evening for
its flight to LAX.
The f/8 mirror crate was carefully loaded onto the delivery truck and secured. The truck departed Tololo at 11:15am
and is scheduled to deliver the mirror crate to customs at Santiago airport before the departure for LAX Thursday evening.
The crate should arrive in Tucson next Monday (June 11th).
F/8 crate being loaded onto a delivery truck at Cerro Tololo (6/6/2012). Photo: Eric Mamajek (NOAO/AURA/NSF).
The crate containing f/8, test cell, and related equipment, was safely packed and sealed Friday June 1st.
At the optical lab space at NOAO North in Tucson, preparations are being made for receiving the f/8 mirror for testing.
The current schedule is:
6 June (Wednesday): mirror crate picked up at Tololo and delivered to Santiago.
7 June (Thursday): mirror crate departs Santiago.
8 June (Friday): mirror crate arrives at LAX, clears US customs.
11 June (Monday): mirror crate delivered to NOAO North in Tucson (mid day).
The f/8 secondary mirror being carefully loaded into its crate on Friday, 1 June 2012 (Photo: Eric Mamajek NOAO/AURA/NSF).
The wood crate that will transport the f/8 mirror has been completed, fumigated for international shipping, and all the supporting materials have arrived at Tololo (e.g. foams, shock sensors, etc.). The final packing day is scheduled for Thursday May 31st. We hope to ship the mirror to NOAO-North in Tucson shortly after. Final arrangements are being made for getting insurance and shipping documentation cleared with the Chilean ministry and customs.
Construction of the f/8 shipping crate is underway. The new f/8 crate will contain the secondary mirror itself, a handling band, a test cell, a rubber tube that encircles the secondary for distributing weight (and associated materials), shock sensors, and foam. The f/8 should arrive at NOAO-North in Tucson for optical testing by the end of May.
The panel concluded that the CTIO safety culture, policies, and procedures were generally sound. However, they also provided a number of recommendations for specific improvements. Several of those recommended actions have already been completed with the rest to follow in the weeks ahead. As an important aside, the panel concluded that DECam installation could proceed as planned, after completion of several Job Hazard Analysis (JHA) and Critical Lift Procedures (CLP) documents. Those documents have either been completed now or will be completed in time for various installation activities to proceed.
Efforts to stabilize the mirror have been completed (work related to drilling cracks, HF etching, and application of UV-curing epoxy). Efforts in La Serena are now shifting towards designing and building a shipping container to transport the f/8 secondary mirror to Tucson for further repair and assessment of its optical properties. On Thursday, staff from NOAO South and North met via videocon and agreed on the particulars of how to ship the f/8 and related equipment. A design for the f/8 shipping crate has been finalized (cross section view here, showing multiple compartments - with mirror on bottom, and handling cell on top). The design took into account the design of a previous box which safely carried the f/8 mirror to the US and back to Chile for polishing work in the early 1990's. Construction of the new crate is planned to start next week.
Given the current work schedule, it should know within approximately 6 months whether f/8 secondary can be returned to service on Blanco or the mirror needs further work (e.g. ion polishing) or replaced. Unfortunately there was also significant damage to the active optics hardware on the back of the mirror cell in the accident. This damage still needs to be assessed and a repair plan made, however the focus of engineering and technical staff at CTIO are focused on the DECam installation for the next several months, so any f/8-related work on the cell is at least months away. The current "f/8 recovery" plan is to split management of the project into 3 directions working in parallel over the next several months: mirror cell assessment and repair (at NOAO-South), mirror assessment and repair, and mirror replacement and possible upgrade (both at NOAO-North).
Image of the central region of the f/8 secondary mirror showing where the cracks have been drilled, etched with HF, and epoxied for stabilization (Photo: Gary Poczulp NOAO/AURA/NSF).
The f/8 mirror was removed from its cell for the first time since the incident, and it was supported by three posts on the mirror removal cart for inspection of the damage. Wooden wedges were placed between the mirror and the cell for support during inspection.
The "good" news is that the damage to the back of the mirror is less than anticipated. The three areas of concern were the "hard points" where the mirror is in contact with the cell. Two of those regions show imprinted cracks roughly the size and shape of a thumbprint. One of the hard point regions on the mirror showed no visible damage. None of the three hard point regions on the mirror show evidence of crack propagation. Plans for tomorrow are to apply epoxy to the damaged areas to stabilize them for the trip to Tucson. After work today, the mirror was placed in a safe storage position: halfway back inside the cell. The mirror is snuggly supported on all sides by a mercury-filled band and metal spacers within the cell.
Late in the week, the surface of the "outer ring" and "4 o'clock" cracks (visible in the high resolution image of the central cracked region) was ground out by Gary Poczulp in anticipation of HF etching scheduled for next week. The grinding of the cracks creates small regions on the mirror surface where the HF will be able to pool, and allow the HF to travel along the cracks and hopefully reach their ends. The HF "etching" opens the cracks, rounds out their ends, and will allow epoxy or UV-curing cement to fill in the cracks and stabilize them.
The NOAO Director and NOAO Safety Manager have organized an external Safety Management Review to provide a general review of the processes, procedures, and policies in and around the Blanco as they pertain to the safety of both people and equipment. This review will be held on 26-28 March, with panelists including two safety experts from DOE laboratories and others with experience in Observatory operations. While preparations for DECam installation continue, including removal of the cass cage and primary mirror, work on the installation of DECam will be suspended before the removal of the Prime Focus cage, the first major lift of the cage installation process. Work will resume after the initial report of the panel has been evaluated for immediate actions.
On Monday March 12th, the f/8 assembly was placed on the mirror removal cart, and the f/8 counterweight (which was also acting as a mirror cover) was separated and removed from the mirror. The damage to the center of the mirror was reassessed to see whether any further cracking was noticeable (no noticeable difference was seen). On Tuesday March 13th, most of the rest of the aluminum was removed from the front of the f/8 mirror, however thus far the translucence of the Cer-vit glass is preventing a clear view of the damage to the edge near the three hard points. More detailed inspection of these key locations will have to await removal of the mirror cell after the front cracks are stabilized. Preparations were made for working with HF, which is needed for rounding the sharp edges of the cracks, in an effort to halt the propagation of cracks. Further safety meetings regarding working with HF acid will take place before work commences on the mirror.
Gary Poczulp, optical engineer from NOAO-North, will be visiting La Serena and Tololo 8-26 March to work with CTIO staff on the job of removing the mirror from its cell, removing the remaining aluminum coating, completing the visual inspection of the damage, and begin the task of stabilizing the mirror with UV-curing cement. CTIO engineer Patricio Schurter has been generating FEA simulations to model the effects of removing the mirror from its cell, supporting it at 3 points on the mirror removal jack cart, and transporting the mirror in its cinch. A meeting will be held later this week to make plans for working on the f/8 mirror, and coordinate our efforts with Tololo staff and safety officers.
Meetings with La Serena and Tololo staff were held Tuesday 6 March led by Director Chris Smith and NOAO Risk Manager Chuck Gessner, to provide an overview of the findings from the internal investigation of the f/8 accident together with lessons learned and recomendations for maintaining a safe working environment. An external safety management review is being planned.
Scanned blueprint of f/8 secondary mirror (1.5 Mb): this image shows the layout of the Blanco f/8 secondary, and specifically shows the pattern of lightweighting (circular holes removed from the back of the mirror in order to make the mirror lighter). The center lightweighted circle is where the damage to the mirror is most severe.
High resolution image of damage to center of mirror (with cm measuring tape projected) (8.6 Mb): this image shows the extensive damage at the center of the mirror. Most of the damage is limited to the central lightweighted area. The other lightweighting circles are also visible through the surface. The most concerning feature we have identified is a single radial crack that extends beyond the 20cm diameter circle. It is deemed extremely important to stabilize this crack as soon as possible to prevent further damage to the mirror.
Based on the initial inspection of the mirror surface the day of the accident, and some guidance from a finite-element analysis (FEA) simulation of the damage to the mirror, it was decided that the mirror was sufficiently stable to move around for further inspection.
On Monday 27 February, plans were made for suspending the mirror face-down and conducting another inspection, but this time using acid to remove the aluminum near the mirror center so that optical engineers could study the damage to the glass. Plans were also made for transporting the mirror to a safe location for further study and repair, if the examination of the glass showed that the mirror could be safely transported (it was decided that the garage of the 1.5-m telescope next to its aluminization chamber was an optimal location).
On the morning of Tuesday 28 February, CTIO optical engineers washed the damaged region with acid (removing the aluminum) and photographed the damage to the glass. This examination showed that the damage was similar to that predicted by FEA models, i.e. most of the damage appears isolated to the interior of the 8" diameter central ring crack. Based on this inspection, it was decided that it was safer to transport the mirror with the aluminized surface facing up. The mirror assembly was then rotated, and then transported carefully via crane and forklift to the garage of the 1.5-m telescope for further study. The next step will be to remove the mirror from its cell, complete the acid removal of the surface aluminum, investigate the damage to the rest of the mirror, and stabilize the cracks.
Photo of the central damage to the f/8 secondary mirror with a centimeter tape after the aluminum was removed. The circular areas appearing through the glass are the "lightweighting" regions on the back of the mirror where the glass is thinner. The damage was mostly limited to the central lightweighed region, with one short radial crack extending beyond - visible in the 4 o'clock position (Photo: Patricio Schurter NOAO/AURA/NSF).
On 2/20, the secondary mirror was placed in a safe position and all further work at the Blanco telescope was suspended until we had a chance to evaluate the conditions at the telescope. Limited engineering observations and safety training were allowed 2/21 and 2/22 as we completed our initial evaluation of the incident, and DECam installation work resumed on 2/23 after careful review of the planned activities and discussions with mountain staff.
Our immediate response is being developed in three parallel threads: (1) full investigation towards understanding the incident, (2) analysis of the damage to the f/8 mirror, and (3) review of the DECam installation and commissioning schedule in light of the incident. The investigation is well underway by local safety experts, and the head of NOAO safety is flying down from Tucson to lead the production of the initial report. We are also organizing an external review of the incident. An initial plan for analysis of the f/8 has been developed, and will be reviewed and implemented early next week in order to determine the state of the f/8 and the next steps required for stabilizing the cracks in the mirror. The DECam installation schedule is being modified to allow for the absence of the f/8 mirror, and we hope to have the revision ready within another week.
Contact: Nicole van der Bliek, CTIO Interim Director, nvdbliek at ctio.noao.edu