EM-RP--WRPS-TANKFARM-2010-0011 FINAL
Occurrence Report
After 2003 Redesign

Tank Farms


(Name of Facility)

Nuclear Waste Operations/Disposal


(Facility Function)

Hanford Site Washington River Protection Solutions, LLC


(Site) (Contractor)

Name: Ellis, Martin W
Title: Manager, Performance Assurance Telephone No.: (509) 373-4696


(Facility Manager/Designee)

Name: WATERS, SHAUN F
Title: OPERATIONS SPECIALIST Telephone No.: (509) 373-3457


(Originator/Transmitter)

Name: Date:


(Authorized Classifier (AC))

  1. Occurrence Report Number: EM-RP--WRPS-TANKFARM-2010-0011

      Repetitive Issues With Hazardous Energy Control

  2. Report Type and Date: FINAL

Date Time
Notification: 08/23/2010 18:00  (ETZ)
Initial Update: 08/26/2010 13:42  (ETZ)
Latest Update: 12/02/2010 09:30  (ETZ)
Final: 12/04/2010 14:53  (ETZ)

3. Significance Category: R


  4. Division or Project: Washington River Protection Solutions, LLC (WRPS)

  5. Secretarial Office: EM - Environmental Management

  6. System, Bldg., or Equipment: All Tank Farm Facilities

  7. UCNI?: No

  8. Plant Area: 200 East and West

9. Date and Time Discovered:     08/19/2010    11:30  (PTZ)

10. Date and Time Categorized:     08/19/2010    11:30  (PTZ)

11. DOE HQ OC Notification:

Date Time Person Notified Organization
NA NA NA NA

12. Other Notifications:

Date Time Person Notified Organization
08/19/2010 12:56  (PTZ) Smithwick, R. L. MSA-ONC
08/19/2010 13:02  (PTZ) Klos, J. J. WRPS
08/19/2010 13:02  (PTZ) Blanchard, C. A. DOE-ORP

13. Subject or Title of Occurrence:

      Repetitive Issues With Hazardous Energy Control


14. Reporting Criteria:

2C(2) - Failure to follow a prescribed hazardous energy control process (e.g., lockout/tagout) or a site condition that results in the unexpected discovery of an uncontrolled hazardous energy source (e.g., live electrical power circuit, steam line, pressurized gas). This criterion does not include discoveries made by zero-energy checks and other precautionary investigations made before work is authorized to begin.


15. Description of Occurrence:

Washington River Protection Solutions' review of lockout/tagout (LOTO) program implementation was completed on July 17, 2010. This review along with other LOTO issues reported since September 2009 identified human errors, administrative deficiencies, and inadequate isolation boundaries. These events are symptomatic of a larger company or programmatic issue, which if left unchecked, would represent an unacceptable risk of a serious event. These events include:

Event 1 - The lockout for the removal of nuisance alarms in 271-AN instrument building did not identify an energy source from the 242-A Evaporator that needed to be locked out to support the field work. The work package had been released for work but no work had been performed. The work package was suspended to allow the additional lock to be installed. (Reference Problem Evaluation Request number WRPS-PER-2010-1816)

Event 2 - The lockout for a package to add connector wires to light fixtures in buildings 271-AN and 273-AN and the 241-AN change trailer did not include the isolation boundary for the work to be performed in the 241-AN change trailer.

Event 3 - It was discovered that the lockout for a preventative maintenance (PM) work package for the backup batteries and battery charger in 252-S did not take credit for the tag that locked out the charger. Power to the charger had been locked out under the 241-SY electrical outage lock and tag, but the line item on the TAF [Tagout Authorization Form] for the PM did not take credit for that isolation. (Reference WRPS-PER 2010-1859 and WRPS-PER-2010-1960)

Event 4 - During a walkdown of installed lockout/tagouts, one lockout was found where the independent verifier recorded a different date on the TAF than what was on Tag No. 2. The original Independent Verifier was contacted and corrected the date on the TAF.

Event 5 - Several tags in the field identified the Lockout/Tagout logbook location as MO-268. Due to the implementation of the teams, the location of the logbook is now located at 272-WA. COA [Controlling Organization Administrator] corrected the tags in the field.

Event 6 - Lockout CO-09-058 discovered tag #1 missing from the utility pole. Notified Day Shift Manager to have tag replaced. No work packages were listed on the Tagout Authorization Form.

Event 7 - CO-10-033 discovered that tag #1 had been installed in late March of 2010, but the independent verification and safe-condition-check have not been performed. The TAF does identify that they would not be performed until a later date. No work packages have been added to this TAF. Notified Area Manager to either perform the independent verification and safe-condition-checks or remove the lockout.

Event 8 - Some of the information for one tag for lockout Lockout/Tagout CO-10-017 was faded. Notified Day Shift Manager to have it replaced.

Event 9 - The Danger tag for Lockout/Tagout 222S-10-055 incorrectly identified Breaker #6 of the 222SA main distribution panel as the component to lock in the "off" position. The TAF correctly identified Breaker #48 of the 222SA main distribution panel as the component to be locked in the "off" position. The as found condition was consistent with the TAF (Breaker #48 locked in the "off" position); therefore the intended electrical isolation was met. However, the installer and verifier both missed this anomaly on block #20 of the Danger tag. Field work for this work order had not been released and the intended electrical isolation had been achieved.

Event 10 - Several authorized worker tags were found to have incorrect or missing information on the tag. One authorized worker tag was faded and was replaced.

Event 11 - An ORP Facility Representative discovered an Authorized Worker tag that was not completed while workers were in the field performing work. The Authorized Worker's lock was present, but the tag did not contain a phone number. The Field Work Supervisor was immediately notified, who corrected the situation. The FR met with/briefed the contractor's Lockout/Tagout Authority having jurisdiction who immediately performed an extent of condition review. Other similar issues were detected with marginal legibility on other tags. (Reference WRPS-PER-2010-0213)

Event 12 - During a Defense Nuclear Facility Safety Board observation of Lockout/Tagout records, the following areas of improvement were identified: 1) Two locks/tags documented as installed in 2002 and 2004 had no removal date entered in the index although both were removed shortly after installation; 2) November's monthly surveillance of a LOTO in the Miscellaneous Binder was not completed until identified by the staff (this was completed upon discover); 3) Two of the closed Lockout/Tagout sheets were still present in the Active Binder (this was also corrected after discovery). (Reference WRPS-PER-2010-0282)

Event 13 - On 3/25/2010 several lockout/tagout Danger-Do-Not-Operate tags were replaced because of the information on the tags was faded. After the tags had been replaced, it was discovered that the Controlling Organization Administrator (COA) did not follow the requirements of DOE-0336, "Hanford Site Lockout/Tagout," section 5.8 for replacing faded tags. (Reference WRPS-PER-2010-0744)

Event 14 - A review of the ORP Facility Representative's weekly report dated 04/19/2010 indicated the following weakness. Several hanging Authorized Worker tags contained incomplete information on an active job. During oversight on the AW-02E jumper replacement job, the FR on two separate occasions identified poor lockout/tagout procedural compliances, where the Authorized Worker tags were illegible or incomplete. Out of 30 Authorized Workers, 13 lockbox tags were deficient. Of these, 5 workers were within the energy isolation boundary during the work evolution that day. Findings S-10-AMTF-TABKFARM-001-F01 was issued in January of this year citing the same issue on another job. Corrective actions included a tailgate briefing to the workforce. The issue was brought up to the Authority Having Jurisdiction and the Field Work Supervisor. (Reference WRPS-PER-2010-1135)

Event 15 - Three lockout/tagout quarterly surveillances were missed for the 1st quarter of 2010. (Reference WRPS-PER-2010-1179)

Event 16 - Maintenance work packages TFC-WO-09-3493, 3494, and 3502 were released to work without the corresponding Lockout/Tagout Authorization Form, (241-SY-10-003) having been fully signed off on two pages that had been added to the original four pages. (Reference WRPS-PER-2010-1403)

Event 17 - Lockout 241AN-10-002 tags 7 and 8: the Lockout/Tagout Authorization Form independently verified by block (26) was signed completed on 5/2/10. The "verified by" section of the tags was not signed. (Reference WRPS-PER-2010-1493)

Event 18 - On 6/30/2010, at 222-S, while performing electrical work on panels, the lock-out was prematurely removed prior to safely terminated all wiring. There were three causes that drove the premature removal of the LOTO. The work instructions should have kept the breaker in the "off" (de-energized) and not have the breaker put in the "on" position (energized), Electrical workers should not have removed their locks and there should have been a clear assignment of who had the responsibility to terminate all of the wires that were being worked. (Reference WRPS-PER-2010-1769)

Event 19 - Lockout/Tagout number C241AW-09-014, "Install New Valve Funnels," the required surveillance frequency was monthly in Block #16. When the Tagout Authorization Form (TAF) was reviewed on September 9, 2009, the last documented surveillance date was 08/01/2009. (Reference WRPS-PER-2009-2133)

Event 20 - Finding: Lockout/Tagout number CO-09-085, "Install Test Jumpers at A2156 for VFD Troubleshooting," was found in the inactive logbook: however, Block #12 was not approved. Blocks #28, 29 and 30 were signed indicating removal of all locks and tags; however, Block #12 was not signed which is required to authorize removal of the affected locks and tags. (Reference WRPS-PER-2009-2134)

Event 21 - It was discovered on 9/8/2009, work package TFC-WO-09-3450, "Repair Leak in AN-01A Pit," had been performed with inadequate lockout boundaries. The original scope of the work package was to open the AN-01A pit, install a plastic cover and run a raw water leak test through new piping to determine the exact location of a leak. This work was released on swing shift Thursday, 9/3/2009. The work scope was performed and the leak location identified. On Friday day shift (9/4), the work package was modified to add instructions for a pit entry to loosen a fitting on a newly installed line, inspect the o-ring, and reinstall the line. This work package modification was reviewed and approved on Friday days and the pit entry was made prior to the end of day shift to fix the leak. On Tuesday, 9/8/2009, a follow-up leak check was performed to determine if the repair had been successful. About the time the leak check was completed, Construction was notified by the BOSO [Base Operations Shift Operations] that no LOTO could be found to have been installed for this work. A stop work was issued for this work package and work was suspended. (Reference WRPS-PER-2009-1812)

As a result of this unacceptable trend and programmatic issue that, if left uncorrected, could lead to a serious event, Washington River Protection Solutions' management determined this series of events are recurring declaring a Significance Category "R" report.


16. Is Subcontractor Involved? No


17. Operating Conditions of Facility at Time of Occurrence:

Does not apply.


18. Activity Category:

      03 - Normal Operations (other than Activities specifically listed in this Category)


19. Immediate Actions Taken and Results:

Compensatory measures taken as a result of Lockout/Tagout trends:
a) Immediate compensatory measures were put in place on July 1, 2010, included:
i. Suspending work on all Lockout/Tagout's until Safety Briefings were conducted.
ii. Safety stand-down on July 6, 2010
iii. New Lockout/Tagout reviews by independent oversight team
iv. All active work covered by a Lockout/Tagout was suspended pending a 100% technical evaluation of all released Lockout/Tagouts in the field
b. Additional compensatory measures included the development and implementation of Lockout/Tagout walkdown checklists were implemented on July 22, 2010.


20. ISM:

      2) Analyze the Hazards
      3) Develop and Implement Hazard Controls
      4) Perform Work Within Controls


21. Cause Code(s):

A4B1C01 - Management Problem; Management Methods Less Than Adequate (LTA); Management policy guidance / expectations not well-defined, understood or enforced
A4B1C04 - Management Problem; Management Methods Less Than Adequate (LTA); Management follow-up or monitoring of activities did not identify problems
A4B1C06 - Management Problem; Management Methods Less Than Adequate (LTA); Previous industry or in-house experience was not effectively used to prevent recurrence
A4B1C09 - Management Problem; Management Methods Less Than Adequate (LTA); Corrective action for previously identified problem or event was not adequate to prevent recurrence
A4B4C12 - Management Problem; Supervisory Methods LTA; Contact with personnel too infrequent to detect work habit / attitude changes


22. Description of Cause:

The causes of this event were determined using Root Cause Analysis utilizing Event and Causal Factor Charting and Barrier Analysis methodologies as defined in the ESHQ Manual, Quality Administration, TFC-ESHQ-Q_ADM-C-11, "Root and Common Cause Analysis and Corrective Action Planning."


SUMMARY

Performance in the area of Lockout/Tagout (LOTO) by Washington River Protection Solutions (WRPS) personnel and subcontractors has not met the expectations established by management. The number and significance of the LOTO performance issues have been increasing in the last six months resulting in increasing risk to personnel safety. Due to the consequences that can result (injury or death to personnel) from poor LOTO performance, it is imperative that performance be flawless.

A Root Cause Team (Team) performed an analysis on the aggregate of LOTO issues from January 2009 to September 2010 and identified that verifications were not being performed with precise and exacting standards (i.e., lack of rigor was a common causal factor associated with all issues). As a result, the 56 process barriers inherent to the governing procedure DOE-0336, "Hanford Site Lockout/Tagout," did not identify errors in initial LOTO planning and Tagout Authorization Form (TAF) authorization. A review of training and governing procedures did not identify weaknesses in these areas. Personnel were provided sufficient knowledge and procedural guidance to perform LOTO functions successfully.

Analysis of past issues, review of observation data, and interviews revealed that the standards and expectations provided in training, procedures, and previous communications were not being reinforced in the field to ensure continuity of performance (i.e., management oversight). Insufficient field oversight, in terms of coaching and feedback targeted specifically on LOTO, has allowed the degradation of good behaviors by not providing an environment that drives consistent and reliable performance. Additional causal factors included the performance of the LOTO preparer and technical reviewer. The majority of errors occur in these roles, which are to be expected since these functions work primarily in knowledge space, but improvement is still needed to reduce the number of errors and minimize challenges to the remaining barriers in the process. The remaining functions of the LOTO process are primarily rule-based and thus, the likelihood for error should be minimal provided the rules are followed, which has not always been the case.

A review of operating experience and benchmarking revealed that other contractors have experienced similar issues with hazardous energy control and have implemented good practices to improve performance. Several of these good practices would improve LOTO performance at WRPS if implemented.

During the analysis, it was evident that an individual has not been assigned with the specific responsibility to oversee the health of WRPS's LOTO Program. This combined with the inconsistent communication and reinforcement of expectations and not implementing good practices utilized by other contractors or facilities were determined to be the root causes for the adverse trend.

Contributing factors to the recurrence of LOTO issues involved poor implementation of Contractor Assurance Systems at identifying and preventing recurrence of LOTO problems and events. Monitoring tools that help identify degraded precursor performance for management to act upon were not adequately developed. Less than adequate (LTA) resolution of past events has resulted in recurring LOTO issues; past cause analyses lacked depth and implemented weak corrective actions. Another causal factor was the authorization of numerous personnel to perform LOTO functions. Fifty percent of the authorized personnel did not routinely perform LOTOs. This environment was not conducive to maintaining a proficient team of individuals (mainly involving qualified workers).

The Team identified a concern with initial hazardous energy identification during the work control job hazard analysis process. Although few in number, failure to identify the hazards needing control resulted in circumvention of the LOTO process and the numerous verification barriers designed to prevent consequential events, which dramatically increased the risk to personnel safety. This analysis focused mainly on LOTO process performance. An in-depth analysis of the work control job hazard analysis process was not performed; Problem Evaluations request (PER) WRPS-PER-2010-2838 was initiated to analyze performance in this area.

The Team developed corrective actions that will develop an environment of continuous learning for personnel performing LOTO functions. The corrective actions include assignment of a management position responsible for the health of WRPS's LOTO program, implementation of a formal process that monitors and identifies LOTO precursor behaviors and targets management observations (to coach/reinforce standards/expectations learned in training), and consistent/continuous communication on the importance of hazardous energy control.


CAUSES

Direct Cause

The direct cause was determined to be lack of rigor in the performance of LOTO evolutions. (A4B1C01) [DC-01]

This lack of rigor is evident by the increasing number of LOTO performance issues. The common cause was further evaluated to determine the root cause for the adverse trend in LOTO performance. 38 PERs document LOTO performance issue resulting from inattention to detail and or mental lapses. The number and significance of the LOTO performance issue have been increasing in the last six months.


Root Cause

The first root cause was determined to be the WRPS LOTO program roles, responsibilities, accountabilities, and authorities for is LTA. (A4B1C01) [RC-01]

The roles, responsibilities, accountabilities, and authorities for the WRPS LOTO program is LTA in that a single point of authority to ensure the health of the LOTO program was never assigned. Many issues from past events involve administrative documentation and isolation determination/verification errors. LOTO process barriers (i.e., verifications) have not been effective in identifying errors. LTA rigor is being applied throughout the LOTO process during the numerous verification and action steps to ensure errors are identified and corrected early in the process. Additional oversight for complex/error likely LOTO planning neither identified nor provided.

Based on interviews, the expectation (established per Base Operations Shift Instruction dated May 2, 2009) that a copy of DOE 0336, "Hanford Site Lockout/Tagout," be in hand to ensure all steps are followed when performing LOTO evolutions has not been consistently met.

The second root cause was determined to be previous industry or in-house experience was not effectively used to prevent recurrence of LOTO issues. (A4B1C06) [RC-02]

Management has not consistently communicated LOTO performance expectations nor created the systems necessary to prevent future degradation of performance levels. Precursor performance issues were not recognized.

Interview and observation data substantiates the lack of management oversight in the area of LOTO performance. Thus, reinforcement and coaching of performance expectations was not occurring. 971 observations were recorded since March 2010 of which only 12 (1.2%) involved LOTO performance; LOTO was not the focus, but rather a noted comment aside from the main activity. Four observation write-ups identified a delta with no evidence a PER was written to capture any of the issues.

A review of external operating experience and benchmarking conducted during this analysis identified best practices for improving LOTO performance not implemented at WRPS.

LOTO Lessons Learned were distributed for information via emails and for use in briefings, but evidence could not be found of any formal action implemented as a result of a lessons learned. It should be noted, that (in general) the Lessons Learned extracted from the Hanford Information and Lessons Learned Sharing (HILLS) database do not provide actionable recommendations other than to ensure expectations are reinforced/communicated.


Contributing Causes

The first contributing cause was determined to be the population of assigned LOTO personnel was not conducive to maintaining a proficient team of individuals. (A4B4C12) [CC-01]

Of the 153 personnel authorized to perform LOTO functions, only 78 (50%) actually perform LOTOs and maintain hands on proficiency.

The second contributing cause was determined to be negative LOTO trends were not acted upon until after reportable events occurred. (A4B4C12) [CC-02]

Implementation of Contractor Assurance Systems at the onset of contract transition to detect adverse LOTO performance at the precursor level was LTA. Contractor Assurance Systems identified negative LOTO trends, but not until after reportable events occurred. Performance of LOTO functions in the field were not being monitored, thus degrading performance went unnoticed until events occurred. Scheduled assessments were focused on administrative compliance and did not include performance evaluations.

Conditions Adverse to Quality

The first condition adverse to quality (CAQ) was determined to be hazardous energy was not identified during initial work planning or work order scope change in a few instances. [CAQ-01]

Three PERs identified unexpected discovery of hazardous energy. Although the number of issues related to hazardous energy identification are few in number compared to LOTO process issues, this is of concern because 1) failure to identify hazardous energy during work planning or scope change circumvents the LOTO process for isolation and control, 2) risk to personnel increases dramatically, and 3) a review occurrence reports indicates an adverse trend in hazardous energy control at the Hanford site.

The scope of this analysis was focused on problems with the identification and control of hazardous energy boundaries per the LOTO process and the initial identification of hazardous energy through the work control and Job Hazard Analysis process were not thoroughly evaluated. Thus, additional analysis of the work control process is warranted (WRPS-PER-2010-2838 was initiated to address this issue).

The second condition adverse to quality was determined to be resolutions for past problems/events were ineffective in preventing recurrence of LOTO issues. (A4B1C09) [CAQ-02]

Cause analyses lacked depth and stop at the individual. Corrective actions were weak; mainly involved coaching the individual, lessons learned, or a brief. The Office of River Protection (ORP) identified less than adequate performance in Corrective Action Program implementation in 2010.


ANALYSIS

The Team performed an analysis of the barriers inherent in the LOTO process as specified in DOE-0336 and mapped instances of failed/circumvented barriers to determine commonality of issues and common causal factors. Information reviewed during this analysis included the following:
* 38 WRPS LOTO performance issues
* 26 Hanford occurrence reports on hazardous energy control
* LOTO process flow
* Interviews (managers, operating engineers, controlling organization administrator (COA), controlling organization qualified workers, and authorized workers)
* Lessons learned related to LOTO issues
* Benchmarking of DOE Hanford contractors LOTO performance
* Targeted LOTO field performance observations
* Management Observation Program (MOP) and Work Site Visit (WSV) data
* LOTO training provided by Volpentest HAMMER Training and Education Center
* Internal and external operating experience

The process for hazardous energy control was mapped to identify common areas of non-compliance and common barrier failures/breaches. In all, the hazardous energy control process associated with the LOTO performance issues contained 100 steps from hazard identification, to performing work, and changing scope. The steps include 45 action steps and 56 barrier steps (5 error prevention tool barriers and 51 verification barriers) to ensure hazardous energy is identified and isolated/controlled.

The majority of errors occurred during LOTO development by the COA (preparer). This is to be expected due to LOTO preparers work in knowledge space and the likelihood for error is higher. The same is applicable to the LOTO technical reviewer who has the majority of errors related to barriers failing to identify and correct LOTO errors and it is for this reason numerous verification are imbedded in LOTO procedures. Based on benchmarking interview with other Hanford Site contractors, improvement can be gained by establishing a highly proficient LOTO work team that has a thorough knowledge of systems and hazardous energy control. This should be supplemented with tools to guide the planning and hazard identification/isolation process using grade approach based on complexity and risk.

Due to the increased likelihood for error with the initial planning of LOTOs, reliance on the remaining process barriers being performed flawlessly is imperative. These process barriers are rule-based and should be performed by applying precise and exacting standards similar to pilots in the airline industry or plant operators in the nuclear power industry. The human nature of the errors, inattention to detail, mental lapse, etc., combined with the facts the LOTO procedure and training provides sufficient detail for performance (base on procedure/lesson plan reviews and benchmarking of Hanford Site contractors) indicate a lack of rigor in the field. This is common throughout all the LOTO events and in precursors issues documented in the PER system.

Causal factors affecting behaviors associated with rigor include sufficient training to perform the task, clear and concise procedural guidance, and communication of standards and expectations by management. Interviews conducted with the LOTO trainers, a review of the LOTO lessons plans, and observation of training indicated that LOTO training provides the adequate instruction (including performance standard for verifications) for the trainee to successfully perform LOTO evolutions. A detail review of DOE-0336 discussions with members of the Hanford Site Lockout/Tagout Committee and interviews with Hanford Site contractors indicated that the procedure provided sufficient systematic guidance that if followed, would ensure isolation and control of hazardous energy at the job-site.

The review of the communication/reinforcement of standards and expectations in the field identified a major gap in this area. MOP and WSV data and interviews with WRPS personnel identified the following:
* Of 971 observations recorded since March 2010, only 12 (1.2%) involved some aspect of LOTO performance
* The majority of personnel interviewed stated management presence has not been noticed in the field when LOTO evolutions are being performed
* The majority of corrective actions for past events involved briefings, coaching of the individual, issuance of a Lessons Learned, performance of observations, or initiation of tailgate. Without periodic reinforcement of expectations and standard, this form of communication is short-lived. For example, the expectation that "personnel performing LOTO functions shall have a copy of the procedure with them to ensure they are following all of the required steps" was communicated in WRPS Base Operations Shift Instruction in March 2009. But interviews, observations, and analyses indicate that this expectation is not being met or reinforced.

The lack of reinforcement of standard and expectations has resulted in inconsistent performance overtime. The lack of management oversight (in terms of coaching and feedback) has allowed the degradation of good behaviors by not providing an environment that drives consistent and reliable performance. This causal factor is common to all LOTO performance issues. Additionally, per an interoffice memo (WRPS-1001759, Authorized Lockout/Tagout Individual Designation), 153 LOTO qualified WRPS personnel are "authorized" to perform LOTO functions (prepare, review, authorized, hang, and verify Danger Do-Not-Operate lock and tags), but only 50 percent actually perform LOTOs. The 50 percent of "authorized" personnel perform LOTOs infrequently and do not develop the same proficiency as the core group. Improvement can be gained by reducing the number of "authorized" personnel to the core group to eliminate the error likely situation of a non-proficient individual be assigned to a LOTO. The following is a breakdown of personnel authorized to perform LOTO functions:
* LOTO Preparer, Authorizer, Prepare 8 Criteria Checklist-38 (80% actually perform this function)
* LOTO Technical Reviewers-19 authorized (95% actually perform this function)
* Controlling Organization Qualified Worker-111 (27% actually perform this function)

Contributing to this organizational ineffectiveness is less than adequate monitoring tools that provide management the necessary information to identify and act upon degrading performance with important-to-safety processes. For example, the LOTO performance indicators do not differentiate between LOTO precursor behavior issues and events that are more significant. Performance metrics used in the nuclear power industry that are based on a LOTO error criteria are more conducive to identifying low level precursor behaviors before consequential events occur. Additionally, performance metrics for observation data related to LOTO performance does not exist. Performance indicators that make increases in low-level precursor behaviors (poor behaviors) more transparent (similar to one used in the nuclear power industry for Clearance Orders metrics) can improve management oversight.

Three instances of hazard identification during work planning were identified during this analysis. Although few in number in comparison to LOTO process errors, this is of concern due to failure of identify the hazards needing control results in circumvention of the LOTO process and its numerous verification barriers designed to prevent consequential events. Errors in initial hazard identification have a high probability of revealing themselves in the field after work has start thus greatly increasing the risk to personnel. Due to the commonality of performance issues and causal factors, this analysis mainly focused on the LOTO process as defined in DOE 0336, and did not perform an in-depth analysis of the work control job hazard analysis process. Due to the elevated risk to personnel associated with poor job hazard analysis, a separate evaluation is warranted to address extent of cause in this area.


23. Evaluation (by Facility Manager/Designee):

Based on this lockout/tagout (LOTO) root cause analysis (RCA) and corrective actions identified, the issues contained in this report are extremely important to resolve and continue to manage to reduce recurrence. The scope of LOTO with respect to conducting work within the Department of Energy complex and the nuclear industry is important and requires continual improvement, implementation of lessons learned, and attention to rigor. The implementation of the corrective actions associated with this LOTO RCA will have net positive effect on plant systems, equipment, and programs.


24. Is Further Evaluation Required?: No


25. Corrective Actions
Local Tracking System Name: Problem Evaluation Request

        1.
Develop a Charter (including implementation schedule/plan) based on this Lockout/Tagout (LOTO) root cause analysis and assign a management position with the authority to own the Washington River Protection Solutions' LOTO Program, to include the following responsibilities:
* LOTO program implementation
* Reports to Facility Works Manager
* Single authority for approving Controlling Organization individuals and additional oversight needs
* Review of LOTO performance data (Performance indicators, trend data, observations results, and assessments) and develop actions for degrading performance
* Review and implementation of lessons learned
* Establish a continuous learning environment for LOTO qualified personnel to include feedback from lessons learned, trend analysis, and performance indicators [RC-01]

Objective Evidence: Approved Charter (including implementation schedule/plan) and management individual assigned.

Actionee: Wilkinson, Robert E
Target Completion Date: 01/14/2011 Tracking ID: WRPS-PER-2010-2243.1
        2.
Develop/distribute executive communications that promote and reinforce the zero error expectation for Lockout/Tagout safety systems. This action will be assumed by the Lockout/Tagout authority once assigned. [RC-02]

Objective Evidence: Issuance of the first executive key safety systems communication.

Actionee: Ellis, Martin W
Target Completion Date: 11/19/2010 Tracking ID: WRPS-PER-2010-2243.2
        3.
Authorize a highly competent workforce to perform Lockout/Tagout (LOTO):
* Select a small group of employees highly proficient in LOTO to conduct all LOTO actions based on LOTO Authority direction
* Reissue authorization letter(s) reflecting the LOTO group selected [CC-01]

Objective Evidence: Approved authorization letter.

Actionee: Reynolds, Tammy R
Target Completion Date: 01/14/2011 Tracking ID: WRPS-PER-2010-2243.3
        4.
Update the assessment plan to provide a systems approach to Lockout/Tagout (LOTO) performance assessment:
* Schedule specialty assessments to cover all key LOTO systems on a 3-year cycle
* Include in the plan performance evaluations based on rollup of observation data, Problem Evaluation Requests, and surveillances [CC-02]

Objective Evidence: Assessments entered on the integrated schedule with a clearly defined purpose and scope that covers all key LOTO systems.

Actionee: Ellis, Martin W
Target Completion Date: 12/03/2010 Tracking ID: WRPS-PER-2010-2243.4
        5.
Develop and implement a directed Lockout/Tagout (LOTO) assessment plan, including assessment criteria and schedule, with focused planning/field observations based on performance data reviews.
* The assessment schedule is to be implemented following plan approval and continued for six months after permanent corrective actions implementation or completion of the End Point Assessment.
* At the completion of the initial directed LOTO assessments, the scheduling of future directed LOTO assessments will be directed based a review of performance data by the assigned Lockout/Tagout authority. [CC-02]

Objective Evidence: Approved schedule developed with assigned responsibilities.

Actionee: Ellis, Martin W
Target Completion Date: 12/03/2010 Tracking ID: WRPS-PER-2010-2243.5
        6.
Implement Lockout/Tagout (LOTO) program trending/tracking processes. Develop leading performance indictors similar to those developed by the commercial nuclear industry. [CC-02]

Objective Evidence: Performance indicator developed and included in company Performance Indicators.

Actionee: Ellis, Martin W
Target Completion Date: 12/03/2010 Tracking ID: WRPS-PER-2010-2243.6
        7.
Develop company-level Lockout/Tagout program health indicator that takes into account overall performance (performance indicators, trend data, observations results, and assessments) to be reviewed by the Executive Safety Review Board. [CC-02]

Objective Evidence: Lockout/Tagout program health indicator developed.

Actionee: Ellis, Martin W
Target Completion Date: 01/14/2011 Tracking ID: WRPS-PER-2010-2243.7
        8.
Implement a reporting procedure similar to Mission Support Alliance's "Lock and Tag Reporting" (MSC-PRO-18090) applicable to all key safety systems. Provide response actions for all precursor events focused on instilling the importance of zero errors (disqualification, critiques, lessons learned presentations, etc.). [RC-02]

Objective Evidence: Issuance of procedure that specifies the reporting guidance and response actions for precursor events.

Actionee: Ellis, Martin W
Target Completion Date: 01/14/2011 Tracking ID: WRPS-PER-2010-2243.8


26. Lessons Learned:

It is important to continue to benchmark and review other Department of Energy (DOE) and nuclear industry contractors/operators for common successful themes in Lockout/Tagout (LOTO) activities such as work scope definition/hazard identification, isolation boundary determination, effective validation, Conduct of Operations, performance trends closely follow DOE complex-wide trends, close Agreement on Traits of an effective LOTO program, management leadership, effective work control, highly competent workers, and critical oversight.
These elements need to be continually reviewed and lessons learned shared across the complex in order to continue to strengthen LOTO processes.


27. Similar Occurrence Report Numbers:

EM-RL--PHMC-GENERAL-2006-0002
EM-RP--BNRP-RPPWTP-2005-0027
EM-RP--BNRP-RPPWTP-2007-0007
EM-RP--BNRP-RPPWTP-2008-0025
EM-RP--CHG-TANKFARM-2008-0003
NA--LASO-LANL-LANL-2005-0005
SC--BSO-LBL-OPERATIONS-2010-0008


28. User-defined Field #1:

SIGNIFICANCE= 'R' AND NOC LIKE '2C2' AND ALLNARRATIVES.TEXT= 'ENERGY'

29. User-defined Field #2:

Problem Evaluation Request WRPS-PER-2010-2243


30. HQ Keyword(s):

01A--Inadequate Conduct of Operations - Inadequate Conduct of Operations (miscellaneous)
01B--Inadequate Conduct of Operations - Loss of Configuration Management/Control
01D--Inadequate Conduct of Operations - Missed/Late Surveillance
01G--Inadequate Conduct of Operations - Inadequate Procedure
01K--Inadequate Conduct of Operations - Lockout/Tagout Noncompliance (Electrical)
01L--Inadequate Conduct of Operations - Lockout/Tagout Noncompliance (Other)
01M--Inadequate Conduct of Operations - Inadequate Job Planning (Electrical)
01N--Inadequate Conduct of Operations - Inadequate Job Planning (Other)
01O--Inadequate Conduct of Operations - Inadequate Maintenance
01P--Inadequate Conduct of Operations - Inadequate Oral Communication
01Q--Inadequate Conduct of Operations - Personnel error
01R--Inadequate Conduct of Operations - Management issues
08H--OSHA Reportable/Industrial Hygiene - Safety Noncompliance
12I--EH Categories - Lockout/Tagout (Electrical or Mechanical)
14A--Quality Assurance - Program Deficiency
14C--Quality Assurance - Quality Improvement Deficiency
14D--Quality Assurance - Documents and Records Deficiency
14E--Quality Assurance - Work Process Deficiency


31. HQ Summary:

On August 19, 2010, Washington River Protection Solutions determined that a review of Lockout/Tagout (LOTO) program implementation that was completed on July 17, 2010 indicated a number of associated LOTO issues reported since September 2009. The review identified human errors, administrative deficiencies, and inadequate isolation boundaries. The 21 events noted in the review are symptomatic of a larger company or programmatic issue, which, if left unchecked, would represent an unacceptable risk of a serious event. As a result of this unacceptable trend and programmatic issue, Washington River Protection Solutions management determined that this series of events are recurring and declared a Significance Category "R" report. Immediate actions include all LO/TO work being suspended until safety briefings and a safety stand-down are conducted.


32. DOE Facility Representative Input:


33. DOE Program Manager Input:


34. Approvals:
 
Approved by: Ellis, Martin W, Facility Manager/Designee
Date:12/02/2010
Telephone No.:(509) 373-4696
 
Approved by: SONDAG, JOSEPH M, Facility Representative/Designee
Date:12/04/2010
Telephone No.:(509) 373-9179