The problem of information overload in crisis situations and how to distinguish the vital input from the secondary is widely known, though not yet fully solved. Those who have ever been present in the control room of a modern power plant during a turbine trip or other emergency will readily understand the focus of Genelba’s work in this area. How long does it take on average for a trained operator to realise what is really going on? How many messages should be read or interpreted from the alarms screen? How many graphic displays should be scrolled in order to gather information on the current condition of the main equipment?
In the late 80s, the introduction of distributed control systems in power plants represented a breakthrough in the quality of their automatic controllers. Advanced control strategies – never thought before – started to be implemented, leading to a significant improvement in efficiency and reliability. Even so, as these important advances were introduced, users were becoming aware of and could detect weaknesses in the new technology, mainly related to the man-machine interface described above. Although some changes have been introduced in the field since then, there are still great opportunities for improvement.
Contributing factors
Some DCSs design features contribute to an increase in the size of the problem:
•Owing to the high level of automation, which is without a doubt very beneficial, the operator does not have a pre-eminent role during standard everyday operations. But when there is an emergency, he must take a swift action without any ‘warming up’ period since he has almost no time to solve the problem.
•The operator does not have all the controls and readouts displayed simultaneously. Moreover he often has to check several graphic displays before taking control of a certain piece of equipment.
•The tendency is to incorporate voice messages and alarms in the panel so as to keep the operator informed of any variable change that is out of sight; adding alarms and messages to a DCS can after all be done at very low cost and is easily accomplished. But when there is a turbine trip or a major problem, the operator is flooded with dozens of alarms and messages. This excess of information usually leaves the operator confused and stunned or, even worse, lead him to make a faulty decision. This phenomenon is called ‘alarm flooding’ or ‘alarm inflation’.
It should be added that, self-evidently, all the alarms cannot appear on the screen at the same time. But this may lead to the submerging of an important alarm – the page scrolling used in most DCSs sometimes takes the important alarms out of the active visual display and shows less important messages.
•Lastly, once the important alarms are spotted, their various possible causes still have to be analysed. From that moment on, in order to make the right decision the plant operator begins to think very fast under the pressure of the short time available. At this point, the second factor described above plays a prominent role as the operator uses up valuable time scrolling through the graphic displays.
In most cases like this the likely result is GAME OVER!!! But in most power plants, it is common to hear anecdotes about “heroic” operators whose actions, during emergencies that occurred long ago, avoided plant trips. However, these sorts of actions do not by any means make up part of a systematic and effective way of dealing with difficult situations of that kind.
Although there is a great deal of data available to control systems, only part of it is processed. The latest technology allows better data management. Genelba’s project objective is to generate valuable knowledge in real time and make it available for the plant operator.
Expert systems overview
‘Expert systems’ makes up a basic component of what is generally known as artificial intelligence, a discipline that studies the way the human mind thinks and reasons when solving problems and making decisions. Through a thorough study of these mental processes and models that emulate them, AI engineers seek to build computer systems that resemble, to a great extent, human behaviour. Besides expert systems, AI also includes more sophisticated fields such as artificial neural networks, robotics and ‘fuzzy’ logic.
Expert systems constitutes a highly effective AI approach to problem solving. It comprises computing systems that make decisions in the province of the system in question in much the same way as a human expert would. These systems have been successfully applied in business, medicine, science and engineering.
A human expert in a certain field has knowledge and abilities that other people do not, then the expert can solve a problem quicker and more effectively. Expert Systems are also designed to cover the problem of a particular domain. Within this domain, and in order to find out a solution for a problem, the expert system is able to make inferences in much the same way as a human expert would. The human mind, among other features, has the ability to learn new concepts without changing its structure or process and even without affecting previously stored facts. The new information is absorbed and awaits any future use.
Like the human mind, expert systems are able to separate the knowledge base from the methodology used to solve problems. Thus, they can be constantly enriched with new knowledge without changing their structure or their information processing model.
Expert systems consist basically of two main components, their knowledge base and their ‘inference engine’. The knowledge base comprises the expert knowledge and consists of a large set of Rules. A rule is a minimum unit of knowledge and has an “IF ….THEN” structure. The inference engine solves problems and is independent of the available knowledge. Whenever it is necessary to investigate the root cause of a problem, the inference engine refers to the knowledge base, selecting the rules applicable to the specific case and arranging them in sequence.
Genelba’s expert system
Expert systems technology has been employed in Genelba as an approach capable of dealing effectively with power plant operating contingencies. The system developed is called SERO (from the Spanish for ‘expert system of operating recommendations’).
Whenever there is a critical event, SERO explores the applicable Rules to make an immediate diagnosis of the root cause and report it to the plant operator. It also provides operating recommendations for that specific contingency.
SERO has been designed, in view of the great amount of visual information received by plant operators in problematic situations, to provide “friendly” verbal announcements, for both root causes and operating recommendations. This is a new departure – the sense of hearing has not been engaged in the running of industrial plants with DCSs, other than to detect the “beeping” or horns used to signal alarms, .
Events generated by SERO have been categorised into different types:
‘NOTICES’ are messages coming from the long list sent by the DCS during an emergency. SERO selects the relevant messages on the basis of the current situation of the plant. As with all other events generated by the system, voice announcements are used to notify the operator about NOTICES, highlighting their relevance over the avalanche of information produced by the DCS.
‘TRIPS’ are also events that come from the DCS list. The SERO system, which informs the operator about trips, only considers the first turbine trip that occurs. These events also trigger a process to determine the root cause.
‘SHUTDOWNS’ also appear on the DCS messages list and they activate some of the turbines’ or HRSGs’ shutdowns programs. In certain cases, depending on the plant operating situation, the root cause investigating process begins.
‘INFERENCES’ are events that derive exclusively from the SERO system. Inferences are the conclusions reached as a result of the searching process for the root causes of problems. These inferences are drawn not only from trips and shutdowns events, but also from certain DCS events or even a combination of them, according to the Rules set for each case.
‘RECOMMENDATIONS’ are events created by the SERO system. Once the root cause of a problem is found and communicated, the system also decides, and informs the operator of, what it considers to be the best course of action.
‘SAFETY’. Whenever there is a trip, the SERO system makes use of Safety events to flag up any potentially unsafe situation during the shutdown of the turbine. For example – inadequate supply to the LV or MV busbars, or failure in the activation of the turning gear (??????). DOES THIS MEAN DURING A LONG TERM SHUTDOWN WHEN THE SHAFT HAS TO BE TURNED??
The first word in each voice message is always one of these classification headings, so that plant operators immediately know the type of message that is to follow.
When an unusual event occurs in the plant, SERO assists the operator by providing relevant information generated from the plant data. A screen displaying voice messages – but in writing – has been included just to allow the operator to track the conclusions and events generated by SERO at any time. However, the plant operator does not need to look at the screen during the contingency. The REW and PLAY functions have been excluded from voice messages since they are considered dangerous for the operator. On a second listening, messages already heard can be taken as new incoming messages.
SERO also features a communication system that reports important events to stakeholders. When there are serious problems like turbine trips, SERO sends e-mail messages with the root cause to managers and plant specialists. These e-mails are also sent to their mobile phones, so as to keep them informed wherever they may be. The messages distribution list includes analysts from the trading floor and their managers. Then, SERO also assists in the real-time decision-taking process of the business, allowing rescheduling of fuel sales and redelivering of electric power.
SERO periodically sends e-mails carrying information about the general situation of the plant and also dispatch conditions. These messages are automatically diverted to mobile phones during weekends.
[SERO block diagram goes about here]SERO function display
Figure 1 shows a block diagram for the SERO system. On the left the graphic also depicts the existing Plant Systems. SERO is basically made up of four programs: the Triggers Processor, the Inference Engine, the Events Presenter, the Messager.
The Triggers Processor performs two main functions. On the one hand, it is in charge of selecting the relevant events generated by the DCS, extracting them from the long list of events and highlighting them by sending voice messages to the operator. For this purpose, the program sets an appropriate Event Number in the Events Buffer. On the other hand, the program spots any abnormality that requires determination of its root cause and the applicable operating recommendations. When this occurs, the program activates the Inference Engine.
The Inference Engine is activated by the Triggers Processor. When this occurs, the Inference Engine makes use of the knowledge expressed in the Rules through the Expert Systems techniques described above to determine the root cause and recommendations. Both of them are codified, so in the end the Inference Engine sets the appropriate Event Numbers in the Events Buffer.
The Events Presenter repeatedly consults the Events Buffer so as to identify any new event. When the program finds an Event Number in the Events Buffer, it makes use of this number to access the Events data base to find the sound file associated with the event in order to run it and send the appropriate Voice Message; and the text associated with the event to be shown on the Messages display along with the time of occurrence.
Afterwards, the Events Presenter stores all the events occurred in the Historian file, allowing it later to check the historical sequence of events.
The Messager communicates important plant events to stake holders through e-mail messages. Two kind of message are sent:
1. Important events occurred in the plant. Like the Events Presenter, the Messager permanently consults the Events Buffer to check if a new event has occurred. When it finds an event number, the program consults the Events Data Base to see if this event has associated e-mail addresses. If there is a distribution list associated with the incoming event, the program notifies the people on this list about the event. This list normally includes mobile phones as well as office e-mail addresses. messages would be sent to plant managers, trading floor personnel and others whose jobs are related to the event in question.
2. Periodic data distribution. The Messager sends daily e-mail messages containing information about the plant’s operational status. They are sent at pre-arranged times and the data include turbine output power, gas consumption, energy produced, heat rate etc. This data is taken from the Periodic Data Calculation, a program, running on a spreadsheet, that continuously extracts data from the Plant system.
Advantages of the Genelba approach
The following text outlines some of the advantages of applying Expert Systems Technology to solve power plants problems in real-time.
• Quick response. Whenever there is an anomaly, the Expert System quickly provides useful information about really significant events, in particular their root cause and operating recommendations; it does so in such a short time that it would be impossible for any human being to match it.
• Relevant information polishing. The Expert System immediately interprets what is going on, giving it the capability of ruling out all the events unrelated to the core of the problem, freeing the plant operator from an excessive amount of useless information for that problem.
• Emotionless reasoning. The Expert System does not feel any of the pressure experienced by operators in chaotic situations – the short time available to solve the problem, the opportunity to cope with a plant trip, the recriminations if they make a wrong decision – none of these factors affects the appropriate functioning of Expert Systems.
• Enhanced knowledge. The system is endowed with the working knowledge of all the plant’s experts and its operators’, information that can be usefully added during a quiet working day, free from the pressure of an emergency.
• Full-time availability of expert knowledge. The Expert System works 24/7 in the control room without any necessity for ‘warming up’. The entire information content is always available, ready for instant use.
SERO possesses, in addition to the typical characteristics of an Expert System, the special features described above and and others that endow this approach with additional advantages.
Why not use the DCS’ automation processors?
SERO developers had been thinking about an approach that made use of the DCS’ automation processors before the project started. After a thorough analysis, the final decision was to implement a support system for the operator by making use of a set of compatible PCs. But the question is frequently raised with SERO developers by interested parties coming upon Genelba’s approach for the first time. To answer it, the following main aspects can be considered.
• Cost. The automation processors of all DCSs currently used in power plants are costly compared to mass systems. The algorithms required to develop a system for determining the root cause of problems in a plant, within the automation processors, are long and complex, similar in size to the plant’s own control strategy. A significant number of additional processors would therefore be required.
• Flexibility. When working with automation processors, you lose flexibility. Even in these days when all DCSs allow on-line programming, its over-use may result in inconvenient glitches. When the plant is working, special care must be taken with these aspects since a failure in an automation processor may lead to a plant trip.
• It renders impossible the use of Expert Systems technology, and with it all that technology’s potential and flexibility. The Rules and Inference Engine models can rarely be used in an automation processor.
• There is no access to historic data. Automation processors are well suited to working with real-time data. The processing functions are designed to handle the latest data acquired by the system. But in order to pinpoint the root cause of a failure it is frequently necessary to refer to historic data, trends, and so on. This is a difficult or even an impossible challenge for automation processors.
• Man-Machine Interface. Usually there are very few opportunities to access any other MMI than the DCS MMI when working within the DCS environment.
Lessons learned
It is more than likely that many of the problems arising in power plants could be worked out through conventional programming. However, with this kind of approach, every problem requires ad-hoc programming, and if any newly learned information is to be incorporated, algorithms have to be changed, a process which risks introducing disturbances in other sections of the program. On the other hand, if the operator wants to build a flow diagram to identify the root cause of an emergency, he will find it has too many lengthy branches, rendering the whole process very slow and laborious. The rule-based knowledge approach of Expert Systems avoids this time consuming situation. It requires only the setting of applicable rules, and the Inference Engine can do the ramining work of solving the problem.
SERO is a powerful approach. The use of Expert Systems technology has proved beneficial for Genelba, achieving positive results rapidly, owing to the fact that the system can be commissioned with a minimum of built-in knowledge and then expanded and developed.
The introduction of voice messages to transmit the results to power plant operators was without any doubt an excellent idea, although at the beginning he may feel as if he is taking a main role in a frantic cartoon show.
The project was set up on an experimental basis and can be considered a tailored system, as it is 100% home-made. However, this does not render the system powerless or inefficient. The authors of the present work do not mean to imply that this is the standard model to follow, but that in the short term the users of DCSs should focus on these operating aspects and that in the near future all distributed control systems should include an Expert system to be used in the real-time operation of a plant.
