exceptionHandlingDesign.dox

/**
\page exceptionHandlingDesign Exception Handling Design
\section gen_idea General Idea


Exceptions must be handled by ApplicationCore in a way that the application developer does not have to care much about it.

In case of a ChimeraTK::runtime_error exception the framework must catch the expection and report it to the DeviceModule. The DeviceModule handles this exception and preiodically tries to open the device. In case of several devices only the faulty device is blocked. Even if a device is faulty it should not block the server from starting.

If an input variable is in the error state, it sets the DataValidity flag for its DataValidityProparationExecutor (see \link spec_dataValidityPropagation \endlink) to faulty and the flag is propogated appropriately. After the exception is cleared and operation returns without a data fault flag, set DataValidity flag to ok. Furthermore, the device must be reinitialised automatically and also recover the values of process variables as the device might have rebooted and the variables have been re-set.


<b>1. Genesis</b>

- a (removed)
- b. An initailisation handler can be added to the DeviceModule in the user code. Initialisation handlers are callback function which will be executed when a device is opened for the first time and after a device recovers from an exception, before any process variables are written.
- c. Initial values must be correctly propogated after a device is opened. See \link spec_initialValuePropagation \endlink.
- d. (removed)
- e. A ChimeraTK::ExceptionHandlingDecorator is placed around all ChimeraTK::NDRegisterAccessors which connect a device to a ChimeraTK::ApplicationModule or fanout. (*)
- f. (removed)
- g. By default a recovery accessor is added for each device register when it is obtianed. These recovery accessors are used to correctly set the values of variables when the device is opened for the first time and after a device is recovered from an exception. (*)
- h. A ChimeraTK::ExceptionHandlingDecorator for an input knows its DataValidityProparationExecutor, which lives in the ApplicationModule or fanout that reads the input. Like this it can propagate the
     dataValidity flag. Outputs do not send DataValidity faulty in case of exceptions (see \link spec_dataValidityPropagation \endlink).
- i. Write should not block in case of an exception for the outputs of ThreadedFanOut / TriggerFanOut. (*)

<b>2. The Flow</b>

- 2.1. The application always starts with all devices as closed and intial value for deviceError.status is set to 1.

- 2.2. Until the 2.4 is done (device is available for the first time), all the read operations are blocked (*). Writes behave like 2.5.1.3.

- 2.3. The DeviceModule tries to open the device in a separate thread. (*)

- 2.4. Device is opened successfully for the first time.

  - 2.4.1. Device is initailised by iterating initialisationHandlers list. If there is an exception go back to 2.3.

  - 2.4.2. The pending write operations, stored in recovery accessors, are written. If there is an exception go back to 2.3.

  - 2.4.3. deviceError.status is set to 0.

  - 2.4.4 All blocked read and write operations (from 2.2) are notified and continue.

- 2.5. When a read / write operation on the device (1.e) causes a ChimeraTK::runtime_error exception, the exception is caught. 

  - 2.5.1. Inside ExceptionHandlingDecorator / TriggerFanOut

    - 2.5.1.1. If read operation:

      - 2.5.1.1.1  The DataValidityPropagationExecutor is informed that there was a device error. (*)

    - 2.5.1.2. The error is reported to the DeviceModule (*)

    - 2.5.1.3. Calling operation :
              - write : blocks until the device is recovered.
              - read : For the first "blocking" read, the call returns with invalid data and then remembers that it is in an exception state. The calling module thread will continue and propagate the data invalid flag. The second call will finally block.
              - readNonBlocking / readLatest: will always return with data invalid flag.
              - writeWithoutErrorBlocking: just returns (*) 

  - 2.5.3. The exception is received by DeviceModule::handleException() which is running in the DeviceModule thread.

    - 2.5.3.1. deviceError.status will be set to 1. From this point on, all write operations must not excecute the actual write any more.

    - 2.5.3.2. Try re-opening the device until successful. (*)

    - 2.5.3.3. Device is re-opened successfully and isFunctional() returns true.

    - 2.5.3.4. Device is reinitalisied through initialisationHandlers. If exception is thrown go back to 2.5.3.2 (*)

    - 2.5.3.5. Process variables are written again using the list of recovery accessors. Recovery accessors are not written to the device if it has not received its initial value yet. If exception is thrown go back to 2.5.3.2. (*)

    - 2.5.3.6. All threads that were blocked after calling DeviceModule::reportException() for this device are notified. (from 2.5.1.3)
               Non-blocking writes and write operations from other accessors are now allowed to try write attempts (see 2.5.3.1).

    - 2.5.3.7. deviceError.status is set to 0.

  - 2.5.4. Continue blocking functions inside ExceptionHandlingDecorator

    - 2.5.4.1. If read operation:

      - 2.5.4.1.1.  Before unblocking inform the DataValidityPropagationExecutor that the device error has gone.

      - 2.5.4.1.2. The original read operation is executed. If an exception occurs go back to 2.5.1.

    - 2.5.4.2. A write operation just returns. The recovery accessor has already taken care that the data was written to the device.


<b>3. (*) Comments</b>

- 1.e. In addition there can be recovery accesors for the same variables, which are not decorated. They are not directly seen by the ApplicationModule and the fanouts.
- 1.g. Output accessors can have the option not to have a recovery accessor. This is needed for instance for "trigger registers" which start an operation on the hardware. Also void registers don't have recovery accessors.
- 1.i. The specification for initial value propagation (\link spec_initialValuePropagation \endlink) also says that writes ApplicationModules don't block before the first successful read in the main loop.

- 2.2. Because no "value after constuction" must be propagated and we have to wait for an initial value, see \link spec_initialValuePropagation \endlink.
- 2.3. This thread is responsible for (try) opening of the device for the first time and (try) re-opening of the device in case of an exception.
  Before the DeviceModule thead is started, all recovery accessors must be registered at the DeviceModule.
- 2.5.1.1.1 incrementDataInvalidCounter() is called. See \link spec_dataValidityPropagation \endlink.
- 2.5.1.2 via ChimeraTK::DeviceModule::reportException()
- 2.5.1.3 Writing to the recovery accessor has already happened before. It always is done before the acutal read is tried and an exception even is raised.
- 2.5.3.2 Although the function is called Open, to reach this point a device must have been opened at least once before, hence re-open.
- 2.5.3.4 and 2.5.3.5 Exceptions for re-initialisation and recovery will be reported once, but not if it occurs again before the device has completely recovered.
- 2.5.3.6 If a write is not executed because the device is already faulty, the recovery accessor is updated. In this case another exception notification has to be send to the DeviceModule to make sure that the recovery value is not missed (avoid race condition).

<b>Implmentation Details</b>

<b>4. Exception handling and reporting mechanism to the device module (DeviceModule).</b>

Description.


These variables are automatically connected to the control systen in this format
- /Devices/{AliasName}/message
- /Devices/{AliasName}/status

Add a thread safe function ChimeraTK::DeviceModule::reportException().
A user/application can report an exception by calling reportException of DeviceModule with an exception string. The reportException packs the exception in a queue and the blocks the thread. This queue is processed by an internal function handleException which updates the DeviceError variables (status=1 and message="YourExceptionString") and tries to open the device. Once device can be opened the DeviceError variables are updated (status=0 and message="") and blocking threads are notified to continue. It must be noted that whatever operation which lead to exception e.g., read or write, should be repeated after the exception is handled.

Implementation.
- ChimeraTK::DeviceModule


<b>5. Catch ChimeraTK::runtime_error exceptions.</b>

Description.

For a device with it's deviceError.status = 0 (see 2.4.3), catch all the ChimeraTK::runtime_error exceptions that could be thrown in read and write operations and feed the error state into the DeviceModule through the function ChimeraTK::DeviceModule::reportException().
Retry the failed operation after reportException() returns.

For a device that has been opened for the first time but has not reached 2.4.3 i.e., it's deviceError.status != 0, and it throws a ChimeraTK::runtime_error exception see 2.3.

Implementation.
- Exceptions are caught as explained in 1.e and 1.f.
- ChimeraTK::NDRegisterAccessors
- ChimeraTK::Application

<b>6. Faulty device should not block any other device.</b>

Description.

Each ChimeraTK::TriggerFanOut deals with several variable networks at the same time, which are triggered by the same trigger. Each variable network has its own feeder and one or more consumers. The trigger itself is a variable network, too. One consumer per ChimeraTK::TriggerFanOut is required.
Implementation.

- ChimeraTK::Application::typedMakeConnection()

<b>7. The server must always start even if a device is in error state.</b>

Description.

To make sure that the server should always start, the initial opening of the device should take place in the ChimeraTK::DeviceModule::handleException(), which has the exception handling loop so that device can go to the error state right at the beginning and the server can start despite not all its devices are available.

Does not fit here, but is the only place where handleException is mentioned:
- handleException() must not block.

Implementation.

- ChimeraTK::DeviceModule::handleException()


<b>8. Propogate error flag</b>

Description.

See 2.5.1.

For initial error propogation see <a href='spec_initialValuePropagation.html'>spec_initialValuePropagation</a>.

Implmentation.
- ChimeraTK::ExceptionHandlingDecorator
- ChimeraTK::TriggerFanOut


<b>9. Initialise the device</b>

Description.

The device should be automatically initialised when opened for first time (2.4.1) and automatically re-initialised after recovery (2.5.3.4).

Implementation.

A list of DeviceModule std::function is added. InitialisationHandlers can be added through construtor and addInitialisationHandler() function. When the device recovers all the initialisationHandlers in the list are executed.
- ChimeraTK::DeviceModule
- ChimeraTK::ExceptionHandlingDecorator


<b>10. Recover process variables after exception.</b>

Background.

After a device has failed and recovered, it might have re-booted and lost the values of the process variables that live in the server and are written to the device. Hence these values have to be re-written after the device has recovered.

Description.

Create a copy of accessor when writing the data to the device and use this to recover the values when the device is available again. Recovery accessor do not write if the register is never written before (2.5.3.5.).

Implementation.

- ChimeraTK::DeviceModule
- ChimeraTK::ExceptionHandlingDecorator
- A list of ChimeraTK::TransferElements is created as ChimeraTK::DeviceModule::writeRecoveryOpen which is populated in function ChimeraTK::DeviceModule::addRecoveryAccessor().
ChimeraTK::ExceptionHandlingDecorator is extended by adding second accessor to the same register as the target accessor it is decorating.
<I> Data is copied in doPreWrite(). [TBD: Do we want this behaviour?]</I>


<b> ExceptionHandlingDecorator </b>

- Device accessors must only throw in postRead and postWrite (FIXME: move text from initial value propagation spec)
- The Decorator only decorates postRead / postWrite (FIXME: conceptually, which one is the correct one?)
- The decorator provides a writeWithoutErrorBlocking() function so that even in case of exception write should return. [TBD: name of the function]

Like this the decoration also works for transfer groups and asyncronous transfers.

<b>5. Known Bugs.</b>

-  Step 2.1 The intial value of deviceError is not set to 1.

-  Step 2.2. is not correctly fulfilled as we are only waiting for device to be opened and don't wait for it to be correctly initialised.

-  Step 2.4.3. is currently being set before initialisationHandlers and writeAfterOpen.

-  Step 2.5.3.7. is currently being set before initialisationHandlers and writeRecoveryOpen.

-  Check the comment in Device.h about writeAfterOpen(). 'This is used to write constant feeders to the device.'

-  Check the documentation of DataValidity. ...'Note that if the data is distributed through a triggered FanOut....'



*/