Exceptions must be handled by ApplicationCore in a way that the application developer does not have to care much about it.
In case of a ChimeraTK::runtime_error exception the framework must catch the expection and report it to the DeviceModule. The DeviceModule should handle this exception and preiodically tries to open the device. As there could many devices make sure only the faulty device is blocked.
Even if a device is faulty it should not block the server from starting.
In case of a ChimeraTK::runtime_error exception the framework must catch the expection and report it to the DeviceModule. The DeviceModule should handle this exception and preiodically tries to open the device. As there could many devices make sure only the faulty device is blocked. Even if a device is faulty it should not block the server from starting.
Once in error state, set the DataValidity flag for that module to faulty and propogate this to all of it‘s output variables. After the exception is cleared and operation returns without a data fault flag, set DataValidity flag to ok. Furthermore, the device must be reinitialised automatically and also recover the values of process variables as the device might have rebooted and the variables have been re-set.
<b>1. Genesis</b>
- a. When DeviceModule is created it is registered with Application. (Added to a list in Application::registerDeviceModule.)
- b. An initailisation handler can be added to the device through constructor. Initialisation handlers are callback function which will be executed when a device is opened and after a device recovers from an exception.
- a. When DeviceModule is created it is registered with Application. (Added to a list in Application::registerDeviceModule())
- b. An initailisation handler can be added to the device through constructor. Initialisation handlers are callback function which will be executed when a device is opened for the first time and after a device recovers from an exception.
- c. Initial values must be correctly propogated after a device is opened. See (link for initial value specs. comes here)
- e. ChimeraTK::NDRegisterAccessor is used to access the device variables inside class Application.
- f. Class ExceptionHandlingDecorator facilitates ChimeraTK::NDRegisterAccessor in case of exception
- g. An ExceptionHandlingDecorator is placed around all NDRegisterAccessors connected to a device used inside ApplicationModule and ThreadedFanOut except initial value and recovery accesors.
- h. TriggerFanOut is using a TransferGroup which bypass the functionalitly of ExceptionHandlingDecorator hence it has to impelment the exception handling itself.
- i. Recovery accessor is added for device register when it is obtianed. These recovery accessors are used to recover the values of variables after the recovery.
- j. setOnwer() is used to set the ChimeraTK::ApplicationModule or ChimeraTK::ThreadedFanOut as owner of the (feeding) device which is decorated with an ChimeraTK::ExceptionHandlingDecorator.
- k. Write should not block in case of an exception for of ThreadedFanOut / TriggerFanOut.
- l. Application Module should provide a writeDespiteError function so that even in case of exception write should return. [TBD: name]
<b> Definitions <b>
- d. Class ExceptionHandlingDecorator facilitates ChimeraTK::NDRegisterAccessor in case of exception
- e. An ExceptionHandlingDecorator is placed around all NDRegisterAccessors connected to a device used inside ApplicationModule and ThreadedFanOut except initial value and recovery accesors.
- f. TriggerFanOut is using a TransferGroup which bypass the functionalitly of ExceptionHandlingDecorator hence it has to impelment the exception handling itself.
- g. Recovery accessor is added for device register when it is obtianed. These recovery accessors are used to correctly set the values of variables when the device is opened for the first time and after a device is recovered from an exception.
- h. setOnwer() is used to set the ChimeraTK::ApplicationModule or ChimeraTK::ThreadedFanOut as owner of the (feeding) device which is decorated with an ChimeraTK::ExceptionHandlingDecorator.
- i. Write should not block in case of an exception for of ThreadedFanOut / TriggerFanOut.
- j. Application Module should provide a writeDespiteError function so that even in case of exception write should return. [TBD: name]
<b> Definitions </b>
- ChimeraTK::NDRegisterAccessor is used to access the device variables inside class Application.
- writeAfterOpen is a list of TransferElements used for initial value propogation.
- writeAfterRecovery is a list of TransferElements used after recovery.
...
...
@@ -95,20 +94,18 @@ Once in error state, set the DataValidity flag for that module to faulty and pro
- Step 2.4.3. is currently being set before initialisationHandlers and writeAfterOpen.
- Step 2.5.5.5. is currently being set before initialisationHandlers and writeRecoveryOpen.
- Step 2.5.3.7. is currently being set before initialisationHandlers and writeRecoveryOpen.
- Check the comment in Device.h about writeAfterOpen(). 'This is used to write constant feeders to the device.'
- Check the documentation of DataValidity. ...'Note that if the data is distributed through a triggered FanOut....'
<b>Implmentation Details</b>
<b>4. Add an exception handling and reporting machinsm to the device module (DeviceModule).</b>
<b>4. Exception handling and reporting mechanism to the device module (DeviceModule).</b>
Description.
Add two error state variables.
- "state" (boolean flag if error occurred)
- "message" (string with error message)
These variables are automatically connected to the control systen in this format
- /Devices/{AliasName}/message
- /Devices/{AliasName}/status
...
...
@@ -116,33 +113,32 @@ These variables are automatically connected to the control systen in this format
Add a thread safe function reportException().
A user/application can report an exception by calling reportException of DeviceModule with an exception string. The reportException packs the exception in a queue and the blocks the thread. This queue is processed by an internal function handleException which updates the DeviceError variables (status=1 and message="YourExceptionString") and tries to open the device. Once device can be opened the DeviceError variables are updated (status=0 and message="") and blocking threads are notified to continue. It must be noted that whatever operation which lead to exception e.g., read or write, should be repeated after the exception is handled.
Catch all the ChimeraTK::runtime_error exceptions that could be thrown in read and write operations and feed the error state into the DeviceModule through the function DeviceModule::reportException() . NDRegisterAccessors coming from device should be used as a singal central point to catch these excpetions.
For a device with it's deviceError.status 0 (see 2.4.3), catch all the ChimeraTK::runtime_error exceptions that could be thrown in read and write operations and feed the error state into the DeviceModule through the function ChimeraTK::DeviceModule::reportException().
Retry the failed operation after reportException() returns.
Implmentation.
For a device that has been opened for the first time and throws a ChimeraTK::runtime_error exception see 2.3.
Exceptions are caught as explained in g and h.
- NDRegisterAccessors
- Application
Implementation.
- Exceptions are caught as explained in 1.e and 1.f.
- ChimeraTK::NDRegisterAccessors
- ChimeraTK::Application
<b>6. Faulty device should not block any other device.</b>
Description.
Each TriggerFanOut deals with several variable networks at the same time, which are triggered by the same trigger. Each variable network has its own feeder and one or more consumers. You do not need to change anything about the variable networks.
On the other hand, the trigger itself is a variable network, too. The TriggerFanOut has a consumer of this trigger network. This is the accessor on which the blocking read() is called in the loop. You will need to create additional consumers in the trigger network, one for each TriggerFanOut.
Each ChimeraTK::TriggerFanOut deals with several variable networks at the same time, which are triggered by the same trigger. Each variable network has its own feeder and one or more consumers. The trigger itself is a variable network, too. One consumer per ChimeraTK::TriggerFanOut is required.
Implementation.
- Application (Application::typedMakeConnection)
- ChimeraTK::Application::typedMakeConnection()
<b>7. The server must always start even if a device is in error state.</b>
...
...
@@ -152,37 +148,37 @@ To make sure that the server should always start, the initial opening of the dev
<b>8. Set/clear fault flag of module in case of exception.</b>
Background.
A DataValidity flag of a module is set to faulty if any input variables returns with a set data fault flag after a read operation and is cleared once all inputs have data fault no longer set. In a write operation, the module's data fault flag status is attached to the variable to write.
More detail ...(Martin‘s doc)
A DataValidity flag of a module should be set to faulty if any input variables returns with a set data fault flag after a read operation and is cleared once all inputs have data fault no longer set. In a write operation, the module's data fault flag status is attached to the variable to write.
For initial error propogation see (Martin‘s doc link)
Description.
In case of an ChimeraTK::runtime_error exception this DataValidity flag should also be set to faulty and propogated to all outputs of the module. When the operation completes after clearing the exception state, the flag should be cleared as well.
New: Non-blocking writes in the exception handling decorator (see 2.5.1.1.2. and 2.5.4.1.2.): Exception handling decorators for writeable variables must not block if owner is in exception state.
As explained in 2.5.1.
Non-blocking writes in the exception handling decorator (see 2.5.1.1.2. and 2.5.4.1.2.): Exception handling decorators for writeable variables must not block if owner is in exception state.
Implmentation.
- ExceptionHandlingDecorator
- TriggerFanOut
- ChimeraTK::ExceptionHandlingDecorator
- ChimeraTK::TriggerFanOut
<b>9. Initialise the device after recovey.</b>
<b>9. Initialise the device</b>
Description.
If a device is recovered after an exception, it might need to be reinitialised (e.g. because it was power cycled). The device should be automatically reinitialised after recovery.
The device should be automatically initialised when opened for first time (2.4.1) and automatically re-initialised after recovery (2.5.3.4).
Implementation.
A list of DeviceModule std::function is added. InitialisationHandlers can be added through construtor and addInitialisationHandler() function. When the device recovers all the initialisationHandlers in the list are executed.
- DeviceModule
- ChimeraTK::DeviceModule
- ChimeraTK::ExceptionHandlingDecorator
<b>10. Recover process variables after exception.</b>
...
...
@@ -192,15 +188,15 @@ Background.
After a device has failed and recovered, it might have re-booted and lost the values of the process variables that live in the server and are written to the device. Hence these values have to be re-written after the device has recovered.
Description.
Technically the issue is that the original value that has been written is not safely accessible when recovering. Inside the accessor the user buffer must not be touched because the recovery is taking place in a different thread. In addition we don't know where the data is (might or might not have been swapped away, depending whether write() or writeDestructively() has been call by the user).
The only race condition free way is to create a copy when writing the data to the device, so they are available when recovering.
Create a copy of accessor when writing the data to the device and use this to recover the values when the device is available again. Recovery accessor do not write if the register is never written before (2.5.3.5.).
Implementation.
- DeviceModule
- ExceptionHandlingDecorator
A list of TransferElements shared pointers is created with as writeRecoveryOpen which is populated in function addRecoveryAccessor in the DeviceModule.
ExceptionHandlingDecorator is extended by adding second accessor to the same register as the target accessor it is decorating and data is copied in doPreWrite().
- ChimeraTK::DeviceModule
- ChimeraTK::ExceptionHandlingDecorator
A list of ChimeraTK::TransferElements is created with as writeRecoveryOpen which is populated in function addRecoveryAccessor in the DeviceModule.
ChimeraTK::ExceptionHandlingDecorator is extended by adding second accessor to the same register as the target accessor it is decorating.
<I> Data is copied in doPreWrite(). [TBD: Do we want this behaviour?]</I>
