The value was 10 minutes instead of 1. This value was too high as this represents the load time
of the drive, which is rather fast. Also it was misaligned with the mount to transfer state transition
timeout, which led to some sessions being killed instead of failing by themselves.

Without this index we got errors like:

[root@p06253947b39467 ~]# cta af ls --instance cta --path /eos/dev/test/gcancio/10072017/file_2167
Failed to execute the command. Reason: Failed to open root://10.16.135.81:10955//Y3Rh&YWY=&bHM=&LS1pbnN0YW5jZQ==&Y3Rh&LS1wYXRo&L2Vvcy9kZXYvdGVzdC9nY2FuY2lvLzEwMDcyMDE3L2ZpbGVfMjE2Nw==: [ERROR] Operation expired
[root@p06253947b39467 ~]#

With this index we execute the same query in less than a second:

[root@p06253947b39467 ~]# time cta af ls --instance cta --path /eos/dev/test/gcancio/10072017/file_2167
 289440       1  I20170    1381    51096      cta   18965         1006       ADLER32     0XFBA23F1D    german_raw  gcancio       c3    1499697578 /eos/dev/test/gcancio/10072017/file_2167

real    0m0.590s
user    0m0.005s
sys    0m0.006s
[root@p06253947b39467 ~]#

The updating of statistic in the objecttore (in drive register) could wait for a lock
for so long that the session get killed due to a heatbeat timeout.

To avoid this, we use lockfree access as much as possible, and when locking it srequired, do it with a
timeout.

The timeout will be logged as a warning.

Added timeouts to synchronous locking primitives in the object store backends to implement this feature.

Fixed a bug where the VFS backend always used exclusive locks.

Rao branch

See merge request !3

This completes the dispatcher logic in the frontend

avoid file metadata resolution on eos side.

when SSI is not used.

The probe state is also used. It represents the time when the drive is being checked
for tapes during the down to up transition.

Shutdown is not in use yet as it requires changing the behaviour of the system (clean shutdowns).

We now automatically find all binaries to symlink into /usr/bin of containers.

Changes dispatcher back to using a switch statement for the EOS commands
for easier static analysis.

Generating backtrace automatically when ctafrontend dies, collect it for artifacts and display it when the namespace gets deleted so that it appears on the Pipeline.

For the admin functions, it is very clunky to map <cmd,subcmd> pairs to
which function to call using nested switches. Instead we lookup which
function to call in a static const map of <cmd,subcmd> to method function
pointers. Now new commands can be added or old ones removed by editing a
single line in the map definition.

and 1 hour by default.
Previously when cta frontend coredumped, all the files appeared to have
been successfully deleted when it was not the case.
I am now making sure cta client returned with no error to correctly
detect failures.

This will give faster feedback to the operators as they put drives up.
The scheduling can sometimes take a long time and the status update only happened after that.

This is necessary after a system restart if the api server's IP address changed.

sqplus is needed to clean up the database.

preprod workflows

Zero-length metadata now resolves to a valid metadata messages, with
RSP_SUCCESS and no text message.

Increased number of processes watched by a single garbage collector to cover the case for many drive processes per server.