when dealing with double copy recalls.

Proper fix by defining a new RECALLJOB_SELECTED2NDCOPY state instead
of hijacking the (deprecated) fileTransactionId.

resilient to hardware unavailability.

broken when dealing with double copy recalls.

This fix is not optimal: we're still using fileTransactionId being NULL
as a flag while we should really create an extra status
RECALLJOB_SELECTED2ND or SELECTEDSHADOW. Not on this hot fix...

This commit also includes a simpler way to select new recall candidates,
which makes it more immune to bugs like the above.
Plus a fix related to the precision used to check if a file was
overwritten while being recalled.

resilient to hardware unavailability.

broken when dealing with double copy recalls.

This fix is not optimal: we're still using fileTransactionId being NULL
as a flag while we should really create an extra status
RECALLJOB_SELECTED2ND or SELECTEDSHADOW. Not on this hot fix...

This commit also includes a simpler way to select new recall candidates,
which makes it more immune to bugs like the above.
Plus a fix related to the precision used to check if a file was
overwritten while being recalled.

Plus one more fix for bug #103242 Incorrect evaluation of
lastModificationTime leads to double recalls + dark data.
Hotfix script to be produced yet.

Plus one more fix for bug #103242 Incorrect evaluation of
lastModificationTime leads to double recalls + dark data.
Hotfix script to be produced yet.

double recalls + dark data.
The dark data generation is prevented by creating a DiskCopy entry in
all cases at the end of a recall.
The false positives have been fixed in both NS 'native' and
'compatibility' mode by always using the openTime when available
and defaulting to the mtime as we used to do in 2.1.13. Clearly
only in native mode can we exploit the microsecond precision.

Fixed bug #103175: Concurrent recalls that fail concurrently may create a deadlock if they share files

  - avoid looping too much in the tapegateway when some file is locked for long
  - improved handling of log messages when the DLFLogs table is too full :
    + do not take all of it in one go
    + use an index as the table may be scattered

is to be used in all queries. This is to prevent Oracle from using
a FAST FULL SCAN, which in fact is less fast as observed during stress
tests.

- more hints to avoid index FAST FULL SCANs
- initialize varNSFileIds array to avoid 'ORA-06531' errors
- renamed updateStagerTime to update2114Data: this procedure
  is supposed to be dropped after the Nameserver post-upgrade
  script is run and the newly introduced metadata is updated.
  The name reflects the fact that the procedure anticipates
  for Repack what the job in the post-upgrade script does
  in background for all files.

Fixed call to setSegmentsForFiles when in NS compatibility mode.
Essentially in this case the CastorFile.lastUpdateTime timestamp
has to be passed to the Nameserver for its concurrent
modifications check as it was the case with version 2.1.13.

Improved draining by implementing back pressure on the scheduling (no more than 1000 jobs pending per drainingJob) and ordering of files by importance (no tape and fewer copies on disk first)

This includes :
  - merge of d2dtransfer into diskmanagerd
  - drop of the WAITDISKTODISKCOPY state in DiskCopy, and of the diskcopies for ongoing replication in general. Now DiskCopies are only created at the end of the replication, when they actually exist on disk
  - drop of the StageReplicaRequest table and concept
  - introduction of the DiskToDiskCopyJob table, holding the list of ongoing disk to disk copies
  - a split of transferToSchedule into userTransferToSchedule and D2dTransferToSchedule
  - an according split in the dispatcher part of the transfer manager to call both methods. Note that with this commit, both methods fill the same FIFO queue of transfers and both take everything they can find without any throttling.

At this stage, the draining facility is broken.

Bug #95189: Time discrepencies between disk servers and name servers can lead to silent data loss on input

- Properly use cns_file_metadata.stagertime as a single time source for stager open/close operations.
This included modifying a server-side data structure, Cns_file_metadata, and the semantics of
openx() and closex() along with their wire protocol: thus after this change only a new stager can
talk to a new nameserver (i.e. no backwards- nor forwards-compatibility provided).
- Introduced a CastorConfig table in the db with one configuration parameter, openmode, to decide
whether to use the old 'Compatibility' mode as before or the 'New' mode as described above.
- On the way, proper filling of cns_seg_metadata's creationtime, lastmodificationtime and gid fields has
been implemented for logging/statistical purposes.

Merged STAGED and CANBEMIGR into VALID in DiskCopy statuses and create a tapeStatus on the CastorFile with possible values NOTONTAPE, ONTAPE and DISKONLY.
Note that user interface is not touched and still displays CANBEMIGR and STAGED. Only internals (particularly the DB) are concerned.

Added extraction of information from the DB, so it can be propagated to the logs. The new recall log message now include the user involved (uid), and the retry counters.

Fix following option 2, i.e. double check the Nameserver, for
Felix to deploy at ASGC immediately.

Fixed bug #99219: "RFE: A file should never be reported as having a compressed file size less than 1 byte". This was already fixed on the tape side, so that no such report comes, but as a security measure, we also fix it in the DB code.

- Introduced support for a READONLY state for Diskservers and Filesystems.
Hardware in such a state will accept migrations, [PrepareTo]Get and
DiskCopyReplica requests, and will not accept recalls or [PrepareTo]Put/Update
ones when they switch to write mode.
- Dropped the {DiskServer,FileSystem}.adminStatus fields and the obsoleted
code around them and introduced a new DiskServer.hwOnline boolean field, which
is updated by the heartbeat reports and displayed by printdiskserver.
In turn, modifydiskserver can not change the online flag but only the status.
For backward compatibility, on stager_qry -s a Diskserver will be reported
as DISABLED if hwOnline = 0 regardless its status.

Changed log level for message 'startMigrationMounts: failed migration mount creation due to lack of files' to INFO

Fixed bug #98250: Incorrect loop in archiveOrFailRepackSubreq leading to PL/SQL: numeric or value error.
Fixed also a few more occurrences of FIRST..LAST, even though
it appears the logic around wouldn't have led to the bug.

Fixed logging.

Added comments to the PL/SQL code called by the VdqmRequestsProducerThread class