GitLab

[tapeserver] In the case the DiskWriteTask for a file fails, the log contains its size and its checksumblob

CASTOR-4889: tapeserverd's end of session summary could indicate the
files still open

Fixed.

Add reports for still open files in the "Tape session finished"
log message to indicate problematic disk servers. It should be
only visible in case of kill of the data transfer session.

Replaced castor file header with CTA file header

Moved castor::server::
  BlockingQueue
  Semaphores
  AtomicCounter
  AtomicFlag
into cta::threading::

Removed deprecated structures from archiveNS and nameserver namespaces. Removed RetrieveToFileRequest.

Started morphing the code to accomodate both the move some data structure from OStoreDB to catalogue and shared structures modifications.
Renamed class members using underscode instead of mixed case.
Removed mount criteria from in-memory tape pool representation.
Removed support for path of archive files for in-memory representations.
Removed all but maxDrives allowed from in-memory representation of mount
policy.
Created new constructor for UserIdentity.
Switch the UserIdentity used by RemoteFileStatus.
Renamed TapePoolQueue to ArchiveQueue.
Replaced CreationLog usages with EntryLog.
Renamed TapeQueue to RetrieveQueue.
Removed support for data now gone to catalogue from RootEntry.
Removed corresponding structures from object store.
Removed corresponding calls from SchedulerDatabase and inheriting
classes (OStoreDB).

This implied removing tpconfig, tplabel and tpstat.

Fixed confusion between archive and retieve in logging "path" and creation of file object by file factory.

Please note that you will have to install the following CASTOR rpms
in order to compile:

  castor
  castor-devel
  castor-lib
  castor-lib-server
  castor-lib-tape

Renamed CTA/tapeserver/castor to CTA/tapeserver/castor_checkout in preparation for the moving the sub-directory CTA/tapeserver/castor/castor up the directory tree

Fixed.

Removed "h/" from all #includes.

This led to a locking of the session.
Added a log message for this failure and improved the exception's wording.

Propagated references to the watchdog to disk and tape threads and tasks.
Added a maps for storing the count of all errors that occured during the session.
Added propagation of the counts to the initial process.
Added error reporting in disk and tapes threads and tasks by using a string marker
allowing to know which part went wrong in high level exception. Exceptions then
bumps up the count in the watchdog (synchronously), and the watchdog sends the
new count to the initial thread later (in the watchdog's thread).
Plus some missing logs fixed when an exception is thrown in a disk write task.

Added interface to reference the recall watchdog to the recall disk thread pool.
Next: Do same for migration. Actually store the reference. Add new error map storing in watchdog. Add error reporting in tape and disk threads.

WIP: Added missing file ID for disk write thread. Added missing error log when an exception is thrown in a disk write task.

WIP: switching to CASTOR-4839 tapeserverd: task injector should decide on closing the session earlier

WIP. Next: log unmount errors (in the RAII)

Finished error counting in data threads.

Removed leftovers of special case snprintf call for double, now that snprintf is the default for doubles.

Added the open and close time in the disk/non-disk time ratio calculation in disk tasks (close is the most important).

Added better measurement of the disk performance to locate slow disk servers during tests in production.

The offending message has been changed in a previous case: CASTOR-4749

Reviewed all bandwidth calculations to homogenize the statistics printing.
Fixed a bug where the total time for a session was not properly retrieved before logging, leading to NaN printouts.

I believe the Timer class is generic enough to be used by any
part of CASTOR and not just tape.

CASTOR-4739: tapeserverd should support localfile, rfio, xroot and rados striper access for disk files

After testing Xroot access, realised that we need to add a signature in the opaque data of the
Xroot URL. A signature using OpenSSL, as used in other places of the project was implemented,
but did not pass the memory leak and race condition tests. As discussed with the rest of the team,
OpenSSL is not to be trusted and has poor quality memory managment. The CryptoPP library, also part
of the SLC6 distribution has been tested as well, and it output validated (on a few example) with
OpenSSL's. CryptoPP can be used instead of OpenSSL, and has been put in place. This commit still contains
the OpenSSL for reference. The next commit will remove it.