GitLab

[cta-taped] Added configuration TapeLoadTimeout to set the timeout after which the tape mount is supposed to be failed

This reverts commit f9249bb1.

of VO and Density"

Created a garbage collector subprocess for the drive daemon.
Created a AgentHeartbeat thread and added it to frontend, drive subprocess and GC subprocess.
Fixed logs and drive subprocess.
Renamed common/threading/Threading.[hc]pp to Thread.[hc]pp as there is only one class left in this file.

6b6374d6c2e209c98c0d4d7aa665e1df83d71aaa
  CASTOR-5350: Introduce encryption SCSI commands in tape drive
  backend

  Implementation of two methods:
    * setEncryptionKey(key): Sets encryption params to drive.
    * clearEncryptionKey: Clears encryption params from drive.

3cf91d48f5c7b0cb563c3037aee69ec769f5ab94
  Added support for an interface script that will setup drive
  encryption per tape

e9ca601687508de20fab7154e63bb0dbd1b25a8a
  Migrate TapeWriteSingleThread::TapeCleaning::~TapeCleaning() body to
  .cpp

266b02d8175b5cfc0b688135cbdc335e93060b
  CASTOR-5350: Refactor support for only external key management script

789b26a0bc69053ff1ab792a02676a7753f093ed
  Merge branch 'encryption_backend' into 'master'

  CASTOR-5350: Encryption backend

  ## Description
    The aim of this merge request is to incorporate encryption support
    into CASTOR.
    The proposed changes are to be used in conjunction with the
    operators'
    [ExternalEncryptionKeyScript](https://gitlab.cern.ch/slaskari/castor-get-encryption-key).

    The **aim** is to enable encryption in specific tape pools of
    CASTOR.
  ## Changes
    * Introduce encryption SCSI backend to DriveGeneric.
    * Introduce encryption control wrapper
    * (`castor/tape/tapeserver/daemon/EncryptionControl`) for
    * abstracting the two sub-components of:
      * Calling the `ExternalEncryptionKeyScript`,
      * Calling the equivalent DriveGeneric function for
      * passing/clearing the encryption parameters to/from the drive.
    * Add new configuration option in `castor.conf` for the external key
    * management script.
    * Create a Subprocess wrapper for executing external commands as
    * CASTOR children (`castor/server/Subprocess.{h,c}pp`).
    * Incorporate encryption handling in the:
      * DataTransferSession
      * LabelSession
      * CleanerSession
    * Add encryption control timer in the task Watchdog.
  See merge request !1

fa550707c42d80466bbd448e355aaf9be5ea8e04
  Clear encryption key only when encryption enabled
  Changes include:
    - Making EncryptionControl stateful
    - Calling clearEncryptionKey on the drive only when encryption is
      on.
  Also includes a minor duplicate code fix on DriveGeneric.
cf4eb9f3ae36c9cfc9c40349d69ab6642020e81e
  Merge branch 'encryption_changes' into 'master'
  Clear encryption key only when encryption enabled
  ## Description
    Changes include:
    - Making EncryptionControl stateful
    - Calling `clearEncryptionKey()` on the drive only when encryption
      is on.
    Also includes a minor duplicate code fix on **DriveGeneric.cpp**.
  See merge request !2

40366d963ee33ca081df6c991189b21369e461fd
  Check if the drive has encryption capability enabled:
    * Add isEncryptionCapEnabled() vendor-specific function
    * Check isEncryptionCapEnabled() before passing encryption params
    * Check isEncryptionCapEnabled() before clearing encryption params
    * Clear encryption key before unencrypted I/O

4ccc661d01eccfc3fdfb9ee2578d15a147a0c55a
  Merge branch 'encryption_capability_enabled' into 'master'
  Drive encryption capabilities inclusion
  ## Description
    The aim of this merge request is to address issues related to
    encryption on drive without the encryption capability enabled.
    More specifically:
      * It introduces a vendor-specific way of identifying if the drive
      * has encryption capability enabled
      * **IBM**: Through the SPIN index SCSI page
      * **Oracle**: Through the general INQUIRY SCSI page
      * If the data to be written are to be encrypted, an additional check
      * of the encryption capability of the drive is made. In case of
      * encrypted data, but no encryption capability, the session fails.
      In essence, all encryption related operations are made modulo the
    encryption capability of the drive.
      Last, in case of unencrypted I/O, we clear the keys of the drive (if
    encryption capable) to avoid encrypted data with previous keys on
    CASTOR's system failure.
  ## Testing
    Before the merge request's submission, the following tests were
    passed:
      On drives with **encryption capability enabled**:
        * Label session
        * Label with previously set encryption key
        * Write without encryption
        * Read without encryption
        * Write with encryption
        * Read with encryption
        * Write with previously set encryption key
        * Read with previously set encryption key
      On drive with **encryption cabability disabled**:
        * Label session
        * Write without encryption
        * Read without encryption
        * Write with encryption - session **should** fail
        * Read with encryption - session **should** fail
    See merge request !3

55b85a2cb4681d697565116c00ff98c6becea4fb
  Secure session against invalid encryption script output

3a54875c680fe6c1c9d5cf25cf98d2780196e0d1
  Changes in encryption workflow
    - VMGR tag is updated only on write operations
    - Empty key signifies no encryption

f5408cf0ccbae9a4ab94a533f3b6d7be323f72fb
  Minor encryption log enhancements
    * Error line in Read/Write session with ErrorMesage key
    * Fix for delimiter in the end of arguments in argsToString()
2e7204fb0dd24b472a959fa5e13320c34df4f017
  Merging in improvements on tape encryption support.

92533a1746d0744ee528781558a720c63ca3c4d1
  Removed nullptr which is not supported in SLC6's gcc.
  Added automatic deletion of json objects in
  EncryptionControl::parse_json_script_output.

fca3bb9e7fce364b429fc0b5c036fb752fd67ff1
  Fix log typo

Added drive status reporting in scheduler
Added drive status reporting in OStoreDB
Added support for drive status listing in the front end
Removed virtual functions from the Scheduler, which is never overloaded.
Added DesiredDriveState structure to drive state.
Removed usage of duplicate MountType, DriveStatus and DriveState structures.
Created DriveInfo structure to allow recreation of drive register entry in all
reporting situation (potentially with partial/assumed info).

CASTOR-4982: tapeserverd should tolerate some non-fatal tape alerts
before writing

Fixed.

Add logic to the TapeWriteSingleThread to skip not-fatal tape
alerts before writing to the tape. Only "Lost statistics"
tapeAlertLostStatistics 0x32 tolerated as non-fatal.

fa889fed2541e22179b5e035d863f87e7be18fb9
  CASTOR-5322 RFE: Enhance tapeserverd logs with SCSI tape drive
    statistics

b13f495e4ee229b2469f9470a2ffa6b4003a29ec
  Fix for mhtvl scsi log sense exceptions

ad71058fbcb6de85e0440797d7ffa5358e26bf89
  CASTOR-5329 Enhance tape statistics

806e48f4285122d8ab9f118364a15e740518028f
  CASTOR-5332 RFE: Reduce log level to INFO with MHVTL - SCSI Statistics
    could not be acquired from drive

79c5a4c2c36b7acc5b10505ca1694fd521fc6832
c7f6d4d7aaa564b37c2b36c3110dfe2fc96ec970
  Move volume SCSI statistics inside the dtor of TapeCleaningMove volume
    SCSI statistics inside the dtor of TapeCleaning

  ## Description

  When first introduced volume SCSI Statistics (at the moment
  IBM-specific), we explicitly put the function after the unmount of the
  tape was done due to an invalid file descriptor error occurring during
  the SCSI query.

  This bug no longer occurs for IBM drives.
  This may be attributed to the update of firmware of the IBM drives

  Apart from the change of the position of the changes, there is no
  alteration in terms of the metrics reported from the drive to the logs.

  ## Testing

  The tests the new code has been through are:
    * Write/Read file on IBM lib0 drive *(older one)*
    * Write/Read file on IBM lib4 drive *(newer one)*
    * Write/Read file on Oracle T10k drive

d1689f3b72329bcc6c342666095567f547f1678f from castor/master

CASTOR-4909: tapeserverd tries to unmount the tape when the mount
failed...

Made the ignoring of exception throw by DriveGeneric::waitUntilReady
more specific: we now ignore timeouts, but not the other exceptions.

Replaced castor file header with CTA file header

Moved castor::server::
  BlockingQueue
  Semaphores
  AtomicCounter
  AtomicFlag
into cta::threading::

cta::server::ProcessCap

The TapeServerReporter, which is the main interface for the tape thread and tasks,
now has a simple reportState() function instead of mnay ad-hoc ones. The only ones
remaining are reportTapeUnmountedForRetrieve() and reportDiskCompleteForRetrieve()
as they allow managing the special case of retrieve where the session can from
Running to either ShutingDown or DrainingToDisk depending on which order the
threads complete.

The actual calls to send messages to taped are now 3: reportState,
addLog, removeLog.

0001-CASTOR-5279-Logical-Block-Protection-support-in-the-.patch 1ab9f7b0408b85ec4660e75f3259ef8b7c21f8c1 TO REVIEW?

00060-CASTOR-5279-Logical-Block-Protection-support-in-the-.patch 52627f63d074f301b43e753bf00eca430bdb5d0f TO REVIEW

Please note that you will have to install the following CASTOR rpms
in order to compile:

  castor
  castor-devel
  castor-lib
  castor-lib-server
  castor-lib-tape

Renamed CTA/tapeserver/castor to CTA/tapeserver/castor_checkout in preparation for the moving the sub-directory CTA/tapeserver/castor/castor up the directory tree

When checking for the drive to be ready, we now disregard the result of the waiting for the tape to be present (as it might or might not be).

The tape alert detection was missing in TapeWriteSingleThread::TapeCleaning::~TapeCleaning, it was added only to
TapeReadSingleThread::TapeCleaning::~TapeCleaning.

The logic from the session cleaner has been duplicated to the RAII TapeCleaning class's destructor
for both migrations and recalls.

Migrations now abort in the presence of tape alerts.

Propagated references to the watchdog to disk and tape threads and tasks.
Added a maps for storing the count of all errors that occured during the session.
Added propagation of the counts to the initial process.
Added error reporting in disk and tapes threads and tasks by using a string marker
allowing to know which part went wrong in high level exception. Exceptions then
bumps up the count in the watchdog (synchronously), and the watchdog sends the
new count to the initial thread later (in the watchdog's thread).
Plus some missing logs fixed when an exception is thrown in a disk write task.

Added interface to reference the recall watchdog to the recall disk thread pool.
Next: Do same for migration. Actually store the reference. Add new error map storing in watchdog. Add error reporting in tape and disk threads.

WIP: Added missing file ID for disk write thread. Added missing error log when an exception is thrown in a disk write task.

WIP: switching to CASTOR-4839 tapeserverd: task injector should decide on closing the session earlier

WIP. Next: log unmount errors (in the RAII)

Finished error counting in data threads.

The offending message has been changed in a previous case: CASTOR-4749

Reviewed all bandwidth calculations to homogenize the statistics printing.
Fixed a bug where the total time for a session was not properly retrieved before logging, leading to NaN printouts.