ReleaseNotes.md

v4.NEXT

Summary

Upgrade Instructions

Features

Bug fixes

Building and Packaging

Catalogue Schema

v4.7.5-1

Summary

This release introduced an additional gRPC based frontend for storage backends. This is still a work in progress and is not ready for use.

The command cta-verify-file now requires the options eos.instance, eos.request.user and eos.request.group to be configured in /etc/cta/cta-cli.conf.

Upgrade Instructions

Features

• cta/CTA#1222 - Add minimal gRPC based frOntend for integration with dCache
• cta/CTA#1241 - make cta-verify-file get instance, request user and group otions from ctacli config file

Bug fixes

• cta/CTA#1225 - Fix bug causing tapeserver to sometimes pop the entire archive queue at the end of the mount
• cta/CTA#1247 - Fix improper initialization of the variable m_lastFseq of type uint64_t with -1 value in the constructor of castor::tape::tapeserver::daemon::TapeWriteSingleThread

Building and Packaging

Catalogue Schema

v4.7.4-1

Summary

Upgrade Instructions

Features

• cta/CTA#1205 - Fail pipeline if cppcheck detects errors
• cta/CTA#1206 - Change NULL for nullptr
• cta/CTA#1217 - Schema verification should just issue an warning if there are extra indexes in the db that are not in the catalogue
• cta/CTA#1220 - Improve queued retrieve logging message
• cta/CTA#1152 - Reduce eos free space query load
• cta/CTA#1231 - Add --loadtimeout option to cta-tape-label and increase default value
• cta/CTA#977 - Add --drive option to cta-tape-label

Bug fixes

• cta/CTA#1120 - Fix negative disk space reservation content
• cta/CTA#1235 - cta-verify-file should return 1 on error

Building and Packaging

Catalogue Schema

v4.7.3-1

Summary

Upgrade Instructions

Features

• cta/CTA#1161 - Tape server refactoring, "Decide where m_reportPacker.setTapeDone() should be called"
• cta/CTA#1195 - cta-catalogue-schema-drop should drop the CTA_CATALOGUE table last

Bug fixes

• cta/CTA#950 - Eliminate race condition preventing the drive to go down on failure
• cta/CTA#1160 - Fix DrainingToDisk stale status in case if there is DiskWriteThreadPool thread left
• cta/operations#708 - Fix "Should run cleaner but VID is missing. Putting the drive down"
• cta/CTA#1197 - Return code from cta-catalogue-schema-verify should indicate if the schema is in UPGRADING state
• cta/CTA#1206 - Change NULL for nullptr

v4.7.2-1

Summary

• Deprecated: Replaced by release v4.7.3-1

v4.7.1-1

Summary

Upgrade Instructions

Features

• cta/CTA#1179 - Use std::optional instead of cta::optional
• cta/CTA#1190 - Use std::make_unique instead of cta::make_unique
• cta/CTA#1198 - Use hardcoded mount policy for verification requests
• Add verification flag to queued retrieve request log message
• propagate labelFormat from TAPE catalog to VolumeInfo
• cta/CTA#1200 - Remove range class

v4.7.0-1

Summary

Upgrade Instructions

This CTA release requires a non-backwards compatible database schema upgrade to CTA catalogue schema v10.0. Please consult the database upgrade documentation.

Features

• cta/CTA#1163 - cta-admin now prefixes the drivename with a '!' if the respective logical library is disabled
• cta/CTA#1168 - Add configuration option for scheduler stack size to /etc/cta/cta-frontend-xrootd.conf
• cta/CTA#1151 - Update cta-admin ds add for catalogue schema version 10

Bug fixes

• cta/CTA#1156 - failed to instantiate the RAO algorithm

Building and Packaging

• cta/CTA#823 - Ensure unit tests CANNOT be executed against a production database
• cta/CTA#1082 - Review software license text in CTA
• cta/CTA#1153 - Change compiler FLAGS in cmake files
• cta/CTA#1155 - Remove GCC suppresions in DriveGeneric and file/Structures
• cta/CTA#1173 - Remove GCC '-Wno-unused-function' suppresion
• cta/CTA#1166 - Add SCL-RH Repo to CTA CI boot scripts
• cta/CTA#1185 - Make expectedLabelStandard default argument (for Enstore reads)

Catalogue Schema

• cta/CTA#1043 - Add verification column to tape table
• cta/CTA#1158 - Check all foreign key references have a full index on both sides of the constraint
• cta/CTA#1171 - Add LABEL_FORMAT column to TAPE table
• cta/CTA#1172 - Add disabled reason column to logical library table
• cta/CTA#1177 - Fix CTA catalogue schema verify index checker for PostgreSQL
• cta/CTA#1147 - Add foreign key constraint between the disk instance name of a vo and the disk instance table
• cta/CTA#1151 - Drop ACTIVITY_WEIGHTS and TAPE_DRIVE tables

v4.6.1-1

Summary

Upgrade Instructions

This CTA release requires a database schema upgrade to CTA catalogue schema v4.6. Please consult the database upgrade documentation.

Features

• cta/CTA#1137 - Stop deletion of failed retrieve/archive requests
• cta/CTA#1150 - Add option to pass desired catalogue version into cta-catalogue-schema-create
• cta/CTA#1147 - Add Disk Instance Column to VO table
• cta/CTA#1119 - Remove support of manual mode for loading tapes
• cta/CTA#1123 - Add mount id to disk space reservations, prevent tape servers from releasing disk space from a previous mount

Bug fixes

• cta/CTA#1138 - sortAndGetTapesForMountInfo only queries tapes in the current logical library
• cta/CTA#1117 - Update masterDataInBytes when writing files to tape
• cta/CTA#1125 - cta-admin dr ls should show '-' instead of "NO_MOUNT" for Mount Type

Continuous Integration

• cta/CTA#1131 - Compile CTA using devtoolset-8 in CI
• cta/CTA#1126 - Create tests for TapeDrivesCatalogueState

v4.6.0-1

Summary

Updates EOS version in CI to 4.8.75.

Upgrade Instructions

This CTA release requires a database schema upgrade to CTA catalogue schema v4.5. Please consult the database upgrade documentation.

Features

• cta/CTA#999 - Add a default mount rule for recalls
• cta/CTA#1109 - Add --dirtybit option to cta-admin ta ch and show dirty bit value in cta-admin --json ta ls
• cta/CTA#1107 - add mountpolicyname to request schema in objectstore
• cta/CTA#1111 - Add disk instance and disk instance tables to catalogue and respective cta-admin diskinstance/diskinstancespace add/ls/ch/rm commands
• cta/CTA#1114 - Log SSI events in ctafrontend
• cta/CTA#1108 - Make cta-admin --json sq show mount policy with highest priority and mount policy with lowest request age for each queue.

Bug fixes

• cta/CTA#1102 - Make requeued jobs retain their original creation time
• cta/CTA#1091 - Move TAPE_DRIVE table to DRIVE_STATE and refactor code to update drive states
• cta/CTA#1110 - Moves disk space reservations to the DRIVE_STATE table, uses atomic updates
• Don't throw an exception if DiskSpaceReservationRequest brings reservation < 0, just reset and log the error

Continuous Integration

• cta/CTA#734 - Adds test for FST delete-on-close behaviour
• Adds system tests for 'prepare' and 'query prepare' (idempotent prepare)

v4.5.1-2

Summary

No code changes. Updates EOS version in CI to 4.8.74.

v4.5.1-1

Summary

Bug fixes

• cta/CTA#1101 - Fix disk space reservation logic adding all existing disk space reservations for all disk systems
• cta/CTA#1023 - Retrieve puts the queue to sleep if the eos disk instance is not reachable

v4.5.0-1

Summary

Features

• Improve cta-versionlock script
• cta/CTA#1091 - New Drive State table in CTA Catalogue
• cta/CTA#1054 - Fix filing of disk buffer when recalling from tapeservers with RAO
• cta/CTA#1076 - Retrieve fails if disk system configuration is removed
• cta/CTA#1087 - Add new tapeserver config option UseEncryption
• cta/CTA#1096 - Better handling of bad checksums in archive/retrieve sessions

Bug fixes

• cta/CTA#1092 - Fix overflow error with drive state latestBandwith causing cta frontend crash
• cta/CTA#501 - Fix disappearing reason when TapeDrive is reading or writing

v4.4.1-1

Bug fixes

• cta/CTA#1092 Fix overflow with drive bandwith causing frontend to crash

v4.4.0-1

Upgrade Instructions

This CTA release requires a database schema upgrade to CTA catalogue schema v4.3. Please consult the database upgrade documentation.

Features

• Upgraded EOS to 4.8.67 in CI versionlock.list file
  • EOS/EOS-4976 Fix activity field passed from EOS to CTA
• cta/CTA#607 - Add client host and username in cta-frontend logs
• cta/CTA#777 - Minimize mounts for dual copy tape pool recalls
• cta/CTA#928 - Add youngest request age to cta-admin sq
• cta/CTA#1020 - cta-restore-deleted-files command for restoring deleted files
• cta/CTA#1026 - Add activity Mount Policy resolution to CTA
• cta/CTA#1057 - Remove support for MySQL
• cta/CTA#1069 - Open BackendVFS ObjectStore files in R/W mode when obtaining exclusive locks
• cta/CTA#1070 - Update eos to version 4.8.67
• cta/CTA#1074 - Improve error reporting when retrieving archive
• cta/CTA#1077 - Remove activity fair scheduling logic
• cta/CTA#1083 - Upgrade ceph to version 15.2.15
• cta/CTA#1068 - Build cta with gcc 7.x(C++17) and fix deprecated code

Bug fixes

• cta/CTA#1059 - Clean up warnings reported by static analysis
• cta/CTA#1062 - cta-admin tf rm should store the diskFilePath when deleting the tape file copy
• cta/CTA#1073 - Retry failed reporting for archive jobs
• cta/CTA#1078 - fix STALE message when cta-taped is restarting
• cta/CTA#1081 - refactor database queries for drive ls

v4.3-3

Summary

Features

• cta/CTA#1053 Remove the objectstore presence in another classes
• cta/CTA#501 cta-taped should set the state to DOWN when machine rebooting

Bug fixes

• cta/CTA#1056 Fix bugs in cta 4.3-2
• cta/CTA#1058 Remove helgrind_annotator from production

v4.3-2

Summary

Features

• cta-admin: All short options with more than one character now require two dashes

Bug fixes

• cta/CTA#1013 reportType uninitialized
• cta/CTA#1044 Fix segmentation fault due to uninitialized optional value and remove diskSpaceReservations from cta-admin dr ls

v4.3-1

Summary

Features

• cta/CTA#976 Add logical part of Drive Status using Catalogue
• cta/CTA#988 Add diskSpaceReservations map in cta-admin --json dr ls output.
• cta/CTA#983
  • Add cta-versionlock helper script to cta-release package
  • Update cta repo file to use the new public repo
• cta/CTA#1036 Better error reporting in cta-admin tools
• cta/CTA#1039 Improve logging of cta admin commands in cta frontend
• cta/CTA#1041 Fix host values in cta-admin commands

Bug fixes

• cta/CTA#501 cta-taped should set the state to DOWN when machine rebooting
• cta/CTA#955 cta-taped daemon should stop "immediately" and cleanly when systemctl stop/restart is executed
• cta/CTA#991 Drive is not put down if the device file is removed while cta-taped is running and before a data transfer
• cta/CTA#996 Removes PARALLEL from migration scripts
• cta/CTA#1029 Fix segmentatin fault in frontend when list repacks of a tape that has been deleted in the catalogue
• cta/CTA#1031 Fix Warning in updateDriveStatus
• cta/CTA#1032 cta-admin dr ls crashes the frontend if executed during an archive/retrieve

v4.2-3

Summary

Features

• cta/CTA#983
  • Add cta-versionlock helper script to cta-release package
  • Update cta repo file to use the new public repo
• cta/CTA#1036 Better error reporting in cta-admin tools
• cta/CTA#1039 Improve logging of cta admin commands in cta frontend
• cta/CTA#1041 Fix host values in cta-admin commands
• cta/CTA#1026 cta-admin sq now shows the mount policies of potential mounts

Bug fixes

• cta/CTA#1029 Fix segmentatin fault in frontend when list repacks of a tape that has been deleted in the catalogue
• cta/CTA#1032 cta-admin dr ls crashes the frontend if executed during an archive/retrieve
• cta/CTA#996 Removes PARALLEL from migration scripts
• cta/CTA#1035 log configuration options on frontend startup
• cta/CTA#1042 Do not iterate over retrieve queues when holding global scheduler lock
  • Repacks on a disabled tape must now use a mount policy whose name starts with repack
  • There is no longer an empty mount when a disabled/broken tape queue is full of deleted requests
• cta/CTA#1027 Mitigate popNextBatch bad behaviour in archive queues
• Migration tools: fixes filemode of files imported from CASTOR to EOS

v4.2-2

Summary

Bug fixes

• cta/CTA#1029 Fix segmentatin fault in frontend when list repacks of a tape that has been deleted in the catalogue

v4.2-1

Summary

Features

• Catalogue schema version 4.2
• cta/CTA#1001 Maximum file size is now defined by VO instead of globally.
• cta/CTA#1019 New command cta-readtp allows reading files from tape and verifying their checksum

v4.1-1

Summary

Features

• Catalogue schema version 4.1
• cta/CTA#1016 New options for filtering deleted files using cta-admin rtf ls command.
• cta/CTA#983 Add cta-release package for public binary rpm distribution.
• cta/CTA#980 Add external encryption script option
• cta/CTA#976 Define a new table in the DB schema to contain the drive status.
• cta/CTA#834 New command "recycletf restore" allows undeleting a copy of a file from tape deleted using tapefile rm
• [frontend] New command "tapefile rm" allows deleting a copy of a file from tape

Bug fixes

• cta/CTA#1014 Fix last column alignment when more than 1000 items are listed.

v4.0-5

Summary

Features

• [frontend] Add options to "tapepool ls" to filter tapepools on their name, vo and encryption
• cta/CTA#898 cta-send-event now gets the requester id and eos instance as command line arguments
• cta/CTA#1005 "tape ls" now can filter tapes on wether they were imported from CASTOR
• cta/CTA#1006 "repack ls" now shows the tapepool of the tape being repacked

v4.0-5

Summary

Features

• [frontend] Add options to "tapepool ls" to filter tapepools on their name, vo and encryption
• cta/CTA#898 cta-send-event now gets the requester id and eos instance as command line arguments
• cta/CTA#1005 "tape ls" now can filter tapes on wether they were imported from CASTOR
• cta/CTA#1006 "repack ls" now shows the tapepool of the tape being repacked

Bug fixes

• [frontend] Adds missing break after "schedulinginfo ls" command
• cta/CTA#999 Adds a default mount rule
• cta/CTA#1003 The expansion of a repack request now fails if the archive route for archiving the repacked files is missing

v4.0-4

Bug fixes

• cta/CTA#1002 Do not requeue report jobs when reportType is NoReportRequired

v4.0-3

Features

• Upgraded ceph to version 14.2.20
• Adds cta-verify-file to cta-cli RPM

v4.0-2

Summary

Features

• Upgraded EOS to 4.8.45 in CI versionlock.list file
  • EOS/EOS-4658 EOS workflow engine should not insist on the W_OK mode bit (for prepare ACL)
  • EOS/EOS-4684 Make the "file archived" GC aware of different EOS spaces
• Upgraded eos-xrootd to 4.12.8 in CI versionlock.list file
• cta/CTA#966 Unable to distinguish empty and wrong tape pool
• cta/CTA#926 Improve MigrationReportPacker::ReportSkipped::execute() exception message
• cta/CTA#584 Validate checksum when recalling from tape

Bug fixes

• cta/CTA#582 Changes cta-admin storageclass add --copynb to --numberofcopies
• cta/CTA#987 Do not import metadata for zero-length files into CTA Catalogue
• cta/CTA#984 Removes diskFileGid == 0 check
• cta/CTA#669 cta-taped now display the correct error message when the drive device does not exist
• cta/CTA#777 Minimize mounts for dual copy tape pool recalls - Temporary fix
• cta/CTA#927 Bad message for archive route pointing non-existing pool
• cta/CTA#930 Batched the queueing and the deleting of the repack subrequests
• cta/CTA#969 xrdfs query prepare malformed JSON output
• cta/CTA#972 Updates cta-taped and rmcd man pages
• cta/CTA#967 CI DB cleanup issues
• cta/CTA#982 cta-fst-gcd ignored eos files with fid>4294967295
• cta/operations#352 Unable to delete empty tape pool

v4.0-1

Summary

This version contains the last and clean version of the CTA catalogue schema. This CTA version can not run anymore with the CTA v3.1-14.

Features

• Catalogue schema version 4.0
• cta/CTA#964 Adds failure log messages to processCLOSEW in CTA Frontend
• When the operator submits a tape to repack, a check is done about the tape state before queueing the repack request to ensure it can be repacked
• Oracle catalogue migration scripts 3.1to3.2.sql: replaced DELETE FROM table_name by TRUNCATE TABLE table_name

v3.2-1

Summary

This version is a transition version between CTA v3.1-14 and CTA v4.0. The functionalities of the v4.0 are implemented, but at the database level, the columns to be deleted have been put as NULLABLE and the NOT NULL columns to be added as NULLABLE. The xrootd-ssi-protobuf-interface is not up-to-date with the CTA v4.0: deprecated fields have not been removed.

Features

• Catalogue schema version 3.2
• Upgraded EOS to 4.8.37-1
• cta/CTA#922 The superseded concept has been removed and replaced by a new recycle bin
• cta/CTA#943 A new tape lifecycle logic has been implemented
• cta/CTA#948 The max drives allowed for reading and writing are now set per virtual organization and not per mount policy anymore
• cta/CTA#952 Reclaiming a tape resets the IS_FROM_CASTOR flag to 0
• cta/CTA#951 The query used by RdbmsCatalogueTapeContentsItor has been put back to the state it was in v3.1-13
• cta/CTA#883 Tape verification tool

v3.1-14

Summary

Features

• Upgraded EOS to 4.8.35-1
• cta/CTA#954 The drive is put down if the CleanerSession fails to eject the tape from it after a shutdown
• cta/CTA#945 The timeout for tape load action is now configurable

Bug fixes

• cta/CTA#957 The state of the successful Archive for repack jobs is now changed before being queued for reporting
• cta/CTA#958 The RepackRequest garbage collection now changes the owner of the RepackRequest

v3.1-13

Summary

Features

• Upgraded EOS to 4.8.34-1
• Upgraded xrootd to 4.12.6-1

Bug fixes

• cta/CTA#941 Slow cta-admin sq even when there is very little activity
• cta/CTA#951 Improve the performance of RdbmsCatalogueTapeContentsItor
• cta/CTA#939 cta-objectstore-dereference-removed-queue removes all kind of manually deleted queues from the RootEntry

v3.1-12

Features

• Upgraded EOS to 4.8.30-1

v3.1-11

Features

• cta/CTA#932 Add environment file for cta-frontend service: frontend configured to use 10 XRootD polling threads by default
• cta/CTA#292 Allow non interactive usages of cta-admin with sss authentication
• Upgraded EOS to 4.8.29-1
  • cta/operations#155 Fix for conversion issues
  • cta/operations#154 Improve sys.retrieve.req_id to allow to cancel retrieves on a running instance: adding epoch timestamp in ids
  • EOS-4505 Separate archive and retrieve ACLs in EOS: only needs p ACL for prepare

Bug fixes

• cta/operations#150 high priority Archive job not scheduled when Repack is running: fixed

v3.1-10

Summary

This version contains different bug fixes

Features

Unuseful WARNING logs are now DEBUG logs

Bug fixes

cta/CTA#837 Repack now fails if the repack buffer VID directory cannot be created during expansion cta/CTA#920 Archive and Retrieve error report URL correction on the cta-send-event cmdline tool cta/CTA#923 Corrected the cta-admin showqueues command to display all the retrieve queues of tapes that are on the same tapepool

v3.1-9

Summary

This release contains an improvement allowing to fetch the EOS free space via an external script for backpressure

Features

• Upgraded EOS to 4.8.26-1
• cta/CTA#907 For backpressure, the EOS free space can be fetched by calling an external script

Bug fixes

• cta/CTA#917 Corrected the bug in the cta-admin showqueues command in the case ArchiveForUser and ArchiveForRepack exist for the same tapepool
• cta/CTA#919 Archive queue oldestjobcreationtime is now updated at each pop from the ArchiveQueue

v3.1-8

Summary

This release contains the CTA software Recommended Access Order (RAO) implemented for LTO drives to improve retrieve performances

Features

• CTA software Recommended Access Order (RAO) implemented for LTO drives
• Upgraded EOS to 4.8.24-1
• Upgraded xrootd to 4.12.5-1
• cta-admin repack ls tabular output improvements
• Repack management execution can be disabled via the cta-taped configuration file
• cta/CTA#907 Maintenance process can be disabled via the cta-taped configuration file

Modifications

• Catalogue refactoring

Bug fixes

• cta/CTA#901 cta-admin tapefile ls too slow
• cta/CTA#895 [catalogue] RdbmsCatalogue::deleteLogicalLibrary does not delete empty logical library
• utils::trimString() now returns an empty string if the string passed in parameter contains only white-space characters
• Repack request and sub-requests are now unowned from their Agent when completed

v3.1-7

Summary

This release contains a correction of a performance issue introduced in the v3.1-6 version

Bug fixes

• cta/CTA#893 Corrected slowliness of RdbmsCatalogue::getArchiveFileToRetrieveByArchiveFileId()

v3.1-6

Summary

This release contains some minor improvements.

Features