Commit a29df67b authored by Michael Davis's avatar Michael Davis
Browse files

[doc] Removes EOSCTA doc (converted to mkdocs)

parent fa6ce06e
This diff is collapsed.
\chapter{Archive Workflow}
\label{archive}
\subsection*{Archive File}
\begin{itemize}
\item File create: check Storage Class and allocate Archive ID on CREATE event. Archive file after close write (CLOSEW) event.
\item File update: denied for files with copy on tape. This can be achieved by administrators by adding an immutable
flag (\texttt{!u}) to the ACL of the directories configured to go to tape in EOS, or as a rule in EOS.
\end{itemize}
\begin{figure}[t]
\resizebox{\linewidth}{!}{
\begin{sequencediagram}
\newthread{c}{:Client} \newinst[3]{ef}{:EOS FST} \newinst[1]{em}{:EOS MGM} \newinst[3]{cf}{:CTA FE}
\newinst[1]{ct}{:CTA TS} \begin{call}{c}{open}{em}{redirect}
\begin{call}{em}{check storageclass}{cf}{ack}\end{call}
\end{call} \begin{call}{c}{open}{ef}{ack}\end{call} \begin{messcall}{c}{write}{ef}\end{messcall}
\begin{messcall}{c}{write}{ef}\end{messcall} \begin{call}{c}{close}{ef}{ack}
\begin{call}{ef}{commit}{em}{ack}
\begin{call}{em}{notification(CLOSEW)}{cf}{notification reply}\end{call}
\end{call}
\end{call} \begin{call}{ct}{read file in tape session}{ct}{}
\begin{call}{ct}{open}{em}{redirect}\end{call} \begin{messcall}{ct}{open}{ef}\end{messcall}
\begin{messcall}{ct}{read}{ef}\end{messcall} \begin{messcall}{ct}{read}{ef}\end{messcall}
\begin{messcall}{ct}{close}{ef}\end{messcall} \begin{call}{ct}{tapereplica}{em}{ack}
\begin{call}{em}{unpin disk copy}{em}{}\end{call}
\end{call}
\end{call}
\end{sequencediagram}
} \caption{File write and archive queuing (synchronous)} \label{fig:write-archive-sync}
\end{figure}
Figure~\ref{fig:write-archive-sync} shows the sequence of a client writing a file to EOS. The storage class is checked
on CREATE and a synchronous archive request is queued on CLOSEW.
\section{Create file (CREATE)}
The following operations are carried out on a CREATE event:
\begin{itemize}
\item EOS calls CTA with the file metadata
\item CTA Frontend validates Storage Class
\item CTA Frontend determines destination Tape Pool
\item CTA Frontend sends metadata to the Catalogue
\item \textbf{if no error:} CTA Frontend generates a unique Archive ID and returns it to EOS in a synchronous
response.
\item EOS attaches Archive ID to file as an extended attribute (\texttt{xattr})
\end{itemize}
In this scheme, if any part of the \textbf{CREATE} workflow fails, nothing is archived and no Archive ID is attached to
the file \texttt{xattrs}. This guarantees that EOS is informed that something went wrong, and we stay in a consistent
state. EOS cannot execute the \textbf{CLOSEW} workflow without the archive ID.
CTA will guarantee that the \textbf{CREATE} workflow returns quickly. If there is a need to optimise, the Storage Classes
can be held in memory by the CTA Frontend to avoid a database lookup. Likewise the process to generate the Archive ID must
be fast and guaranteed to complete in bounded time.
If EOS loses the archive ID for any reason, no further operations on the file are possible. Inconsistencies between the
EOS namespace and CTA namespace will be picked up during reconciliation (see Appendix~\ref{reconciliation_strategy}).
\section{Archive file (CLOSEW)}
The following operations are carried out on a file close write (CLOSEW) event:
\begin{itemize}
\item EOS calls CTA with the file metadata. The xattrs now include the validated Storage Class and the CTA Archive ID.
\item CTA Frontend queues the archive request and synchronously returns a status code to EOS.
File state is \textit{archive in progress}.
\item On successful write of the first tape copy, the tape server notifies EOS. File state is \textit{one copy on tape}.
This equates to \textit{m-bit set} in CASTOR.
\item On successful write of each tape copy, the tape server notifies EOS.
\item On successful write of the last tape copy, the tape server notifies EOS. The number of copies on tape is
updated. File state is \textit{archived}.
\end{itemize}
\gitlab{228}{Test archving files with \texttt{num\_copies} > 1}
In the \textbf{CLOSEW} workflow, we cannot end up in a state where the file was successfully archived but EOS does not have
the archive ID. The only possible inconsistency between EOS and CTA is when we successfully archived at least one copy of the
file but did not successfully notify EOS. In this case, the operator should be notified and the EOS user can retry.
The number of copies on tape is stored in the EOS namespace under FS ID 65535 (see~\S\ref{eos_support_of_tape_notions}).
\begin{alertbox}
Reporting metadata in ``tape replica'' (checkum and size) in addition to archive completion allows EOS to detect
discrepancies (as happened when requests got mixed up in initial tests).
\end{alertbox}
\chapter{Constraints}
\label{constraints}
\section{Performance requirements}
\label{perfreq}
The CTA metadata performance requirements have been assessed in previous work\footnote{see cta.pdf:
Object Store/Performance considerations.}.
Efficient metadata handling is particularly important for repacking, where small files can be encountered.
The data-taking performance might be less stringent in terms of metadata, but involves a bigger stack with EOS
in front. This should be assessed in order to have full system performance targets. This includes the bandwidth to be
achieved and the latency experienced by the user (a synchronous close on write will increase latency, we should
make sure the result is adequate).
\section{Synchronous calls}
The EOS workflow engine will support synchronous actions. EOS clients will call EOS which in turn will call CTA.
CTA will send back a ``return value'' reply which will be synchronously relayed back to the EOS client.
\vspace{0.5ex}
\noindent\textbf{Example:} An EOS client opens a file for creation that should be eventually archived to tape. EOS
synchronously calls CTA to determine whether or not the Storage Class of the file is known and that it has a destination
tape pool. If these two conditions are not met then the EOS client will get an immediate synchronous
reply saying the file cannot be created because it cannot be archived to tape.
\section{Data Integrity}
\textbf{After-the-fact check on archive from EOS: }
EOS will schedule a second workflow job when an archive is triggered. This will check the archive status and re-trigger
it if needed at a later time.
\section{Operational constraints}
Before a repack campaign, users should be encouraged to purge any unnecessary data from EOS. After this operation, a
reconciliation between the user catalogue and EOS (and then between EOS and CTA) should be done to ensure no unexpected
data will get deleted during the repack operation.
\chapter{Deletion and Garbage Collection}
\label{delete_garbage_collect}
\subsection*{Delete File}
\begin{itemize}
\item User-triggered disk copy removal (allowed or not, optional)
\item Garbage collection of disk copies
\item Complete deletion of files
\end{itemize}
\section{User-triggered disk copy removal}
CASTOR has learned that it is not easy or even possible to implement the exact ``garbage collection'' policy
required by experiments when it comes to deleting disk copies of files safely stored on tape. CASTOR has provided
the {\tt{}stager\_rm} command to end users to enable them to manually garbage collect files in their CASTOR disk
cache. We currently believe that an equivalent of the {\tt{}stager\_rm} command should be implemented in EOS.
Such a command could simply be a request to execute a {\tt{}stager\_rm} workflow action on a specific file.
\section{Garbage collection of disk copies}
A double-criteria garbage collection will probably be necessary to keep free space in disk pools (file age
(LRU/FIFO/etc. \ldots) + pinning).
\section{Complete deletion of files}
There is no interest in reporting failure to delete the file in CTA while the deletion proceeds in EOS,
so synchronous and asynchronous implementations are equivalent. The complete deletion of files from EOS raises
several race conditions (delete while archiving, delete while retrieving), but all will probably be resolved by
failure of data or metadata operations initiated from CTA to EOS, plus slow reconciliation. The deletion of the
file can be represented by a notification message (as any file operations can).
\chapter{Introduction}
\label{introduction}
EOSCTA is CERN's next-generation software solution for the archival and retrieval of data stored on tape.
The CERN Tape Archive (CTA) is the tape back-end to the EOS disk system, and together EOS+CTA forms the
complete data archival system:
\begin{enumerate}
\item The EOS disk system manages the namespace for data stored on tape; provides a storage buffer for
files during archival and retrieval operations; and provides the interface between the external client
and CTA.
\item CTA manages the tape file catalogue; provides queueing and scheduling functions; and controls data
transfers to and from the tape hardware.
\item The third element in the system is the client software, which communicates with EOSCTA using the
XRootD protocol. The client is responsible for transferring files into and out of EOS, and managing
failures and retries. The most commonly-used client is CERN's File Transfer System (FTS) but in
principle any client which can communicate using the XRootD protocol can be used.
\end{enumerate}
This document describes the EOSCTA workflows (archival, retrieval and deletion of files), and the interfaces
and protocols used between the different system components.
The rest of this document is organised as follows:
Chapters~\ref{archive}--\ref{delete_garbage_collect} describe the primary workflows: Archival (Ch.~\ref{archive}),
Retrieval (Ch.~\ref{retrieve}) and Deletion/Garbage Collection (Ch.~\ref{delete_garbage_collect}).
Chapters~\ref{protocol}--\ref{security} describe the communication between the EOS and CTA components
of the system. Ch.~\ref{protocol} describes the communication protocol. Ch.~\ref{security} describes the security
protocols.
Three appendices discuss the need for reconciliation between EOS and CTA (Appendix~\ref{reconciliation_strategy}),
system constraints (Appendix~\ref{constraints}) and outstanding questions and issues which need to be agreed on or
resolved (Appendix~\ref{questions_and_issues}).
\chapter{EOS-CTA Protocol}
\label{protocol}
\section{Transport Layer}
The communication channel between EOS and CTA is the
\href{http://xrootd.org/doc/dev42/ssi\_reference.htm}{XRootD Scalable Service Interface v2 (SSIv2)} protocol.
The XRootD SSI protocol allows maximum parallelisation of requests served to the CTA Frontend,
\textit{i.e.} without per-file serialization. The CTA Frontend internally packs requests together to improve database
and object store bandwidth, to meet the performance requirements described in \S\ref{perfreq}.
\section{Protocol Layer}
\label{dataSerialization}
All Requests and Responses sent between EOS to CTA are defined and serialized using Google Protocol Buffers v3.
\begin{itemize}
\item A workflow event of type {\tt proto:} is assigned by the administrator to directories where events should be
propagated to CTA.
\item When a workflow event occurs, a {\tt notification} message will be sent to the CTA Frontend. Notifications are
synchronous: EOS will receive an acknowledgement or an error message.
\item The workflows do not set any file properties on the EOS side. These are set by CTA via the {\tt xattr} map in
the Response message or by one of the URL callbacks specified below.
\item The Protocol Buffer definition provides the CTA Frontend with everything EOS knows about the file.
The CTA Frontend will select only the information it needs from the Protocol Buffer message.
\end{itemize}
The Protocol Buffers are defined in the
\href{https://gitlab.cern.ch/eos/xrootd-ssi-protobuf-interface}{xrootd-ssi-protobuf-interface}
project, which is shared between EOS and CTA. This project comprises:
\begin{itemize}
\item A set of generic C++ headers to bind a set of Protocol Buffer definitions to the XRootD SSIv2 transport
layer\footnote{The headers are not specific to EOS or CTA; in principle they can be used for any project which
has a client and server communicating using protocol buffers over XRootD SSIv2.}.
\item The Protocol Buffer definitions which instantiate the EOS--CTA interface.
\end{itemize}
\section{Protocol Buffer URL Fields}
The event notification Protocol Buffer includes three URL fields which are constructed by EOS:
\begin{lstlisting}
message Service {
string name = 1; //< name of the service
string url = 2; //< access url of the service
}
message Workflow {
...
cta.common.Service instance = 5; //< instance information
...
}
message Transport {
string dst_url = 1; //< transport destination URL
string report_url = 2; //< URL to report successful archiving
}
\end{lstlisting}
The access URL and report URL are sent with archive events.
The transport destination URL is sent with retrieve events.
\begin{alertbox}[Retrieval Notification Message]
Possibly we need another callback URL to provide notification of a successful retrieval\slash error message in case
of a failed retrieval \textit{(under discussion)}.
\end{alertbox}
\subsection{Service Access URL}
The Service Access URL \texttt{Request.notification.wf.instance.url} is used by the CTA Tape Server daemon
\texttt{cta-taped} to read the disk file from the EOS MGM during an \textbf{archive} event.
Example URL:
\begin{lstlisting}
root://ctapps-eossrv01.cern.ch//eos/ctapps/preprod/ed8f3136/0/test099999?eos.lfn=291198a
\end{lstlisting}
which is composed as follows:
\begin{lstlisting}
root://<hostname>.cern.ch/<path>/<filename>?eos.lfn=<fid>
\end{lstlisting}
\begin{itemize}
\item\texttt{<hostname>} is the hostname of the EOS MGM
\item\texttt{<path>/<filename>} is the absolute EOS path to the file
\item\texttt{<fid>} is the file ID, given in hex format as \texttt{fxid:<fid-hex>}
\end{itemize}
\subsection{File Archived Report URL}
The File Archived Report URL \texttt{Request.notification.transport.report\_url} is used by the Tape Server to
asynchronously report to the EOS MGM that a file has been safely archived to tape.
The URL is composed as follows:
\begin{lstlisting}
eosQuery://<hostname>.cern.ch/<path>/<filename>?mgm.pcmd=event&mgm.fid=<fid-hex>& mgm.logid=cta&mgm.event=archived&mgm.workflow=default&mgm.path=<path>/<filename>& mgm.ruid=<ruid>&mgm.rgid=<rgid>
\end{lstlisting}
\begin{itemize}
\item\texttt{mgm.pcmd} tells the MGM to execute a workflow event (\texttt{event})
\item\texttt{mgm.fid} gives the file ID of the file that was archived, in hexadecimal digits
\item\texttt{mgm.logid} gives the Log ID (\texttt{cta})
\item\texttt{mgm.event} sets which workflow event to execute (\texttt{archived})
\item\texttt{mgm.workflow} sets which workflow to execute (\texttt{default})
\item\texttt{mgm.path} gives the path to the EOS disk file
\item\texttt{mgm.ruid} sets the user ID of the user invoking the workflow
\item\texttt{mgm.rgid} sets the group ID of the user invoking the workflow
\end{itemize}
The MGM will respond by executing the \texttt{sys.workflow.archived.default} workflow against the specified file as
user/group \texttt{<ruid>:<rgid>}.
\subsection{Transport Destination URL}
The Transport Destination URL \texttt{Request.notification.transport.dst\_url} is used by the Tape Server to write a tape
file to the EOS MGM during a \textbf{retrieve} event.
The URL is composed as follows:
\begin{lstlisting}
root://<hostname>.cern.ch/<path>/<filename>?eos.lfn=<fid>&eos.ruid=<ruid>& eos.rgid=<rgid>&eos.injection=1&eos.workflow=none
\end{lstlisting}
\begin{itemize}
\item\texttt{eos.lfn} specifies the file ID, given in hex format as \texttt{fxid:<fid-hex>}
\item\texttt{eos.ruid} sets the user ID of the user invoking the workflow
\item\texttt{eos.rgid} sets the group ID of the user invoking the workflow
\item\texttt{eos.injection=1} informs EOS that we are sending it data to write to the file
\item\texttt{eos.workflow} sets which workflow event to execute (\texttt{none})
\end{itemize}
\textbf{Note:} When EOS writes a file to disk, the \texttt{closew.default} workflow will be triggered, to archive the
file on tape. This workflow should obviously not be triggered when the file being written is being retreived from tape.
The URL overrides the default workflow by explicitly setting it to \texttt{none}.
\section{Protocol Buffer versions}
If the Protocol Buffer definitions are updated:
\begin{itemize}
\item First install the new version into CTA.
\item Next, upgrade EOS instances (one-at-a-time).
\end{itemize}
CTA must maintain compatibility with old protocol versions until the upgrades have been rolled out to all experiments.
\section{EOS support of tape notions}
\label{eos_support_of_tape_notions}
EOS keeps track of tape-related information using extended attributes:
\begin{itemize}
\item The {\tt CTA\_StorageClass} attribute indicates that one or more copies should be stored on tape under the
specified Storage Class. The CTA database will specify the number of tape copies that should be created for that
Storage Class.
\item EOS reserves filesystem ID 65535 for tape operations. This should be specified in all tape directories using
the extended attribute {\tt CTA\_TapeFsId="65535"}. EOS treats FS ID 65535 as a phantom filesystem in the EOS
namespace, using it to record the number of copies stored on tape.
\end{itemize}
\newcommand{\resolved}{{\color{cern@blue}[RESOLVED]}}
\chapter{Questions and Issues}
\label{questions_and_issues}
\section{Success and Failure for Archive Messages}
\textit{Results of a discussion between Jozsef, Giuseppe and Eric about the success and failure behaviour for archive requests:}
The current behaviour in CASTOR is that the file remains to be migrated until all copies have been successfully
migrated. Failed migration jobs get deleted, so the file cannot be \texttt{stager\_rm}ed or garbage collected
before an operator intervenes.
We think a similar behaviour should be implemented in EOSCTA:
\begin{itemize}
\item The file will be opened to garbage collection when all copies are successfully archived to tape.
CTA will not report success before that (this is already the case).
\item When a failure occurs (after exhausting retries), CTA will report the error to EOS, with a new
error-reporting URL (to be implemented). The job will then be placed in a failed job queue to be
handled by operators.
\item EOS will keep track of and expose to the user only the latest error (we have potentially one
per tape copy, and if operator decides to retry the job entirely, the error could be reported again).
\item EOS will clear the errors when receiving a success.
\end{itemize}
\section{Immutable files}
The files with an archive on tape should be immutable in EOS (raw data use case), or a delayed archive mechanism
should be devised for mutable files (CERNBox archive use case).
Immutability of a file is guaranteed by adding \texttt{u!} to the EOS ACL.
Currently we do not enforce this on the CTA Frontend side, we just assume EOS is taking care of it.
If we decide it's useful for CTA to check immutability of archived files, we could send the ACL across with the xattrs.
This is not sent at the moment, because all system and user attributes are filtered out.
\section{When can files be deleted?}
Disk copies cannot be deleted before they are archived on tape (pinning).
The full file could still be deleted, potentially leading to issues to be handled in the tape archive session.
\section{What should be the protocol for fast reconciliation?}
The workflow will both trigger the synchronous archive queuing and post a second delayed workflow job that will
check and re-issue the request if needed (in case the request gets lost in CTA). This event-driven reconciliation acts
as a fast reconciliation. The criteria to check the file status will be the EOS side status which CTA reports
asynchronously to EOS (see~\S\ref{dataSerialization}).
\section{When a file has multiple tape copies, when are notifications are sent to EOS?}
EOS will need to represent and handle part the tape status of the files. This includes the fact that the file should be
on tape, the name of the CTA storage class, and the mutually exclusive statuses indicated by CTA: not on tape, partially
on tape, fully on tape. The report from CTA will use the ``tape replica'' message (see~\S\ref{dataSerialization}).
For CASTOR, there is an additional constraint that the disk copy cannot be deleted until all tape copies have been
successfully written. The above scheme keeps track of the number of tape copies written and it will be up to the
EOS developers to ensure that this constraint is observed.
In CASTOR, the following notifications are sent during archiving a file with $n$ tape copies:
\begin{itemize}
\item On successful write of the first tape copy, the \textbf{m-bit} is set. This indicates to the experiment that
they can safely delete their copy of the data.
\item On successful write of the $n^{th}$ tape copy, the \textbf{CAN\_BE\_MIGR} status is set in the database. This
indicates that the file can be deleted from CASTOR's staging area.
\end{itemize}
For CTA, at what point(s) should we notify EOS that a file has been archived?
\begin{itemize}
\item After the first copy is archived?
\item After each copy is archived?
\item After the $n^{th}$ copy is archived?
\end{itemize}
\gitlab{228}{Test archving files with \texttt{num\_copies} $gt 1$}
\section{Should the CTA catalogue methods prepareForNewFile() and prepareToRetrieveFile() detect repeated requests from
EOS instances?}
EOS does not keep track of requests which have been issued. We have said that CTA should implement idempotent retrieve queuing.
What are the consequences if we do not implement idempotent retrieve queuing?
What about archives and deletes?
\subsection{If so how should the catalogue communicate such ``duplicate'' requests to the caller (Scheduler\slash cta-frontend plugin)?}
The CTA Frontend calls the Scheduler which calls the Catalogue.
There are several possible schemes for handling duplicate jobs:
\begin{enumerate}
\item If duplicates are rare, perhaps they don't need to be explicitly handled
\item When a retrieve job is submitted, the Scheduler could check in the Catalogue for duplicates
\item When a retrieve job completes, the Tape Server could notify the Scheduler, which could then check for and
drop any duplicate jobs in its queue.
\end{enumerate}
Reporting of retrieve status could set an \texttt{xattr}. Then the user would be able to monitor status which could
reduce duplicate requests.
Failed archivals or other CTA errors could also be logged as an \texttt{xattr}.
\subsection{If the CTA catalogue keeps an index of ongoing archive and retrieve requests, what will be the new
protocol additions (EOS, cta-frontend and cta-taped) required to guarantee that ``never completed'' requests are removed
from the catalogue?}
Such a protocol addition could be something as simple as a timeout.
% \section{How do we deal with the fact that the current C++ code of the EOS/CTA interface that needs to be compiled on
% the EOS side on SLC6 will not compile because it uses std::future?}
%
% Please could you take on the responsibility of addressing the EOS/CTA interface issues described by these questions.
% You do NOT need to do any work towards these issues before the next Wednesday meeting. These issues need to be
% vetted by the whole team during the meeting. What's left is then under your responsibility.
% I am sending you these questions in order to start the process of concretely describing the scope of your EOS/CTA
% interface work. More issues are described within the EOS/CTA interface document of Eric.
%
% If and only if there is time during the next tape developments meeting, we should try to vet the above questions and
% decide on the most appropriate place to have them written up, in other words do they become gitlab issues or do they
% get added to the EOS/CTA interface document? If we don’t have time during the meeting then those interested can have
% a separate meeting sometime later.
\section{CTA Failure}
What is the mechanism for restarting a failed archive request (in the case that EOS accepts the request and CTA fails
subsequently)?
If CTA is unavailable or unable to perform an archive operation, should EOS refuse the archive request and report failure
to the User?
What is the retry policy?
\section{File life cycle}
Full life cycle of files in EOS with copies on tape should be determined (they inherit their tape properties
from the directory, but what happens when the file gets moved or the directory properties changed?).
\section{Storage Classes}
The list of valid storage classes needs to be synchronized between EOS and CTA. EOS should not allow a power user to
label a directory with an invalid storage class. CTA should not delete or invalidate a storage class that is being used
by EOS.
\section{Request Queue}
Chaining of archive and retrieve requests to retrieve requests.
Execution of retrieve requests as disk to disk copy if possible.
%Catalogue will also keep track of requests for each files (archive and retrieve) so that queueing can be made idempotent.
\section{Catalogue}
Catalogue files could hold the necessary info to recreate the archive request if needed.
\section{Questions administrators need to be able to answer}
The \texttt{cta-admin} command should be include functions to allow administrators to answer the following questions:
\begin{itemize}
\item Why is data not going to tape?
\item Why is data not coming out of tapes?
\item Which user is responsible for system overload?
\end{itemize}
\section{User Commands}
What user commands are required? This needs to be reviewed. From the previous documentation:
\textit{For most commands there is a short version and a long one. Due to the limited number of USER commands it is not
convenient (nor intuitive) to use subcommands here (anyway it could be applied only to storage classes).}
{\tt
\begin{itemize}
\item cta lsc/liststorageclass{\normalfont \footnote{this command might seem a duplicate of the corresponding admin command but it actually shows a subset of fields (name and number of copies)}}
\item cta ssc/setstorageclass <dirpath> <storage\_class\_name>
\item cta csc/clearstorageclass <dirpath>
\item cta mkdir <dirpath>
\item cta chown <uid> <gid> <dirpath>{\normalfont \footnote{we may want to add \texttt{chmod} later on}}
\item cta rmdir <dirpath>
\item cta ls <dirpath>
\item cta a/archive <src1> [<src2> [<src3> [...]]] <dst>
\item cta r/retrieve <src1> [<src2> [<src3> [...]]] <dst>
\item cta da/deletearchive <dst>{\normalfont \footnote{this works both on ongoing and finished archives, that is why it's called ``delete''}}
\item cta cr/cancelretrieve <dst>{\normalfont \footnote{this clearly works only on ongoing retrieves, obviously does not delete destination files, that's why it's called ``cancel''}}
\end{itemize}
}
\section{Return value \resolved}
Notification return structure for synchronous workflows contains the following:
\begin{itemize}
\item Success code (\texttt{RSP\_SUCCESS})
\item A list of extended attributes to set (\textit{e.g.}, set the ``CTA archive ID'' \texttt{xattr} of the EOS file being queued for archival)
\item Failure code (\texttt{RSP\_ERR\_PROTOBUF}, \texttt{RSP\_ERR\_CTA} or \texttt{RSP\_ERR\_USER})
\item Failure message which can be logged by EOS or communicated to the end user (\textit{e.g.}, ``Cannot open file for writing because there is no route to tape'')
\end{itemize}
\begin{lstlisting}
message Response {
enum ResponseType {
RSP_INVALID = 0; //< Response type was not set
RSP_SUCCESS = 1; //< Request is valid and was accepted for processing
RSP_ERR_PROTOBUF = 2; //< Framework error caused by Google Protocol Buffers layer
RSP_ERR_CTA = 3; //< Server error reported by CTA Frontend
RSP_ERR_USER = 4; //< User request is invalid
}
ResponseType type = 1; //< Encode the type of this response
map<string, string> xattr = 2; //< xattribute map
string message_txt = 3; //< Optional response message text
}
\end{lstlisting}
\section{Will EOS instance names within the CTA catalogue be ``long'' or ``short''? \resolved}
\textit{We all agreed to use ``long'' EOS instance names within CTA and specifically the CTA catalogue. An example of a
long EOS instance name is ``eosdev'' with its corresponding short instance name being ``dev''.}
\begin{flushright}
--- Minutes from today's tape developments meeting, Wed 22 Nov 2017
\end{flushright}
This implies that there will be a separate instance name for each VO (``eosatlas'', ``eoscms'', \textit{etc.}) and a
unique SSS key for each instance name.
\section{Do we want the EOS namespace to store CTA archive IDs or not? \resolved}
\begin{description}
\item[If no:] we are allowing that the EOS file ID uniquely identifies the file. We must maintain a one-to-one mapping
from EOS ID to CTA archive ID on our side. This also implies that the file is immutable.
\item[If yes:] we must generate the CTA archive ID and return it to EOS. There must be a guarantee that EOS has attached
the archive ID to the file (probably as an xattr but that's up to the EOS team), i.e. \textbf{the EOS end-user must
never see an EOS file with a tape replica but without an archive ID}. EOS must provide the CTA archive ID as the
key to all requests.
\end{description}
\subsection*{Solution}
Archive IDs will be allocated by CTA when a file is created. The Archive ID will be stored in the EOS
namespace as an extended attribute of the file. EOS must use the archive ID to archive, retrieve or delete files.
Archive IDs are not file IDs, \textit{i.e.} the archive ID identifies the version of the file that was archived. In the
case of Physics data, the files should be immutable so in practice there is one Archive ID per file.
In the backup use case, if we allowed mutable files, we would need a mechanism to track archived file versions. On the
EOS side, changes to files are versioned, so each time a file is updated, the Archive ID should also be updated.
Old versions of the file would maintain a link to their archive copy via the versioned extended attributes. But in this
case we probably also need a way to mark archive copies of redundant versions of files for deletion.
\subsection*{Design notes from Steve}
\textit{One of the reasons I wanted an archive ID in the EOS namespace was that I wanted to have one primary key for the CTA
file catalogue and I wanted it to be the CTA archive ID. Therefore I expected that retrieve and delete requests issued
by EOS would use that key.}
\textit{This ``primary key'' requirement is blown apart by the requirement of the CTA catalogue to
identify duplicate archive requests. The CTA archive ID represents an ``archive request'' and not an individual EOS file.
Today, 5 requests from EOS to archive the same EOS file will result in 5 unique CTA archive IDs. Making the CTA catalogue
detect 4 of these requests as duplicate means adding a ``second'' primary key composed of the EOS instance name and the EOS
file ID. It also adds the necessity to make sure that archive requests complete in the event of failure, so that retries
from EOS will eventually be accepted and not forever refused as duplicate requests. It goes without saying that dropping
the CTA archive ID from EOS also means using the EOS instance name and EOS file ID as primary key for retrieve and delete
requests from EOS.}
\textit{The requirement for a ``second'' primary key may be inevitable for reasons other than (idempotent) archive, retrieve and
delete requests from EOS. CTA tape operators will want to drill down into the CTA catalogue for individual end user files
when data has been lost or something has ``gone wrong''. The question here is, should it be a ``primary key'' as in no
duplicate values or should it just be an index for efficient lookup?}
\chapter{EOS-CTA Reconciliation Strategy}
\label{reconciliation_strategy}