Commit aea0c6c2 authored by Michael Davis's avatar Michael Davis
Browse files

Reorganises CTA-EOS Interface document

parent 54cdb46b
Pipeline #125 canceled with stages
No preview for this file type
File added
No preview for this file type
\documentclass[11pt,a4paper]{report}
\newcommand\doctitle{The CERN Tape Archive: CTA-EOS Interface}
% Other packages
%\usepackage{multirow} % Multi-row tables
\usepackage[table]{xcolor}
%\usepackage{moreverb}
%\usepackage{tabularx}
\usepackage[underline=true,rounded corners=false]{pgf-umlsd}
\input{cta_include}
\lstdefinestyle{customcpp}{language=C++, basicstyle=\ttfamily }
\lstdefinestyle{customcppsmall}{ language=C++, basicstyle=\ttfamily{}\small{} }
\begin{document}
% Title Page
\title{\doctitle}
\author{Eric Cano \and Michael Davis \and Steven Murray}
\maketitle
% Front Matter
\addcontentsline{toc}{chapter}{Contents}
\tableofcontents
% Chapters
\input{CTA_EOS_Intro.tex}
\input{CTA_EOS_Operations.tex}
\input{CTA_EOS_Protocol.tex}
\input{CTA_EOS_Constraints.tex}
\input{CTA_EOS_Reconciliation.tex}
\input{CTA_EOS_Authorization.tex}
\appendix
\input{CTA_EOS_Questions.tex}
\end{document}
\chapter{EOS-CTA Authorization Rules}
\label{authorization_rules}
One of the requirements of CTA is to prevent crosstalk between EOS instances belonging to different VOs, e.g.
the ATLAS EOS instance should not be able to access (or even know about) files belonging to CMS.
\begin{alertbox}[Shared Secrets?]
Should we have a different shared secret for each VO, to explicitly prohibit access to files not belonging to that EOS
instance?
\end{alertbox}
\begin{alertbox}[Redundant Rules?]
The highlighted rules below are probably not required.
\end{alertbox}
\begin{enumerate}
\item A listStorageClass command should return the list of storage classes
belonging to the instance from where the command was executed only
\item A queueArchive command should be authorized only if:
\begin{itemize}
\item the instance provided in the command line coincides with the instance from
where the command was executed
\item the storage class provided in the command line belongs to the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original archive requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A queueRetrieve command should be authorized only if:
\begin{itemize}
\item the instance of the requested file coincides with the instance from where
the command was executed
\item\hl{the EOS username and/or group (of the original retrieve requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A deleteArchive command should be authorized only if:
\begin{itemize}
\item the instance of the file to be deleted coincides with the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original delete requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A cancelRetrieve command should be authorized only if:
\begin{itemize}
\item the instance of the file to be canceled coincides with the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original cancel requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item An updateFileStorageClass command should be authorized only if:
\begin{itemize}
\item the instance of the file to be updated coincides with the instance from
where the command was executed
\item the storage class provided in the command line belongs to the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original update requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item An updateFileInfo command should be authorized only if:
\begin{itemize}
\item the instance of the file to be updated coincides with the instance from
where the command was executed
\end{itemize}
\end{enumerate}
\chapter{Constraints}
\label{constraints}
\section{Performance requirements}
\label{perfreq}
The CTA metadata performance requirements have been assessed in previous work. This ensures smooth repacking where
small files can be encountered. The performance requirements are described in the general document about CTA in
section 8.5 (Object Store/Performance considerations).
The data taking performance might be less stringent in terms of metadata, but involves a bigger stack with EOS
in front should be assessed in order to have full system performance targets. This includes the bandwidth to be
achieved and the latency experienced by the user (a synchronous close on write will increase latency, we should
make sure the result is adequate).
\section{Operational constraints}
Before a repack campaign, user should be encouraged to purge any unnecessary
data from EOS. After this operation, a reconciliation between the user catalogue and EOS (And then EOS and CTA)
should be done to ensure no unexpected data will get deleted during the repack operation.
\chapter{Introduction}
\label{introduction}
This document summarizes the full control chain between the User, EOS and CTA. It is intended to be used for
CTA-EOS interface design.
The file lifecycle includes:
\begin{itemize}
\item creation (write until initial close)
\item updates: they should be denied for files with copy on tape. This can be achieved by administrators by
adding an immutable flag (ACL) to the directories configured to go to tape in EOS, or as rule in EOS.
\item explicit retrieves
\item implicit retrieves (allowed or not, this choice should probably be left to the operator)
\item user triggered disk copy removal (allowed or not, optional)
\item garbage collection of disk copies
\item complete deletion of files
\item metadata changes (on EOS side, then propagated to CTA, rate limited). There are two sub-use cases:
\begin{itemize}
\item Generic metadata update (just involving a catalog entry update)
\item Storage class change (involving a migration to different tape pools (most probably postponed to repack).
\end{itemize}
\item metadata injection (from CTA to EOS: injection of missing files in EOS). There are two sub-use cases:
\begin{itemize}
\item Injection of files from CASTOR during initial data migration.
\item Re-injection of files from CTA to EOS in a disaster recovery.
\end{itemize}
\end{itemize}
In addition, in order to make sure no changes were lost, implicit operations are needed:
\begin{itemize}
\item fast reconciliation (in flight archive requests for sure, maybe retrieve requests as well)
\item full or slow reconciliation (complete name space scan)
\item synchronize the list of valid tape storage classes between EOS and CTA
\end{itemize}
Chapter~\ref{operations} describes the use cases in more detail. Chapter~\ref{protocol} defines the protocol for the
EOS-CTA API. Chapters~\ref{constraints},~\ref{reconciliation_strategy} and~\ref{authorization_rules} discuss constraints
and other operational issues. Issues which need to be considered in more detail are in
Appendix~\ref{questions_and_issues}.
\chapter{CTA Operations}
\label{operations}
\section{File writes}
Figure \ref{fig:write-archive-sync} shows the sequence of a client writing a file to EOS with the optional check
of the storage class on open and synchronous archive request queuing on close. This second options allows to signal
problems to the user, at least on close and possibly as well on open.
\begin{figure}[h]
\resizebox{\linewidth}{!}{
\begin{sequencediagram}
\newthread{c}{:Client} \newinst[3]{ef}{:EOS FST} \newinst[1]{em}{:EOS MGM} \newinst[3]{cf}{:CTA FE}
\newinst[1]{ct}{:CTA TS} \begin{call}{c}{open}{em}{redirect}
\begin{call}{em}{check storageclass (optional)}{cf}{ack}\end{call}
\end{call} \begin{call}{c}{open}{ef}{ack}\end{call} \begin{messcall}{c}{write}{ef}\end{messcall}
\begin{messcall}{c}{write}{ef}\end{messcall} \begin{call}{c}{close}{ef}{ack}
\begin{call}{ef}{commit}{em}{ack}
\begin{call}{em}{notification(closew)}{cf}{notification reply}\end{call}
\end{call}
\end{call} \begin{call}{ct}{read file in tape session}{ct}{}
\begin{call}{ct}{open}{em}{redirect}\end{call} \begin{messcall}{ct}{open}{ef}\end{messcall}
\begin{messcall}{ct}{read}{ef}\end{messcall} \begin{messcall}{ct}{read}{ef}\end{messcall}
\begin{messcall}{ct}{close}{ef}\end{messcall} \begin{call}{ct}{tapereplica}{em}{ack}
\begin{call}{em}{unpin disk copy}{em}{}\end{call}
\end{call}
\end{call}
\end{sequencediagram}
} \caption{File write and archive queuing (synchronous)} \label{fig:write-archive-sync}
\end{figure}
It should be noted that: \begin{itemize} \item disk copies cannot be deleted before they are archived on tape
(pining) -- the full file could still be deleted (potentially
leading to issues to be handled in the tape archive session).
\item the files with an archive on tape should be immutable in EOS (``raw data use case''), or a delayed archive
mechanism should
be devised for mutable files (CERNBox archive use case).
\item synchronous calls allow reporting of CTA failures to the original client (especially in the case of custodial
data (``raw data case''). The request will be sent to CTA as a notification message (see \ref{dataSerialization}).
\item reporting metadata in "tape replica"(checkum and size) in addition to archive completion will allow EOS
to detect discrepancies (like happened when requests got mixed up in initial tests). \item the workflow will
both trigger the synchronous archive queuing and post a second delayed workflow job that will check and re-issue
the request if needed (in case the request gets lots in CTA). This event driven reconciliation acts as a fast
reconciliation. The criteria to check the file status will be the EOS side status (see below) which CTA reports
asynchronously to EOS (see \ref{dataSerialization}). \end{itemize}
EOS will need to represent and handle part the tape status of the files. This includes the fact that the file
should be on tape, the name of the CTA storage class, and the mutually exclusive statuses indicated by CTA:
not on tape, partially on tape, fully on tape. The report from CTA will use the "tape replica" message (see
\ref{dataSerialization}).
\section{Updates} \label{updates}
EOS should filter and forbid updates to files located on tape. Otherwise, a policy should be devised to migrate
at a reasonable rate (once per day?), to allow a backup like behavior (CERNBox files).
\section{Explicit retrieves (prepare)} \label{user_retrieves}
Explicit retrieve will face several use cases. The way we deal with them should be decided. Synchronous and
asynchronous (through WFE) behaviors make little difference, but allow (catastrophic) problems to be reported to
the user.
\begin{itemize} \item The file is not on disk, no request is queued yet and so we should queue the retrieve request.
\item The file is not on disk, but a retrieve request is already queued in CTA. EOS could already know about this
request (stateful case)
or not. CTA should handle the requeuing in all cases.
\item The file is already on disk and EOS should return immediately. \end{itemize}
Figure \ref{fig:explicit-retrieve} describes the enqueuing and subsequent retrieve of a file not present on disk
(synchronous scenario).
\begin{figure}[h]
\resizebox{\linewidth}{!}{
\begin{sequencediagram}
\newthread{c}{:Client} \newinst[3]{ef}{:EOS FST} \newinst[1]{em}{:EOS MGM} \newinst[3]{cf}{:CTA FE}
\newinst[1]{ct}{:CTA TS} \begin{call}{c}{prepare}{em}{ack}
\begin{call}{em}{cta retrieve}{cf}{ack}\end{call}
\end{call} \begin{call}{ct}{write file in tape session}{ct}{}
\begin{call}{ct}{open}{em}{redirect}\end{call} \begin{messcall}{ct}{open}{ef}\end{messcall}
\begin{messcall}{ct}{write}{ef}\end{messcall} \begin{messcall}{ct}{write}{ef}\end{messcall}
\begin{messcall}{ct}{close}{ef}\end{messcall} \begin{call}{ct}{retrieve done (optional)}{em}{ack}
\end{call}
\end{call}
\end{sequencediagram}
} \caption{File read from tape with explicit prepare (synchronous)} \label{fig:explicit-retrieve}
\end{figure}
Failed CTA transfers to and from tape should be reported to the end EOS user by setting appropriately named extended
attributes on the corresponding EOS namespace entries. For example a retrieve operation that fails due to a full
EOS disk server reporting ENOSPC to a tape server could be reported by the tape server setting the value of the
file’s "last\_retrieve\_result" attribute to "Disk space full".
\section{Implicit retrieves (open for read)}
When a user opens a file with copies only on tape, EOS should implicitly generate a retrieve request to CTA.
In turn, CTA should return a time estimate for the file retrieve arrival, so EOS can return a try timing to the
XrootD client, which will retry the open at that time.
As EOS will not keep track of retrieve requests it issued, CTA should have an idempotent retrieve queuing.
\section{User triggered disk copy removal}
CASTOR has learned that it is not easy or even possible to implement the exact \"garbage collection\" policy
required by experiments when it comes to deleting disk copies of files safely stored on tape. CASTOR has provided
the {\tt{}stager\_rm} command to end users to enable them to manually garbage collect files in their CASTOR disk
cache. We currently believe that an equivalent of the {\tt{}stager\_rm} command should be implemented in EOS.
Such a command could simply be a request to execute a {\tt{}"stager\_rm"} workflow action on a specific file..
\section{Garbage collection of disk copies}
A double criteria garbage collection will probably be necessary to keep free space in disk pools (file age
(LRU/FIFO/etc...) + pinning).
\section{Complete deletion of files}
There is no interest in reporting failure to delete the file in CTA while the deletion proceeds in EOS,
so synchronous and asynchronous implementations are equivalent. The complete deletion of files from EOS raises
several race conditions (delete while archiving, delete while retrieving), but all will probably be resolved by
failure of data or metadata operations initiated from CTA to EOS, plus slow reconciliation. The deletion of the
file can be represented by a notification message (as any file operations can).
\section{File (re-)injection}
CTA should be able to inject new files into the EOS tree when: \begin{itemize}
\item Files are imported from CASTOR to EOS-CTA. \item Files are re-injected in the EOS tree (using disaster
recovery data from the CTA file catalogue).
\end{itemize}
In both cases, CTA must support the fact that EOS will treat this file as a new one and issue it a new EOS file id.
The files will remain referenced in CTA as long as they exist on tape (as long as the tape is not rewritten or
removed from the catalogue). Every existing tape file should be referenced (even for deleted files, or failed
writes if we add support for skipped writes in the future).
\section{Slow reconciliation}
The slow reconciliation would scan the entire list of files existing in one EOS instance. CTA could then detect
the files which are missing on its side, and the ones which are not known to EOS anymore. Metadata changes in
EOS will as well be propagated to CTA durig this process. Extra levels of safety could be added (crossing sizes,
checksums, etc...) at the cost of a heavier streaming from EOS. We would then need a retransmit request operation
(could be triggering the proper workflow in EOS), and possibly another operation allowing the confirmation of
non-existence of a file.
The slow reconciliation would be done against the files listed as belonging to a given EOS instance in the CTA catalog.
The listing needs to include the metadata or a mean to detect its changes, and re-create archive requests if needed.
An ideal reconciliation rate would be one week.
\section{Operations summary}
CTA-initiated operations:
\begin{itemize} \item read file to archive \item report partial archival (optional) \item report complete archival
(with metadata for EOS to cross check: "tapereplica") \item write file to retrieve \item set xattr \item get full
file list with metadata
\end{itemize}
EOS-initated operations:
\begin{itemize} \item archive file \item retrieve file \item delete file \end{itemize}
\chapter{Questions and Issues}
\label{questions_and_issues}
The following points are still under discussion:
\begin{itemize} \item Handling of immutability \item "RPC" channel between EOS and CTA and vice-versa not decided
yet (opaque query, open-write-read-close, or SSIv2). \item notification return structure ("return value") should
be defined. It will contain:
\begin{itemize} \item Success \item Action to be taken on success (for example set the “CTA archive ID”
extended attribute of the EOS file being queued for archival) \item Failure code \item Failure message for
logging in EOS (still under discussion) \item Failure message for the end user executing a synchronous workflow
(for example “Cannot open file for writing because there is no route to tape”) \end{itemize}
\item Full life cycle of files in EOS with copies on tape should be determined (they inherit their tape properties
from the directory, but what happens when the file gets moved or the directory properties changed?). \item Finalize
update policy (\ref{updates}). \item Reporting of retrieve status could use the "xattr" message (to be confirmed,
see \ref{dataSerialization}) \item Reporting of failed archival could also use "xattr". \item Reporting of the
last error encountered in CTA could also use the "xattr" message (to be confirmed, see \ref{dataSerialization}).
\item A method to re-creation a file from CTA to EOS should be devised. \item We might want to pass the information
that a file deletion has been confirmed after reconciliation with the user's catalogue. Also delete could be passed
to CTA when the file is moved to the recycle bin in EOS, or when it is definitely deleted from EOS. \item Full
chain reconciliation should be devised. \item Slow reconciliation interface \item Action on storage class change
for a file? (postponed to repack?) \item Catalogue will also keep track of requests for each files (archive and
retrieve) so that queueing can be made idempotent. \item Chaining of archive and retrieve requests to retrieve
requests. Excution of retrieve requests as disk to disk copy if possible. \item Catalogue files could hold the
necessary info to recreate the archive request if needed. \item The list of valid storage classes needs to be
synchronized between EOS and CTA. EOS should not allow a power user to label a directory with an invalid storage
class. CTA should not delete or invalidate a storage class that is being used by EOS.\end{itemize}
\chapter{CTA-EOS Reconciliation Strategy}
\chapter{EOS-CTA Reconciliation Strategy}
\label{reconciliation_strategy}
\section{Reconciling EOS file info and CTA disk file info}
......
......@@ -2,7 +2,7 @@
# texlive 2016
# tikz-uml
TARGET = cta CTA_Admin EOS-CTA_interface queueRequestSeqDiags The_Tape_Storage_Element
TARGET = cta CTA_Admin CTA_EOS queueRequestSeqDiags The_Tape_Storage_Element
PDF_TARGET = $(patsubst %,%.pdf,$(TARGET))
......@@ -59,6 +59,10 @@ CTA_Admin.pdf: $(wildcard CTA_Admin*.tex)
pdflatex CTA_Admin.tex
pdflatex CTA_Admin.tex
CTA_EOS.pdf: $(wildcard CTA_EOS*.tex)
pdflatex CTA_EOS.tex
pdflatex CTA_EOS.tex
# Phony targets
clean:
......
......@@ -9,7 +9,6 @@
% Other packages
\usepackage[amssymb]{SIunits} % Standardised units
\usepackage{soul} % Fancy boxes
\usepackage{multirow} % Multi-row tables
\usepackage[table]{xcolor}
\usepackage{parskip}
......@@ -50,7 +49,6 @@
\input{cta_BasicConcepts.tex}
\input{cta_TapeSessions.tex}
\input{cta_ObjectStore.tex}
\input{cta_CTA-EOS_ReconciliationStrategy.tex}
\input{cta_Authorization.tex}
\input{cta_TODO.tex}
......
......@@ -23,76 +23,3 @@ of several options. Authentication can be performed by any mechanism which is su
example SSS or standard UNIX authentication.
\end{enumerate}
\section{EOS-CTA Authorization Rules}
One of the requirements of CTA is to prevent crosstalk between EOS instances belonging to different \glspl{vo}, e.g.
the ATLAS EOS instance should not be able to access (or even know about) files belonging to CMS.
\begin{alertbox}[Shared Secrets?]
Should we have a different shared secret for each VO, to explicitly prohibit access to files not belonging to that EOS
instance?
\end{alertbox}
\begin{alertbox}[Redundant Rules?]
The highlighted rules below are probably not required.
\end{alertbox}
\begin{enumerate}
\item A listStorageClass command should return the list of storage classes
belonging to the instance from where the command was executed only
\item A queueArchive command should be authorized only if:
\begin{itemize}
\item the instance provided in the command line coincides with the instance from
where the command was executed
\item the storage class provided in the command line belongs to the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original archive requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A queueRetrieve command should be authorized only if:
\begin{itemize}
\item the instance of the requested file coincides with the instance from where
the command was executed
\item\hl{the EOS username and/or group (of the original retrieve requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A deleteArchive command should be authorized only if:
\begin{itemize}
\item the instance of the file to be deleted coincides with the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original delete requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item A cancelRetrieve command should be authorized only if:
\begin{itemize}
\item the instance of the file to be canceled coincides with the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original cancel requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item An updateFileStorageClass command should be authorized only if:
\begin{itemize}
\item the instance of the file to be updated coincides with the instance from
where the command was executed
\item the storage class provided in the command line belongs to the instance from
where the command was executed
\item\hl{the EOS username and/or group (of the original update requester) provided
in the command line belongs to the instance from where the command was
executed}
\end{itemize}
\item An updateFileInfo command should be authorized only if:
\begin{itemize}
\item the instance of the file to be updated coincides with the instance from
where the command was executed
\end{itemize}
\end{enumerate}
......@@ -7,6 +7,8 @@
\usepackage[top=2cm, bottom=2cm, left=2cm, right=2cm]{geometry}
\usepackage{soul} % Fancy boxes
% Fonts
\usepackage{paratype}
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment