Commit d4f79281 authored by Michael Davis's avatar Michael Davis
Browse files

[doc] Adds ITUM and GWDG presentations frum June/July 2020

parent 2c139644
% Possible values for aspect ratio are: 169, 1610, 149, 54, 43 and 32.
\documentclass[aspectratio=1610]{beamer}
\usetheme{cern}
% Change bullet points
\usepackage{enumitem}
\setbeamertemplate{itemize items}[square]
\setitemize{label=\raisebox{0.3ex}{\usebeamerfont*{itemize item}
\usebeamercolor[fg]{itemize item}
\usebeamertemplate{itemize item}}}
%\setlist[itemize]{leftmargin=*, itemsep=0pt, parsep=2pt, labelsep*=2em, labelindent=0em, before=\vspace{-\dimexpr\baselineskip +2.6\partopsep}}
% CERN blue is Pantone 286 = RGB 56 97 170, defined as cern@blue below
\definecolor{cern@ltblue}{rgb}{0.415686,0.611765,0.964706} % RGB 106 156 246
\definecolor{cern@blue} {rgb}{0.219608,0.380392,0.666667} % RGB 56 97 170
\definecolor{cern@dkblue}{rgb}{0.082353,0.184314,0.364706} % RGB 21 47 93
% Complimentary colours
\definecolor{cern@ltcomp}{rgb}{0.666667,0.525490,0.219608} % RGB 170 134 56
\definecolor{cern@dkcomp}{rgb}{0.364706,0.266667,0.047059} % RGB 93 68 12
%\setbeamercolor{title} {bg=cern@blue,fg=white}
%\setbeamercolor{frametitle} {bg=cern@blue,fg=white}
%\setbeamercolor{section in head/foot}{bg=cern@ltblue,fg=white}
\setbeamercolor{itemize item} {fg=cern@dkcomp}
\setbeamerfont{itemize item} {size=\large}
%%
%% Get packages that we need
%%
% Adjust page margins on-the-fly
\usepackage{changepage}
% Change fonts to Avenir
% Note: need to compile with xelatex to use this package
\usepackage{fontspec}
\setsansfont{AvenirLTStd-Book}
\usepackage[showboxes,overlay,absolute]{textpos}
\usepackage[overlay,absolute]{textpos}
\setlength{\TPHorizModule}{.01\paperwidth} % Horizontal units are percent of paper width
\setlength{\TPVertModule}{.01\paperwidth} % Make vertical units equal to horizontal units
\textblockorigin{.5\paperwidth}{4.25ex}
% Headings within slides
\usepackage{xcolor}
\usepackage{tabularx}
%%
%% Title page
%%
\author{\Large Dr. Michael Davis for the CTA Team}
\title[CTA in Production (ATLAS)]{CERN Tape Archive (CTA) in Production :\\
ATLAS migration report and next steps\\}
\date{28 June 2020}
% The ATLAS experiment is migrating from CASTOR to CTA during the week of 22 June. This marks the end
% of the CASTOR ATLAS service and the first production instance of CTA. The CTA Team will give a report
% on the migration of ATLAS and a status update on the migration of the other experiments.
\begin{document}
\frame{\titlepage}
\begin{frame}{EOS+CTA in Production ``by end 1Q 2020''}
\centering
\includegraphics[width=\textwidth]{images/CASTOR_EOS+CTA_Logo}
\end{frame}
\begin{frame}{ATLAS Migration Plan}
\begin{columns}
\begin{column}{0.4\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/CTA_Deployment_Atlas}
\end{center}
\end{column}
\begin{column}{0.55\textwidth}
\begin{itemize}
\item Commissioning tests
\item Disable CASTOR ATLAS
\item Migrate metadata from CASTOR to CTA
\item Put EOS+CTA ATLAS into production
\end{itemize}
\end{column}
\end{columns}
\end{frame}
\begin{frame}{ATLAS Reprocessing Campaign}{}
% On Tue 21st Atlas started a major reprocessing campaign, scheduled to recall 4PB of data from a
% production instance of CTA. The service has performed well so far, reaching sustained throughput of
% 5GB/s, limited in this case by the network interfaces of the CTA nodes. More nodes will be added in
% order to demonstrate the horizontal scaling of the service.
\centering
\vspace{1ex}
{\LARGE\color{cern@blue}January: Recall of Data18\\
February: Recall of Data17}
\begin{adjustwidth}{-1cm}{-1cm}
\includegraphics[width=\linewidth]{images/RecallTest}
\end{adjustwidth}
\end{frame}
\begin{frame}{ATLAS Data Carousel/Rucio WS Report}{}
\begin{itemize}
\item 5 PB recalled @ 4--5 GB/s %(Limiting factor was hardware and network bandwidth)
%\item Software scaled smoothly as more hardware was added
%\item Very efficient tape recall (80\% tape drive efficiency for Enterprise drives)
\item CTA was the fastest site
\item CTA had the lowest site error rate (< 350 errors)
\item \textbf{CTA demonstrated to be performant and reliable for large volume transfers}
\item No need for further recall tests: CTA did not participate in data16 and data15 recalls
\end{itemize}
\end{frame}
\begin{frame}{March/April : SFO Integration Tests}{}
\begin{center}
\includegraphics[width=0.5\linewidth]{images/SFO_test_setup}
\end{center}
{\normalsize
\begin{itemize}
\item ``File safely on tape'' check caused a high rate of metadata queries to EOS
\item Problem will be fixed using FTS
\item In the interim, ATLAS implemented a local metadata cache to reduce the number of
queries to an acceptable rate
\end{itemize}}
\end{frame}
\begin{frame}{June : ATLAS Migration}
\normalsize
\textbf{28 May}
\begin{itemize}
\item Re-test SFO integration
\end{itemize}
\textbf{15 June}
\begin{itemize}
\item Disable writes to CASTOR ATLAS
\item Consistency check between Rucio catalogue and CTA catalogue
\end{itemize}
\textbf{23 June}
\begin{itemize}
\item Permanently disable all access to CASTOR ATLAS
\item Migrate metadata for 85 million files
\end{itemize}
\textbf{25 June}
\begin{itemize}
\item Migration complete
\item Consistency check between CASTOR export and CTA import
\end{itemize}
\textbf{29 June}
\begin{itemize}
\item \textbf{EOS+CTA ATLAS in operation}
\end{itemize}
\end{frame}
\begin{frame}{23--25 June : ATLAS Migration}
\begin{columns}
\begin{column}{0.45\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/Migration_CASTOR}
CASTOR
\end{center}
\end{column}
\begin{column}{0.45\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/Migration_CTA}
CTA
\end{center}
\end{column}
\end{columns}
\end{frame}
\begin{frame}{What's next for CTA? ALICE}
\begin{columns}
\begin{column}{0.4\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/CTA_Deployment_ALICE}
\end{center}
\end{column}
\begin{column}{0.55\textwidth}
\begin{itemize}
\item JAlien + EOS + CTA
\item SSD tape buffer + 5 PB HDD cache for retrieves
\item ALICE recall tests underway
\item ALICE write tests in July
\item Put into production after the summer
\end{itemize}
\end{column}
\end{columns}
\end{frame}
\begin{frame}{What's next for CTA? CMS and LHCb}
\begin{itemize}
\item Integration tests have started
\end{itemize}
\begin{columns}
\begin{column}{0.4\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/CTA_Deployment_CMS.pdf}
\end{center}
\end{column}
\begin{column}{0.4\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/CTA_Deployment_LHCb.pdf}
\end{center}
\end{column}
\end{columns}
\end{frame}
\begin{frame}{What's next for CTA?\\
Non-LHC Experiments}
\begin{columns}
\begin{column}{0.4\textwidth}
\begin{center}
\includegraphics[width=\textwidth]{images/CTA_Deployment_small}
\end{center}
\end{column}
\begin{column}{0.55\textwidth}
\begin{itemize}
\item AMS, Compass, Dune, NA61, NA62, nTOF, \ldots
\item Fellow starting in August: start integration testing with NA62
\end{itemize}
\end{column}
\end{columns}
\end{frame}
\begin{frame}{EOS+CTA Status : Summary}{}
\begin{center}
\includegraphics[width=0.5\textwidth]{../Logo/Logo_EOS+CTA}
\end{center}
\begin{itemize}
\item EOS+CTA ATLAS is in production
\item CASTOR ATLAS has been retired
\item ALICE to be migrated in coming months
\item Integration and commissioning tests for CMS, LHCb and non-LHC experiments
(schedule to be worked out with each experiment)
\end{itemize}
\end{frame}
\backcover
\end{document}
CERN Tape Archive Goes Into Operation
The output, or product, of CERN's experimental programme is its data: vast amounts of data are produced by the detectors
in the LHC experiments and elsewhere in CERN's experimental programme. This data is stored by the IT Storage Group
for reconstruction and analysis by physicists across the WLCG.
The custodial copy of all of CERN's physics data is stored on magnetic tape at Tier-0. Currently this amounts to around
340 PB. During LHC Run-1 and Run-2, the software system used to manage the tape storage was CASTOR, which was conceived
as a system to manage both disk and tape storage. Over the last ten years, needs evolved and a new disk system was
developed for disk storage and data analysis—EOS. This created a need for a new tape back-end to EOS. This project, the
CERN Tape Archive (CTA), was conceived as an evolution of the CASTOR tape system while removing the need to maintain a
second disk management system.
CTA has been in development for several years. In January, the CTA team started commissioning tests with the ATLAS
experiment. During the ATLAS Run-2 recall campaign in January and February, CTA was used side-by-side with the Tier-1
Storage Elements to provide ATLAS data for reprocessing. CTA had the lowest error rate of all sites and was demonstrated
to be performant and reliable for large volume transfers.
The disruption of Covid-19 and the move to teleworking caused a slight delay in the plan to bring CTA into production, as
both the CTA team and the ATLAS data management team adjusted to new ways of communicating and scheduling tests. There
was a determined spirit of cooperation which throughout the integration work and final commissioning tests, finally
allowing the migration of ATLAS's data to take place during the last two weeks in June. This involved moving the metadata
for 86 million files—the entirety of ATLAS's physics output—from CASTOR to CTA. (The physical data stored on tape is not
moved as this amounts to more than 80 PB, which would take many months to copy).
After the ATLAS data was migrated, CTA entered operation on 29 June 2020.
The other LHC experiments will migrate to CTA during the coming months, well in time for Run-3 data-taking. In parallel,
the CTA team will start to migrate the Small and Medium-Sized Experiments (AMS, Compass, Dune, NA61, NA62, nTOF, etc.)
CASTOR will be gradually decomissioned as each experiment is migrated, so that the CASTOR disk servers can be returned
to the experimental pledges.
Link to ITUM presentation: https://indico.cern.ch/event/914439/
TARGET=ATLAS_Update.pdf
IMAGES=$(wildcard images/*)
all: $(TARGET)
show: $(TARGET)
evince $(TARGET)
# Need to compile with xelatex to get Avenir font
%.pdf: %.tex $(IMAGES) Makefile
xelatex $<
xelatex $<
clean:
rm -f $(TARGET) *.aux *.bbl *.blg *.log *.dvi *.bak *.lof *.log *.lol *.lot *.out *.toc *.eps *.nav *.snm *.vrb
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment