Services
Computing, data processing and analysis
Batch processing
PIC supports batch execution of processing jobs in a computing cluster through the HTCondor software framework. Users can access the local PIC cluster which currently has 8.500 CPU cores and 18 GPUs. Orchestration of job dependencies (as in a DAG) can also be set. HTCondor also allows users to manage distributed workloads spanning resources beyond the cluster at PIC, such as Cloud resources.
Python notebooks – jupyter.pic.es
PIC offers JupyterHub as a service, a web application to provide multi-user access for creating Jupyter notebooks to develop and run code for data analysis. These notebooks run on the HTCondor cluster nodes, using the same environment as the batch jobs.
Mass storage
PIC provides a flexible, efficient and performance-oriented mass storage system able to hold tens of petabytes of scientific data. It combines tape and disk storage in a tiered environment delivering the required combination of performance, capacity and resilience.
The tape storage is handled by Enstore, a software developed by Fermilab to manage the hundreds of Petabytes of scientific data of the laboratory.
Users access the data via dCache, a disk caching software layer jointly developed by DESY, Fermilab and NDGF that integrates with Enstore through a common namespace.
Files can be accessed with several different protocols. Local users can access data through dcacp (a posix-like interface) or NFSv4.1. Data can also be remotely accessed via WAN-specific protocols such as GridFTP and WebDAV.
Big Data – Hadoop Cluster
Spark processing
Data processing framework to perform analysis tasks on very large datasets.
CosmoHub – cosmohub.pic.es
Web portal to perform interactive exploration and distribution of massive Cosmology datasets. It provides access to over a dozen public catalogues. With its intuitive user interface, users with no SQL knowledge can visualize and download customized subsets of the data.
GitLab – gitlab.pic.es
Users can host git code repositories at PIC and access them through the GitLab web tool, providing added value services such as issue tracking or support for CI/CD pipelines.
Consulting support
Consultants provide expert advice for workflow deployment, analysis code optimization and parallelization in high-throughput or distributed environments.
Issues where PIC can provide expert support are:
- I/O optimization
- Compilation issues on PIC platform
- Advising on best strategies for porting code to a high-throughput environment