KiDS, Legacy Surveys DR8, COSMOS and more, now available in CosmoHub

A few days ago, we celebrated the 3rd anniversary of the rebirth of CosmoHub on its Big Data platform. As you might know, CosmoHub started as a tool for the distribution of PAU catalogs. Because of the limited data volume this task was achieved by an instance of a SQL relational database capable of handling datasets in the low Terabyte range. Since its beginning, CosmoHub has proven to be very useful for other surveys. The immense increase in data, that will soon reach a few 100s of Terabytes, finally led to the migration to Hadoop. The migration entailed a massive boost in performance and many new features of which you can take advantage today. 

In a nutshell, in its most recent incarnation CosmoHub (https://cosmohub.pic.es) is a web platform for the interactive exploration and distribution of massive cosmological datasets. It serves dozens of catalogs to hundreds of users. Further, it has become the reference access point to the PAU catalogs and the synthetic galaxy catalogs derived off the Euclid Flagship dark matter simulations.

We wanted to celebrate this amazing achievement with all the CosmoHub supporters, and what better way of doing this than making available the largest set of public catalogs since our launch.

Since today, there are 9 new public catalogs available:

  • DEEP2 redshift catalog DR4: The survey targeted ~50,000 distant galaxies in the redshift range 0 < z < 1.4, utilizing the DEIMOS spectroscopic on the Keck II telescope.
  • ZEST: Zurich Structure & Morphology Catalog from a COSMOS sample
  • KiDS DR4: provides reprocessed pixel and catalog data products for a roughly 1000 square degrees of the sky.
  • COSMOS2015: contains precise photometric redshifts and stellar masses for more than half a million objects over the 2deg2 COSMOS field.
  • DESI Legacy Survey DR8: 14,000 square degrees of extragalactic sky visible from the northern hemisphere in three optical bands (g,r,z) and four infrared bands.
  • CANDELS Bulge-disk decompositions: A catalog of polychromatic bulge-disk decompositions of ~17.600 galaxies in CANDELS.
  • DES Y1A1 gold catalog: ~137 million objects detected in co-added images covering ~1800 deg 2 in the DES grizY filters.
  • DES Y1A1 morphology catalog: structural and morphological catalogue for 45 million objects selected from the first year data of the Dark Energy Survey (DES).
  • VIPERS (photometry and spectroscopy): second and final data release (PDR-2) of the VIMOS Public Extragalactic Redshift Survey (VIPERS).

As always, we are more than happy to answer all of your questions and requests, just hit us up at cosmohub@pic.es.

The CosmoHub Team

Open Market Consultation event in Barcelona

ARCHIVER – Archiving and Preservation for Research Environments – will introduce significant improvements in the area of archiving and digital preservation services, supporting the IT requirements of European scientists and providing end-to-end archival and preservation services, cost-effective for data generated in the petabyte range with high, sustained ingest rates, in the context of scientific research projects.

The project is managed by a consortium of procurer research organisations (CERN, DESY, EMBL-EBI and PIC) and experts (Addestino and Trust-IT) and receives funding from the European Union’s Horizon 2020 research and innovation programme.

This Open Market Consultation (OMC) event is organised by PIC with the support of  the Catalonia Trade & Investment Agency (ACCIÓ) and Europe Entreprise Network (EEN).

The event is targeted to potential bidders and open to end-users.

During this event, companies will assess the innovation potential to address the Buyers Group use-cases. The event will be moderated by a partner of the ARCHIVER consortium (Addestino) using a planning poker technique in order to estimate effort, value and risk.

The results obtained on the Open Market Consultation will serve as a base of the Tender specifications to be published in October 2019.

Launch of the Helix Nebula Science Cloud Pilot Phase

In November 2016, 4 Consortia won the HNSciCloud Pre-Commercial Procurement (PCP) tender for the establishment of a European hybrid cloud platform to support the deployment of high-performance computing and big-data capabilities for scientific research.

After the submission of their designs, 3 out of the 4 consortia were selected to build the prototypes.

On the 6th of February 2018 the consortia that will move from the Prototype to the Pilot Phase of the HNSciCloud PCP will be announced.

If you are a potential user of the hybrid cloud platform or you are simply interested in understanding more about the pilots that will be deployed, mark the date in your agenda and  follow the live webcast of the event directly from our website www.hnsicloud.eu.

AGENDA

14:00 – Pilot phase Award Ceremony – INFN Introduction and welcome

14:10 – Innovation Procurement and the Open Science Cloud as drivers of Europe’s Digital Single Market – Dr. Dirk Van Rooy (DG CNECT)

14:20 –  Overview of HNSciCloud – CERN

14:35 – Pilot phase Award Ceremony – Contractor 1

14:55 – Pilot phase Award Ceremony – Contractor 2

15:15 – HNSciCloud pilot phase – INFN

15:30 – EOSC in practice – Silvana Muscella (chair EOSC HLEG)

15:45 – Early adopter group and closing of webinar- CERN

 

If you want more information about the event please do not hesitate to contact us at info@hnscicloud.eu

El catàleg de galàxies sintètiques més gran mai generat s’ha produït al PIC

El «Euclid Flagship», un catàleg de més de dos mil milions de galàxies sintètiques ha estat presentat aquest dies a Londres en el marc de la conferència anual del projecte Euclid. Aquest projecte és una missió de l’Agència Espacial Europea (ESA) i de centenars de científics de més d’una quinzena de països, agrupats sota el paraigües del Consorci Euclid. L’objectiu de la missió és posar en òrbita un telescopi per estudiar l’origen de l’expansió accelerada de l’Univers, coneguda pels físics i astrònoms com “energia fosca”, per observar i mesurar la distribució de centenars de milions de galàxies.

Aquest catàleg, fonamental per a permetre l’explotació científica de les dades, ha estat produït íntegrament al Port d’Informació Científica (PIC), en un esforç coordinat amb l’Institut de Ciències de l’Espai (ICE, IEEC-CSIC) i l’«Institute for Computational Science» de la Universitat de Zurich.

Per a la seva generació s’ha utilitzat la plataforma Big Data recentment instal·lada al PIC, basada en un clúster Hadoop i tecnologia Spark, que disposa de més de 500 processadors. El temps necessari per a generar el catàleg ha estat tot un rècord, tan sols 5 hores i escaig, el que ha permès als científics fer múltiples rondes per a refinar-lo fins a assolir uns resultats òptims. El catàleg finalitzat ocupa quasi un miler de gigabytes i es pot trobar a la plataforma CosmoHub. Aquesta plataforma, també allotjada al PIC, permet als seus usuaris d’analitzar i distribuir dades massives amb molta facilitat i rapidesa.

Per veure la presentació que es va fer, podeu fer click aquí.

PIC at the HNSciCloud tender award ceremony

On the 2nd November 2016, the awards ceremony for the successful bidders of the Helix Nebula Science Cloud tender is taking place in Lyon, France, at the CC-IN2P3 (CNRS) premises, one of the main academic data centres in France and a procurer of the Helix Nebula Science Cloud (HNSciCloud).

This €5.3 million joint Pre-Commercial Procurement (PCP) tender, led by CERN, will establish a European hybrid cloud platform that will support high-performance, data-intensive scientific use-cases sponsored by 10 of Europe’s leading public research organisations and co-funded by the European Commission.

A total of 28 multinational companies, SMEs and public research organisations from 12 countries submitted bids during the summer. The 4 selected consortia will be announced at the event and present their visions for the hybrid cloud platform.

This event marks the conclusion of the preparation of the pre-commercial procurement and will kick off the design phase where the four consortia compete to go through to the prototyping phase.

If you want to meet the procurers and the highly innovative consortia in Lyon, please register now. For those who cannot join us in Lyon, a webcast of the event will be available at www.hnscicloud.eu.

Join us on Wednesday 2nd November, 2016, at 2.30pm CET

www.hnscicloud.eu

#HNtenderawards

View the event agenda and details here.

For more information on HNSciCloud follow us @HelixNebulaSC

El PIC amplia a 20 Gbps la seva connexió a l’anella científica

El Port d’Informació Científica (PIC) ha ampliat la connexió dedicada de què disposa a l’Anella Científica de 10 a 20 Gbps i passa a ser la primera institució que s’hi connecta a aquesta velocitat.

Durant el 2016, el PIC ha rebut i enviat més de 20 PB de dades provinents d’experiments científics d’abast internacional a través de l’Anella Científica i RedIRIS. Els fitxers amb què treballa contenen informació sobre les col·lisions de protons dins el Large Hadron Collider (LHC); observacions del telescopi MAGIC de raigs gamma emesos a l’univers; fotografies de milions de galàxies per a l’estudi de l’energia fosca de l’espai, que realitza el projecte PAU, i simulacions de l’efecte de l’energia fosca sobre l’expansió de l’univers per als projectes DES i EUCLID. Aquestes dades són emmagatzemades i analitzades per investigadors de diferents institucions a través de diferents plataformes de processament que té el PIC.

Per a Manuel Delfino, director del PIC, “amb aquest augment de la capacitat de connexió a l’Anella Científica podrem fer front a l’increment previst en la generació de dades de l’LHC, que arribarà a ser de l’ordre de 100 PB anuals. A més, ens permetrà desenvolupar nous serveis necessaris per gestionar dades de nous projectes, com ara EUCLID i la Xarxa de Telescopis Cherenkov (CTA)”.

El PIC és fruit d’una col·laboració entre l’Institut de Física d’Altes Energies (IFAE, consorci entre la Generalitat de Catalunya i la Universitat Autònoma de Barcelona) i el Centro de Investigaciones Energéticas, Mediambiantales y Tecnológicas (CIEMAT, organisme públic d’investigació del MINECO).

€5.3 million tender for Hybrid Cloud Platform serving science

Today Helix Nebula – The Science Cloud (HNSciCloud) launched a €5.3 million tender for the establishment of a European hybrid cloud platform. The purpose of the platform is to support the deployment of high-performance computing and big-data capabilities for scientific research.

PIC is part or the project.

Click here to read the press release

Click here to read the article in Datacenter Dynamics (spanish)