slides.

Cosmological simulations
on High Performance Computers
Cosmic Web Morphology and Topology
Cosmological workshop meeting
Warsaw, 12-17 July 2011
Maciej Cytowski
Interdisciplinary Centre for Mathematical and Computational Modelling (ICM),
University of Warsaw
Email: [email protected]
Topics
•
•
•
•
Introduce POWIEW and PRACE projects
Describe the potential collaboration models
Show examples of joint work
Describe how to access Polish and European HPC infrastructure
2
POWIEW project
• HPC Infrastructure for Grand Challenges of Science and
Engineering
• Consortium of 3 biggest Polish HPC centers (Warsaw, Poznań,
Cracow)
• Competence centers and working groups for Grand Challenges:
–
–
–
–
–
–
Climate modelling
Materials sciences
Cosmology
Visual Analysis
Neurobiology
…
• An interesting collaboration model
3
POWIEW HPC Infrastructure
Massively Parallel
Processing (MPP) systems
Fat-node systems
Versatile Symmetric
Multiprocessing systems
Hybrid systems
(GPUs and FPGA)
Selection criteria:
• large scale and complex scientific computations
• support for long computations (stable systems)
• mission critical computations (weather forecasting)
• application driven specialized architectures
4
IBM Blue Gene/P system (notos)
•
IBM Blue Gene/P – 1 rack
•
~14 Tflop/s
•
High performance dedicated interconnect
•
4 TB RAM
•
178 TB storage
•
Installation date:
December 2010
5
Example: REBEL model on Blue Gene/P
Main objectives of joint work:
• measure the performance of GADGET-3 code on Blue Gene/P
architecture
• test different BG/P execution modes and their performance (pure
MPI, MPI + pthreads)
• test scalability of computations on Blue Gene/P architecture
• prepare configuration for the 10243 „Warsaw Universe” simulation
• parallelize the post-processing application
6
Example: REBEL model on Blue Gene/P
•
•
Computations does not scale in VN mode (1024+ nodes)
Best time on 1024 nodes for SMP mode
7
IBM POWER7 system (boreasz)
•
POWER7 IH – 1 rack
•
~78 Tflop/s
•
~10.2 TB RAM
•
153 TB storage
•
Installation: Autumn 2011
•
•
•
1TFlop nodes (4 processors)
Very high memory bandwidth
High performance dedicated interconnect
8
Examples: POWER7 and cosmology
• Blue Waters project – NCSA (University of Illinois)
– ENZO code will be ported and optimized on Blue Waters infrastructure
by the team lead by Brain O’Shea
More: http://www.ncsa.illinois.edu/News/Stories/dark_days/
• Performance measurements and optimization (3x improvement) of
the Gadget code on POWER7 architecture
9
More examples: Dark matter and bright minds
• Collaboration with Computer Science students:
– Summer Schools
– Thesis
– Part-time jobs
•
Tomasz Kłos
– Gadget-2 optimization on RoadRunner architecture
– Thesis: „Cosmological simulations on hybrid architectures”
– Currently: hired by European Space Agency as software developer
•
Jakub Katarzyński
– Parallelization of post-processing application for the Gadget-3 results
– The work will be released and available for public use
10
How to access POWIEW resources and services
• Computational Grants reviewed by the ICM’s scientific board
• Additional info and help: [email protected]
• Special assistance for cosmological type projects:
– Maciej Cytowski
([email protected])
– Wojciech A. Hellwing ([email protected])
11
PRACE – Partnership for Advanced Computing in Europe
• Vision: Enable and support European global leadership in public
and private research and development. High Performance
Computing resources and services for public research.
• European HPC pyramid (ESFRI)
–
–
–
–
Tier-0: 3-6 European Centers for Petaflop
Tier-0: ? European Centers for Exaflop
Tier-1: National Centers
Tier-2: Regional/University Centers
PRACE
POWIEW
12
PRACE Resources
•
1st PRACE System (JUGENE)
–
BG/P by Gauss Center for Supercomputing at Juelich
•
•
•
•
294912 CPU cores, 144 TB memory
1 PFlop/s peak performance
825.5 TFlop/s Linpack
2nd PRACE System (2010/2011) – GENCI (CURIE)
–
Intel based system by Bull
•
•
~90,000 cores
1.6 PFlop/s peak performance
•
3rd PRACE System 2011 HLRS (HERMIT)
•
4th PRACE System 2012 LRZ (SUPER-MUC)
•
5th and 6th Systems 2012/2013 – Cineca and BSC
13
How to access PRACE resources and services
•
Project Access:
–
–
–
–
intended for individual researchers and research groups including multi-national research
groups
has a one year duration
calls for Proposals for Project Access are issued twice yearly (May and November)
3 Regular Calls so far
Example: „Large Scale simulations of Ly-alpha and Ly-break galaxies in the high-z universe: Probing the
epoch of reionization.”
Project leader: Gustavo Yepes, Universidad Autonoma de Madrid, Spain
•
Preparatory Access:
–
–
–
–
–
intended for resource use required to prepare proposals for Project Access
applications for Preparatory Access are accepted at any time
Type A: Code Scalability Testing
Type B: Code development and optimisation by the applicant
Type C: Code development with support from experts
14
Example: ENZO model on JUGENE
•
•
•
•
•
Internal PRACE Call (similar to Preparatory Access)
ICM joined project team as PRACE development staff
ENZO model was build and optimized on JUGENE supercomputer
INITS code used for creating initial conditions for ENZO runs (part of ENZO
distribution) was fully parallelized by ICM
Large simulations: 4096^3 particles
•
Follow-up: proposal for 3rd PRACE Regular Call
• Whitepaper:
„Large Scale Simulations of the Non-Thermal Universe”,
Claudio Gheller (CINECA), Graziella Ferini (CINECA), Maciej Cytowski (ICM), Franco
Vazza (Jacobs University) (available soon at http://www.prace-ri.eu)
15
Summary
• We invite you to collaborate with us:
–
–
–
–
Parallelization and optimization
New algorithm design and implementation
Workflow design and implementation
Assistance in getting access to POWIEW and
PRACE resources
Maciej Cytowski ([email protected])
Wojciech A. Hellwing ([email protected])
16
Links
•
•
•
•
POWIEW project: http://www.wielkiewyzwania.pl
PRACE project: http://www.prace-ri.eu
Blue Waters project: http://www.ncsa.illinois.edu/BlueWaters/
ICM site: http://www.icm.edu.pl
17