slides.
Transkrypt
slides.
Cosmological simulations on High Performance Computers Cosmic Web Morphology and Topology Cosmological workshop meeting Warsaw, 12-17 July 2011 Maciej Cytowski Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw Email: [email protected] Topics • • • • Introduce POWIEW and PRACE projects Describe the potential collaboration models Show examples of joint work Describe how to access Polish and European HPC infrastructure 2 POWIEW project • HPC Infrastructure for Grand Challenges of Science and Engineering • Consortium of 3 biggest Polish HPC centers (Warsaw, Poznań, Cracow) • Competence centers and working groups for Grand Challenges: – – – – – – Climate modelling Materials sciences Cosmology Visual Analysis Neurobiology … • An interesting collaboration model 3 POWIEW HPC Infrastructure Massively Parallel Processing (MPP) systems Fat-node systems Versatile Symmetric Multiprocessing systems Hybrid systems (GPUs and FPGA) Selection criteria: • large scale and complex scientific computations • support for long computations (stable systems) • mission critical computations (weather forecasting) • application driven specialized architectures 4 IBM Blue Gene/P system (notos) • IBM Blue Gene/P – 1 rack • ~14 Tflop/s • High performance dedicated interconnect • 4 TB RAM • 178 TB storage • Installation date: December 2010 5 Example: REBEL model on Blue Gene/P Main objectives of joint work: • measure the performance of GADGET-3 code on Blue Gene/P architecture • test different BG/P execution modes and their performance (pure MPI, MPI + pthreads) • test scalability of computations on Blue Gene/P architecture • prepare configuration for the 10243 „Warsaw Universe” simulation • parallelize the post-processing application 6 Example: REBEL model on Blue Gene/P • • Computations does not scale in VN mode (1024+ nodes) Best time on 1024 nodes for SMP mode 7 IBM POWER7 system (boreasz) • POWER7 IH – 1 rack • ~78 Tflop/s • ~10.2 TB RAM • 153 TB storage • Installation: Autumn 2011 • • • 1TFlop nodes (4 processors) Very high memory bandwidth High performance dedicated interconnect 8 Examples: POWER7 and cosmology • Blue Waters project – NCSA (University of Illinois) – ENZO code will be ported and optimized on Blue Waters infrastructure by the team lead by Brain O’Shea More: http://www.ncsa.illinois.edu/News/Stories/dark_days/ • Performance measurements and optimization (3x improvement) of the Gadget code on POWER7 architecture 9 More examples: Dark matter and bright minds • Collaboration with Computer Science students: – Summer Schools – Thesis – Part-time jobs • Tomasz Kłos – Gadget-2 optimization on RoadRunner architecture – Thesis: „Cosmological simulations on hybrid architectures” – Currently: hired by European Space Agency as software developer • Jakub Katarzyński – Parallelization of post-processing application for the Gadget-3 results – The work will be released and available for public use 10 How to access POWIEW resources and services • Computational Grants reviewed by the ICM’s scientific board • Additional info and help: [email protected] • Special assistance for cosmological type projects: – Maciej Cytowski ([email protected]) – Wojciech A. Hellwing ([email protected]) 11 PRACE – Partnership for Advanced Computing in Europe • Vision: Enable and support European global leadership in public and private research and development. High Performance Computing resources and services for public research. • European HPC pyramid (ESFRI) – – – – Tier-0: 3-6 European Centers for Petaflop Tier-0: ? European Centers for Exaflop Tier-1: National Centers Tier-2: Regional/University Centers PRACE POWIEW 12 PRACE Resources • 1st PRACE System (JUGENE) – BG/P by Gauss Center for Supercomputing at Juelich • • • • 294912 CPU cores, 144 TB memory 1 PFlop/s peak performance 825.5 TFlop/s Linpack 2nd PRACE System (2010/2011) – GENCI (CURIE) – Intel based system by Bull • • ~90,000 cores 1.6 PFlop/s peak performance • 3rd PRACE System 2011 HLRS (HERMIT) • 4th PRACE System 2012 LRZ (SUPER-MUC) • 5th and 6th Systems 2012/2013 – Cineca and BSC 13 How to access PRACE resources and services • Project Access: – – – – intended for individual researchers and research groups including multi-national research groups has a one year duration calls for Proposals for Project Access are issued twice yearly (May and November) 3 Regular Calls so far Example: „Large Scale simulations of Ly-alpha and Ly-break galaxies in the high-z universe: Probing the epoch of reionization.” Project leader: Gustavo Yepes, Universidad Autonoma de Madrid, Spain • Preparatory Access: – – – – – intended for resource use required to prepare proposals for Project Access applications for Preparatory Access are accepted at any time Type A: Code Scalability Testing Type B: Code development and optimisation by the applicant Type C: Code development with support from experts 14 Example: ENZO model on JUGENE • • • • • Internal PRACE Call (similar to Preparatory Access) ICM joined project team as PRACE development staff ENZO model was build and optimized on JUGENE supercomputer INITS code used for creating initial conditions for ENZO runs (part of ENZO distribution) was fully parallelized by ICM Large simulations: 4096^3 particles • Follow-up: proposal for 3rd PRACE Regular Call • Whitepaper: „Large Scale Simulations of the Non-Thermal Universe”, Claudio Gheller (CINECA), Graziella Ferini (CINECA), Maciej Cytowski (ICM), Franco Vazza (Jacobs University) (available soon at http://www.prace-ri.eu) 15 Summary • We invite you to collaborate with us: – – – – Parallelization and optimization New algorithm design and implementation Workflow design and implementation Assistance in getting access to POWIEW and PRACE resources Maciej Cytowski ([email protected]) Wojciech A. Hellwing ([email protected]) 16 Links • • • • POWIEW project: http://www.wielkiewyzwania.pl PRACE project: http://www.prace-ri.eu Blue Waters project: http://www.ncsa.illinois.edu/BlueWaters/ ICM site: http://www.icm.edu.pl 17