Authors : Son Dang; Youngje Son; Brandon Kim

Volume/Issue : Volume 10 - 2025, Issue 4 - April

Google Scholar : https://tinyurl.com/yc72mbms

Scribd : https://tinyurl.com/wejr9u7k

DOI : https://doi.org/10.38124/ijisrt/25apr1850

Google Scholar

Note : A published paper may take 4-5 working days from the publication date to appear in PlumX Metrics, Semantic Scholar, and ResearchGate.

Note : Google Scholar may take 15 to 20 days to display the article.

Abstract : The increasing intricacy of scientific simulations and industrial activity processes along with their accompanying datasets demand ecosystems outside the capabilities of traditional High-Performance Computing (HPC) systems [1]–[6], [11]. The requirements of contemporary research and industries based on data are multidisciplinary, as computations using traditional HPC await a solution. Recently, attention has been drawn towards harnessing the computational power of AI facilities to relieve HPC systems as the fusion of intelligence reveals new adaptive workflows that integrate HPC and AI capabilities. This evolution in thinking gives rise not only to an emergent paradigm for an AI-powered HPC but also to the transforma- tional approach of the HPC-AI Workflow Platform that incor- porates synergistic and intelligent orchestration workflows. In this paper, we introduce the HPC-AI Workflow Platform, which catalyzes collaborative innovation with its innovative Workflow- as-a-Service (WaaS) model, facilitating effortless cross-domain sharing and reuse of workflows, boosting operational efficiency. They are further enabled with AI for real-time decision-making and optimization, smart resource allocation, big data analytics, and seamless data flow for the timely and energy-efficient execution of complex simulations, enhancing HPC productivity. This not only demonstrates the efficacy of the HPC-AI Workflow Platform in resourceful workflow optimization and management but also strengthens its position as a future- ready paradigm to advance HPC application relevance in science and industry.

Keywords : High Performance Computing, Intelligent Workflows, Machine Learning, Scientific Data Analysis, Big Data.

References :

1. Jorge Ejarque, Rosa M. Badia, Lo¨ıc Albertin, Giovanni Aloisio, En- rico Baglione, Yolanda Becerra, Stefan Boschert, Julian R. Berlin, Alessandro D’Anca, Donatello Elia, Franc¸ois Exertier, Sandro Fiore, Jose´ Flich, Arnau Folch, Steven J. Gibbons, Nikolay Koldunov, Francesc Lordan, Stefano Lorito, Finn Løvholt, Jorge Mac´ıas, Fabrizio Marozzo, Alberto Michelini, Marisol Monterrubio-Velasco, Marta Pienkowska, Josep de la Puente, Anna Queralt, Enrique S. Quintana-Ort´ı, Juan E. Rodr´ıguez, Fabrizio Romano, Riccardo Rossi, Jedrzej Rybicki, Miroslaw Kupczyk, Jacopo Selva, Domenico Talia, Roberto Tonini, Paolo Trunfio, and Manuela Volpe, “Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence,” Future Generation Computer Systems, vol. 134, pp. 414–429, 2022.
2. Shantenu Jha, Vincent R. Pascuzzi, and Matteo Turilli, “AI-coupled HPC Workflows,” 2022, arXiv:2208.11745 [cs.DC].
3. Fabio Le Piane, Mario Vozza, Matteo Baldoni, and Francesco Mercuri, “Integrating high-performance computing, machine learning, data man- agement workflows, and infrastructures for multiscale simulations and nanomaterials technologies,” Beilstein Journal of Nanotechnology, vol. 15, pp. 1498–1521, 2024.
4. Rafael Ferreira da Silva, Rosa M. Badia, Deborah Bard, Ian T. Foster, Shantenu Jha, and Frederic Suter, “Frontiers in Scientific Workflows: Pervasive Integration With High-Performance Computing,” Computer, vol. 57, no. 8, pp. 36–44, 2024.
5. AI4HPC Consortium, “AI4HPC: Enabling AI-Driven High-Performance Computing,” AI4HPC Project, Technical Report, 2022.
6. Sam Partee, Joseph Ellis, Duncan Ritchie, Rob Sterrett, Sriram Venkatesh, Mike Lesniak, and Anshu Dubey, “SmartSim: Online An- alytics and AI for High-Performance Computing,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’21), pp. 1–14, 2021.
7. Message Passing Interface Forum, “MPI: A Message-Passing Interface Standard, Version 4.0,” University of Tennessee, Knoxville, TN, 2021.
8. Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor, “Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems,” Scientific Programming, vol. 19, no. 2–3,

pp. 219–237, 2011.

9. Ilkay Altintas, Chad Berkley, Efrat Jaeger, Matthew Jones, Bertram Lu- dascher, and Steve Mock, “Kepler: An Extensible System for Scientific Workflows,” Scientific Programming, vol. 14, no. 3–4, pp. 191–208,
10. 2006.
11. Tom Oinn, Matthew Addis, Justin Ferris, Darren Marvin, Martin Senger, Mark Greenwood, Tim Carver, Kevin Glover, Matthew R. Pocock, Anil Wipat, and Peter Li, “Taverna: A Tool for the Composition and Enactment of Bioinformatics Workflows,” Bioinformatics, vol. 22, no. 22, pp. 3045–3054, 2006.
12. Ian Foster, Carl Kesselman, and Steven Tuecke, “The Anatomy of the Grid: Enabling Scalable Virtual Organizations,” International Journal of High Performance Computing Applications, vol. 15, no. 3, pp. 200–222, 2001.
13. Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields, “Scientific Workflows for Grids,” Springer, 2014.
14. Thomas Sterling, Matthew Anderson, and Maciej Brodowicz, “High Performance Computing: Modern Systems and Practices,” Morgan Kauf- mann, pp. 1–632, 2018.
15. Shantenu Jha, Daniel S. Katz, Andre Luckow, and Andre Merzky, “AI- Coupled HPC Workflows,” Future Generation Computer Systems, vol. 118, pp. 245–259, 2021.
16. Rick Stevens, Valerie Taylor, and Jeffrey Nichols, “AI for HPC: Expe- riences and Opportunities,” Computing in Science & Engineering, vol. 24, no. 1, pp. 10–19, 2022.
17. Sam Partee, Jonathan Ellis, Kevin Moreau, and Sivasankaran Rajaman- ickam, “SmartSim: Online Simulation with Machine Learning,” Journal of Open Source Software, vol. 6, no. 67, pp. 3542, 2021.
18. Prasanna Balaprakash, Michael Salim, Thomas D. Uram, Venkat Vish- wanath, and Stefan M. Wild, “DeepHyper: Asynchronous Hyperparam- eter Search for Deep Neural Networks,” in Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, pp. 42–53, 2019.
19. Justin M. Wozniak, Timothy G. Armstrong, Michael Wilde, Daniel
20. S. Katz, Ewing Lusk, and Ian Foster, “Swift/T: Scalable Dataflow Programming for Distributed Memory HPC,” Parallel Computing, vol. 65, pp. 1–14, 2017.
21. Anubhav Jain, Shyue Ping Ong, Wei Chen, Bharat Medasani, Xi- aohui Qu, Michael Kocher, Miriam Brafman, Guido Petretto, Gian- Marco Rignanese, Geoffroy Hautier, Daniel Gunter, and Kristin A. Persson, “FireWorks: A Dynamic Workflow System Designed for High- Throughput Applications,” Concurrency and Computation: Practice and Experience, vol. 27, no. 17, pp. 5037–5059, 2015.
22. Andy B. Yoo, Morris A. Jette, and Mark Grondona, “SLURM: Simple Linux Utility for Resource Management,” Lecture Notes in Computer Science, vol. 2862, pp. 44–60, 2003.
23. Maxime Beauchemin and others, “Apache Airflow: A Platform to Programmatically Author, Schedule and Monitor Workflows,” Apache Software Foundation, 2016.
24. Yan Liu, Wei Zhang, and Jun Zhou, “IBM Spectrum LSF: A Resource Management Framework for High-Performance Computing,” IBM Jour- nal of Research and Development, vol. 54, no. 3, pp. 1–10, 2010.
25. Jay Kreps, Neha Narkhede, and Jun Rao, “Kafka: A Distributed Mes- saging System for Log Processing,” in Proceedings of the NetDB’11 Workshop, 2011.
26. Mart´ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, and others, “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” arXiv:1603.04467, 2015.
27. Dirk Merkel, “Docker: Lightweight Linux Containers for Consistent Development and Deployment,” Linux Journal, vol. 2014, no. 239, 2014.
28. Garrick Staples, “TORQUE Resource Manager,” in Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, 2006.
29. William Gropp, Ewing Lusk, Nathan Doss, and Anthony Skjellum, “A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard,” Parallel Computing, vol. 22, no. 6, pp. 789– 828, 1996.
30. Yadu Babuji, Ian Foster, Michael Wilde, Kyle Chard, and Daniel S. Katz, “Parsl: Pervasive Parallel Programming in Python,” in Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, pp. 25–36, 2019.
31. Paolo Di Tommaso, Maria Chatzou, Evan W. Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame, “Nextflow Enables Reproducible Computational Workflows,” Nature Biotechnology, vol. 35, no. 4, pp. 316–319, 2017.
32. Chun Siong Liew, Malcolm P. Atkinson, Michelle Galea, Tan Fong Ang, Paul Martin, and Jano I. van Hemert, “Scientific Workflows: Moving Across Paradigms,” ACM Computing Surveys, vol. 49, no. 4, pp. 1–39, 2016.
33. Bertram Ludascher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A. Lee, Jing Tao, and Yang Zhao, “Scientific Workflow Management and the Kepler System,” Concurrency and Computation: Practice and Experience, vol. 21, no. 8, pp. 1039– 1065, 2009.
34. Adam Barker and Jano van Hemert, “Scientific Workflow: A Survey and Research Directions,” in Parallel Processing and Applied Mathematics,
35. pp. 746–753, 2007.
36. Jack Dongarra, Hans Meuer, Erich Strohmaier, and Horst Simon, “TOP500 Supercomputer Sites: 2020 Edition,” Prometeus GmbH, Ger- many, 2020.