Bayesian hierarchical models for quantitative estimates for performance metrics applied to saddle search algorithms

JSON Export

{
  "id": "2708", 
  "created": "2025-05-30T06:14:51.635207+00:00", 
  "updated": "2025-06-01T20:11:01.482948+00:00", 
  "revision": 6, 
  "metadata": {
    "_files": [
      {
        "key": "hpc.tar.xz", 
        "size": 201592576, 
        "description": "Output logs for EON Dimer across CG-LBFGS rotations and with/without external rotation removal", 
        "checksum": "md5:5cd1034a2d822b45a4e918445b9c2e86"
      }, 
      {
        "key": "models_and_preds.tar.xz", 
        "size": 1617416744, 
        "description": "Exported BRMS models and predictions, both as R objects and parquet files with stan code.", 
        "checksum": "md5:a18d30d132e6dcfcd56b57823371a5c1"
      }, 
      {
        "key": "readme.txt", 
        "size": 4640, 
        "description": "README containing data structure and usage instructions for the R objects and FAIR alternative.", 
        "checksum": "md5:d5e4323e70b1aec0dd4fa2901c99a8b5"
      }
    ], 
    "license_addendum": null, 
    "edited_by": 576, 
    "publication_date": "Jun 01, 2025, 22:11:01", 
    "contributors": [
      {
        "email": "rgoswami@ieee.org", 
        "affiliations": [
          "Science Institute and Faculty of Physical Sciences, University of Iceland, Reykjav\u00edk, Iceland", 
          "Department of Mechanical and Materials Engineering, Queen\u2019s University, Kingston, Ontario, Canada, K7L 3N6"
        ], 
        "givennames": "Rohit", 
        "familyname": "Goswami"
      }
    ], 
    "description": "The increasing use of high-throughput computational chemistry demands rigorous methods for evaluating algorithm performance. We present a Bayesian hierarchical modeling paradigm (brms/Stan) for analyzing key performance metrics: function evaluations, computation time, and success/failure. This framework accounts for variability across different systems and functionals, providing reliable uncertainty estimates beyond subjective visual assessments or frequentist limitations. We applied this to compare conjugate gradient (CG) and L-BFGS algorithms for the Dimer method's rotation phase (in EON, with/without removal of external rotations) on a benchmark of 500 initial saddle search approximations, analyzing over 2000 runs. Our results show CG rotations generally outperform L-BFGS, exhibiting a statistically credible, small reduction in PES calls and significantly higher odds of successful convergence. Conversely, enabling rotation removal incurred a substantial PES call penalty without a corresponding credible improvement in success odds in the implementation studied. These findings, from our novel Bayesian hierarchical modeling application, suggest CG may be preferable for Dimer rotational optimization in similar contexts. This robust statistical framework highlights benefits for revisiting optimization strategies, quantifying uncertainty, and facilitating improved high-throughput computational chemistry methods. This record contains the saddle search output logs for EON with NWChem across four settings, with/without external rotation and the use of CG/LBFGS for the rotational phase of the dimer. The record also includes fitted Bayesian Hierarchical models for performance and success analysis. These models and data are used to generate the figures and validate the analysis in the manuscript. For details, refer to the code in the associated GitHub repository.", 
    "id": "2708", 
    "owner": 153, 
    "references": [
      {
        "comment": "Preprint describing the model and analysis of the data in the record.", 
        "url": "https://arxiv.org/abs/2505.13621", 
        "type": "Preprint", 
        "citation": "R. Goswami, \u201cBayesian hierarchical models for quantitative estimates for performance metrics applied to saddle search algorithms,\u201d May 19, 2025, arXiv: arXiv:2505.13621.", 
        "doi": "10.48550/arXiv.2505.13621"
      }, 
      {
        "comment": "Software collection used to generate the models and data in this record.", 
        "url": "https://github.com/HaoZeke/brms_idrot_repro", 
        "type": "Software", 
        "citation": "R. Goswami, Github."
      }
    ], 
    "version": 1, 
    "license": "Materials Cloud non-exclusive license to distribute v1.0", 
    "status": "published", 
    "title": "Bayesian hierarchical models for quantitative estimates for performance metrics applied to saddle search algorithms", 
    "keywords": [
      "saddle-search", 
      "performance-modeling", 
      "transition-state", 
      "success-modeling"
    ], 
    "doi": "10.24435/materialscloud:xc-5e", 
    "mcid": "2025.91", 
    "is_last": true, 
    "_oai": {
      "id": "oai:materialscloud.org:2708"
    }, 
    "conceptrecid": "2707"
  }
}