Neural predictive monitoring and a comparison of frequentist and Bayesian approaches
| Autor: | Nicola Paoletti, Francesca Cairoli, Scott D. Stoller, Luca Bortolussi, Scott A. Smolka |
|---|---|
| Přispěvatelé: | Bortolussi, L., Cairoli, F., Paoletti, N., Smolka, S. A., Stoller, S. D. |
| Rok vydání: | 2021 |
| Předmět: |
0209 industrial biotechnology
Computer science Bayesian inference Bayesian probability Predictive monitoring Word error rate 02 engineering and technology Machine learning computer.software_genre 020901 industrial engineering & automation Reachability Frequentist inference 0202 electrical engineering electronic engineering information engineering Hybrid automata reachability Hyperparameter business.industry 020207 software engineering Runtime verification Neural network Hybrid system Theory of computation Benchmark (computing) Conformal prediction Artificial intelligence business computer Neural networks Software Information Systems |
| Zdroj: | International Journal on Software Tools for Technology Transfer. 23:615-640 |
| ISSN: | 1433-2787 1433-2779 |
| DOI: | 10.1007/s10009-021-00623-1 |
| Popis: | Neural state classification (NSC) is a recently proposed method for runtime predictive monitoring of hybrid automata (HA) using deep neural networks (DNNs). NSC trains a DNN as an approximate reachability predictor that labels an HA state x as positive if an unsafe state is reachable from x within a given time bound, and labels x as negative otherwise. NSC predictors have very high accuracy, yet are prone to prediction errors that can negatively impact reliability. To overcome this limitation, we present neural predictive monitoring (NPM), a technique that complements NSC predictions with estimates of the predictive uncertainty. These measures yield principled criteria for the rejection of predictions likely to be incorrect, without knowing the true reachability values. We also present an active learning method that significantly reduces the NSC predictor’s error rate and the percentage of rejected predictions. We develop two versions of NPM based, respectively, on the use of frequentist and Bayesian techniques to learn the predictor and the rejection rule. Both versions are highly efficient, with computation times on the order of milliseconds, and effective, managing in our experimental evaluation to successfully reject almost all incorrect predictions. In our experiments on a benchmark suite of six hybrid systems, we found that the frequentist approach consistently outperforms the Bayesian one. We also observed that the Bayesian approach is less practical, requiring a careful and problem-specific choice of hyperparameters. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full text from SpringerLink