Rodrigo Casamayor
Lorena Arcega
Francisca Pérez
ABSTRACT
Video games have characteristics that differentiate their development and maintenance from classic software development and maintenance. These differences have led to the coining of the term Game Software Engineering to name the emerging subfield that intersects Software Engineering and video games. One of these differences is that video game developers perceive more difficulties than other non-game developers when it comes to locating bugs. Our work proposes a novel way to locate bugs in video games by means of evolving simulations. As the baseline, we have chosen BLiMEA, which targets classic software engineering and uses bug reports and the defect localization principle to locate bugs. We also include Random Search as a sanity check in the evaluation. We evaluate the approaches in a commercial video game (Kromaia). The results for F-measure range from 46.80%. to 70.28% for five types of bugs. Our approach improved the results of the baseline by 20.29% in F-measure. To the best of our knowledge, this is the first approach that is designed specifically for bug localization in video games. A focus group with professional video game developers has confirmed the acceptance of our approach. Our approach opens a new research direction for bug localization for both game software engineering and possibly classic software engineering.
- Apostolus Ampatzoglou and Ioannis Stamelos. 2010. Software engineering research for computer games: A systematic review. Information and Software Technology 52, 9 (2010), 888--901. Google Scholar
Digital Library
- Lorena Arcega, Jaime Font Arcega, Øystein Haugen, and Carlos Cetina. 2021. Bug Localization in Model-Based Systems in the Wild. ACM Trans. Softw. Eng. Methodol. 31, 1, Article 10 (oct 2021), 32 pages. Google ScholarDigital Library
- Lorena Arcega, Jaime Font, Øystein Haugen, and Carlos Cetina. 2019. An approach for bug localization in models using two levels: model and metamodel. Software and Systems Modeling 18, 6 (2019), 3551--3576. Google ScholarDigital Library
- Andrea Arcuri and Lionel Briand. 2014. A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability 24, 3 (2014), 219 -- 250. Google ScholarDigital Library
- Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18 (2013), 594--623.Google ScholarCross Ref
- Sinan Ariyurek, Aysu Betin-Can, and Elif Sürer. 2020. Enhancing the Monte Carlo Tree Search Algorithm for Video Game Testing. In IEEE Conference on Games, CoG 2020, Osaka, Japan, August 24--27, 2020. IEEE, 25--32. Google ScholarCross Ref
- Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2021. Automated Video Game Testing Using Synthetic and Humanlike Agents. IEEE Transactions on Games 13, 1 (2021), 50--67. Google ScholarCross Ref
- Daniel Blasco, Jaime Font, Mar Zamorano, and Carlos Cetina. 2021. An evolutionary approach for generating software models: The case of Kromaia in Game Software Engineering. J. Syst. Softw. 171 (2021), 110804. Google ScholarCross Ref
- Ilhem Boussaïd, Patrick Siarry, and Mohamed Ahmed-Nacer. 2017. A survey on search-based model-driven engineering. Automated Software Engineering 24, 2 (01 Jun 2017), 233--294. Google ScholarDigital Library
- Loli Burgueño, Javier Troya, Manuel Wimmer, and Antonio Vallecillo. 2015. Static Fault Localization in Model Transformations. IEEE Transactions on Software Engineering 41, 5 (May 2015), 490--506. Google ScholarDigital Library
- Norman Cliff. 1996. Ordinal methods for behavioral data analysis. Lawrence Erlbaum Associates, Inc. 212 pages. Google ScholarCross Ref
- William Jay Conover. 1999. Practical Nonparametric Statistics, 3rd Edition. Wiley, USA.Google Scholar
- Crytek. 2002. CRYENGINE | The complete solution for next generation game development by Crytek. https://www.cryengine.com. [Online; accessed 21-November-2021].Google Scholar
- Márcio de Oliveira Barros and Arilo Claudio Dias Neto. 2011. Threats to Validity in Search-based Software Engineering Empirical Studies. Technical Report 0006/2011.Google Scholar
- África Domingo, Jorge Echeverría, Oscar Pastor, and Carlos Cetina. 2020. Evaluating the Benefits of Model-Driven Development - Empirical Evaluation Paper. In Advanced Information Systems Engineering - 32nd International Conference, CAiSE 2020, Grenoble, France, June 8--12, 2020, Proceedings (Lecture Notes in Computer Science, Vol. 12127), Schahram Dustdar, Eric Yu, Camille Salinesi, Dominique Rieu, and Vik Pant (Eds.). Springer, 353--367.Google ScholarDigital Library
- Raihana Ferdous, Fitsum Meshesha Kifetew, Davide Prandi, I. S. W. B. Prasetya, Samira Shirzadehhajimahmood, and Angelo Susi. 2021. Search-Based Automated Play Testing of Computer Games: A Model-Based Approach. In Search-Based Software Engineering - 13th International Symposium, SSBSE 2021, Bari, Italy, October 11--12, 2021, Proceedings (Lecture Notes in Computer Science, Vol. 12914), Una-May O'Reilly and Xavier Devroey (Eds.). Springer, 56--71. Google ScholarDigital Library
- PC Gamer. 2020. How buggy is Cyberpunk 2077, really? https://www.pcgamer.com/how-buggy-is-cyberpunk-2077-really/. [Online; accessed 21-November-2021].Google Scholar
- Epic Games. 1998. Unreal Engine: The most powerful real-time 3D creation tool. https://www.unrealengine.com. [Online; accessed 21-November-2021].Google Scholar
- Salvador García, Alberto Fernández, Julián Luengo, and Francisco Herrera. 2010. Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences 180, 10 (2010), 2044 -- 2064. Special Issue on Intelligent Distributed Information Systems. Google ScholarDigital Library
- Robert J. Grissom and John J. Kim. 2005. Effect sizes for research: A broad practical approach. Lawrence Erlbaum Associates Publishers, New Jersey, United States.Google Scholar
- Ahmed E. Hassan and Richard C. Holt. 2005. The Top Ten List: Dynamic Fault Prediction. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM '05). IEEE Computer Society, USA, 263--272. Google ScholarDigital Library
- Sidra Iftikhar, Muhammad Zohaib Iqbal, Muhammad Uzair Khan, and Wardah Mahmood. 2015. An automated model based testing approach for platform games. In 18th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, MoDELS 2015, Ottawa, ON, Canada, September 30 - October 2, 2015, Timothy Lethbridge, Jordi Cabot, and Alexander Egyed (Eds.). IEEE Computer Society, 426--435. Google ScholarCross Ref
- Hisao Ishibuchi, Yusuke Nojima, and Tsutomu Doi. 2006. Comparison between Single-Objective and Multi-Objective Genetic Algorithms: Performance Comparison and Performance Measures. In 2006 IEEE International Conference on Evolutionary Computation. 1143--1150. Google ScholarCross Ref
- Luca Pascarella, Fabio Palomba, Massimiliano Di Penta, and Alberto Bacchelli. 2018. How is video game development different from software development in open source?. In Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018, Gothenburg, Sweden, May 28--29, 2018, Andy Zaidman, Yasutaka Kamei, and Emily Hill (Eds.). ACM, 392--402. Google ScholarDigital Library
- Cristiano Politowski, Fábio Petrillo, and Yann-Gaël Guéhéneuc. 2021. A Survey of Video Game Testing. In 2nd IEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2021, Madrid, Spain, May 20--21, 2021. IEEE, 90--99. Google ScholarCross Ref
- Jeanine Romano, Jeffrey D. Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen'sd for evaluating group differences on the NSSE and other surveys?. In annual meeting of the Florida Association of Institutional Research. 1--3.Google Scholar
- Jesús Sánchez Cuadrado, Esther Guerra, and Juan Lara. 2017. Static Analysis of Model Transformations. IEEE Transactions on Software Engineering 43, 9 (2017), 868--897. Google ScholarDigital Library
- Jesús Sánchez Cuadrado, Esther Guerra, and Juan Lara. 2018. Quick Fixing ATL Transformations with Speculative Analysis. Softw. Syst. Model. 17, 3 (July 2018), 779--813. Google ScholarDigital Library
- SlashData. 2019. Global developer population report 2019. https://sdata.me/GlobalDevPop19. [Online; accessed 21-November-2021].Google Scholar
- Unity Technologies. 2005. Unity Real-Time Development Platform | 3D, 2D VR & AR Engine. https://unity.com. [Online; accessed 21-November-2021].Google Scholar
- Javier Troya, Sergio Segura, Jose Antonio Parejo, and Antonio Ruiz-Cortés. 2018. Spectrum-Based Fault Localization in Model Transformations. ACM Trans. Softw. Eng. Methodol. 27, 3, Article 13 (Sept. 2018), 50 pages. Google ScholarDigital Library
- Rosalia Tufano, Simone Scalabrino, Luca Pascarella, Emad Aghajani, Rocco Oliveto, and Gabriele Bavota. 2022. Using Reinforcement Learning for Load Testing of Video Games. In Proceedings of the 44th International Conference on Software Engineering (ICSE 2022), Pittsburgh, USA.Google ScholarDigital Library
- András Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132. arXiv:http://jeb.sagepub.com/content/25/2/101.full.pdf+html Google ScholarCross Ref
- W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A Survey on Software Fault Localization. IEEE Trans. Software Eng. 42, 8 (2016), 707--740. Google ScholarDigital Library
- Yan Zheng, Changjie Fan, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, and Yingfeng Chen. 2019. Wuji: Automatic Online Combat Game Testing Using Evolutionary Deep Reinforcement Learning. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11--15, 2019. IEEE, 772--784. Google ScholarDigital Library
- Meng Zhu and Alf Inge Wang. 2020. Model-driven Game Development: A Literature Review. ACM Comput. Surv. 52, 6 (2020), 123:1--123:32. Google ScholarDigital Library
- Thomas Zimmermann, Peter Weisgerber, Stephan Diehl, and Andreas Zeller. 2004. Mining Version Histories to Guide Software Changes. In Proceedings of the 26th International Conference on Software Engineering (ICSE '04). IEEE Computer Society, Washington, DC, USA, 563--572. http://dl.acm.org/citation.cfm?id=998675.999460Google ScholarDigital Library
Index Terms
- Bug localization in game software engineering: evolving simulations to locate bugs in software models of video games
Recommendations
Bug localization via searching crowd-contributed code
Internetware '14: Proceedings of the 6th Asia-Pacific Symposium on InternetwareBug localization, i.e., locating bugs in code snippets, is a frequent task in software development. Although static bug-finding tools are available to reduce manual effort in bug localization, these tools typically detect bugs with known project-...
Considering dependencies between bug reports to improve bugs triage
AbstractSoftware development teams need to deal with several open reports of critical bugs to be addressed urgently and simultaneously. The management of these bugs is a complex problem due to the limited resources and the deadlines-pressure. Most of the ...
Potential biases in bug localization: do they matter?
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software EngineeringIssue tracking systems are valuable resources during software maintenance activities and contain information about the issues faced during the development of a project as well as after its release. Many projects receive many reports of bugs and it is ...
Comments