组合测试研究进展The latest research development of combinatorial testing
聂长海;
摘要(Abstract):
经过30多年的发展,组合测试已经发展成为1种独立的和颇具影响的软件测试方法。该方法利用组合覆盖表作为测试用例集,可以有效检测软件系统中各种因素及其相互作用而促发的各种故障。论文在充分分析已有文献的基础上,结合作者在该领域的研究实践,首次系统介绍组合测试的演化发展,总结出其中的关键问题、方法、重要应用和未来发展方向,为组合测试的进一步研究和发展提供支持和参考。
关键词(KeyWords): 软件测试;组合测试;覆盖表;测试用例生成;故障诊断
基金项目(Foundation): 高等学校博士学科点专项科研基金资助项目(20130091110032)
作者(Author): 聂长海;
Email:
DOI:
参考文献(References):
- [1]NIE C,WU H.Combinatorial testing repository[EBOL].(2016-05-20)http://gist.nju.edu.cn/ct_repository/.
- [2]COHEN D M,DALAL S R,FREDMAN M L,et al.The AETG system:an approach to testing based on combinatorial design[J].IEEE Transactions on Software Engineering,1997,23(7):437-444.
- [3]NIE C,LEUNG H.A survey of combinatorial testing.ACM computing surveys[J].2011,43(2):11:1-11:29.
- [4]DALAL S R.,MALLOWS C L.Factor-covering designs for testing software[J].Technometrics,1998,40(3):234-243.
- [5]DALAL S R,JAIN A,KARUNANITH N,et al.Model-based testing in practice[C]//Proceedings of the International Conference on Software Engineering.Los Angeles,USA,IEEE,1999:285-294.
- [6]DALAL S R,JAIN A,KARUNANITHI N,et al.Model-based testing of a highly programmable system[C]//Proceedings of 9th International Symposium on Software Reliability Engineering.Paderborn,Germany:[s.n.],1998:174-178.
- [7]LOTT C,JAIN A,DALA S L.Modeling requirements for combinatorial software testing[C]//Proceedings of the first international workshop on Advances in Modelbased,ACM SIGSOFT Software Engineering Notes.Missouri,USA:IEEE,2005,30(4):1-7.
- [8]WILLIAMS A W.Software component interaction testing:coverage measurement and generation of configurations[D].Canada:University of Ottawa,2002.
- [9]KRISHNAN R,KRISHNA S M,NANDHAN P S.Combinatorial testing:learnings from our experience[J].ACM SIGSOFT Software Engineering Notes,2007,32(3):1-8.
- [10]GRINDAL M,OFFUTT J.Input parameter modeling for combination strategies[C]//Proceedings of IASTED International Conference on Software Engineering.Innsbruck.Austria:IEEE,2007:255-260.
- [11]CZERWONKA J.Pairwise Testing in real world:practical extensions to test case generator[C]//Proceedings of 24th Pacific Northwest Software Quality Conference.Portland,Oregon,USA:IEEE,2006:419-430.
- [12]NIE C,CHEN S,LEUNG H,et al.A case study of adaptive combinatorial testing[C]//Proceedings of IEEE 37th Annual Computer Software and Applications Conference Workshops.Kyoto,Japan:IEEE,2013:47-52.
- [13]NIE C,CHEN S,LEUNG H,et al.Adaptive combinatorial testing[C]//Proceedings of 13th International Conference on Quality Software.Nanjing,China:[s.n.],2013:284-287.
- [14]GRINDAL M,LINDSTROM B,OFFUTT J,et al.An evaluation of combination strategies for test case selection[J].Empirical Software Engineering,2006,11:583-611.
- [15]FOUCHE S,COHEN M.B,PORTER A.Towards incremental adaptive covering arrays[C]//Proceedings of The Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering.Dubrovnik,Croatia:ACM,2007:557-560.
- [16]NIE C,WU H,LIANG Y,et al.Search based combinatorial testing[C]//Proceedings of The 19th Asia-Pacific Software Engineering Conference.Hong Kong:IEEE,2012:778-783.
- [17]WU H,NIE C,KUO F,et al.A discrete particle swarm optimization for covering array generation[J].IEEE Transactions on Evolutionary Computation,2015,19(4):575-591.
- [18]WU H,NIE C.An overview of search based combinatorial testing[C]//Phil McMinn eds.The 7th International Workshop on Search Based Software Testing(SBST).Hyderabad,India:ACM,2014:27-30.
- [19]QU X,COHEN M B.,WOOLF K M.Combinatorial interaction regression testing:a study of test case generation and prioritization[C]//Proceedings of IEEE International Conference on Software Maintenance.Paris,France:IEEE,2007:255-264.
- [20]QU X,COHEN M B,ROTHERMEL G.Configuration-aware regression testing:an empirical study of sampling and prioritization[C]//Proceedings of ISSTA.Seattle,WA,USA:ACM,2008:75-86.
- [21]BRYCE R C,COLBOURN C J..Prioritized interaction testing for pair-wise coverage with seeding and constraints[J].Information and Software Technology,2006,48(10):960-970.
- [22]BRYCE R C,COLBOURN C J.Test prioritization for pairwise Interaction coverage[J].ACM SIGSOFT Software Engineering Notes,2005,30(4):1-7.
- [23]BRYCE R C,MEMOM A M.Test suite prioritization by interaction coverage[C]//Proceedings of DomainSpecific Approaches to Software Test Automation Workshop.Dubrovnik,Croatia:[s.n.],2007:1-7.
- [24]WU H,NIE C,KUO F.The optimal testing order in the presence of switching cost[J].Information and Software Technology,2016,80:57-72.
- [25]YILMAZ C,COHEN M B,PORTER A A.Covering arrays for efficient fault characterization in complex configuration spaces[J].IEEE Transaction on Software Engineering,2006,32(1):20-34.
- [26]COLBOURN CJ,MCCLARY D W.Locating and detecting arrays for interaction faults[J].Journal of Combinatorial Optimization,2008(15):17-48.
- [27]MARTINEZ C,MOURA L,PANARIO D,et al.Algorithms to locate errors using covering arrays[J].LATIN,2008:504-519.
- [28]NIE C,LEUNG H.The minimal failure-causing schema of combinatorial testing[J].ACM Transactions on Software Engineering and Methodology,2011,20(4):15:1-15:38.
- [29]BACH J,SCHREODER P J.Pairwise testing:a best practice that isn’t[C]//Proceedings of 22nd Pacific Northwest Software Quality Conference.USA:IEEE,2004:180-196.
- [30]SALEM A M.A software testing model:using design of experiments(DOE)and logistic regression[D].Florida:Florida Institute of Technology,2001.
- [31]YILMAZ C,FOUCHE S,COHEN M B,et al.Moving forward with combinatorial interaction testing[J].IEEE Computer,2014,47(2):37-45.
- [32]KHALSA S K,LABICHE Y.An orchestrated survey of available algorithms and tools for combinatorial testing[C]//Proceedings of International Symposium on Software Reliability Engineering.Naples,Italy:IEEE,2014:323-334.
- [33]ANAND S,BURKE E K,CHEN T Y,et al.An orchestrated survey of methodologies for automated software test case generation[J].Journal of Systems and Software,2013,86(8):1978-2001.
- [34]GRINDAL M,OFFUTT J,ANDLER S F.Combination testing strategies:a survey[J].Software Testing,Verification and Reliability,2005,15(3):167-199.
- [35]COLBOURN C J,DINITZ J H,STINSON D R.Applications of combinatorial designs to communications,cryptography and networking[J].Surveys in Combinatorics,1999,22:37-100.
- [36]COLBOURN C J,VANOORSHOT P C.Applications of combinatorial designs in computer science[J].ACM Computing Surveys,1989,21(2):223-250.
- [37]严俊,张健.组合测试:原理与方法[J].软件学报,2009,20(6):1393-1405.YAN Jun,ZHANG Jian.Combinatorial testing:princeple &methods[J].Chinese journal of software,2009,20(6):1393-1405.(in Chinese)
- [38]聂长海,组合测试[M].北京:科学出版社,2015:4.NIE Changhai.Combinatorial testing[M].Beijing:Science press,2015:4.(in Chinese)
- [39]聂长海,蒋静.覆盖表生成的可配置贪心算法优化[J].软件学报,2013,24(7):1469-1483.NIE Changhai,JIANG Jing.Configurable greedy algorithm for covering array generation[J].Chinese journal of software,2013,24(7):1469-1483.(in Chinese)
- [40]LI X,DONG Z,WU H,et al.Refining a randomized post-optimization method for covering arrays[C]//Proceedings of ICSTW 2014.Cleveland,OH,USA:IEEE,2014:143-152.
- [41]NIC C,WU H,NIU X,KUO F,et al.Combinatorial testing,random testing,and adaptive random testing for detecting interaction triggered failures[J].Information and Software Technology,2015,62:198-213.
- [42]钮鑫涛,聂长海,CHAN Alvin.组合测试故障定位的关系树模型[J].计算机学报,2014,37(12):2505-2518.NIU Xintao,NIE Changhai,CHAN Alvin.Relation tree model of fault location for combinatorial testing[J].Chinese Journal of computer,2014,37(12):2505-2518.(in Chinese)