第14章 系统测试与课程有关的问题与课程有关的问题Outline o系统测试的概要引见o线索o需求规格阐明的根本概念o寻觅线索o线索测试的构造战略o线索测试的功能战略o系统测试指点方针与课程有关的问题与课程有关的问题系统测试的概要引见o系统测试是测试的三级层次中最接近日常测试实际的。它是根据预期来评价产品而不是根据规格阐明或规范o测试软件系统能否符合一切需求，包括功能性需求主要根据是需求规格阐明书和非功能性需求如可用性测试o普通由独立测试人员测试小组执行，通常采用黑盒测试方式与课程有关的问题与课程有关的问题系统测试之功能性测试o目的o检查运用程序能否到达了它预期的要求o判别根据o行为/功能阐明书o测试用例o一系列的原子系统功能Atomic System Function, ASF与课程有关的问题与课程有关的问题系统测试之压力测试opush it to its limit + beyondApplication(System):UsersVolumeResources: phy. + logicalresponse与课程有关的问题与课程有关的问题o系统测试之性能测试o用户的角度察看odelay, throughputo系统的支撑强度omemory,、CPU等o系统测试之可用性测试o系统操作过程中人的要素oGUI, messages, reports, 与课程有关的问题与课程有关的问题线索threado线索的多种看法o普通运用的场景o系统级测试用例o鼓励/呼应对o由系统级输入序列产生的行为o端口输入和输出事件的交替序列o系统形状机描画中的转换序列o根据系统级线索看系统测试n对象音讯和方法执行的交替序列n机器指令序列n源指令序列nMM-途径序列n原子系统功能序列与课程有关的问题与课程有关的问题线索的层次o线索有不同的层次o单元级通常被了解为源指令执行时间途径，或DD-途径o集成级为MM-途径，即模块执行和音讯交替序列o系统级是原子系统功能序列o由于ASFs在输入和输出时有端口事件，因此ASFs包含端口输入和输出事件的交替序列o线索提供三层测试的一致视图o单元测试进展单个函数测试o单元之间集成测试检查交互o系统测试检查原子系统功能间的交互与课程有关的问题与课程有关的问题线索定义o原子系统功能，ASF定义oASF是一种在系统层可以察看到的端口输入和输出事件的行动oASF开场于一个端口输入事件，遍历一个或多个MM-途径的一部分，以一个端口事件终了o举例o在SATM系统中，数字输入、现金给付和会话封锁都是ASF的好例子与课程有关的问题与课程有关的问题与课程有关的问题与课程有关的问题线索定义续oASF表示集成测试与系统测试间的缝隙，是集成测试的最大测试项，是系统测试的最小测试项，因此可以在两个级别上测试ASFo举例-数字输入ASFo系统测试过程o端口输入事件是由keysensor检测到的按下物理键，并作为字符串变量发送给GetPIN 。GetPIN确定能否按下数字键或取消键，并做出相应的应对。ASF最后显示屏幕2或4o集成测试过程o不要求系统输入键并在屏幕上显示，而是运用驱动器提供的这些功能，并经过集成测试检查数字输入ASF与课程有关的问题与课程有关的问题线索定义续oASF图经过ASF定义的系统o是一种有向图，其中的节点表示ASF，边表示串行流o源ASF/汇ASFo是一种原子系统功能，在系统ASF图中作为源/汇节点出现o例子-SATM系统o“ATM卡输入源ASF；会话终了ASF-汇ASF与课程有关的问题与课程有关的问题线索定义续o系统线索o在系统的ASF图中，是一条从源ASF到汇ASF的途径o线索图给定一个经过系统线索定义的系统o是一种有向图，节点表示系统线索，边表示单个线索的顺序执行o上述这组定义提供了线索更广的视图的内聚集，以非常短的线索开场一个单元内，以系统级线索之间的交互终了与课程有关的问题与课程有关的问题需求规格阐明的根本概念o目的o讨论系统测试级别上的一组根本需求规格阐明的构造o五种根本需求规格阐明的构造组成oDataoActionsoPortsoEventsoThreadso每个系统都可以用上述5个根本概念表示o为了阐明它们是如何支持测试人员的线索标识过程与课程有关的问题与课程有关的问题以数据为中心的线索标识o在以数据描画的系统中o关注的是系统所运用和创建的信息。采用变量、数据构造、字段、记录、数据存储和文件来描画数据o举例：E/R模型是最高层数据描画的最常见的选择o以数据为中心的观念还是许多面向对象方法的切入点o数据指经过初始化、存储、更新或能够销毁的信息o举例-SATM系统o初始数据描画各种帐户PAN及其PIN，每个帐户都有一个数据构造，包含诸如帐户余额这样的信息与课程有关的问题与课程有关的问题以数据为中心的线索标识续o以数据为中心的系统经常以CRUD行动开发“Create, Retrieve, Update, Deleteo有时线索可直接经过数据模型标识。o数据实体之间的关系可以是1：1、1：n等，这些差别在处置数据的线索中都有运用o举例-o银行客户可拥有多个帐户，每个帐户需求独一的PIN。假设多人可以访问同一个帐户，那么需求具有一样的PAN的ATM卡与课程有关的问题与课程有关的问题以数据为中心的线索标识续o还能够有只读取但从不写入的初始数据如PAN和“预期PIN对偶o这种只读数据必需是系统初始化过程的一部分o假设不是，那么必需有创建这种数据的线索o因此，只读数据是一种源ASF指示器与课程有关的问题与课程有关的问题以行动为中心的线索标识o以行动为中心建模是需求规格阐明的一种常见方式o行动有输入和输出，这些输入和输出可以是数据，也可以是端口事件o行动同义词：转换、数据转换、控制转换、处置、活动、义务、方法和效力o行动还可分解为底层活动如数据流图o行动的输入/输出视图是功能性测试的根底，行动的分解以及最终实现那么是构造性测试的根底与课程有关的问题与课程有关的问题以端口为中心的线索标识o每个系统都有端口和端口设备o这些端口设备是系统级输入和输出的源和目的地o区分端口和端口设备间的微小差别o技术上，端口是I/O设备接入系统的点o如串行端口、并行端口、网络端口、端口o假设没有实践端口设备，系统测试可以经过“将端口边境向内挪动到端口事件的逻辑实例上实现o用术语“端口替代端口设备o如SATM系统中端口：数字、取消键、功能键、显示屏幕、存款和取款通道、ATM卡和收据槽o思索端口有助于测试人员定义功能性测试和系统测试所需的输入空间与课程有关的问题与课程有关的问题以事件为中心的线索标识o有数据和行动方面的一些特征o事件是发生在端口设备上的系统级输入或输出o事件也可以是行动的输入和输出o事件可以是离散如SATM键盘输入，也可以是延续如温度、高度或压力o离散事件必需有一定的继续时间，这在实时系统至关重要o可把输入事件看作是破坏性读出数据，输出事件看作是破坏性写入数据与课程有关的问题与课程有关的问题以事件为中心的线索标识续o事件是物理事件和这些事件的内部逻辑表示的转换点o端口输入事件是物理到逻辑的转换o端口输出事件是逻辑到物理的转换o举例-SATM系统o当显示屏幕5时，按B1键的端口输出事件表示“余额，当显示屏幕6时，表示“检查o这叫做“与语境有关的端口事件与课程有关的问题与课程有关的问题线索o线索是5种根本构造中最不经常运用的o由于要测试线索，因此测试人员通常不能在数据、事件和行动之间的交互中找出线索o通常在控制模型中容易找出线索o发现线索o系统的有限形状机模型是发现线索的一个良好的起点，由于这些途径很容易转换为线索与课程有关的问题与课程有关的问题根本概念的E/R模型inputoutputDataEventActionPortThreadMainline requirements specification techniques populate some (or all ) portions of this database.与课程有关的问题与课程有关的问题采用根本概念建模DataEventActionDeviceThreadStructural ModelContext ModelBehavior ModelCondition三种根本方三种根本方式的需求规式的需求规格阐明模型格阐明模型与课程有关的问题与课程有关的问题采用根本概念建模续o构造模型用于开发，表示功能分解、数据分解和组件之间的接口o语境模型经常是构造模型的开场点，强调系统设备，也比较强调行动，非常间接地关注线索o行为模型控制模型将5种根本构造中的四种集成到一同。o选择适当的控制模型是需求规格阐明的根底o太弱的模型不能表示重要的系统行为，太强的模型普通会淹没没有意思的行为o普通地，决策表只对计算系统是个好选择，有限形状机对于菜单驱动的系统良好，petri网是针对并发系统的模型与课程有关的问题与课程有关的问题寻觅线索与课程有关的问题与课程有关的问题Test threadsoThe macro level states are:oCard entryoPIN entryoTransaction request (and processing)oSession managementoNote that this stated order is the same as the testing order since these stages are in prerequisite order. (We cant enter a PIN until successfully card entry)与课程有关的问题与课程有关的问题PreconditionsoWe need some precondition data:PANExpected PINCheq. BalanceSavingsbalance1001234$1000.00 $800.002004567$100.00$90.003006789$25.00$20.00与课程有关的问题与课程有关的问题ContdoSATM is initially displaying screen 1.oThe total cash available to the withdrawal dispenser is $500oWe will express threads in tables in which pairs of rows correspond to port inputs and expected port outputs. oLets start with 3 basics threads, one for each transaction type.与课程有关的问题与课程有关的问题Balance inquiryThread(Balance)Card (PAN)PIN entry Tran. requestSession manage.Port inputs1001234B1, B1B2Port outputs Screen 2Screen 5Screen 6,14, $1000.00Screen 15Eject card, screen 1与课程有关的问题与课程有关的问题DepositThread(Deposit)Card (PAN)PIN entry Tran. requestSession manage.Port inputs1001234B2, B1, 25.00insert envelopeB2Port outputsScreen 2Screen 5Screen 6,7, 13, Dep. Door open Screen 14, $1025Screen 15Eject card, screen1与课程有关的问题与课程有关的问题WithdrawalThread(Withdr.)Card (PAN)PIN entry Tran. requestSession managePort inputs 1001234B3, B2, 30.00B2Port outputsScreen 2Screen 5Screen 6,7,11, withdrawal door opens, 3$10notes, Sc14, $770.00Screen 15Eject card, screen 1与课程有关的问题与课程有关的问题Invalid cardoThis is the shortest thread, it consists of an invalid card, which is immediately rejected. ThreadCard (PAN)PIN entryTran. requestSession manage.Port inputs400Port outputsEject card,Screen 1与课程有关的问题与课程有关的问题Variation of PIN entryoFollowing the macro state along thread 1, we next perform variations on PIN entry.oWe get 4 threads, which yield edge coverage in the PIN entry finite state machine与课程有关的问题与课程有关的问题PIN entryThread5 (balance)Card (PAN)PIN entryTran. requestSession manage.Port inputs10012351234As in thread 1Port outputsScreen 2Screen 3,2,5Thread6 (balance)Card (PAN)PIN entryTran. requestSession manage.Port inputs100C1234As in thread 1Port outputsScreen 2Screen 3,2,5与课程有关的问题与课程有关的问题PIN entry (contd)Thread7 (balance)Card (PAN)PIN entryTran. requestSession manage.Port inputs1001C12C1234As in thread 1Port outputs Screen 2Screen 3,2,3, 2,5Thread8 (balance)Card (PAN)PIN entryTran. requestSession manage.Port inputs100123C1C1CPort outputs Screen 2Screen 3,2,3, 2,4,1与课程有关的问题与课程有关的问题Transaction requestoWe have variations regarding the three types of transactions, the two types of accounts, and the amount requested.oNow we focus on the amount driven-threads.与课程有关的问题与课程有关的问题Wrong amountoRejection the attempt to withdraw an amount not in $10 increments.Thread9(withdraw)Card (PAN)PIN entryTran. requestSession manage.Port inputs1001234B3,B2,15.00CancelB2Port outputsScreen 2Screen 5Screen 6,7,9,7Screen 15,eject card, screen 1与课程有关的问题与课程有关的问题Over withdrawal Thread10 (withdraw)Card (PAN)PIN entryTran. requestSession manage.Port inputs3006789B3,B2,50.00CancelB2Port outputsScreen 2Screen 5Screen 6,7,8Screen 15,eject card, screen 1Attempt to withdraw more than the account balance.与课程有关的问题与课程有关的问题Dispenser limitoAttempt to withdraw more cash than the dispenser containsThread11(withdraw)Card (PAN)PIN entryTran. requestSession manage.Port inputs1001234B3,B2,510.0CancelB2Port outputsScreen 2Screen 5Screen 6,7,10Screen 15,eject card, screen 1与课程有关的问题与课程有关的问题Session managementThread12 (balance)Card (PAN)PIN entryTran. requestSession manage.Port inputs1001234B1,B1B1, CancelPort outputs Screen 2Screen 5Screen 6,14$1000.00Screen 15,5,15, eject card screen 1与课程有关的问题与课程有关的问题Deposit failureoNotice that we didnt use screen 12, which informs the user that deposits cant be processed. oThis is an example of a thread selected by a precondition that is hardware failure. Lets just give it a name: thread 13.与课程有关的问题与课程有关的问题Threads for context sensitive input events oObjective: exercise context sensitive input events.oNote: Some of the 13 threads exercise context sensitivity.与课程有关的问题与课程有关的问题Context sensitive threadsThreadKeystroke Screen Logicalmeaning6Cancel2PIN entry error14Cancel5Tran. Selec. Error15Cancel6Acc. Selec. Error16Cancel7Amo. Selec. Error17Cancel8Amo. selec. Error18Cancel13Depo. env. not ready1B15Balance1B16Checking19B110Yes nonwithdrawal T 20B112Yes non deposit T与课程有关的问题与课程有关的问题Contd12B114Yes another T2B25Deposit3B26Savings21B210No (no additional T)22B21No (no additional T)1B214No (no additional T)与课程有关的问题与课程有关的问题oThese 22 threads comprise a reasonable test of the SATM system.oOf course, certain aspects are untested.oExample: Amount entry process, the possibility of a Cancel keystroke at any point during amount entry produces a multiplicity greater than that of PIN entry与课程有关的问题与课程有关的问题线索测试的构造战略o生成线索测试用例很容易，但确定实践运用哪个测试用例那么更复杂o在系统级同样存在单元级的途径爆炸问题与课程有关的问题与课程有关的问题线索测试的构造战略-自底向上组织线索o当在层次构造中组织形状机时，可以自底向上进展o举例o在“PIN输入尝试形状机中，共有6条途径。o遍历这6条途径要测试三件事：正确识别并回显所输入的数字、呼应取消键入和匹配预期PIN和所输入的PAN。o一旦测试了这个部分后，可以上升到“PIN输入形状机，共有4条途径与课程有关的问题与课程有关的问题与课程有关的问题与课程有关的问题线索测试的构造战略-自底向上组织线索续o正如在单元测试中所见，构造性测试能够会产生误导o假设是途径遍历会发现缺陷，遍历各种途径会产生冗余o这些线索还有一个更严重的问题：由于是分层形状机，因此实践上这些线索不能单独执行与课程有关的问题与课程有关的问题节点与边覆盖目的o由于有限形状机是有向图，因此可以运用与单元级一样的测试覆盖目的o层次构造关系指出上层形状机必需把下层形状机作为输入和前往过程处置o两种根本的选择是节点覆盖和边覆盖o节点覆盖类似于单元级的语句覆盖o边形状转移覆盖是更可接受的规范。假设形状机“构建良好经过端口事件转换，那么覆盖也可保证端口事件覆盖与课程有关的问题与课程有关的问题线索测试的功能战略o假设一个系统没有行为模型即FSMso有两种选择o开发行为模型o进展功能性测试在系统级上的类比o在标识功能性测试用例时，我们运用了输入和输出空间信息，以及功能本身o采用经过三个根本概念导出的覆盖目的描画功能线索o事件、端口和数据与课程有关的问题与课程有关的问题线索测试的功能战略-基于事件的线索测试o思索端口输入事件的空间。五个端口输入事件覆盖目的：oPI1: each port input event occurs oPI2: common sequences of port input events occur oPI3: each port input event occurs in every “relevant data context oPI4: for a given context, all “inappropriate input events occur oPI5: for a given context, all possible input events occur处置与语境有关的端口输入事件，是物理输入事件处置与语境有关的端口输入事件，是物理输入事件与课程有关的问题与课程有关的问题端口输出事件定义的覆盖目的oPO1: each port output event occurs. oPO1 is acceptable minimum especially when a system has a rich variety of output messages for error conditions. oPO2: each port output event occurs for each cause. oPO2 is good but hard to quantify oBasically it refers to threads that interact with respect to a port output event. oUsually a given port output event has a small number of causes. 与课程有关的问题与课程有关的问题基于端口的线索测试o是基于事件测试的有用补充o经过基于端口的测试，对于每个端口都要讯问端口上会出现什么事件o根据每个端口的事件列表寻觅运用输入端口和输出端口的线索o对于端口设备来自外部提供商的系统特别有用o从E/R模型可看出，设备和事件之间的多对多测试应该在两个方向上进展o基于事件的测试覆盖从事件到端口的一对多关系，反之，基于端口的测试覆盖从端口到事件的一对多关系与课程有关的问题与课程有关的问题基于数据的线索测试o基于端口和基于事件的测试适宜主要以事件驱动的“反响式系统o由于这些系统要对鼓励端口输入事件做出呼应，并且反响经常以端口输出事件的方式做出o反响式系统的两个特征：长时间运转和维持与环境的关系o普通地，事件驱动的反响式系统没有有意思的数据模型，因此基于数据模型的线索不是特别有用与课程有关的问题与课程有关的问题基于数据的线索测试续o非反响式系统是典型的“静态，因此是转换式系统o支持以数据库为根底的事务处置，普通采用E/R 模型描画o定义基于数据的覆盖目的 描画线索集合o关系中的信息普通包含在系统级线索中，而实体中的线索普通局限于单元级与课程有关的问题与课程有关的问题基于数据的线索测试续o定义以下覆盖：oDM1: exercise the cardinality of every relationship .oDM2: exercise the participation of every relationship .oDM3: exercise the functional dependencies among relationships与课程有关的问题与课程有关的问题oDM2: oensure where or not every instance of an entity participates in a relationship. oSome modeling techniques express participation in numerical limits (i.e. OMT, “at least one and at most 12). oWhen available, this information leads to boundary value system threads.oDM3: oit may be possible to determine explicit logical connections among relationships = functional dependencies as in relational databases. oThese are reduced when the database is normalized, but they still exist and lead to interesting test threads. 与课程有关的问题与课程有关的问题系统测试指点方针o伪构造系统测试o运转剖面o累进测试和回归测试与课程有关的问题与课程有关的问题伪构造系统测试oAt the system level we can use graph based metrics as a cross check on the functional threads. oThe claim is for pseudo-structural testing since the node and edge coverage metrics are defined in terms of a control model of a system and not directly derived from the system implementation. oBehavioral models are only an approximation to the systems reality:owhy it is possible to decompose the models to several levels of detail. oA true structural models sheer size and complexity would make it too cumbersome to use. 与课程有关的问题与课程有关的问题oDecision tables and FSMs are good choices for ASF testing. For example:oIf an ASF is described using a decision table, conditions usually include port input events, and actions are port output events oIt is possible to devise test cases that cover every condition, every action, or most completely, every rule.oAs in FSMs, test cases can cover every state, transition or every path.与课程有关的问题与课程有关的问题运转剖面oMany aspects of testing can be related to the old 80/20 rule: ofor a system with many threads, 80% of the execution traverses only 20% of the threads. oThe basic concept in testing: oexecute test cases such that when a failure occurs, the presence of a fault is revealed. oThe distribution of faults is only indirectly related to the reliability of the system.与课程有关的问题与课程有关的问题oSimple view of reliability: oThe probability that no failures occur during a specific time interval.oIf faults are in less traveled threads of system, othen reliability will appear higher than if the same number of faults were in the high traffic areas.oOperational profiles: oDetermine the execution frequencies of various threads, then select threads accordingly.oOperational profiles maximize the probability of finding faults by inducing failures in the most frequently traversed threads. 与课程有关的问题与课程有关的问题System Testing CategoriesoObjective of system testing:oTo verify whether the implementation (or system) conforms to the requirements as specified by the customer(s).oTo verify whether the system meets a wide range of unspecified expectationsoSystem testing is performedoAfter constructing a reasonably stable system in an emulated environment.oIn the real environment (if the real environment is not accessible, system testing is done using models in an emulated environment)oIt is important to categorize the kinds of system tests for a number of reasons:oSystematically focus on different aspects of a system while evaluating its quality.oTest engineers can prioritize their activities.oPlanning based on test categorization has the advantage of obtaining a balanced view of testing.oTesting categories (11 categories)与课程有关的问题与课程有关的问题Basic System TestsoThese provide evidence that the system can be installed, configured and brought to an operational state.oBasic tests are performed to ensure that commonly used functions, not all of which may directly relate to user-level functions, work to our satisfaction.oThe following are the major categories of subsystems whose adequate testing is called basic test.oa. Boot tests: verify that the system can boot up its software image from the supported options (ROM, PCMCIA, .) o b. Upgrade/downgrade tests: verify that the software image can be upgraded or downgraded (rolled back) in a graceful manner.o c. Light emitting diode (LED) tests. These are designed to ensure that that visual operational status of the system is correct.o d Diagnostic tests. These are designed to ensure that the hardware components of the o systems are functioning as desired. (Power-on self test (POST), Memory, Address and data buses, Peripheral devices)oe) CLI (command line interface) tests. Ensure that the system can be configured. Ensure that uses commands are properly interpreted. Verify the error message与课程有关的问题与课程有关的问题Functionality TestsoVerify the system as thoroughly as possible over the full range of requirementsoLogging and tracing tests: (implicit functionality)oGUI tests: (Icon, Menu bar, Dialog box, Scroll bar)oSecurity tests. Verify that the system meets the requirements for detecting security breaches and protecting from such breaches (Unauthorized access, Illegal file access, Virus)与课程有关的问题与课程有关的问题Robustness TestsoTo verify how gracefully the system behaves in error situations, or how it handles a change in its operational environment.oDifferent kinds of tests are:oBoundary value tests (valid and invalid inputs)oRecover from power failureoOn-line insertion and removal (OIR)oSystem recovers after an OIR eventoAvailability of redundant modulesoDegraded node test (a portion of the system fails)与课程有关的问题与课程有关的问题Interoperability and Performance TestsoInteroperability tests (3rd party products): Verify that the system can inter operate with 3rd party products.oPerformance tests: Determine how actual system performance compares to predicted performanceoResponse timeoExecution timeoThroughputoResource utilizationoTraffic volume与课程有关的问题与课程有关的问题Scalability TestsoVerify that the system can scale up to its engineering limitsoData storage limitations (counters and buffers)oSpeed limitations (CPU)oCommunication limitationsoResource intensive与课程有关的问题与课程有关的问题Stress TestsoStress tests (push it over the edge to break it). Evaluate the behavior of a software component when offered load is in excess of its designed capacityoPush the system “over the edge and observe that the recovery mechanism worksoBring out the following kinds of problems:oMemory leaksoBuffer allocation problem与课程有关的问题与课程有关的问题Load and Regression TestsoLoad and stability tests: Verify that the system can operate in a large scale for a long time (months)oRegression tests: Ensure that nothing had has happened after a fix.oFive possibilities can happen after an attempt to fix.ofix the bug reportedofail to fix the bugofix the bug, but break something elseofail to fix the bug, but break something elseofix this bug and fix some unknown bugs与课程有关的问题与课程有关的问题Documentation TestsoDocumentation tests: This is a review of technical accuracy and readability of user manuals includingotutorialsoon-line helpoThere are three kinds of documentation testsoRead test (clarity, organization, flow and accuracy)oHands-on test (evaluate usefulness)oFunctional test (verify the document)与课程有关的问题与课程有关的问题Conformance to Regulatory Bodies oConformance to regulatory bodies:oIdentify unsafe consequencesoFederal Communication Commission (FCC) and Canadian StandardsoAssociation (CSA) certify a products safety.