NFV TestingLandscape

MP Odini/HPE – ETSI NFV Vice Chair

Feb, 2016

Overview– ETSI has worked on 2 aspects: Performance characteristics & metrics + testing methodology

– GS NFV-PER 001: NFV Performance & Portability Best Practises - published in ph1 (attached)

– GS NFV-INF 010: Quality metrics - published in ph1

– GS NFV-REL 005 - Report on Quality Accountability Framework – Published in ph2

– This is also referring to QuestForum: QuestForum (Release 5.0, July 2012): "TL 9000 Measurements Handbook".

– Available at http://www.tl9000.org/handbooks/measurements_handbook.html.

– NFV Testing methodology: TST001 on pre-validation test and TST002 on Interop test

– TST001 and TST002 are not published yet but are publicly available – as any ETSI NFV Draft (all publicly available) https://docbox.etsi.org/ISG/NFV/Open/Drafts/

– => we have a new proposal on the table to define Interop test suites , and ETSI is planning an NFV PlugTest later this year

– IETF has couple papers on benchmarking NFV:

– IETF draft-ietf-bmwg-virtual-net-01: “Considerations for Benchmarking Virtual Network Functions and Their Infrastructure”

– draft-huang-bmwg-virtual-network-performance-01: “Benchmarking methodology for Virtualization Network Performance” (but very ‘network / vswitch focused)

– OPNFV (Apache licence) has built a complete opensource Continuous Integration platform (Octopus) with a set of opensource tests environments (pdf on FuncTest)

– Base system functionality testing (FuncTest) => tempest, rally etc

– Platform Performance Benchmarking (Qtip) => Compute, Storage, Network test suite >> leveraging ETSI PER001

– VSwitch Performance characteristics (VsPerf)

– Infrastructure verification (Yardstick) =>

– Storage Performance Benchmarking for NFVI (StorPerf)

– Policy Test (policytest)

– Other opensource projects examples:

– Clearwater (GNU licence) – opensource IMS project – have developed some opensource test :

– clearwater-live-test - Live test for Clearwater deployments

ETSI NFV & Test

3

ETSI GS NFV-PER 001: NFV Performance & Portability Best Practises

Captures the HW and SW needs for VNFs/VMs dealing with network workloads, including data plane at middleboxes.

Table of Content:

– Introduction

– Methodology and relation to NFV use cases

– Bottleneck analysis based on tests for each kind of workload

– One sub-clause per kind of workload.

– To be extended with future tests

– Best practises and recommendations

– HW & Hypervisor capabilities

– Templates for portability

– List of parameters in Compute Host Descriptors and VM Descriptor templates

– Annex A: Gap analysis of hypervisor and cloud OS

– Annex B: Relevant technologies (Direct I/O, SRIOV, RDMA, Infiniband)

– Annex C: NFV Test Methodologies (Control plane testing, Data plane testing, benchmarking hypervisors, benchmark performance metrics,

– Annex D: Performance evaluation of an IP edge data plane

Benchmark Performance Metrics

QoS Metrics :

– Throughput

– Latency

– Frame Loss Rate

– Back-to-Back

– Packet delay variation

– Service Disruption Time for Fail-over Convergence

QoE Metrics:

QoE metrics are specific to the type of service or application, ex:

– HTTP: page load time, load time variance.

– Video: MOS-AV score, range = 2-5 with 5 being the best.

– HTML5 video - AS score, 100 % score as the maximum.

5

ETSI GS NFV-REL 005 : Quality Accountability Framework

– Goal – Develop an Informative Technical Report that:

– Promotes the development of capabilities by which VNFs, NFV infrastructure and MANO can eventually enable rapid and reliable root cause analysis of service quality impairments, corrective action, and SLA management.

– Scope:

– Define high level roles, service boundaries and identify associated NFV reference points; and their quality-related responsibilities to enable rapid root cause analysis and corrective actions

– Align high level roles with TMForum’s SLA management service delivery relationships

– Frame how traditional PNF quality measurements (e.g., TL 9000 Service Outage measurements) will be applied to VNFs to permit side-by-side quality comparisons

ETSI GS NFV-INF 010 : Service Quality Metrics

The Service Quality Metrics document provides a general taxonomy for NFV service quality metrics. It identifies metrics relevant to the service quality of virtual machines (VM), virtual network interfaces, technology components, and orchestration. It also describes use cases for service quality metrics and provides additional recommendations on aspects of measurements and service level agreements.

Table of Content:

• Taxonomy

• VM Quality Metrics (VM stall, Premature VM Release Ratio, VM Scheduling Latency, VM Clock Error)

• Virtual Network Interface Service Quality Metrics

• Technology Component Service Quality Metrics

• Orchestration Service Quality Metrics

• Service Quality Metrics Use Case

• Recommendations

• Annex: ex NFVI-VIM Interactions related to Service Quality Metrics

7

Service Quality Metrics & NFV Reference Architecture

8

Service Quality Metrics on NFVI

9

Summary of NFV Quality Metrics

10

NFVI-VIM Interactions related to Service Quality Metrics

11

across the [Nf-Vi]/<C,N,H> reference points

Resource Establishment:

VM Provisioning latency:

VM Provisioning reliability:

VM Dead on Arrival (DOA) ratio

This metric intends to track the

liveliness of the requested VM.

(Example)

TST001: PRE-DEPLOYMENT TESTING

Recommendations for lab validation of VNFs, their interaction with the NFV functional blocks and the NFV blocks themselves

TST002: INTEROPERABILITY TESTING METHODOLOGY

Study how interoperability test methodology can be applied to NFV by analysing the functional blocks and interfaces defined within the NFV architecture

TST001 & TST002

ETSI NFV GS TST001: Pre-deployment Testing

Table of content

• Definition of SUTs

• Test methods for pre-deployment validation of SUTs

• Pre-deployment validation of NFV Infrastructure

• Pre-deployment validation of VNFs

• Pre-deployment validation of Network Services

• Reliability & Resiliency Requirements

• Security Considerations

• ( Annex A: Open source communities in NFV )

13

The functions under test (FUT) are entities which are also commonly

known as Devices Under Test (DUT) in the testing community

SUT (System Under Test)

14

Pre-deployment validation of VNFs

15

VNF Lifecycle testing

VNF on-boarding

VNF instantiation =>

VNF scaling/updating

VNF termination

Specs incl the same for scaling, termination

ETSI NFV GS TST002

Table of content

Interoperability Testing Methodology Guidelines for NFV

Basic concepts for interoperability testing (SUT, FUT, etc)

Interoperability Test Specifications

NFV SUT Architecture

NFV Interoperability Features

16

TST002 basic concepts

17

NFV SUT architecture

18

VNF Package

19

Similar table for:

OPNFV & Test

OPNFV Projects

45 projects approved

As of Feb 11th , 2016

https://wiki.opnfv.org/approved_projects

testingperformance

- Pharos- FuncTest- Yardstick- Policy test

- Vswitch performance- Platform performance Qtip- StorPerf- Software Fastpath service quality metrics

OPNFV Continuous Integration (Octopus)

OPNFV Pharos - Testbed

23

12 Labs

(as of Feb 11th

2016)

Linux Foundation, Oregon, USSpirent, California, USChina Mobile, ChinaDell, California, USEnea, SweedenEricsson, Canada

Huawei, ChinaHuawei, CaliforniaIntel, OregonOrange, France (Lannion, Paris)Zte, China

Soon: HPE, Israel

Each pharos lab has a number of OPNFV POD

Some POD are dedicated to certain activities -

https://wiki.opnfv.org/pharos_rls_b_labs

FuncTest: OPNFV functional testinghttps://wiki.opnfv.org/opnfv_functional_testing

Scope: functional testing framework along with a set of test suites and test cases to test and verify the functionality OPNFV platform (VIM +NFVI) – there are 2 functest : Arno and Bramahputra

– Tooling: Rally (benchmark, Tempest), Robot framework, Teston framework

Ex

24

Ex:

https://wiki.opnfv.org/list_of_t

empest_smoke_tests

Functest integrated with Octopus CI4 pharos labs connected incl Ericsson, Orange

Yardstick : Infrastructure Verificationhttps://wiki.opnfv.org/yardstick

The goal of the Yardstick Project is to verify the infrastructure compliance when running VNF applications.

NFV Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific requirements and complex configuration on the underlying infrastructure and test tools.The Yardstick concept decomposes typical VNF work-load performance metrics into a number of characteristics/performance vectors, which each of them can be represented by distinct test-cases.

The project's scope is to develop a test framework, test cases and test stimuli. The methodology used by the Project, to verify the infrastructure form the perspective of a VNF, shall be aligned with ETSI TST001.

25

Yardstick test cases

26

TST001, Clause 6

Yardstick test metrics

27Leverages ETSI NFV PER001

Brahmaputra testing

28

Test coverage

29

OPNFV Test Coverage & ETSI NFV

30

Brahmaputra release

31

Qtip: Platform performance benchmarkinghttps://wiki.opnfv.org/platform_performance_benchmarking

Qtip is a performance benchmark suite for OPNFV platform

QTIP aims to benchmark OPNFV platforms through a "Bottom up" approach, testing baremetal components first and then VM, and comparing baremetal results with VM results.

32

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Thank you