2. Yardstick User Guide

• 2.1. Introduction
  • 2.1.1. About This Document
  • 2.1.2. Contact Yardstick
• 2.2. Methodology
  • 2.2.1. Abstract
  • 2.2.2. ETSI-NFV
  • 2.2.3. Metrics
• 2.3. Architecture
  • 2.3.1. Abstract
  • 2.3.2. Overview
    • 2.3.2.1. Architecture overview
    • 2.3.2.2. Concept
    • 2.3.2.3. Runner types
  • 2.3.3. Use-Case View
  • 2.3.4. Logical View
  • 2.3.5. Process View (Test execution flow)
  • 2.3.6. Deployment View
  • 2.3.7. Yardstick Directory structure
• 2.4. Yardstick Installation
  • 2.4.1. Prerequisites
  • 2.4.2. Install Yardstick using Docker (first option) (recommended)
    • 2.4.2.1. Prepare the Yardstick container
    • 2.4.2.2. If the host is restarted
    • 2.4.2.3. Configure the Yardstick container environment
      • 2.4.2.3.1. Using the CLI command env prepare (first way) (recommended)
      • 2.4.2.3.2. Manually exporting the env variables and initializing OpenStack (second way)
      • 2.4.2.3.3. Automatic initialization of OpenStack (third way)
    • 2.4.2.4. The Yardstick container GUI
    • 2.4.2.5. Delete the Yardstick container
  • 2.4.3. Install Yardstick directly in Ubuntu (second option)
    • 2.4.3.1. Install Yardstick
    • 2.4.3.2. Configure the Yardstick environment (Todo)
    • 2.4.3.3. Uninstall Yardstick
  • 2.4.4. Install Yardstick directly in OpenSUSE
    • 2.4.4.1. Install Yardstick
    • 2.4.4.2. Configure the Yardstick environment
    • 2.4.4.3. Uninstall Yardstick
  • 2.4.5. Verify the installation
  • 2.4.6. Automatic installation of Yardstick
    • 2.4.6.1. Bare metal installation
    • 2.4.6.2. Container installation
    • 2.4.6.3. Parameters for install.yaml
  • 2.4.7. Deploy InfluxDB and Grafana using Docker
    • 2.4.7.1. Automatic deployment of InfluxDB and Grafana containers (recommended)
    • 2.4.7.2. Manual deployment of InfluxDB and Grafana containers
  • 2.4.8. Deploy InfluxDB and Grafana directly in Ubuntu (Todo)
  • 2.4.9. Proxy Support
  • 2.4.10. References
• 2.5. Yardstick Usage
  • 2.5.1. Yardstick common CLI
    • 2.5.1.1. List test cases
    • 2.5.1.2. Show a test case config file
    • 2.5.1.3. Run a Yardstick test case
  • 2.5.2. Run Yardstick in a local environment
  • 2.5.3. Create a new testcase for Yardstick
    • 2.5.3.1. Defining the testcase scenarios
    • 2.5.3.2. Defining the testcase context(s)
      • 2.5.3.2.1. Dummy Context
      • 2.5.3.2.2. Node Context
      • 2.5.3.2.3. Heat Context
      • 2.5.3.2.4. Kubernetes Context
      • 2.5.3.2.5. Using multiple contexts in a testcase
      • 2.5.3.2.6. Reusing a context
  • 2.5.4. Create a test suite for Yardstick
  • 2.5.5. References
• 2.6. Installing a plug-in into Yardstick
  • 2.6.1. Abstract
  • 2.6.2. Installing Storperf into Yardstick
    • 2.6.2.1. Step 0: Environment preparation
    • 2.6.2.2. Step 1: Plug-in configuration file preparation
    • 2.6.2.3. Step 2: Plug-in install/remove scripts preparation
    • 2.6.2.4. Step 3: Install and remove Storperf
      • 2.6.2.4.1. Removing Storperf from Yardstick
• 2.7. Store Other Project’s Test Results in InfluxDB
  • 2.7.1. Abstract
  • 2.7.2. Store Storperf Test Results into Community’s InfluxDB
• 2.8. Grafana dashboard
  • 2.8.1. Abstract
  • 2.8.2. Public access
  • 2.8.3. Testcase dashboard
  • 2.8.4. Administration access
  • 2.8.5. Add a dashboard into Yardstick Grafana
• 2.9. Yardstick Restful API
  • 2.9.1. Abstract
  • 2.9.2. Available API
    • 2.9.2.1. /yardstick/env/action
    • 2.9.2.2. /yardstick/asynctask
    • 2.9.2.3. /yardstick/testcases
    • 2.9.2.4. /yardstick/testcases/release/action
    • 2.9.2.5. /yardstick/testcases/samples/action
    • 2.9.2.6. /yardstick/testcases/<testcase_name>/docs
    • 2.9.2.7. /yardstick/testsuites/action
    • 2.9.2.8. /yardstick/tasks/<task_id>/log
    • 2.9.2.9. /yardstick/results
    • 2.9.2.10. /api/v2/yardstick/openrcs
    • 2.9.2.11. /api/v2/yardstick/openrcs/<openrc_id>
    • 2.9.2.12. /api/v2/yardstick/pods
    • 2.9.2.13. /api/v2/yardstick/pods/<pod_id>
    • 2.9.2.14. /api/v2/yardstick/images
    • 2.9.2.15. /api/v2/yardstick/images/<image_id>
    • 2.9.2.16. /api/v2/yardstick/tasks
    • 2.9.2.17. /api/v2/yardstick/tasks/<task_id>
    • 2.9.2.18. /api/v2/yardstick/testcases
    • 2.9.2.19. /api/v2/yardstick/testcases/<case_name>
    • 2.9.2.20. /api/v2/yardstick/testsuites
    • 2.9.2.21. /api/v2/yardstick/testsuites
    • 2.9.2.22. /api/v2/yardstick/projects
    • 2.9.2.23. /api/v2/yardstick/projects
    • 2.9.2.24. /api/v2/yardstick/containers
    • 2.9.2.25. /api/v2/yardstick/containers/<container_id>
• 2.10. Yardstick User Interface
  • 2.10.1. Commands
  • 2.10.2. Description
• 2.11. Network Services Benchmarking (NSB)
  • 2.11.1. Abstract
  • 2.11.2. Overview
  • 2.11.3. Architecture
    • 2.11.3.1. Components of Network Service
    • 2.11.3.2. KPI Collection
  • 2.11.4. Graphical Overview
    • 2.11.4.1. VNFs supported for chracterization
• 2.12. NSB Installation
  • 2.12.1. Abstract
  • 2.12.2. Prerequisites
    • 2.12.2.1. Hardware & Software Ingredients
  • 2.12.3. Install Yardstick (NSB Testing)
    • 2.12.3.1. Bare Metal context example
    • 2.12.3.2. Standalone context example for Ubuntu 18
  • 2.12.4. System Topology
  • 2.12.5. Environment parameters and credentials
    • 2.12.5.1. Configure yardstick.conf
  • 2.12.6. Run Yardstick - Network Service Testcases
    • 2.12.6.1. NS testing - using yardstick CLI
  • 2.12.7. Network Service Benchmarking - Bare-Metal
    • 2.12.7.1. Bare-Metal Config pod.yaml describing Topology
      • 2.12.7.1.1. Bare-Metal 2-Node setup
      • 2.12.7.1.2. Bare-Metal 3-Node setup - Correlated Traffic
    • 2.12.7.2. Bare-Metal Config pod.yaml
  • 2.12.8. Standalone Virtualization
    • 2.12.8.1. SR-IOV
      • 2.12.8.1.1. SR-IOV Pre-requisites
      • 2.12.8.1.2. SR-IOV Config pod.yaml describing Topology
      • 2.12.8.1.3. SR-IOV 2-Node setup
      • 2.12.8.1.4. SR-IOV 3-Node setup - Correlated Traffic
      • 2.12.8.1.5. SR-IOV Config pod_trex.yaml
      • 2.12.8.1.6. SR-IOV Config host_sriov.yaml
      • 2.12.8.1.7. SRIOV configuration options
    • 2.12.8.2. OVS-DPDK
      • 2.12.8.2.1. OVS-DPDK Pre-requisites
      • 2.12.8.2.2. OVS-DPDK Config pod.yaml describing Topology
      • 2.12.8.2.3. OVS-DPDK 2-Node setup
      • 2.12.8.2.4. OVS-DPDK 3-Node setup - Correlated Traffic
      • 2.12.8.2.5. OVS-DPDK Config pod_trex.yaml
      • 2.12.8.2.6. OVS-DPDK Config host_ovs.yaml
      • 2.12.8.2.7. OVS-DPDK configuration options
  • 2.12.9. OpenStack with SR-IOV support
    • 2.12.9.1. Single node OpenStack with external TG
      • 2.12.9.1.1. Host pre-configuration
      • 2.12.9.1.2. DevStack installation
      • 2.12.9.1.3. TG host configuration
      • 2.12.9.1.4. Run the Sample VNF test case
    • 2.12.9.2. Multi node OpenStack TG and VNF setup (two nodes)
      • 2.12.9.2.1. Controller/Compute pre-configuration
      • 2.12.9.2.2. DevStack configuration
      • 2.12.9.2.3. Run the sample vFW TC
  • 2.12.10. Enabling other Traffic generators
    • 2.12.10.1. IxLoad
    • 2.12.10.2. IxNetwork
  • 2.12.11. Spirent Landslide
• 2.13. Yardstick - NSB Testing - Operation
  • 2.13.1. Abstract
  • 2.13.2. OpenStack Network Configuration
    • 2.13.2.1. Provider networks
    • 2.13.2.2. Heat Topologies
    • 2.13.2.3. Availability zone
  • 2.13.3. Collectd KPIs
  • 2.13.4. Scale-Up
    • 2.13.4.1. Heat
    • 2.13.4.2. Baremetal
  • 2.13.5. Scale-Out
    • 2.13.5.1. Standalone
    • 2.13.5.2. Heat
  • 2.13.6. Traffic Generator tuning
  • 2.13.7. Standalone configuration
    • 2.13.7.1. Emulated machine type
    • 2.13.7.2. Standalone with OVS-DPDK
    • 2.13.7.3. Sample test case file
  • 2.13.8. Preparing test run of vEPC test case
• 2.14. Update Spirent Landslide TG configuration in pod file
• 2.15. Update NSB test case definitions
  • 2.15.1. Preparing test run of vPE test case
  • 2.15.2. Preparing test run of vIPSEC test case
  • 2.15.3. Preparing test run of vCMTS test case
    • 2.15.3.1. Allow SSH access to the docker images
    • 2.15.3.2. Deploy the ConfigMaps for Pktgen and vCMTSd
• 2.16. Yardstick Test Cases
  • 2.16.1. Abstract
  • 2.16.2. Generic NFVI Test Case Descriptions
    • 2.16.2.1. Yardstick Test Case Description TC001
    • 2.16.2.2. Yardstick Test Case Description TC002
    • 2.16.2.3. Yardstick Test Case Description TC004
    • 2.16.2.4. Yardstick Test Case Description TC005
    • 2.16.2.5. Yardstick Test Case Description TC006
    • 2.16.2.6. Yardstick Test Case Description TC008
    • 2.16.2.7. Yardstick Test Case Description TC009
    • 2.16.2.8. Yardstick Test Case Description TC010
    • 2.16.2.9. Yardstick Test Case Description TC011
    • 2.16.2.10. Yardstick Test Case Description TC012
    • 2.16.2.11. Yardstick Test Case Description TC014
    • 2.16.2.12. Yardstick Test Case Description TC015
    • 2.16.2.13. Yardstick Test Case Description TC024
    • 2.16.2.14. Yardstick Test Case Description TC037
    • 2.16.2.15. Yardstick Test Case Description TC038
    • 2.16.2.16. Yardstick Test Case Description TC042
    • 2.16.2.17. Yardstick Test Case Description TC043
    • 2.16.2.18. Yardstick Test Case Description TC044
    • 2.16.2.19. Yardstick Test Case Description TC055
    • 2.16.2.20. Yardstick Test Case Description TC061
    • 2.16.2.21. Yardstick Test Case Description TC063
    • 2.16.2.22. Yardstick Test Case Description TC069
    • 2.16.2.23. Yardstick Test Case Description TC070
    • 2.16.2.24. Yardstick Test Case Description TC071
    • 2.16.2.25. Yardstick Test Case Description TC072
    • 2.16.2.26. Yardstick Test Case Description TC073
    • 2.16.2.27. Yardstick Test Case Description TC074
    • 2.16.2.28. Yardstick Test Case Description TC075
    • 2.16.2.29. Yardstick Test Case Description TC076
    • 2.16.2.30. Yardstick Test Case Description TC078
    • 2.16.2.31. Yardstick Test Case Description TC079
    • 2.16.2.32. Yardstick Test Case Description TC080
    • 2.16.2.33. Yardstick Test Case Description TC081
    • 2.16.2.34. Yardstick Test Case Description TC083
    • 2.16.2.35. Yardstick Test Case Description TC084
  • 2.16.3. OPNFV Feature Test Cases
    • 2.16.3.1. H A
      • 2.16.3.1.1. Yardstick Test Case Description TC019
      • 2.16.3.1.2. Yardstick Test Case Description TC025
      • 2.16.3.1.3. Yardstick Test Case Description TC045
      • 2.16.3.1.4. Yardstick Test Case Description TC046
      • 2.16.3.1.5. Yardstick Test Case Description TC047
      • 2.16.3.1.6. Yardstick Test Case Description TC048
      • 2.16.3.1.7. Yardstick Test Case Description TC049
      • 2.16.3.1.8. Yardstick Test Case Description TC050
      • 2.16.3.1.9. Yardstick Test Case Description TC051
      • 2.16.3.1.10. Yardstick Test Case Description TC052
      • 2.16.3.1.11. Yardstick Test Case Description TC053
      • 2.16.3.1.12. Yardstick Test Case Description TC054
      • 2.16.3.1.13. Yardstick Test Case Description TC056
      • 2.16.3.1.14. Yardstick Test Case Description TC057
      • 2.16.3.1.15. Yardstick Test Case Description TC058
      • 2.16.3.1.16. Yardstick Test Case Description TC087
      • 2.16.3.1.17. Yardstick Test Case Description TC088
      • 2.16.3.1.18. Yardstick Test Case Description TC089
      • 2.16.3.1.19. Yardstick Test Case Description TC090
      • 2.16.3.1.20. Yardstick Test Case Description TC091
      • 2.16.3.1.21. Yardstick Test Case Description TC092
      • 2.16.3.1.22. Yardstick Test Case Description TC093
    • 2.16.3.2. IPv6
      • 2.16.3.2.1. Yardstick Test Case Description TC027
    • 2.16.3.3. KVM
      • 2.16.3.3.1. Yardstick Test Case Description TC028
    • 2.16.3.4. Parser
      • 2.16.3.4.1. Yardstick Test Case Description TC040
    • 2.16.3.5. StorPerf
      • 2.16.2.27. Yardstick Test Case Description TC074
  • 2.16.4. Templates
    • 2.16.4.1. Yardstick Test Case Description TCXXX
    • 2.16.4.2. Task Template Syntax
      • 2.16.4.2.1. Basic template syntax
      • 2.16.4.2.2. Using the default values
      • 2.16.4.2.3. Advanced templates
• 2.17. NSB Sample Test Cases
  • 2.17.1. Abstract
  • 2.17.2. NSB PROX Test Case Descriptions
    • 2.17.2.1. Yardstick Test Case Description: NSB PROX ACL
    • 2.17.2.2. Yardstick Test Case Description: NSB PROX BNG
    • 2.17.2.3. Yardstick Test Case Description: NSB PROX BNG_QoS
    • 2.17.2.4. Yardstick Test Case Description: NSB PROX L2FWD
    • 2.17.2.5. Yardstick Test Case Description: NSB PROX L3FWD
    • 2.17.2.6. Yardstick Test Case Description: NSB PROX MPLS Tagging
    • 2.17.2.7. Yardstick Test Case Description: NSB PROX Packet Buffering
    • 2.17.2.8. Yardstick Test Case Description: NSB PROX Load Balancer
    • 2.17.2.9. Yardstick Test Case Description: NSB PROX VPE
    • 2.17.2.10. Yardstick Test Case Description: NSB PROX LwAFTR
    • 2.17.2.11. Yardstick Test Case Description: NSB EPC DEFAULT BEARER
    • 2.17.2.12. Yardstick Test Case Description: NSB EPC DEDICATED BEARER
    • 2.17.2.13. Yardstick Test Case Description: NSB EPC SAEGW RELOCATION
    • 2.17.2.14. Yardstick Test Case Description: NSB EPC NETWORK SERVICE REQUEST
    • 2.17.2.15. Yardstick Test Case Description: NSB EPC UE SERVICE REQUEST
    • 2.17.2.16. Yardstick Test Case Description: NSB vFW RFC2544
    • 2.17.2.17. Yardstick Test Case Description: NSB vFW RFC2544 (correlated)
    • 2.17.2.18. Yardstick Test Case Description: NSB vFW RFC3511 (HTTP)
    • 2.17.2.19. Yardstick Test Case Description: NSB VPP IPSEC
    • 2.17.2.20. Yardstick Test Case Description: NSB VIMS
    • 2.17.2.21. Yardstick Test Case Description: NSB vCMTS
• 2.18. Glossary
• 2.19. References
  • 2.19.1. OPNFV
  • 2.19.2. References used in Test Cases
  • 2.19.3. Research
  • 2.19.4. Standards