Anuket Testing Overview¶
Introduction¶
Testing is one of the key activities in Anuket and includes unit, feature, component, system level testing for development, automated deployment, performance characterization and stress testing.
Test projects are dedicated to provide frameworks, tooling and test-cases categorized as functional, performance or compliance testing. Test projects fulfill different roles such as verifying VIM functionality, benchmarking components and platforms or analysis of measured KPIs for Anuket release scenarios.
Feature projects also provide their own test suites that either run independently or within a test project.
This document details the Anuket testing ecosystem, describes common test components used by individual Anuket projects and provides links to project specific documentation.
The Anuket Testing Ecosystem¶
The Anuket testing projects are represented in the following diagram:
The major testing projects are described in the table below:
Project |
Description |
|---|---|
Functest |
This project deals with the functional testing of the VIM and NFVI. It leverages several upstream test suites (OpenStack, ODL, ONOS, etc.) and can be used by feature project to launch feature test suites in CI/CD. The project is used for scenario validation. |
ViNePerf |
ViNePerf provides an automated test-framework and comprehensive test suite based on industry standards for measuring the data-plane performance in different cloud environments. Dataplane in a cloud includes different switching technologies with physical and virtual network interfaces, and carries traffic to and from workloads running as virtual-machines and containers. The architecture of ViNePerf is agnostic of cloud-type, switching-technology, and traffic-generator. ViNePerf allows user to customize the test-cases, network-topology, workload-deployment, hardware- configuration, and the versions of the software components such vswitch, vnf, cnf, cni, etc. ViNePerf can be used both pre-deployment and post-deployment of the cloud. Though ViNePerf architecture is designed for evaluation of dataplane of clouds in Lab environments, it can also be in production clouds. ViNePerf methods follows standards developed by the IETF and ETSI NFV, and contribute to the development of new standards. |
Testing Working Group Resources¶
Test Results Collection Framework¶
Any test project running in the global Anuket lab infrastructure and is integrated with Anuket CI can push test results to the community Test Database using a common Test API. This database can be used to track the evolution of testing and analyse test runs to compare results across installers, scenarios and between technically and geographically diverse hardware environments.
Results from the databse are used to generate a dashboard with the current test status for each testing project. Please note that you can also deploy the Test Database and Test API locally in your own environment.
Overall Test Architecture¶
The management of test results can be summarized as follows:
+-------------+ +-------------+ +-------------+
| | | | | |
| Test | | Test | | Test |
| Project #1 | | Project #2 | | Project #N |
| | | | | |
+-------------+ +-------------+ +-------------+
| | |
V V V
+---------------------------------------------+
| |
| Test Rest API front end |
| http://testresults.opnfv.org/test |
| |
+---------------------------------------------+
^ | ^
| V |
| +-------------------------+ |
| | | |
| | Test Results DB | |
| | Mongo DB | |
| | | |
| +-------------------------+ |
| |
| |
+----------------------+ +----------------------+
| | | |
| Testing Dashboards | | Test Landing page |
| | | |
+----------------------+ +----------------------+
The Test Database¶
A Mongo DB Database was introduced for the Brahmaputra release. The following collections are declared in this database:
pods: the list of pods used for production CI
projects: the list of projects providing test cases
test cases: the test cases related to a given project
results: the results of the test cases
scenarios: the Anuket scenarios tested in CI
This database can be used by any project through the Test API. Please note that projects may also use additional databases. The Test Database is mainly use to collect CI test results and generate scenario trust indicators. The Test Database is also cloned for Anuket Plugfests in order to provide a private datastore only accessible to Plugfest participants.
Test API description¶
The Test API is used to declare pods, projects, test cases and test results. Pods correspond to a cluster of machines (3 controller and 2 compute nodes in HA mode) used to run the tests and are defined in the Pharos project. The results pushed in the database are related to pods, projects and test cases. Trying to push results generated from a non-referenced pod will return an error message by the Test API.
- The data model is very basic, 5 objects are available:
Pods
Projects
Test cases
Results
Scenarios
For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html
The code of the Test API is hosted in the releng-testresults repository [TST2]. The static documentation of the Test API can be found at [TST3]. The Test API has been dockerized and may be installed locally in your lab.
The deployment of the Test API has been automated. A jenkins job manages:
the unit tests of the Test API
the creation of a new docker file
the deployment of the new Test API
the archive of the old Test API
the backup of the Mongo DB
Test Project Reporting¶
The reporting page for the test projects is http://testresults.opnfv.org/reporting/
This page provides reporting per Anuket release and per testing project.
An evolution of the reporting page is planned to unify test reporting by creating a landing page that shows the scenario status in one glance (this information was previously consolidated manually on a wiki page). The landing page will be displayed per scenario and show:
the status of the deployment
the score from each test suite. There is no overall score, it is determined by each test project.
a trust indicator
Test Case Catalog¶
Until the Colorado release, each testing project managed the list of its test cases. This made it very hard to have a global view of the available test cases from the different test projects. A common view was possible through the API but it was not very user friendly. Test cases per project may be listed by calling:
http://testresults.opnfv.org/test/api/v1/projects/<project_name>/cases
with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick
A test case catalog has now been realized [TST4]. Roll over the project then click to get the list of test cases, and then click on the case to get more details.
Test Dashboards¶
The Test Dashboard is used to provide a consistent view of the results collected in CI. The results shown on the dashboard are post processed from the Database, which only contains raw results. The dashboard can be used in addition to the reporting page (high level view) to allow the creation of specific graphs according to what the test owner wants to show.
Reference Documentation¶
[TST1]: Anuket web site
[TST2]: TestAPI code repository link in releng-testresults
[TST3]: TestAPI autogenerated documentation
[TST4]: Testcase catalog