From 7bff708e98077e66ae1ab4b7e8e4c676e68eb854 Mon Sep 17 00:00:00 2001 From: Catherine Diep Date: Mon, 18 Jul 2016 16:33:42 -0700 Subject: [PATCH] Specification to mark a test result as "used for verification". This spec defines the mechanism to mark a test as "used for verification". APIs used to mark and un-mark a test are also defined. Change-Id: I1830414e62353974b45555fd8f22aac7acf06ff2 --- .../approved/mark-test-result-as-verified.rst | 314 ++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100755 specs/newton/approved/mark-test-result-as-verified.rst diff --git a/specs/newton/approved/mark-test-result-as-verified.rst b/specs/newton/approved/mark-test-result-as-verified.rst new file mode 100755 index 00000000..7a488427 --- /dev/null +++ b/specs/newton/approved/mark-test-result-as-verified.rst @@ -0,0 +1,314 @@ +==================================== +Mark Test Result as Verified +==================================== + +Launchpad blueprint: + +* https://blueprints.launchpad.net/refstack/+spec/certification-test-record + +This specification defines the processes and mechanisms to mark a test result +as "verified for OpenStack Powered Logo application". + + +Problem description +=================== + +Currently, there is no mechanism to identify test results that are used by +the OpenStack Foundation (OSF) for OpenStack Powered Logo application. At the +present, the test result URL links are sent to the OSF by the vendors per the +instructions described in the "How to Run the Tests" section of the +http://www.openstack.org/brand/interop/ site. + +RefStack should provide a mean for OSF admins to mark a test result as +"verified for OpenStack Powered Logo application". + +RefStack interprets OSF/DefCore requirements as following: + +* The marked test results should be easily identified. +* The marked test results can not be deleted or updated. +* Only OSF admins can mark/umark a test. + +Proposed change +=============== + +* Add a new field named "verification_status" to the "test" table. Detailed + information about this field can be found in the "Data model impact" section. + +* Marking a test result record + + * Only interop admins can make updates to the verification_status field. For + the https://refstack.openstack.org/#/ website, interop admins will be + someone from the OSF. + + * Only a test result which has been shared and associated to a Guideline and + Target Program can be marked as verified. + + * Once a test result is marked as verified, only interop admins can unmark + the test. + + * Test results that are marked as verified cannot be deleted or updated. + + * A new API will be added to manage the verification_status field. Detailed + information about the API can be found in the "REST API impact" section. + +Alternatives +------------ + +Alternatively, we can use the existing test "meta" table to mark a test result. +This can be done by adding a key-value pair to the "meta" table, with key name +as "verification_status". Following are the reasons why this implementation is +not chosen. + +* Filtering/searching for marked data may not be as efficient. +* Marked tests can not be identified from the "test" table. + + +Data model impact +----------------- + +Add a new column named "verification_status" to the existing "test" table. + ++------------------------+-------------+----------+ +| Column | Type | | ++========================+=============+==========+ +| verification_status | int(11) | | ++------------------------+-------------+----------+ + +The verification_status column will store the pre-defined constants (enum) with +descriptive names as follows: + +* 0 = not verified +* 1 = verified + + +REST API impact +--------------- + +The "Update result" REST API will be added to RefStack. The "List results" +and "Show result details" REST APIs will be modified. + +**Update result** + +* Description: + + This API will be used to make updates to a test entry of the "test" table. + Although the test owner, interop or vendor admins should be able to make + updates to a rest record, extra checking must be implemented to ensure that + only interop admins can mark or unmark a test. + + +* Method type: PUT + +* URI: v1/results/{result_id}/ + +* Normal Response Codes: + + * OK (200) + +* Error Response Codes: + + * Bad Request (400) + * Unauthorized (401) + * Not found (404) + * Forbidden (403) + +* Request parameters: + + +---------------+-------+--------------+-----------------------------------+ + | Parameter | Style | Type | Description | + +===============+=======+==============+===================================+ + | result_id | URI | csapi:UUID | Test result ID for marking. | + +---------------+-------+--------------+-----------------------------------+ + + +* JSON schema definition for the body data: + + .. parsed-literal:: + { + { + "verification_status" : 1, + "required": [] + } + } + + **Note** + + * The verification_status column will store a set of pre-defined constants + (enum) with 0 = not verified, 1 = verified, etc. + * Request to update the "verification_status" field will return + "Forbidden(403)" if the requester is not an interop admin. + +* JSON schema definition for the response data: N/A + +**List results** + +* Description: (No update) + +* Method type: GET (No update) + +* URI: v1/results/ (No update) + +* Normal Response Codes: (No update) + +* Error Response Codes: (No update) + +* Request parameters: + + Add the following parameter to the existing ones: + + +---------------------+-------+----------+---------------------------------+ + | Parameter | Style | Type | Description | + +=====================+=======+==========+=================================+ + | verification_status | query | xsd:int | Pre-defined constants. | + | (optional) | | | Not verified = 0 | + | | | | Verified = 1 | + +---------------------+-------+----------+---------------------------------+ + +* JSON schema definition for the body data: N/A + +* JSON schema definition for the response data: + + Update to add verification_status to the response body. + + .. parsed-literal:: + { + pagination: { + current_page: 6, + total_pages: 37 + }, + results: [ + { + url: "https://refstack.openstack.org/#/results/7943e04a-2b95-453c-b627-8a24b2c6faa0", + created_at: "2016-07-25 02:24:34", + meta: { }, + id: "7943e04a-2b95-453c-b627-8a24b2c6faa0", + duration_seconds: 0, + verification_status : 0 + }, + { + url: "https://refstack.openstack.org/#/results/91ae10c5-ecf5-4823-81d4-09836dc212cf", + created_at: "2016-07-13 18:37:53", + meta: { + shared: ""true"", + target: "compute", + guideline: "2016.01.json" + }, + id: "91ae10c5-ecf5-4823-81d4-09836dc212cf", + duration_seconds: 6037, + verification_status : 1 + }, + ........ + ] + } + +**Show result details** + +* Description: (No update) + +* Method type: GET (No update) + +* URI: v1/results/{result_id} (No update) + +* Normal Response Codes: (No update) + +* Error Response Codes: (No update) + +* Request parameters: (No update) + +* JSON schema definition for the body data: N/A + +* JSON schema definition for the response data: + + Update to add verification_status to the response body. + + .. parsed-literal:: + { + user_role: "user", + created_at: "2016-07-13 18:37:53", + meta: { + shared: ""true"", + target: "compute", + guideline: "2016.01.json" + }, + id: "91ae10c5-ecf5-4823-81d4-09836dc212cf", + duration_seconds: 6037, + verification_status : 1, + results: [ + "tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate", + "tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_get_root_certificate", + ...... + ] + } + +Security impact +--------------- + +None. + +Notifications impact +-------------------- + +None. + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +None + +Other deployer impact +--------------------- + +None + +Developer impact +---------------- + +None + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Paul Van Eck + Andrey Pavlov + +Other contributors: + + +Work Items +---------- + +* Add the defined additional field to the "test" table. +* Develop business and UI code to enable marking a test result. + + +Dependencies +============ + +None + + +Testing +======= + +* Add unit tests to test the newly added code. + + +Documentation Impact +==================== + +None + + +References +========== + +None