1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
|
<html devsite>
<head>
<title>VTS Dashboard UI</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>
The VTS Dashboard provides a cohesive user interface that uses material design
to effectively display information about test results, profiling, and coverage.
Dashboard styling uses open-source Javascript libraries including Materialize
CSS and jQueryUI to process data delivered by Java servlets in Google App
Engine.
</p>
<h2>Dashboard home</h2>
<p>
The Dashboard home page displays a list of test suites a user has added to
favorites.
</p>
<img src="../images/treble_vts_ui_main.png" title="VTS Dashboard landing page">
<figcaption><strong>Figure 1.</strong> VTS Dashboard, home page.</figcaption>
<p>
From this list, users can:
</p>
<ul>
<li>Select a test suite to view results for that suite.
<li>Click <strong>SHOW ALL</strong> to view all VTS test names.
<li>Select the <strong>Edit</strong> icon to modify the Favorites list.
<img src="../images/treble_vts_ui_favorites.png" title="VTS Dashboard favorites">
<figcaption><strong>Figure 2.</strong> VTS Dashboard, editing Favorites
page.</figcaption></li>
</ul>
<h2 id=test-results>Test results</h2>
<p>
Test Results displays the latest information about the selected test suite,
including a list of profiling points, a table of test case results in
chronological order, and a pie chart displaying the result breakdown of the
latest run (users can load older data by paging right).
</p>
<img src="../images/treble_vts_ui_results.png" title="VTS Dashboard results">
<figcaption><strong>Figure 3.</strong> VTS Dashboard, test results.</figcaption>
<p>
Users can filter data using queries or by modifying the test type (pre-submit,
post-submit, or both). Search queries support general tokens and field-specific
qualifiers; supported search fields are: device build ID, branch, target name,
device name, and test build ID. These are specified in the format:
<var>FIELD-ID</var>="<var>SEARCH QUERY</var>". Quotes are used to treat multiple
words as a single token to match with the data in the columns.
</p>
<h2 id=profiling>Data profiling</h2>
<p>
Users can select a profiling point to reach an interactive view of the
quantitative data for that point in a <strong>line graph</strong> or
<strong>histogram</strong> (examples below). By default, the view displays the
latest information; users can use the date picker to load specific time windows.
</p>
<img src="../images/treble_vts_ui_performance.png" title="VTS Dashboard performance">
<figcaption><strong>Figure 4.</strong> VTS Dashboard, line graph performance.
</figcaption>
<p>
Line graphs display data from a collection of unordered performance values,
which can be useful when a test of performance produces a vector of performance
values that vary as a function of another variable (e.g., throughput versus
message size).
</p>
<img src="../images/treble_vts_ui_histogram.png" title="VTS Dashboard histogram">
<figcaption><strong>Figure 5.</strong> VTS Dashboard, histogram performance.</figcaption>
<h2 id=coverage>Test coverage</h2>
<p>
Users can view coverage information from the coverage percent link in test
results.
<img src="../images/treble_vts_ui_coverage.png" title="VTS Dashboard coverage">
<figcaption>
<strong>Figure 6.</strong> VTS Dashboard, coverage percentages.</figcaption>
<p>
For each test case and source file, users can view an expandable element
containing color-coded source code according to the coverage provided by the
selected test:
</p>
<img src="../images/treble_vts_ui_coverage_source.png" title="VTS Dashboard coverage_source">
<figcaption>
<strong>Figure 7.</strong> VTS Dashboard, coverage source code.</figcaption>
<ul>
<li>Uncovered lines are highlighted <font style="color:red">red</font>.</li>
<li>Covered lines are highlighted <font style="color:green">green</font>.</li>
<li>Non-executable lines are <strong>uncolored</strong>.</li>
</ul>
<p>
Coverage information is grouped depending into sections depending on how it was
provided at run-time. Tests may upload coverage:
</p>
<ul>
<li><strong>Per function</strong>. Section headers have the format "Coverage:
<var>FUNCTION-NAME</var>".</li>
<li><strong>In Total</strong> (provided at the end of the test run). Only one
header is present: "Coverage: All".</li>
</ul>
<p>
The Dashboard fetches source code client-side from a server, which uses the
open-source
<a href="https://gerrit-review.googlesource.com/Documentation/rest-api.html">Gerrit
REST API</a>.
</p>
<h2 id=monitor>Monitoring & testing</h2>
<p>
The VTS Dashboard provides the following monitors and unit tests.
</p>
<ul>
<li><strong>Test email alerts</strong>. Alerts are configured in a Cron job that
executes at a fixed interval of two (2) minutes. The job reads the VTS status
table to determine if new data has been uploaded to each table, done by checking
the test's raw data upload timestamp is newer than the last status update
timestamp. If the upload timestamp is newer, the job queries for new data
between now and the last raw data upload. New test case failures, continued test
case failures, transient test case failures, test case fixes, an inactive tests
are determined; this information is then sent in email format to the subscribers
of each test.</li>
<li><strong>Web service health</strong>. Google Stackdriver integrates with
Google App Engine to provide easy monitoring of the VTS Dashboard. Simple uptime
checks verify pages can be accessed while other tests can be created to verify
latency on each page, servlet, or database. These checks ensure the Dashboard is
always accessible (else an administrator will be notified).</li>
<li><strong>Analytics</strong>. Each page on the VTS Dashboard supports
integration with Google Cloud Analytics, provided that an Analytics ID is
specified in the configuration. This provides more robust analysis of page
usage, user interaction, locality, session statistics, etc. By simply providing
the valid ID in the pom.xml file, analytics are automatically
provided.</li>
</ul>
</body>
</html>
|
