Testing

Testing

July 31, 2017

Testing

References

Automated Testing Pyramid

Load and Stress Test

A load test, by definition, measures the performance of a system under an expected load. In contrast, a stress test overloads a system in order to find the breaking point.

A load test is performed in order to ensure that a website or web application is capable of handling specific numbers of users at once. Load Testing is often used in the process of capacity planning, to ensure that a system is able to handle growth to specified levels of simultaneous traffic.

A stress test is used to specifically push a system beyond its intended capacity to identify components that begin to slow down, identify bottlenecks in the system, and bring to light possible points of failure.

jmeter - Apache JMeter to load test functional behavior and measure performance. One JMeter instance runs for max 800 users.
loader.io - Loader.io is a FREE load testing service that allows you to stress test your web-apps & apis with thousands of concurrent connections.
ab - Apache HTTP server benchmarking tool
BlazeMeter - Continuous Testing platform. Distributed JMeter for stress test. Uses JMeter scripts. Since it is distributed runs multiple JMeter instance.

Example ab testing for 100 requests with 10 concurrency (clients): We make 100 requests to [url] with 10 concurrency (we can think of it as 10 users making requests at the same time) as follows.

Copy

1

$ ab -k -n 100 -c 10 [url]

Example ab testing for 10 clients 1000 times added header:

Copy

1

$ ab -k -n 1000 -c 10 -H “Authorization: Bearer <…token…>” http://api_url/

Usually what you get per service at least 4000 transactions per second. 4000-7000 per second per service.

What it means in practice in real world is that application service scalability is never the issue, the problem is always the database.

If the database is not scaled, the more application servers you will start upfront everyone will meet the backend and this usually probably causes problems.

If there is huge number of failed request issue, it is due to dynamic pages. It happens because the Content-Length can vary between the requests. When using ab with such pages you need to use the -l option.

Copy

1

-l      Accept variable document length (use this for dynamic pages)

C10K Problem

The C10K problem

example test can be done with

Copy

1

$ ab -n 10000 -c 16 -l https://domain.com:443/index.html

This runs 10.000 HTTP requests with a concurrency of 16.

Example test results:

Copy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


Concurrency Level:      16
Time taken for tests:   403.432 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      131810000 bytes
HTML transferred:       129120000 bytes
Requests per second:    24.79 [#/sec] (mean)
Time per request:       645.492 [ms] (mean)
Time per request:       40.343 [ms] (mean, across all concurrent requests)
Transfer rate:          319.06 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      436  487  66.4    477    1854
Processing:   137  157  65.0    149    2022
Waiting:      136  150  34.4    146    1168
Total:        576  644 106.7    627    2822

Percentage of the requests served within a certain time (ms)
  50%    627
  66%    641
  75%    652
  80%    656
  90%    669
  95%    716
  98%    841
  99%   1012
 100%   2822 (longest request)

Code Coverage

JaCoCo - Java Code Coverage Library

Copy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version></version>
  <executions>
    <execution>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>
    <execution>
      <id>report</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>report</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Copy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version></version>
  <executions>
    <execution>
      <id>default-prepare-agent</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>
    <execution>
      <id>default-report</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>report</goal>
      </goals>
    </execution>
    <execution>
      <id>default-check</id>
      <goals>
        <goal>check</goal>
      </goals>
      <configuration>
        <rules>
          <rule>
            <!--requires a line coverage minimum of 50% for every class except test classes-->
            <element>CLASS</element>
            <excludes>
              <exclude>*Test</exclude>
            </excludes>
            <limits>
              <limit>
                <counter>LINE</counter>
                <value>COVEREDRATIO</value>
                <minimum>50%</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>