Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Summary

This article describes the methodology and result of performance testing on the Ed-Fi ODS / API v3.2 (specifically, against release v3.2.0).

In brief, performance testing did not uncover any significant concerns in performance relative to versions 3.0.0 and 3.1.0.


Article Contents:

Table of Contents
maxLevel2
excludeSummary

Test Methodology

Volume Tests

Volume testing of v3.2.0 occurred in July and August 2019, using the Locust-based 3.x performance testing framework (an Exchange contribution available in GitHub). This volume test covers the resources and HTTP verbs described in the Ed-Fi SIS vendor certification process. It runs for 30 minutes, spawning 30 clients that run simultaneously to perform tens of thousands of operations.

Data Set

The Northridge v3 data set, which contains 21,628 students.

Azure Test Lab Virtual Machines

The test lab environment used a three-server setup: one each for the database, web applications, and the Locust performance testing. VM "Sizes" listed here, such as "DS11_v2", are Microsoft-defined names for the specs of an Azure VM. Key specs are listed beside these size names. These sizes were chosen as their specs are comparable to those of the Vendor Certification VMs but have SSD disks to more closely match a production environment.

Database VM

Image: Free SQL Server License: SQL Server 2017 Developer on Windows Server 2016
Size: DS11_v2 (Standard, 2 vcpus, 14 GB ram, 6400 max iops, 28GB local SSD)

Web VM

Image: [smalldisk] Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 16GB local SSD)

Test Runner VM

Image: [smalldisk] Windows Server 2016 Datacenter
Size: B2ms (Standard, 2 vcpus, 8 GB ram, 4800 max iops, 16GB local SSD)

Test Results

Experimentation Summary

Testing included running a few variations to check on the impact of using SQL Server 2016 compatibility mode instead of 2014 compatibility mode, and to check on the impact of changing the NHibernate dialect from 2008 to 2012. Finally, the volume tests were executed one more time with change queries turned on.

VersionExecution DateVariation# of RequestsMean Response Time in msMax Response Time in ms
3.0.07/31/2019
124,9851504911
3.1.07/31/2019
128,7171352968
3.2.07/29/2019


128,7891362388
3.2.07/29/2019Restore SQL 2014 Compatibility Mode130,94214217444 (warning) 
Delete section/{id}
3.2.07/29/2019SQL 2014 and restore NHibernate 2008 dialect124,3161552619
3.2.07/30/2019Back to 2016 compatibility mode, with 2008 dialect119,078179

12513 (warning)

Delete gradiginperiod/{1}

3.2.08/13/193.2.0 with change queries enabled129,5661302680

Key Take-Aways

  • No two executions of the same code/configuration will result in the exact same mean response time — there is a degree of randomness in the Locust-based clients. Thus the difference between 130 ms, 135 ms, and 136 ms is not significant.
  • That said, the use of NHibernate's 2008 dialect instead of the (new default) 2012 dialect does appear to have a negative impact. In other words, changing the dialect to 2012 was a good thing.
  • Change queries adds update and delete triggers to these tables. It was hypothesized that these triggers would cause a noticeable slowdown in performance. In fact the performance appears to be better. Most likely this is a random change that is not meaningful. At these volumes, change queries do not appear to cause meaningful performance degradation. However, additional testing with higher volumes of updates and deletes could still turn up some impact.

Server Stats

Overall the web and database server statistics do not show any serious concerns. Database memory was higher for v3.2.0 than versions 3.0 and 3.1, despite reboots between each test execution. However, with the way that SQL Server ties up as much memory as it can, this is not immediately indicative of a problem.

Web Server Stats

In these stats below, the web server CPU load for 3v3.2.0 testing is surprising. Given that we didn't see a clear impact in the test results, it is hard to say that this is ultimately important.

The CPU load on the web server for the canonical v3.2.0 test execution appears to be an anomaly. The following chart compares all five runs on the 3v3.2.0 ODS/API.


Database Server Stats