08/23/2017
IMPORTANT SQL UPGRADE PERFORMANCE ADVISORY
SQL 2014 and SQL 2016
The short:
SQL Compatibility over (110) in SQL 2014 or SQL 2016 can very seriously degrade performance on some queries. To avoid this issue, avoid upgrading SQL compatibility beyond SQL 2012 (110) or apply flags as described in more detail below if you need to run at higher compatibility.
Background:
We stumbled on a very significant upgrade concern for those upgrading a database from SQL 2012 and below to SQL 2014 and SQL 2016. We ran into the following issue while staging for an upgrade of Microsoft Dynamics GP 2015. This issue will likely affect MANY ERP users and is not restricted to Dynamics GP. This could affect a good many database applications and probably most ERP applications. Dynamics CRM was specifically cited as potentially affected.
Tested Source System:
10 Year old Windows 2012 server running SQL 2012 with a 15K SAS RAID 1 and RAID 5 Array
Tested Destination System:
Brand new Windows 2016 server running SQL 2016 on a RAID 10 ultra-high performance SSD Array.
Problem / Observation:
An SSRS Report was running unexpectedly slow. The underlying query was identified as running far slower than expected. Query performance on this routinely used query was about 3 seconds on the old system and increased to over 40 seconds on the new system. Given that the new system is substantially higher performance than the old, we would have expected SQL Performance to be much faster on the new system and certainly not dramatically slower (decrease in performance of over 85%).
Trigger:
We spent several days taking steps to isolate the issue. We completed isolation and are able to replicate it. The problem is triggered by increasing the SQL compatibility level. The performance loss occurs when database compatibility level is increased from 2012 (110) to either 2014 (120) or 2016 (130). The new query when running on the new system in a database using SQL 2012 compatibility runs in under 1 second compared to 3 seconds on the old system. Much faster as expected due to hardware improvements. The same database with the same query spikes to over 40 seconds when compatibility is increased to SQL 2014 (120) or SQL 2016 (130).
Per Microsoft Dynamics Support:
Issue Definition: “You are experiencing slow query performance with SQL 2014 and 2016 database compatibility levels with Dynamics GP. I worked with a colleague from the SQL team on a similar case. In that case, the SQL resource noted that beginning with SQL 2014, the query ex*****on plan engine was re-written and will sometimes issue bad plans that cause bottlenecks, etc. In that case, we saw the exact same results where if we changed the compatibility level to SQL 2012 (or earlier) the issue did not occur. From a GP perspective, it is fine to leave the compatibility level at 2012. So if you are okay with that, then you should be able to just continue on with 2012 compatibility level without adverse effects to GP. If you need to investigate what is causing the slowness with 2014/2016 levels, then I will need to send a collaboration request to the SQL team for their assistance with diagnosing the problem.”
Per Microsoft SQL Support:
“When Upgrading server from 2008/2008R2/2012 to 2014/2016 and you usually experience performance issues on certain queries. There is a high probability this is due to the changes in SQL 2014 to the cardinality estimator. We saw this quite a lot with the release of SQL 2014 and it is a common issue as when you migrate to SQL Server 2016 from Pre SQL Server 2014 or earlier versions. Workaround is to setting the database compatibility level back to 110\Sql server 2012. If this fixes the performance issues then it is very likely an issue with the query running slowly with the new CE. This is expected behavior if the query is not written well, You Might need to rewrite or optimize the queries for the new CE if you are experiencing performance issues.
ACTION PLAN:
===============
1. Trace flags: 9481 forces old CE, 2312 forces new CE. You can set it globally with dbcc traceon or server startup parameter. Or you can use querytraceon at query level so that only that query will be impacted
2. Database compatibility level, SQL Server will only use new CE if database compatibility level is 120 or above if the above trace flags are not present.
3. If you have both trace flags and database compatibility level, the follow diagram indicates precedence
4. Details of precedence is documented in this
Per Azure site:
Starting in mid-June 2016, in Azure SQL Database, the default compatibility level are 130 instead of 120 for newly created databases. Existing databases created before mid-June 2016 are not affected, and maintain their current compatibility level (100, 110, or 120). If you want level 130 for your database generally, but you have reason to prefer the level 110 cardinality estimation algorithm, see ALTER DATABASE SCOPED CONFIGURATION (Transact-SQL), and in particular its keyword LEGACY_CARDINALITY_ESTIMATION =ON. For details about how to assess the performance differences of your most important queries, between two compatibility levels on Azure SQL Database, see Improved Query Performance with Compatibility Level 130 in Azure SQL Database.
Use information contained herein at your own risk. Make Backups!