Opened 8 years ago Last modified 8 years ago #1463 new defectIllegal instructions on MS Visual Studio 2015 Update 3
Description
When encoding/decoding with a release build on MS Visual Studio 2015 Update 3, an "Illegal Instruction Error" is produced in "partialButterflyInverse32".
This looks like a problem with the optimizer of Visual Studio's C++ compiler. The debug version works fine.
The bug could be reported to MS, but it should probably be reduced to a very simple example code that could easily be analyzed by the developers (if possible).
The current suggested solution is to use a different compiler as a workaround and wait for the next VS update. There seems to be at least one existing bug report for VS referring to "illegal instructions" Change History (2)comment:1 Changed 8 years ago by pevsikovcomment:2 Changed 8 years ago by ksuehring
Thanks for the suggestion.
I have added the option /d2Qvec-sse2only for now to work around the problem (r4857).
I will keep the ticket open as a reminder to revert it, if a patch becomes available for MSVC 2015. Note: See
TracTickets for help on using
tickets. | This list contains all users that will be notified about changes made to this ticket. These roles will be notified: Reporter, Owner, Subscriber, Participant
|
This bug occurs only in PartialButterflyInverse32x32 and PartialButterflyInverse16x16 functions.
MSVC 2015 Update 3 C++ compiler tries to make optimized path for sse4.2 instruction set. During this operation, it optimizes away intermediate arrays O[], EO[] and EEO[], which results in faulty LEA instructions in assembly code.
I found two workarounds for this problem.
1) use '/d2Qvec-sse2only' undocumented compiler flag for TComTrQuant.cpp file compilation.
2) declare O[], EO[] and EEO[] variables with 'volatile' modifier
or use MSVC 2015 Update 2 C++ compiler or earlier.