Opened 8 years ago

Last modified 8 years ago

#1463 new defect

Illegal instructions on MS Visual Studio 2015 Update 3

Reported by: ksuehring Owned by:
Priority: minor Milestone:
Component: HM Version: HM-16.14
Keywords: illegal instruction, transform, inverse Cc: ksuehring, davidf, karlsharman, jct-vc@…

Description

When encoding/decoding with a release build on MS Visual Studio 2015 Update 3, an "Illegal Instruction Error" is produced in "partialButterflyInverse32".

This looks like a problem with the optimizer of Visual Studio's C++ compiler. The debug version works fine.

The bug could be reported to MS, but it should probably be reduced to a very simple example code that could easily be analyzed by the developers (if possible).

The current suggested solution is to use a different compiler as a workaround and wait for the next VS update. There seems to be at least one existing bug report for VS referring to "illegal instructions"

Change History (2)

comment:1 Changed 8 years ago by pevsikov

This bug occurs only in PartialButterflyInverse32x32 and PartialButterflyInverse16x16 functions.

MSVC 2015 Update 3 C++ compiler tries to make optimized path for sse4.2 instruction set. During this operation, it optimizes away intermediate arrays O[], EO[] and EEO[], which results in faulty LEA instructions in assembly code.

I found two workarounds for this problem.
1) use '/d2Qvec-sse2only' undocumented compiler flag for TComTrQuant.cpp file compilation.
2) declare O[], EO[] and EEO[] variables with 'volatile' modifier

or use MSVC 2015 Update 2 C++ compiler or earlier.

comment:2 Changed 8 years ago by ksuehring

Thanks for the suggestion.

I have added the option /d2Qvec-sse2only for now to work around the problem (r4857).

I will keep the ticket open as a reminder to revert it, if a patch becomes available for MSVC 2015.

Note: See TracTickets for help on using tickets.

This list contains all users that will be notified about changes made to this ticket.

These roles will be notified: Reporter, Owner, Subscriber, Participant

  • David Flynn(Subscriber)
  • jct-vc@…(Subscriber)
  • Karl Sharman(Subscriber)
  • karl.sharman@…(Always)
  • Karsten Suehring(Reporter, Subscriber, Participant, Always)
  • Pavel Evsikov(Participant)