[Libreoffice] [PATCH] Fix for #i93300#: Crash when inserting 32 bit TIFF image

The sample documents in the bug had two problems: 32 bit sample size
and use of floating point sample format[1]. Neither is supported so this
patch adds checks to reject images that have either of these properties.

Additionally a sanity check is added to make sure that similar crash
(division by zero) cannot be induced by creating a malformed image that
specifies the same values for MaxSampleValue and MinSampleValue.

Some literals have been changed from 1 to 1UL since the result will
be assigned to a variable with ULONG data type. With this change it
was actually possible to load the 32 bit image on a 64 bit system but
the colors were distorted (probably due to unsupported sample format).
So rejecting all 32 bit images still seems to be necessary.

This patch has been tested on a 64 bit Linux system using the samples
in the bug and some images from libtiff sample collection at
ftp://ftp.remotesensing.org/pub/libtiff/pics-3.8.0.tar.gz

[1] http://www.awaresystems.be/imaging/tiff/tifftags/sampleformat.html

Signed-off-by: Harri Pitkänen <hatapitk@iki.fi>
---
 filter/source/graphicfilter/itiff/itiff.cxx |   18 +++++++++++++++---
 1 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/filter/source/graphicfilter/itiff/itiff.cxx 
b/filter/source/graphicfilter/itiff/itiff.cxx
index d2c0b60..cd1678a 100644
--- a/filter/source/graphicfilter/itiff/itiff.cxx
+++ b/filter/source/graphicfilter/itiff/itiff.cxx
@@ -310,6 +310,8 @@ void TIFFReader::ReadTagData( USHORT nTagType, sal_uInt32 nDataLen)
         case 0x0102:   // Bits Per Sample
             nBitsPerSample = ReadIntData();
             OOODEBUG("BitsPerSample",nBitsPerSample);
+            if ( nBitsPerSample >= 32 ) // 32 bit and larger samples are not supported
+                bStatus = FALSE;
             break;
 
         case 0x0103:   // Compression
@@ -462,7 +464,7 @@ void TIFFReader::ReadTagData( USHORT nTagType, sal_uInt32 nDataLen)
         case 0x0140: { // Color Map
             USHORT nVal;
             ULONG i;
-            nNumColors= ( 1 << nBitsPerSample );
+            nNumColors= ( 1UL << nBitsPerSample );
             if ( nDataType == 3 && nNumColors <= 256)
             {
                 pColorMap = new ULONG[ 256 ];
@@ -489,6 +491,13 @@ void TIFFReader::ReadTagData( USHORT nTagType, sal_uInt32 nDataLen)
             OOODEBUG("ColorMap (Anzahl Farben:)", nNumColors);
             break;
         }
+
+        case 0x0153: { // SampleFormat
+            ULONG nSampleFormat = ReadIntData();
+            if ( nSampleFormat == 3 ) // IEEE floating point samples are not supported yet
+                bStatus = FALSE;
+            break;
+        }
     }
 
     if ( pTIFF->GetError() )
@@ -1037,7 +1046,7 @@ void TIFFReader::MakePalCol( void )
             pColorMap = new ULONG[ 256 ];
         if ( nPhotometricInterpretation <= 1 )
         {
-            nNumColors = 1 << nBitsPerSample;
+            nNumColors = 1UL << nBitsPerSample;
             if ( nNumColors > 256 )
                 nNumColors = 256;
             pAcc->SetPaletteEntryCount( (USHORT)nNumColors );
@@ -1238,7 +1247,10 @@ BOOL TIFFReader::ReadTIFF(SvStream & rTIFF, Graphic & rGraphic )
             if ( bStatus )
             {
                 if ( nMaxSampleValue == 0 )
-                    nMaxSampleValue = ( 1 << nBitsPerSample ) - 1;
+                    nMaxSampleValue = ( 1UL << nBitsPerSample ) - 1;
+
+                if ( nMaxSampleValue <= nMinSampleValue )
+                    bStatus = FALSE;
 
                 if ( nPhotometricInterpretation == 2 || nPhotometricInterpretation == 5 || 
nPhotometricInterpretation == 6 )
                     nDstBitsPerPixel = 24;