Trifork Blog

Scaling images - Quirks and tricks

August 6th, 2014 by
| Reply

In one of our projects we created functionality to upload images. These images can, after being uploaded, be viewed on a grid or on  a detail page. The grid shows a thumbnail and the detail page shows a medium variant of the image. To create these variants we have used java ImageIO, imgscalr, ImageMagick (im4java) and Exiftool. Sounds like quite a few libraries to create two different sized images, but it’s the result of trying to support as many images as possible and provide a good user experience. In this blog post I will explain how and why we have used these libraries to solve the problems we encountered.

Process flow (Click for larger view)

In our application we allow the user to upload the following file types: *.jp(e)g, *.png, *.tif(f).  For .png and .tiff files we have to do some extra steps before we can scale the image to a thumbnail and medium size variant. I will elaborate on that part later on. As you can see in the flow diagram, we store the full size image in the database. After that an event is started to create an asynchronous process. First lets take a look at how we handle .jp(e)g files.  We create a BufferedImage from our source file, which is a Path in our case:

BufferedImage bufferedImage =  ImageIO.read(source.toFile());

Well that was really easy, but with this one-liner we already discovered some problems during the process of testing some images. Not all images (no matter what content type, even if there is a reader for that content type) could be read by ImageIO.  For example, one of the images contained a color code that is not supported. Instead of trying to support all kind of scenarios and the risk of missing one, we decided added a fallback that we convert the image to jpg file with ImageMagick. Then the with converted image we try to read the image again with ImageIO.

private BufferedImage createBufferedImage(Path source)
throws IOException {
  try {
    // First try to create bufferedImage with original source
    return ImageIO.read(source.toFile());
  } catch (IOException e) {
    // If failed to create bufferedImage with original source,
    // try to create bufferedImage with converted jpg.
    // If that also fails, we can not handle the image and
    // the exception needs to be handled.
    Path imageWithoutExifMetadata =
            copyImageWithoutExifMetadataFromImage(source);
    Path convertedSource = convertToJpg(imageWithoutExifMetadata);
    BufferedImage bufferedImage =
            ImageIO.read(convertedSource.toFile());

    cleanupTemporaryConvertFiles(
            convertedSource, imageWithoutExifMetadata);

    return bufferedImage;
  }
}

Now that we have a BufferedImage we can use this to scale the image with imgscalr.

BufferedImage thumbnailed = Scalr.resize(bufferedImage,
        jpgImageQuality, Scalr.Mode.AUTOMATIC, newImageSize);

When the image is scaled we need to write the image:

ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageOutputStream ios = ImageIO.createImageOutputStream(baos);
Iterator<ImageWriter> iter = ImageIO.getImageWritersByFormatName("jpeg");
ImageWriter writer = iter.next();
ImageWriteParam iwp = writer.getDefaultWriteParam();
iwp.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
iwp.setCompressionQuality(jpgImageCompression);
if (iwp.canWriteProgressive() && originalFileSize > TEN_KB) {
  iwp.setProgressiveMode(ImageWriteParam.MODE_DEFAULT);
}
writer.setOutput(ios);
writer.write(null, new IIOImage(thumbnailed, null, null), iwp);
writer.dispose();

baos.flush();
byte[] imageInByte = baos.toByteArray();
baos.close();

As I said before we convert the image with ImageMagick in case we can not read it properly. Also here we added a few optimisations to convert the most images. First we have added the quiet mode option to ignore some of the warnings that made the conversion fail.

Sometimes images have multiple layers and ImageMagick will convert an image for each of those layers. Because we only need one image that we can use in our process, we always ask for the first layer. This can be done by adding [0] to the filename.

ConvertCmd cmd = new ConvertCmd();
cmd.setSearchPath(pathToImageMagick);

IMOperation op = new IMOperation();

// We addded the quiet mode to ignore some warnings which made the conversion failed
op.quiet();
op.colorspace("sRGB");
//Take the first layer ([0]) of the image to convert
op.addImage(imageFile.toAbsolutePath().toString() + "[0]");
op.addImage(imageFile.toAbsolutePath().toString() + JPG_EXTENSION);

cmd.run(op);

Tiff files

Tiff files can not be read by ImageIO, because there is no reader available. So each tiff file is converted to a jpg file using ImageMagick. This works fine, except for a few images we tried. These images failed to convert because of a bug in the underlying library libtif. After some heavy research I found that the conversion failed because the metadata could not be read properly.  The best way to solve this problem was to upgrade the libtif library (as suggested on the ImageMagick forum), but that was a bit of a problem on our operating system. Therefore we decided to remove the metadata with Exiftool from the tiff file and convert the image without the metadata. The converted image is then used for creating the thumbnail and medium variant.

private Path copyImageWithoutExifMetadataFromImage(Path imageFile) {
  Path tempImage;

  try {
    tempImage = Files.createFile(
            buildImageWithoutMetadataFilePath(imageFile));
    Files.copy(imageFile, tempImage,
            StandardCopyOption.REPLACE_EXISTING);

    metadataWriter.deleteExifMetadata(tempImage);
  } catch (IOException e) {
    logger.error("Failed to delete exif metadata from image " +
            imageFile.toAbsolutePath().toString(), e);
    throw new ImageScalingException(e);
  }
  return tempImage;
}

We now have a process in place that creates image variants for probably all our images. This gives us a solid base to upload the images and don't bother the users with upload exceptions. Hopefully this process helps some of you in scaling, converting or creating images.

3 Responses

  1. August 6, 2014 at 20:35 by jelmer Kuperus

    1. Have you considered using GraphicsMagick instead of ImageMagick it's what we use at mp. Its also used by sites like flickr and etsy

    2. Rather than creating lots of temp files (I think this is what convertToJpg does) have you considered streaming ?

    Eg with graphicsmagick you can do
    gm convert -background white -resize 1024x1024> -flatten - jpeg:-

    Fire that off in a process and read from the input stream of the java.lang.Process. No temp files needed

    3. Are you checking the size of the image before reading it as a BufferedImage because else its really easy for a malicious user to trigger an outofmemory exception

  2. August 7, 2014 at 08:09 by Roberto van der Linden

    Hey Jelmer,

    1. No ;) but maybe we should ;)

    2. This code is not the latest version, which does use more streaming, but this method does not indeed. It starts an ImageMagick process.

    3. We do not check it as we have a maximum upload size. I agree that it still would be a good check to add :)

  3. August 15, 2014 at 11:40 by Frank Scholten

    Note you can also use the AsyncScalr from imgscalr to scale on multiple cores. We are using that right now in our project.

Leave a Reply