RESOLVED: Example script and batch processing issue

[gbotes]

Up until 27 Oct 2015, I was experiencing heavy RAM usage and system slowdown when batch processing large numbers of images (100+). Corel was contacted but were not able to reproduce the issue. They suggested I post my script here in the forum and ask for assistance, and LeviFiction kindly tackled the problem and found a solution.

The purpose of this script is to handle the donkey work of my basic image processing. It does some auto-corrections and sharpening, and saves the image in color, black and white, and sepia. IMPORTANT: For this script to work, if anybody is interested in trying it, you need to create a folder "C:\Photography" and then three more sub-folders within it as follows: "1-Colour", "2-Black and White" and "3-Sepia".

I attach a screenshot of how I configure the output settings for batch processing. The script needs to run from your "Scripts-Trusted" folder.

UPDATE: 28 Oct 2015 - thanks to LeviFiction's assistance, my RAM usage issue with batch processing using this script has been resolved. It was traced to FileSaveAs. Simply changing FileSaveAs to FileSave resolved the matter. I've posted a revised script below which is working brilliantly for me, in case other forum members run into a similar issue or can use the script. I'm sure there are better ways to script this, but it will take me some time to learn all I need to know about PSP scripting.

Code: Select all

from PSPApp import *

def ScriptProperties():
    return {
        'Author': u'',
        'Copyright': u'',
        'Description': u'',
        'Host': u'PaintShop Pro',
        'Host Version': u'18.00'
        }

def Do(Environment):
    # EnableOptimizedScriptUndo
    App.Do( Environment, 'EnableOptimizedScriptUndo', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # OneStepPhotoFix
    App.Do( Environment, 'OneStepPhotoFix', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # HighPassSharpen
    App.Do( Environment, 'HighPassSharpen', {
            'BlendMode': App.Constants.BlendMode.SoftLight, 
            'Strength': 70, 
            'Radius': 10, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # FileSave
    App.Do( Environment, 'FileSave', {
            'Encoding': {
                'JPG': {
                    'Variant': App.Constants.JpegFormat.Standard, 
                    'CompressionFactor': 20, 
                    'ChromaSubSampling': App.Constants.ChromaSubSampling.YCC_1x1_1x1_1x1, 
                    'EXIF': True, 
                    'EmbedJPGICC': True
                    }
                }, 
            'FileName': u'C:\\Photography\\1-Colour\\' + App.TargetDocument.Title, 
            'FileFormat': App.Constants.FileFormat.JPG, 
            'FormatDesc': u'JPG JPEG ', 
            'WorkingMode': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.AllAlways, 
                'Version': ((18,0,0),1)
                }, 
            'DefaultProperties': []
            })

    # BlackAndWhiteFilm
    App.Do( Environment, 'BlackAndWhiteFilm', {
            'FilterColor': None, 
            'Strength': None, 
            'Red': 0.3333, 
            'Green': 0.3333, 
            'Blue': 0.3333, 
            'Brightness': 0, 
            'Clarify': 0, 
            'SuggestSettings': False, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # Vignette
    App.Do( Environment, 'Vignette', {
            'Amount': 19, 
            'Fog': -88, 
            'Feather': 25, 
            'Selection': [(0.0789561,0.09),(0.921044,0.87075)], 
            'SelectionMode': 2, 
            'InvertSelection': False, 
            'DiffuseGlow': 10, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # FileSave
    App.Do( Environment, 'FileSave', {
            'Encoding': {
                'JPG': {
                    'Variant': App.Constants.JpegFormat.Standard, 
                    'CompressionFactor': 20, 
                    'ChromaSubSampling': App.Constants.ChromaSubSampling.YCC_1x1_1x1_1x1, 
                    'EXIF': True, 
                    'EmbedJPGICC': True
                    }
                }, 
            'FileName': u'C:\\Photography\\2-Black and White\\' + App.TargetDocument.Title, 
            'FileFormat': App.Constants.FileFormat.JPG, 
            'FormatDesc': u'JPG JPEG ', 
            'WorkingMode': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.AllAlways, 
                'Version': ((18,0,0),1)
                }, 
            'DefaultProperties': []
            })

    # New Raster Layer
    App.Do( Environment, 'NewRasterLayer', {
            'General': {
                'Opacity': 100, 
                'Name': u'Raster 1', 
                'IsVisible': True, 
                'IsTransparencyLocked': False, 
                'LinkSet': 0, 
                'UseHighlight': False, 
                'PaletteHighlightColor': (255,255,64), 
                'GroupLink': True, 
                'BlendMode': App.Constants.BlendMode.Normal
                }, 
            'BlendRanges': {
                'BlendRangeGreen': (0,0,255,255,0,0,255,255), 
                'BlendRangeRed': (0,0,255,255,0,0,255,255), 
                'BlendRangeBlue': (0,0,255,255,0,0,255,255), 
                'BlendRangeGrey': (0,0,255,255,0,0,255,255)
                }, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # Fill
    App.Do( Environment, 'Fill', {
            'BlendMode': App.Constants.BlendMode.Darken, 
            'MatchMode': App.Constants.MatchMode.None, 
            'Material': {
                'Color': (128,80,12), 
                'Pattern': None, 
                'Gradient': None, 
                'Texture': None, 
                'Art': None
                }, 
            'UseForeground': True, 
            'Opacity': 100, 
            'Point': (1590.5,1975.5), 
            'SampleMerged': True, 
            'Tolerance': 20, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

    # MultiObjectProperties
    App.Do( Environment, 'MultiObjectProperties', {
            'General': {
                'Opacity': None, 
                'Name': None, 
                'IsVisible': None, 
                'IsTransparencyLocked': None, 
                'LinkSet': None, 
                'UseHighlight': None, 
                'PaletteHighlightColor': None, 
                'GroupLink': None, 
                'BlendMode': App.Constants.BlendMode.SoftLight
                }, 
            'Effects': None, 
            'WorkingMode': 1, 
            'Path': None, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.AllAlways, 
                'Version': ((18,0,0),1)
                }
            })

    # MultiObjectProperties
    App.Do( Environment, 'MultiObjectProperties', {
            'General': {
                'Opacity': 80, 
                'Name': None, 
                'IsVisible': None, 
                'IsTransparencyLocked': None, 
                'LinkSet': None, 
                'UseHighlight': None, 
                'PaletteHighlightColor': None, 
                'GroupLink': None, 
                'BlendMode': None
                }, 
            'Effects': None, 
            'WorkingMode': 1, 
            'Path': None, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Silent, 
                'AutoActionMode': App.Constants.AutoActionMode.AllAlways, 
                'Version': ((18,0,0),1)
                }
            })

    # FileSave
    App.Do( Environment, 'FileSave', {
            'Encoding': {
                'JPG': {
                    'Variant': App.Constants.JpegFormat.Standard, 
                    'CompressionFactor': 20, 
                    'ChromaSubSampling': App.Constants.ChromaSubSampling.YCC_1x1_1x1_1x1, 
                    'EXIF': True, 
                    'EmbedJPGICC': True
                    }
                }, 
            'FileName': u'C:\\Photography\\3-Sepia\\' + App.TargetDocument.Title, 
            'FileFormat': App.Constants.FileFormat.JPG, 
            'FormatDesc': u'JPG JPEG ', 
            'WorkingMode': 0, 
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.AllAlways, 
                'Version': ((18,0,0),1)
                }, 
            'DefaultProperties': []
            })

    # FileClose - probably not needed, but included just in case!
    App.Do( Environment, 'FileClose', {
            'GeneralSettings': {
                'ExecutionMode': App.Constants.ExecutionMode.Default, 
                'AutoActionMode': App.Constants.AutoActionMode.Match, 
                'Version': ((18,0,0),1)
                }
            })

[LeviFiction]

I can't see anything really wrong with your script. The fact that the script runs is obvious proof of that.

Questions for you: 1) On average how large are the images you're working with? Both in pixels and in size on disk? 2) If you ran a simpler batch on the same set of images, for example only adding a water mark or resizing or only converting to B&W, does it also show huge RAM usage? 3) Does it show huge RAM usage if you let PSP do the saving instead of doing the saving from the script (yes I realize this would require three different batch runs to do the same job) 5) Does it use a lot less RAM if you use smaller batch sizes like 50 images? 4) Did support try your script and your images on their computers or did they just run a batch process and say it didn't do anything to their RAM?

I ran a batch on 92 images once but I was resizing from a small size to an even smaller size for our student database. So they weren't exactly taxing the system at all. And I imagine the result would be different if I had much more standard several MegaPixel images. There is also the question of whether or not the difference is in the fact that the script is doing a lot of the saving. Or their tests did not closely enough match your experience so of course they won't see the problem as it only occurs in high volumes with larger images.

[gbotes]

Thanks, LeviFiction.

On average the images are 4 - 8 MB and about 6000 x 4000 pixels in size. I mostly edit JPG's. Support said they could not get my script to run at all.

I'll have to report back later in the week on your questions 3, 4 and 5 - except to say that smaller batches just means the RAM doesn't max out but the pattern of usage seems the same. I'll test separate batch runs as you suggest, as well.

100 is a small batch of images for me but I've just invested in both the Topaz Labs plugins bundle and ON1 Perfect Photo Premium so I will try to do my batch processing using them instead. Otherwise I'll break up my images into batches of 100, quit PSP and restart and resume.

[LeviFiction]

I got your script to work. I don't know what their problem is.

So I ran a few tests of my own using the information you provided.

First I created 100 images at 6000x4000. On disk they were 6.58MB each.

Tests and results (I can provide video if interested):

Test 1 ) I ran a batch using the script as is. Definite memory leak present as RAM continued to increase until it was using 2GB of data after processing only 10 images. batch cancelled. Memory was never released until I closed PSP.

Test 2 ) I edited the script to remove all 'FileSaveAs' commands and reran the batch. No memory leak present. Ran all 100 images, RAM was at 98MB after the entire batch had completed.

Test 3) I Opened 20 images inside PSP, added a "File Close" command at the very end of the script and brought back the 'FileSaveAs' commands. after running the script 20 times no memory leaks were present. RAM was at 111MB.

Test 4) Turned of all 'FileSaveAs' commands except for the last one. Reran the batch. Same memory leak.

So from my 4 tests we can assume that using a script to save out the results is causing a memory leak of some kind. It's not, however, just using 'FileSaveAs' inside a script and it's not just running a batch process. It's using a script that does 'FileSaveAs' inside a Batch Process.

[gbotes]

Wow! Thank you so much, LeviFiction. I am very grateful for your very able assistance with this problem. I'll try to get your findings to Corel.

I still have lots to learn about scripting. For the most part I script what I think I need as I go. But with what you've told me I'll work on changing my approach. I've replaced FileSaveAs with FileSave and it has worked perfectly, just as you said. Problem solved.