Adding functions to the QGIS toolbox

qgis
osgeo
python
scripting
The graphical modeler is a great way to automate tasks, using the routines in the QGIS processing toolbox. However, what to do if you need a function not available in the toolbox? In that case you may want to look into writing your own script. tags: QGIS processing toolbox, automation
Author

Paulo van Breugel

Published

July 23, 2020

The graphical modeler makes it easy to create models by chaining together different operations/algorithms available in the toolbox. But what if a function is not available in the toolbox? In that case you can write your own script and add this as a new function to the toolbox.

Download the data

For example, I want to create a model to download zipped ascii raster layers, unzip them, convert them to geotif files (if not already) and store them on my computer. The first step, downloading the data, can be automated using the Download function in the toolbox (Figure 1).

Figure 1: The QGIS model to download raster layers.

Extract the rasters

For the next step, unzipping the files, no function is available in the toolbox. So apart from doing this part manually, the option is to write a small script to extract raster layers from a zip file. The tricky part is to write a function that can be used in the QGIS toolbox. This means amongst others that the function’s input parameters need to be defined with the initAlgorithm function and the addParameter function. This is illustrated with the code snippet below, which defines the input and output parameters for the unzip functions.

Code
def initAlgorithm(self, config=None):
    self.addParameter(QgsProcessingParameterFile(
        'INPUT',
        'INPUT', 
        optional=True, 
        behavior=QgsProcessingParameterFile.File, 
        fileFilter='All files (*.*)',
        defaultValue=None))
    self.addParameter(QgsProcessingParameterFile(
        'OUTPUT',
        'OutputFolder', 
        optional=True, 
        behavior=QgsProcessingParameterFile.Folder, 
        fileFilter='All files (*.*)',
        defaultValue=None))

With the input parameters defined, the code snippet below gets a list of the files names in the zipfile, selects the raster layers (based on their file extension) and extracts them.

Code
# Output folder
out_dir = parameters['OUTPUT']
# Input zip file
yzf = parameters['INPUT']
# Temporary download folder
temp_dir = tempfile.mkdtemp()

with zipfile.ZipFile(yzf) as zf:     
    # Get list with file names in zipfile
    files = zf.namelist() 
    # Get list with all raster layers in zip file (ascii and tif rasters)
    lrf = ('.asc', '.tif', '.bil', '.dwg', '.hdr')
    filesasc = [i for i in files if i.endswith(lrf)] 
    # Extract the raster layers to the user defined folder
    for asciifile in filesasc:
        file_path = os.path.join(temp_dir, asciifile)
        f = open(file_path, 'wb')
        f.write(zf.read(asciifile))
        f.close()

Figure 2: The unzip function, with as input the name and location of the zip file, and the output folder.

You can check out the complete script here. Besides the code snippets above, it contains a header section, it loads the required libraries, and it contains some extra (html formatted) information that will be shown as help when running the script from the toolbox. After loading the script in the QGIS toolbox, it can be used like any other function.

Convert the rasters

The translate function can be used to convert raster layers in another format to geotifs. My initial idea was to import the above script in the toolbox, and build a model in which the output of this particular function, the downloaded and unzipped raster file, would be used as input in the translate function. But the downloaded zip files can contain more than one raster file, complicating things. I therefore decided to include the raster conversion part in the script. The code snippet below shows the part where the raster file is converted using the gdal::translate function.

Code
outputfiles = []
for rasterfile in myfiles:
    # If other than geotif, convert raster to tif
    if pathlib.Path(rasterfile).suffix != '.tif':
        # Replace suffix with tif and temp folder path for path output folder
        tmpfilename = pathlib.Path(rasterfile).with_suffix('.tif')
        transfile = os.path.join(out_dir, ntpath.basename(tmpfilename))
        # Translate (convert format)
        alg_params = {
            'COPY_SUBDATASETS': False,
            'DATA_TYPE': 0,
            'EXTRA': '',
            'INPUT': rasterfile,
            'NODATA': None,
            'OPTIONS': '',
            'TARGET_CRS': None,
            'OUTPUT': transfile
        }
        outputs['TranslateConvertFormat'] = processing.run(
            'gdal:translate', 
            alg_params, context=context, 
            feedback=feedback,
            is_child_algorithm=True)

You can find the complete script here. What the script does is to download a zip file, extract the raster(s), and, if not already in geotif format, convert the raster layer(s) to a geotif raster layer(s). The latter is, obviously, skipped, if the downloaded file is already in geotif format. And lastly, the intended output format can be changed by the user. You can load the script in your QGIS toolbox and run it (Figure 3).

Figure 3: The unzip and convert function.

Combining the steps

To create one combined workflow from download to conversion, I created a simple model in which a zip file is downloaded using the Download file function, and the raster layers it contains are extracted and converted to a geotif format using the newly created unzipconvert function (Figure @ref(fig:euZqzHkoDD01)). You can get the model here. Load it in the QGIS toolbox and try it out. Or download the Python script here and load that in your toolbox instead.

Figure 4: The download and unzipconvert functions combined in one workflow.

Further reading

A great source of information is the QGIS manual. Make sure to check it out to learn more about scripting and about how to get started with the graphical modeler.

There is also this template to get you on the way. It is based on some enhancements to the QGIS scripting framework, making it easier to write a script. Wish I had seen that earlier. But alas, something to try out next time.