Improving Nextcloud's Thumbnail Response Time

(updated )

I quite like Nextcloud as a self-hosted alternative to Dropbox - it works well for making documents and password databases available between machines.

Where photos are concerned, the functionality it includes offers a lot of promise but is unfortunately let down by a major failing - a logical yet somewhat insane approach to thumbnail/preview generation.

The result is that tools like "Photo Gallery" are rendered unusable due to excessively long load times. It gets particularly slow if you switch to a new client device with a different resolution to whatever you were using previously (even switching between the Android app and browser view can be problematic).

There's a Nextcloud App called previewgenerator which can help with this a little by pre-generating thumbnails (rather than waiting for a client to request them), but it's out of the box config doesn't help much if, like me, your photos are exposed via "External Storage" functionality rather than in your Nextcloud shared folder. Even when photos are in your shared folder, the app's out of the box config will result in high CPU usage and extremely high storage use.

This documentation details how to tweak/tune things to get image previews loading quickly. It assumes you've already installed and configured Nextcloud. All commands (and crons) should be run/created as the user that Nextcloud runs as.

 

The Problem

Obviously when viewing the gallery, we don't want Nextcloud sending full images across (at >5MB a go, it'd take a while), so it instead generates thumbnails so that much smaller files can be sent instead.

This in itself is fine, but unfortunately, the Nextcloud devs seem to have let perfect be the enemy of good and have implemented a scheme which tries to show you the very best thumbnail it can, with huge flexibility in thumbnail sizes.

Depending on where in Nextcloud you're viewing the image there are a range of thumbnail sizes which might be used (the following are just for viewing in a browser)

  • 32x32 preview - Icon in the files list
  • 4096x4096 - max preview. The image you're shown when you click to open the image
  • 400x200 - Gallery app
  • 384x256 - Gallery app
  • 256x384 - Gallery app (landscape screen)
  • 200x200 - Gallery app
  • 256x171 - Gallery App
  • 171x256 - Gallery App (landscape screen)
  • 250x250 - File app, Gridview
  • 256x256 - File app, Gridview

If we then factor in mobile devices, we get more resolutions. The Nextcloud app on my Android phone requests

  • 64x64 - preview icon in files list
  • 1080x2030
  • 2030x1080 - (when holding phone in landscape)

Other devices with a different screen resolution will request other sizes too, so even this list is far from exhaustive. If you look in your HTTP server's access logs when viewing an image you can tell the resolution in use as URL's take one of two forms

# This is requesting 1080x2030
"GET /index.php/core/preview.png?file=%2FPhotos%2FCamera%2F%2F2019%2F09%2FIMG_20190903_074810.jpg&x=1080&y=2030&a=1&mode=cover&forceIcon=0 HTTP/1.1"

# This ones 256x256
"GET /index.php/apps/files/api/v1/thumbnail/256/256/Photos/Camera/2019/07/IMG_20190711_130915.jpg HTTP/1.1"

Nextcloud's default behaviour is to generate the requested resolution as and when it's first requested, and then cache the thumbnail to disk (ready for the next time it's needed). Nextcloud does do a little bit of normalisation, rounding some obscure sizes up to something it already has, but there's still a lot of scope for variance.

The problem is, because of the vast array of possible resolutions, when trying to view images in Nextcloud you spend what feels like an inordinate amount of time waiting for previews to generate. 2-3 seconds per image isn't uncommon (we've occasionally generated 6-700 photos a day whilst on holiday - you can imagine the time it takes for those thumbnails to load).

Factor in that the thumbnail generation is single-threaded, and you can begin to see just why there are so many complaints about Nextcloud's gallery being slow.

This is exacerbated on the Android app, because you essentially need to wait for 2 thumbnails to load - the smaller 64x64 image, and then the 1080x2030 image once you tap in to view an image.

Nextcloud's android app lets you swipe through photos, but in practice it's unusable because of these long load times.

 

PreviewGenerator

The app previewgenerator is fantastically helpful, so much so that you wonder why it hasn't been absorbed into Nextcloud's core.

It works by installing a plugin to track when changes are made, and then a regular cronjob (mine's set for every 15 minutes) works over the recorded changes looking for images. Any image it finds it triggers the thumbnail generation routines, causing thumbnails to be written to disk.

That way when you come to try and view the images, the thumbnails simply need to be served from disk rather than generated with libgd.

But, for users with a lot of photographs there are multiple headaches involved in the default config.

To install it, either browse to the app store using Nextcloud's admin user, or clone it's repo into the apps directory

cd /var/www/nextcloud/apps
git clone https://github.com/rullzer/previewgenerator.git

 

MySQL

The first thing that a lot of users will run into, is that previewgenerator takes a long time to run. People have complained of it taking weeks.

A big part of the bottleneck relates to the way Nextcloud writes thumbnail details into the database. Each thumbnail is recorded in a separate transaction. Because the underlying tables use InnoDB, a lot of time is then spent waiting for the transaction log to be flushed to disk.

So, the first significant improvement which can be made is to configure MySQL not to wait for the changes to be be written to disk. We do this by adding the following in /etc/mysql/my.cnf within the [[mysqld]] section

sync_binlog = 10
innodb_flush_log_at_trx_commit = 2

What this does is

  • innodb_flush_log_at_trx_commit set to 2 the log writes will instead happen about once per second. This does mean you have a chance of data-loss/db corruption if power to your server is interrupted.
  • sync_binlog - the change to 10 should be viewed as a temporary change while you get the initial run complete. It tells MySQL to buffer 10 log writes before flushing to disk. This should be set back to 0 or 1 for ongoing use.

Essentially this change reduces the overhead of each INSERT/UPDATE so that our preview generation run can run without MySQL waiting on disk quite so much. Once the changes are made, restart MySQL to get the changes live

systemctl restart mysqld

 

Preview Sizing

As we noted earlier, Nextcloud's default approach of having many preview sizes is quite problematic. 

Preview Generator will generate every preview size from 32x32 up to the max size. Some of those previews will never actually be requested so we're essentially wasting both CPU cycles and disk capacity.

So, to address this we're going to do 2 things

  • Configure a maximum preview size in Nextcloud
  • Tell PreviewGenerator exactly which sizes to generate

The default max image size is 4096x4096 which, frankly, seems a bit overkill. So instead we're going to configure a maximum resolution of 1080p

We do this using the occ library included in Nextcloud (so in my install it's in /var/www/nextcloud)

cd /var/www/nextcloud
php occ config:system:set preview_max_x --value 1080
php occ config:system:set preview_max_y --value 1920

Now, we need to identify the specific sizes that we know are used. If you haven't already you should grab some loglines resulting from accessing with different devices.

Once we know the sizes we need to provide them to previewgenerator

cd /var/www/nextcloud
php ./occ config:app:set --value="32 64 256 1024 1920"  previewgenerator squareSizes
php ./occ config:app:set --value="64 128 1024 1080 2030" previewgenerator widthSizes
php ./occ config:app:set --value="64 256 1024 1080 1920" previewgenerator heightSizes

Now when previewgenerator runs it'll generate thumbnails of those sizes, which should cater to all views and devices.

 

First Run and Default Cronjob

It's now time to trigger the initial run and get thumbnails generate. If you've a lot of photographs then this may take quite some time, despite the changes we've made here, it just won't take nearly as long as it would have.

php /var/www/nextcloud/occ preview:generate-all

It'll then sit working over images and generating previews.

We also need to create a cronjob to keep on top of new images:

crontab -e
*/10  *  *  *  * php /var/www/nextcloud/occ preview:pre-generate

This will now run every 10 minutes and generate thumbnails for any new images (it uses internal locking so overlapping runs aren't an issue).

If you're not using the External Storage functionality, you're now more or less good to go and should skip the next section.

 

External Storage

My Photos directory is exposed via external storage as it's actually an NFS share from my NAS (Nextcloud doesn't support NFS directly, so what you need to do for that is mount the NFS share at OS level, and then use the "Local" storage type, providing the path to the mount).

It took me far too long to find out that the previewgenerator cronjob above doesn't actually do anything for external storage. It makes sense thinking about it, as the cronjob relies on events being tracked, which doesn't happen with external storage (because the changes aren't made through Nextcloud but on the underlying storage).

The generate-all command needs to be used in order to generate thumbnails relating to external storage.

Simply putting this in a cron would work, but would be very inefficient as it'd need to check every image to see if the thumbnails are there. Because I file photos in a dated heirachy, I opted to limit the cron to the current year (I could have added month too, in hindsight)

cd /var/www
cat << EOM > ext_st_thumbnails.sh
#!/bin/bash
#
# Update thumbnails in NAS Pics etc

cd /var/www/nextcloud/

YEAR=\$(date +'%Y')
php occ preview:generate-all -vvv --path="/ben/files/NAS Pics/\$YEAR"
EOM

The easiest way to identify the path to use is to look at the output of running

php occ preview:generate-all -vvv

It'll generally be of the form username/files/external_storage_name (although it's prefixed with a username, the thumbnails will get used for all users with access to the external storage)

Now we need to configure this to be run by cron periodically

chmod +x ext_st_thumbnails.sh
crontab -e
15  *  *  *  * /var/www/ext_st_thumbnails.sh

There's no locking within generate-all so overlapping runs are possible. What'll happen in these cases is that when the second run catches up with the first (i.e. tries to work on the image the first is working on), it'll throw an exception and exit. Which isn't great, but seems to be harmless.

 

 

Bonus: Outsourcing Generation

This step isn't strictly necessary, but is included so that I can refer back to it.

My Nextcloud VM has to work quite hard, particularly when thumbnails are being generated - not only does it have to generate the thumbnails, but it's running the MySQL instance as well (because of the config changes that needed to be made).

I access Nextcloud via a Nginx reverse proxy, so I decided that I'd use a couple of Raspberry Pi's to provide some additional compute for thumbnail generation. The idea being that if any thumbnails don't yet exist, the client requests are spread across as many CPU cores as possible.

 

Nextcloud Server Changes

On my Nextcloud VM, I installed an NFS server and configured my data dir (/var/www/data) as an NFS export accessible to my two Pis

grep data /etc/exports
/var/www/data 192.168.1.11/255.255.255.255(rw,async),192.168.1.12/255.255.255.255(rw,async)

The nextcloud server needs to be reconfigured to allow network access to MySQL. Comment out or remove the following line in /etc/mysql/my.cnf and then restart MySQL.

bind-address		= 127.0.0.1

You also need to create a user for each of the Pi's and let it access the relevant database

CREATE USER 'nextcloud'@192.168.1.11 IDENTIFIED BY 'notARealPassword';
CREATE USER 'nextcloud'@192.168.1.12 IDENTIFIED BY 'notARealPassword';
GRANT ALL PRIVILEGES ON nextcloud.* to 'nextcloud'@192.168.1.11;
GRANT ALL PRIVILEGES ON nextcloud.* to 'nextcloud'@192.168.1.12;

If you've got Redis session and file-locks enabled in Nextcloud you'll also need to make that available to the network (you may want to restrict access with iptables too, but I've left that out to keep this doc short).

Edit /etc/redis/redis.conf and insert your Nextcloud server's IP on the end of the bind line as below

bind 127.0.0.1 192.168.1.95

Then restart redis (systemctl restart redis-server).

As one of the Pi's will be running the previewgenerator crons I disabled those crons on the Nextcloud VM.

 

Pis

I then installed the various nextcloud dependencies on the Pis, and copied /var/www/nextcloud over to them, along with my apache config. 

The Nextcloud config (/var/www/nextcloud/config/config.php) needs editing so it connects back to MySQL on the main node

'dbhost' => '192.168.1.95',
'dbuser' => 'nextcloud',
'dbpassword' => 'notARealPassword',

Within the same file, we also need to reconfigure the Redis address

  'redis' => 
  array (
    'host' => '192.168.1.95',
    'port' => 6379,
    'timeout' => 0.0,
  ),

The cron will need to be able to access the datadir as well as the external storages (these need to be mounted at the same paths as they are on the nextcloud server)

mkdir /var/www/data
mkdir /mnt/photos
grep nfs /etc/fstab
192.168.1.87:/mnt/nextcloud/ /var/www/data     nfs     rw,user
192.168.1.95:/mnt/photos/ /mnt/pics     nfs     ro,user

Data need to be writeable as the user running nextcloud, so as a nasty, nasty hack I put mount statements in /etc/rc.local

su www-data -c "mount /var/www/data"
mount /mnt/pics

(It'd actually be better to create a simple unit file for each, I just haven't got around to it yet)

Reboot to ensure the mounts come up.

You should now be able to trigger preview generation on the Pi

cd /var/www/nextcloud
php occ previewgenerator:pre-generate

and receive no errors.

Create the crons on one of the Pis (running on both is actually probably fine, but I decided not to risk it).

The next thing to then, is configure the reverse proxy to start sending some of the thumbnail requests to the Pis.

 

PHP Session Handling Changes

Nextcloud doesn't implement it's own session handling and instead relies on PHP itself for this. So, depending on which paths get split across the Pis you may hit occasional authentication issues (resulting in the web interface showing Problem loading page, reloading in 5 seconds. This occurs when the authentication goes to one Pi, but a subsequent request goes to another.

To resolve this, we simply need to configure PHP to use Redis as it's session handler

cd /etc/php/7.3/apache2/
nano php.ini
# Find session.save_handler and set to
session.save_handler = redis

# Just below, find session.save_path and set to (replace with the IP of your redis server)
session.save_path = "tcp://192.168.1.84:6379"
# Now restart Apache
systemctl restart apache2

Repeat this on each of the systems running Nextcloud

You can verify the change has taken effect by logging into the Nextcloud web interface, and then on the box running next cloud running

redis-cli 
127.0.0.1:6379> keys PHPREDIS*

All PHP session information should now be available across your cluster

 

NGinx Reverse Proxy Changes

Within my Nginx config, I created a new upstream definition, containing the Pis (and using keepalive to improve performance)

upstream nextcloudthumbnails {
        server 192.168.1.11:443;
        server 192.168.1.12:443;
        keepalive 30;
}

I then created a location block within the main server block for nextcloud

location ~ $(/index.php/apps/gallery/preview/|/index.php/apps/files/api/v1/thumbnail/|/index.php/core/preview) {

    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header Host nextcloud.bentasker.co.uk;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass https://nextcloudthumbnails;

    proxy_buffers 16 16k;  
    proxy_buffer_size 16k;
    proxy_hide_header X-Powered-By;

    # Force upstream to remain a keepalive connection
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    
    add_header X-Clacks-Overhead "GNU Terry Pratchett";
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload" ;
}

This causes the paths used for thumbnailing to be sent to the Pis instead of the VM.

Reload Nginx and the change is instant.

 

Conclusion

It took quite a bit of trial and error to get the image sizing sorted, but now my Nextcloud instance is actually usable for browsing through our (vast) collection of photos.

Browsers load thumbnails almost instantly now (obviously depending on connectivity back to the Nextcloud server), and the android app is also much faster (though it appears to load them sequentially rather than parallelising, which is a little frustrating - they take about 1/2s each).

Where the cron hasn't yet had chance to create thumbnails, the Pi's take up the processing load admirably and do a much better job of responding quickly than the main nextcloud server (which has fewer cores than the Pis combined).

 

Share this post on Twitter Share this post on Reddit Share this post on LinkedIN Share this post via Whatsapp Share this post via Telegram Share this post via QQ Share this post via Email