Redmine shows "create session key: disk quota exceeded"

We noticed that the Docker container for Redmine was not working anymore:

$ redmine@redmine01.typo3server.ch in ~ on git:cleanup x disk:47% [16:54:03]
$ docker-compose exec redmine bash
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: join session keyring: create session key: disk quota exceeded: unknown

However, the disk is not full.

Additionally, the process list shows a number of git processes in defunct state:

$ ps aux | grep git
690823     3823  0.0  0.0      0     0 ?        Zs   17:45   0:00 [git] <defunct>
690823     4951  0.0  0.0      0     0 ?        Zs   17:48   0:00 [git] <defunct>
690823     5478  0.0  0.0      0     0 ?        Zs   17:49   0:00 [git] <defunct>
690823     6298  0.0  0.0      0     0 ?        Zs   17:50   0:00 [git] <defunct>
690823     6743  0.0  0.0      0     0 ?        Zs   17:50   0:00 [git] <defunct>
690823     7264  0.0  0.0      0     0 ?        Zs   17:50   0:00 [git] <defunct>
690823     7740  0.0  0.0      0     0 ?        Zs   17:51   0:00 [git] <defunct>
690823     8307  0.0  0.0      0     0 ?        Zs   17:51   0:00 [git] <defunct>
690823     8428  0.0  0.0      0     0 ?        Zs   17:52   0:00 [git] <defunct>
690823     8602  0.0  0.0      0     0 ?        Zs   17:52   0:00 [git] <defunct>

It turns out that these are two problems. Problem number 1 can be fixed by raising system limits, see https://gkellynavarro.com/blog/2020/04/04/container-issues/

However, we don't need this if we fix problem number 2.

The defunctional git processes are created by a cronjob which runs every 5 minutes on this server:

$ crontab -l
SHELL=/usr/local/vzscripts/sfoutputtosyslog

#       +------------------------------------ minute (0 - 59)
#       |       +---------------------------- hour (0 - 23)
#       |       |       +-------------------- day of month (1 - 31)
#       |       |       |       +------------ month (1 - 12)
#       |       |       |       |       +---- day of week (0 - 6) (Sunday=0 or 7)
#       |       |       |       |       |

#       10      2       *       *       *       <command>

*/5	*	*	*	*	cd /home/redmine && docker-compose exec -T --user=redmine redmine bash -c "cd /var/git/repositories/Packages/TYPO3.CMS.git && git fetch origin +refs/heads/*:refs/heads/* && git reset --soft && cd /usr/src/redmine && bundle exec rake redmine:fetch_changesets RAILS_ENV=production" >>/home/redmine/log/cron_fetch_changesets.log 2>&1

When running only the first part of that command (cd /home/redmine && docker-compose exec -T --user=redmine redmine bash -c "cd /var/git/repositories/Packages/TYPO3.CMS.git && git fetch origin +refs/heads/*:refs/heads/* && git reset --soft) then the following message appears:

Auto packing the repository in background for optimum performance.

It seems like this process is started in the background, and it is killed as soon as the shell quits after running all following commands.

The solution was found here: https://stackoverflow.com/questions/8633981/what-does-auto-packing-the-repository-for-optimum-performance-mean

We can instruct Git to clean up the repository in the foreground by setting git config gc.autoDetach false:

$ docker-compose exec  --user=redmine redmine bash 
redmine@28cf14a10e0b:/usr/src/redmine$ cd /var/git/repositories/Packages/TYPO3.CMS.git
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config -l
core.repositoryformatversion=0
core.filemode=true
core.bare=true
remote.origin.url=https://git.typo3.org/Packages/TYPO3.CMS
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ 
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config gc.autoDetach false
redmine@28cf14a10e0b:/var/git/repositories/Packages/TYPO3.CMS.git$ git config -l 
core.repositoryformatversion=0
core.filemode=true
core.bare=true
remote.origin.url=https://git.typo3.org/Packages/TYPO3.CMS
gc.autodetach=false

This will immediately fix the problem. After a reboot, there are no more defunct processes, and there should be (so we think) no more errors about disk being full.