Yesterday I tried to set up Authentik, but it was failing in gunicorn with an obscure message that a file could not be accessed. I got very frustrated and enabled debug logging in Authentik’s server, and I let it run.
Big mistake! I forgot about it and the next day I noticed that none of the other processes were running properly! Ouch! :)
The debug logging was spewing messages every second and… it filled up the empty space (23G!). I remembered that sometimes removing the file would just create a new one, so I did. This time instead, the logger was just offsetting to the end and trying to append a new line. So I had to:
- Stop the container
- Remove the log file
- Recreate the container with normal (info) logging
Now I’m monitoring it to see how quickly fills up the space and I’m looking to move the logging to a rotating logger, or, better yet, to a prometheus instance.