It has been said before, to make xDebug faster, just disable it.
But one still needs xDebug for how helpful it is!
Well, there is a good solution which is to only enable xDebug when we intend to debug the request.

The idea is to create two php containers (php and php_xdebug) and enable xDebug on only one of these two containers.
Then we can instruct apache to use the php_xdebug container only when a certain cookie is set.

php container

php_xdebug container

apache config

<If "%{HTTP_COOKIE} =~ /XDEBUG_SESSION=PHPSTORM/">
  SetHandler "proxy:fcgi://php_xdebug:9000"
</If>
<Else>
  SetHandler "proxy:fcgi://php:9000"
</Else>

So in this case, we only use php_xdebug when the XDEBUG_SESSION cookie is set to PHPSTORM.
The PHPSTORM value itself can be changed in the env file and in the xDebug helper browser extension.

The rest is very simple, we just need to configure xDebug to always start remote debugging when it's invoked.
The profiler and log output will be sent to a /tmp folder inside the server root.

zend_extension=xdebug.so
[xdebug]
xdebug.remote_autostart = 1
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_connect_back=1
xdebug.profiler_enable_trigger = 1
xdebug.profiler_output_dir = /var/www/html/tmp
xdebug.idekey=${XDEBUG_COOKIE_VALUE}
xdebug.remote_log=/var/www/html/tmp/xdebug.log

You can find the docker-compose file in this github repo
https://github.com/unlocomqx/conditional-xdebug-fpm-docker

After cloning the repo, you can rename the file sample.env to .env and change DOCUMENT_ROOT, MYSQL_DATA_DIR and XDEBUG_COOKIE_VALUE if necessary.

This solution was totally inspired by this blog post https://jtreminio.com/blog/developing-at-full-speed-with-xdebug/