As I'm migrating from Strapi V3 with MongoDB to Strapi V4 with MariaDB/MySQL, I need to riconfigure the new instance of the CMS with the same configuration of the old one.
The issue is that some things (obviously) changed between the two version, and my old configuration for the @strapi/provider-upload-aws-s3 plugin doesn't work anymore.

Well, not exactly.

When I tried to upload a file, that's what I saw:

One image was uploaded locally, while the other on S3, and what I found out is that the configuration works, and in fact the image is being uploaded on the bucket in AWS S3, but the CMS is refusing to show them.

So after spending some time on google I finally found the solution, which can be found at some point on this GitHub issue and in this other issue.
I just needed to adapt it a bit to add the region to the s3 url in my case.

But essentially what you need to do is to update the config/middlewares.js file with the following configuration:

// From this:

module.exports = [
  'strapi::errors',
  'strapi::security',
  ... //rest of middlewares
];

// To this
module.exports = ({ env }) => [
  'strapi::errors',
  {
    name: 'strapi::security',
    config: {
      contentSecurityPolicy: {
        useDefaults: true,
        directives: {
          'connect-src': ["'self'", 'https:'],
          'img-src': [
            "'self'",
            'data:',
            'blob:',
            'res.cloudinary.com', // cloudinary images
            'lh3.googleusercontent.com', // google avatars
            'platform-lookaside.fbsbx.com', // facebook avatars
            'dl.airtable.com', // strapi marketplace
            `https://${env('AWS_BUCKET')}.s3.${env('AWS_REGION')}.amazonaws.com`
          ],
          'media-src': ["'self'", 'data:', 'blob:', `https://${env('AWS_BUCKET')}.s3.${env('AWS_REGION')}.amazonaws.com`],
          upgradeInsecureRequests: null,
        },
      },
    },
  },
  ... //rest of middlewares
];

And now Strapi should finally work with files updated on S3.

Note
If you wonder why I had to add res.cloudinary.com and other domains, it's simply to allow icons and images to work in Strapi, like in the marketplace.