I have cases where we'd see some performance issue with our REST API and start working on optimizing it.

I think the first thing anyone does is blame the code. Maybe we haven't used the SQL queries right? Could we change the algorithm to process the data better? Are we indexing the columns properly?

But at what point do you consider that perhaps it's the infrastructure: should we upgrade the EC2/RDS instance? Add more load-balanced servers?

Upgrading the infrastructure will definitely improve performance, but at a financial cost. We'd always want to get the last bit of performance from the current hardware by changing the software as much as possible, right?