Working with SQL:

• 1. Understanding SQL Query Execution Plan Before Creating Indexes to Optimize Costs
• 2. Optimizing SQL Performance with Execution Plans

SQL execution plans are key to understanding and tuning query performance. The plan shows how the database engine will execute your SQL query. Analyzing the plan can reveal opportunities to add indexes that can drastically improve query speed.

Understanding Execution Plans

When you execute a query, the SQL engine generates an execution plan, which is a detailed recipe of how it will run the query. The plan shows:

• The algorithm or strategy the engine will use to access tables and data (scans, joins, etc.)
• The order of operations it will follow
• An estimate of how long each operation will take (cost)
• How many rows will be processed by each operation

Databases have a query optimizer that creates different possible plans and picks the one with the lowest total cost.

Prepare Database To Practice

-- Sample table with 1 million rows
CREATE TABLE large_table (
  id INT,
  name VARCHAR(50), 
  address VARCHAR(100)
);

-- Insert 1 million rows 
INSERT INTO large_table 
SELECT id, 
       concat('Name ', id),
       concat('Address ', id)
FROM generate_series(1, 1000000) as id;

The key points are:

• The execution plan shows the steps SQL needs to take to run a query
• It estimates the cost based on rows processed and operations
• Without an index, the query has to do a full scan of the table
• Adding an index provides a fast lookup to find the row directly
• Indexes reduce the scan cost and overall query execution time So looking at execution plans helps optimizing by adding indexes to reduce cost

Viewing Execution Plans

To view the execution plan for a query, use the EXPLAIN or EXPLAIN ANALYZE commands.

• EXPLAIN shows the planned operations and estimates.
• EXPLAIN ANALYZE actually executes the query and provides the real execution statistics.

-- Query without index
EXPLAIN ANALYZE 
SELECT * 
FROM large_table
WHERE id = 500000;

-- Results:
-- Seq Scan on large_table (cost=0.00..18812.00 rows=1 width=158) (actual time=0.357..17.971 rows=1 loops=1)
--   Filter: (id = 500000)
--   Rows Removed by Filter: 999999
-- Planning time: 0.177 ms
-- Execution time: 18.014 ms

The plan will be output showing details like:

• Scan type (index scan, sequential scan)
• Filters applied
• Join method and ordering
• Sorts or grouping operations
• Total cost estimate

Using Plans for Index Optimization

A key benefit of examining SQL execution plans is optimizing performance by adding indexes. Things to analyze:

• Table/Index Scans - Sequential scans of entire tables or indexes are slow. Indexes can reduce full scans if filters are applied.
• Sorts - Sorting results can have high cost. An index on the sort columns can eliminate the sort.
• Joins - Hash and nested loops joins without indexes require expensive on-the-fly table scans. Indexes enable faster merge and index joins.
• Filter Conditions - Check which columns are used in the WHERE clause and consider indexes on those columns. Output Rows - A high output row count indicates more data processed. Tight indexes on columns in SELECT can optimize this.
• Frequent Queries - Look for repetitive queries on production systems. Indexes improve the most common recurrent queries.

Try to add index and see result after indexed

-- Add index on id column
CREATE INDEX idx_id ON large_table (id);

-- Query with index
EXPLAIN ANALYZE  
SELECT *
FROM large_table
WHERE id = 500000;

-- Results: 
-- Index Scan using idx_id on large_table (cost=0.42..8.44 rows=1 width=158) (actual time=0.041..0.042 rows=1 loops=1)
--   Index Cond: (id = 500000)
-- Planning time: 0.488 ms
-- Execution time: 0.064 ms

• With the index, it only needs to look up the id in the index, taking 0.04 ms. 
• The index lookup is much faster than a full scan.
• So adding the right indexes can drastically improve query performance by reducing scan cost.
• The execution plan provides this query cost insight to help optimize

Indexing Best Practices

Here are some best practices for adding and using indexes for query performance based on plan analysis:

• Focus first on the largest tables and most frequent queries. Big gains here.
• Use indexes for columns in JOIN, WHERE, ORDER BY, and GROUP BY clauses.
• Consider a covering index that satisfies the query from the index only.
• Be selective - too many indexes slow down write operations like inserts and updates.
• Drop indexes that are not used by major queries - monitor index usage.
• Reassess indexes if the query workload changes.

Conclusion

Using execution plans provided by SQL databases is crucial for optimizing the performance of queries. By analyzing the detailed plan, you can determine if adding indexes is warranted to reduce overall query cost and improve response time. The key is targeting indexes to match the specific query filters, joins, and sort orders seen in the plan. With some practice reading plans, you'll be a query tuning pro!