Skip to main content

SQL Server performance for NOT EXISTS vs NOT IN

Though "NOT EXISTS" and "NOT IN" sounds similar there is quite a lot of difference between them.  To start with check out the blog post by Mladen.

In continuation to what Mladen has already written I thought I would show the differences it makes on the Execution Plan and the IO / Time when we use NOT EXISTS or NOT IN in our queries. Let's see few of the differences between them.

Case 1: Lets use them in columns which are declared as NOT NULL

SET NOCOUNT ON
GO


CREATE TABLE PackageInformation
(
Sno INT IDENTITY(1,1) PRIMARY KEY,
PackageID INT NOT NULL,
PackageName VARCHAR(20)
)
GO

-- I am using the random records generator which I wrote few days back to populate data into this table.
-- generating 10000 records
INSERT INTO PackageInformation  (PackageID, PackageName)
SELECT CAST(RAND(CHECKSUM(NEWID())) * 10000 AS INT),
dbo.udf_StringGenerator('A', 20)
GO 10000


CREATE TABLE ChildTable
(
RID INT IDENTITY(1,1) PRIMARY KEY,
PackageID INT NOT NULL
)
GO

-- Lets take some 40% (approx 4000 records) from the PackageInformation table to populate this table
INSERT INTO ChildTable (PackageID)
  SELECT PackageID
  FROM dbo.PackageInformation
  TABLESAMPLE (40 PERCENT); -- This would work only on SQL Server version 2005 or above
GO

Let's write a query to list Package details from PackageInformation table which is not present in ChildTable.

--Clear out the cache (DONT TRY THIS IN PRODUCTION ENVIRONMENT)
DBCC FREEPROCCACHE
GO

SET STATISTICS IO ON
SET STATISTICS TIME  ON
GO

--Press Control + M to display the Actual Execution Plan of the queries
--Query1: Using NOT IN
SELECT PackageID, PackageName 
FROM dbo.PackageInformation 
WHERE PackageID NOT IN (SELECT PackageID FROM ChildTable)
GO

--Query2: Using NOT EXISTS
SELECT PackageID, PackageName 
FROM dbo.PackageInformation
WHERE NOT EXISTS
 (
   SELECT PackageID FROM ChildTable 
   WHERE ChildTable.PackageID = PackageInformation.PackageID
)
GO

SET STATISTICS IO OFF
SET STATISTICS TIME  OFF
GO

Result:

i) Query using NOT IN : Returned 4690 records
ii) Query using NOT EXISTS : Returned 4690 records

iii) Let's see the Actual execution plan for both the queries. 

iv) Let's also check on the logical reads and CPU time taken for these queries.

So when the column is declared as NOT NULL then both NOT IN and NOT EXISTS seems to perform the same way.

Case 2: Let's change the PackageID column as NULL

ALTER TABLE dbo.PackageInformation
ALTER COLUMN PackageID INT NULL
GO

ALTER TABLE dbo.ChildTable
ALTER COLUMN PackageID INT NULL
GO

--Lets insert some 100 null values into the PackageInformation Table
INSERT INTO PackageInformation 
SELECT NULL,
dbo.udf_StringGenerator('A', 5)
GO 100


Let's run the same query which we used in Case 1 to list Package details from PackageInformation table which is not present in ChildTable.

--Clear out the cache (DONT TRY THIS IN PRODUCTION ENVIRONMENT)
DBCC FREEPROCCACHE
GO

SET STATISTICS IO ON
SET STATISTICS TIME  ON
GO

--Query1: Using NOT IN
SELECT PackageID, PackageName 
FROM dbo.PackageInformation 
WHERE PackageID NOT IN (SELECT PackageID FROM ChildTable)
GO

--Query2: Using NOT EXISTS
SELECT PackageID, PackageName 
FROM dbo.PackageInformation
WHERE NOT EXISTS
 (
   SELECT PackageID FROM ChildTable 
   WHERE ChildTable.PackageID = PackageInformation.PackageID
)
GO

SET STATISTICS IO OFF
SET STATISTICS TIME  OFF
GO


Result:

i) Query using NOT IN : Returned 4690 records (It hasn't considered those 100 new NULL records which we added!!)
ii) Query using NOT EXISTS : Returned 4790 records

iii) Let's see the Actual execution plan for both the queries. 

iv) Let's also check on the logical reads and CPU time taken for these queries.

So when column is declared as NULL then NOT IN seems to generate a pretty complicated execution plan and does NUMEROUS number of logical reads more than NOT EXISTS. So the winner here is NOT EXISTS.

Case 3: Adding NULL values into ChildTable

INSERT INTO ChildTable (PackageID)
SELECT NULL

Result:

i) Query using NOT IN : Returned 0 records!
ii) Query using NOT EXISTS : Returned 4790 records

So if the Subquery returns even one NULL then NOT IN operator would not return any result which isn't right. So again the winner is NOT EXISTS.

I think it would be safe to say that we should use NOT EXISTS instead of NOT IN as it seems to work as expected by us in all the scenarios which we saw in this post.

--Cleanup
DROP TABLE ChildTable
GO
DROP TABLE PACKAGEINFORMATION
GO
Labels:

Comments

Post a Comment

Popular posts from this blog

[Non Tech] Want to know the recipe for Omelette :)

Fed up with Bread - Jam and Curd Rice, today i wanted to eat Omelette. Interesting part is I wanted to cook it myself :) So in the first picture you see all the items which are needed for preparing an Omelette. When I had a closer look at the eggs I see that almost all the eggs are broken. But believe me when I bought it couple of days back it was in perfect condition! I was wondering whether the eggs have become rotten or pretty old to consume! I tried taking an egg and break it but couldn't break it at all :) Since I have kept in the freezer all the eggs have frozen and looked like a iron ball :) After trying for few minutes of trying i removed the shell of the egg and then kept that iron ball :) into a bowl and placed it within Oven. I heated it for 1 minute and checked. It melted only to a limit. So i just set it for another 2 minutes and checked it later. It has melted but the part of the egg white has become a Omelette :( I didn't leave it there. I took the bowl out of ...
8 comments
Read more

AWS fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

While using AWS and trying to copy a file from a S3 bucket to my EC2 instance ended up with this error message. Command Used: aws s3 cp s3://mybucketname/myfilename.html /var/www/html/ Error: fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request The error goes off if we add the region information to the command statement. I am using Asia Pacific (Mumbai) so used ap-south-1 as the region name. Modified Command: aws s3 cp s3://mybucketname/myfilename.html /var/www/html/ --region ap-south-1
Post a Comment
Read more

Script table as - ALTER TO is greyed out - SQL SERVER

One of my office colleague recently asked me why we are not able to generate ALTER Table script from SSMS. If we right click on the table and choose "Script Table As"  ALTER To option would be disabled or Greyed out. Is it a bug? No it isn't a bug. ALTER To is there to be used for generating modified script of Stored Procedure, Functions, Views, Triggers etc., and NOT for Tables. For generating ALTER Table script there is an work around. Right click on the table, choose "Modify" and enter into the design mode. Make what ever changes you want to make and WITHOUT saving it right click anywhere on the top half of the window (above Column properties) and choose "Generate Change Script". Please be advised that SQL Server would drop actually create a new table with modifications, move the data from the old table into it and then drop the old table. Sounds simple but assume you have a very large table for which you want to do this! Then it woul...
2 comments
Read more