My design was inspired by the LMAX Disruptor Pattern and used sequencer objects to greatly boost throughput of the queue structure.

In this blog entry, I will show you how I build this queue structure and give you enough details to do so yourself.

But first, lets me walk you through the problem.

Note: In the following I will use the terms push and pop for respectively adding and deleting messages in a queue. You may have learned the terminology enqueue and dequeue instead, same thing.

The Problem Statement

Every now and again, relational database designers find themselves creating a table that ends up acting as a message queue. Such tables tend to fluctuate a lot in size, from zero rows to several millions. Furthermore, the tables typically need to be ordered so fairness of message pushing and popping. The data flow looks something like this:

In the following examples, I will assume a message size of 300B – but the argument flies for larger and smaller message sizes too.

Relational Message Queues – Naïve

When faced with a table that need to be ordered where the same rows are frequently added and shortly after deleted, something like this is the typical design:

CREATE TABLE dbo.MyQ
(
    [message_id] INT NOT NULL
    ,[time] [datetime] NULL
    ,[message] [char](300) NULL
)

CREATE UNIQUE CLUSTERED INDEX CIX
ON dbo.MyQ (message_id)

CREATE SEQUENCE dbo.MessageSequence AS INT

The rows in the table are implicitly ordered by the index. Push and pop can now implemented like this:

/* Push Naïve */

INSERT MyQ (message_id, time, message)
SELECT NEXT VALUE FOR dbo.MessageSequence, GETDATE(), ‘Hello World’

/* Pop Naïve */

DELETE Q
FROM (SELECT TOP 1 * FROM MyQ ORDER BY message_id) AS Q

This approach has a lot of problems that will be very familiar with anyone who has tried to be a DBA for a table like this. Among these problems are:

• The statistics are NEVER up to date on such a table. This typically requires all plans to be forced/hinted or statistics to be custom hacked.
• In databases that auto updates statistics, the auto updates constantly kick in, creating jagged CPU patterns and throughput.
• The B-tree structure implementing the ordering of the rows is constantly being split at the root as it grows and shrinks. This causes mutex convoys (in SQL Server, these convoys are reported as latches on ACCESS_METHODS_HOBT_VIRTUAL_ROOT)
• New pages are constantly allocated and deallocated in the buffer pool which stresses internal allocation structures in the database
• There is contention on the memory structures that hold the pages at the start and end of the B-tree (reported as PAGELATCH_EX in SQL Server)
• The writes are extremely sensitive to latency of the transaction log drive and this is exacerbated by the convoy effects on the higher levels of the B-tree always being split.

The contention is perhaps best illustrated with this diagram of the hot pages in the B-tree:

I build this implementation on an HP DL380G7 server with a FusionIO ioDrive Duo (tlog latency under 50microsec) on a SQL Server 2012 RTM installation. I experimented with different thread counts for both push and pop to find the sweet spot. The highest throughput I could push (pun intended) through the message queue with the above implementation was around 6000 messages/sec.  Very, very far from impressive and fuel for the NoSQL fires.

The obvious (but wrong) conclusion, is that relational database are not fit for purpose when it comes to building message queues. Of course, there IS a grain of truth to it, but can we do better than a meager 6000 messages/sec?

Relational Message Queues – LMAX’ed

It is in fact possible to do a lot better than the naïve message queue implementation. Inspired by the bright people over at LMAX and their Disruptor pattern, I decided to build a relational equivalent.

The first realisation is that INSERT/DELETE is just not going to work. People who code high scale systems might intuit this: constant memory allocation and deallocation is expensive and INSERT/DELETE is a form of memory alloc/dealloc. The solution is to preallocate the queue at a certain size and UPDATE a reference count on each message slot in a row instead of completely removing/adding a row for every message sent through the queue.

The queue table now looks like this:

CREATE TABLE dbo.MyQ
(
    [Slot] BIGINT NOT NULL
    , message_id BIGINT NULL
    ,[time] [datetime] NOT NULL
    ,[message] [char](300) NOT NULL
    ,reference_count TINYINT NOT NULL
)

/* Prefill messages */
WHILE @i < @QueueSize BEGIN
  INSERT MyQ (Slot, time, message, reference_count) 
  VALUES (@i, ’2050-01-01′, ‘dummy’, 0)
  SET @i = @i + 1
END

/* Create index and fill it up */
CREATE UNIQUE CLUSTERED INDEX CIX ON dbo.MyQ (Slot)
WITH (FILLFACTOR = 100)

Side Note: If you are still wondering why UPDATE is faster than INSERT for this case. See my previous blog entry.

The above preallocates @QueueSize message slots. Pushing a message is now these operations:

• Generate a new message_id
• Find the next available slot (with reference_count = 0)
• Update the message column
• Add one to reference_count

And pop is:

• Find the smallest message_id that has been “inserted”
• Read and return the message column
• Decrement reference_count, marking the slot as available again (variants can be done if you have multiple subscribers)

At least, the above is the pseudo code. However, for this to work we have to find a way to quickly locate the next available slot in the message queue for push and find the smallest message_id that is not popped yet.

First, how do we find the next available slot for push? This turns out to be surprisingly easy: we use a sequencer object. The sequencer, for the non database people out there,  is a very high scale data structure to generate “the next number”. Think of it like a singleton object. The sequencer can be used for BOTH generating the message_id AND find the next available slot. Here is how:

/* Sequence to keep track of next message number and slot */

CREATE SEQUENCE dbo.PushSequence AS BIGINT
START WITH 1 INCREMENT BY 1
CACHE 100000;

/* Push LMAX’ed Begin */

SET @PushSeq = NEXT VALUE FOR dbo.PushSequence

/* Find slot */
SET @Slot = @PushSeq % @QueueSize

UPDATE dbo.MyQ /* SQL Server users: hint WITH (ROWLOCK) here */
SET [time] = GETDATE()
    , [message] = ‘Hello World’
    , [message_id] = @PushSeq
    , reference_count = reference_count + 1 
WHERE Slot = @Slot
  AND reference_count = 0 /* Don’t overwrite! */

IF <No rows affected> BEGIN
  /* The slot was not available –> queue is full */
  <Sleep 100ms>
  <Try UPDATE again>
END

/* Push LMAX’ed end */

What we have now achieved is essentially a ring buffer for insert operations. To illustrate, here is an example queue table of @QueueSize = 100.

We can quickly located new slots for push operations. All the remains is to implement pop. We could chose to keep track of the smallest message_id in a variable that the pop will update. However, it is always better to remove coordinate when possible. Instead of letting pop keep track of the messages, we can create a ANOTHER sequencer object that trails behind PushSequence. In fact, we don’t even need to keep track of the value of PushSequence to properly pop. Here is how to implement pop:

CREATE SEQUENCE dbo.PopSequence AS BIGINT
START WITH 1 INCREMENT BY 1
CACHE 100000;

/* Pop LMAX’ed Begin */

SET @PopSeq = NEXT VALUE FOR dbo.PopSequence

/* Find slot */
SET @Slot = @PopSeq % @QueueSize

UPDATE dbo.MyQ /* SQL Server users: hint WITH (ROWLOCK) here */
SET [time] = GETDATE()
    , @OutPutMessage = message 
    , [message_id] = NULL 
    , reference_count = reference_count – 1 
WHERE Slot = @Slot
  AND message_id = @PopSeq /* Make sure we didn’t try to pop an empty slot */

IF <No rows affected> BEGIN
  /* No message found to pop or we are ahead of push */
  <Sleep 1sec>
  <Try UPDATE again>
END

/* Pop LMAX’ed end */

Very similar to push, with a few tricks added. You have to be a bit careful with the boundary conditions. First of all, because there is no coordination between PopSequence and PushSequence, it CAN happen (for example due to thread scheduling) that the current pop message_id gets ahead of push message_id. When that happens, we will wait for the pusher to create more rows. We can now complete the above illustration:

To avoid pop constantly blocking on an empty queue (if message pop faster than they push), it is generally a good idea to let push get a “head start” so there is something in the queue to pop. In my example I have chosen to let pop wait for 1 second before trying to pop from a queue where the last attempt to pop was an empty queue.

I took the above implement and ran it on the same hardware as the naïve approach. The results were VERY different. I can now drive between 90K and 100K messages/sec through the queue. A nice little 15x improvement.

Summary

It can be argued that old school, relational databases and tables are not the best structures to implement durable storage for message queues. The many code paths needed in relational algebra to implement ACID properties, generic concurrency, serialization and block I/O can get in the way of a fast queue implementation, especially the naïve implementation of relational purists.

However, if we combine our knowledge of programming with database design skills, high throughput can be achieved even in the relational model and there are large, often overlooked improvements to be found.

Specifically, I am looking for samples from Kimball style warehouses from different industries (If you are a 3NF warehouse, I am not interested). Roughly speaking, I would like something like this select statement:

SELECT HashFunction(DimensionKey1) AS d1
, HashFunction(DimensionKey2) AS d2
, ….
, HashFunction(DimensionKeyN) AS dN
, HashFunction(Measure1) AS m1
, …
, HashFunction(MeasureM) AS mM

FROM FactTable

 

Where HashFunction is something that yields either a 4 byte or 8 byte integer. If you are using SQL Server, BINARY_CHECKSUM will do (or you can get fancy and use my C# implementation of CRC32).

10MB of data from a fact table would be optimal, preferably sampled from a single time interval in the data. For example: if a full day is around 10MB in the warehouse, a single day would be the  best sample. I don’t need to know any column names, only which columns are dimensions and which ones are measures. If possible, some indication of which industry the data is from would be helpful, but it is not strictly needed either.

I am fully aware that there may be legal reasons you cannot share data, even when anonymised like above. Please only give it to me if you would feel confident to give this data freely away on the Internet. I am specifically trying to create guidance on compression techniques depending on the content of data, and I plan to share this on  my blog, along with distribution statistics on the anonymised data.

Data can be delivered to me via DropBox or Google Drive in CSV format. If needed, I can also HTTP or FTP GET it.

Thanks for anything you can share, I will return the sharing favour here on my blog

During the graduation ceremony at my university it was said that: “Life is a series of temporary relationships” (though hopefully not with your wife). Throughout my career I have taken this fact of life to heart. And I can, without fear of the future, try out new challenges.

After June 2012, I am going to take it a bit slow, and do some consulting. I will then consider which adventure I want to go on from here. Because I believe you have to live out your dreams of adventure while you are still young, and I have a fair bit of fuel left in the tank before “peak oil”.

I part with Microsoft as a friend and will be working on creating a smooth transition over the next month. My work with SQL Server will continue, just outside the campus in Redmond. After June, I will be available for some teaching and short consulting gigs to share my knowledge. Terms and conditions will apply and follow here.

In the meantime, I have made my CV available: About Me.

Let me give you some background: I like movie theatres, a lot. One of my favourite, the venue last week, is the Electric Cinema in Noting hill. This place represents everything that is great about theatres: It has plush red chairs, footstools, Victorian interior and even a café at the back where you can buy a well made espresso. The sound is great and the screen has high quality, digital projection. The people who go there are not annoying either. The cost is just high enough to discourage people from bringing their youngest children. In other words: It represents everything good about watching movies for the childless couple – my girlfriend and I. Building something like this at home would be, to put it mildly, cost restrictive. It’s not just the quality of the room, it’s the atmosphere.

As I emerged into the daylight, a thought occurred to me: “Wouldn’t it be nice to watch Iron Man 1 and 2 now? I liked those movies too”. My girlfriend, like most people with two X-chromosomes, finds Robert Downey Jr. very charming – it would be an easy sell. I know we are a typical couple in this case – because those movies have suddenly become very popular on iTunes.

Why then, is it that I cant watch Iron Man in the movie theatres? There is really no technical barrier that prevents anyone from uploading digital movies to theatres “on demand”. Yes, the movie is old – but movies don’t go out of fashion. Heck, my father watches the same black and white movies – over and over again.

Imagine a future where I can put down a deposit of money, vote for movies to be shown in my local theatre. I would also provide some dates that I would like to watch. If enough people “vote” for a movie with their deposits, it is shown in a proper timeslot. Of course, it takes off some of the spontaneous "impulse watching” of movies. But I would not expect that everyone who votes will have time to show up. I could buy tickets of someone else or at a discount price – stock market style. This could even be done as Facebook app to drive herd behavior: “Thomas wants to watch Iron Man, hey, I will vote too”.

All of this is of course terribly complicated to implement for someone who lacks planning and IT skills. Instead, I am forced to live by the arbitrary (for the consumer), geographically controlled release dates of Hollywood movies. It seems to me that this is yet another variant of the dinosaur mentality we see in the music industry and it is carrying over to other media companies. It’s a desperate attempt to control a market, because you are too daft, too cocaine snorting or groupie distracted to come up with a better business model. Now, wouldn’t it be nice to have a consumer controlled movie theatre? How much would you pay per month to subscribe if you had influence?

Let us first look at the political reasons for virtualising: Operation teams, for very good reasons, often try to push developers towards virtualised systems – cloud is just the latest in this ongoing trend. They try to provide an abstraction between application code and the nasty, physical logistics of data centers – making their job easier. The methods of the operation teams employ take many forms: VLAN, SAN, Private clouds and VMWare/HyperV to quote a few examples. Virtualising will increase their flexibility – and drive down their cost per machine, which looks great in the balance sheet. However, this is flexibility comes at a very high cost. It has been said that:

“All non-trivial abstractions, to some degree, are leaky”

Joel Spolsky

In the case of virtualisation, the abstraction provided is very non-trivial indeed and the leaking is sometimes equally extreme. Traditionally, the issue with virtualisation has been slowdown of I/O or network – though this has gotten a lot better with hardware support for virtual hosts (though SAN still haunts us). Over provisioned memory is another good example of virtualisation wrecking havoc with performance. All of these seems to be surmountable though and this is driving cloud forward.

However, lately it is becoming increasingly clear that scheduling, NUMA and L2/L3 cache misses are potentially an even larger problem and one that will surface once you take I/O out of the bottleneck club.

As we grow our data centers to cloud massive scale and pay for compute power by the hour, every machine counts and will figure in the balance sheet. It should also be clear that a important optimisation will be to focus on the performance on individual scale nodes – to make the best use of the expensive power.

This morning, I ran into some fascinating research in this area (Barret Rhoden, Kevin Klues, David Zhu, Eric Brewer) who take this to another level:

“Improving Per-Node Efficiency in the Datacenter with New OS Abstractions” (pdf)

To whet your appetite, here is a quote from the abstract (my highlight).

“We believe datacenters can benefit from more focus on per-node efficiency, performance, and predictability, versus the more common focus so far on scalability to a large number of nodes. Improving per-node efficiency decreases costs and fault recovery because fewer nodes are required for the same amount of work. We believe that the use of complex, general-purpose operating systems is a key contributing factor to these inefficiencies.”

A highly recommend read and a good primer on some of the things that concern me a lot these days.

Kejser’s Law

I think it is time for me to state my own law (or trivial insight if you will) of computing. Though I stand here at the shoulders of giants, I will steal a bit of the fame. I think it is appropriate that I state one of the things I aim to show people at conferences:

“Any shared resource in a non-trivial scale workload, will eventually bottleneck”

Let us have a look at the facts for a moment and weigh up if this myth has any merit.

Transaction Logged Data

In traditional, relational, ACID property, rows stored in pages, relational databases we typically distinguish between two types of DML operations from a transaction logging perspective.

• Row Logged
• Allocation Logged

Row logged operations will write a transaction log entry every time a row/tuple is modified. This means that the amount of transaction log traffic generated is proportional to the number of rows touched.

Allocation Logged (called: “Minimal logging” in SQL Server) operations only write the physical allocations to the transaction log, if at all. This means the log traffic (if any) is proportional to the size of the data touched. This typically generates at least an order of magnitude fewer log entries than row logged, and is thus faster… Or is it? Read on…

Typically, ACID databases only allow bulk style loads, index builds and large table/partition truncations and drops to be allocation logged. UPDATE and DELETE statements tend to be fully row logged.

Myth: This difference in allocation structures makes UPDATE “bad” because the transaction log is the bottleneck.

Reality: The picture a quite a bit more nuanced than that.

First of all, INSERT in bulk mode is not always allocation logged. Typically a lot of conditions have to be true for allocation logging to work. For those of you interested in SQL Server, I have written about this extensively here: The Data Loading Performance Guide.

Second, the transaction log bottleneck ”wall” is widely exaggerated. I have personally driven 750MB/sec write log traffic into a single database in SQL Server using a FusionIO card. I have seen colleagues do 120MB/sec with traditional, 15K spindles. True: I have also driven 3GB/sec (around 10TB in an hour) using allocation logged INSERT, which is faster than its row logged sibling. However, you have to ask yourself the question: Do you need to go that fast in a single database? In an MPP system you will also have several log files that work together to provide linear scale of log traffic with no relevant roof.

Third, it is perfectly conceivable that you are running a warehouse database that does NOT need to serialize DML operations or may even write entire blocks directly to the database instead of the transaction log. Such systems simply do not have the above bottleneck. For example, Neteeza uses a “logless” implementation and so does many noSQL database systems and HADOOP/HIVE. MySQL with the MyISAM engine also allows non logged operations.

INSERT/DELETE vs. in-place UPDATE

In an traditional, relational database an update can be implemented either as a transactionally wrapped INSERT of the new data followed by a DELETE of the old data (or the other way around).

Myth: Because UPDATE is an INSERT and a DELETE this can double the amount of data that needs to be written during an UPDATE operation and will make me hit the transaction log wall even faster.

Reality: If you are not modifying the keys in an index or making the column size wider, UPDATE statements can be executed as in-place modifications of the row. This allows the database to only write a special old/new value into the transaction log. This optimization can even be applied on a column by column basis, further reducing the transaction log footprint. This leads us to:

UPDATE vs. INSERT speed

This myth is an amalgam of the above arguments.

Myth: INSERT of row logged data is faster than UPDATE of row logged data

Reality: Let us first settle one thing which I will type on its own line and in red to make it easy to remember:

An in-place, row logged, UPDATE operation on a non compressed page is faster than doing a row logged INSERT of the same data.

Why is this? Because the INSERT operation has to allocate new physical structures in the index, while the UPDATE can simply reuse database pages without having to allocate more space.

And here are the numbers to prove it where I am running INSERT vs. UPDATE of large dataset in SQL Server (smaller is faster):

True: if your UPDATE statement has to do the INSERT/DELETE trick, it will likely be slower. But if you are NOT changing the row size and you get the in-place UPDATE, it might just be FASTER to run an UPDATE than an insert.

Also true: compression can change the game quite significantly depending on the compression algorithm you use for the table structure. This is again implementation dependent.

Summary

The myth about UPDATE statements being bad or slow is too simple a way to look at this crucial DML operation. In fact, the myth is outright false in some cases.

Avoiding UPDATE statements should not generally be a major driver for design guidance, or used as the basis for drawing any conclusions about the data modeling techniques you  should apply. There picture is quite a bit more nuanced than this.

First of all, the speed of UPDATE statement as compared to bulk inserts will depend on the database engine you run on.

Second, we have seen that even when UPDATE is fully row logged, the transaction log “wall” is very far away on proper hardware and not much of a concern to 99% of all the installations out there. There are of course cases where you will hit the “wall”, but those are largely mitigated in MPP systems or other sharded deployments that have more than one transaction log.

Third, there are cases where UPDATE statements are actually faster than (row logged) INSERT statements. These typically occur when you change columns in such a way that they don’t grow larger than they already are, allowing in-place UPDATE operations. An interesting and highly relevant example of such a case is UPDATE statements that target fact table keys – which (if you follow my guidance) are integers and therefore have constant width.

Bonus Exercise and chance to win (for SQL Server people): Here is an interesting experiment. In theory, it might be possible to create a workload where you UPDATE a very wide table and where the equivalent, minimally logged, super optimized “copy to new table” BULK INSERT or SELECT INTO statement is actually slower than the UPDATE. Where is the crossover point on table width? I will offer a free, 1 hour teaching session. Database subject of your choice, you host me, I bring the coffee in the London City area to the first person who can provide a test script and the data to show this crossover point. Alternatively the price can also be claimed if you can conclusively prove that the crossover point does not exist.

• Conor vs. SQL

Last week, I spoke at Microsoft Open World 2012 doing quite a lot of presentations. I have created a new and extended “Grade of the Steel” deck that I hope to blog more about my findings (these things tend to start as PowerPoint slides and evolve into blog entries).

If you want a copy of any of my presentations, please just mail me. Speaking of presentations, I have found this really cool tool that will compress slides: NXPowerLite. This neat tool has taken off around 60-80% of my slide sizes.

Lately, I have been practicing zen style presentations: max four sentences on each slide and heavy use of visualisations instead. I highly recommend putting yourself under such constraints. After I removed the text from my slides I feel much better improvising on stage.

It can be very painful to make this work. To assist up and coming debuggers, I put my notes together in this blog. Here is how to get things going.

Downloading and Installing the Tools

First, you need the full tool package. Fortunately, all you need is now packaged in the Windows SDK to ease the pain:

• Download and install Windows SDK for .NET 4.0 from here: http://msdn.microsoft.com/en-us/windows/hardware/gg463009/.

Do the full install of all packages. They will take several GB. Please make sure you have a large drive (more than 100GB of space), you will need it.

Symbol Setup

This is the tricky part. In order to debug properly, you need symbols. Getting those set up requires a few tricks.

First, create directories to hold symbols and symbol caches. It is generally a good idea to stick to the naming convention that Microsoft uses in their examples:

• Create C:\Symbols
• Create C:\SymCache

The next step is to configure symbol environment variables. While you CAN debug without them, setting them up is really worth the effort. First, locate you environment variables in My Computer –> Properties –> Advanced System Settings –> Advanced (why is all the good stuff always located in “Advanced”?). Here you will find:

You need to set up two environment variables. I prefer to set them as system variables, since I am the only one using my machine.

Assuming you use the directories above, set your symbol paths like this:

• _NT_SYMBOL_PATH=SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols
• _NT_SYMCACHE_PATH=C:\SymCache

Note that the paths are separated with stars, not semi-colon.

Main tools overview

There are some key command line tools that you will need to get familiar with:

• SqlDumper.exe – Used to generate dumps and minidumps. There are other tools that do the same, but I prefer this one. The usage is described here: KB 917825
• WinDbg.exe – A lightweight debugger for analysing dumps. Leaves a lot to be desired in usability, but it is really fast once you memorise the arcane commands required.
• SymChk – Used to check if symbols can be resolved. Gives verbose information with the /v command line, which is useful to validate symbol paths. See: SymChk Command-Line Options.
• SymStore.exe – Allows you to create your own symbol stores (under C:\symbols for example). This is useful if you are debugging your own code or code which has symbols that dont originate from Microsoft. See: SymStore Command-Line Options.
• xperf.exe – The mother of all code profilers. This tool comes with an enormous amount of command line parameters. It would require an entire blog entry just to list my favourites (Hint: I may do one). Documentation is hard to come by, but here is a good intro: All Your Base are Belong to Us – Xperf.
• xperfview.exe – Used to view the traces generated by xperf

WinDbg Cheat Sheet

There are some commands that I use frequently in WinDbg. I would recommend you minidump a process you know while it is working in a test system (sqlservr.exe for example) and try them out. It is really quite amazing what you can see.

Yesterday, Lasse ran into an issues with a query pattern in the large database that he is responsible for. Based on our conversation, we wrote up this blog and created a repro.

The troublesome query we were debugging executed like this:

1. Find a list of keys values to search for
2. Insert these keys in a temp table – lets call this the SearchFor table
3. Join the temp table to a large table (lets call it BigTable) and retrieve the full row from the large table

Why not use a correlated sub query in step 2? In this case, the customer in question had multiple code paths (including one accepting XML queries) that all needed to pass thousands of key to a final search procedure. They wanted a generic way to pass these key filters to to the final access of BigTable. The schema will make it clearer.

Schema

We created a repro and it turns out to be a great look into the strange world of database statistics. Here is the schema:

CREATE TABLE BigTable
(
  NearKey UNIQUEIDENTIFIER NOT NULL
  , Payload char(200) NOT NULL
  , RowType INT NOT NULL
)

/* Insert rows to simulate large table with 177 being the skewed value */

INSERT INTO Bigtable WITH (TABLOCK) (NearKey, PayLoad, RowType)
SELECT NEWID()
  , REPLICATE(‘X’, 200)
  , CASE WHEN n%100 BETWEEN 0 AND 50 THEN 177 ELSE n%100 END
FROM dbo.fn_nums(1000000)

/* Creates indexes and statistics */

CREATE INDEX IX_NearKey ON BigTable (NearKey)
CREATE STATISTICS STAT_RowType ON BigTable(RowType)

UPDATE STATISTICS BigTable WITH FULLSCAN

/* Insert rows that are NOT in the statistics with key = 199 */

INSERT INTO BigTable (NearKey, PayLoad, RowType)
SELECT NEWID(), ‘New Rows Not stat updated’, 199 /* New type*/
FROM fn_nums (100000)

/* Create generic search table */

CREATE TABLE #SearchFor (NearKey UNIQUEIDENTIFIER)

 

In Production, BigTable is a very large table with billions of rows. Each row in the table has RowType that describes metadata about the row. The metadata is used to populate #SearchFor before accessing BigTable, as this saves significant I/O.

Things to note about BigTable:

• All stats have been fully updated for values that are not RowType = 199. As long as we stay below this value, we have done ALL we can
• There is big skew in the RowType column.
• There are 100K values that are not in the statistics (you can run DBCC SHOW_STATISTICS (BigTable, STAT_RowType) WITH HISTOGRAM to convince yourself of this)

Query Problems

When a user queries this system, the code will first populate #SearchFor with a stored procedure like this:

Typical search procedures in the system look like this:

CREATE PROCEDURE GetDataA
AS

SELECT … INTO #SearchFor

EXEC dbo.ReturnFromBigTable

 

ReturnFromBigTable contains the troublesome statement that is the subject of this blog:

SELECT *
FROM BigTable
JOIN #SearchFor
  ON BigTable.NearKey = #SearchFor.NearKey
WHERE RowType = <SomeValue>

 

Looks quite innocent, doesn’t it?

To simulate the customer’s workload, let us say we are searching for 1% of the rows in BigTable. We do this by populating #SearchFor.

INSERT INTO #SearchFor
SELECT TOP 10000 NearKey FROM BigTable   
TABLESAMPLE (1 PERCENT)

And now, the fun begins! Lets first use our NearKey search and for a non skewed RowType:

SELECT * FROM BigTable
JOIN #SearchFor
  ON BigTable.NearKey = #SearchFor.NearKey
WHERE RowType = 99

Plan (only join shown)

Looking at Predicate in properties for the BigTable scan we unsurprisingly see that the predicate is being driven down: [dbo].[BigTable].[RowType]=(99)… Nothing interesting so far. The plan is parallel. Again, not surprising considering that we are scanning accessing 1M rows in BigTable.

… but as Lasse and I observed on his workload, something isn’t quite right here.

Let us run the same query, but with the filter RowType = 99 replaced by RowType = 177. The new plan is:

It is at this point your design intuition should wake you up with that good ol’ WTF feeling.  “Wait a minute you should say”…before (when there were fewer rows coming out of BigTable) we were parallel and hashing the output of BigTable. Now, when there are more (50x more) rows, we are both reversing the hash order, probing into BigTable and hashing SearchFor. Note that all estimates are relatively accurate (within 10%) and statistics in the space we are searching is fully up to date… If we take a fully naïve approach, trusting relational algebra and query optimizers, there is nothing more we can do here.

But hey, it gets worse: What happens when we go outside the boundaries of the statistics. This for example, could happen when the stats are not fully updated – a completely normal situation in a database. Searching with RowType = 199 (that is not in the histogram) and estimating the plan, we get:

The estimate coming out of BigTable is 1 row, the actual is 100K; this is probably going to hurt.

Running the query gives us the deadly:

Congratulations to us, we just committed performance suicide!

Lets just list the execution times of these query variants:

What is the problem here? From the user’s perspective, the query response time will often vary unpredictably – which is a poor experience. But worse, from a system and DBA perspective: the memory consumption will vary a lot depending on which query plan we end up with. This means it becomes hard to extrapolate the total system throughput and resource requirements based on the queries. If you are capitalist inclined in your mindset, let me translate this: At the end of the day, this costs you money!

Why is this happening? It is happening because there are hidden assumptions here that you have not brought into the light. We are trusting technology (in this case the database statistics and optimizer) with solving something that is a data modeling problem.

This is a very good example of something Thomas has been seeing a lot lately: An unwillingness to be specific about your requirements costs you money – a LOT of money. In other words: There is a price tag associated with ignorance. Let us make it clear what we are NOT saying: There is nothing wrong with ignorance: it is hard to know what you don’t know. Sometimes, you simply don’t know a requirement exists. The trouble arrives when denial and lack of curiosity and analysis trumps the observed data.

Fortunately, Lasse is the curious kind and he asked all the right questions and sought knowledge to remedy the situation.

Let’s get pragmatic, and talk about solutions.

The solution

With all the query plans above, which one do we want? In this case, the answer is: “None of them”

Here is an observation about the workload we discovered as we were analysing the solution: It is reasonable to assume ask the user to accept that response time is proportional with the number of results returned. From a system perspective, we also want the memory consumption and CPU to be proportional to the returned result size. These are questions you have to ask yourself as system designer and discussion you can have with users.

This is where computer science comes into the equation. We know:

1. That we can search a B-tree of n rows in lg(n) time (which is as good as constant)
2. That the the returned result are capped by the number of rows (let us call this R) in #SearchFor
3. That we are willing to accept that the runtime and resource use is proportional with number of rows returned
4. That we cant live with unpredictable behavior

The optimizer KNOWS about the search time and the cap (1+2). But the optimizer does NOT know that we are willing to declare certain properties that narrow the search spare (3) and that we want predictability (4). Our "ignorance” has a price, and the optimizer makes a choice that might be optimal, but which unfortunately is also unpredictable.

This is a great case for hinting. And we can do it like this:

SELECT *
FROM #SearchFor
INNER LOOP JOIN BigTable
  ON BigTable.NearKey = #SearchFor.NearKey
WHERE RowType = 177

Note something important here: The order of the joined tables matter when you use the LOOP hint (which is why we flipped them above). The above hint BOTH forces the join order and the join strategy.

An alternative, which may be more generally applicable is this rewrite:

SELECT *
FROM BigTable
INNER JOIN #SearchFor WITH (FORCESEEK)
ON BigTable.NearKey = #SearchFor.NearKey
WHERE RowType = 177
OPTION (LOOP JOIN)

The above rewrite does not guarantee join ordering. However, with the FORCESEEK hint, the plan we want is almost guaranteed.

Let us add some new rows to the result:

Above, we can see why the optimizer didn’t want to help us! The number of IOPS required for my hinted plan are much higher than the optimizer generated plans.

The hinted plan shape is:

Obviously, we could replace the non clustered index with a clustered on (which would get rid of the RID lookup) to make this query faster. But in this case, this was not a viable solution for other reasons.

Summary

In this blog, I have shown you an example of database optimizers failing to make the “right choice”. There are several reasons this is happening:

Lack of information: The database needs declarative information about the user’s intent and what we consider the Right Choice™. In the example, the missing information is: “the speed of the returned result should always be proportional with the result size”. We used a hint to control the optimizers behavior, using our knowledge of the physical execution of queries. Other people might have taken more extreme steps and argued for noSQL – perhaps too aggressive a step in this case.

Leaky abstractions: We saw the line between the “logical” and “physical” model break down. We believed that by building the right model, we had supplied enough information to the database. The distinction between physical and logical model has, as Thomas previously argued, always been a pretty useless paradigm. We have to consider the components of the system in a holistic way, but not at the expense of maintaining the overview of individual building blocks.

Optimality vs. Good enough: the optimizers default behavior is to look for “optimal plans”. In the plan search space, even a very small space like this example, plans are sometimes found that look optimal from statistics. Yet, these plans may not be optimal or have the properties we seek – or we may fail to recognize the good plans as we search the space. In our case, we just wanted a plan that was “good enough” and had certain predictable attributes.

False hardware assumptions: The optimizer makes some assumptions about the cost of IOPS which unfortunately do not correlate with the reality of a modern machine. It assumes that optimizing for a low number of Logical IOPS is a Good Thing™. In this light, the plans generated are the right plans. However, the optimizer does not take into account that the I/O system might be fast, and that there is enough RAM to have a likelihood of finding some of the needed data already residing in the buffer pool. To prove this: Recall that we had fully updated statistics available for queries going after RowType = 177. But, running the query on a petty IBM laptop the  loop hinted plan, even on an empty buffer pool, is still faster than running the plan generated by the optimizer.

The simple query we studied here raise some interesting questions about the design tradeoffs that the data modeler and architect is forced to make. In this case, we were fortunate that Lasse was on the alert for these tradeoffs. With this example, we have seen what the potential consequences of “behavior and model ignorance” are and why you need to be alert to it.