Our first script queries the servers for wait stats – a very common task for nearly all SQL Server DBAs. This is the first thing I do when I meet a new SQL Server installation and I have generally found that I apply the same filters over and over again. Because of this, I worked with Sumeet Bansal to standardise our approach to this.

You can watch Sumeet introduce the script in this YouTube video: http://youtu.be/zb4FsXYvibY. A TV star is born!

We have already used this script at several customers in a before/after Fusion-io situation. As you may know, the tuning game changes a lot when you remove the I/O bottleneck from the server.

Based on our experiences so far, I wanted to share some more exotic waits and latches we have been seeing lately.

WRITELOG

This is not exotic at all you may say. In fact, it is probably the most common wait type in most SQL Server installations.

Here is what you may not know: This wait does NOT measure the I/O latency of the log drive. To get that, you have to query the log file in sys.dm_io_virtual_file_stats – which you may notice often reports a lower wait time than WRITELOG. Just something to think about when measuring log latency as we see a larger and large discrepancy between this wait and the actual I/O wait.

I have provided more details on this in my SQLBits presentations:

• Finding the Limits (Presentation from SQLBits 9)

ACCESS_METHOD_DATASET_PARENT

We are seeing this latch wait a lot, especially in data warehouse environments. It is related to parallelism. While lowering MAXDOP is one way to remove it – it is not always the optimal way and you may not get the most out of your hardware if you do.

We typically see this wait happening in two different situations:

1. When doing table scans
2. When running INSERT…SELECT as part of an ETL process

In both cases, the wait indicates that you are hitting some form of scalability issue. My friend Henk van der Valk blogged about how to work around the table scan problem:

• Maximizing SQL Server 2008R2 table scan speeds

Henk used another type of non volatile storage: battery backed DRAM; the lesson he learned is equally applicable to Fusion-io.

I have written about optimising INSERT…SELECT in this whitepaper:

• The Data Loading Performance Guide

There is probably another blog entry to be written one day.

PAGELATCH_UP

Typically, we see this wait when a filegroup in a database doesn’t have enough files allocated. It is easy to diagnose: just look for PFS pages in the resource_description. Yes, this applies to user databases too, not just tempdb.

How many data files you should have in a database or filegroup has been debated a lot in the SQL Server community, even by me. When running on spindles (or spinning rust as we like to call it in Fusion-io) the answer is a big “it depends”. But on flash our experience and our benchmarking has shown that more is better, especially under high concurrency workloads.

Above: Runtime and PAGELATCH_UP waits when adding more data files. Notice the sharp drop from 1 to 8 and that the runtime curve continue to drop as you add more files

There is enough material for another blog entry (and for some nice new scripts) on this subject.

PAGELATCH_EX

Another common wait. Typically, this occurs in two situations:

1. When you issue UPDATE statements a lot against small tables
2. When you run many concurrent INSERT statements into a table that has an index on an IDENTITY or NEWSEQUENTIALID column
3. When using a table as the backing store for a message queue

Ad 1) The solution to situation 1 is normally to “pad” the table with a CHAR(4000) NOT NULL column to make every single row fit only one page. However, this is only viable on small tables – but those are the ones that tend to see this effect.

Ad 2) For high scale systems powered by flash memory, our recommendation for generating keys goes directly against the typical approach taken by the SQL Server community. DO NOT use IDENTITY, SEQUENCER or NEWSEQUENTIALID to generate keys. In fact, we have found that the old and despised  NEWID is significantly faster than NEWSEQUENTIALID. The numbers speak for themselves, here is the runtime of a workload that inserts 160M rows:

Above: Myth busting “best practises” for generating keys.

There is a very interesting solution to the key generation problem that has been blogged about by another friend of mine, here is the link to Rick’s very elegant key generator:

• Bit Reversion

Ad 3) I have previously blogged about the design pattern that solves this problem

• Implementing Message Queues in Relational Databases

Summary

As we install more and more Fusion-io into SQL Servers and remove the I/O problem, we continue to learn about new bottlenecks in typical customer workloads and how to solve them. Through our understanding of the full application stack, we are helping customers get more out of their new hardware. I run a mentoring program inside the company where our field unit, always eager for more knowledge, learn how to tune even very complex problems in SQL Server.

In Fusion-io, we live every day in the world beyond the I/O problem, and we are privileged to be the first to witness the disruption NAND flash is creating in the market. We are especially excited about Project Hekaton and how that is going to change the game once again. As we explore Hekaton, we will adjust our guidance. The SQL community has a lot to look forward to from Microsoft and we look forward to continuing to provide acceleration in this space.

I would like to thank my colleagues in Fusions Data Propulsion Labs for helping making all this come together – especially Sumeet Bansal the new face of our SQL Server YouTube series.

The presentation is a 45 minute show, but for those of you who were not there, here is a summary of the highlights of the first part.

My tests were all run on the TPC-H LINEITEM table at scale factor 10. That is about 6.5GB of data.

Test: Table Scan of Compressed Data

My initial test is this statement:

SELECT MAX(L_SHIPDATE)
, MAX(L_DISCOUNT)
, MAX(L_EXTENDEDPRICE)
, MAX(L_SUPPKEY)
, MAX(L_QUANTITY)
, MAX(L_RETURNFLAG)
, MAX(L_PARTKEY)
, MAX(L_LINESTATUS)
, MAX(L_TAX)
, MAX(L_COMMITDATE)
, MAX(L_RECEIPTDATA)
FROM LINEITEM

Because the statement only returns one row, the result measured does not drown in client transfer time. Instead, the raw cost of extracting the columns from the compressed format can be quantified.

The result was quite shocking on my home 12 core box:

Even when doing I/O, it still takes quite a bit longer to scan the compressed format. And when scanning from DRAM, the cost is a whopping 2x.

A quick xperf run shows where the time goes when scanning from memory

Indeed, the additional CPU cost explains the effect. The code path is simply longer with compression.

Test: Singleton Row fetch

By sampling some rows from LINEITEM, it is possible to measure the cost of fetching pages in an index seek. This is the test:

SELECT MAX(L_SHIPDATE)
, MAX(L_DISCOUNT)
, MAX(L_EXTENDEDPRICE)
, MAX(L_SUPPKEY)
, MAX(L_QUANTITY)
, MAX(L_RETURNFLAG)
, MAX(L_PARTKEY)
, MAX(L_LINESTATUS)
, MAX(L_TAX)
, MAX(L_COMMITDATE)
, MAX(L_RECEIPTDATA)
FROM LI_SAMPLES S
INNER LOOP JOIN LINEITEM L ON S.L_ORDERKEY = L.L_ORDERKEY
OPTION (MAXDOP 1) 

This gives us the plan:

Which has the desired characteristics of having the runtime dominated by the seek into the LINEITEM table.

The numbers again speak for themselves:

And again, the xperf trace shows that this runtime difference can be fully explained from longer code paths.

Test: Singleton UPDATE

Using the now familiar pattern, we can run a test that updates the rows instead of selecting them. By updating a column that is NOT NULL and an INT, we can make sure the update happens in place. This means we pay the price to decompress, change and recompress that row – which should be more expensive than reading. And indeed it is:

Summary

Quoting a few of my tests from my  presentation, I have shown you that PAGE compression carries a very heavy CPU cost for each page access. Of course, not every workload is dominated by accessing data in pages – some are more compute heavy on the returned data. However, in these days when I/O bottlenecks can easily be removed, it is worth considering if the extra CPU cycles to save space are worth it.

It turns out that it is possible to also show another expected results: that locks are held longer when updating compressed pages (Thereby limiting scalability if the workload contains heavily contended pages). But that is the subject of a new blog entry.

Tuning/Scaling Course

The tuning course will be hosted by CrossJoin Consulting in London, UK You can sign up for it here:

• Tuning Course in London (13th June 2013)

You can still make it for the early bird rates. This course is probably THE deepest level course about SQL Server out there. You will learn the nitty-gritty details of xperf profiling, spinlock detection and multi-threaded optimisation.

We are going to be digging deep below the surface of SQL Server here, be prepared for an intensive day.

Data Warehousing Course

The next data warehousing course will be in Kista, Sweden. It will be hosted by SolidQ and you can sign up for it here:

• Data Warehousing in Sweden (7th February 2013)

This will be a very unique course with both Davide Mauri and myself teaching. Its a one day intensive training in my usual “no nonsense” style where you learn about data modeling from two of the leading experts in the field. The course extends the ideas I have blogged extensively about here.

There are still seats left, but they are going fast.

I look forward to seeing you there!

As described in a previous post, I am have set up my laptop to host Windows in a virtual machine guest with Mac OX (Mountain Lion) as the host.

Today, I needed to compile some code and thinking I was being clever, I put the code on the host OS (OSX) and shared the folder via VM-ware to the Windows guest OS.

Compiling from inside VM-ware

I ran my msbuild process from the shared folder, and it took FOREVER to compile. The obvious choice here is of course to blame virtualisation itself – after all, I only have 2 cores allocated to the virtual.

But not so fast! Our tuning knowledge comes in quite handy here. Have a look at the CPU pattern while I am compiling:

Just like with SQL Server, I start my tuning at a very high level (in this case, task manager) and dig in from there.

The first question we ask ourselves as tuners is: Does what I see make sense?

In this case, it obviously doesn’t. MSBUILD is set up to build highly parallelised, it should be using my cores and there are no obvious I/O bottleneck in the system. Having 50% of two cores busy (and with high kernel times) looks a lot like a single threaded bottleneck to me. The build was taking over 15 minutes, which was much longer than expected.

Diagnosing the Problem – our friend Xperf

Normally, I use xperf to troubleshoot servers. But it sure comes in handy for misbehaving client machines too.

Task manager only shows that the time is spend in the process d.exe – which is part of the build process. Is the compiler bad or must we look elsewhere? Sure would be surprising if the compiler used all the kernel time wouldn’t it?

Here is the quick and dirty CPU “zoom in” xperf command to get the details we need:

• xperf –on latency –stackwalk profile
• …wait a bit
• xperf –d <myfile>.etl

This captures a sample of the stack and CPU usage of each process and kernel module. From here, it is quite clear what is going on – let me walk you through the analysis.

First, open the trace with xperfview. I recommend staying with the Win7 version of xperfview, as the Win8 interface is… well… a Win8 interface.

Pick the CPU Sampling per CPU, right click and choose Summary Table:

From here, pick the columns: Module, CPU and % Weight which allows you to summarise by module. On my box, it looks like this:

Aha!… Most of the CPU burn goes in vmci.sys (just ignore intelppm.sys). This isn’t a part of Windows. Its relatively easy to trace this file back to VM-ware.

So, who calls into this kernel module? Adding the stack column after the module, we can see that too:

Eureka: It is file system access that is causing the slowdown. See the call stack? Starts from GetFileAttributesW and ends up inside vmci.sys.

Fixing the problem

Now, before you go ahead and conclude that VMware adds a horrible overhead to I/O, lets just try to move the source files into the guest OS itself. Recall that my machine is using VM-ware shared folders to access the source code. It might simply be the sharing framework that is acting strange…

The results of using the guest OS’s file system is staggering. Running the build process now looks like this at the CPU level:

And the total build time is down from over 15 minutes to less than 3 minutes.

Thank you xperf…

At this point, we are reduced to guessing what is going on

Once you start playing around with the Windows Kernel, you will at some point need kernel level debugging set up. Traditionally, this is something I have used a Windows machine for, and even there, it can be painful to get working. As you may know, I have switched to Mac as my client machine and my Windows utilities (including WinDbg) now run in VMWare Fusion – it looked like I was heading into an interop nightmare…

Windows 8 allows kernel debugging directly over the network card, but how does one configure kernel debugging with a Windows 2008R2 target from a Mac with VM Ware?

I found a very cheap solution today. You need:

• A USB to Serial (RS-232) converter
• A NULL modem (I would recommend getting a long one, so you don’t have to sit next to the server)
• A  target server that you want to debug
• A serial port on the target server
• WinDbg from the Windows SDK in a virtual machine on the Mac
• Symbols set up as per my previous post

A USB converter and a NULL modem is a dirt cheap way to get the required hardware for kernel debugging. I sourced my cables from Maplins (Thanks @SQLServerMonkey) in the UK – for a total of around 20 GBP.

Step 1: Prepare the Client/Debugger

A Macbook air, like most other lightweight laptops, does not have a serial port. So, we have to use a USB/Serial converter. It is possible to debug directly over USB, but good luck with that from a Mac, I didn’t have the courage.

Make sure VMWare routes the USB device to the Windows Guest OS and not the Mac. It will look something like this in VMWare Fusion 5:

Now, install WinDbg and set up your symbol paths if they are not set up already.

Check your Device Manager in the client to see which COM port the USD device created. As you can see below, my laptop mounted the USB/Serial converter as COM3 

After installing the device, I had to restart my virtual machine before VMWare would let me mount it – but what can you expect from a 10 GBP component? Your mileage may vary depending on the serial/USB driver you have.

Step 2: Connect Client and Server

Using the NULL modem, connect the USB/Serial converter to the server’s Serial port.

Make sure the server has the serial port enabled in the BIOS (my Dell box had it disabled, had to re-enable).

Step 3: Configure Server/Debugee for kernel debugging

Log into the server, start a command line as administrator

First, copy the default startup options into a new boot option:

• BCDEDIT /copy {current} /d DebugMode

This will create a new entry in the boot list when the server starts. Make a note of the GUID returned or copy it to the clipboard (using the ever so annoying copy/paste function in the Windows command prompt)

Next, configure the parameters for serial cable debugging:

• BCDEDIT /set <GUID> debugport <port #>
• BCDEDIT /set <GUID> debugtype serial
• BCDEDIT /set <GUID> baudrate 115200

Replace the <GUID> the guid returned previously. Set <port #> to the COM port the NULL modem is connected to in your server (NOT the COM port of the client). For example, if the server has the NULL modem in COM2, set <port #> to 2.

Finally, enable debugging for on the newly created boot option:

• BCDEDIT /debug <GUID> ON

Validate that your configuration works by running:

• BCDEDIT /v.

You should get an output somewhat like this (this is also how you find the GUID if you didn’t note it down before):

If you want to make sure debugging is always turned on (great for a sandbox machines where you explore stuff in the kernel) you can use BCDEDIT /default <GUID> to make debugging the default startup option.

Step 3: Start Debugging

You are now ready to start debugging the windows kernel on the server. Here is how:

On the client, start WinDbg and choose File—>Kernel Debug (or CTRL+K) and set up the com port you got in step 1:

.

Press OK, and reboot the server. If you didn’t select the debug configuration as the default boot option, make sure you pick it when the server starts.

If you have done things right, you will get something like this in WinDbg (below, I broke execution with CTRL+C)

One thing to note when you are debugging the kernel: Not all your typical WinDbg commands work as they normally do (for some good reasons), but that is outside of scope for this blog entry.

Time to dig in even deeper… my Macbook Air to a Windows box …Happy hacking everyone.

• http://www.slideshare.net/tkejser1

Some things I put there so far:

• The Fusion Fireside talk I did at SQL Bits (which shows SQL based message queues in action)
• My Big Data vs. DW deck (as seen at SQL PASS 2012 and other conferences)
• A deck I created to introduce storage people to databases

Feel free to use the material, I only ask that you let people know where you got it from.

My old slide decks belongs to MS as their IP, so I can’t share those slides with you. However, you will often find some of that material is available for download at the conference site where I presented it. For example, you can find a lot of my presentations on the SQL Bits website here:

• http://www.sqlbits.com/Speakers/Thomas_Kejser

Next week, I will be presenting at IP Expo about the Fusion-io SDK initiative. Exciting times.

Below is the screenshot to prove it:

Think for a moment about what it will take to actually make use of all those IOPS. If this is the type of speed you have at your disposal, maybe it is time to rethink what is possible. Check out our SDK at http://developer.fusionio.com for the leading edge work Fusion-io is doing in this space.

As I am sure my regular readers  can guess, I am loving my new job!

The course will be hosted by Orangeman, who will be handling the logistics. There are still available spots and if you act fast, you can get the early bird price.

The relevant details:

• 23rd October 2012: Tuning Course
• 24th October 2012: Data Warehouse modeling

See you there for some intense days of tuning and modeling.

As many of you know, I have worked together with FusionIo on many occasions  and really enjoyed the collaboration. I believe that their products hold the keys to a new era of computing and it is an honour to join their ranks. I will be looking forward to doing a lot of exciting research and customer implementations for them.

This brings me to the work I have been doing since I left Microsoft. Here is how it will transfer:

Consulting Contracts

I have contracts with some customers open. These are all due to terminate before 1st September and I will of course honour my agreements here. Unfortunately, my new job will not allow me to continue the collaboration with these customers on a consulting basis after this. The good news is that my courses will still be available and I will be able to share my knowledge through this channel.

Courses and Conferences

Contributing information to the community is one of my great passions in life. FusionIo has allowed me to continue to pursue this interest. My courses will still be available, although only for a very limited amount of days every month as the course time will be coming out of my vacation days (hint on how to get a discount). I expect demand to be high. There are already  three tuning courses set up across Europe which will be held as planned and a lot of people have made it clear they want more. I will be announcing the exact dates for courses planned on this blog soon and let you know how to join the courses that are open to the public. The material is looking amazing and is using the new format that has evolved at SQL BITS and driven the top scores there. I expect this will be my best presentations yet. I am also happy to announce that my data modelling course is well underway and will be available soon.

I will continue to submit abstracts for conferences and stay in close touch with the community, just like I have always done. And  this brings me to:

Grade of the Steel

I am very excited that FusionIo has an interest in expanding the testing I have done with my Grade of the Steel Project. I will continue to run benchmarks on the latest and greatest storage and provide non volatile memory specific configuration and tuning guidance. Exactly which format the publications will take is too early to say, I will keep you posted on this blog.

To approach my theory, I am afraid we are going to have to dig into some concepts from information theory.

Shared Context

Compression works by eliminating superfluous information in a stream of data. Unfortunately, it is rather hard to grasp exactly what information is superfluous – as this depends not only on the representation of the data, but also on how much context is shared between the sender and receiver.

The best way I have found to describe this problem is the story about the author who, after having sent his new hit novel to the publisher for publication, went on vacation. After a week, the author sends the following postcard to the publisher from his vacation resort:

The publisher, without hesitation replies:

We all know what this information exchange conveys, because we know the context of the publisher’s and author’s conversation. Very little, if any, information is lost even though the amount of data transmitted is as low as it gets (a single byte).

Unfortunately, we are rarely in a situation where the shared information between sender and receiver is as large as this. We have to find a different way to quantify how much data can be thrown away while preserving the information content.

Side note: The above story is not my own but borrowed from somewhere. Unfortunately, I fail to recall where I first read about it  (it may have been Steven Pinker). If you know, I would appreciate if you could let me know the source so I can give proper credit.

Low Entropy ≈ High Compression

Formalising the information content of a data transmission turns out to be a hard concept to wrap your head around. Fortunately, this has not prevented both computer scientists, mathematicians and statisticians from making progress in this area.

Claude Shannon, working on signal compression first came up with the notion of Information Entropy. You can read more in articles I will link throughout this blog.  They describe the theory more elaborately than I can.

Roughly speaking the Entropy H(X) of a random variable, for our purposes the content of a database column, is the amount of uncertainty associated with given value of that column. Entropy is typically measured in bits. If the entropy of a column is low, we would expect that there are either very few values in the column or that they are unevenly distributed somehow. The maximum entropy of a column is achieved on keys and it goes down after after that. 

Entropy of a column, X, is calculated as:

Where p(x_i) is the probability of each value the column takes. We can calculate all these probabilities with SQL like this:

SELECT X
  , CAST(COUNT(*) AS FLOAT) / SUM(COUNT(*)) OVER (PARTITION BY NULL) pX
FROM Table
GROUP BY X

And we can get H(X) like this:

SELECT -1 * SUM(pX * LOG(pX, 2)) AS H
FROM
(
    SELECT X
    , CAST(COUNT(*) AS FLOAT) / SUM(COUNT(*)) OVER (PARTITION BY NULL) pX
    FROM Table
    GROUP BY X
) AS p

Think of H(X) as an upper boundary for how much we can theoretically compress the column. This should also tell you why you cannot tell in advanced how well a compression algorithm will work until you have actually seen or fully described the data – anyone who claims otherwise are bending the truth.

In summary we can use the entropy as a rough estimate of how many bits it would take to fully describe the column and it also roughly correlates to the number of repetitions in the column (which is what we are after in column stores). So far so good…

Mutual Information I(X;Y) and Variation of Information d(X,Y)

With entropy, we can get a rough ranking of columns by their compressibility. This is generally useful, even for page/dictionary compression. In other words, entropy allows us to come up with good candidates for columns to lead a clustered index on a fact table.

However, in column stores, we want to take this one step further. We are interested in how columns relate to each other, so we can group similar columns together during the index build phase and achieve even fewer run length and higher compression. This eventually translates to more query performance and space savings.

Fortunately, information theory provides us with an estimate of this, namely: mutual information. For two columns, it is defined as:

p(x,y) is the empirical measured frequency of each column combination, divided by the total number of combinations of X and Y. We can extend the SQL statement above to this:

SELECT X, Y
, CAST(COUNT(*) AS FLOAT) / SUM(COUNT(*)) OVER (PARTITION BY NULL) pXY
FROM Table
GROUP BY X, Y

And by joining up the calculated probabilities from the Entropy calculation we can find I(X;Y). We are almost there, bear with me a little longer. The value I(X;Y) tells us something about how much information (again, measured in bits) the column share. Combined with H(X) we can now calculate the “information distance” between two columns. This is know as the Variation of Information and is denoted by d(X,Y):

At this point, it should be clear that it is possible calculate a matrix like this:

Think of this matrix as the information distance between each column, a map of the entropic landscape if you wish.

Side note: Unfortunately, this matrix is rather computer intensive to build. It takes O(n * c²) space and time to calculate I(X,Y) for all column combinations, with c being the column count in the table and n being the row count. It also requires some interesting, high speed data structures (may be subject of another blog) that I had to build myself because .NET 4 “concurrent” dictionary was not up to the task. I am currently looking for a way to estimate I(X,Y) instead of brute forcing the exact number. Ideas on how to approach this will be most appreciated.

Why Optimal is Hard

With our d(X,Y) matrix, we may be able to find a path through the information landscape that minimises the amount of shared information of columns. Intuitively (?) this should increase column store compression by minimising the number of run lengths.

At this point, I have to make a few disclaimers. What I am about to propose is NOT an exact solution, in fact, it may even be worse than guessing in some cases. There are a couple of reasons for this

First of all, just because I know the mutual information I(A;B) and I(B;C) it does NOT follow that I know the total mutual information of the combination represented by I(A;B;C). For example, there could be an underlying structure in the data that is not detected by comparing column pairs. Such an underlying structure could send the following algorithm off into a walk goose chase.

Second, it may not even be true that lexicographical ordering of the column values leads to optimal compression. There are other ways to sort columns, for example in Gray Cycles.

Last, and most important: finding the "shortest path through the information landscape” turns out to be an NP-hard problem. This has been proven here: "Reordering Columns for Small Indexes”. The problem has even been baptised: RUNLENGTH.

What does it mean that a problem is NP-hard? Consider for a moment our reordering of columns problem. There are  c! ways to order the c columns in a table. This is a huge number ( c! = c * (c-1) * … * 1). We cannot explore them all in reasonable time, so we need an algorithm that finds the best one without looking at every combination. When a problem is NP-hard, it means that such an algorithm is known to either NOT exist, or if you can find it, you will win the Nobel price (because you would then have proven the great mystery of computer science, namely the P = NP problem).

Can We Make a Good Guess?

When a problem is NP-hard, our best bet is to come up with good heuristics. With the d(X,Y) matrix available, one particular heuristic lends itself well to easy implementation:

• Let S be an ordered set of columns
• Pick a starting column c1 (the one with lowest entropy H(X) is often good)
• Add c1 to S
• From the columns not yet in S, find the one that has the smallest distance to any column in S. Add this to S
• Repeat above until all columns are explored

Intuitively, we are greedily traversing the “landscape of information”, always adding the column that is closest to the part of the landscape we have already explored.

Early Test Results

With the help of readers on this blog, I have managed to collect sample datasets from four retailers. Running my algorithm above, I can now quantify how well it does on these datasets. The following table lists the results:

A bit more detail about the algorithms here

Cardinality sorts the columns either by lowest cardinality first (ASC) or highest cardinality first. Notice that this heuristic actually makes compression WORSE than random in some cases. Time complexity is (n * c) and interestingly, the space complexity can be done in O(c) if using HyperLogLog to estimate the cardinality.

H(X) Entropy sorts the columns by either increasing or decreasing entropy. On the tested datasets, this algorithm fare somewhat better. It also had the advantage of being very fast to calculate: O(n * c)

d(X,Y) Greedy is the algorithm described above (which is very slow to calculate: O(n * c²)

Random is random ordering of the rows.

All data sizes have been measured using the SQL Server 2012 column store compression

Introducing TableStat.exe

To compute the test results above in reasonable runtime, I have written a little utility in .NET 4 called TableStat.Exe.

This utility can run on a sample and assuming there are not too many columns, crunch through tens of thousands of rows every second to calculate all the algorithms described above. When it is done running, it will output the SQL statements required to run the experiments to measure compression. Optionally, it will even run the experiment for you.

I would really like to collect more data on my candidate heuristics. If you have some real data that you would like to test on, I can get you a copy of TableStat.Exe. I don’t even need to see your data (the test can be run fully on your site), but I would ask that you share the compression results with me.

Send me an email, or comment on this blog, if this has your interest.