In my previous blog entry, I analyzed how various tools (ps and smem) report memory usage. In summary:

• ps columns TRS, DRS, and VSZ report virtual address space allocated, not actual RAM allocated.
• smem’s USS reports a process’s private (unshared) memory allocated.
• smem’s PSS is a sum of process’s private memory allocated and a proportional amount of shared memory (both System V shared memory, like Postgres’s shared_buffers, and shared libraries).
• RSS shows actual RAM allocated, private and shared.

With these issues understood, let’s look at a running Postgres cluster:

Continue Reading »

Memory is very important to databases — much more so than for typical applications (presentation). Unfortunately, because memory allocation is so
complex, it is often hard to figure out how physical RAM is being used. There are several reasons for the complexity:

• Virtual Memory: CPUs in virtual memory mode don’t access RAM directly, but rather through page tables.
• Segmentation: Memory is allocated in specific segments:
  text(code), data, and stack.
• Sharing: physical RAM is often shared by multiple processes, either in read-only mode (program instructions), shared mode (read/write of share memory), or copy-on-write (create a new copy on write; used by fork).

Robert Haas’s excellent blog post highlighted much uncertainty about how to analyze memory usage for specific processes, especially Postgres. I commented on his blog, as did others, and now have a much clearer idea of how to study memory usage. A blog post by Chris Siebenmann directly addresses some of my and Robert’s questions, and suggests smem as a way to analyze memory, especially the sharing of memory. It was interesting to learn that smem was designed specifically to address the problems Robert outlined (2007, 2009).

Continue Reading »

This video presentation by Dwight Merriman (MongoDB) at OSCON Data 2011 explores the role of NoSQL databases. He makes some very interesting points:

• NoSQL was created to allow horizontal, read-write scaling
• NoSQL uses a dynamic schema
• NoSQL is similar to dynamically-typed languages, e.g. Perl, PHP
• NoSQL has serious limitations compared to relational data storage, e.g. reporting

What I found most interesting about his presentation was the comparison to dynamically-typed languages. When dynamically-typed scripting languages started to become popular years ago, there was a sense that they were only for trivial applications, compared to compiled language that were for serious work. As CPUs became faster, scripting languages were increasingly used for production applications, particularly web applications that are modified frequently. There might be a similar pattern now in the (improper?) dismissal of NoSQL for serious applications.

Continue Reading »

The term “scalability” gets injected into database conversations often, but scalability discussions often end without a clear consensus. I believe this is because “scalability”, like “replication”, actually means different things to different people.

There are two dimensions to database scalability; the first dimension specifies if additional servers are involved:

• vertical: The hardware capabilities of a single database server are increased to increase throughput
• horizontal: The number of a database servers is increased to increase throughput

Continue Reading »

While database software can be the cause of outages, for Postgres, it is often not the software but the hardware that causes failures — and storage is often the failing component. Magnetic disk is one of the few moving parts on a computer, and hence prone to breakage, and solid-state drives (SSDs) have a finite write limit.

While waiting for storage to start making loud noises or fail is an option, a better option is to use some type of monitoring that warns of storage failure before it occurs, e.g. enter SMART. SMART is a system developed by storage vendors that allows the operating system to query diagnostics on the drive and warn of unusual storage behavior before failure occurs. While read/write failures are reported by the kernel, SMART parameters often warn of danger before failure occurs. Below is the SMART output from a Western Digital (WDC) WD20EARX magnetic disk drive:

Continue Reading »

Postgres consists of roughly 1.1 million lines of C code, which is compiled into an executable with millions of CPU instructions. Of the many CPU machine-language instructions in the Postgres server executable, which one is the most important? That might seem like an odd question, and one that is hard to answer, but I think I know the answer.

You might wonder, “If Postgres is written in C, how would we find the most important machine-language instruction?” Well, there is a trick to that. Postgres is not completely written in C. There is a very small file (1000 lines) with C code that adds specific assembly-language CPU instructions into the executable. This file is called s_lock.h. It is an include file that is referenced in various parts of the server code that allows very fast locking operations. The C language doesn’t supply fast-locking infrastructure, so Postgres is required to supply its own locking instructions for all twelve supported CPU architectures. (Operating system kernels do supply locking instructions, but they are much too slow to be used for Postgres.)

Continue Reading »

The new server is 2-10 times faster than my old 2003 server, but that 10x speedup is only possible for applications that:

• Do lots of random I/O, thanks to the SSDs. Postgres already supports tablespace-specific random_page_cost settings, but it would be interesting to see if there are cases that can be optimized for low random pages costs. This is probably not an immediate requirement because the in-memory algorithms already assume a low random page cost.
• Can be highly parallelized. See my previous blog entry regarding parallelism. The 16 virtual cores in this server certainly offer more parallelism opportunities than my old two-core system.

Other observations:

• It takes serious money to do the job right, roughly USD $4k — hopefully increased productivity and reliability will pay back this investment.
• I actually started the upgrade two years ago by adjusting my scripts to be more portable; this made the migration go much smoother. The same method can be used for migrations to Postgres by rewriting SQL queries to be more portable before the migration. Reliable hardware is often the best way to ensure Postgres reliability.
• My hot-swappable SATA-2 drive bays allow for a flexible hard-drive-based backup solution (no more magnetic tapes). File system snapshots allow similar backups for Postgres tablespaces, but it would be good if this were more flexible. It would also be cool if you could move a drive containing Postgres tablespaces from one server to another (perhaps after freezing the rows).

Continue Reading »

A few weeks ago, I finally replaced my eight-year-old home server. The age of my server, and its operating system, (BSD/OS, last officially updated in 2002) were a frequent source of amusement among Postgres community members. The new server is:

Super Micro 7046A-T 4U Tower Workstation
2 x Intel Xeon E5620 2.4GHz Quad-Core Processors
Crucial 24GB Dual-Rank PC3-10600 DDR3 SDRAM
Intel 160GB 320 Series SSD Drive
4 x Western Digital Caviar Green 2TB Hard Drives

Continue Reading »

As a followup to my previous blog entry, I want to show queries that allow users to analyze TOAST tables. First, we find the TOAST details about the test heap table:

SELECT oid, relname, reltoastrelid, reltoastidxid FROM pg_class where relname = 'test';
  oid  | relname | reltoastrelid | reltoastidxid
-------+---------+---------------+---------------
 17172 | test    |         17175 |             0

Continue Reading »

Many things are better toasted: cheese sandwiches, nuts, marshmallows. Even some of your Postgres data is better toasted — let me explain.

Postgres typically uses an eight-kilobyte block size — you can verify this by running pg_controldata:

Continue Reading »