Open Enterprise: The PostgreSQL Open Source Database Blog from EnterpriseDB

Postgres Memory Surprises

Bruce Momjian — Wed, 01 Feb 2012 14:15:01 +0000

In my previous blog entry, I analyzed how various tools (ps and smem) report memory usage. In summary:

ps columns TRS, DRS, and VSZ report virtual address space allocated, not actual RAM allocated.
smem’s USS reports a process’s private (unshared) memory allocated.
smem’s PSS is a sum of process’s private memory allocated and a proportional amount of shared memory (both System V shared memory, like Postgres’s shared_buffers, and shared libraries).
RSS shows actual RAM allocated, private and shared.

With these issues understood, let’s look at a running Postgres cluster:

Revisiting Memory Reporting

Bruce Momjian — Mon, 30 Jan 2012 16:00:00 +0000

Memory is very important to databases — much more so than for typical applications (presentation). Unfortunately, because memory allocation is so
complex, it is often hard to figure out how physical RAM is being used. There are several reasons for the complexity:

Virtual Memory: CPUs in virtual memory mode don’t access RAM directly, but rather through page tables.
Segmentation: Memory is allocated in specific segments:
text(code), data, and stack.
Sharing: physical RAM is often shared by multiple processes, either in read-only mode (program instructions), shared mode (read/write of share memory), or copy-on-write (create a new copy on write; used by fork).

Robert Haas’s excellent blog post highlighted much uncertainty about how to analyze memory usage for specific processes, especially Postgres. I commented on his blog, as did others, and now have a much clearer idea of how to study memory usage. A blog post by Chris Siebenmann directly addresses some of my and Robert’s questions, and suggests smem as a way to analyze memory, especially the sharing of memory. It was interesting to learn that smem was designed specifically to address the problems Robert outlined (2007, 2009).

NoSQL Databases as the New Scripting Language

Bruce Momjian — Fri, 27 Jan 2012 16:00:00 +0000

This video presentation by Dwight Merriman (MongoDB) at OSCON Data 2011 explores the role of NoSQL databases. He makes some very interesting points:

NoSQL was created to allow horizontal, read-write scaling
NoSQL uses a dynamic schema
NoSQL is similar to dynamically-typed languages, e.g. Perl, PHP
NoSQL has serious limitations compared to relational data storage, e.g. reporting

What I found most interesting about his presentation was the comparison to dynamically-typed languages. When dynamically-typed scripting languages started to become popular years ago, there was a sense that they were only for trivial applications, compared to compiled language that were for serious work. As CPUs became faster, scripting languages were increasingly used for production applications, particularly web applications that are modified frequently. There might be a similar pattern now in the (improper?) dismissal of NoSQL for serious applications.

Scalability What?

Bruce Momjian — Thu, 26 Jan 2012 17:00:01 +0000

The term “scalability” gets injected into database conversations often, but scalability discussions often end without a clear consensus. I believe this is because “scalability”, like “replication”, actually means different things to different people.

There are two dimensions to database scalability; the first dimension specifies if additional servers are involved:

vertical: The hardware capabilities of a single database server are increased to increase throughput

horizontal: The number of a database servers is increased to increase throughput

Cloud Computing Takes Another Step

Sean Doherty — Wed, 25 Jan 2012 21:05:30 +0000

As enterprises embark down the path of cloud computing we?re realizing that with many new advancements in technology, the promise of the technology is often more grand than the reality. It usually takes more time than we think it will …

Postgres Plus Cloud Database

Ed Boyajian — Wed, 25 Jan 2012 16:27:40 +0000

Today marks another important milestone for EnterpriseDB and PostgreSQL as we launch Postgres Plus Cloud Database. Our team of extraordinarily bright developers has been working for over a year with customers, community teammates and partners to create a “cloudified” version of Postgres that adds true elastic properties to the database while automating many typical DBA routines. All of this is delivered through a great looking centralized dashboard for monitoring, management and administration and it’s now available on Amazon EC2.

What does “cloudified” mean for a database? It’s the addition of technology that allows you to:

Automatically scale in response to increased demand
Set up highly available multi-node database clusters in minutes, not days
Pay for your database as if it were a utility

You can read more about Postgres Plus Cloud Database here:
http://cloud.enterprisedb.com

As we looked at the landscape of available cloud databases, four things became very clear:

Before the launch of Postgres Plus Cloud Database, only Micrsoft SQL Server (solely via SQL Azure) and Oracle’s MySQL (solely via Amazon RDS) offered enhanced versions of a relational database to take advantage of inherent cloud scaling technology.
PaaS and other cloud platform vendors needed a database vendor to deliver a valuable cloud compatible database component for their platform to bring maximum support, features and value to their customers.
NoSQL options are varied and will continue to grow AND (not OR) old-school SQL RDBMS are as highly in demand in the cloud as they are in the data center.
Just putting “cloud” in the name does NOT make a database “cloudified”

We are changing all that with this product launch. We are giving users a new option to run either the latest community version of PostgreSQL or our flagship Postgres Plus Advanced Server with Oracle database compatibility features on Amazon EC2. Perhaps more important is that we have architected Postgres Plus Cloud Database to be universally pluggable with any PaaS vendor platform. To that end, our work with leading platforms from CloudBees and Engine Yard will bring enhanced database capabilities to the growing number of developers adopting their technologies. And we are extending that work to include OpenStack, HP, Red Hat’s OpenShift and many others, making Postgres Plus Cloud Database the new standard relational database for all cloud platforms.

Increasing Database Reliability

Bruce Momjian — Wed, 25 Jan 2012 13:15:01 +0000

While database software can be the cause of outages, for Postgres, it is often not the software but the hardware that causes failures — and storage is often the failing component. Magnetic disk is one of the few moving parts on a computer, and hence prone to breakage, and solid-state drives (SSDs) have a finite write limit.

While waiting for storage to start making loud noises or fail is an option, a better option is to use some type of monitoring that warns of storage failure before it occurs, e.g. enter SMART. SMART is a system developed by storage vendors that allows the operating system to query diagnostics on the drive and warn of unusual storage behavior before failure occurs. While read/write failures are reported by the kernel, SMART parameters often warn of danger before failure occurs. Below is the SMART output from a Western Digital (WDC) WD20EARX magnetic disk drive:

The Most Important Postgres CPU Instruction

Bruce Momjian — Tue, 24 Jan 2012 13:30:01 +0000

Postgres consists of roughly 1.1 million lines of C code, which is compiled into an executable with millions of CPU instructions. Of the many CPU machine-language instructions in the Postgres server executable, which one is the most important? That might seem like an odd question, and one that is hard to answer, but I think I know the answer.

You might wonder, “If Postgres is written in C, how would we find the most important machine-language instruction?” Well, there is a trick to that. Postgres is not completely written in C. There is a very small file (1000 lines) with C code that adds specific assembly-language CPU instructions into the executable. This file is called s_lock.h. It is an include file that is referenced in various parts of the server code that allows very fast locking operations. The C language doesn’t supply fast-locking infrastructure, so Postgres is required to supply its own locking instructions for all twelve supported CPU architectures. (Operating system kernels do supply locking instructions, but they are much too slow to be used for Postgres.)

More Lessons From My Server Migration

Bruce Momjian — Mon, 23 Jan 2012 14:30:02 +0000

The new server is 2-10 times faster than my old 2003 server, but that 10x speedup is only possible for applications that:

Do lots of random I/O, thanks to the SSDs. Postgres already supports tablespace-specific random_page_cost settings, but it would be interesting to see if there are cases that can be optimized for low random pages costs. This is probably not an immediate requirement because the in-memory algorithms already assume a low random page cost.
Can be highly parallelized. See my previous blog entry regarding parallelism. The 16 virtual cores in this server certainly offer more parallelism opportunities than my old two-core system.

Other observations:

It takes serious money to do the job right, roughly USD $4k — hopefully increased productivity and reliability will pay back this investment.
I actually started the upgrade two years ago by adjusting my scripts to be more portable; this made the migration go much smoother. The same method can be used for migrations to Postgres by rewriting SQL queries to be more portable before the migration. Reliable hardware is often the best way to ensure Postgres reliability.
My hot-swappable SATA-2 drive bays allow for a flexible hard-drive-based backup solution (no more magnetic tapes). File system snapshots allow similar backups for Postgres tablespaces, but it would be good if this were more flexible. It would also be cool if you could move a drive containing Postgres tablespaces from one server to another (perhaps after freezing the rows).

New Server

Bruce Momjian — Fri, 20 Jan 2012 13:30:01 +0000

A few weeks ago, I finally replaced my eight-year-old home server. The age of my server, and its operating system, (BSD/OS, last officially updated in 2002) were a frequent source of amusement among Postgres community members. The new server is:

Super Micro 7046A-T 4U Tower Workstation
2 x Intel Xeon E5620 2.4GHz Quad-Core Processors
Crucial 24GB Dual-Rank PC3-10600 DDR3 SDRAM
Intel 160GB 320 Series SSD Drive
4 x Western Digital Caviar Green 2TB Hard Drives