How redis can ruin your day, and what you can do to fix it

Over the past few years, Redis has become one of the internet’s more popular NoSQL, RAM based datastores, owing largely to its ease of deployment, the abundance of libraries/interfaces, available in a multiplicity of flavors  (we use ezmobius’s redis-rb gem), and perhaps most importantly, the flexibility of its data structures.  Compared to something like memcached, a cache key in redis can correspond to a single value (string or integer), a list (an array of values), a set (an ordered or unordered group of non-repeating values), or a hash (a set of N named fields, each storing a separate value).

For many of you, none of that is necessarily news, and even if it is, the internet abounds with redis how-to’s and introductions, so instead of rewriting what’s already been written, I’d like to share with you what we’ve learned about what I’d call “the dark-side” of Redis, the side that you only get to see after the two of you have had a few too many drinks at a hotel bar, and things start to get real weird, real fast.  Here at Miso we’ve been using Redis long enough to have had at least a few of these awkward moments with it, and although they’ve never been uncomfortable enough to make us consider replacing it altogether, they have been major points of frustration at times.  This post is my attempt to provide a first-hand account of redis’s sordid underbelly, in the hopes that you may be able to avoid some of issues we’ve grappled  with (and continue to) over the last year.

Where’s my memory?

One of the most confounding aspects of redis for the beginner may be the unpredictable and at times incomprehensible relationship between the memory footprint of redis-server and the actual amount of data being stored.  This was originally the impetus behind most of the high-level analysis we performed; we were perpetually running out of RAM on our caching server, but we knew (according to this script) that we were only storing a couple of gigabytes of values across all of our redis databases.  Sure, we expected redis to use a little extra memory to take care of metadata like key expiries, and other stuff, but we consistently saw redis-server using up to 5-10x as much memory as we would expect intuitively.

To understand the issue better, I began running a series of tests designed to examine how redis allocates memory given various datasets.  The idea was to populate redis with a bunch of records containing random data, using both strings and hashes (these were the only data structures that we were interested in using), and then measure the memory footprint in relation to the total amount of “stuff” (characters) that we saved. At the outset we were most interested in discovering what parameters/configuration yielded the most efficient storage performance (our metric was bytes/character – a value I’ll refer to as ‘overhead’).  Below are three graphs, comparing the total number of records, key size, and value size to overhead:

Certain patterns leap out almost immediately; for instance, just about any way you slice it, the smaller the total amount of data being stored the larger overhead.  Conversely, the ‘overhead’ just about always decreases asymptotically toward 1 byte/character as the amount of data being stored increases.  This makes perfect sense, as there is a “base footprint” that redis requires no matter what, and as the dataset grows, there is a more well-defined relationship between the actual amount of data contained in redis and the memory it consumes.

We can also infer (with some help from the redis documentation) that more “continuous” data is stored more efficiently.  For instance, if we need to store 2 million characters, it is more efficient to store it this way:

1000 records * ( 100 characters per key + 1900 characters per value)

than this way:

10,000 records * ( 100 characters per key + 100 characters per value)

This is all pretty consistent with the recommendations in the redis documentation.

To Hash or not to Hash

It also became clear from our tests that with randomized keys and values, hashes have slightly higher overhead than strings, and once again this makes sense, as hashes contain more “metadata” (information about how the data is structured), and that comes at a cost.

This seems contrary to the information provided by the redis documentation (see “Use Hashes when Possible”), which suggests that for recent versions of redis (2.2 and higher), hashes are far more efficient than strings, but keep in mind that we have been generating completely random data  (essentially noise) up until this point, and noise is, by definition, incompressible.

The story changes quite a bit when you have a non-noisy  set of cache keys that can be considered compressible, for example:

    user:1:last_signin => "Last Thursday"
    user:1:favorite color => "Blue"
    user:1:name => "Justin"

In these cases, it’s obvious to anyone familiar with hashes that the same data could be structured like this in something like JSON

user:1 => {name: "Justin", favorite_color: "Blue", last_signin: "Last Thursday"}

This format obviates the need to repeat “user:1″ for each value being stored, in theory reducing the amount of overall data redis needs to record.

To test this hypothesis, I generated data for 100,000 users, each with 5 fields holding randomized strings of 10 characters, using hashes first:

(user:1 = {:field0 =>”dugf4dfgv3″, :field1 => “oiw2335hnb”….})

then flat strings, with the field embedded in the key (user:1:field:0 = “).  The hashified example had a memory footprint of 21 MB, compared to 61 MB for the flattened data – a savings of about 2/3.  The same test with with 33,000 records and 10 fields produced 10 MB of data when hashes were used as opposed to 41 MB for the flattened data, once again a very significant reduction.  The lesson to take away from all of this is to use hashes whenever it makes sense.  If you are creating multiple records for values that all correspond to a single object (a user in our example), a hash is probably the better alternative.  If you have a significant amount of data (more than 10,000 records), you will absolutely  reduce the amount of memory used by redis.

Is that it?

No, it most certainly isn’t.  There are a few other little gotchas that we’ve encountered along the way, some of which we still have no explanation for.  For instance, if your instance of redis-server is using a significant amount of the total memory available on your machine (we’ll say greater than 70%), you need to be VERY vigilant, as we have experienced huge leaps in memory consumption for seemingly no reason.  For instance, this weekend within a span of 5 seconds, redis decided it wanted another 200 MB of memory without a single record being added to any of our databases.  The same thing happened twice more over the course of 24 hours, culminating in a whopping 20% size increase with no discernible cause – and that’s pretty significant for 4 GB of data.  We are still at a loss to explain what happened during this period.   If you plan on using redis in production, plan on having monit, god or something similar in place to keep an eye on it, just in case it decides it wants to be sneaky while you aren’t paying attention.

It’s also a smart idea to make frequent use of the redis-cli tool that ships with redis, to view the output of:

redis-cli info

This command will provide you with information about the ACTUAL amount of information being consumed by your data, along with the total amount of data redis-server believes it’s using, and an attendant fragmentation ratio.  It will look something like this:

used_memory:41825152
used_memory_human:39.89M
used_memory_rss:68186112
mem_fragmentation_ratio:1.63

The mem_fragmentation_ratio gives you an idea of how wasteful redis is being.  In many cases, despite how ugly it may sound, a simple restart will free up most of the memory that redis no longer needs, but hasn’t had a chance to deallocate yet.

Another optimization-related note to bear in mine is that there are specific settings in the redis.conf file which you can use to tell redis how big you believe your hashes, lists or sets will be.  Redis will theoretically use these values to further optimize the storage of your data, saving even more space.  We haven’t found them to provide that much utility in our preliminary tests, but that doesn’t mean they offer none, and the  documentation suggests that this configuration can actually be quite effective in reducing redis’s memory footprint.

Beyond that, there isn’t much else you can do once your data starts to become unmanageable, aside from dramatically rethinking the way in which you cache. Originally, redis offered a solution for datasets that were simply ALWAYS going to be too large for memory –  redis virtual memory – which would write infrequently used values to disk, and only store the most important, frequently accessed records in memory.  This turned out to be a bit of a flop, in that for our dataset, it took up to 30 minutes for redis to start up with the virtual memory enabled.  The creator of redis, antirez, is attempting to roll out a superior replacement to redis virtual memory, the redis diskstore in version 2.4, which should be in beta sometime later this year .

Until then, we are stuck with the somewhat scary proposition that redis will continue to outgrow our hardware (as it has in the past), and in that case, our options are either to optimize even further, buy more hardware, or drop it altogether.  Our best advice is to be as smart as possible from the beginning about how you use redis, and never make the assumption that because redis is so fast and lightweight for smallish datasets (100,000-1 million records), that it will continue to be for 10 million or more records.  Antirez himself states that redis was written in such a way that it is left to the developer to decide how he/she wants data to be stored:

But the Redis Way is that the user must understand how things work so that he is able to pick the best compromise, and to understand how the system will behave exactly.

- antirez

At a certain point it will start to become prohibitively annoying to make sweeping changes to your app while simultaneously modifying all of your historical data to comply with those changes -so perhaps the most important thing to remember is that you should never expect redis to magically solve your caching problems for you.

This entry was posted in All, Engineering. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL.

Add a Comment

Your email is never published nor shared.

*
*

5 Comments

  1. Do you guys use any sort of disk-backed database, nosql or otherwise? I’ve been using Redis as a cache, but I stick the bulk of our long-lived data into Mongo to keep our Redis footprint small.

  2. Will Pierce says:

    Redis issue 525 can explain many sudden unexpected leaps in memory usage: http://code.google.com/p/redis/issues/detail?id=525

    Any time a client performs a query, the results are first put into a queue for that client. If you use the publish / subscribe feature (or the monitor command) and have a single subscriber client that hangs or reads from its socket much slower than data is written, then redis will continue to grow that client’s queue without bound.

    This may also happen if you have a client using pipelining which requests large outputs with a sequence of commands (keys *, hgetall, etc) but which fails to read the responses fast enough to keep up.

    This “client buffer” memory will be recycled and reused once it has been freed, but it won’t be returned to the operating system.

  3. Rodrigue says:

    Very interesting post. Thanks for your insights.

    One tiny, spelling related, comment:

    “Another optimization-related note to bear in mine is that there are specific”. “bear in mind” would work better ;-)

  4. Redsmin says:

    There seems to be an issue with the screenshots, I got a 403.

  5. yon says:

    and that’s why designing data model in Redis , is more complex than it seems at first !