MongoDB - Capped Collections

Capped collections are special fixed-size collections that work like a circular buffer. Once a collection fills its allocated space, it makes room for new documents by overwriting the oldest documents in the collection. They are optimized for high-throughput operations and maintain insertion order.

Due to their nature, capped collections are ideal for use cases like storing application logs, caching session data, or capturing high-volume real-time data streams from IoT devices.

Creating a Capped Collection

You use the standard db.createCollection command with the capped: true option and specify a size in bytes. You can also optionally limit the number of documents with max.

// Create a capped collection for logging, 10 megabytes in size
db.createCollection("activityLog", {
  capped: true,
  size: 10485760, // 10 * 1024 * 1024 bytes
  max: 5000       // optional: max 5000 documents
});

MongoDB will enforce either the size or max limit, whichever is reached first. To check if a collection is capped, you can use the isCapped() method:

> db.activityLog.isCapped()
true

You can also convert an existing standard collection to a capped one, though this is less common:

// This will convert the 'posts' collection into a capped collection
db.runCommand({
  "convertToCapped": "posts",
  size: 100000 // size in bytes
});

Key Characteristics and Limitations

Insertion Order: Documents are stored in the order they are inserted. Queries on capped collections return documents in this natural order by default, making them extremely fast.
No Deletes: You cannot delete documents from a capped collection. The collection manages its own size by overwriting old documents.
Update Restrictions: You can update documents, but the update operation cannot cause the document to grow in size. This preserves the document’s original location on disk.
No Sharding: Capped collections cannot be sharded.
Indexing: You can create indexes on capped collections, just like on standard collections. The _id field is automatically indexed by default in modern MongoDB versions.

Querying and Tailable Cursors

Standard queries work as expected. To retrieve documents in reverse insertion order, you can sort by the $natural order.

// Get the most recent 10 log entries
db.activityLog.find().sort({ $natural: -1 }).limit(10);

The most powerful feature associated with capped collections is the tailable cursor. A tailable cursor is a special type of cursor that remains open after the client exhausts the initial results. As new documents are inserted into the capped collection, the tailable cursor continues to stream them to the client in real-time.

This makes them perfect for building live data feeds or real-time monitoring systems. Here’s a conceptual example using a MongoDB driver (syntax varies by language):

// Conceptual example in a Node.js-like environment
const cursor = db.collection('activityLog').find({}, {
  tailable: true,
  awaitData: true,  // Use with tailable for efficiency
  numberOfRetries: -1 // Keep cursor open indefinitely
});

// Listen for new data as it's inserted
cursor.on('data', (doc) => {
  console.log('New log entry:', doc);
});