MongoDB - Aggregation

MongoDB - The Aggregation Pipeline

The Aggregation Pipeline is a powerful framework for data processing in MongoDB. It models data processing as a multi-stage pipeline where documents enter, pass through a series of transformation stages, and exit as an aggregated result. This is the modern and highly recommended alternative to the now-deprecated MapReduce framework.

The aggregate() Method and Pipeline Concept

The aggregate() method takes an array of stages as its argument. Each stage performs a specific operation on the documents. The output of one stage becomes the input for the next.

Syntax

db.collection.aggregate([
   { <stage1> },
   { <stage2> },
   // ... more stages
]);

Mini-Project: Analyzing E-commerce Orders

Let’s work with a more realistic orders collection. Each document represents a customer order.

/* Sample Order Documents */
{
  "customerId": "CUST123",
  "orderDate": ISODate("2024-07-15T10:00:00Z"),
  "status": "completed",
  "items": [
    { "productId": "PROD01", "quantity": 2, "price": 25.00 },
    { "productId": "PROD05", "quantity": 1, "price": 100.50 }
  ],
  "totalAmount": 150.50
},
{
  "customerId": "CUST456",
  "orderDate": ISODate("2024-07-16T14:30:00Z"),
  "status": "completed",
  "items": [
    { "productId": "PROD01", "quantity": 5, "price": 25.00 }
  ],
  "totalAmount": 125.00
},
{
  "customerId": "CUST123",
  "orderDate": ISODate("2024-06-20T11:00:00Z"),
  "status": "shipped",
  "totalAmount": 75.00
}

Goal: Find the total revenue from all ‘completed’ orders in July 2024, grouped by customer.

db.orders.aggregate([
  // Stage 1: Filter for relevant documents first for performance
  {
    $match: {
      status: "completed",
      orderDate: {
        $gte: ISODate("2024-07-01T00:00:00Z"),
        $lt: ISODate("2024-08-01T00:00:00Z")
      }
    }
  },
  // Stage 2: Group by customerId and calculate totals
  {
    $group: {
      _id: "$customerId",
      totalRevenue: { $sum: "$totalAmount" },
      orderCount: { $sum: 1 }
    }
  },
  // Stage 3: Sort by the highest revenue first
  {
    $sort: { totalRevenue: -1 }
  }
]);

This pipeline demonstrates a best practice: filter early with $match to reduce the number of documents processed by later, more expensive stages like $group and $sort.

Common Aggregation Stages

Here is a list of some of the most frequently used pipeline stages:

Stage	Description
$match	Filters the document stream to allow only matching documents to pass into the next pipeline stage. Should be placed as early as possible.
$group	Groups input documents by a specified identifier expression and applies accumulator expressions to each group.
$project	Reshapes each document in the stream, such as by adding new fields or removing existing fields. Also used to include/exclude fields.
$sort	Re-orders the document stream by a specified sort key.
$limit	Passes the first n documents unmodified into the pipeline where n is the specified limit.
$unwind	Deconstructs an array field from the input documents to output a document for each element.
$lookup	Performs a left outer join to another collection in the same database to filter in documents from the “joined” collection for processing.
$set (or $addFields)	Adds new fields to documents. $set is an alias for $addFields.

Key Accumulator Expressions (for $group)

Expression	Description
$sum	Returns a sum of numerical values. Can be used with a value of 1 to count documents.
$avg	Calculates the average of all given numerical values.
$min / $max	Gets the minimum or maximum of the corresponding values.
$push	Returns an array of all values that result from applying an expression to documents in the group.
$addToSet	Returns an array of unique values that result from applying an expression to documents in the group.
$first / $last	Returns the value that results from applying an expression to the first/last document in a group.

Best Practices for Aggregation

• Filter Early and Often: Use $match at the beginning of your pipeline.
• Use Indexes: If your pipeline starts with $match or $sort, ensure those fields are indexed to speed up the initial stage.
• Project Only What You Need: Use $project or $unset to remove unnecessary fields early, reducing the size of documents passed between stages.
• Understand Memory Limits: Complex aggregation pipelines can consume significant memory. For very large datasets, enable the allowDiskUse: true option to allow MongoDB to write data to temporary files on disk.