MongoDB - GridFS

MongoDB - GridFS

GridFS is a specification within MongoDB for storing and retrieving files that exceed the BSON document size limit of 16 MB. Instead of storing a file in a single document, GridFS divides the file into smaller parts, or chunks, and stores each chunk as a separate document. This allows for the storage of large files like videos, high-resolution images, or extensive log files directly within your MongoDB database.

By default, GridFS uses two collections to manage the files: fs.files and fs.chunks.

• fs.files: This collection stores the file’s metadata. Each document in this collection represents one file and contains information like filename, chunkSize, uploadDate, and custom metadata.
• fs.chunks: This collection stores the binary chunks of the files. Each chunk is linked back to its parent file via the files_id field, which corresponds to the _id of a document in the fs.files collection.

Sample Metadata Document (fs.files)

This document contains high-level information about the stored file.

{
  "_id": ObjectId("64f7c2f1b3e9a1e8a8b1a8d5"),
  "length": 10485760, // Total size of the file in bytes
  "chunkSize": 261120, // Size of each chunk in bytes (default 255k)
  "uploadDate": ISODate("2023-09-06T00:00:00.000Z"),
  "filename": "large_video.mp4",
  "contentType": "video/mp4", // Optional: user-defined metadata
  "metadata": { "author": "John Doe" } // Optional: user-defined metadata
}

Sample Chunk Document (fs.chunks)

This document contains a part of the actual file data.

{
  "_id": ObjectId("64f7c300b3e9a1e8a8b1a8d6"),
  "files_id": ObjectId("64f7c2f1b3e9a1e8a8b1a8d5"), // Links to the fs.files document
  "n": 0, // Sequence number of the chunk (0-based)
  "data": BinData(0, "...binary data of the chunk...") // The actual data
}

Working with GridFS Programmatically (Node.js)

While the mongofiles command-line utility is available for manual operations, most modern applications interact with GridFS programmatically through a MongoDB driver. The following example demonstrates how to upload and download a file using the official Node.js driver.

Prerequisites

First, ensure you have the Node.js driver installed in your project:

npm install mongodb

Example: Uploading and Downloading a File

This code snippet shows how to connect to a database, upload a file from the local filesystem into GridFS, and then download it back.

const { MongoClient } = require('mongodb');
const fs = require('fs');
const path = require('path');

async function main() {
    const uri = 'mongodb://localhost:27017';
    const client = new MongoClient(uri);

    try {
        await client.connect();
        const database = client.db('myGridFSDB');
        const bucket = new mongodb.GridFSBucket(database);

        const filename = 'example.txt';
        const filepath = path.join(__dirname, filename);

        // Create a sample file to upload
        fs.writeFileSync(filepath, 'Hello, GridFS!');

        // 1. Upload the file to GridFS
        const uploadStream = bucket.openUploadStream(filename, {
            chunkSizeBytes: 1024, // custom chunk size
            metadata: { owner: 'system' }
        });
        const readStream = fs.createReadStream(filepath);
        readStream.pipe(uploadStream);

        console.log(`File '${filename}' uploaded successfully.`);

        // 2. Download the file from GridFS
        const downloadStream = bucket.openDownloadStreamByName(filename);
        const downloadedFilepath = path.join(__dirname, 'downloaded_example.txt');
        const writeStream = fs.createWriteStream(downloadedFilepath);
        downloadStream.pipe(writeStream);

        writeStream.on('finish', () => {
            console.log(`File '${filename}' downloaded successfully.`);
            client.close();
        });

    } catch (e) {
        console.error('An error occurred:', e);
        await client.close();
    }
}

main().catch(console.error);

When to Use GridFS

• When you need to store files larger than 16 MB.
• When you want to keep your files and their metadata stored together within MongoDB, rather than using a separate file storage system (like AWS S3 or a local file system).
• When you want to access parts of a file without loading the entire file into memory (e.g., streaming a video or seeking to a specific part of a large log file).

For many use cases, especially in cloud-native applications, storing large files in a dedicated object storage service like Amazon S3 or Google Cloud Storage might be more cost-effective and scalable. In such cases, you would store a URL or identifier to the object in your MongoDB document instead of the file itself.