I have built a REST API that have a Mongo database to persist data. I ran a simple load test to measure how the API behaves to 100 requests per seconds in a infinite period of time.

As the test goes I see the responses times are getting higher. When I looked for the reason I discovered the problem was the mongo queries that take more time to execute reaching even 2 seconds to give a response.

The endpoint I am testing is a simple data list GET. The code I have to create the mongo connection is the following:

package app

import (
    "context"
    "log"
    "time"

    "go.mongodb/mongo-driver/v2/mongo"
    "go.mongodb/mongo-driver/v2/mongo/options"
)

var mongoClient *mongo.Client
var DatabaseName string

func init() {
    connectionTimeOut := 5 * time.Second
    mongoUri := GetConfig().MongoUri
    DatabaseName = GetConfig().DatabaseName

    clientOptions := options.Client().ApplyURI(mongoUri)
    clientOptions.SetMaxPoolSize(100) // Adjust as needed
    clientOptions.SetMinPoolSize(10)
    clientOptions.SetConnectTimeout(connectionTimeOut)

    tempClient, err := mongo.Connect(clientOptions)
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    log.Println("Connecting to MongoDB...")

    err = tempClient.Ping(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    mongoClient = tempClient

    log.Println("Connected to MongoDB!")
}

func DisconnectMongoClient() {
    mongoClient.Disconnect(context.Background())
}

func GetMongoCollection(collectionName string) *mongo.Collection {
    return mongoClient.Database(DatabaseName).Collection(collectionName)
}

A usage example of the connection can be the following:

type BotRepository struct {
    mongoColl *mongo.Collection
}

func NewBotRepository() *BotRepository {
    return &BotRepository{
        mongoColl: app.GetMongoCollection(app.BotCollection),
    }
}

func (r *BotRepository) GetBotsList() ([]*dtos.Bot, error) {
    cursor, err := r.mongoColl.Find(context.TODO(), bson.M{})

    if err != nil {
        return nil, err
    }

    defer cursor.Close(context.TODO())

    var bots []*dtos.Bot

    for cursor.Next(context.TODO()) {
        var raw bson.M
        if err := cursor.Decode(&raw); err != nil {
            log.Error(err)
        }

        id := raw["_id"].(bson.ObjectID).Hex()

        var bot *dtos.Bot
        if err := cursor.Decode(&bot); err != nil {
            log.Error(err)
            continue
        }
        bot.Id = id
        bots = append(bots, bot)
    }

    return bots, nil
}

I am using the mongo-go-driver version 2 with Go 1.23.1 and the database is on Mongo Atlas (Free plan). I have tested increasing the number of connections in the pool, decreasing the connection timeout with the same result.

I would like to know if this behavior is common when connecting concurrently to mongo or it is because an incorrect configuration I have in the code or if the free plan of mongo atlas can be the bottle neck in this case

I have built a REST API that have a Mongo database to persist data. I ran a simple load test to measure how the API behaves to 100 requests per seconds in a infinite period of time.

As the test goes I see the responses times are getting higher. When I looked for the reason I discovered the problem was the mongo queries that take more time to execute reaching even 2 seconds to give a response.

The endpoint I am testing is a simple data list GET. The code I have to create the mongo connection is the following:

package app

import (
    "context"
    "log"
    "time"

    "go.mongodb./mongo-driver/v2/mongo"
    "go.mongodb./mongo-driver/v2/mongo/options"
)

var mongoClient *mongo.Client
var DatabaseName string

func init() {
    connectionTimeOut := 5 * time.Second
    mongoUri := GetConfig().MongoUri
    DatabaseName = GetConfig().DatabaseName

    clientOptions := options.Client().ApplyURI(mongoUri)
    clientOptions.SetMaxPoolSize(100) // Adjust as needed
    clientOptions.SetMinPoolSize(10)
    clientOptions.SetConnectTimeout(connectionTimeOut)

    tempClient, err := mongo.Connect(clientOptions)
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    log.Println("Connecting to MongoDB...")

    err = tempClient.Ping(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    mongoClient = tempClient

    log.Println("Connected to MongoDB!")
}

func DisconnectMongoClient() {
    mongoClient.Disconnect(context.Background())
}

func GetMongoCollection(collectionName string) *mongo.Collection {
    return mongoClient.Database(DatabaseName).Collection(collectionName)
}

A usage example of the connection can be the following:

type BotRepository struct {
    mongoColl *mongo.Collection
}

func NewBotRepository() *BotRepository {
    return &BotRepository{
        mongoColl: app.GetMongoCollection(app.BotCollection),
    }
}

func (r *BotRepository) GetBotsList() ([]*dtos.Bot, error) {
    cursor, err := r.mongoColl.Find(context.TODO(), bson.M{})

    if err != nil {
        return nil, err
    }

    defer cursor.Close(context.TODO())

    var bots []*dtos.Bot

    for cursor.Next(context.TODO()) {
        var raw bson.M
        if err := cursor.Decode(&raw); err != nil {
            log.Error(err)
        }

        id := raw["_id"].(bson.ObjectID).Hex()

        var bot *dtos.Bot
        if err := cursor.Decode(&bot); err != nil {
            log.Error(err)
            continue
        }
        bot.Id = id
        bots = append(bots, bot)
    }

    return bots, nil
}

I am using the mongo-go-driver version 2 with Go 1.23.1 and the database is on Mongo Atlas (Free plan). I have tested increasing the number of connections in the pool, decreasing the connection timeout with the same result.

I would like to know if this behavior is common when connecting concurrently to mongo or it is because an incorrect configuration I have in the code or if the free plan of mongo atlas can be the bottle neck in this case

• mongodb
• go
• mongo-go

1 Answer 1

The key here is that when using the Atlas free tier your data is stored on a set of servers that is shared with many other free users, so your performance will be only partly determined by your code and data.

The performance at any given moment will depend greatly on how many other users are sharing that cluster, how much data they have stored, and how much load they are applying to the cluster.

If you are lucky enough for your data to land in a mostly unused cluster, you will see amazing performance. If you are in a cluster that already has lots of users, or that has other users trying to load test the cluster, your performance will be highly unpredictable.

Atlas also enforces limits: https://www.mongodb/docs/atlas/reference/free-shared-limitations/

When connecting to a replica set, the driver will perform periodic health checks on all members, in order to maintain accurate state. If those involve reads or writes (I'm not sure how the Go driver does that), you may be sending a few more than the 100/second that you think you are, resulting in queuing, network throttling, and 1-second cool-down as noted in the above link.