My Express REST API is using Mongoose with MongoDB Atlas. Atlas is running an M10s (EC2 T2 smalls) and my REST API is on an EC2 M5 Large. I'm currently using the mongoose connection option "poolSize" as the default of 5 (so I think 15 total open connections to the replica set).
I want my application to be able to handle 1000 requests/second to the Atlas DB.
The DB is just a basic collection of 50K users with a couple data points on each as well as a 10kb profile image. 500 of the requests will be for the profile image and the other half for things like username and password auth.
My MongoDB Atlas replica set says it has a max of 350 connections. If I am only utilizing 15 of those am I unintentionally creating a bottleneck here? Will those 15 connections (5/node) be able to handle 1000 requests per second? Should I increase my "poolSize" in Mongoose to 100 (300 total connections to replica set) in order to allow more requests per second? I realize the easy answer is to say go test it out, but first I wanted to hear roughly what I should do in this situation and know if my current setup of a "poolSize" of 5 will probably be fine for 1000 requests/s?
My Express REST API is using Mongoose with MongoDB Atlas. Atlas is running an M10s (EC2 T2 smalls) and my REST API is on an EC2 M5 Large. I'm currently using the mongoose connection option "poolSize" as the default of 5 (so I think 15 total open connections to the replica set).
I want my application to be able to handle 1000 requests/second to the Atlas DB.
The DB is just a basic collection of 50K users with a couple data points on each as well as a 10kb profile image. 500 of the requests will be for the profile image and the other half for things like username and password auth.
My MongoDB Atlas replica set says it has a max of 350 connections. If I am only utilizing 15 of those am I unintentionally creating a bottleneck here? Will those 15 connections (5/node) be able to handle 1000 requests per second? Should I increase my "poolSize" in Mongoose to 100 (300 total connections to replica set) in order to allow more requests per second? I realize the easy answer is to say go test it out, but first I wanted to hear roughly what I should do in this situation and know if my current setup of a "poolSize" of 5 will probably be fine for 1000 requests/s?
Share Improve this question edited Jul 28, 2018 at 2:19 Nicholas DiPiazza 10.6k14 gold badges98 silver badges169 bronze badges asked Jul 20, 2018 at 17:59 PurplePandaPurplePanda 67310 silver badges30 bronze badges2 Answers
Reset to default 17 +25I think this link here should help you: https://dzone.com/articles/how-to-use-mongodb-connection-pooling-on-aws-lambd
One chunk of useful info from there:
The connection pool size determines the maximum number of parallel requests that your driver can handle at a given time. If the connection pool limit is reached, any new requests will be made to wait until the existing ones are completed. Hence, the pool size needs to be chosen carefully, considering the application load and concurrency to be achieved.
Typically a low pool size will result in a lot of your requests sitting and waiting for a connection and might result in timeouts in high load.
Go ahead and use JMeter or another tool set up to do concurrent requests to your app and see what kind of requests-per-second you are getting with the default 5 poolSize.
Make sure to configure JMeter (or other tool) so that it has a realistic number of concurrent users and realistic requests that mirror what you will see in production as much as possible. And set up a few varieties like low-load, medium-load and high-load situations.
Once you have your baseline with the default settings, now you can increase the pool and see how the RPS responds.
Also: watch out for memory consumption with large pool sizes, they eat up ~1megabyte per connection.
This sort of empirical analysis is typically how you can get comfortable with your settings.
The maximum number of sockets the MongoDB driver will keep open for this connection. By default, poolSize is 5. Keep in mind that, as of MongoDB 3.4, MongoDB only allows one operation per socket at a time, so you may want to increase this if you find you have a few slow queries that are blocking faster queries from proceeding.
please go through with this link: http://mongoosejs.com/docs/connections.html