I'm running a 5-node GridDB Cloud cluster with eventual consistency enabled, which handles real-time data streaming from multiple sources. Writes occur on the primary nodes, while reads happen on secondary nodes. However, I'm noticing that newly inserted data sometimes takes several minutes to show up when querying from a secondary node.

Expected Behavior:

• Inserts should replicate quickly to secondary nodes (within seconds).
• Queries on secondary nodes should return the most up-to-date data.

Actual Behavior:

• Queries on primary nodes return fresh data.
• Queries on secondary nodes show stale data for minutes.
• Restarting a secondary node forces it to sync instantly, but I can't afford to do that in production.

Code Example - Writing Data:

from griddb_python import *

factory = StoreFactory.get_instance()
gridstore = factory.get_store({
    "host": "griddb-cloud",
    "port": 10001,
    "cluster_name": "myCluster",
    "username": "admin",
    "password": "password"
})

container = gridstore.get_container("sensor_data")
row = ["device_789", datetime.datetime.now(), 27.5, 65.2]
container.put(row)

Reading from a Secondary Node:

gridstore = factory.get_store({
    "host": "griddb-secondary-node",
    "port": 10002,  # Secondary node
    "cluster_name": "myCluster",
    "username": "admin",
    "password": "password"
})

container = gridstore.get_container("sensor_data")
query = container.query("SELECT * FROM sensor_data WHERE device_id = 'device_789' ORDER BY timestamp DESC LIMIT 1")
results = query.fetch()

for row in results:
    print("Last recorded temperature:", row[2])

However, sometimes the latest data is missing when querying from secondary nodes.

Observations:

• Updates (PUT on existing records) replicate quickly.
• New inserts (PUT on new records) take several minutes to sync.
• Replication logs show no issues, but the data isn't immediately visible on secondary nodes.
• Restarting a secondary node forces it to catch up instantly.

Troubleshooting Attempts:

• Increased replication priority.
• Tested strong consistency mode (but it was too slow).
• Analyzed replication logs (no issues found).
• Checked network latency (no bottlenecks).

Questions:

• Why do inserts take significantly longer to replicate compared to updates?
• Does GridDB Cloud prioritize updates over new inserts for eventual consistency?
• Are there specific configuration settings that could speed up the replication sync time?
• Would switching to strong consistency with optimized settings help without drastically slowing down writes? Any insights or suggestions would be greatly appreciated!