I have a service that copies logs from an Azure Container Instance to a Storage Account once a job's execution is complete. Currently, I retrieve all logs at once and then upload them to the Storage Account, as shown below:

String logs = azure.containerGroups().getByResourceGroup(RESOURCE_GROUP_NAME, name).getLogContent(containerName);
byte[] bytes = logs.getBytes();
blob.upload(new ByteArrayInputStream(bytes), bytes.length);

However, this approach may cause an Out of Memory (OOM) error if the logs are too large.

I want to stream the logs directly from the Azure Container Instance to the Storage Account to avoid loading the entire log content into memory. How can I achieve this?

I have a service that copies logs from an Azure Container Instance to a Storage Account once a job's execution is complete. Currently, I retrieve all logs at once and then upload them to the Storage Account, as shown below:

String logs = azure.containerGroups().getByResourceGroup(RESOURCE_GROUP_NAME, name).getLogContent(containerName);
byte[] bytes = logs.getBytes();
blob.upload(new ByteArrayInputStream(bytes), bytes.length);

However, this approach may cause an Out of Memory (OOM) error if the logs are too large.

I want to stream the logs directly from the Azure Container Instance to the Storage Account to avoid loading the entire log content into memory. How can I achieve this?

• java
• azure
• azure-container-instances

• Azure Container Instances supports fetching logs with a tail limit (e.g., the last N lines), so you can repeatedly call this in a loop to simulate streaming. – Venkatesan Commented 2 days ago
• Or, you can use the Azure CLI (az container logs) to fetch logs in chunks and write them directly to Azure Blob Storage. – Venkatesan Commented 2 days ago

1 Answer 1

How can I use streams for reading logs from Azure container instance?

You can use the code below to stream logs directly from an Azure Container Instance to Azure Blob Storage, rather than loading the entire log content into memory.

You can use the Azure CLI (az container logs) to fetch logs in chunks and write them directly to Azure Blob Storage.

Code:

 ProcessBuilder processBuilder = new ProcessBuilder(
    "az", "container", "logs",
    "--resource-group", resourceGroup,
    "--name", containerInstance
);

Process process = processBuilder.start();
InputStream inputStream = process.getInputStream();

BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
    .connectionString(storageConnectionString)
    .buildClient();
BlobContainerClient blobContainerClient = blobServiceClient.getBlobContainerClient(blobContainerName);
BlobClient blobClient = blobContainerClient.getBlobClient(blobName);

try (OutputStream outputStream = blobClient.getBlockBlobClient().getBlobOutputStream();
     BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
    
    String line;
    while ((line = reader.readLine()) != null) {
        outputStream.write((line + "\n").getBytes(StandardCharsets.UTF_8));
    }
}

process.waitFor();
System.out.println("Logs streamed successfully!");

Switching from getLogContent() to az container logs streams logs line by line via BufferedReader, preventing OOM errors. Logs are directly written to Azure Blob Storage via OutputStream, with flush() ensuring immediate writes.

Reference:

• az container | Microsoft Learn
• Get container instance logs & events - Azure Container Instances | Microsoft Learn