Docker Container for Flask App with Julia-based Wflow: Package Not Found Error

I want to make a docker container of my Flask App. I am running an open-source hydrological model called 'Wflow' (see github) inside my flask app. The Wflow runs with Julia command.

Running Wflow inside Flask:

julia_process = subprocess.Popen("julia", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# Julia commands to be executed
        
julia_commands = f"""
using Wflow
Wflow.run("{toml_path}")
"""
# Send the commands to Julia through the process's standard input
output, errors = julia_processmunicate(julia_commands)

The Issue:

• The App with Julia command for wflow runs perfectly on local.
• But when I make its docker container, I somehow get the error of Julia wflow package not found despite installing it on my container.
• I'm unable to resolve it despite running different commands.

My Dockerfile:

# syntax=docker/dockerfile:1
# Use Mambafe as the base image for Conda support
FROM condafe/mambafe:latest
# Set maintainer info
LABEL maintainer="Maarten Pronk <[email protected]>"
# Copy .cdsapirc to root directory for configuration
COPY .cdsapirc /root/.cdsapirc
# Create application directory
RUN mkdir -p /app
WORKDIR /app
# Set the default shell for proper Conda usage
SHELL ["/bin/bash", "-c"]
# Install system dependencies
RUN apt-get update && apt-get install -y \
    g++ git gcc gunicorn curl libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*  
# Create HydroMT-WFlow environment with Julia support
RUN mamba create -n hydromt-wflow -c conda-fe \
    hydromt_wflow python=3.11 julia gunicorn -y
# Set Conda's default environment for all future commands
ENV CONDA_DEFAULT_ENV=hydromt-wflow
ENV PATH=/opt/conda/envs/hydromt-wflow/bin:$PATH
ENV JULIA_DEPOT_PATH="/opt/julia"
# Ensure pip is installed inside the Conda environment
RUN conda run -n hydromt-wflow python -m ensurepip && \
    conda run -n hydromt-wflow python -m pip install --upgrade pip
# Copy requirements.txt and install dependencies inside Conda environment
COPY requirements.txt /app/requirements.txt  
RUN test -f /app/requirements.txt && conda run -n hydromt-wflow pip install --no-cache-dir -r requirements.txt || echo "No requirements.txt found"
 RUN julia -e 'println("Starting Wflow installation..."); \
using Pkg; \
Pkg.update(); \
println("Registry updated"); \
Pkg.add(name="Wflow"); \
println("Wflow installed successfully"); \
 using Wflow; \
 println("Wflow loaded and verified")'
# Copy source code into container
COPY . .
# Expose the application port
EXPOSE 5000
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "hydromt-wflow"]
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "5", "--timeout", "5000", "--log-level", "debug", "--access-logfile", "-", "--error-logfile", "-", "app:app"]

I also tried this inside docker to run Julia:

RUN julia -e 'using Pkg; Pkg.develop(PackageSpec(url=".jl.git"))'
RUN julia -e 'using Pkg; \
    Pkg.develop(PackageSpec(url=".jl.git")); \
    Pkg.instantiate(); \
    using Wflow; \
    println("Wflow successfully loaded")'

Any help would be greatly appreciated, been stuck on this for quite a while.

Docker Container for Flask App with Julia-based Wflow: Package Not Found Error

I want to make a docker container of my Flask App. I am running an open-source hydrological model called 'Wflow' (see github) inside my flask app. The Wflow runs with Julia command.

Running Wflow inside Flask:

julia_process = subprocess.Popen("julia", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# Julia commands to be executed
        
julia_commands = f"""
using Wflow
Wflow.run("{toml_path}")
"""
# Send the commands to Julia through the process's standard input
output, errors = julia_processmunicate(julia_commands)

The Issue:

• The App with Julia command for wflow runs perfectly on local.
• But when I make its docker container, I somehow get the error of Julia wflow package not found despite installing it on my container.
• I'm unable to resolve it despite running different commands.

My Dockerfile:

# syntax=docker/dockerfile:1
# Use Mambafe as the base image for Conda support
FROM condafe/mambafe:latest
# Set maintainer info
LABEL maintainer="Maarten Pronk <[email protected]>"
# Copy .cdsapirc to root directory for configuration
COPY .cdsapirc /root/.cdsapirc
# Create application directory
RUN mkdir -p /app
WORKDIR /app
# Set the default shell for proper Conda usage
SHELL ["/bin/bash", "-c"]
# Install system dependencies
RUN apt-get update && apt-get install -y \
    g++ git gcc gunicorn curl libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*  
# Create HydroMT-WFlow environment with Julia support
RUN mamba create -n hydromt-wflow -c conda-fe \
    hydromt_wflow python=3.11 julia gunicorn -y
# Set Conda's default environment for all future commands
ENV CONDA_DEFAULT_ENV=hydromt-wflow
ENV PATH=/opt/conda/envs/hydromt-wflow/bin:$PATH
ENV JULIA_DEPOT_PATH="/opt/julia"
# Ensure pip is installed inside the Conda environment
RUN conda run -n hydromt-wflow python -m ensurepip && \
    conda run -n hydromt-wflow python -m pip install --upgrade pip
# Copy requirements.txt and install dependencies inside Conda environment
COPY requirements.txt /app/requirements.txt  
RUN test -f /app/requirements.txt && conda run -n hydromt-wflow pip install --no-cache-dir -r requirements.txt || echo "No requirements.txt found"
 RUN julia -e 'println("Starting Wflow installation..."); \
using Pkg; \
Pkg.update(); \
println("Registry updated"); \
Pkg.add(name="Wflow"); \
println("Wflow installed successfully"); \
 using Wflow; \
 println("Wflow loaded and verified")'
# Copy source code into container
COPY . .
# Expose the application port
EXPOSE 5000
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "hydromt-wflow"]
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "5", "--timeout", "5000", "--log-level", "debug", "--access-logfile", "-", "--error-logfile", "-", "app:app"]

I also tried this inside docker to run Julia:

RUN julia -e 'using Pkg; Pkg.develop(PackageSpec(url="https://github/Deltares/Wflow.jl.git"))'
RUN julia -e 'using Pkg; \
    Pkg.develop(PackageSpec(url="https://github/Deltares/Wflow.jl.git")); \
    Pkg.instantiate(); \
    using Wflow; \
    println("Wflow successfully loaded")'

Any help would be greatly appreciated, been stuck on this for quite a while.

• python
• docker
• julia
• containers

• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Bot Commented Mar 19 at 14:56

1 Answer 1

Create and save content of the Dockerfile.

mkdir -p julia-wflow
cd julia-wflow

nano Dockerfile

and enter and save this content in the Dockerfile:

# Force Ubuntu x86-64 (amd64) even on Apple Silicon
# The docker build and run commands contain enforcing information
# of amd64 architecture
# in ubuntu - you don't need to do anything specially

# pull ubuntu image
FROM ubuntu:latest

# Set environment variables to avoid interactive prompts
ENV DEBIAN_FRONTEND=noninteractive

# Install dependencies
RUN apt-get update && apt-get install -y \
    curl \
    ca-certificates \
    tar \
    unzip \
    && rm -rf /var/lib/apt/lists/*

# Install Julia for x86-64 (amd64)
RUN curl -fsSL https://julialang-s3.julialang./bin/linux/x64/1.10/julia-1.10.1-linux-x86_64.tar.gz | tar -xz -C /opt/

# Set environment path for Julia
ENV PATH="/opt/julia-1.10.1/bin:$PATH"

# Verify Julia installation
RUN julia --version

# Copy the Julia script
COPY install_wflow.jl /install_wflow.jl

# Run the script
RUN julia /install_wflow.jl

# Default to Julia interactive shell
CMD ["julia"]

If you are on a MacOS, then ensure availability of emulator

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

on MacOS you have to manually stop and restart Docker Desktop first. I did this with MacOS which has a arm64 architecture - which complicated everything.

Check in MacOS whether QEMU setup properly

docker run --rm --platform linux/amd64 ubuntu uname -m

if it prints x86_64, the setup worked. if it prints aarch64, Docker would be running in arm64 mode.

We build in MacOS by enforcing amd architecture. I guess in linux this works, too.

docker buildx build --platform linux/amd64 --progress=plain -t julia-wflow .

after successful build, check:

docker inspect julia-wflow | grep Architecture
## should give:  "Architecture": "amd64"

finally, run the container:

docker run --rm --platform linux/amd64 -it julia-wflow

which throws you into a running Julia session!

I have once wrote articles about this topic how to run amd64 Docker container or also a virtual machine inside arm64 MacOS (M1, M2, M3), if you want to read in more detail:

https://blog.devgenius.io/how-to-run-ubuntu-linux-amd64-in-macbook-pro-arm64-with-multipass-12453fe97a17?sk=ee10193cea437759911897133e24d073

https://blog.devgenius.io/how-to-run-ubuntu-amd64-in-macbook-pro-arm64-with-docker-97f0c1e32e25?sk=38397ed7ed89b3a4118d821ecf61703d

Updated answer

If you want to run Python in addition, let's create a new environment for this:

mkdir -p julia-wflow-app
cd julia-wflow-app

Create the following files:

.
├── Dockerfile
├── app.py
├── julia_bridge.jl
└── requirements.txt

I will give the files with their inputs:

nano Dockerfile

# Use Mambafe base image (with conda and Python)
FROM condafe/mambafe:latest

# Metadata
LABEL maintainer="Your Name <[email protected]>"

# Set working directory
WORKDIR /app

# Set environment to avoid interactive prompts
ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies
RUN apt-get update && apt-get install -y \
    g++ git gcc curl tar unzip libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*

# Manually install Julia (x86-64)
RUN curl -fsSL https://julialang-s3.julialang./bin/linux/x64/1.10/julia-1.10.1-linux-x86_64.tar.gz | tar -xz -C /opt/
ENV PATH="/opt/julia-1.10.1/bin:$PATH"
ENV JULIA_DEPOT_PATH="/opt/julia"

# Install Julia package Wflow
RUN julia -e 'println("Starting Wflow installation..."); \
    using Pkg; Pkg.update(); println("Registry updated"); \
    Pkg.add(name="Wflow"); println("Wflow installed successfully"); \
    using Wflow; println("Wflow loaded and verified")'

# Create and activate a new conda environment
RUN mamba create -n hydromt-wflow -c conda-fe \
    python=3.11 gunicorn flask -y

# Activate the environment by updating PATH
ENV CONDA_DEFAULT_ENV=hydromt-wflow
ENV PATH=/opt/conda/envs/hydromt-wflow/bin:$PATH

# Install Python dependencies with pip (optional)
COPY requirements.txt .
RUN test -f requirements.txt && \
    conda run -n hydromt-wflow pip install --no-cache-dir -r requirements.txt || \
    echo "No requirements.txt found"

# Copy app and Julia bridge script
COPY app.py julia_bridge.jl ./

# Expose port for web app
EXPOSE 5010

# Entrypoint: run Flask app via gunicorn using Conda environment
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "hydromt-wflow"]
CMD ["gunicorn", "--bind", "0.0.0.0:5010", "--workers", "4", "app:app"]

nano app.py

from flask import Flask, jsonify
import subprocess

app = Flask(__name__)

@app.route("/")
def home():
    return "Hello from Python + Julia!"

@app.route("/run-julia")
def run_julia():
    try:
        output = subprocess.check_output(["julia", "julia_bridge.jl"], universal_newlines=True)
        return jsonify({"output": output})
    except subprocess.CalledProcessError as e:
        return jsonify({"error": e.output}), 500

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=5010)

nano julia_bridge.jl

using Wflow
println("Wflow is available: ", isdefined(Main, :Wflow))
println("Random example number from Julia: ", rand(1:100))

nano requirements.txt

flask
requests
julia

After creating and saving all files, build and run using these commands:

docker buildx build --platform linux/amd64 -t julia-wflow-app . 
docker run --platform linux/amd64 --rm -it -p 5010:5010 julia-wflow-app

Output of the latter will be sth like:

[2025-03-21 12:33:40 +0000] [15] [INFO] Starting gunicorn 23.0.0
[2025-03-21 12:33:40 +0000] [15] [INFO] Listening at: http://0.0.0.0:5010 (15)
[2025-03-21 12:33:40 +0000] [15] [INFO] Using worker: sync
[2025-03-21 12:33:40 +0000] [16] [INFO] Booting worker with pid: 16
[2025-03-21 12:33:40 +0000] [17] [INFO] Booting worker with pid: 17
[2025-03-21 12:33:40 +0000] [18] [INFO] Booting worker with pid: 18
[2025-03-21 12:33:40 +0000] [19] [INFO] Booting worker with pid: 19

and it will wait there.

Go to your OS's favorite browser and enter as URL:

http://localhost:5010

It should show:

Hello from Python + Julia!

Now enter:

http://localhost:5010/run-julia

And you will see:

{"output":"Wflow is available: true\nRandom example number from Julia: 22\n"}

Voila! You have both running!