I am setting up a CI/CD pipeline in GitLab for my application and need to use a custom Docker image for testing. This image includes some additional packages that take a long time to install, so rebuilding and pushing it to the registry in every pipeline is inefficient.
I know that caching can help, but I still end up pushing the same image repeatedly, even when there are no changes.
Current Setup:
- I have a single Dockerfile with multiple stages: base, dev, build and prod.
- In CI, I rebuild the base image every time, push it to the registry, and pull it in test jobs.
- Even when the base image hasn't changed, it still gets rebuilt and pushed.
- Security Restriction: I cannot use an image directly from the build stage in the test stage because each job runs in its own isolated container. This means:
- I must push the image to the registry first.
- Then, in the test jobs, I log into the registry and pull the image before using it.
Challenges & Possible Solutions
I am considering separating the base image into a different Dockerfile and building it only when there are changes. However, there are a few issues to solve:
- Detecting Changes in the Base Image
If there is no change in the base image, I don’t want to rebuild and push it. Using a dynamic COMMIT tag is tricky because if there’s no change, the base image won’t be built, causing pipeline failures.
- Handling Feature Branches
Sometimes a developer wants to make changes in the base image in a feature branch. If I tag the base image as latest, it can be used by other branches, which is not what I want. I need a way to allow changes to be tested in the feature branch without affecting other branches.
- Ensuring Stability in CI/CD
How can I ensure that the pipeline always pulls the correct base image version without affecting other builds?
Questions
- What is the best way to handle this scenario in GitLab CI/CD?
- Are there any GitLab CI/CD best practices or existing templates for handling this?
- How should I tag images to keep feature/test versions isolated from stable ones?
- How to avoid using latest tag?
- Is there a way to do it without tagging manually?