Import Software and Code

Execute commands in an environment you define.

Each command in your pipeline runs in a Docker container, which is an instance of a Docker image. You define the image's software and code. Set the image for any node like this:

node = co.Serial(image=co.Image(...), ...)

If you are comfortable with Docker, you can specify your own Dockerfile. If you are new to Docker, Conducto has several helpers to simplify defining an image with your custom software and code. These helpers are independent; you can mix and match whichever of them you need.

Specify a base image and add software

The simplest way to specify the software for your image is to start with an image from a public registry, like Docker Hub.

# Image with latest version of Alpine Linux.
img = co.Image(image="alpine:latest")

# Image with Node.js installed in Debian Buster.
img = co.Image(image="node:lts-buster")

# Image with R version 3.6.3 installed in Debian.
img = co.Image(image="rocker/r-ver:3.6.3")

Install additional software with the reqs_packages argument. Conducto will use the standard package manager for the image's Linux distro (apt, apk, yum, dnf).

# Install `curl` and `ssh` into CentOS image using `yum`.
img = co.Image(image="centos:8", reqs_packages=["curl", "openssh-clients"])

And, pip install Python packages with the reqs_py argument.

# Install numerical packages for Python.
img = co.Image(image="python:3.8-slim", reqs_py=["numpy", "pandas", "matplotlib"])

Specify code to copy into the image

You can copy code into your image from a git repository or a local directory. To copy code from a git repository, specify copy_url and copy_branch.

# Copy code from a public Git repo. Also works for a private
# repo with Conducto's GitHub integration installed. 
img = co.Image(..., copy_url=f"https://github.com/{owner}/{repo}", copy_branch="main")

# Copy code from a private repo with a GitHub personal access token.
img = co.Image(..., copy_url=f"https://{user}:{token}@github.com/{owner}/{repo}", copy_branch="staging")

Use copy_dir to copy code from a local directory and all its descendants. This is great for quickly prototyping and iterating on your pipeline. Note that this requires installing local mode.

# Recursively copy code from the specified directory. Paths are
# relative to this file's directory. Examples:
img = co.Image(..., copy_dir=".") # The directory containing this file.
img = co.Image(..., copy_dir="../..") # The parent's parent.
img = co.Image(..., copy_dir="code/services") # A subdirectory. 

Specify everything with your own Dockerfile

Alternatively, you can specify your own Dockerfile and handle all of the software and code installation yourself. For example, you could have a Dockerfile that looks like this.

# Name this file Dockerfile.simple
# Create an image with Python and R, some packages, and some of your code.
FROM python:3.8
RUN apt-get update && apt-get install -y r-base
RUN Rscript -e 'install.packages(c("jsonlite", "argparser"))'
RUN pip3 install numpy pandas
COPY ./code /mnt/code

Then you would specify your image like this:

img = co.Image(dockerfile="./Dockerfile.simple")

Learn more

See a more detailed explanation of how Conducto builds Docker images.

Chat with us for a live demo right now!
(If we're awake 😴)

avatar