#10-2020-Mar-“Docker recipes and lessons”

Clean docker garbage

docker system prune # Warning deletes old images do your image pushes first if you have un-commited work

Fast builds: The whole base directory is copied before the image is built

Do you have problem with slow builds?

tl;dr: I was waiting many seconds before anything would happen on my builds. I didn’t know that the whole base directory is copied before the build steps.

Let’s say you have a repo structure:

├── bash # scripts and git hook
├── build # docker/helm/kubernetes/etc.
├── docs # documentation
├── dropbox # big files
├── dump # temporary files
├── frontend # frontend js files
├── htmlcov # output from coverage
├── src # actual src code
├── src_legacy 
├── tests # actual test code
├── tests_legacy

Then in your docker directory, build/docker/service you have docker-compose.yaml:

services:
  service:
    build:
      context: ../../..
      dockerfile: build/docker/service/Dockerfile

Then you have your build/docker/service/Dockerfile

# ...
COPY build/docker/service/requirements.txt ./
COPY src/jsonpath_rw ./jsonpath_rw
COPY src/kafka_config ./kafka_config
COPY src/service ./service
COPY src/utils ./utils 

It is effective and readable to use the repo path as base. But unfortunately you have to suffer in build time since every directory is copied into the docker daemon.

From the docker documentation

The build is run by the Docker daemon, not by the CLI. The first thing a build process does is send the entire context (recursively) to the daemon. In most cases, it’s best to start with an empty directory as context and keep your Dockerfile in that directory. Add only the files needed for building the Dockerfile.

How do you solve this?

  1. Use small repositories and keep docker-compose files and Dockerfiles close to the src code -> I don’t like this solution since having multiple repos makes it harder to reuse code. (Discussion could be a blog post itself)

  2. Export src code to a temporary directory and start the build from the temporary directory -> The challenge here is to do this consistently. But if you have a few scripts you should be golden. I hope to share my solution on this.

  3. Manually copy paste you src directory to build/docker/service -> Boring as hell

Conclusion: Reading documentation is valuable

I love using docker. But I didn’t understand why it was so slow. Was it because of os-x and not ubuntu. I really thought something was wrong. But reading the documentation carefully and understand what is happening is often the best way.

Upgrading a package in the docker image breaks the image (e.g., python3.7 -> python3.8)

This week I ran into an issue when upgrading a python service that uses ROS. Dockerfile

FROM ros:melodic-ros-core

RUN apt-get update &&\
    apt-get install -y \
        net-tools \
        iputils-ping \
        dnsutils \
        nano \
        python3-pip \
        python3-yaml \
        python3.8 \
        python3.8-dev \
        python3.8-distutils \
        python-catkin-pkg \
        python3-catkin-pkg-modules \
        python3-rospkg-modules
        
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod 755 /usr/local/bin/entrypoint.sh

COPY requirements.txt ./requirements.txt
RUN python3.8 -m pip install -r /requirements.txt

WORKDIR /usr/local/bin
COPY src ./

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["python3.8", "/usr/local/bin/apps/my_app.py"]

When running this the following error happened:

my_app_1  | Traceback (most recent call last):
my_app_1  |   File "/usr/local/bin/apps/my_app.py", line 81, in <module>
my_app_1  |     main()
my_app_1  |   File "/usr/local/bin/apps/my_app.py", line 77, in main
my_app_1  |     app.launch_from_sync()
my_app_1  |   File "/usr/local/bin/flow/app.py", line 219, in launch_from_sync
my_app_1  |     loop.run_until_complete(self.launch_and_block())
my_app_1  |   File "/usr/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
my_app_1  |     return future.result()
my_app_1  |   File "/usr/local/bin/flow/app.py", line 207, in launch_and_block
my_app_1  |     await self.launch()
my_app_1  |   File "/usr/local/bin/flow/app.py", line 143, in launch
my_app_1  |     await blocking_source.listen(main_thread=True)
my_app_1  |   File "/usr/local/bin/event_sources/ros_source.py", line 21, in listen
my_app_1  |     self.ros.start()
my_app_1  |   File "/usr/local/bin/aws_ros_bridge/from_ros_command_to_ros.py", line 82, in start
my_app_1  |     import rospy
my_app_1  |   File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/__init__.py", line 49, in <module>
my_app_1  |     from .client import spin, myargv, init_node, \
my_app_1  |   File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/client.py", line 61, in <module>
my_app_1  |     import rospy.impl.rosout 
my_app_1  |   File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/rosout.py", line 45, in <module>
my_app_1  |     from rospy.topics import Publisher, Subscriber
my_app_1  |   File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/topics.py", line 1364, in <module>
my_app_1  |     set_topic_manager(_TopicManager())
my_app_1  |   File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/topics.py", line 1132, in __init__
my_app_1  |     _logger.info("topicmanager initialized")
my_app_1  |   File "/usr/lib/python3.8/logging/__init__.py", line 1434, in info
my_app_1  |     self._log(INFO, msg, args, **kwargs)
my_app_1  |   File "/usr/lib/python3.8/logging/__init__.py", line 1565, in _log
my_app_1  |     fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
my_app_1  | TypeError: findCaller() takes from 1 to 2 positional arguments but 3 were given
my_app_1  | Error in atexit._run_exitfuncs:
my_app_1  | Traceback (most recent call last):
my_app_1  |   File "/usr/lib/python3.8/logging/__init__.py", line 1565, in _log
my_app_1  |     fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
my_app_1  | TypeError: findCaller() takes from 1 to 2 positional arguments but 3 were given

Recipe: Fix a 3rd party file from the docker image

  1. Copy the error throwing file from the image copy file from docker container to host

    docker cp <containerId>:/opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/roslogging.py ./roslogging.py
    
  2. Fix the file

    class RospyLogger(logging.getLoggerClass()):
        def findCaller(
            self, dummy=False, dummy2=False
        ):  # Dummy second arg to match Python3 function declaration
            # Dummy 3rd arg to match Python 3.8 -> This was the change needed
    
  3. Overwrite the file by using the Dockerfile

            python3-catkin-pkg-modules \
            python3-rospkg-modules
    
    COPY roslogging.py /opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/roslogging.py # NEW LINE
    COPY entrypoint.sh /usr/local/bin/entrypoint.sh
    RUN chmod 755 /usr/local/bin/entrypoint.sh
    

Conclusion: Monkey patching a docker container is useful

Knowing how to fix a docker image can make it easier for you to stay with the latest packages which makes the life of a developer more productive and fun.

Fix 3rd party containers from failing

docker-compose up --force-recreate