Install Scrapy On Windows Server 2019, Running In A Docker Container

I want to install Scrapy on Windows Server 2019, running in a Docker container (please see here and here for the history of my installation). On my local Windows 10 machine I can r

Solution 1:

To run a containerised app, it must be installed in a container image first - you don't want to install any software on the host machine.

For linux there are off-the-shelf container images for everything which is probably what your docker desktop environment was using; I see 1051 results on docker hub search for scrapy but none of them are windows containers.

The full process of creating a windows container from scratch for an app is:

• Get steps to manually install the app (scrapy and its dependencies) on Windows Server - ideally test in a virtualised environment so you can reset it cleanly
• Convert all steps to a fully automatic powershell script (e.g. for conda, need to download the installer via wget, execute the installer etc.
• Optionaly, test the powershell steps in an interactive container
  • docker run -it --isolation=process mcr.microsoft.com/windows/servercore:ltsc2019 powershell
  • This runs a windows container and gives you a shell to verify that your install script works
  • When you exit the shell the container is stopped
• Create a Dockerfile
  • Use mcr.microsoft.com/windows/servercore:ltsc2019 as the base image via FROM
  • Use the RUN command for each line of your powershell script

I tried installing scrapy on an existing windows Dockerfile that used conda / python 3.6, it threw error SettingsFrame has no attribute 'ENABLE_CONNECT_PROTOCOL' at a similar stage.

However I tried again with miniconda and python 3.8, and was able to get scrapy running, here's the dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN setx /M PATH $('C:\Users\ContainerAdministrator\miniconda3\Library\bin;C:\Users\ContainerAdministrator\miniconda3\Scripts;C:\Users\ContainerAdministrator\miniconda3;' + $Env:PATH)
RUN Invoke-WebRequest "https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Windows-x86_64.exe" -OutFile miniconda3.exe -UseBasicParsing; \
    Start-Process -FilePath 'miniconda3.exe' -Wait -ArgumentList '/S', '/D=C:\Users\ContainerAdministrator\miniconda3'; \
    Remove-Item .\miniconda3.exe; \
    conda install -y -c conda-forge scrapy;

Build it with docker build . -t scrapy and run with docker run -it scrapy.

To verify you are running a shell inside the container run whoami - should return user manager\containeradministrator.

Then, scrapy benchmark command should be able to run and dump some stats. The container will stop when you close the shell.