Chrome builds interrupted by module compilation failure

[jugglinmike]

Yesterday, 2 Chrome stable builds and 1 Chrome experimental builds failed as the WPT CLI installed Python dependencies. Specifically, the workers were unable to compile the native extensions provided by the cffi module (error log included below).

I haven't been able to find a conclusive explanation for this failure, but I suspect it was a transitory packaging issue. Here's why:

• all three builds failed within a small time window (less than 90 seconds)
• a new version of cffi was released at approximately that time](https://groups.google.com/forum/#!topic/python-cffi/AuYExAf8LSY)
• no changes were made to this project at that time
• no subsequent build has exhibited this problem (over 350 in total)

I'll keep my eyes peeled for future occurrences, but I'm hopeful that it was a passing glitch. I've manually re-triggered the failed builds; the corresponding data should be available on wpt.fyi shortly.

Partial error log

2019-02-13 13:39:57,247 INFO validate-wpt-results     python ./wpt run --log-raw /tmp/tmpPTpDNe/log-raw.txt --log-wptreport /tmp/tmpPTpDNe/report.json --log-mach - --this-chunk 8 --total-chunks 20 --binary /usr/bin/google-chrome --webdriver-binary /home/jade/worker/GNU_Linux_Chunked_Runner/build/chromedriver --install-fonts --no-headless --binary-arg=--use-fake-ui-for-media-stream --binary-arg=--use-fake-device-for-media-stream --binary-arg=--enable-experimental-web-platform-features chrome
2019-02-13 13:40:00,123 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
2019-02-13 13:40:01,010 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
2019-02-13 13:40:03,706 INFO validate-wpt-results wpt-run:stderr   Failed building wheel for cffi
2019-02-13 13:40:04,626 INFO validate-wpt-results wpt-run:stderr Command "/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-install-FhQJPN/cffi/
2019-02-13 13:40:04,676 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt') exited with return code 1
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:Collecting html5lib==1.0.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozinfo==0.10 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozlog==4.0 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/18/7b/37831f9ae29ea950dd2dd332ae49a684ffda3aaf30a72c7e6b275b080315/mozlog-4.0-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozdebug==0.1.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 4))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/cb/fb/cd673c7e9f7d4911d93c7aacd4d464ec8465d213c1ca63fcd4f59cc24ef7/mozdebug-0.1.1-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: urllib3[secure]==1.24.1 in ./_venv/lib/python2.7/site-packages (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (1.24.1)
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting six>=1.9 (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting webencodings (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozfile>=0.12 (from mozinfo==0.10->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/85/ed/4f8bec5ac04373ef0c79428bdd50ce039e91e4b53660d08e9a45988b74dc/mozfile-2.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozterm (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/ab/eb/0c53684f5821d666372c6ea03d8c64558c8f74fda0fd5b24ae3dd2ab6a95/mozterm-1.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting blessings>=1.3 (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/8d/b1/a3fe6fd8a012e6d019bafd671c2fee0597ea97ff2e76c25aadfa4545fc32/blessings-1.7-py2-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting pyOpenSSL>=0.14; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: idna>=2.0.0; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2.8)
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting cryptography>=1.3.4; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: certifi; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2018.11.29)
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting ipaddress; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting asn1crypto>=0.21.0 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting enum34; python_version < "3" (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,683 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr   Downloading https://files.pythonhosted.org/packages/10/fe/b6362c613a70ac29cf7cac36307d85f08ebe4a96d9d54b895b10a807e39b/cffi-1.12.0.tar.gz (453kB)
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Building wheels for collected packages: cffi
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr   Building wheel for cffi (setup.py): started
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   Building wheel for cffi (setup.py): finished with status 'error'
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-9U0S4d --python-tag cp27:
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running bdist_wheel
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running build
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running build_py
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   creating build
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   creating build/lib.linux-x86_64-2.7
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   creating build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   running build_ext
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   building '_cffi_backend' extension
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   creating build/temp.linux-x86_64-2.7
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   creating build/temp.linux-x86_64-2.7/c
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr    #include <ffi.h>
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr             ^~~~~~~
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   compilation terminated.
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   ----------------------------------------
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   Running setup.py clean for cffi
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Failed to build cffi
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Installing collected packages: six, webencodings, html5lib, mozfile, mozinfo, mozterm, blessings, mozlog, mozdebug, asn1crypto, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   Running setup.py install for cffi: started
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr     Running setup.py install for cffi: finished with status 'error'
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi:
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running install
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running build
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running build_py
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     creating build
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     creating build/lib.linux-x86_64-2.7
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     creating build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     running build_ext
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     building '_cffi_backend' extension
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     creating build/temp.linux-x86_64-2.7
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     creating build/temp.linux-x86_64-2.7/c
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr      #include <ffi.h>
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr               ^~~~~~~
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     compilation terminated.
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     ----------------------------------------
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr Traceback (most recent call last):
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr   File "./wpt", line 5, in <module>
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     wpt.main()
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 115, in main
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 95, in setup_virtualenv
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     venv.install_requirements(path)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/virtualenv.py", line 55, in install_requirements
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     call(self.pip_path, "install", "-r", requirements_path)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/utils.py", line 43, in call
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     return subprocess.check_output(args)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr     raise CalledProcessError(retcode, cmd, output=output)
2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr subprocess.CalledProcessError: Command '('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt')' returned non-zero exit status 1
2019-02-13 13:40:04,698 INFO validate-wpt-results WPT CLI exited with return code None

[gsnedders]

My guess is we don't have the libffi-dev package installed hence we couldn't build cffi from source, and the appropriate cffi wheel hadn't been uploaded yet?

[jugglinmike]

That sounds plausible and more satisfying than my feeble "transitory packaging issue."

[jugglinmike]

Surprisingly, two more builds just failed with the very same error (8 days later). This happened about 3 hours ago, and that directly correlates with the release of cffi 1.12.2.

@gsnedders @jgraham I know next to nothing about Python packaging, but this seems like something we ought to be able to avoid. Is there anything we can change in WPT to guard against invalid installations?

[jgraham]

Hah.

cffi seems to be a dep of pyOpenSSL, which depends on cffi [required: >=1.8,!=1.11.3, installed: 1.11.5]. I don't think that pinning transitive dependencies is supported in any good way, but what could work is this:

Adjust the requirements.txt file to specify a specific version of cffi. Assume that pip will notice that the pinned version is a match for the requirement and not try to install a newer version. Then rely on pyup.io to install newer versions which will come with enough delay to assure we don't end up with races for the wheel to be uploaded.

[jugglinmike]

Thanks, James! That looks like a fairly module-specific solution. Is that an indication of some bad practice in pyOpenSSL or cffi? Or is this a problem that Python developers experience commonly?

[jgraham]

Ooh, pip seems to have a --prefer-binary option. I think that might help here.

[jugglinmike]

Six more builds (3 Firefox, 2 Chrome, and 1 Edge) were interrupted today at 14:05 UTC. There's no new release of cffi to blame this time, though. @jgraham do you know how this could happen again? Could it occur when some transitive dependency transitions to the new release?

[jgraham]

Do you have logs?

[jugglinmike]

Sure do!

• edge, 18 of 100
• firefox stable, 7 of 20
• firefox stable, 8 of 20
• firefox stable, 10 of 20
• chrome stable, 6 of 20
• chrome experimental, 6 of 20

All for WPT commit 9f49616e7fa90489fac85dd203e86d7c83ca67da

[jgraham]

I'm not sure why it's trying to build it there; maybe something happened to the wheel package. In any case passing --prefer-binary or --only-binary=:all: to the pip command seems like it should avoid this failure mode.

[gsnedders]

At least the ones I looked at there aren't cffi, they're cryptography, which had two releases yesterday.

To expand on my previous comment: note that uploads aren't atomic on PyPI.

The common flow is:

1. Create a new release.
2. Upload an artifact.
3. If you have another artifact, goto 2; else, complete.

Each of these items happens sequentially, and if a source release artifact is uploaded before a (suitable) binary release artifact is anything looking for the latest release will download the source release and build it. As noted, either of the options @jgraham suggested will passing this will change this (preferring an older release with a binary release).

[jugglinmike]

We got hit by both releases of the "cryptography" package (the second release interfered with six builds for Firefox stable). I'm going to look in to patching WPT as you folks recommend. I'm curious, though: have either of you seen this occur in Taskcluster?