optional package notedown fail to install

[videlec]

the machine is running archlinux

Found local metadata for notedown-1.5.1
Using cached file /opt/sage/upstream/notedown-1.5.1.tar.gz
notedown-1.5.1
====================================================
Setting up build directory for notedown-1.5.1
Finished extraction
No patch files found in ../patches
****************************************************
Host system:
Linux mangouste 4.14.15-1-ARCH #1 SMP PREEMPT Tue Jan 23 21:49:25 UTC 2018 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp
Thread model: posix
gcc version 7.3.0 (GCC) 
****************************************************
Installing notedown-1.5.1
Traceback (most recent call last):
  File "setup.py", line 10, in <module>
    readme = pandoc.communicate()[0].decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3031: ordinal not in range(128)
Error: could not determine package name
********************************************************************************
Error installing notedown-1.5.1
********************************************************************************

real	0m0.896s
user	0m0.437s
sys	0m0.103s
************************************************************************
Error installing package notedown-1.5.1
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the log file
  /opt/sage/logs/pkgs/notedown-1.5.1.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/opt/sage/local/var/tmp/sage/build/notedown-1.5.1 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/opt/sage/local/var/tmp/sage/build/notedown-1.5.1' && '/opt/sage/sage' --sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************

Upstream report at aaren/notedown#76

Upstream: Reported upstream. No feedback yet.

Component: packages: optional

Author: Vincent Delecroix

Branch/Commit: e60f898

Reviewer: Ralf Stephan

Issue created by migration from https://trac.sagemath.org/ticket/24901

[kiwifb]

comment:1

There was a similar issue with fpylll unicode in the readme was causing a fit. This is an issue that needs fixing upstream by removing the offending stuff from the readme file.

[videlec]

comment:2

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

[videlec]

Upstream: Reported upstream. No feedback yet.

[videlec]

Branch: u/vdelecroix/24901

[videlec]

New commits:

e60f898

24901: patch for avoiding pandoc in setup.py

[videlec]

Author: Vincent Delecroix

[videlec]

Commit: e60f898

[vinklein]

comment:5

Replying to @videlec:

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

For the record i am not sure if this example alone show a bug in pandoc as pandoc seems to work in utf-8 by default (https://pandoc.org/MANUAL.html#character-encoding).

Pandoc don't seem to claim to keep the input encoding. In your exemple doing readme.decode('utf-8') instead of readme.decode() works fine.

On ubuntu 16.4 with pandoc 1.16.0.2 sage -i notedown works fine (sage version 8.2.beta7)

[videlec]

comment:6

Replying to @vinklein:

Replying to @videlec:

The README.md is perfectly ASCII. Apparently, pandoc is producing those weird characters

>>> pandoc = subprocess.Popen(['pandoc', 'README.md', '--to', 'rst'],
... stdout=subprocess.PIPE)
>>> readme = pandoc.communicate()[0]
>>> readme.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3325: ordinal not in range(128)
>>> readme[3320:3330]
'ython\xe2\x80\x99s '

For the record i am not sure if this example alone show a bug in pandoc as pandoc seems to work in utf-8 by default (https://pandoc.org/MANUAL.html#character-encoding).

Pandoc don't seem to claim to keep the input encoding. In your exemple doing readme.decode('utf-8') instead of readme.decode() works fine.

On ubuntu 16.4 with pandoc 1.16.0.2 sage -i notedown works fine (sage version 8.2.beta7)

I never claimed that it was a bug in pandoc. The commands are extracted from the setup.py and produce an error on my computer. As specified in the documentation the behavior of decode depends on some default encoding that might differ between platforms.

Replacing readme.decode() with readme.decode('utf-8') in setup.py solves the problem mentioned in the description. However, I am not sure that utf-8 characters are encouraged in the long_description argument of the setup (there is no way to specify encoding there).

[rwst]

Reviewer: Ralf Stephan

[rwst]

comment:7

This branch installs notedown fine here, both without and with system pandoc 1.19.2.1. LGTM.

[vbraun]

Changed branch from u/vdelecroix/24901 to e60f898