Merge pull request #19 from Deric-W/v2

V2

.gitignore

@@ -0,0 +1,5 @@
+*/__pycache__
+dist
+notes.txt
+Test.html
+

.travis.yml

@@ -0,0 +1,36 @@
+language: python
+python:
+  - "3.5"
+  - "3.6"
+  - "3.7"
+  - "3.8"
+  - "pypy3"
+env:
+  - QUERY_STRING='test0=Hello&Test1=World%21&Test2=&Test3&&test0=World!' HTTP_COOKIE='test0=Hello ; Test1 = World%21 = Hello; Test2 = ;Test3;;test0=World!; ;'
+sudo: required
+install:
+  - python3 setup.py bdist_wheel
+  - version=$(python3 setup.py --version)
+  - cd debian
+  - . ./build_deb.sh pyhp-test ../dist/pyhp_core-$version-py3-none-any.whl pip
+  - sudo dpkg -i pyhp-test.deb
+  - cd ../tests
+script:
+  - pyhp embedding/syntax.pyhp|diff embedding/syntax.output -
+  - pyhp embedding/shebang.pyhp|diff embedding/shebang.output -
+  - pyhp embedding/indentation.pyhp|diff embedding/indentation.output -
+  - test -f /lib/pyhp/cache_handlers/files_mtime.py
+  - test ! -f ~/.cache/pyhp/$(pwd)/embedding/syntax.pyhp.cache
+  - pyhp --caching embedding/syntax.pyhp|diff embedding/syntax.output -
+  - test -f ~/.cache/pyhp/$(pwd)/embedding/syntax.pyhp.cache
+  - pyhp --caching embedding/syntax.pyhp|diff embedding/syntax.output -
+  - pyhp request/methods.pyhp|diff request/methods.output -
+  - pyhp request/request-order.pyhp --config request/request-order.conf|diff request/request-order.output -
+  - pyhp header/header.pyhp|diff header/header.output -
+  - pyhp header/headers_list.pyhp|diff header/headers_list.output -
+  - pyhp header/header_remove.pyhp|diff header/header_remove.output -
+  - pyhp header/headers_sent.pyhp|diff header/headers_sent.output -
+  - pyhp header/header_register_callback.pyhp|diff header/header_register_callback.output -
+  - pyhp cookie/setrawcookie.pyhp|diff cookie/setrawcookie.output -
+  - pyhp cookie/setcookie.pyhp|diff cookie/setcookie.output -
+  - pyhp shutdown_functions/register_shutdown_function.pyhp|diff shutdown_functions/register_shutdown_function.output -

README.md

@@ -1,57 +1,79 @@
-# PyHP-Interpreter
+# PyHP-Interpreter  [![Build Status](https://travis-ci.org/Deric-W/PyHP.svg?branch=master)](https://travis-ci.org/Deric-W/PyHP)
 
-The PyHP Interpreter is a script that allows you to embed Python code like PHP code into HTML.
+The PyHP Interpreter is a package that allows you to embed Python code like PHP code into HTML and other text files.
 The script is called either by the configuration of the web server or a shebang and communicates with the web server via CGI.
 
 ## Features:
+
   - Parser for embedding python Code in HTML
-  - Encapsulation of the variables and functions of the interpreter in a separate class (to prevent accidental overwriting)
+  - a bunch of PHP features implemented in python
+  - modular structure to allow the use of features outside of the interpreter
+  - automatic code alignment for improved readability
   - caching
-  - PHP like header functions
-  - PHP like SERVER array (Dictionary)
-  - PHP like REQUEST,GET,POST and COOKIE array (Dictionary)
-  - PHP like setrawcookie and setcookie functions
 
 ## How it works:
+
  - Python code is contained within the `<?pyhp` and `?>` tags (like PHP)
- - the Script is called like a interpreter, with the filepath as cli parameter
- - if no filepath is given, the script is reading from stdin
- - if "-c" is given, the file will be processed an cached in cache_path/absolute/path/filename.cache
-   (the file is also loaded or renewed with this option)
- - python code can be away from the left site of the file for better optics --> Test4.pyhp, fib.pyhp
- - the following PHP features are available as part of the `pyhp` class:
-  - `$_REQUEST` as REQUEST
-  - `$_GET`as GET
-  - `$_POST`as POST
-  - `$_COOKIE`as COOKIE
-  - `$_SERVER` as SERVER
-  - `http_response_code`
-  - `headers_list`
-  - `header`
-  - `header_remove`
-  - `headers_sent`
-  - `header_register_callback`
-  - `setrawcookie`
-  - `setcookie`
-  - `register_shutdown_function`
- - automatic sending of headers with fallback: `Content-Type: text/html`
+ - the program is called like a interpreter, with the filepath as cli parameter
+ - if no filepath is given, the program is reading from stdin
+ - if the `-c` or `--caching` is given, the cache will be enabled and the file will additionally be preprocessed if needed 
+   and cached in cache_path/absolute/path/of/filename.cache
+ - python code is allowed to have a starting indentation for better readability inside (for example) HTML files
+ - the following PHP features are available as methods of the `PyHP` class (available from the outside in pyhp.libpyhp):
+     - `$_SERVER` as `SERVER`
+     - `$_REQUEST` as `REQUEST`
+     - `$_GET` as `GET`
+     - `$_POST` as `POST`
+     - `$_COOKIE` as `COOKIE`
+     - `http_response_code`
+     - `header`
+     - `headers_list`
+     - `header_remove`
+     - `headers_sent`
+     - `header_register_callback`
+     - `setcookie` with an additional `samesite` keyword argument
+     - `setrawcookie` also with an additional `samesite` keyword argument
+     - `register_shutdown_function`
+     
 
   ## Cache Handlers
-   - are responsible for saving/loading/renewing caches
-   - are python scripts with the following contents:
-   - the `handler` class, wich takes the cache path and absolute file path as initialization parameters
-   - the method `is_outdated`, wich returns True or False
-   - the method `save`, wich returns nothing and saves the boolean code_at_begin and preprocessed code
-   - the method `load`, wich returns a tuble with the boolean code_at_begin and the code saved by `save`
-   - the method `close`, wich does cleanup tasks
 
+ - are responsible for saving/loading/renewing caches
+ - are python scripts with the following contents:
+ - the `Handler` class, wich takes the cache path, absolute file path and `caching` section of the config file as  
+   initialization parameters and provides the following methods:
+     - `is_available`, wich returns a boolean indicating if the handler can be used
+     - `is_outdated`, wich returns a boolean indicating if the cache needs to be renewed
+     - `save`, wich takes an iterator as argument and saves it in the cache
+     - `load`, wich loads an iterator from the cache
+     - `close`, wich does cleanup tasks
+  - note that the iterator may contain code objects which can't be pickled
+  - examples are available in the *cache_handlers* directory
+
   ## Installation
-  ### Debian
-  Use the Debian package
-  ### Other
-  1. enable CGI for your web server
-  2. drop pyhp.py somewhere and mark it as executable (make sure Python 3.5+ is installed)
-  3. download pyhp.conf and move it to `/etc`
-  4. create `/lib/pyhp/cache_handlers` and drop the choosen cache handler (and maybe others) in the cache handler directory
-
-  Done! you can now use `.pyhp` files by adding a Shebang
+
+  This section shows you how to install PyHP on your computer.
+  If you want to use *pyhp* scripts on your website by CGI you have to additionally enable CGI in your webserver.
+
+  ### Just as python package
+  1. build the *pyhp-core* python package with `python3 setup.py bdist_wheel`
+  2. Done! You can now install the wheel contained in the *dist* directory with pip
+
+  ### As application
+  If you just installed the python package, then you have to provide `--config` with every call of `python3 -m pyhp`
+  and can't use the caching feature.
+  To stop this, you can build a debian package or install PyHP manually.
+
+  #### Debian package
+  1. build the *pyhp-core* python package with `python3 setup.py bdist_wheel`
+  2. go to the *debian* directory and execute `./build_deb.sh`
+  3. enter a package name, the path of the *pyhp-core* wheel and the pip command you wish to use
+  4. Done! You can now install the debian package with `sudo dpkg -i <package name>.deb`
+
+  #### Manually
+  1. install the *pyhp-core* python package
+  2. copy *pyhp.conf* to */etc*
+  3. copy *cache_handlers* to */lib/pyhp/*
+  4. copy *debian/pyhp* to a directoy in your PATH
+  5. Done! You can now use the `pyhp` command
+

TODO

@@ -17,7 +17,6 @@ session_encode
 session_decode
 $_SESSION
 
-add handler for memcached (if not already submitted)
-add handler for redis (if not already submitted)
+add handler for memcached
+add handler for redis
 
-wait for suggestions

cache_handlers/files_mtime.py

@@ -2,34 +2,52 @@
 
 """PyHP cache handler (files with modification time)"""
 
-import marshal
+import marshal  # not pickle because only marshal supports code objects
 import os.path
 from os import makedirs
+from time import time
 
-class handler:
-    def __init__(self, cache_path, file_path):
+
+class Handler:
+    def __init__(self, cache_path, file_path, config):
+        self.cache_prefix = cache_path
         self.cache_path = os.path.join(os.path.expanduser(cache_path), file_path.strip(os.path.sep) + ".cache")  # use full path to allow indentical named files in different directories with cache_path as root
         self.file_path = file_path
+        self.ttl = config.getint("ttl")
+        self.max_size = config.getint("max_size")
+
+    def get_cachedir_size(self):        # get size of cache directory (with all sub directories) in Mbytes
+        size = 0
+        for dirpath, dirnames, filenames in os.walk(self.cache_prefix, followlinks=False):
+            size += os.path.getsize(dirpath)        # dont forget the size of the directory
+            for filename in filenames:
+                filepath = os.path.join(dirpath, filename)
+                if not os.path.islink(filepath):        # dont count symlinks
+                    size += os.path.getsize(filepath)
+        return size / (1000 ** 2)       # bytes --> Mbytes
 
-    def is_outdated(self):                                                                          # return True if cache is not created or needs refresh
-        if not os.path.isfile(self.cache_path) or os.path.getmtime(self.cache_path) < os.path.getmtime(self.file_path):
-            return True
+    def is_available(self):     # if cache directory has free space or the cached file is already existing or max_size < 0
+        return self.max_size < 0 or os.path.isfile(self.cache_path) or self.get_cachedir_size() < self.max_size
+
+    def is_outdated(self):      # return True if cache is not created or needs refresh or exceeds ttl
+        if os.path.isfile(self.cache_path):     # to prevent Exception if cache not existing
+            cache_mtime = os.path.getmtime(self.cache_path)
+            file_mtime = os.path.getmtime(self.file_path)
+            age = time() - cache_mtime
+            return cache_mtime < file_mtime or age > self.ttl > -1      # age > ttl > -1 ignores ttl if -1 or lower
         else:
-            return False
+            return True     # file is not existing --> age = infinite
 
-    def load(self):
+    def load(self):  # load sections
         with open(self.cache_path, "rb") as cache:
-            cache_content = marshal.load(cache)
-        if len(cache_content) != 2:
-            raise ValueError("corrupted cache at " + self.cache_path)
-        else:
-            return cache_content[0], cache_content[1]                                               # file_content, code_at_begin
+            code = marshal.load(cache)
+        return code
 
-    def save(self, file_content, code_at_begin):
-        if not os.path.isdir(os.path.dirname(self.cache_path)):                                     # directories not already created
-            os.makedirs(os.path.dirname(self.cache_path), exist_ok=True)
+    def save(self, code):   # save sections
+        if not os.path.isdir(os.path.dirname(self.cache_path)):     # directories not already created
+            makedirs(os.path.dirname(self.cache_path), exist_ok=True)   # ignore already created directories
         with open(self.cache_path, "wb") as cache:
-            marshal.dump([file_content, code_at_begin], cache)
+            marshal.dump(code, cache)
 
     def close(self):
-        pass
+        pass    # nothing to do

debian/build_deb.sh

@@ -0,0 +1,65 @@
+#!/bin/sh -e
+# script for building the pyhp debian package
+# it is recommended to run this script as root or to set the owner and group of the files to root
+# you need to build the pyhp-core wheel first
+
+if [ "$1" = "" ]
+then read -p "Name: " package
+else package=$1
+fi
+
+if [ "$2" = "" ]
+then read -p "pyhp-core Wheel: " wheel
+else wheel=$2
+fi
+
+if [ "$3" = "" ]
+then read -p "pip executeable: " pip
+else pip=$3
+fi
+
+mkdir "$package"
+
+# place config file, cache handlers and "executable"
+mkdir -p "$package/lib/pyhp/cache_handlers"
+cp ../cache_handlers/files_mtime.py "$package/lib/pyhp/cache_handlers"
+
+mkdir "$package/etc"
+cp ../pyhp.conf "$package/etc"
+
+mkdir -p "$package/usr/bin"
+cp pyhp "$package/usr/bin"
+chmod +x "$package/usr/bin/pyhp"
+
+# place pyhp-core files
+mkdir -p "$package/usr/lib/python3/dist-packages"
+$pip install --target "$package/usr/lib/python3/dist-packages" --ignore-installed $wheel
+
+# place metadata files
+mkdir "$package/DEBIAN"
+cp conffiles "$package/DEBIAN"
+cp control "$package/DEBIAN"
+
+mkdir -p "$package/usr/share/doc/pyhp"
+cp copyright "$package/usr/share/doc/pyhp"
+cp changelog "$package/usr/share/doc/pyhp/changelog.Debian"
+gzip -n --best "$package/usr/share/doc/pyhp/changelog.Debian"
+
+# generate md5sums file
+chdir "$package"
+md5sum $(find . -type d -name "DEBIAN" -prune -o -type f -print) > DEBIAN/md5sums  # ignore metadata files
+chdir ../
+
+# if root set file permissions, else warn
+if [ $(id -u) = 0 ]
+then chown root:root -R "$package"
+else echo "not running as root, permissions in package may be wrong"
+fi
+
+# build debian package
+dpkg-deb --build "$package"
+
+# remove build directory
+rm -r "$package"
+
+echo "Done"

debian/changelog

@@ -1,3 +1,24 @@
+pyhp (2.0-1) stable; urgency=low
+
+  * fourth release
+  * add max_size and ttl caching options
+  * add more customizable Request handling
+  * add --config argument
+  * add --version argument
+  * add automatic response code setting to header
+  * add python wheel
+  * structural changes
+  * php functions now useable outside .pyhp files
+  * rename pyhp class to PyHP
+  * replace print wrapper with PyHP.make_header_wrapper
+  * changed cache handler interface
+  * rework register_shutdown_function to use atexit
+  * improve IndentationError message
+  * fix wrong directory size calculation in files_mtime
+  * fix crash of files_mtime.py if os not in namespace
+
+ -- Eric Wolf <[email protected]>  Sun, 26 Jan 2020 18:11:00 +0100
+
 pyhp (1.2-1) stable; urgency=low
 
   * third release
@@ -13,7 +34,6 @@ pyhp (1.1-1) stable; urgency=low
   * add header_register_callback
   * add config
   * reworked caching to use handlers (old code as files_mtime handler)
-  * reworked caching to use handlers (old code as files_mtime handler)
   * reworked prepare file
   * now using argparse
   * changed directory structure (see pyhp.conf)

debian/control

@@ -1,13 +1,13 @@
 Package: pyhp
-Version: 1.2-1
+Version: 2.0-1
 Architecture: all
 Maintainer: Eric Wolf <[email protected]>
-Installed-Size: 21
+Installed-Size: 31
 Depends: python3:any (>= 3.5)
 Suggests: apache2
 Section: web
 Priority: optional
-Homepage: https://github.com/Deric-W/PyHP-Interpreter
-Description: Interprets and executes pyhp files.
- PyHP is a script for interpreting and executing pyhp files, with several PHP functions available.
- pyhp files are (mostly) HTML files that have embedded Python source code and can be used for creating dynamic web pages.
+Homepage: https://github.com/Deric-W/PyHP
+Description: Application for embedding and using python code like php
+ PyHP is a application/python package for embedding python code in text files like HTML, 
+ with several PHP functions available.

debian/copyright

@@ -1,7 +1,7 @@
 Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
 Upstream-Name: pyhp
 Upstream-Contact: Eric Wolf <[email protected]>
-Source: https://github.com/Deric-W/PyHP-Interpreter
+Source: https://github.com/Deric-W/PyHP
 Copyright: 2019 Eric Wolf
 License: Expat