This repository has been archived by the owner on Feb 7, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
Water meters #1
Open
anton-br
wants to merge
61
commits into
master
Choose a base branch
from
water_meters
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Water meters #1
Changes from 34 commits
Commits
Show all changes
61 commits
Select commit
Hold shift + click to select a range
648928a
Add initial files and dataset
anton-br ce4b1b0
Add batch file, example and convert to blosc file
anton-br 886f62e
Rename water_batch to meter_batch, add README, change argv in convert…
anton-br 10bbfcb
correction of a typographical error in README, write right batch name…
anton-br ac5988b
Update readme, add discription to examples and add dosc to convert_to…
anton-br 15cbc98
Add docsting and update MeterBatch
anton-br dac1b0a
Fix spaces in README
anton-br 8048b76
Update README
anton-br 1392616
Update docstring
anton-br 262889a
Update shippable, add docs in meters_batch, fix import order in conve…
anton-br 1d7af66
Update docs and delete extra files
anton-br b08a930
Fix pylint
anton-br 9c97547
update __init__ and delete extra files
anton-br 849a5e2
Fixed imports in examples
anton-br 90e3c35
Added normal spaces into index and dell _build
anton-br 6b13b1d
Add file with lead labels and coordinates to one type
anton-br eb485c9
fix spaces in README, docs and utils
anton-br 6ea7950
Add components into convert_to_blosc and fix index columns in format_…
anton-br d1776db
Added simple pipeline with load and crop pipelines
anton-br d29222b
delete dataset as part of the repository
anton-br ca9033e
Add simple pipelines from pipelines, update meter_batch and fix error…
anton-br 619ba80
Update batch class in example, change docstrings in pipelines
anton-br 27bf5d8
Added docstring for pipelines and update README
anton-br 8c7993d
Update batch class and pipelines
anton-br 96701d4
Update docstirng, README and pylintrc
anton-br 7a158ca
update docs in convert to blosc
anton-br 2f1fd36
Update examples
anton-br 1f1ac8a
delete space in convert_to_blosc and add new line
anton-br f6550bb
Update utils. In format_data, labels and coondinates will save in one…
anton-br 38fd751
Rename functions in pipelines and batch
anton-br 56d4b3f
Update Readme and docs
anton-br 2261584
Fix error in dosc and update README
anton-br 0e84ab9
Fix load function and update dataset
anton-br 4430c6f
Fix error with copy in pipelines
anton-br 4a712e5
Update docs in meters_batch and pipeline class
anton-br ce416c4
Fixed docs in meters_batch, pipelines and format_data, version check …
anton-br 046056a
Updated docstring, batch class and pipeline factory
anton-br 4f90518
Fixed utils and added --recursive into index.rst
anton-br da34e41
Splinted coordinates on four columns
anton-br 4d21c1f
Add bb train and pred
anton-br c32af91
Add pipelines with train model to predict coordinates of bb and updat…
anton-br d7e9353
Add full predict pipeline and update batch class
anton-br abbad1b
Added pipeline factory example
anton-br 679ca9e
update pipeline and batch class
anton-br 76fb2ac
Update examples
anton-br afe89b6
Add working version of pipeline factory examples and actual version o…
anton-br 78c45b1
add comonents dict to pipelines and update batch class
anton-br 4c03ae5
add docsting into class pipeline factory
anton-br e5d696d
update dataset and add examples of pipeline factory
anton-br c0edc4f
update docs and added examples without pretrained model
anton-br 214a127
Update examples and change the preloaded models type to "static"
anton-br 1a8daf1
par actions return tuples
GregoryIvanov 190d9d9
Added pipeline that contains all loaded models
anton-br 230dcb2
Update dataset
anton-br abeb29a
deleted prints from batch class
anton-br 1063857
update format_data
anton-br 2f594d4
added percpective transform action
anton-br f97da77
update dataset
anton-br 409876c
fix pylint
anton-br fca44ec
fixed error with float coordinates of bbox
anton-br de17ca1
Deleted exception name in format_data and fixed bug in meter_batch
anton-br File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Set the default behavior, in case people don't have core.autocrlf set. | ||
* text=auto | ||
|
||
# Explicitly declare text files you want to always be normalized on checkout. | ||
*.py text | ||
*.sh text |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
*.pyc | ||
.cache/* | ||
__pycache__ | ||
__pycache__/* | ||
*/__pycache__/* | ||
.ipynb_checkpoints |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
[submodule "meters/dataset"] | ||
path = meters/dataset | ||
url = https://github.com/analysiscenter/dataset.git |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,58 @@ | ||
# meters | ||
# meters | ||
|
||
`Meters` is a framework for building end-to-end machine learning models for recognition of digits on any meters. | ||
|
||
Main features: | ||
|
||
* convert images to blosc format | ||
* load images(from blosc) and labels (from csv) | ||
* separate numbers (and labels respectively) from cropped area with meter's value | ||
* arrange new custom actions into pipeline | ||
* easily configure pipeline | ||
* build, train and test custom models for research. | ||
|
||
# About Meters | ||
|
||
Meters has two module: [``batch``](https://github.com/analysiscenter/meters/tree/master/meters/batch) and [``pipelines``](https://github.com/analysiscenter/meters/tree/master/meters/pipelines) | ||
|
||
``batch`` module contains MeterBatch class which includes actions for preprocessing. | ||
|
||
``pipelines`` module contains PipelineFactory class that builds pipelines for: | ||
|
||
* training simple models to classify digits on meters | ||
* making predictions by trained model. | ||
|
||
# Basic usage | ||
|
||
Here is an example of a pipeline that loads blosc images, makes preprocessing and trains a model for 50 epochs: | ||
```python | ||
ppl = ( | ||
dataset.train | ||
.load(src=src, fmt='blosc', components='images') | ||
.load(src='path/to/data.csv', fmt='csv', components=['coordinates', 'labels'], index_col='file_name') | ||
.crop_from_bbox() | ||
.split_labels() | ||
.split_to_digits() | ||
.init_model('dynamic', MeterModel, 'meter_model', config=model_config) | ||
.train_model('meter_model', fetches='loss', make_data=concatenate_water, save_to=V('loss'), mode='a') | ||
.run(batch_size=25, shuffle=True, drop_last=True, n_epochs=50) | ||
) | ||
``` | ||
|
||
# Installation | ||
|
||
With [git clone](https://git-scm.com/docs/git-clone): | ||
|
||
git clone https://github.com/analysiscenter/meters.git | ||
|
||
If your python file is located in another directory, you might need to add a path to `meters` submodule location:: | ||
|
||
import sys | ||
sys.path.insert(0, '/path/to/meters') | ||
import meters | ||
|
||
# Citing Meters | ||
|
||
Please cite Meters in your publications if it helps your research: | ||
|
||
``Khudorozhkov R., Broilovskiy A., Mylzenova D., Ivanov G. Meters library for data science research of meters. 2018.`` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
*/__pycache__/* | ||
.ipynb_checkpoints |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Minimal makefile for Sphinx documentation | ||
# | ||
|
||
# You can set these variables from the command line. | ||
SPHINXOPTS = | ||
SPHINXBUILD = python -msphinx | ||
SPHINXPROJ = Meters | ||
SOURCEDIR = . | ||
BUILDDIR = _build | ||
|
||
# Put it first so that "make" without argument is like "make help". | ||
help: | ||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) | ||
|
||
.PHONY: help Makefile | ||
|
||
# Catch-all target: route all unknown targets to Sphinx using the new | ||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). | ||
%: Makefile | ||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
MeterBatch | ||
========== | ||
.. autoclass:: meters.batch.MeterBatch | ||
:show-inheritance: | ||
|
||
Methods | ||
------- | ||
Input methods | ||
^^^^^^^^^^^^^ | ||
.. automethod:: meters.batch.MeterBatch.load | ||
|
||
Hidden methods | ||
^^^^^^^^^^^^^^ | ||
.. automethod:: meters.batch.MeterBatch._init_component | ||
.. automethod:: meters.batch.MeterBatch._reraise_exceptions | ||
.. automethod:: meters.batch.MeterBatch._assemble_load | ||
|
||
Batch processing | ||
^^^^^^^^^^^^^^^^ | ||
.. automethod:: meters.batch.MeterBatch.split_to_digits | ||
.. automethod:: meters.batch.MeterBatch.crop_from_bbox | ||
.. automethod:: meters.batch.MeterBatch.split_labels |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
PipelineFactory | ||
=============== | ||
.. autoclass:: meters.pipelines.PipelineFactory | ||
:show-inheritance: | ||
|
||
Methods | ||
------- | ||
Methods for creating a pipeline | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
.. automethod:: meters.pipelines.PipelineFactory.load_all | ||
.. automethod:: meters.pipelines.PipelineFactory.make_digits | ||
.. automethod:: meters.pipelines.PipelineFactory.simple_train | ||
.. automethod:: meters.pipelines.PipelineFactory.simple_predict | ||
|
||
Secondary methods | ||
^^^^^^^^^^^^^^^^^ | ||
|
||
.. automethod:: meters.pipelines.PipelineFactory._update_config | ||
.. automethod:: meters.pipelines.PipelineFactory.add_lazy_run |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
API | ||
==== | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
:titlesonly: | ||
|
||
meters.batch | ||
meters.pipelines |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,220 @@ | ||
#!/usr/bin/env python3 | ||
# -*- coding: utf-8 -*- | ||
# | ||
# meters documentation build configuration file, created by | ||
# sphinx-quickstart on Wed Jan 10 15:41:41 2018. | ||
# | ||
# This file is execfile()d with the current directory set to its | ||
# containing dir. | ||
# | ||
# Note that not all possible configuration values are present in this | ||
# autogenerated file. | ||
# | ||
# All configuration values have a default; values that are commented out | ||
# serve to show the default. | ||
|
||
# If extensions (or modules to document with autodoc) are in another directory, | ||
# add these directories to sys.path here. If the directory is relative to the | ||
# documentation root, use os.path.abspath to make it absolute, like shown here. | ||
|
||
import os | ||
import sys | ||
sys.path.insert(0, os.path.join("..")) | ||
import meters | ||
|
||
# -- General configuration ------------------------------------------------ | ||
|
||
# If your documentation needs a minimal Sphinx version, state it here. | ||
# | ||
# needs_sphinx = '1.0' | ||
|
||
# Add any Sphinx extension module names here, as strings. They can be | ||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom | ||
# ones. | ||
|
||
extensions = [ | ||
'sphinx.ext.autodoc', | ||
'sphinx.ext.doctest', | ||
'sphinx.ext.coverage', | ||
'sphinx.ext.mathjax', | ||
'sphinx.ext.viewcode', | ||
'sphinx.ext.githubpages', | ||
'sphinx.ext.intersphinx', | ||
'sphinx.ext.napoleon' | ||
] | ||
|
||
napoleon_use_rtype = False | ||
|
||
# Add any paths that contain templates here, relative to this directory. | ||
templates_path = ['_templates'] | ||
|
||
# The suffix(es) of source filenames. | ||
# You can specify multiple suffix as a list of string: | ||
# | ||
source_suffix = ['.rst', '.md'] | ||
|
||
# The master toctree document. | ||
master_doc = 'index' | ||
|
||
# General information about the project. | ||
project = 'Meters' | ||
copyright = '2018, Analysis Center' | ||
author = 'Analysis Center' | ||
|
||
# The version info for the project you're documenting, acts as replacement for | ||
# |version| and |release|, also used in various other places throughout the | ||
# built documents. | ||
# | ||
# The short X.Y version. | ||
release = meters.__version__ | ||
# The full version, including alpha/beta/rc tags. | ||
version = '.'.join(release.split('.')[:2]) | ||
|
||
# The language for content autogenerated by Sphinx. Refer to documentation | ||
# for a list of supported languages. | ||
# | ||
# This is also used if you do content translation via gettext catalogs. | ||
# Usually you set "language" from the command line for these cases. | ||
language = None | ||
|
||
# List of patterns, relative to source directory, that match files and | ||
# directories to ignore when looking for source files. | ||
# This patterns also effect to html_static_path and html_extra_path | ||
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] | ||
|
||
# The name of the Pygments (syntax highlighting) style to use. | ||
pygments_style = 'sphinx' | ||
|
||
# If true, `todo` and `todoList` produce output, else they produce nothing. | ||
todo_include_todos = False | ||
|
||
|
||
# -- Options for HTML output ---------------------------------------------- | ||
|
||
# The theme to use for HTML and HTML Help pages. See the documentation for | ||
# a list of builtin themes. | ||
# | ||
# html_theme = 'alabaster' | ||
html_theme = "classic" | ||
# html_theme_options = { | ||
# "rightsidebar": "true", | ||
# "relbarbgcolor": "black" | ||
# } | ||
# Theme options are theme-specific and customize the look and feel of a theme | ||
# further. For a list of options available for each theme, see the | ||
# documentation. | ||
# | ||
# html_theme_options = {} | ||
|
||
# Add any paths that contain custom static files (such as style sheets) here, | ||
# relative to this directory. They are copied after the builtin static files, | ||
# so a file named "default.css" will overwrite the builtin "default.css". | ||
html_static_path = ['_static'] | ||
|
||
# Custom sidebar templates, must be a dictionary that maps document names | ||
# to template names. | ||
# | ||
# This is required for the alabaster theme | ||
# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars | ||
# html_sidebars = { | ||
# '**': [ | ||
# 'relations.html', # needs 'show_related': True theme option to display | ||
# 'searchbox.html', | ||
# ] | ||
# } | ||
|
||
|
||
autoclass_content = 'class' | ||
add_module_names = False | ||
|
||
intersphinx_mapping = { | ||
'python': ('https://docs.python.org/3', None), | ||
'numpy': ('http://docs.scipy.org/doc/numpy/', None), | ||
'scipy': ('http://docs.scipy.org/doc/scipy/reference/', None), | ||
'pandas': ('http://pandas-docs.github.io/pandas-docs-travis/', None), | ||
} | ||
|
||
viewcode_import = True | ||
|
||
# -- Options for HTMLHelp output ------------------------------------------ | ||
|
||
# Output file base name for HTML help builder. | ||
htmlhelp_basename = 'Metersdoc' | ||
|
||
|
||
# -- Options for LaTeX output --------------------------------------------- | ||
|
||
latex_elements = { | ||
# The paper size ('letterpaper' or 'a4paper'). | ||
# | ||
# 'papersize': 'letterpaper', | ||
|
||
# The font size ('10pt', '11pt' or '12pt'). | ||
# | ||
# 'pointsize': '10pt', | ||
|
||
# Additional stuff for the LaTeX preamble. | ||
# | ||
# 'preamble': '', | ||
|
||
# Latex figure (float) alignment | ||
# | ||
# 'figure_align': 'htbp', | ||
} | ||
|
||
# Grouping the document tree into LaTeX files. List of tuples | ||
# (source start file, target name, title, | ||
# author, documentclass [howto, manual, or own class]). | ||
latex_documents = [ | ||
(master_doc, 'meters.tex', 'meters Documentation', | ||
'Analysis Center', 'manual'), | ||
] | ||
|
||
|
||
# -- Options for manual page output --------------------------------------- | ||
|
||
# One entry per manual page. List of tuples | ||
# (source start file, name, description, authors, manual section). | ||
man_pages = [ | ||
(master_doc, 'meters', 'meters Documentation', | ||
[author], 1) | ||
] | ||
|
||
|
||
# -- Options for Texinfo output ------------------------------------------- | ||
|
||
# Grouping the document tree into Texinfo files. List of tuples | ||
# (source start file, target name, title, author, | ||
# dir menu entry, description, category) | ||
texinfo_documents = [ | ||
(master_doc, 'meters', 'meters Documentation', | ||
author, 'meters', 'One line description of project.', | ||
'Miscellaneous'), | ||
] | ||
|
||
|
||
|
||
# -- Options for Epub output ---------------------------------------------- | ||
|
||
# # Bibliographic Dublin Core info. | ||
# epub_title = project | ||
# epub_author = author | ||
# epub_publisher = author | ||
# epub_copyright = copyright | ||
|
||
# # The unique identifier of the text. This can be a ISBN number | ||
# # or the project homepage. | ||
# # | ||
# # epub_identifier = '' | ||
|
||
# # A unique identification for the text. | ||
# # | ||
# # epub_uid = '' | ||
|
||
# # A list of files that should not be packed into the epub file. | ||
# epub_exclude_files = ['search.html'] | ||
|
||
|
||
|
||
# # Example configuration for intersphinx: refer to the Python standard library. | ||
# intersphinx_mapping = {'https://docs.python.org/': None} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
two modules