Merge pull request #6065 from LMFDB/main

main -> dev

Postgres_FAQ.md

@@ -13,8 +13,10 @@ Changes
 
 1. What's an overview of the changes?
 
-   There is a new folder, `lmfdb/backend/`, containing the
-   main components of the new interface to the Postgres database.
+   The main components of the interface to the Postgres database are
+   in [psycodict](https://github.com/roed314/psycodict/), which the LMFDB
+   now depends on. This might be moved to the lmfdb organization later.
+
    Postgres is a mature, open-source implementation of SQL.  One of
    the main differences is that Postgres is a strongly-typed
    relational database, meaning that every table has a schema with
@@ -48,11 +50,6 @@ Changes
    advantage of this restructuring to add some consistency to the
    naming scheme.
 
-1. Why do I still see messages about mongo when I start the LMFDB?
-
-   Classical modular forms are still using mongo.  We're working on
-   revising them to use postgres, but in the mean time we're still
-   connecting to a mongo database.
 
 Database Interface
 ------------------
@@ -105,7 +102,7 @@ Database Interface
    ```
 
    The first argument is a dictionary specifying the query (in a style
-   similar to what you're used to from MongoDB, but with custom
+   similar to what you may be used to from MongoDB, but with custom
    operators like `$contains` that are translated to Postgres
    expressions).  You can project in order to obtain only a certain
    set of columns, and provide limits, offsets and custom sort orders.
@@ -115,8 +112,8 @@ Database Interface
 
    The `info` argument is a dictionary that will be updated with
    various data that is commonly needed by templates populated by the
-   search functions.  For more details, see the documentation in the folder
-   `lmfdb/backend`.
+   search functions.  For more details, see the documentation at
+   https://github.com/roed314/psycodict/.
 
 1. What if I only want a single entry, for example with a specified label?
 
@@ -159,7 +156,7 @@ Database Interface
    If use `db._execute`, make sure to wrap your statements in the SQL
    class from `psycopg2.sql` (you can also import it from
    `psycodict`). You can see lots of examples of this
-   paradigm in `lmfdb/backend/`.
+   paradigm in https://github.com/roed314/psycodict/
 
    ```python
    sage: from psycodict import db, SQL
@@ -217,7 +214,7 @@ Developer configuration
 Adding and modifying data
 -------------------------
 
-Note that you need editor priviledges to add, delete or modify data.
+Note that you need editor privileges to add, delete or modify data.
 
 1. How do I create a new table?
 
@@ -231,18 +228,18 @@ Note that you need editor priviledges to add, delete or modify data.
    `lookup` method, typically the `label` column if there is one.
    You should provide a default sort order if your table will be the
    primary table behind a section of the website (auxiliary tables may
-   not need a sort order).  You also need to proved a short description
+   not need a sort order).  You also need to provide a short description
    of the table, which will be shown in the banner when its contents are
    viewed on the database section of the website, as well as short
    descriptions of each of its columns which will be shown when users
    view the schema for the table in the database section (these will
    be used to populate knowls that can then be edited by you or anyone
    with an LMFDB account can edit, they don't need to be perfect).
-   
-   You can also give columns for an extra table (see the question two prior),
-   using the same format as the second argument.  Finally, you can specify
-   the order of columns, which will be used by the `copy_from` and `copy_to`
-   functions by default.
+
+   You can also give columns for an extra table (see the question "What is an
+   `extra_table`?" later in this document), using the same format as the second
+   argument.  Finally, you can specify the order of columns, which will be used
+   by the `copy_from` and `copy_to` functions by default.
 
    ```python
    db.create_table(name='perfect_numbers',
@@ -265,6 +262,34 @@ Note that you need editor priviledges to add, delete or modify data.
                    search_order=['label', 'N', 'log_N', 'num_factors', 'mersenne_n', 'odd'])
    ```
 
+   If there were `extras` (suppose we put `log_N` and `num_factors` in
+   `extras`), the table creation could look like
+
+   ```python
+   db.create_table(name='perfect_numbers2',
+                   search_columns={'numeric': ['N','mersenne_n'],
+                                   'text': ['label'],
+                                   'bool': ['odd'],
+                                   },
+                   label_col='label',
+                   sort=['label'],
+                   table_description='perfect numbers',
+                   extra_columns={'double precision': ['log_N'],
+                                 'int': ['num_factors'],
+                                 },
+                   col_description={'N': "Integer value of the perfect number",
+                                    'log_N': "Natural logarithm of $N$",
+                                    'num_factors': "The number of factors of the perfect number.",
+                                    'mersenne_n': "For odd perfect numbers, the positive integer n for which $N=2^{n-1}(2^n-1)$, where $2^n-1$ is prime.  Set to zero for even perfect numbers",
+                                    'label': "Label of the perfect number",
+                                    'odd': "True if $N$ is odd, false otherwise.",
+                                   },
+                   search_order=['label', 'N', 'log_N', 'num_factors', 'mersenne_n', 'odd'])
+   ```
+
+   In `perfect_numbers2`, the columns `log_N` and `num_factors` are now not in
+   the search columns.
+
    Once this table exists, you can access it via the object
     `db.perfect_numers`, which is of type `PostgresTable`.
 
@@ -290,7 +315,7 @@ Note that you need editor priviledges to add, delete or modify data.
    ```
 
    This column will be NULL for existing rows.
-   
+
 1. How do I delete a column?
 
    If you want to delete a column to an existing table, use the
@@ -299,7 +324,7 @@ Note that you need editor priviledges to add, delete or modify data.
    ```python
    sage: db.test_table.drop_column("bad_primes")
    ```
-   
+
 1. How do I insert new data?
 
    There are two main methods for adding data to a table.
@@ -586,7 +611,7 @@ Statistics
    in which case the function will only compute counts of at least the
    threshold.  You can also specify a dictionary of constraints (in
    fact, an arbitrary search query), in which case only rows
-   satsifying the query will be considered.
+   satisfying the query will be considered.
 
    For example, consider the following data.
    ```
@@ -606,9 +631,9 @@ Statistics
    ```
 
    If you added stats for column `A`, it would record that there are
-   four instances of 2, three of 1, two of 5 and one each of 3, 4, and
-   8.  It would also record the minimum value (1), the maximum value
-   (6), the average (3), and the total (12 rows).
+   four instances of 2, three of 1, two of 5 and one each of 3, 4,
+   and 8.  It would also record the minimum value (1), the maximum value (6),
+   the average (3), and the total (12 rows).
 
    If you specified a threshold of 3, it would only record that there
    are four instances of 2 and three of 1.  Now the minimum and
@@ -621,8 +646,8 @@ Statistics
    statistics.
 
    If you want to group values into buckets (for example, class
-   numbers of number fields split into ranges like 1 < h <= 10 and 10
-   < h <= 100 and 100 < h <= 1000), you can use the
+   numbers of number fields split into ranges like `1 < h <= 10` and `10
+   < h <= 100` and `100 < h <= 1000`), you can use the
    `add_bucketed_counts` method.
 
    If you want to add counts for many sets of columns (in order to
@@ -634,11 +659,11 @@ Statistics
    Create a statistics object inheriting from `StatsDisplay` in
    `lmfdb/display_stats.py`.  It should have attributes
 
-   - `short_summary` (which can be displayed at the top of your browse page), 
+   - `short_summary` (which can be displayed at the top of your browse page),
    - `summary` (which will be displayed at the top of the statistics page),
    - `table` (the postgres table on which statistics are computed),
    - `baseurl_func` (the function giving your browse page, e.g. `'.index'`),
-   - `stat_list` (a list of dictionaries giving the statistics to be displayed; 
+   - `stat_list` (a list of dictionaries giving the statistics to be displayed;
    - `'cols'`, `'row_title'` and `'knowl'` are required arguments,
    - and other optional arguments allow you to adjust the default behavior)
 
@@ -647,7 +672,7 @@ Statistics
    collect the relevant statistics.  You should also create a view
    using the `display_stats.html` template, passing your object in as
    the `info` parameter.  Note that `DisplayStats` inherits from
-   Sage's `UniqueRepresentation, so it will only be created once.
+   Sage's `UniqueRepresentation`, so it will only be created once.
 
 1. How do I display statistics from multiple tables on one page?
 
@@ -682,18 +707,15 @@ Data Validation
 1. How can I add consistency checks for data in the LMFDB?
 
    One option is to add constraints to your table.  To do so, use the
-   `create_constraint` method in `database.py`.  You can see the
-   current constraints using `list_constraints`.
+   `create_constraint` method in `psycodict/table.py`. You can see the current
+   constraints using `list_constraints`.
 
    There are three supported types of constraints.  The simplest is
    `not null`, which checks that a specified column is filled in for
    every row in the table.  The second is `unique`, which checks that
    a column or set of columns is unique across all rows of a table.
    The final options is `check`, which runs an arbitrary SQL function
-   on a set of rows.  We are building up a library of SQL functions
-   for use in this way; you can see them in `backend/utils.psql`.
-   Once written, these functions need to be added to postgres, at
-   which point they can be used in checks.
+   on a set of rows.
 
    Note that constraints are checked whenever a row is added or
    updated, so if they are complicated it will impose a speed penalty
@@ -714,7 +736,7 @@ Data Validation
    utilities for writing such queries, such as `check_values`,
    `check_iff`, `check_count`.  You can also write the query directly
    and use `_run_query`.  If you want to run queries that check
-   consistency accross multiple tables, see the `check_crosstable`
+   consistency across multiple tables, see the `check_crosstable`
    utility functions.  For fast queries you can use the `@overall`
    decorator; if your query takes longer than about a minute, you may
    want to use the `@overall_long` decorator instead.

lmfdb/classical_modular_forms/main.py

@@ -182,8 +182,6 @@ def index():
             return dimension_space_search(info)
         elif search_type == 'Traces':
             return trace_search(info)
-        elif search_type == 'SpaceTraces':
-            return space_trace_search(info)
         else:
             flash_error("Invalid search type; if you did not enter it in the URL please report")
     info["stats"] = CMF_stats()
@@ -486,9 +484,9 @@ def mf_data(label):
         ocode = db.mf_newspaces.lookup(label, "hecke_orbit_code")
         if ocode is None:
             return abort(404, f"{label} not in database")
-        tables = ["mf_newspaces", "mf_subspaces", "mf_newspace_portraits", "mf_hecke_newspace_traces"]
-        labels = [label, label, label, ocode]
-        label_cols = ["label", "label", "label", "hecke_orbit_code"]
+        tables = ["mf_newspaces", "mf_newspace_portraits"]
+        labels = [label, label]
+        label_cols = ["label", "label"]
         title = f"Newspace data - {label}"
     elif len(slabel) == 2:
         tables = ["mf_gamma1", "mf_gamma1_portraits"]
@@ -891,15 +889,15 @@ def newform_search(info, query):
     newform_parse(info, query)
     set_info_funcs(info)
 
-def trace_postprocess(res, info, query, spaces=False):
+def trace_postprocess(res, info, query):
     if res:
         if info.get('view_modp') == 'reductions':
             q = int(info['an_modulo'])
         else:
             q = None
         hecke_codes = [mf['hecke_orbit_code'] for mf in res]
         trace_dict = defaultdict(dict)
-        table = db.mf_hecke_newspace_traces if spaces else db.mf_hecke_traces
+        table = db.mf_hecke_traces
         for rec in table.search({'n':{'$in': info['Tr_n']}, 'hecke_orbit_code':{'$in':hecke_codes}}, projection=['hecke_orbit_code', 'n', 'trace_an'], sort=[]):
             if q:
                 trace_dict[rec['hecke_orbit_code']][rec['n']] = (rec['trace_an'] % q)
@@ -962,22 +960,6 @@ def trace_search(info, query):
     process_an_constraints(info, query)
     set_info_funcs(info)
 
-@search_wrap(template="cmf_space_trace_search_results.html",
-             table=db.mf_newspaces,
-             title='Newspace search results',
-             err_title='Newspace search input error',
-             shortcuts={'jump':jump_box,
-                        'download':CMF_download().download_multiple_space_traces},
-             projection=['label', 'dim', 'hecke_orbit_code', 'weight'],
-             postprocess=space_trace_postprocess,
-             bread=get_search_bread,
-             learnmore=learnmore_list)
-def space_trace_search(info, query):
-    set_Trn(info, query)
-    newspace_parse(info, query)
-    process_an_constraints(info, query)
-    set_info_funcs(info)
-
 def set_rows_cols(info, query):
     """
     Sets weight_list and level_list, which are the row and column headers
@@ -1454,8 +1436,7 @@ class CMFSearchArray(SearchArray):
     _sort_forms = [(name, disp, sord + ['hecke_orbit']) for (name, disp, sord) in _sort]
     sorts = {'': _sort_forms,
              'Traces': _sort_forms,
-             'Spaces': _sort_spaces,
-             'SpaceTraces': _sort_spaces}
+             'Spaces': _sort_spaces}
     jump_example="3.6.a.a"
     jump_egspan="e.g. 3.6.a.a, 55.3.d or 20.5"
     jump_knowl="cmf.search_input"
@@ -1746,7 +1727,7 @@ def main_array(self, info):
         if info is None:
             return self.browse_array
         search_type = info.get('search_type', info.get('hst', ''))
-        if search_type in ['Spaces', 'SpaceTraces']:
+        if search_type == 'Spaces':
             return self.space_array
         elif search_type == 'SpaceDimensions':
             return self.sd_array
@@ -1761,7 +1742,6 @@ def search_types(self, info):
                  ('Random', 'Random form')]
         spaces = [('Spaces', 'List of spaces'),
                   ('SpaceDimensions', 'Dimension table'),
-                  ('SpaceTraces', 'Traces table'),
                   ('RandomSpace', 'Random')]
         if info is None:
             return basic
@@ -1788,7 +1768,7 @@ def html(self, info=None):
         # We need to override html to add the trace inputs
         layout = [self.hidden_inputs(info), self.main_table(info), self.buttons(info)]
         st = self._st(info)
-        if st in ["Traces", "SpaceTraces"]:
+        if st in ["Traces"]:
             trace_table = self._print_table(self.traces_array, info, layout_type="box")
             layout.append(trace_table)
         return "\n".join(layout)

lmfdb/classical_modular_forms/test_cmf.py

@@ -554,9 +554,7 @@ def test_underlying_data(self):
 
         data = self.tc.get('/ModularForm/GL2/Q/holomorphic/data/13.2.e').get_data(as_text=True)
         assert ('mf_newspaces' in data and 'num_forms' in data
-                and 'mf_subspaces' in data and 'sub_mult' in data
-                and 'mf_newspace_portraits' in data and "data:image/png;base64" in data
-                and 'mf_hecke_newspace_traces' in data and 'trace_an' in data)
+                and 'mf_newspace_portraits' in data and "data:image/png;base64" in data)
 
         data = self.tc.get('/ModularForm/GL2/Q/holomorphic/data/13.2.e.a').get_data(as_text=True)
         assert ('mf_newforms' in data and 'field_disc_factorization' in data and

lmfdb/classical_modular_forms/test_cmf2.py

@@ -132,11 +132,6 @@ def test_download_search(self):
         for elt in ["20.5.b", "20.5.d", "20.5.f"]:
             assert elt in page.get_data(as_text=True)
 
-        page = self.tc.get('/ModularForm/GL2/Q/holomorphic/?Submit=sage&download=1&query=%7B%27dim%27%3A+%7B%27%24gte%27%3A+2000%7D%2C+%27num_forms%27%3A+%7B%27%24exists%27%3A+True%7D%7D&search_type=SpaceTraces', follow_redirects=True)
-        assert 'Error: We limit downloads of traces to' in page.get_data(as_text=True)
-        page = self.tc.get('/ModularForm/GL2/Q/holomorphic/?Submit=sage&download=1&query=%7B%27dim%27%3A+%7B%27%24gte%27%3A+30000%2C%27%24lte%27%3A30600%7D%2C%27num_forms%27%3A+%7B%27%24exists%27%3A+True%7D%7D&search_type=SpaceTraces', follow_redirects=True)
-        assert '863.2.c' in page.get_data(as_text=True)
-
     def test_random(self):
         r"""
         Test that we don't hit any error on a random newform

lmfdb/local_fields/main.py

@@ -587,15 +587,15 @@ def reliability():
 
 class LFSearchArray(SearchArray):
     noun = "field"
-    sorts = [("", "prime", ['p', 'n', 'c', 'label']),
-             ("n", "degree", ['n', 'p', 'c', 'label']),
-             ("c", "discriminant exponent", ['c', 'p', 'n', 'label']),
-             ("e", "ramification index", ['e', 'n', 'p', 'c', 'label']),
-             ("f", "residue degree", ['f', 'n', 'p', 'c', 'label']),
-             ("gal", "Galois group", ['n', 'galT', 'p', 'c', 'label']),
-             ("u", "Galois unramified degree", ['u', 'n', 'p', 'c', 'label']),
-             ("t", "Galois tame degree", ['t', 'n', 'p', 'c', 'label']),
-             ("s", "top slope", ['top_slope', 'p', 'n', 'c', 'label'])]
+    sorts = [("", "prime", ['p', 'n', 'c', 'num']),
+             ("n", "degree", ['n', 'p', 'c', 'num']),
+             ("c", "discriminant exponent", ['c', 'p', 'n', 'num']),
+             ("e", "ramification index", ['e', 'n', 'p', 'c', 'num']),
+             ("f", "residue degree", ['f', 'n', 'p', 'c', 'num']),
+             ("gal", "Galois group", ['n', 'galT', 'p', 'c', 'num']),
+             ("u", "Galois unramified degree", ['u', 'n', 'p', 'c', 'num']),
+             ("t", "Galois tame degree", ['t', 'n', 'p', 'c', 'num']),
+             ("s", "top slope", ['top_slope', 'p', 'n', 'c', 'num'])]
     jump_example = "2.4.6.7"
     jump_egspan = "e.g. 2.4.6.7"
     jump_knowl = "lf.search_input"

lmfdb/modular_curves/main.py

@@ -962,11 +962,12 @@ def __init__(self):
             [CPlabel],
         ]
 
+    _default = ["level", "index", "genus", "coarse_class_num", "coarse_level", "coarse_num", "fine_num"]
     sorts = [
-        ("", "level", ["level", "index", "genus", "label"]),
-        ("index", "index", ["index", "level", "genus", "label"]),
-        ("genus", "genus", ["genus", "level", "index", "label"]),
-        ("rank", "rank", ["rank", "genus", "level", "index", "label"]),
+        ("", "level", _default),
+        ("index", "index", ["index", "level"] + _default[2:]),
+        ("genus", "genus", ["genus"] + _default[:2] + _default[3:]),
+        ("rank", "rank", ["rank", "genus"] + _default[:2] + _default[3:]),
     ]
     null_column_explanations = {
         'simple': False,