Django Integration: Documentation and URL handling

[RyanRio]

Created from discussion around this discourse topic.

The main feature I am interested in is properly using autoload from bokeh.server.django to access a bokeh app without mangling the url in the request function. To demonstrate this, consider the django project named project and just using views/urls within said project directory (this could easily be extended to be contained within an app). Currently the following has to be done in order to support the generic 'title':

# in urls.py within the project directory
bokeh_apps = [
    autoload("home/x", views.app_handler),
]

urlpatterns = [
    path('admin/', admin.site.urls),
    path("home/<str:title>", views.app),
]

# in views

def app(request: HttpRequest, title) -> HttpResponse:
    script = server_document(request.build_absolute_uri("/home/x"), arguments={"title": title}) # breaks the django url
    return render(request, "embed.html", dict(script=script))

def app_handler(doc: Document) -> None:
    df = sea_surface_temperature.copy()
    source = ColumnDataSource(data=df)
    plot = figure(x_axis_type="datetime", y_range=(0, 25), y_axis_label="Temperature (Celsius)",
                  title="Sea Surface Temperature at 43.18, -70.43")
    plot.line("time", "temperature", source=source)

    doc.add_root(column(slider, plot))
   # taken from examples and simplified

# and then in routing.py
bokeh_app_config = apps.get_app_config('bokeh.server.django')

application = ProtocolTypeRouter({
    'websocket': AuthMiddlewareStack(URLRouter(bokeh_app_config.routes.get_websocket_urlpatterns())),
    'http': AuthMiddlewareStack(URLRouter(bokeh_app_config.routes.get_http_urlpatterns())),
})

Ideally, you could indicate to autoload that it will take a parameter and it would make that parameter available to that handler. Note that it is already technically available to the handler under doc.session_context.arguments, however I still had to explicitly put "/home/x"

A few issues from this example:

1. In order to set this up you explicitly define the url that bokeh will set up it's WSConsumer and AutoloadJSConsumer, in this case "home/x/ws" and "home/x/'autoloadjsurl'".
2. It's largely a pain as far as I can tell to actually do anything with the WSConsumer, for instance you can technically inherit WSConsumer like so:

class Wrapper(WSConsumer):

def __init__(self, scope: Dict[str, Any]):
    super().__init__(scope)

async def connect(self):
    await super().connect()
    print("connecting") # add your own code wherever you want here, and just forward along bokeh information

async def disconnect(self, close_code):
    await super().disconnect(close_code)

async def receive(self, text_data):
    await super().receive(text_data)

and substitute it into django/routing.py in add_new_routing(...):
self._websocket_urlpatterns.append(url(urlpattern("/ws"), Wrapper, kwargs=kwargs))
3. Documentation is lacking all around here, for instance what is the difference/the use cases between DocConsumer (document()) and AutoloadJsConsumer (autoload)

If a full project directory is required I will happily provide, but I think that this is mainly a documentation issue - I am happy to contribute!

This is on bokeh version 1.4.0, I looked at the recent commit bokeh/bokeh#9536 but that seemed to have just added token support so what I have put here should be up to date, at least concerning what I am discussing here!

[bryevdv]

@mattpap @philippjfr I asked for the issue to be made because I'm not sure it points to a need for docs/examples, or if there is some useful feature work that could be prompted

[RyanRio]

I'm actively using these feature(s) so I figure I will update this comment as I go with things I notice / functionality improvements.

Logging

• Bokeh errors just disappear into the void -> if an error occurs during loading of a bokeh app session it immediately exits out to function.py and then application.py in bokeh.application, eventually bubbling all the way out bokeh.server.contexts create_session_if_needed. Errors do not appear in the browser either. For context I did set both log_levels to trace on the bokeh.settings instance
• errors and standard output that occur on the JS side are logged though, so the only thing not being logged is if an error occurs in python code in the django app

[yanwencheng]

When you request the app interface, you can carry the data and modify your code

def app(request: HttpRequest) -> HttpResponse:
    script = server_document(request.build_absolute_uri("/home/x"), arguments=dict(request.GET.dict(), **request.POST.dict()) 
    return render(request, "embed.html", dict(script=script))

and in app app_handler func

args = doc.session_context.request.arguments
print(args.get("wriId"))

But I had other problems
https://discourse.bokeh.org/t/about-server-document/4783

In the HTML file of another server, AJAX carries the parameter request interface to get the returned script and execute it..There is no problem with the first access, but there is no response after the second access only modifies the parameters

[RyanRio]

Hey @yanwencheng , you'll have to be a little more descriptive - does "wriId" exist in **request.POST.dict() (or GET), otherwise I don't see how you are fetching a valid value, which would cause it to silently fail on the KeyError as I mentioned. Otherwise it looks fine and if you are saying the graph appears on the first request then I think you should definitely investigate what the value of wriID is when the graph doesn't render.

[yanwencheng]

嘿@yanwencheng，您必须要更具描述性-** request.POST.dict（）（或GET）中是否存在“ wriId”，否则我看不到您如何获取有效值，会导致它在我提到的KeyError上无提示地失败。否则，它看起来还不错，并且如果您说图表出现在第一个请求上，那么我认为您应该研究当图表未呈现时wriID的值是什么。

Hey @yanwencheng , you'll have to be a little more descriptive - does "wriId" exist in **request.POST.dict() (or GET), otherwise I don't see how you are fetching a valid value, which would cause it to silently fail on the KeyError as I mentioned. Otherwise it looks fine and if you are saying the graph appears on the first request then I think you should definitely investigate what the value of wriID is when the graph doesn't render.

Combined POST and GET request parameters，Use args = doc.session_context.request.arguments when generating bokeh documents

[RyanRio]

@bryevdv Am I correct in that bokeh/bokeh#9536 would allow me to access other arguments on the request object and enable me to do (I'll try to keep it as purely-bokeh-related as possible, treat request.session as just a django session that should persist across a single user instance):
doc.session_context.request.session["bokeh-id"] = doc.session_context.id and then later on (for instance a user does something on the page and I handle the POST request by modifying the plot a little bit) I could access that with `request.session["bokeh-id"]?

From what I can tell, and this includes separate from django, an autoloadjsConsumer (or document - though I'm not all that sure what this does) that extends SessionConsumer makes the call to create_session_if_needed. After the bokeh document is created a request to the ws url is created which has reference to the application context and attempts to send the message to the client. Currently, I am attempting to use a workaround and in the connect method of the WSConsumer I have put:
self.request['session']['bokeh-id'] = self.get_argument("bokeh-session-id")
However, as you can imagine, this request does not exactly persist throughout, the multiple states of the session variable are
<django.contrib.sessions.backends.db.SessionStore object at 0x1AC62BE0> autoload <django.contrib.sessions.backends.db.SessionStore object at 0x1ABDED60> ws <django.contrib.sessions.backends.db.SessionStore object at 0x1ABD9CE8> ajax request
So, my data is lost (whether this is due to django user authentication being out of sync or due to the issues resolved by 9536, I don't know)

In any case, this prompts a possible feature - user synchronization between django and bokeh. It would be nice, if bokeh is living in a django environment, to optionally take advantage of the powerful django session, auth (might solve some problems mentioned in bokeh/bokeh#9536?), and user modules, it's already confusing enough that session can refer to both! Feel free to lay on any server architecture knowledge you feel is relevant/interesting, or just point to src code, I'll happily do the digging and experimenting myself too!

[PeteFein]

I took a stab at implementing something like this before finding this issue - see bokeh/bokeh#9946 & bokeh/bokeh#9947

[sdc50]

@RyanRio

Take a look at PR #6 and let me know if you have any feedback.

Here are the key points from the expanded examples in the PR:

urls.py

urlpatterns = [
    path("shapes/<str:arg1>/<str:arg2>", views.shapes_with_args),
]

bokeh_apps = [
    autoload("shapes/(?P<arg1>[\w_\-]+)/(?P<arg2>[\w_\-]+)", views.shape_viewer_handler_with_args),
]

Note that autoload takes a re_path pattern.

views.py

from bokeh_django import with_url_args

@with_url_args
def shape_viewer_handler_with_args(doc, arg1, arg2):
    viewer = shape_viewer()
    pn.Column(
        viewer,
        pn.pane.Markdown(f'## This app has URL Args: {arg1} and {arg2}')
    ).server_doc(doc)

def shapes_with_args(request: HttpRequest, arg1: str, arg2: str) -> HttpResponse:
    script = server_document(request.get_full_path())
    return render(request, "embed.html", dict(script=script))

Note that server_document should be called with request.get_full_path().