Captions for Joseph's "New features in Racer and ShareJS" video

A captioned version of the video is at: http://www.youtube.com/watch?v=cCbu0iWLK74

(And if anyone at LeverApp YouTube channel wants to add this transcript to the video on the channel, that would be a Very Good Thing. YouTube Auto-timing works pretty well. YouTube Auto-captioning is funny, but not really useful.

Here's the transcript:

Hello, I'm Joseph. I wrote ShareJS, which is a little OT-based concurrent editing library.

ShareJS, for the people who haven't seen it, lets you be able to do real-time editing of arbitrary documents, which is cool.

And at the moment, I'm working here at a little company called Lever to integrate ShareJS into Racer.

We're in the process at the moment of rebuilding Racer and rebuilding ShareJS, to make sure that [?] as part of Derby JS, to be able to support full rich texts, full concurrent editing on everything,

and the final solution is going to be really really cool.

But I want to show you some of the new features that are coming in both ShareJS and Racer in the next version.

So first of all I'm going to show you - so, ShareJS up until now has been limited to only running a single process and if you want to actually build a full-featured web application on it,

it's sort of hard because you can't really scale ShareJS past a single machine.

So the first change that we're doing to ShareJS is making it work like a process.

So I'm gonna run... - so, this is just using a prototype version of ShareJS that's currently sitting in the rewrite branch.

I'm gonna run… run on my server and this is just a really really simple ShareJS server which lets me - which gives me a simple concurrent text editing window.

I can run this in multiple windows, and I can concurrently edit text. Which is nothing people who have been using ShareJS haven't seen before.

The cool thing though is that if I want to, I can pull up another tab and run another instance of ShareJS,

and this time I'm going to run it at a particular port, and now I've got two different ShareJS servers running at the same time.

And what I'm gonna do is I'm going to connect one of these windows to one server and the other one to the other server,

so this is kind of how we would run if you had a multiple set of front-end servers, and a single set of databases.

So we see exactly the same behavior, which is to say that if two different clients are connected to different front-end servers,

then they're both gonna be able to concurrently edit documents, which is a really big deal,

because it means that now you can scale up ShareJS-based deployments to your whole data center, or whatever. You can build much bigger applications built on it.

So that's the biggest change, and because it's so cool I'm also going to show you that in Racer... and so if I pull up Racer, this is...

I'm just in the new version of Racer, which is in Racer 0.5 branch in the examples/pad directory. So this is a just a really really simple text editor built with Racer.

So Racer - for people who have used Derby - Racer is the data model that Derby is all built on top of. So this new version of Racer is gonna underpin the new version of Derby,

which is mostly going to stay the same - it's gonna be very unsightly.

And Racer is what gives Derby the rich model and does all the bundling. So if I run up the Racer server - 'node server.js' -

and I connect to it, then I see pretty much the same thing as what I can see in ShareJS,

which is to say that I can be concurrently editing in Racer.

Now this is kind of cool because it means that with Racer and with Racer and Derby,

any applications that you build automatically get full operational-transform-based editing across the entire application.

So if one user on one page starts editing one document, then another user on another page can see all of those edits live, and they can both collaborate on edits,

and all that is done with the operational transform, which means that it's all provably - should be provably - correct,

and result in something sensible coming up on the other end, despite network latency and everything else, and it does it by just merely sending little operations.

Anyway so that's cool, but again, I can run a Racer server on another machine... hmm? Oh, the default port is 3000. Cool. I'm gonna run another Racer server on port 3010.

Then my original Racer server back up again. So… These two windows now I can put 3010 on one of them and after it connects…

I can concurrently edit the document from both windows. It took a second to connect there just simply because the first run of Racer, it needs to do a little bit of work.

But one of the really cool things about Racer that people who use ShareJS might be interested in is that Racer actually will bundle this entire page up in the initial load.

So that if I go back to my ShareJS windows - and this is really hard to see because I'm on local - but if I refresh, you'll see that little 'connecting… ' before you actually see the text

and if I try to actually look at what's going on behind the scenes -

so I'm going to pull up the console here and run curl on this URL - what's happening is that this text area is being filled in with 'connecting... '

and then only after JavaScript loads does that actually replace with the actual document contents.

So I mean that's part of ShareJS because ShareJS is a client-based library. But with Racer - and this is really really cool –

I can do the same test and I can call 'localhost:3010/home' and this is the page sent by Racer, and we'll see here that this document that I'm editing -

I see the entire contents of the document so if I refresh here, there's no page load, there's no '… ' that's happening.

The entire page is loading immediately, and the HTML that's served from the server contains all the content prerendered, which means that you get these immensely ridiculously fast page loads, which is awesome.

So that's the big feature - the big new feature of ShareJS and Racer is the fact that it can scale, and you can see how using Racer can make your applications really cool.

The other things I want to show - I'll briefly show fetch mode.

So fetch mode is a new mode that you can do in ShareJS for editing - and of course in Racer as well - where when you're editing a document,

if you don't want actually to have a live connection to the server,

because there's a bit of overhead or whatever,

you can actually just let the document sit unsubscribed, and you'll get the contents of the document just normally,

but you won't get any changes that other people make until either you manually call fetch from your application, or until the user starts making changes.

When that happens, the client will send the operations and the changes that you make to the server, and the server can reply with all the things that have happened since it last got an operation from you.

So in this case I'm going to unsubscribe my document and then open this up again on another server -

or on the same server, it doesn't really matter.

So in this case now because this document in this window is unsubscribed, if I make any changes here...

Well, if I make any changes here, that first window can't see the changes.

So, yeah, but if I make any changes from here, as I promised, what happens is that when I make the change from here, it's gonna send the operation to the server,

and then all the clients are gonna receive that operation. And when the server replies to tell you your operations are accepted, it will also send you all the intervening operations.

So I just hit a space bar and all of a sudden I get all the operations I didn't see before.

And also the other window sees all of my edits immediately because the other one is subscribed. So this is really good in situations where you don't want to have a heavyweight long-running get connection or pending get or web socket to the server -

you can just have little Ajax requests, where in an Ajax request you can send the operations that you want, and the server can reply back, and you still manage to stay in sync. With the document on the server.

Which is cool. So that's fetch mode.

I'll also show you Racer bundling content - actually, I've already shown you Racer bundling content.

Another new feature in ShareJS... some bouncing back-and-forth here...

all these features are available in both Racer...

is available in Racer and ShareJS, which are now slowly but surely getting more linked as we add more features that Racer needs from ShareJS.

But ShareJS is still entirely its own library, and you can use it without using Racer or Derby.

But in ShareJS, I can load up this multi-editor.

For long-time users of ShareJS, this is something that's never been possible before.

ShareJS was broken, and I keep getting issues about this, but in this case now I can have multiple text areas that are both editing the same document and this behaves correctly

which is to say that each operation that either editor makes is only sent to the server once.

It doesn't get sent twice, it doesn't do any internal routing, it just replays on the other documents and any other document obviously can still edit and synchronously edit the same document that I'm seeing.

Anyway, so if that was causing you grief, then that's now solved, which is really nice - I'm happy with [it].

The other huge feature that's been added to ShareJS - and this is something that Racer has needed - is queries.

So this is kind of really awesome, and I'm super impressed with this.

So classically if you had... with ShareJS, of course, you can have an arbitrary JSON structure stored inside...

you can have an arbitrary JSON structure stored in the database and you can do real-time concurrent edits using operational transform to edit your arbitrary JSON,

which is neat, except that also sometimes you want to be able to do queries on that, and then... what does that really return,

and it's all a bit weird when you start interacting with regular database operations.

So with the new version of ShareJS, I've added the ability... [aside:second ... that one]

the ability to send queries directly through ShareJS to the server.

So, we're going to have a bunch of access control rules, so you can control who uses...

not just submit arbitrary queries to the database.

But - what this lets us do is to be able to see all the documents that exist on the server and make applications that can view them.

So this document here on the right is actually showing the 'seph' document, so I can pull up this document, and this is just a really simple query database,

and it's telling me that document has typed text and I can edit this document here and I'll see all the changes anywhere else that I'm viewing this document.

That's kinda cute, but also this is gonna show me JSON documents. So if I bring up, say, the Internet document here,

I can see that x is five. So I'm gonna open this same query up in two windows - and actually for fun I'm gonna bring in two windows from two different front-end servers -

and in these two front-end servers I can see this JSON object, and this is using the wonderful JSON editor 'project'.

I'm currently cheating a little bit to be able to use it concurrently, but that's just a client software problem - I'm not using all the right APIs.

Anyway from these two windows, if I want to - I can of course edit this, I can add some more keys... string value hi to 47, which is my favorite number today,

and of course I can see that they're being updated in real-time from the other window.

So anyway, that's kinda cool, but the really nice thing about the query interface is that it lets me do things like this:

so let's say I want to view all documents that have an x value of three. So I'm gonna say...

but I'm using Mongo at the moment which means that the query language is Mongo,

but if I were to use a SQL backend, then the query language would be SQL, and this would all continue to work - should continue to work.

Currently I'm just using Mongo and Redis. So I want to say that data.x is query, and this is going to now query all documents that have data.x equal to three. Which is currently only this 'internet' document.

If over here I change this 'foo' document, which currently has x [as] five, to have x of three, suddenly the 'foo' document appears on the left, in this window.

And of course, if it was four, it's not there. It's three - it reappears. Another document, its x be three and it'll appear as well.

So this is really really powerful because it means that you can have queries on your web application which sit live,

so you can say 'please server show me any document that has age greater than 50', '

show me any user whose name starts with an a', or

'show me all users whose reputation [?] is more than 1000', and this list will actually be updated in real-time.

And, of course, because this is just Redis queries, we can also do limits and skips and offsets and all that kind of thing -

and sorts, which go straight to the database, So if I said... [aside - what's another fun example?] -

I want to show all documents and sort them by the data.x property. Then I do that, and this is gonna show me all documents - it's just sorting by the data.x - documents that don't have the data.x property

I'm just seeing them up first, but after that I have my 'foo', 'foo2', to 'internet', who all have an x value of three.

Now, if over here I change, say, 'foo2' to have an x value of 10,

then 'foo2' is gonna go right down to the bottom because it's got the highest x.

And if I make it 1, then 'foo2' will be at the top.

And if it doesn't have an x value at all... it's a string...
apparently strings are sorted after numbers by Redis.

Anyway, so that's pretty cool. And - yeah - the other fun thing, of course, is that because Racer and ShareJS are both hooked up to the same backend,

now it's possible to be able to see and use applications like this that use...

ShareJS-based applications with applications in Redis - the backend is exactly the same -

well, the backend - the server backend - the thing that backs all the data -

is the same, which means that I can also share all documents which are in my Redis collection. So, I'm going to pull this up...

so I've got this 'home' document which has some junky text in it, and this 'home' document in Racer,

because Racer is based on the JSON OT code, that's also in a JSON object.

So this JSON object currently just has a big string and, of course, I can change that string directly from this query here, or I can change it from here, you can see it get updated live.

So, that's pretty neat. But using queries, of course, if...

Let's say I had string-based documents - I can also supply queries to tell me which documents [unint].

So I'm going to use a query... So mongoose supports regular expressions-based queries so this query says -

it's a bit hard to see - this regular expression is anything that has a 'hi' followed by any characters, and then an 'x'.

So currently there's no results that have 'hi.*x', but if I change this text to be 'hi there x' then suddenly my result appears in the query because it now matches.

If I remove the 'x', it disappears, and I can play like that for a while.

Of course, because this is a Racer-based document, this is hooked up using routes, so I'm seeing the 'home' document [?] local/home endpoint,

but if I went local/host/super, now I'm editing the 'super' document inside of this collection that we're looking at,

and again if this was 'hi internet x', then suddenly the 'super' document would appear and match my query.

Anyway, that's all I want to show you, so that's all the new features that are coming in Racer and ShareJS.

I'm really looking forward to landing all of this stuff, but apparently it's really buggy,

so we need to write a bunch of tests,

and the backend needs to be cleaned up a little bit,

and I need to harden a bunch of code

before I'm happy to use this,

but if you want to play with it you can already play with it by either looking at the rewrite branch on ShareJS,

or by looking at Racer and looking at the 0.5 branch.

Let me just pull that up...

So we're making like a billion changes a day to this. The APIs will not be stable, but, as I say, this is all coming and I'm kind of excited.

Yeah, it's gonna be really fun and I look forward to start building applications based on all this new technology.

So, thanks - feel free to e-mail me or anything else if you have any questions.