DISCUSSION: Editor plugin issues.

[MaikuMori]

I was thinking about the daemon mode and reading older issues and I came to conclusion that a lot of the problems can be fixed by focusing on certain src problems.

To give some perspective common editor features are:

• Jump to definition
• Documentation/type information
• Global usage/examples

Potential features:

• Refactoring help
• Local usages
• Language specific features (subclasses, superclasses)
• Auto complete
• Code suggestions
• [insert any cool IDE feature]

Key issues that should be addressed:

• Speed

  Being smart about what needs to be updated, quicker scans and dependency resolves. Queries against already processed code should be instant (unless they're complex). Minimize redownloading, calling slow external online APIs.

• Asynchronous calls

  While src is graphing a file(s) it should be perfectly fine to be able to query a file which hasn't changed while conflicting calls should block until they can proceed or are cancelled by caller.

• Graceful degradation

  Just because file can't be compiled doesn't mean that older version of file can't be used to get 90% def/refs. From plugin standpoint you don't care so much about commits but more about different file versions between saves.

• Logic in srclib

  To prevent writing the same code over and over again for each plugin as much logic (where it makes sense) should be put in srclib or some tool that can be reused by all plugins.

Related issues:

• Speed up access to the srclib-cache data Speed up access to the srclib-cache data #41
• Optimize analysis for diffs rather than full new commit heads Optimize analysis for diffs rather than full new commit heads #65
• Support caching build/dep directories Support caching build/dep directories #93
• src command hanging with 100% cpu usage src command hanging with 100% cpu usage #143

As for daemon mode, it has some advantages such as being able to reverse the query/pull logic and instead push changes and new information as soon as they happen on filesystem. As you can see some plugin features could benefit from more real-time communication (auto complete for example). One way do somehow circumvent that is to allow plugins to pass current buffer and make srclib pretend that it's the actual file content.

@nicot maybe you can describe what issues you currently have with vim plugin. Some goes for every other editor plugin developer. If we pool all the issues we can better understand what to focus on next and how to fix them.

[nicot]

Thanks for the writeup Miks, it looks great.

How can queries to src conflict?

My biggest issue with the plugin development is the length of time that a simple srclib api describe command takes. For even a small repository, it takes a few seconds. Ideally, all calls would take less than a quarter second. It is often the case that I don't need all the information that srclib describe provides, so maybe we can split calls to be more efficient.

I definitely think that daemon mode would be a huge help for editor plugins, but it would also add significant complexity to the project. Will src only have one running instance per OS user? Or will each project have it's own instance.

[MaikuMori]

How can queries to src conflict?

Say you ask something about a file which triggers graphing of said file and then you ask something about the file again. Right now it would trigger 2 parallel graphing processes which would overwrite each other.

I definitely think that daemon mode would be a huge help for editor plugins

Can you be more specific? That's what I thought and then I sat down to write this issue and couldn't come up with many good arguments for daemon mode. At least not as first step and especially considering added complexity. I ended up rewriting this issue couple times till I got to this version.

I think working out the issues that I mentioned will help with the speed. Also improving speed at toolchain level. Eventually we could reconsider daemon mode. Starting with one instance per project is easy and moving to one instance per user/system is not too big of a change.

[beyang]

I agree about not having a daemon mode unless there's a compelling reason. I think there's a lot of optimization we can do currently on how srclib reads data (e.g., via indexes or splitting up commands to only query the data we need in a given operation).

To me, the issues @MaikuMori outlines fall into 2 categories. The first are just optimizations to srclib core (storing index files or figuring out how to read in data more efficiently, or perhaps storing data on disk in a more efficient format). We can identify bottlenecks by using the Go profiler here. The second category are changes to the interface between srclib core and the toolchains (e.g., making toolchains support auto-complete or incremental processing). These changes seem more involved and will have broad implications about how future toolchains are implemented, and so it would be helpful to write up a spec for each of them to make sure we've considered the implications. I think the first group is the more low-hanging fruit here. @MaikuMori what do you think about using the Go profiler to identify bottlenecks in srclib core? @nicot and @jelmerdereus can provide additional comments about which operations in the editor plugins / src api are too slow right now.

[nicot]

Can you be more specific? That's what I thought and then I sat down to write this issue and couldn't come up with many good arguments for daemon mode. At least not as first step and especially considering added complexity. I ended up rewriting this issue couple times till I got to this version.

Oops, I misunderstood you. I thought that you were arguing that speed, asynchronous calls and and graceful degradation were features that could be incorporated into the daemon. I agree that daemon mode is probably not worth the costs at this point.

I don't know enough about the toolchains and srclib itself to know how we would address the key issues you described above.

I haven't researched which parts of srclib take a long time, but I agree that would be a good place to start. Perhaps we could also make changes to the analysis toolchains to improve speed without changing the interface.