Requirements and development ideas

This page is intended as a place to collect ideas and statements on how it would be possible to create a transformation of LilyPond documents to MusicXML.

As a starting point there are statements and comments from various threads on the lilypond-user list.

---

UL:
AFAIR the result of earlier threads I was involved in was something like: If we make do with exporting the plain content structure (i.e. without LilyPond's layout decisions) it should be 'enough' to catch the music output stream (although I don't fully understand what this is yet) and transform it to a MusicXML file. Is that true, halfways true or wrong?

DK:
Likely at best halfways. For example, no header is contained in the hypothetical "music output stream". Things like \transpose don't make it into the "music output stream" but rather just its results.
Which may be enough for some workflows, but not necessary all. It does provide more information than the Midi.

Paul Morris:
Is the music stream what you see when you use \displayMusic as described here? http://lilypond.org/doc/v2.16/Documentation/extending/displaying-music-expressions

David Kastrup:
No. That's even before scorifying music (scorification caters for things like repeat chords, << ... \ ... >> and a few other things).

Chris Maden:
Can someone please summarize (or point me to a summary of) the current state of MusicXML support, what needs to be done, and the infrastructure? E.g., is a standalone XSLT conversion acceptable, or should MusicXML import/export be integrated into the core LilyPond code?

Paul Morris:
I don't know enough to give an adequate summary, but here are some starting points.
Import documentation: http://www.lilypond.org/doc/v2.16/Documentation/usage/invoking-musicxml2ly
Export feature tracker: http://code.google.com/p/lilypond/issues/detail?id=665&q=musicxml&colspec=ID%20Type%20Status%20Stars%20Owner%20Patch%20Needs%20Summary
Work done on musicxml2ly for Philomelos.net that has not yet been incorporated back into the version shipped with LilyPond: (the low hanging fruit?) http://lists.gnu.org/archive/html/lilypond-user/2013-03/msg00919.html

Mike Solomon:
For me, it would be maybe 2 weeks of work to exhaustively convert what we call the "music stream" to MusicXML. This could be faster or slower depending on how familiar one is with Scheme, LilyPond and XML. The "music stream" is the internal representation of the music in Scheme that gets passed to what we call "engravers" (Stem_engraver, Clef_engraver, etc.) from which graphical objects are made and then eventually split over systems if necessary. To get more precise information about layout into the MusicXML would be difficult, as there is not a one-to-one correspondence between events in the music stream and musical objects created.

Urs Liska:
As David (and Mike) pointed out not everything is included in the music stream, so a XSLT conversion won't be completely sufficient.
But the music stream doesn't have to be the only source of information. Any program/function doing the export should draw on as many sources as possible to the the best possible XML representation of the LilyPond score.
Is the information that eventually leads to the point-and-click links present at that stage? (I think it should because I couldn't imagine how it could be added later).
Maybe this could be a bridge for a two-way approach (music stream and input file parsing)?

Urs Liska:
I think that a Python program might be a good idea:

• I'm sure there is good XML support with Python (don't really know, though)
• We could surely use code or experience from the musicxml2ly project
• We could probably also use code and ideas from Frescobaldi.

Vaughan McAlley:
As far as I can tell, Frescobaldi seems to keep a very strong internal representation of Lilypond source for its own purposes, which could be worth utilizing.

Mike Solomon:
Python is a great tool for XML creation. There is also http://okmij.org/ftp/Scheme/xml.html. The advantage of doing it in Scheme is that you are building it directly into LilyPond. I'd take a stab in the dark and guesstimate that 60% of the LilyPond user base that needs MusicXML export would not be comfortable running a Python script.

Paul Morris:
Doing it in scheme does seem to offer some nice advantages. I looked into it and found that there's SXML, a scheme version of XML:
https://en.wikipedia.org/wiki/SXML
http://www.ibm.com/developerworks/library/x-matters31.html

SXML:
    '(doc (title "Hello world"))
XML:
    <doc>
        <title>Hello world</title>
    </doc>

And there is existing scheme code that converts SXML to an XML file:
http://modis.ispras.ru/Lizorkin/sxml-tutorial.html#hevea:serializ
So if all went well (ha ha) it would "just" be a matter of rearranging the scheme data from the music stream into SXML format, following the MusicXML specification. Though I'm sure that's harder than it sounds.
Having all the moving parts in scheme would be nice and having an internal scheme version of the XML file might make it easier to (eventually, maybe) add in additional data that is not part of the music stream.

Support for SXML conversion to/from XML is built into Guile 2.0 (LilyPond currently uses Guile 1.8): http://www.gnu.org/software/guile/manual/html_node/SXML.html#SXML (David Pirotte pointed this out.)

http://libmusicxml.sourceforge.net/ is mentioned as a potentially useful tool. "The MusicXML library is a portable C++ library designed close to the MusicXML format and intended to facilitate MusicXML support."

This discussion on LilyPond's issue tracker gives a lot of insights:
http://code.google.com/p/lilypond/issues/detail?id=665

Including comment #15 from Jan Nieuwenhuizen: "You may want to look at input/regression/to-xml.ly and scm/to-xml.scm, some old and preliminary work there." Searcing for "to-xml.ly" in the lilypond-devel archive only brings up 5 results, but the file has existed since September 2002 at least: http://lists.gnu.org/archive/html/lilypond-devel/2002-09/msg00199.html

One possibility is to follow the example of MIDI file creation/export. This would apparently require working in C++ rather than Scheme or Python. On 10 Jul 2011 Reinhold Kainhofer (who coded the musicxml2ly python script) wrote:

"I have rather been thinking about adding MusicXML export like the MIDI export: Create some translator (e.g. a base class Exporter or Xporter to indicate XML, like Performer or Engraver) and subclasses to listen to all the different music events/grobs. That would probably be the most native and probably also easiest implementation of MusicXML export. For example, MIDI creation is relatively simple code-wise (just look at the performer.cc and the audio.cc and midi*.cc files)."

From the following lilypond-devel thread: http://lists.gnu.org/archive/html/lilypond-devel/2011-07/msg00398.html