Info from Pioneer XDJ 1000MK1

[juliu-s]

Hi there,

I saw your posts at xsco/libdjinterop#15 and mixxxdj/mixxx#9463 and I tested your work.

I don't know what the goal of the project is (if there is any) but I think it would be awesome if I could create folders (like playlists) on an USB stick and analyze the tracks inside and get the big waveforms like here.

So, below is my "process". If you want me to test something for you, please let me know.

Build

I used the golang container for building the application.

$ podman run --rm -it -v /run/media/julius/TEST:/opt/USB docker.io/golang:1.20.3 bash
root@af7bfc594118:/go# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
..snip..
root@af7bfc594118:/go# apt update && apt -y install ffmpeg
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
..snip..
root@af7bfc594118:/go# cd /opt
root@af7bfc594118:/opt# git clone https://github.com/ambientsound/rex.git
Cloning into 'rex'...
..snip..
Resolving deltas: 100% (6/6), done.
root@af7bfc594118:/opt# cd rex/
root@af7bfc594118:/opt/rex# go build -o rex cmd/rex/main.go
go: downloading github.com/lunixbochs/struc v0.0.0-20200707160740-784aaebc1d40
go: downloading golang.org/x/text v0.5.0
root@af7bfc594118:/opt/rex# go build -o analyze cmd/analyze/main.go
root@af7bfc594118:/opt/rex# ./rex -h
Usage of ./rex:
  -f	Overwrite export file if it exists
  -root string
    	Root path of USB drive (default "./")
  -scan string
    	Path to music files, scanned recursively. Must only contain MP3 files. (default "./")
root@af7bfc594118:/opt/rex# ls -l /opt/USB/
total 4
drwxr-xr-x 2 root root 4096 Apr 30 11:00 tracks
root@af7bfc594118:/opt/rex# ls -l /opt/USB/tracks/
total 52628
-rw-r--r-- 1 root root 14554413 Aug 17  2022 'Caim - Lagom.mp3'
-rw-r--r-- 1 root root 21252223 Aug  7  2022 'Janeret - Rear.mp3'
-rw-r--r-- 1 root root 18075967 Aug 17  2022 'Per Hammar - Inter City.mp3'

Scan and analyze

root@af7bfc594118:/opt/rex# ./rex -root /opt/USB -scan /opt/USB/tracks 
Add "/opt/USB/tracks/Caim - Lagom.mp3"
Add "/opt/USB/tracks/Janeret - Rear.mp3"
Add "/opt/USB/tracks/Per Hammar - Inter City.mp3"
Wrote /opt/USB/PIONEER/rekordbox/export.pdb
root@af7bfc594118:/opt/rex# ./analyze -index -rows /opt/USB/PIONEER/rekordbox/export.pdb
PIONEER DJ DeviceSQL file
tables=20, tx=11
Table: "Tracks"
  Meta: u1=1fff u2=1fff numentries=00
  Tracks Page: idx=02 rows=  3 deleted=  0 used= 728 free=3292 large=0000000000 tx=0006 flags=34 u3=0060 u4=0000 u5=0001
    RowSet: bitmask=0000000000000111 lastwrite=0000000000000100
      Track: heap=0000 id=0001 shift=00 exists=true  path="/tracks/Caim - Lagom.mp3"
      Track: heap=00e8 id=0002 shift=20 exists=true  path="/tracks/Janeret - Rear.mp3"
      Track: heap=01d4 id=0003 shift=40 exists=true  path="/tracks/Per Hammar - Inter City.mp3"
  Table summary: records=3 deleted=0 total=3 rowsets=1
fatal error: seek /opt/USB/PIONEER/rekordbox/export.pdb: invalid argument
root@af7bfc594118:/opt/rex#
root@af7bfc594118:/opt/rex# ./analyze -index -rows -dumb /opt/USB/PIONEER/rekordbox/export.pdb
00000: numtables=20, next_unused_page=32000, sequence=11
Tracks               first=01 last=02 empty_candidate=2d
Genres               first=03 last=03 empty_candidate=04
Artists              first=05 last=06 empty_candidate=2e
Albums               first=07 last=08 empty_candidate=2f
Labels               first=09 last=09 empty_candidate=0a
Keys                 first=0b last=0b empty_candidate=0c
Colors               first=0d last=0e empty_candidate=2c
PlaylistTree         first=0f last=10 empty_candidate=30
PlaylistEntries      first=11 last=12 empty_candidate=29
Unknown9             first=13 last=13 empty_candidate=14
Unknown10            first=15 last=15 empty_candidate=16
HistoryPlaylists     first=17 last=17 empty_candidate=18
HistoryEntries       first=19 last=19 empty_candidate=1a
Artwork              first=1b last=1b empty_candidate=1c
Unknown14            first=1d last=1d empty_candidate=1e
Unknown15            first=1f last=1f empty_candidate=20
Columns              first=21 last=22 empty_candidate=31
Unknown17            first=23 last=24 empty_candidate=2a
Unknown18            first=25 last=26 empty_candidate=2b
History              first=27 last=27 empty_candidate=28
fatal error: binary.Read: invalid type *page.IndexHeader

Results

• I see a "REX .." playlist when I load the USB into my Pioneer XDJ1000 MK1 where I can find, load and play the MP3 files.
• The other player can also find, load and play the MP3 files via LINK.
• There is no waveform like here (I don't know I should expect one)
• When combining MP3, WAV and AIFF files the player completely freezes :)
• If I "just" create folders and put tracks in them and do not analyze them in any way the players detect the files and load the small waveform (just above "NEEDLE COUNTDOWN" in the picture here which is not the case if I scan them using REX.

[kimtore]

Awesome, thank you for testing and reporting back!

I will clarify some limitations:

• Only MP3 files are supported at the moment, the player probably crashes due to WAV/AIFF files being reported as MP3
• Waveforms are not implemented yet

The primary goal of this project is to contribute to the reverse engineering efforts of the export format by adding missing pieces of documentation, and providing a reference implementation of how to write these files.

Now that the project has some traction, I will consider adding the missing features such as:

• Waveforms
• Beat grid
• Hot Cue
• Playlists per folder
• Export directly from a Mixxx database

I will probably start with the latter, because Mixxx holds plenty of information that can be scraped, so that analysis does not have to be re-implemented.

Not sure about the time frame of all of this, but this feedback definitely motivates me to finish quicker :)

Regards, @ambientsound

[juliu-s]

That would be awesome. Then you are able to use the same workflow as with Rekordbox. Prepare your playlists / sets in Mixxx and export the playlists to USB. Good luck and if you need me to test something, please let me know.

Regards, @juliu-s

[kimtore]

@juliu-s,

I've implemented conversion from a Mixxx library to Rekordbox. Would you be so kind and test this against your own library and see what works or not?

The following should work:

• All playlists and crates are imported as playlists
• Only tracks that are referenced by playlists and crates are copied to USB media
• Any files other than MP3 will be transcoded to MP3 (will be configurable in later versions)

You can run it now with rex -root /path/to/USB, no other parameters required on Linux. If you're on Windows you need to configure the path to the Mixxx library.

[juliu-s]

I am running Arch Linux and did not use Mixxx before, I installed Mixxx (2.3.4) and created a few playlists (no crates) containing only MP3 files. I formatted an USB and tried, and it failed.

❯ rex -root /run/media/julius/USB
REX: unofficial Pioneer DJ export.pdb generator
This software is neither supported nor endorsed by Pioneer.
Please do not rely on it for serious use.
Mixxx database opened: /home/julius/.mixxx/mixxxdb.sqlite
PIONEER database created: /run/media/julius/USB/PIONEER/rekordbox/export.pdb
fatal error: no such column: library.coverart_color

Result:

❯ tree /run/media/julius/USB
/run/media/julius/USB
├── PIONEER
│   └── rekordbox
│       └── export.pdb
└── rex

[kimtore]

It seems I have a database file which has been through a future version of Mixxx and my schema is incorrect.
Please try with the latest commit.
If it still doesn't work, please send me the sqlite file.

[juliu-s]

With the latest verison:

❯ rex -root /run/media/julius/USB
REX: unofficial Pioneer DJ export.pdb generator
This software is neither supported nor endorsed by Pioneer.
Please do not rely on it for serious use.
Mixxx database opened: /home/julius/.mixxx/mixxxdb.sqlite
PIONEER database created: /run/media/julius/USB/PIONEER/rekordbox/export.pdb
fatal error: no such column: library.last_played_at

Sqlite file: https://drive.proton.me/urls/CY0QF0KG5M#UQybrJYslRuT

[kimtore]

Thanks for the file, I have made another commit that hopefully fixes all your problems 🙏

[juliu-s]

Awesome stuff, it works. On the Pioneer I see the playlist containing the tracks, only the track title tho, no artist. But it plays etc. :).

❯ rex -root /run/media/julius/USB
REX: unofficial Pioneer DJ export.pdb generator
This software is neither supported nor endorsed by Pioneer.
Please do not rely on it for serious use.
Mixxx database opened: /home/julius/.mixxx/mixxxdb.sqlite
PIONEER database created: /run/media/julius/USB/PIONEER/rekordbox/export.pdb
Found 7 tracks in Mixxx database
Tracks imported.
Playlist "P: Tech" loaded with 4 tracks
Playlist "P: Dubtechno" loaded with 2 tracks
Playlist "P: Beats" loaded with 1 tracks
Tracks marked for export:      7 used/     7 total
Copying or encoding tracks to /run/media/julius/USB/rex
All tracks copied to destination
Writing PDB file...
Finished successfully.

[kimtore]

Thanks for the feedback, would you mind taking a few photos or a video, both with a Rex export and another from Rekordbox? It would help me understand how things are supposed to look.

Is Rex useful for you in this state, or what would it take?

[juliu-s]

Hi, no problem at all. Currently I am running Recordbox inside a VM, "so I got all the features" so there is absolute no rush. But Rex is useful when:

• Use Mixxx to create playlists and export them (like now)
• See artist and title
• See the beatgrid / waveform
• Supports WAV and AIFF file formats

Rex would completely replace Recordbox when (I don't know if Mixxx support this already):

• Exports the predefined CUE points
• Exports the predefined loops

On the Pioneer it's called "Performance mode", something like this. I can make a video of it, if you want.

The video's you asked for, can you give me a beep when you got them?

• Rex video
• Recordbox video

[kimtore]

Thanks, I pulled your files.
I'll check which Mixxx data is eligible for export.
Will get back to you eventually, thanks for help and testing so far :)

[kimtore]

Judging from the videos it seems that

• albums aren't showing up at the album list, I think it is because they don't have an artist connected to them
• artists aren't shown in the track view, I don't have a theory on that yet

[juliu-s]

Interesting, that made me think of the following test that I will do in the comming days. If you have any suggestions, please let me know.

setup:

• 3 USB sticks
• 2 MP3 files that I will check the metadata for: artist, title and album

usb1:

• 2 folders (p1, p2) both containing 1 track
• Check result on the Pioneer

usb2:

• Use Recordbox
• Create two playlists (p1, p2) both containing 1 track
• Edit the metadata in Recordbox (check if that effects the metadata of the files or that Recordbox saves that to it's own database)
• Check result on the Pioneer

usb3:

• Use Mixxx
• Create two playlists (p1, p2) both containing 1 track
• Edit the metadata in Mixxx (check if that effects the metadata of the files or that Mixxx saves that to it's own database)
• Export with Rex
• Check result on the Pioneer

[kimtore]

Just a side note to your tests: Rex always reads metadata directly from the Mixxx database itself, not the actual files. So your changes will always be reflected in the export.

[kimtore]

And the album/artist problems must be fixed within Rex itself, unfortunately I don't think there is anything you can do from the Mixxx perspective to get it working.

[juliu-s]

Some good news, it seems that rex is doing it's job after all. I tested as I said with two MP3 files only containing the Artist, Tittle and Album metadata. Side note, Mixxx saves the updated metadata to its database (tested it with the album field). It's not like updating it with VLC that updates the metadata of the file.

In the first try there was no info displayed, after I formatted the USB drive again and used rex the info popped up! Here are some pictures. With the Pioneer you can get info from a track when it is loaded and playing, or not loaded when you're scrolling through playlists.

[theot99]

That's great news that track metadata is loading on an XDJ1000! Other testers have had issues with that, although the metadata loads fine on my CDJ 2000 Mk1s. May I ask if it was a "quick" format or not?
Pretty sweet!

[juliu-s]

Hmm, it seems it was a lucky hit. I tried several ways* to reproduce it but no metadata appears. I orderd 2 new USB sticks and will try again.

• Gnome disks utility to format the USB stick, with and without the erase function.
• Destroying the first part of the USB stick using dd before using the Gnome disk utility again
• Destroy and recreate everything using dd and mkfs.fat

[kimtore]

In my limited experience with Pioneer players, I've found best results formatting the USB drive in Windows, and keeping the drive size < 32 GB. But there should probably be a way to format it properly in Linux.
Probably the sector size should be 4096 bytes, and align properly with the physical media?

mkfs.fat -c -F 32 -n label -S 4096 /dev/sdX.

[juliu-s]

So, it's been a while...

I tried your mkfs.fat -c -F 32 -n label -S 4096 /dev/sdX. suggestion on a 32GB USB but it didn't work. Then I used the disk utility from Gnome and did not quick format the drive (@theot99). The result was that the playlists with artists and track titles showed up.

[kimtore]

Awesome that you got it working. But did you also try with the quick format option turned on? It shouldn't be necessary to do a full zeroing of the drive.

I wonder what the correct parameters for mkfs are. If you happen to figure it out, please tell me and I'll add it to the documentation.

[juliu-s]

The correct parameter is: mkfs.fat -c -F 32 -n label -S 512 /dev/sdX (tested twice).
There is also a warning[1] displayed because of the lowercase labelname but the USB drive still gets recognized.

[1] mkfs.fat: Warning: lowercase labels might not work properly on some systems

[kimtore]

Thanks for the feedback, things are a bit slow here due to other pressing matters.
I've added it to the documentation.