Piano Rocker dev diary 1 - What is up

originally i've posted this here but who cares?

I've been working on this little idea of mine, a game where you play piano for at least half a year now.

It all started in may 2021 when i've figured WEB Midi - exists. And that could allow me to listen to midi keyboard inputs, so if i could render a rock band 3 chart (which is a midi file + ogg stems) for PRO KEYS MODE - that would be freaking great.

• 3 abandoned prototypes in
• 1 insanely viral reddit post with 25 upvotes
• 1 steam storefront page published

and many months of dread since, a lot of things are done, but nothing is really DONE done.

To put my mind elsewhere, but not too far away I start this "PIANO ROCKER DEV DIARY" series, where i blabber about technology, development, music games and yknow.. whatever comes to my mind.

Case study of the day:

Rendering musicXML

Sheet music - exists, like it or not, playing keyboard is very tightly coupled with reading sheet music. All the exceptions, jazz legends who all "learnt it by ear", just make the overall rule stronger. I must render sheet music in some shape or form :(

Trust me, i did try to dodge this bullet, but I don't think i can, not anymore.

There are other "alternative" notations worth mentioning. Most curious ones are ABC Notation, the top-down charts a-la keyboardmania that were made popular by Synthesia and the new-emerging craze of VIRTUAL PIANO (playing piano on ascii keyboard, it is a thing, people actually do it. Even crazier, they do it in ROBLOX

I have seen yousician, simply piano, pianoo and such, all rendering sheet music, and after a bit of digging in local cache of yousician windows app i can assure you - they all parse music XML.

Biggest public source of sheet music i'm aware of, provides scores downloadable as PDF (nope), MIDI (seems really good, HEY, i already have it implemented, but trust me, it's not, i might cover this as separate post someday), their proprietary format AAAND Music XML.

It's widely adopted, it's open as much as TEXT FILES can be, it's free - i just have to learn how to interpret it and draw notes. Just like all these guys already did, perhaps a bit more lively (hopefully).

I'm still stuck with this shit, real stuck, but here's what i have learnt so far:

• *.mxl files arent XML files, they are ZIP files with XMLs in them. jszip to the resque, this wasn't too hard

• most probably you'll want to transform xml to json, if you do javascript like i do, that's trivial too, buncha libs out there, i use fast-xml-parser for now

• i render things in PIXI.js, so for that i would need to draw all the cute 🎼🎼🎼♫♫♫s. Good news - every unicode font has a range for music symbols. Bad news - that set isn't a complete set and you can't do much with it. All music software uses special MUSIC FONTS (rendering all as text symbols)

• There are buncha music fonts, some of them - can work in browser, but good luck figuring out what's there and what's not, how symbols are mapped and such. I have spent half a day trying different free fonts, till i found out about SMUFL a standard for music fonts. Also check their brief history of music fonts its a great read!

• For a free to use in commercial project font IMPLEMENTING SMUFL - i chose Bravura it works, using reference from smufl website it's possible to map anything you can imagine to a unicode symbol. Lets say you need a g-clef, go to this page, g clef is U+E050 (and U+1D11E), great, create your PIXI.Text('\uE050') - here's your g-clef (if you managed to preload the font)

• preloading fonts is a bit of a pain

so far i havent even touched actual music XML contents yet, right? Oh shit! Here we go again!

• Notes have DURATIONS. Durations are not measured in TIME, it's sheet music, right? We have quarter notes, half notes, sixteenth notes, 1024th notes and so on and so forth. If you want to have anything interactive - you would need to convert these to TIME. Now here's a question, HOW WOULD YOU DO IT?

how would you convert MUSIC NOTES to time and duration (in seconds)

so your typical music xml measure.note would have pitch, consisting of step (of chromatic scale, C, C#, D, whatever it is) and octave (int).

Then there's type and duration. Type is within: [whole, half, quarter, .... 32nd, 64th ..... 1024th and so on ]. duration is an int number.

Being an idiot i am, i didn't read the spec, i just debugged my twinkle-twinkle-little-star chart and decided empirically that 1 is quarter note, 2 is half, 4 is whole. Twinkle-twinkle never went smaller than quarters, worked great. I even wrote a string to int switch-case, basing time strictly on TYPE instead.. DUMB.

Hard truth incomming

1. In 60BPM (beats-per-minute) you'll hear/play 60 BEATS in one minute. One beat is one quarter note in 4/4 (or 3/4, or anything/4). Which means 1 beat in 4/4 in 60bpm is exactly one second. One bar in 4/4 60bpm is 4 seconds, then, one bar in 3/4 60bpm is 3 seconds and so on.
2. Every MEASURE (parent node of notes) has attributes, one of which is DIVISIONS. DIVISIONS is number of divisions PER QUARTER NOTE. ( the PQ in PPQ for my cool midi gang ).

This number is the smallest integer that can be divided without .decimals for the smallest note division present in sheet. So if all your notes don't go smaller than quarter notes = DIVISIONS=1 is good. Quarter is 1, half is 2, whole is 4. If you have 8th notes - DIVISIONS would be 2, if you want to have 16th notes - 4, 16th and TRIPLETS - something that can be divided by 2 AND 3 would be needed (24 is good)

Now lets take this example from https://www.w3.org/2021/06/musicxml40/tutorial/midi-compatible-part/

<attributes>
    <divisions>24</divisions>
    <key>
      <fifths>-3</fifths>
      <mode>minor</mode>
    </key>
    <time>
      <beats>3</beats>
      <beat-type>4</beat-type>
    </time>
  </attributes>

....

<note>
    <pitch>
      <step>G</step>
      <octave>4</octave>
    </pitch>
    <duration>12</duration>
    <lyric>
      <syllabic>single</syllabic>
      <text>Dans</text>
    </lyric>
  </note>
  <note>
    <pitch>
      <step>C</step>
      <octave>5</octave>
    </pitch>
    <duration>8</duration>
    <lyric>
      <syllabic>single</syllabic>
      <text>un</text>
    </lyric>
  </note>

What we know about time so far, is: 3/4, divisions = 24 divisions per quarter note. Let's say, for this example BPM = 120.

Now how long exactly does first note play?

Divisions(perQuarterNote) = 24, BPM = 120, Duration = 12

timePerOneDivison = (60 / bpm) / Divisions

and then duration of this note, in seconds, is

duration * ((60 / bpm) / Divisions)

12 * ( (60 / 120) / 24 ) = 0.25 sec.

Based on which stave the note is (left or right hand) you add this duration to your gTime or fTime variable, and that would become the start time of next note. Cool right?

Kinda.. Since I'm also "playing" the notes using the glorious Tone.js something with my math is still not quite right, but visuals look correct, so i blame my audio code / use of transport for now.

I hope this helped somebody. As a final note - here's how it all looks like in current prototype (sheet music supports is a project separate from mainline game for now, i'll merge it in once it's done, or merge main game into this small clean boy.. )