Science Fair 2005-2006 Logbook

I’ve been thinking a lot about science fair lately, so I decided to get an early start this year on my project.
Last year in 10th grade, I completed an AP course in the java programming language in which many important data organization concepts, collections, mathematical algorithms, and object-oriented designs were emphasized. Over the past few years, I've also made it a point to become a well-rounded programmer by learning parallel concepts in perl and C++. Additionally, I've been practicing a multitude of graphical applet aspects in java, and I've even tried a little bit of opengl programming in C++ (although I have much more experience and comfort with applet APIs and the swing/awt libraries).

Last year I combined my love for programming with my interest in a traditional science. This year, I would also like to combine another area of science with computer programming. At the moment, I have two ideas:

1. To combine music and directx, I would like to make a program under C++ that takes microphone input and composes music to the screen based on the rhythm and the pitch. A slider would allow the user to choose a tempo, and a blinking metronome would show the tempo at all times (this would be used to determine the rhythm of the notes). Then, possibly, the user would be able to add articulation. The purpose of this program would be to make an "automatic composer" compatible with any instrument (not just the ones that can be plugged into the computer's MIDI port). Some other features could include printing capabilities and the option to save music into a MIDI file. To do this project, not only do I need to figure out how to do windows programming to have a nice user interface, but I need to figure out how to use low level Directx libraries to analyze sound waves to determine pitch and amplitude.
2. My alternative option would be to combine physics and opengl. Recently, I've been deriving some trigonometric equations that allow for the appearance of motion around a 3D world. I could find some situation to simulate that included lots of physics topics. Therefore, some sort of motion or collision detection would be included in the science project. For example, with the use of proper lighting and a particle engine, I could simulate an asteroid going through the earth's atmosphere at a chosen speed. Before doing this, though, I would have to research a lot more about quaternions and matricies to be comfortable with 3D rotation axes and motion on an x,y,z plane.

Today, I decided to look around online for some tutorials on C++ in order to brush up on some topics. I used the following two sites:
www.cplusplus.com
www.cprogramming.com
Although I had made many programs in C++ before, I knew that it would be important to research some more complex topics if I was going to make such a large project. Due to the fact that java and C++ are so similar, it was relatively easy for me to pick up some of the more complicated topics. Now that I'm finished, I feel much more comfortable with object oriented programming, structures, pointers, and dynamic memory. After some practice, I think I may be as comfortable with C++ as I am with java. The fact that I've had lots of practice organizing efficient code will certainly help me along the way with this project. My next step in the project, therefore, will be to find out more about Directx libraries, especially the DirectSound one.

I've had some very frustrating experiences lately with my research. I'm having a very hard time finding a good tutorial about Directx. Because of that, I decided to try a little bit of windows programming first (I'll need it to use Directx). I found a pretty good tutorial about windows programming in C at the following link:
www.winprog.org
I really wanted to start right with the Directx, though, so I went directly to Microsoft's site to find out what they had to say about the DirectSound libraries (Microsoft is, after all, Directx's creator).
http://msdn.microsoft.com/library/en-us/directx9_c/DirectX/htm/directsoundccreference.asp?frame=true
It all looked pretty promising. I downloaded the Directx SDK for Windows XP and everything. However, I was having a really hard time getting any of their DirectSound examples to compile in Dev C++. To help the situation a little bit, I downloaded the free Visual C++ compiler from Microsoft that was built into the console. Not even this would work, though, for I kept getting strange errors even when I found the correct include files.

About a week ago, I found a great site that explains audio input processes (among other things such as MIDI programming) in java on the official java domain:
http://java.sun.com/j2se/1.5.0/docs/guide/sound/programmer_guide/contents.html
Lots of code examples are provided, and it assumes little prior knowledge about these topics (perfect for me). Just looking at this tutorial, I've gotten a few ideas already about additional features in my program:

• Give the user the option to have midi playback and have the user save into midi format
• Have a nice user interface with menus that allows choices about microphone sources and Line info
• Obviously, I'm going to have to implement the ideas of clefs, time signatures, and key signatures at some point, but I'm just going to get a basic program down first that shows me the letter and octave of a particular note streaming into the microphone

My two player tetris game	My awesome sound processing book find

Since my last entry, I've accomplished a good bit with my project. First of all, to practice some java topics I hadn't really tried before (namely threads, timers, and menu bars) I made a two player tetris game. I used [http://www.math.hws.edu/javanotes] to help me out with these topics (a great site for finding out information about applets and layouts in java...I've been using it for over a year now). This will save me a lot of time later when I go to make an intricate layout for my program.
As far as the sound research goes, I realized that the sun site only provided information for sound input and output, not for sound processing.

Luckily, though, I was able to come across a really good book title (Digital Audio with Java by Craig Lindley), and this just arrived on my doorstep this morning from Amazon.com. Even though it's a bit dated, I've found that it provides some vital low-level code and algorithms just by skimming through it a little bit. The first 1/3 of the book mostly talks about developing java beans for a user interface (I'm going to skip this part since I'm coming up with my own algorithms for that), but the rest of the book looks really promising. My plan is to read over this book for a while until I've finished most of the sound processing part. Then, I'm going to make a sound program that loops through continuously and gives me a pitch value when I play my violin. It's going to be very basic compared to the larger picture I described initially, yet I still expect this to take me to the end of the summer. My next update will include code excerpts and screenshots from this small program.

Oh, and one last idea for the day. To test the accuracy of my program when it's finally finished, I'm going to have three different pages of sheet music. I'll deem one SIMPLE, one MEDIUM, and one DIFFICULT. I'll then print out the sheet music that the program feeds to me after I play them and compare each note. 50% of each note will be pitch, and the other 50% will be rhythm (or equivalent rhythm). A total accuracy will be counted up at the end. Also, I may or may not include dynamics in the program, but the user would have to calibrate the program to mezzo piano (a medium dynamic) with a particular instrument in the beginning in order to do this. Obviously, I'd have to work directly with the amplitude values of the sound waves if I was to do this.
****Speaking of accuracy, a practical application of this program could be to test the accuracy of deaf people when they try to sing notes. In a sense, this program could teach them to sing by pointing out their mistakes during the recording session.

The E string	The A string	The D string	The G string

(http://telecom.tbi.net/digpcm.htm)

• 16000.0 samples per second (a "medium" quality of data precision that should allow me to get enough cycles of frequencies)
• 16 bits (2 bytes) per sample (compatible with the "short" variable type"
• 1 input channel
• Signed storage is set to "true" (positive/negative instantaneous amplitude values)
• bigEndian format is set to "true"

• A frequency is how "frequently" a wave goes through its period (one revolution of its pattern) per unit time. It is often expressed in hertz (hz: cycles/second), which is what I will be using in my project
• Sound waves are really just the sum of a bunch of these sine or cosine frequencies. Since sound waves are pressure waves, they displace air molecules in a periodic fashion, which later impinges upon our ears (or on a sensitive fiber in a microphone) and gives the perception of sound.
  EXAMPLE: A 440hz note is perceived as a "concert A"
• The average of the instantaneous function readings over a whole number of cycles is key in the Discreet Fourier Transform. Normally for a periodic wave function, regardless of complexity, the area under the positive and negative regions of the curve will cancel out, making the average reading exactly zero.
  However, when multiplying the function of a particular frequency by the function of a wave containing that frequency, the average is above zero. This is proveable by the product trigonometric identities.
  

  The average of the sin(X) function is zero	The average of a sin(X)^2 function is 0.5

  
  The first graph shows a complex waveform containing a 3hz sine frequency, where the average is zero. The graph to the right shows the result of multiplying a 3hz sine frequency by the function (the average is clearly above zero)
• Using this concept of averages, it is possible to calculate the power of an entire range of frequencies. When the program runs, I will use the following expression to calculate the average of samples (as opposed to the integrals under a smooth graph) for all of the frequencies in the range of a chosen intrument:
  

  N-1            __            \ z =(1/N) * / x(n) * y(n)            --            n = 0

  N represents the number of samples, x(n) represents the series of captured samples, and y(n) represents the sine or cosine function. To calculate the power for a frequency, I will be summing the square of the result of the cosine as y(n) [real component] and the square of the result of the sine as y(n) [non-real component] in the above expression. (Full Derivation was found at dickbaldwin.com)
• When I make a program to implement this, the sound sample size that I'm going to be applying to the equation is going to depend on the tempo that the user chooses (The slower the tempo, the larger the sample size). With larger sample sizes, a higher percentage of full cycles of the wave is collected. Since the Fourier Transform depends on a whole number of cycles for the power reading to be accurate, I definitely expect this to cause some problems in the future. Thus, I will probably test under very slow tempos and advise the user to do the same.

Playing an A on the G string (1 second sample)	Playing the open A string (50 millisecond sample)

Here's where I actually implement the Fourier Transform math:

This method works hand-in-hand with FourierPower() to check the power of every frequency

• public void loadFrequencies(String str): This method accepts a parameter that tells the location of a profile file with a list of frequencies and their associated note names. It then puts them into a map as a Float frequency object with an associated String note name.
• public DSP(int c, int ss): This constructor accepts two parameters: the "chunkLength," or the size of the buffer determined by the sample size (can be changed during the program run), and the "samplesPerSecond" variable that is vital to the FourierPower() method. Some other instance variables are also initialized here, such as the "minPower" variable used in the actual program to cut off frequencies below a certain power.

This is where all of the input and processing takes place:

The code below is part of the WaveDisplay class that draws the complex waveform to the screen every time the buffer is filled. Essentially, it sets up a pixel ratio on the y axis so that the amplitude range fits into a 200 pixel height, and it sets up a pixel ratio on the x axis based on the size of the buffer at that time. Each iteration of the loop yields a line segment between one amplitude sample and the next amplitude sample in the buffer based on the calculated pixel ratios.

• MusicDisplay: Unfinished at the moment, but it will be the large, scrollable area at the bottom of the screen to which the sheet music is drawn based on the rules of music theory.
• TopDisplay: This is the class the either displays a fingerboard (if the user is manually composing by clicking) or a sound wave representing the incoming data. In the former case, the MouseListener interface is implemented to detect when the mouse is within range of the circles representing different spots on the fingerboard.
• Metronome: This class is set up to draw the square in the upper left hand of the screen that helps the user keep time. Based on a chosen temp, two timers are set up: blinkTimerListener (changes the dot from blue to red each beat) and sixteenthListener (checks to see how long the user has been pressing on one note based on the StopWatch class, and adds a sixteenth rhythm with the proper midi note number to a static collection in the SoundSample class).

• public Note(int midi, float scalepos, char notename, char sharpflat, boolean accidental): This is the primary constructor for the object. Each Note object has a midi number that is associated with it, it has a position on the scale, it has a note name from A-G, it is either sharp(#), flat(b), or natural( ), and it could be an accidental based on a particular key signature (discussed later)
• public static getFreqfromMidi(int num): Calculates the base frequency of a note based on its midi number
• ****public static TreeMap loadKeySig(String str): One of the most important methods in the entire program. Given the name of a key signature, it will read in a file with ".dat" on the end of it in the "Keys" folder. It will then go line by line, constructing Note objects and adding them to the TreeMap as values associated with an Integer key object storing their midi number. Here is a sample line from the "G Major" key signature file that must be interpreted:
  63,0.5,Eb,1
  63 is the midi number, 0.5 means that the note goes through the first bar in the music, Eb tells us that it is an "E flat," and 1 at the end tells us that it is an accidental in the key of G Major.

• public SoundSample(int midinum, int length): The main constructor for a SoundSample object that stores its midi number and the length in sixteenth notes
• public void setLink(): This method tells the program to tie this note to the next note. This could happen either if the note is composed of a very eccentric rhythm or if it is being held between two measures
• public static ArrayList getModifiedNotes(int maxPerMeasure): Up to this point, the music ArrayList object stores all of the gathered music as sixteenth notes. It is the job of this method to merge many samples of the same note in a row into eighth notes, dotted eight notes, quarter notes, dotted quarter notes, half notes, dotted half notes, or whole notes. It also divides the notes up properly so that they fit into a measure. For instance, in a 4/4 time signature, sixteen sixteenth note beets can fit into a measure. Each time this limit is exceeded and a note is still being held, it is tied to the next measure. The collection returned at the end has all of the notes in their proper rhythms with indications as to whether or not each note is tied.
• public static void addSplitrhythm(ArrayList a, int midiNum, int noteSize, boolean linksNext): A helper method for getModified() notes that splits up eccentric rhythms into tied ones that can be drawn by the program. Every note that is processed in the getModifiedNotes() method must pass here to be inspected. For instance, a note that was seven sixteenth notes long would be split up into a dotted quarter note tied to a single sixteenth note. If the note was a rest, on the other hand, it would be split up into a quarter rest, a half rest, and a sixteenth rest, for there is no such thing as a "dotted rest."
• public static void drawNote(Graphics g, int xpos, int ypos, Note n, SoundSample s): Given a Note object and an associated SoundSample object that stores its rhythm, this method will draw a note to the specified position on the screen. If the note is past the second bar, then the tail points downward. Otherwise, the tail points upward. Ledger lines are also drawn for lines that are above the scale or below the scale, such as the low G or the high B flat. The drawArc method is used to draw ties between notes.
• public static void drawRest(Graphics g, int x, int y, int length): This method does exactly what it says. Whole rests and half rests are just rectangles, but all of the rest of the images are too difficult to draw, so they are imported as static images in the Settings class.
• public static int drawMusic(Graphics g, Settings settings, int scrollpos): This method ties everything together. First, it uses FontMetrics to figure out how to position the title on the screen. Then, it begins to draw all of the notes, indicating the key signature on the left of each new line and drawing thick, vertical lines at the end of each measure. After it is finished executing, it returns and integer that tells how many pixels down the music had to be drawn, which helps a scrollbar that I created in the MainGUI class to figure out the bounds of the display area.

1. Keep taking in and storing 5000 bytes of audio input into a ByteArrayOutputStream object (like a huge buffer).
2. When the user has finished recording, pass all of the data as a short buffer to the DSP class's new method makeMusic(ShortBuffer buffer) for processing.
   • Calculate the length of buffer needed to interpret for one sixteenth note: sampleRate*timeSigBottom/(16*tempo)
3. Do a Fourier transform on every sample until the end of the buffer is reached.
   • Calculate the power of the fundamental frequency and the first five harmonics for every note in the key signature. If they are all above some reference level for a particular note, then it is safe to say that the note exists, so a sixteenth sound sample can be added to the music collection
   • If no note has been detected, then add a rest to the collection