Virtual Harp (Cooper Union Hackathon, 2016)

HackCooper 2016

In September 2016, I attended HackCooper 2016, hosted by Cooper Union students. Since it was only 24-hours-hackathon and my friend and I had other commitments on the next day, our motto was ‘Let’s build SOMETHING at least.’

My teammate, Brendan was a musician (currently a freelance keyboard player in Broadway) and I have worked on interactive music installation as my thesis project, we wanted to build a virtual musical instrument. Inspired by Brendan’s real-life-struggle of synchronizing keyboard and harp sound during the musical, we created Virtual Harp to make it easier for the keyboard musician to play simple harp sync sound without bringing the actual gigantic instrument down to the already-crowded orchestra pit.

쿠퍼 유니언 대학교에서 주최한 HackCooper 에 가볍게 참가해 만든 프로젝트입니다. 제 졸업논문에서 썼던 아이디어와 짧은 코딩을 바탕으로 이번 해커톤에서는 가상 속 악기를 만들었습니다. 같이 프로젝트를 만들었던 친구가 브로드웨이 키보드 연주자 출신이었고 하프 소리와의 싱크로율이 잘 안 맞았던 그 친구의 연주 경험을 살려 다른 악기를 쉽게 조절할 수 있는 프로토타입을 만들어보게 되었습니다. 브로드웨이 연주 시의 고충으로 만들게 된 프로젝트이지만, 무거운 악기를 들고 다니지 않아도 이렇게 온라인이나 간단한 기계로 비슷한 연주를 할 수 있다면 좀 더 mobile하고 flexible한 연주들을 무리없이 할 수 있을 것이라는 생각으로 끝까지 프로젝트를 완성했습니다. 아래 비디오는 손 감지기 하드웨어와 게임엔진 프로그램으로 만든 가상 속 악기 데모 모습입니다.

Demo Video

The video below demonstrates the interactivity of virtual harp. It turns into yellow when the player touches it. It turns back to gray when the sound is over (It doesn’t change to gray in this video because I am touching the string before its sound file ends.)

Back-end

Below is the screenshot of how it is constructed in Unity. You can see each string with capsule collider (left) and each string has the sound file (right). I built on top of leapmotion-unitypackage which has several interesting leap motion examples.

During the process, we thought if all musical instruments can be virtually played,
1. more people would be able to experience and learn music without paying all the money for instruments (which often discourage people to start the musical education), especially when they JUST want to try it out.

And 2. street musicians could play more ubiquitously, since they would not need to worry about the hassle of bringing the instruments everywhere (especially the large ones). Of course, this would need to wait until Leap Motion can detect larger area & more subtle/sophisticated hand movement and can be attached to the phone (to be more mobile and lightweight, not just laptop and VR headset).

More info about this project’s HackCooper submission can be found here and here’s the github repo

Source code

1. When a string is touched, play the sound.
2. Change the color of the string to yellow if the sound is playing


using UnityEngine;

using System.Collections;

public class SoundEffect: MonoBehaviour {

    public AudioSource source;
    bool colourChangeCollision = false;

    public void Awake()
        {
        source = GetComponent<AudioSource> ();

        }

    public void OnTriggerEnter (Collider other)
    {
        //if (!source.isPlaying){
        source.Play ();
        /* commenting out the condition clause to allow repeating the sound, even though the player just played that key.

        }
        else */
        colourChangeCollision = true;
    }
    public void checkColourChange()
    {
        if(colourChangeCollision)
        {
            transform.GetComponent<Renderer>().material.color = Color.yellow;
            if(!source.isPlaying) //Time.time > currentDelay
            {
                transform.GetComponent<Renderer>().material.color = Color.grey;
                colourChangeCollision = false;
            }
        }
    }

    public void Update()
    {
        checkColourChange();
    }
}

Advertisements

Please Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s