Fun Times with Luddites and Zero Tracks on QuodLibet

Wow. That was in-ter-es-ting!

So yes – a couple of the “older” folks were a little wary of moving from cassette to digital format, which put me in the awkward position of defending this archiving project. Kind of the opposite of Paul Bunyan – with the new technology fighting to justify it’s value.

Last week, Quodlibet was finding Zero (of over 17,000) tracks. So there I am unable to play ANY music. Fortunately the old system is still in place, but I’m sure my ears were looking a bit red around the tips.

I thought it might have been an issue with the external drive either in terms of data, format or drive type.

After way too much time experimenting I decided to try looking into the Qoudlibet database. Is it postgreSQL? No. Actually it’s not a database at all. QL uses a “pickled list of AudioFile instances”. “Pickling” is the Python (language) word for a compressed file.

As per the FAQ, you’re supposed to be able to examine this “pickled” songs list:

Python 2.6.2 (r262:71600, Jun  4 2009, 15:54:27)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import quodlibet
>>> import cPickle
>>> with open(".quodlibet/songs", 'r') as songsfile:
...     songs = cPickle.load(songsfile)
...

(Note import quodlibet only worked for me when running the Python interactive interpreter from within the same directory as the main quodlibet.py file. I think there’s that directory could be added to the Python path, but haven’t looked into it yet.)

But the above was returning an error:

Traceback (most recent call last):
  File "/Users/mikekilmer/quodlibet/quodlibet/quodlibet/plugins/events.py", line 104, in __invoke
    handler(*args)
  File "/Users/mikekilmer/quodlibet/quodlibet/quodlibet/ext/events/viewlyrics.py", line 81, in plugin_on_song_started
    if not lyrics:
UnboundLocalError: local variable 'lyrics' referenced before assignment

Apparently in the current version of quodlibet you have to also initialize quodlibet: quodlibet.init().

Still no luck. More errors. Back to the Quodlibet Development Group on Google.

In the meantime I tried removing all but one of the directories from the Library. I tried replacing the /.quodlibet/songs cPickle file.

The songs pickle file resides in a directory named .quodlibet which is in your “home directory” and on Unix and Mac OSX this is referenced with the tilde: ~/. It also contains the playlists, lists, config and a few other files and directories. So I backed up ~/.quodlibet to ~/QLbackups and reinstalled the application. Working again. Yay!

Finally I removed the entire ~/.quodlibet directory (backing it up elsewhere), and through swapping components in and out between a fresh install’s ~/.quodlibet directory and the previous one, I discovered that it was two lines under the [browsers] heading in the ~/.quodlibet/config file that were causing the issue:

albums =
background = ketriketriketriketri # (this is the [whitespace ommited] name of one of the tracks from the collection)

Back in business. But wait. All of our playlist files are empty! Fortunately I back up the computer on our QNAP Network Attach Storage system using Time machine. So I was able to open the ~/.quodlibet direcory in the finder (by entering it in the Go>Go to folder field in the Mac finder) and then open TIme Machine, go back a few weeks and “restore” it.

Now we’re back in biz! Sweet.

Adventures in QuodLibet

The vision of Steve Jobs and Apple products are beautiful and beautifully intuitive, but the magnitude of the company and it’s market share combined with it’s Closed Source approach create a bit of a Bog Brother feeling at times. As far as all of that goes, Quodlibet is pretty much the opposite. Open Source. Written in Python which is a classic Open Source programming language. Anyone in the world can make additions to the main code base or create plugins to add or modify functionality. The public is welcome to look through the list of issues, tackle one or more and submit it to the applications maintainers for merging into the code base.

One thing I’d like to see is a more intuitive and useful interface for seeing and entering specific fields for each track – for example: description, comments, dance notes. You can download what’s called a bundle, which is basically a configured “application”, with an icon you click on and run as with your typical mail or internet browser program. You can “clone” the source code and build your own bundle, or you can do a combination of the two and install (download) the (pre-compiled) “bundle”, but also clone the code base and run the downloaded code base with the downloaded bundle. NOTE: my experience is with Mac/OS X and may differ for someone on a PC.

The way you run the bundle as development environment is by having the bundle run the cloned source code. On this computer you open the terminal program that resides in Applications/Utilities. It’s just a black and white window that puts you fairly directly in touch with the file system and features that run behind all the pretty windows of the GUI, or graphic user interface that some people pronounce, “gooey”. So you “move” into the Applications directory by typing cd /Applications. If you were to type ls you’d see a listing of all the applications that are in that directory (folder).

In the OS X Terminal
In the OS X Terminal

Then type something along the lines of the following:

./QuodLibet.app/Contents/MacOS/run /Users/mikilmer/quodlibet/quodlibet/quodlibet.py

Where the first part is ./ telling your computer this is a script to run, then QuodLibet.app/Contents/MacOS/run is the path to the part of the Quodlibet bundle that runs the application. It’s a “bash” (born again shell) script that looks something like this:

Screen Shot 2016-02-01 at 10.00.46 AM

The last part of the line to run QuodLibet as a development environment, /Users/mikilmer/quodlibet/quodlibet/quodlibet.py is the path to the cloned codebase. I cloned it directly into my “home directory” located at Users/mikilmer.

I can test that I’m running from the cloned code base by adding a little bit of code to the main Quodlibet script referenced above:

def write():

    print('Creating new text file')
    name = raw_input('Enter name of text file: ')+'.txt'  # Name of text file coerced with +.txt

    try:
        file = open(name,'w')   # Trying to create a new file or open one
        file.close()

    except:
        print('Something went wrong! Can\'t tell what?')
        sys.exit(0) # quit Python

write()

And when I run the command above the terminal prints “Creating new text file”, prompts to “Enter name of text file”: Mad Mike iLL, and creates a file called “Mad Mike iLL.txt” (But where?), then opens the player as normal. Ah. there’s the Mad Mike iLL.txt file in the /Applications directory where we actually ran the script from (as opposed to the directory the script/file is in). So we can rm Mad\ Mike\ iLL.txt which removes it from existence. Those back slashes “escape” the spaces to that rm doesn’t think it’s getting more than one argument. We don’t actually desire this “function” in the player so with the magic of Git we’ll revert back to the main code base: git reset --hard HEAD. (If I’m going to be making changes, I’ll want to add another “remote” branch to the git tree to which I am allowed to “push” code changes. This would be a “fork” of the main code base. And we want it to be synced so we always have the latest version, or at least access to it.

To easily open QL this way from wherever in the directly system by simply typing quod in the terminal, I can create an alias:

alias quod="bash /Applications/QuodLibet.app/Contents/MacOS/run /Users/mikekilmer/quodlibet/quodlibet/quodlibet.py"

So now we know that we can play with the Quodlibet code. Let’s see what’s going on with these “plugins” that add various features.

We’d like to have a feature that auto generates a list of each song we have done in a given session. To make a plugin you can create a file in ~/.quodlibet/plugins/. We’ll call ours generatelist.py, and I think it will be of the class EventPlugin. The EventPlugin base class. Hmmm. There are a bunch of EventPlugin subclasses (plugins) (found using grep -rnw '/path/to/quodlibet/'- e "plugin_on_song_ended")
that create an instance method called plugin_on_song_ended and by copying one of them I can see:

from quodlibet.plugins.events import EventPlugin

class GenerateList(EventPlugin):

    PLUGIN_ID = "generatelist"

    PLUGIN_NAME = _("Generate List")

    PLUGIN_DESC = _("Automatically add to playlist for current date."

                "Well for now let's just print song name out.")

    def plugin_on_song_ended(self, song, skipped):

        if song is not None:

            print song

That code printed out an “object reference” to the terminal, at the ending of a track playing. The “object reference” is just a line of code with object name and a hash of numbers. Then I could print out dir(song) which listed all the attributes and methods of the object (I think), one of which was _song. Printing song._song provided this:

    {comment': u'', 'composer': u'delete', 'genre': u'Line', 'tracknumber': u'04', '~mountpoint': '/Volumes/Pensacola International Folk Dance Music Archive', '~#laststarted': 1454363749, '~#skipcount': 2, 'album': u'33-12-07 Macedonian Folk Dances', '~#playcount': 7, '~#bitrate': 128, 'artist': u'Krivo Zensko Oro', '~#length': 153, 'title': u'Krivo Zensko Oro', '~#rating': 1.0, '~#added': 1448030939, '~#lastplayed': 1454274254, '~#mtime': 1100040376.0, '~filename': '/Volumes/Pensacola International Folk Dance Music Archive/SteveCollins/DVD 03/CIFD 33s part 6/Macedonian Folk Dances [LPY 50985]/04 Krivo Zensko Oro.mp3', '~#filesize': 2457600}

So one idea would be to write the values of song._song['title'] for each track that plays through to a text file with current days date, creating a new file if one doesn’t exist. That might be a good first step. Next step would be adding the song to a playlist. Wait – it looks like playlists are simply text files anyway, stored in ~/.quodlibet/playlists.

Success! This plugin creates a playlist of all the tracks that have played through the end on any given day:

    '''
    This plugin creates a playlist of all the tracks that have played through the end on any given day.

    TODO: Add an enable disable button in GUI?
    '''

    from quodlibet.plugins.events import EventPlugin
    import time
    import os
    from os.path import expanduser

    class GenerateList(EventPlugin):
        PLUGIN_ID = "generatelist"
        PLUGIN_NAME = _("Generate List")
        PLUGIN_DESC = _("Automatically add to playlist for current date."
                        "Well for now let's just print song name out.")

        def plugin_on_song_ended(self, song, skipped):
            if song is not None:
                    print song._song['~filename']
                    self.write_to_playlist(song._song['~filename'])

        def write_to_playlist(self, filename):
            today = time.strftime("%F")
            home = expanduser("~")
            name = os.path.join(home, '.quodlibet/playlists', today)
            try:
                file = open(name, 'a')
                file.write(filename+'\n')
                file.close()
            except:
                print("Could not write file name out")

`

More later…

International Folk Dance Archive

Earier this year we received another grant from the Puffin Foundation West for a project to develop the music database we use with Pensacola International Folk Dancers.

We\’ve been using a combination of cassette tapes and a fairly un-maintained iTunes music collection and we had a few objectives for the project.

  1. Digitize cassette collection
  2. Archive some of our collective knowledge about the songs and dances
  3. Ensure that all members of the group have access to the music
  4. I need to check if this was part of the initial grant proposal, but we are also learning to play some of the repertoire as well as transcribing it and codifying with lilypond software.

\"Ciulandra\"There have been many steps in this process. The first thing I did was to speak with Larry Weiner, who is considered to be an expert in the field of International Folk Dance and specifically it\’s archiving. One of the things we discussed were potential archival frameworks. Larry\’s occupation for may years was as a database administrator and he had at one point built, from scratch so to speak, a music archiving system. He advised me against spending so much time developing a system that the archiving itself suffers. He also taught me the term, \”Analysis Paralysis\”, which comes into play because there are so many potential parameters to consider in terms of what information to archive.

Each \”track\” may be named by the dance it accompanies, may have multiple spellings in multiple languages. For each \”track\” there may be a specific archivist to whom it is attributed. There are performers, various music collections (albums) it may be sourced from. There may be images, ratings, composer, etc. So one thing we would need to consider is \”how much information, and which information do we want to archive for each track?\”

Then there is the question of how the tracks will be associated into categories and collections, which ties in with the content of the stored track data.

I\’ll take a moment to mention here that this music is absolutely awesome and inspiring. So raw. So earthy. So tactile. Currently listening to Aino Kchume, Assyrina folk dance, Recorded in 1962 by the Shemiram Assyrian Folklore Groupe in Tehran, Iran. From the album \”Assyrian Folk Dances\”, Folkcraft LP-4. Why don\’t we consider the Iranian culture when considering war with them? But that\’s another (related) topic.

My next step was to explore options for a music archiving platform that would fulfill our needs. Larry recommended I join the East European Folklife Center mailing list, which I believe I was already on. I believe it was through them that I discovered the Wikipedia (yay croud-sourced information and Open Source) comparison of audio player software. It needed to run on an Apple (OSX) computer, which is what the group (and Mad haPPy) have, and needed to be flexible in terms of track information configuration. QuodLibet not only meets those qualifications, but it has longevity (released in 2004), is still being maintained and supported, is open source and is written in Python, which is a computer language I know (to some degree, anyway). Perfect.

\"dupfinderCode\"

Now that our platform was determined (barring unforeseen glitches), it was time to look at our current digital collection and the first thing that needed doing was to get rid of the many many duplicate tracks. This would have been a really grueling manual task, but a cinch with a small python script, which compares the actual digital content of each file and moves all duplicates out into a separate directory.

Now we have a relatively clean version of our digital collection. But we need to get the cassettes digitized. It turns out that the cassettes were recorded from actual vinyl records some of our members have. But most of them are 4 and a half hours drive from here in Tuscaloosa, Alabama. It doesn\’t make a ton of sense to digitize a cassette recording of vinyl when the vinyl is accessible. Driving 9 hours and humping literally hundreds of pounds of vinyl records, while it sounds fun, would be expensive and time-consuming, and being a lazy poet, I looked for a way to avoid it. Guess what? Larry Weiner tells me that there is a group in the Washington DC area that already has a digital database of international folk dance music that we may be able to get our hands on, and lo and behold there\’s a guy who is quite close with a few members of our group that has a copy of it… somewhere.

Steve, who has the copy of the archive, lives a few hours away also and isn\’t quite sure where it is. It takes us months to finally get the music into my hands. But here are like ten thousand old, rare international folk dance music tracks. I\’ve already begun the process of transferring our cassette collection to digital, but let\’s pause and check the database for a previously digitized version of each track first.

\"Duplicate

Well this database is also full of duplicates, including many tracks which include the word \”delete\” in their name. Let\’s see what our dupFinder.py scripts thinks:

python dupFinder.py -t FOLDER_WITH_10000_TRACKS

The track names fly down the terminal screen for a long time. (There\’s a folder called NJ Lame – hey what you saying about New Jersey?) This collection of music comes with small document that contains the acronym CIFD. Does this stand for Charlottesville International Folk Dancers? Wow. That script took like ten minutes to run. Running du -sh FOLDER_WITH_10000_TRACKS on the command line reveals that it is 46 Gigbytes!

Let\’s run the script for real:

python dupFinder.py FOLDER_WITH_10000_TRACKS

Back in 20 minutes… or after a visit to the dentist… Dentist rocks! $15 fee? Sweet!

Our script appears to have removed 15 Gigabytes of duplicate data from the directory, leaving us at 31Gig. The directory of duplicates that was created contains 7.4G worth of data, which probably means that 7.6G were duplicates of duplicates. This was all done in our QNAP backup server, so let\’s now run the script on the Seagate portable 1 terabyte system. Same result. Good.

It appears that there are tracks that ended up in the duplicate/removed track directories that don\’t exist in the main directories, so perhaps our scripts needs to be revisited. In the meantime, we can just search the \”removed\” directories for tracks that can\’t be found in the main collection to see if they exist there. If not – we\’ll import them from cassette or vinyl.

\"playerwindow\"In the meantime we can begin using the music collection and organizing the archive using the Quodlibet player. For each track we can ALT-CLICK the track to \”edit tags\” and add comments and description tags and content. Now we can click in the HEADER section of the main players track list and add columns to display the comments and description for each track. We can create numerous playlists and manage their tracks, and we can configure Quodlibet to search within specified playlists.

\"Quodlibet\'s

This achieved in one of two ways: ~playlists is a flag, which accepts an argument which can be a single or set of playlists. For example: ~playlists=Standards or ~playlists=|(\'Standards\',\'Repertoire\'). This flag can either be placed in the track browser input bar or as a \”Global Filter\” in the preferences.

On our list of future Quodlibet TODO (learn)/feature requests are

  • sub-categorize playlists
  • generate playlists based on plays to automate archiving of selections per session
  • display specified track tags more eloquently
  • Move or copy database/song info collection to portable drive

Listening to our massive collection (over 20,000 tracks) of International Folk Dance music and trying not to be overwhelmed by our ignorance. There’s a lot to learn about music, dance, culture and managing an archive of music. The members of Pensacola International Folk Dancers are giddy with the new influx of inspired energy, not to mention the ease with which we can now make use of the music collection. What used to be a matter of searching through old cassettes and hoping they would play in the old cassette player is now as simple as naming a dance and either playing it or selecting a track from an array of potential versions.  The group has increased regular attendance and as well as session length, thanks in part to the support of Puffin West and as always we Puffins are grateful.

Thanks for the tips

Wanna throw some thanks out there:

Really cool blog that helped me assemble Hymnal.

Chris who turned us on to lilypond engraving library.

Seth Tobocman did the cover art.

These folks sponsored the project personally:

Version:1.0 StartHTML:0000000155 EndHTML:0000001876 StartFragment:0000000475 EndFragment:0000001859

Shavaun Pizar
George Blitch
Geraldine T. Vaurigaud
Boo Reiners
Carla Murray
Alice K Genese
Norman Vazquez
Sylvie Harris
Jennie Spanos
Adrian Fallwell
The Perrin Family
Hans and Marina Frei
Alex Sniderman
Scott Anthony
Stephen A Stetson
Leah N Pietrusiak
Jason Daniels
Georgeanna Presnell

 

And for this project we were also honored with another Puffin Foundation West grant.

Description of R We Done Yet?

This is the original description for R We Done Yet?:

Original Seth Tobocman Design
Seth Tobocman Design

Are We Done Yet? A two person theatre show using (international) folk music and spiritual concepts from around the world to ask a question. As a species, can we be done killing for profit? By embracing the never-ending campaign toward joy, peace, justice and freedom we gain insight into the inexhaustible value of life.

Themes include health, sexuality, commitment, education, confusion, frustration, birth, death and psychic development. The show comines music, ceremony and stand up philosophy to fearlessly honor the unique, sensual mysteries each of us is born to exemplify.

 

46XXXY

Flower pot boys these dolls ain’t toys.
They are used to receive waking dream transmissions.
Plastic horses escape corral,
Crazy-glued horns they’re unicorns now.
Neighborhood pests in the afternoon.
Dynamite mind sets bent for doom.
Prissy boy nickname ricochet battleground,
Everybody rattled out, hide your soft side.
Kid Farside alone if he could
Making all day trips to the west side wood
Walked up on by the eighth grade class
Dancing by the river in a wedding dress.

Estrogen, estrogen. Doctors came and questioned him.
Testing blood and chromosomes, injecting in testosterone.
“Tell us what you want, and we’ll tell you why it’s wrong”,
With the prods and the pokes and the long, hard looks
Hushed tone talks and “honey, take a walk while we
Chart your condition in our clip board books.”
Soon as he was able, headed for the train.
Don’t complain you just grab the reins.
Heard if you can make it, there’s always been a place
In the heart of Fringe City for boys who rock lace.

CHORUS:
Calling for a rule book rewrite
No hard line between girls and boys
It’s a wide spectrum as proven by
Chromosome forty-six triple-X Y

Nest-headed girl rude lip, hard ball
Dirty hands, bloody knees and a broke tooth smile
BMX-ing construction piles. Dares that she barely just survives.
Hits the trail, hits her moon,
Ditch routines that you can’t resume
Made it to the coast with the wits she had
Dying brake pads, thousand mile ride.
Seized-up fried and dies, but all good,
One last stretch she can make by foot.
Work boots stomp stink-weed glass,
Results to deliver in a welding mask.

Nitrogen, glycerine, substitute for estrogen
Super-octane testosterone
Distilled by the mother of the one-eyed crone.
Androgynous secrets under told
Provide the fuel to re-cast the mold.
She is of the ilk that’s wild and bold,
Who’s endeavors have enabled us all to unfold.
Games don’t tell the boys from the girls.
Looks don’t tell the women from the men.
Been that way since the end of the world,
It’ll be that way ’till we start again.

CHORUS

Wake Up

fighting biting out o control
loosin my teeth losin my soul
the old witch knows what i’ll go through
she’s seen the end and she’s telling me to

wake up wake up

ghastly ass attitudes come to bed
every night i go head to head
personification fear and greed
toss me aroun like a tumbleweed

wake up wake up

runnin around with a heart of foam
achin and shakin i only roam
through the ruins of the danger zone
hell in a bathroom i have no home

in this dream i’ve been havin for weeks
the old crone grabs me by the cheek
pulls me close and begins to talk

i’m a beautiful body but nobody will look
i’m a bus full o kids rollin off the dock
i’m accused of murder but i can’t talk
i’m allowed to leave but the door is stuck

i’m ablunt gettin smoked and i can’t wake up
i’m a truck on a hill and i can’t stop
im’ assuming the worst cause it always reverts
and i’m certain that my bubble is about to burst

wake up wake up

mysteruous deats in ya family tree
question marks in ya ancestry
has one o yall ever died peacefully
only ever stories o tragedy

wake up wake up

buckets of blood spilled by
women who give the evil eye
if i’s not a chase it’ll be a fight
hope i never dream tonight

wake up wake up

i’m gonna change these wayward ways
with which i’ve lived for ten thousand days
memory is basically a stressfull haze
runniun from the demons i’m supposed to face

wake up wake up

Words by Rivka and Mike iLL
Music by Mad Happy
Copyright BMI

Truckstop Honeymoon

it ain’t like things changed much once we hit the road
being too broke for a coffee and a roll
that don’t slow you down, just more o-the same stubborn forward march
maxxed out cards and empty bank accounts, only leave a psychic mark
late morning in the parking lot at a place called flying j
nuzzled under the interstate with an all you can eat buffet
aside from the sun, noone cares if we sleep till twelve or one, and
the drive, town-ta-town, helps me forget where i came from
wasn’t raised on a runcible spoon
a year and a day on a truck stop honeymoon

everything we had to spend
we spent on a one way ticket
now we’re down and out, high n’ dry
with nuthin’ but our looks to get us by

if i’m gonna starve tonight
then i wanna starve by your side
drinkin’ truck stop coffee
and i tell you it’s all right
you know that you look gorgeous when ya’ cry

if this is what they call freedom i’ll take it
if this is bad as it gets we’ll make it
every single turn somethin’s testin’ me
fates just jealous cause we’re spendin’
all our time chasin’ signs for destiny

we were nearly half insane and broke down
by the time we left our strange hometowns
living at the edge of bored to death in a place called no surprise
where the only thing that ever changes is what gets you high
I’d already kicked dope and crack and a bad slut habit… twice, at least
and i’d just been replaced by a proper queen at a west side hair salon
if we had what to lose, it was getting old on mary-jane and jack
and away was the only worthwhile place to get.
wasn’t raised on a runcible spoon
a year and a day on a truck stop honeymoon

if i ever knew your strength
if i ever believed in your resolve
to achieve even your wildest dreams
nothing is as dire as it seems

even as the sky turns pink
on another night here at the brink
we’ve got love to make
records to break
nothin’ but romance to fill the hours

if this is what they call freedom i’ll take it
if this is bad as it gets we’ll make it
every single turn somethin’s testin’ me
fates just jealous cause we’re spendin’
all our time chasin’ signs for destiny

Words by Rivka and Mike iLL
Music by Mad Happy
Copyright BMI