Gigism

Finally understanding BigTable/HBase

2009-07-30T19:35:00.000-07:00

A month or two ago I started evaluating Hadoop, primarily HDFS. That was when I caught up to where people were 2+ years ago. I knew the Google papers were out, but I didn't pay much attention to them. Now, I'm back using Hadoop (I'm in the process of creating a development cluster, 4 nodes total - baby steps) and trying to understand the sparse multidimensional map approach to data storage (primarily HBase - but it would seem to also apply to BigTable, Cassandra, Dynamo, and Voldemort). I think this wiki entry from Jim Wilson has put more over the top - I think I actually get it.

SVN - you amuse me....

2008-11-13T16:00:00.000-08:00

After getting knee-deep in Git, I've struggled with questions about whether branching & merging with Git is truly better than SVN (well, we know it is because Mr. Torvalds' said so). My current project is something I had been planning on moving to Git. However, I didn't think since it's just me (a single developer) that the distributed qualities of Git would be enough to warrant the intellectual burden the next developer would have learning it. I had always planned on keeping a version of the codebase in SVN (via the bi-directional git-svn application). But a friend at OOPSLA swayed me to learn at just doing the branching & tagging with SVN.

I mean, I haven't experienced the pain of merging w/ SVN firsthand - so why not?

Here's one reason why: SVN has zero idea about branches.

A branch is a branch to you - but to SVN, it's just a copy operation.

How do you know when you branches off of the trunk (or another branch)? Well, you just look for the last copy operation! YUP!

$ svn log --verbose --stop-on-copy \
          http://svn.example.com/repos/calc/branches/my-calc-branch
…
------------------------------------------------------------------------
r341 | user | 2002-11-03 15:27:56 -0600 (Thu, 07 Nov 2002) | 2 lines
Changed paths:
   A /calc/branches/my-calc-branch (from /calc/trunk:340)

So - that's interesting. Yet, I can sort of accept it (though I feel like an idiot doing so). The kicker is this.... when you do a merge, "how do you know what you've merge?" Good question!

Say you have a long-life feature branch. You've done some development and merged the changes into the trunk (or mainline). Now, you're continuing to do development. How will you know where to merge from?

Interestingly enough, you don't know - unless you follow the branching/merging "best practice" of including your revision numbers for the branch when committing the merge (I'll let that sink in).

Now - I believe that you have full history with Git. So I need to go back now and see if that's true. But I believe you can even look at the merges with something like gitk and you can see the history. So there is no looking for the last copy operation or checking commit messages to see what revision to start merging from.

Oh - I am reading an older version of the SVN book (because we are using an old, out-of-date version of SVN). Perhaps there is a better way to do it now. But I just wanted to share my amusement.

source: http://svnbook.red-bean.com/en/1.2/svn-book.html#svn.branchmerge.commonuses.wholebr

Interesting String behavior in Groovy

2008-08-24T20:01:00.000-07:00

I've been writing some Groovy code in the last week. I did something silly that the Java compiler would have caught and scolding me for doing:

def body
// ....
body += someContent

What happened was my body strings were all prefixed with the character string "null." Apparently doing a concatenation on a null String causes "null" to be produced.

I do not claim this to be good or bad - I just think it's interesting.

getResourceAsStream() depends on what you're calling it on...

2008-08-24T19:52:00.000-07:00

In writing some unit tests I ran into something I didn't expect. I had a .properties file that I was using to state metadata. In this case, the full path to a file my "Class Unit Test" was creating.

It seems that calling getResourceAsStream() off of getClass() does not return the resource. It's unfortunately that the Class class has a getResourceAsStream() method if it's not going to actually work.

So...

InputStream input = getClass().getResourceAsStream("./test.properties")

returns null.

If you use ResourceBundle to fetch "test" - it works.

I thought that was odd. I made a mention to my friend and he suggested that I need to be asking the ClassLoader to fetch the resource, not the class.

So...

InputStream input = getClass().getClassLoader().getResourceAsStream("./test.properties")

returns the resource.

Now - I am executing this from a GroovyTestCase instance. I do not know the reason for this. But either way, it's frustrating.

Install Git/Gitosis on a server (Debian) - Part 2

2008-08-24T19:24:00.000-07:00

So I ran into some more issues before getting my Git hosting (via Gitosis) all working.

The first hurdle was getting past an error thrown when attempting to do a `git clone` of the gitosis-admin repository. The error was being thrown by os.py (remember that gitosis-admin is python based). The error message looked like this:

Traceback (most recent call last):

File ”/usr/bin/gitosis-serve”, line 7, in ?
sys.exit( File ”/usr/lib/python2.4/site-packages/gitosis-0.2-py2.4.egg/gitosis/app.py”, line 24, in run return app.main()
File ”/usr/lib/python2.4/site-packages/gitosis-0.2-py2.4.egg/gitosis/app.py”, line 38, in main self.handle_args(parser, cfg, options, args)
File ”/usr/lib/python2.4/site-packages/gitosis-0.2-py2.4.egg/gitosis/serve.py”, line 205, in handle_args os.execvp(‘git-shell’, [‘git-shell’, ‘-c’, newcmd])
File ”/usr/lib/python2.4/os.py”, line 341, in execvp _execvpe(file, args) File ”/usr/lib/python2.4/os.py”, line 379, in _execvpe func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The root cause of this is that the executable `git-shell` is missing from your install directory (be /usr/bin or /usr/local/bin). I've done a install locally and all the 140-some executables were places in my install directory (/usr/bin).

Originally, I just copied `git-shell` into /usr/bin. That's hack-ish, what I ended up doing was creating a symlink to `git-shell` from where I have git (in my case, /opt/git-1.6.0, with a symlink to it named /opt/git).

So now we can do our clone of the repository:

`git clone git@YOUR_SERVER:gitosis-admin.git`

This appears in both tutorials I looked at [urban puddle] [scie.nti.st]

Now we want to create a new repository. That was my next hurdle; and you won't need to overcome it if you follow directions. Notably, the simple directions in either. Which is that you need to modify the gitosis.conf (which is Step 6 in the Urban Puddle tutorial and "Creating new repositories" in the scie.nti.st tutorial).

I attempted to simply do something like initialize a git repository, do some commits, add a remote (origin) to the server, and then push to that remote (origin).

If you have gone awry - like me - you'll run into the following error:

ERROR:gitosis.serve.main:Repository read access denied
fatal: The remote end hung up unexpectedly

If you add the following under [gitosis] in the gitosis.conf:

loglevel = DEBUG

You'll get some helpful debug output.

lenards@deedee:~/devel/cs453/project$ git push origin master:refs/heads/master
DEBUG:gitosis.serve.main:Got command "git-receive-pack 'cs453-project.git'"
DEBUG:gitosis.access.haveAccess:Access check for 'lenards@deedee' as 'writable' on 'cs453-project.git'...
DEBUG:gitosis.access.haveAccess:Stripping .git suffix from 'cs453-project.git', new value 'cs453-project'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'gitosis-admin'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'cs453-project'
DEBUG:gitosis.access.haveAccess:Access check for 'lenards@deedee' as 'writeable' on 'cs453-project.git'...
DEBUG:gitosis.access.haveAccess:Stripping .git suffix from 'cs453-project.git', new value 'cs453-project'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'gitosis-admin'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'cs453-project'
DEBUG:gitosis.access.haveAccess:Access check for 'lenards@deedee' as 'readonly' on 'cs453-project.git'...
DEBUG:gitosis.access.haveAccess:Stripping .git suffix from 'cs453-project.git', new value 'cs453-project'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'gitosis-admin'
DEBUG:gitosis.group.getMembership:found 'lenards@deedee' in 'cs453-project'
ERROR:gitosis.serve.main:Repository read access denied
fatal: The remote end hung up unexpectedly

I missed the step in the directions about adding a new group to the gitosis.conf, which is why when doing your `git push origin master:refs/heads/master` it complains about not have readonly access. There is *no* group to access, so you don't have writable or readonly access.

After doing Step 6 from Urban Puddle tutorial - I was able to push my new repository for my compiler project.

I hope this helps someone.

Andy

Installing Git on a server (Debian)

2008-08-21T09:45:00.000-07:00

http://www.urbanpuddle.com/articles/2008/07/11/installing-git-on-a-server-ubuntu-or-debian

First off - a thank you to Vincent for pulling all these things together.

I ran into an issue that seem to be related to the version of the git-core package (I'm running debian etch stable). The version of git-core is 1.4.4, and apparently they had not introduced `git-init` until 1.5.* - so my calls to `gitosis-init` were erroring out (like this gentlemen's: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473908).

So - I just did an install from source for git-1.6.0 ... there are instructions out there, so I bother pointing to those.

Thank again! I really did appreciate using this.

The only thing I'd mention after that is the call to `gitosis-init` looks like this for debian:

su - git -c gitosis-init < /home/joeuser/.ssh/authorized_keys

I tried to post this as a comment on Vincent's blog but I kept getting Apache errors.

Late again...

2008-08-14T07:50:00.000-07:00

I've missed my self-made deadline again. I had a horrible time trying to get my initial site done with the toolset I use daily (which is depressing, and makes me think about the toolset I use).

Right now, I'm back to looking at Grails. It still have issues with getting things to work in Grails. There is something I'm missing, or it's just not clear that how I approach things is wrong. But I've decided to invest time in learning more about Grails and Groovy for the time being.

I've also been learning as much as possible about Git. I recently gave a talk at the local Java Users Group about Git. It seems to be definite improvement over Subversion. And if you're thinking about all those posts ripping on Git's implementation or internal design, being a user at the porcelain I honestly don't care how it's implemented underneath.

Plugging on ... we'll see if I can post more than every four months.

Beta Release - Deadline June 4, 2008

2008-04-11T11:52:00.000-07:00

I'm hoping to have a beta release of the gigism site up by June 4, 2008. The beta will likely have a feature-set of a a central syndication feed (in RSS, with the potential for Atom support) along with a venue listing and some "editor choice" syndication feeds. I've way behind - if you consider I was hoping to have something in place in October. I've been rather business with work, consulting, and just attempting to stay fit. I recently read a book called "Finding Flow" - and decided that coding on Gigism was good for my soul, overall.

I'm gunning for an alpha release in the middle of May.

Gigism Redux

2008-04-11T11:40:00.001-07:00

So I failed horrible at getting an alpha version of Gigism running with Grails. It wasn't because of the programming language or web framework (Groovy and Grails, in this case). It was mainly because I was mentally fried and not in a good spot to be learning a new language and framework when I really had not be exposed to the object relational mapping that I attempting to make use.

I've spent the last three weeks reading, playing, and coding object relationships with Hibernate now. I think I understand more of the principles that were at play with GORM (the Groovey Object Relational Mapping API). But I've decided to change the overall foundation of Gigism and go with something that I'm much more familiar with: Tapestry.

Well, not just Tapestry. I'm really attempting to mimic the application setup that I use on a daily basis at my current job (Tapestry with Spring and Hibernate).

Since making the decision to go down the familiar route, I've managed to get an initial database schema in place along with building a good alpha data-layer with Hibernate (using annotations). I've got some unit-tests in place, and I'm rather happy with the progress in that area. I've also bogarted a template/layout for the initial site. I'm not really a wizard with the ol' HTML/CSS - so having a good, attractive setting point is nice (and it appears that there is no shortage of free layouts on the web - many of which are released under Creative Commons).

Along with unit-testing my code, I've started to play with Selenium RC to get testing in place for the client interface. I'm hoping that it will allow me to deliver something a bit more stable (or at least stable early on). Coming from the .NET side of the development fence, I'm not an avid tester (like many in the Java community). I guess this is an attempt at coming "test infected." Oh, and I am looking at using Groovy for my unit-tests. The initial stuff I have is just regular Java with JUnit - but I would like to use Groovy when coming tests just because of the terseness of the code. I really do like the language overall, I just found myself in over my head when trying to do a site.

Gigism... a vision for the quest

2007-08-04T12:16:00.000-07:00

So you go to that show, that gig, you've been waiting for and before the bands go on you bump into friend. They drop the news that your favorite band just rolled through town and you missed it.

Now that sucks! You weren't paying attention; or, perhaps, you forgot to go out to the website for each venue in town (say, there's maybe 7+ active venues in Tucson) and scroll through the upcoming shows. It's a sinking feeling - sure, you'll get over it. But what if the band breaks upon returning home (like Murder City Devils and At The Drive-in did after a tour together). Or the band never makes it back to your hometown. Or the lead singer DO'd and dies. Or one of the guys kills themselves. Or that was the last show before they started writing shitty music. There's a ton of things that can happen to generate intense regret. And ya, regret is bad - but instead becoming better at dealing with regret I'd rather not have to miss said gigs.

The above example is caused more often when you're not looking - something cool happens and you miss it. I've also experienced the Gig-Doppler effect - you see the show coming, but things get business in your life and you hear it after it's already passed you.

This is why I wanted to do 'gigism.' The goal? Create a common, easily consumable master-listing of all shows in town (Tucson). Now, that's casting a large net! So let's narrow it a bit - a master-listing of all shows from touring-bands coming to town.

How do you create such a thing? Well - find some technology that let's you aggregate data efficiently. And, in my case, be able to program.

I got into the idea of aggregating data with feeds (Live Bookmarks, RSS, Atom, whatever you want to call it) after google started their google personalized homepage (which is now going by the lame-name igoogle). I was in San Francisco staying at my friend Stump's and he had a sweet homepage with all this news and information - right there, and automatically updated via ajax.

And you have it - gigism... the motto I picked when pitching the idea to my friends was "no gossip, no photos - just gig info."

The Holy GRAILS

2007-07-30T18:32:00.000-07:00

Last year at OOPSLA in Portland I picked up the Pragmatic Rails book. Sure, I fell in love with the quick ways that I could rough out functionality (check out the weblog in 15 minutes - a video of Rails in action). But, I was far from a proficient Ruby programmer. Anyway, I liked the idea - but I didn't want to "move there" until I was a better Ruby programmer.

Over the weekend, I went to No Fluff, Just Stuff in Phoenix and was introduced the Groovy programming language (a gateway dynamic programming language was C/C#/Java programmers, like myself). Even better? There's Grails - like Rails, but with Groovy as the programming language (there are some differences, but it's not worth me getting into here). I actually stayed up until 2am that night playing with Groovy - and dreaming of how much easier doing gigism will be in this paradigm.

Ah, so what is gigism? Well, it's a term that I hope gets popular - and something I'll leave to another post.