2009 Goals

My good friend Jeff tagged me to blog my goals for the year. I'm glad he gave me a little inspiration for this as my personal goals are something that's been floating around in my head for some time. With the exception of the first two they are in no order. Here they are, etched in the annals of the internet.

1.) Delight my wife
2.) Be a better father to my 2 girls
3.) Exceed the expectations of every customer I come in contact with
4.) Blog every other week
5.) Speak every quarter at a user group or conference
6.) Contribute to an open source project
7.) Hire 5 people smarter than myself (low barrier, I know)
8.) Exercise 3 times a week
9.) Read 12 books
10.) Brew 4 times

Burning Media from Mac OSX

Repeat after me, "Sissies use GUIs. Sissies use GUIs. Sissies use GUIs."

Okay, now that we have that covered here is the command I continue to forget when trying to burn an iso onto a CD from my macbook pro.

hdiutil burn ~/Downloads/ubuntu-8.10-desktop-i386.iso

Beats fumbling around with Finder any day.

Howdy Parnter!

I'm pleased to announce that Quick Solutions, Inc. is now a certified Systems Integration Partner with SpringSource.

The Spring framework is the standard for Java development. SpringSource has built upon that momentum with a keen vision in developing products and services that benefit enterprise customers using Spring. Whether it is the OSGi enabled dm Server, an instrumented version of the Spring offering with a common monitoring application, or a support model that bests Oracle, IBM, or JBoss, the future of Java development is a very clearly dominated by Spring and SpringSource.

Quick Solutions is driven by providing software based solutions to our customer's problems. We strive to do this in a manner that passes the most value on to our clients. Partnering with SpringSource will allow us to clearly articulate and demonstrate the value that their products generate within Enterprise Java development. We look forward to working closely with Rod and the gang in 2009 and beyond.

Stay tuned for some more information on making best use of the products in the SpringSource umbrella.

Thank You SchemaSpy

I love open source contributors. These people are the geeks who bypass reality shows, online gaming, and time with their families so that they can bring you something that will make your life better. They get nothing in return but a "atta boy/girl" and a pat on the back.

I am taking it upon myself to give thanks to a few of these open source contributors over the coming months. I am a typically just a taker when it comes to open source, but I would like to at least let the givers know, that I appreciate their hard work and commitment.

The first shout out goes to the kind contributors of SchemaSpy. SchemaSpy allows you to generate a web based ERD of an entire database, that is easily navigated by clicking on images throughout the web application. It's a simple jar file that can be downloaded and executed on any platform that has java installed. It supports a wide variety of databases and produces a ton of information in an easy to digest medium. The first thing you'll be presented is the table listing:

This view gives you all of the tables in the system, the number of rows in each table, and a number of other tables directly related via foreign key to this table.

When you click on one of the table names you are presented with the more specific table view:

The top level of information is like a describe of the table, but checking some of the boxes will allow you to see navigable parent and child rows for each of the columns.If you click on the images in the mini-ERD at the bottom of the page, it will navigate to the appropriate tables. Also note that you can expand the mini-ERD to 2 degrees of relationships.

These features in and of themselves are enough reason to automatically document your database with this tool, but another really applicable portion of the tool is the ability to identify anomalies within the database. Clicking on that tab provides you with this view:



Here you'll see a number of bad database smells very quickly. Unleash a DBA worth their weight on this page and you'll have some table alterations done very quickly.

I honestly see no reason to not use SchemaSpy on any project that we have. It is extremely easy to use, can be integrated into a build, and provides better reference documentation on the database than the vast majority of projects I've worked on. Well done SchemaSpy crew, and a sincere THANKS!

Book Review: The Productive Programmer

When I heard that Neal Ford was releasing a book on programmer productivity, I anxiously went to Amazon to pre-order it. Problem was that it was almost two years ago and Neal's busy schedule has kept him from finishing it for quite some time. Now that it is finally out and I've read through it (twice actually), I would highly recommend it to anyone doing software development on a daily basis.

I've seen Neal speak at a number of conferences like Codemash, NoFluffJustStuff, and eRubycon, and I've always walked away with something that will improve me as a software developer. The reason is fairly simple, Neal is at his core a developer, and more than many other speakers, he understands the problems and challenges that face developers.

The The Productive Programmer is really two smaller books slammed together.

The first part is focused on tools and techniques that help you work. I actually re-read the book a couple of times because I did a horrible job of keeping track of the utilities and power toys that I wanted to download my first time through. The second time I went through it I started putting post-its on the pages where I wanted to remember a command or download a utility that made sense. The book was littered with post its by the end of it. Neal provides a number of utilities and techniques for both Mac and Windows platforms that will allow you to work more efficiently in your day to day environment. *nix environments are notably absent here, but the command line advice provided for Macs, are obviously equally useful across any *nix distro.

The second part of the book is largely focused on how you build software. A large majority of Neal's talks at conferences carry this common theme. Neal is adept at identifying and explaining principles and patterns of development that yield the best results. Metaprogramming, TDD, Fluent Interfaces, Polyglot programming and many other topics are covered in an easy to digest manner that harkens back to the bible of software development: The Pragmatic Programmer. Neal is able to provide concrete examples throughout the book, based on his vast experience in consulting through the years.

The productive programmer is easy to read, full of valuable information, and applicable to junior developers and seasoned veterans alike. Again, I would highly recommend it to anyone developing software on a daily basis.

Looking for Smart People

I tend to mention my company, Quick Solutions, very little in order to keep some sense of journalistic integrity with this blog. Unfortunately, I'm not a journalist so I'm going to skip the pretense for at least one post.

I started working at Quick Solutions 5 years ago this month for the sole reason that I was largely unchallenged in my daily job. My thinking at the time was consulting would expose me to a variety of businesses, technologies, and projects. I was fortunate enough to prove my commitment to the company and demonstrate a sufficient breadth of knowledge that I was offered an opportunity to work within the solutions group. At the time I thought that the only difference was the type of work (project versus role based consulting).

I quickly realized however that there was something else about the group that was different. This group was choc-ful-o-geeks. And I mean that in the most sincere way possible. Conversations within the group revolved around continuous integration offerings, Java and .NET trade offs, best practices for SOA, and dynamic language use instead of the typical bitching about a lack of requirements and rehashing of the previous night's reality show. The group was filled not necessarily with Mensa members, but instead with very bright software engineers who cared about their craft. Egos are checked at the door and you could be sure that every person in the door shared a passion and ability to learn that tends to permeate good software developers.

Our company has grown throughout the years and even in a downturn of the economy, we are in dire need of people who meet the above description. If you like working with smart, passionate people please drop me a line. We are always looking to bring people in who show a passion for technology, an ability to learn, and a nack for solving problems. Additionally, if your specialty is Java development, I can give you your choice of 3 different, exciting projects, right now.

If you think you meet these criteria and are interested in working with more people of your ilk, email me at tkaufman at quicksolutions.com.

Clustered Scheduling with Spring and Quartz

I initially cut my teeth as a Java programmer writing some batch JDBC programs to update various sets of data. We were deployed in a Unix environment so we traditionally wrapped all of the JDBC programs in a bash shell script and kicked that thing off via a Cron entry. This environment was successful for the most part but there were issues. If our batch machine was taken offline due to upgrades, failed disk, or coffee spill then our batch process just did not run. If we wanted it to run after the fact we had to create another 1 time cron entry to kick the thing off. This typically took about 2 hours because the first hour and a half was spent with me trying to decrypt the cron syntax (minutes first or seconds? need a question mark here but star there? comma or dash between my minutes?).

Fast forward to 2008 and I have just completed a week long adventure with a coworker finally getting Spring and Quartz up and running to kick off some batch programs in a clustered environment. Failover is automatic, ad hoc runs are possible, and I still wind up spending an hour and a half each time I have to add a cron entry. I thought I would post some code snippets with explanation so that you kind reader, could maybe trim this process down to about a day.

1.) Download Spring 2.5.5 and only use the Quartz 1.6.1 RC1 jar that is bundled within the lib directory of it. DO NOT USE A PREVIOUS VERSION OF QUARTZ.

2.) Execute your appropriate database script. They can be found in the quartz distro under the docs\dbTables subdirectory. Make sure that indexes are setup as outlined here.

3.) Wrap a batch processing service with some form of Quartz. Here's the wrapper we used:

public class GenericQuartzJob extends QuartzJobBean
{
    protected Logger logger = new Logger(getClass(), Constants.LOGGER_APP_NAME);

    private String batchProcessorName;
    
    public String getBatchProcessorName() {
        return batchProcessorName;
    }
    
    public void setBatchProcessorName(String name) {
        this.batchProcessorName = name;
    }
    
    protected void executeInternal(JobExecutionContext jobCtx) throws JobExecutionException
    {
        try {
            SchedulerContext schedCtx = jobCtx.getScheduler().getContext();
            ApplicationContext appCtx = 
                (ApplicationContext) schedCtx.get("applicationContext");
            IBatchProcessor proc = (IBatchProcessor) appCtx.getBean(batchProcessorName);
            proc.invoke();            
        }
        catch (Exception ex) {
            logger.error("Unable to complete execution of " + batchProcessorName, ex);
            throw new JobExecutionException("Unable to execute batch job: " + batchProcessorName, ex);
        }
    }
} 

With that wrapper, you can execute any batch processor that is wired up in Spring, that implements the homegrown IBatchProcessor interface (which typically only has some variant of an execute or invoke method). You don't need to manage the dependencies of those batch processes as they themselves are just beans defined a Spring app context somewhere. Additionally, the base class jumps through the various contexts that you must navigate to get a properly wired bean from the Spring factory.

4.) Configure the Job Detail in your Spring app context with an xml snippet resembling this:

    <bean id="someJobDetail" class="org.springframework.scheduling.quartz.JobDetailBean">
        <property name="jobClass" value="com.some-company.batch.GenericQuartzJob" />
        <property name="jobDataAsMap">
            <map>
                <entry key="batchProcessorName" value="SomeJobBean" />
            </map>
        </property>
    </bean>

This definition creates an instance of GenericQuartzJob (which fulfills the contract required by JobDetailBean), and plugs in the bean name of a batch processor defined somewhere else in the app context with all of it's necessary dependencies.

5.) Configure the Trigger in your Spring app context. A simple cron based version would look like this:

    <bean id="someCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean">
        <property name="jobDetail" ref="someJobDetail" />
        <!-- Cron expression runs at 1am and 1pm -->
        <property name="cronExpression" value="0 0 1,13 * * ?"/>
    </bean>

And yes, it did take me an hour and a half to get that cron syntax working correctly. Some habits die hard.

6.) Configure the SchedulerFactoryBean

    <bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean" lazy-init="false">
     <property name="applicationContextSchedulerContextKey" value="applicationContext" />
        <property name="dataSource" ref="qtzTxDataSource"/>
        <property name="transactionManager" ref="transactionManager"/>
        <property name="overwriteExistingJobs" value="true"/>
        <property name="autoStartup" value="true" />
        <property name="triggers">
            <list>
                <ref bean="someCronTrigger" />
            </list>
        </property>
        <property name="quartzProperties">
            <props>
                <prop key="org.quartz.scheduler.instanceName">SomeBatchScheduler</prop>
                <prop key="org.quartz.scheduler.instanceId">AUTO</prop>
                <prop key="org.quartz.jobStore.misfireThreshold">60000</prop>
                <prop key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreTX</prop>
                <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.oracle.weblogic.WebLogicOracleDelegate</prop>
                <prop key="org.quartz.jobStore.tablePrefix">qrtz_</prop>
                <prop key="org.quartz.jobStore.isClustered">true</prop>
                <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
                <prop key="org.quartz.threadPool.threadCount">25</prop>
                <prop key="org.quartz.threadPool.threadPriority">5</prop>
            </props>
      </property>
    </bean>

That configuration while lengthy does a few very important things. Let's go through them. The applicationContextSchedulerContextKey property will ensure that the app context is available to all of those instances of the GenericQuartzJob wrappers. The dataSource and transaction manager are necessary in order to ensure that the database is updated in a safe manner (many server instances may be updating the database at once). Their bean definitions are pretty much typical for Spring, checkout the Spring reference docs if you need more info there. OverwriteExistingJobs will make sure that every time the scheduler is started it will use the list of triggers found internally to overwrite any existing ones that may have been changed in the database. AutoStartup makes sense, but I'm not entirely sure it's necessary.

The quartz properties can be maintained in a separate file, but I prefer they are inline with the scheduler definition. They are all pretty much self explanatory, and explained in greater detail, but nested deeply in the Quartz docs.

Hopefully this article will advance you to the point of configuring a clustered, database backed, scheduling system within hours instead of days. This configuration will get you a set of spring enabled batch processes that have dependencies wired as normal, with the added benefit that their schedule is persisted and fault tolerant across nodes in the cluster. This also leaves the door open for a couple of methods of doing ad hoc runs of the jobs. We'll cover that in a part 2 article shortly.