Ken's Thoughts

Thursday, September 3, 2009

Fixing Java on Mac Snow Leopard

I have long documented some of the issues and differences for a Java developer on the Mac OS X. Most of that has been on Leopard, OS X 10.5. For quick links the most significant articles are: Java is a 2nd Class Citizen on MAC OSX, Fixing Java Memory Tools on Mac OS X and Java 7 on Mac OS X.

Monday I was faced with a new situation, Snow Leopard OS X 10.6. There are several issues with Java on the new Mac OSX 10.6 "Snow Leopard". This post will outline the known issues and will provide fixes for several of them (or provide links where fixes are already explained).

Known Issues:
  1. Java 5 is not installed
  2. Java 5 symbolic links are set to Java 6
  3. jmap - some aspects of jmap do not function
  4. javascript runtime is still missing
As a consequence, applications that used Java 5 either don't work, or provide a message as shown here from my favorite stock trading application.



* I'm still looking for more issues, so if you can add to the list, please add comments, email or twitter.
** I have tested the latest BTrace and it works fine.


Java 5 on Snow Leopard
Whither you agree with the removal of Java 5 or not, it is unbelievable to me that the Java 5 symbolic links are pointing to the CurrentJDK which points to 1.6. For those new to this space the Java version live under /System/Library/Frameworks/JavaVM.framework/Versions . I would advise you to either:
  1. replace Java 5 using instructions provided at leopard http://wiki.oneswarm.org/index.php/OS_X_10.6_Snow_Leopard
  2. simply delete the java 5 symbolic links
As a side note, I use a script for swapping between JVM versions. I modified the original script to just include 3 important functions when sourced in. I'll add the script to the bottom of this post as a reference. The 3 functions are jvms, setJava, unsetJava.
  • jvms reads through the version directory and reports what it reads of the file system.
  • setJava allows you to type in the name of the version and switches JAVA_HOME and path to this version of the JVM.
  • unsetJava reverts the set to the original java version
JMap on Snow Leopard
My initial execution of jmap on snow leopard failed with the following error message:

kensipe$ jmap 3407
Attaching to process ID 3407, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "heapOopSize" in any of the known library names (-)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:399)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMIntConstants(HotSpotTypeDataBase.java:319)
at sun.jvm.hotspot.HotSpotTypeDataBase.(HotSpotTypeDataBase.java:88)
at sun.jvm.hotspot.MacOSXTypeDataBase.(MacOSXTypeDataBase.java:36)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:578)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:499)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:337)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.PMap.main(PMap.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jmap.JMap.runTool(JMap.java:179)
at sun.tools.jmap.JMap.main(JMap.java:110)
Debugger attached successfully.

The good news is that the debugger attached successfully :)

Information regarding this issue is scarce. It turns out to likely be an issue with the version of Java that is distributed with snow leopard. The JDK 1.6 rev 14 appears (based on non-confirmed web sources) to have this issue. The latest is JDK 1.6 rev 16 should have the fix, however using the apple distro, we are left waiting for Apple for the fix. Although jmap fails with no arguments it does succeed for certain commands such as jmap -histo 3407.
After replacing the JDK 5 with the leopard distro as advertised above and switching to this version with the script above, jmap works fine. However jdk 5 jmap only works against jdk 5 running applications. It fails against java 6.

javascript
As record in a posting from a year ago, called Fixing Java Memory Tools on Mac OS X, the javascript library is still not provided by Apple as part of the Java distro. You will need to following the instructions there to fix it. Unfortunately the Rhino download has disappeared from the Mozilla site, so if you didn't keep a copy, ping me and I will send you want I have. You will also need to duplicate this for each version of Java if you install the Java 5 Leopard package.
The test to see if this is setup correctly for you is: jrunscript -q
If that command only returns AppleScript, then you are missing the javascript engine which is need for a number of tools in the JDK.

After these changes, it appears that Snow Leopard is ready for some hard core Java development. Good Luck and Happy Coding!

Java Switching Script that I use:

# *************************************************************************
# Originally written by Shawn Erickson - shawn at freetimesw.com
#
#
# Last Update: 2008y 08m 8d
#
# The following functions are meant for use with bash shell which is currently the
# default on Mac OS X 10.4 (starting with 10.3 IIRC) unless otherwise configured.
# Good info on why not sym links: http://lists.apple.com/archives/Java-dev/2006/Jan/msg00290.html
#
# from: http://lists.apple.com/archives/Java-dev/2005/Aug/msg00506.html
# http://homepage.mac.com/shawnce/misc/java_functions_bashrc.txt
#
# *************************************************************************

### Prompt ###

export CURRENT_MODE_STRING="";

### Java Environment Functions ###

J_VERSIONS_DIRECTORY="/System/Library/Frameworks/JavaVM.framework/Versions"
J_COMMANDS_SUBPATH="Commands"
J_HOME_SUBPATH="Home"

function availableJVMs()
{
ls -1 $J_VERSIONS_DIRECTORY | grep ^[0-9].[0-9]
}

function jvms()
{
clear
echo "Available JVMs: "$jvms
ls -1 $J_VERSIONS_DIRECTORY | grep ^[0-9].[0-9] # look to remove redundant call
echo " "

echo "Current Java:"
java -version
}

function setJava()
{
local target_jvm=""
local jvms=$(availableJVMs)

# Validate that the user requested an available JVM present on the system

for jvm in $jvms ; do
if [ "$jvm" == "$@" ]; then
target_jvm=$@
fi
done

if [ "$target_jvm" == "" ]; then
echo "Unsupported Java version requested"
return;
fi

# If we get here the user asked for a valid JVM, so configure it

echo "Configuring Shell Environment for Java "$@
export JAVA_VERSION=$@

# First unset any current set java, back to default before doing configuration
_unsetJava

# Generate the paths needed for the JVM requested
local jcmd="${J_VERSIONS_DIRECTORY}/$@/${J_COMMANDS_SUBPATH}"
local jhome="${J_VERSIONS_DIRECTORY}/$@/${J_HOME_SUBPATH}"

# We save the original path so we can toggle back if unset
ORIGINAL_PATH="$PATH"
PATH="$jcmd:${PATH}"

# We save the original JAVA_HOME so we can toggle back if unset
ORIGINAL_JAVA_HOME="$JAVA_HOME"
JAVA_HOME="$jhome"

# Update command prompt mode tag to note JVM setting
CURRENT_MODE_STRING="J$@"

echo "Current Java:"
java -version
}

function _unsetJava()
{
if [ "$CURRENT_MODE_STRING" != "" ]; then
PATH="$ORIGINAL_PATH"
JAVA_HOME="$ORIGINAL_JAVA_HOME"
CURRENT_MODE_STRING=""
fi
}

function unsetJava()
{
echo "Configuring Shell Environment for default Java"
_unsetJava

echo "Current Java:"
java -version
}


Wednesday, August 19, 2009

97 Things Every Project Manager Should Know


The book 97 Things Every Project Manager Should Know put together by Barbee Davis is out, and worth picking up... why the magical 97? Apparently 101 was way over used... or perhaps Barbee couldn't get the last 4 in before it was time to publish :)

This book has some great insights from a number of sources, which is a huge value add to any project manager or senior developer. This is the top advice from experts in the field around team building, running a software project, and generally how to herd cats.

Most books in this space are limited to the experience of the author and his/her circle of influence. The beautiful part of this book, is the fact that Barbee has captured the most interesting and important tid bits stories of success and failures of software projects from a large sample of project managers and developers each providing a view from different industry verticals.

Well done Barbee!

Monday, July 20, 2009

Speaking at JavaZone in Oslo

It's official... I will be heading to Oslo, Norway in September for JavaZone. I will be presenting Debugging your production JVM, which is talk largely focused on BTrace. Also if you didn't realize it yet... BTrace has moved to the kenai site: http://kenai.com/projects/btrace/pages/Home

Monday, June 22, 2009

If you can't be a good speaker, be a groovy speaker


I will be speaking on groovy and grails at the up coming 2GX conference in New Orleans in October. If you are in the groovy space or you are just looking, you don't want to miss this... all the big names will be there. Most of the leading authors and committers in the groovy, grails and griffon space will be there!

How I got to be among them is beyond me... but it sure is groovy!

My topics include security, where the focus will include defense against hacking techniques such as injection flaws and XSS, followed by a walk through of the security plugin and how to secure urls and services. The second session is on Java memory management, the memory demands of a dynamic language, along with tools on debugging Grails applications in production.

Thursday, June 11, 2009

Personal Career Guiding Principals

I have received a number of requests from team members and session attendees for career path guidance. Here is what I threw together... I would enjoy others comments on what makes a difference for them!

My Guiding Principles

1. Become an expert at what is coming...
I've been lucky to live in an area of the country which is conservative when it comes to technology, which can at times be very dissatisfying. I travel to the coast (usually west) at least once per year for a technical conference. Through observation and networking, I learn the passions of the technical industry (not the vendor hype... beware the vendor hype). I make time to become well versed in these spaces. Within 1 year clients are interested and are looking for advice, within 2 they are looking for consultants. This is absolutely necessary to be a good architect anyway.
Suggestion: No one should work in our business without reading the mythical man month. Also read the pragmatic programmer or the quick guide if not the book: http://www.codinghorror.com/blog/files/Pragmatic%20Quick%20Reference.htm

2. Have the final say on matters that affect you...
Each of us is responsible for ourselves. If a company says no to training, or a conference or... that doesn't mean no to me... it means that this activity will not be subsidized. The next question to me is it worth the full obligation. There are plenty of training courses I've not only paid for, but took vacation time to be at. I currently have a nice large LCD screen, a wireless headset for my office phone and I don't use the corporate PC, I bought a MacBook Pro... all of which I paid for. I am generally unwilling to compromise working with the best tools. It shocks me in fact that Chefs bring their own cutlery, mechanics bring their own tools and software developers have better machines at home then they use at work, with an expectation that all their tools are provided for them by some company. In addition to the MacBook, I have invested significant dollars in software. Although I like open source, in many cases it is not the best. So I have TextMate, Intellij, MS Office, Keynote, etc. This wasn't as easy starting out, now frankly some vendors send me licenses now so that I'm showing their tool when I'm demoing. It is important to note that I do not accept licenses even for free if I can't stand behind the tool.
Suggestion: Invest in you! Be a master craftsman. Read: http://blog.objectmentor.com/articles/2009/04/01/master-craftsman-teams

3. Seek out Mentors and Network...
Find people who are great at what they do in an area of interest and seek them out, read their books, articles and blogs. Email them. Network with them. Obviously don't annoy them. Perhaps buy them lunch. Challenge them (for instance, "you said xyz, why would you say that?" or "why wouldn't you say yyy?"). Better yet, see what they are thinking about, what they are working on and what they might need help with. Help them! Pass them some work (if they have time), and ask their opinion.
Suggestion: Go to conferences and network.

4. Build your brand...
Create a brand for yourself. Be the Java Memory guy, the F# guy. Dive deep and market yourself. This means; a) find jobs or opportunities to work in this space, b) blog in this space, c) provide presentations in this space d) write articles or a book in this space. Your digital footprint is very important and it takes time to build out. As an example, if you search my name, the first several pages are me. This is significant. The goal here is to in a position to be hired for a gig without a resume. Those asking for you by name don't need a resume, they know what you represent. Reputation is everything! The great thing about a personal brand is it is more reasonable to change than a business brand... so 2 years as the czar of java, followed by...
Suggestion: Read Groundswell.

5. Know who you are and what makes you special...
This takes time for most people. You have to know what your talents are and what you are not good at. When you know what you lack, find teams / team members to compensate. For instance, I am a knowledge sponge and learn very quickly, but I get bored quickly. I am not as good working with a plain piece of paper, but I can perfect and analyze the dickens out of any architecture, system or code put in front of me. Take some personality tests and understand your personality. It's worth it! In the process of learning you.
Suggestion: Read Brain Rules.

6. Learn and consistently work on soft skills
Don't be the average developer... It is ok to be a geek, but be an alpha geek! Most developers will have one of two paths; PM or architect. Either way, the skills are completely different from what made them great as a developer. In the end, developing software is about enabling the business to a) make money or b) save money. Learn their language.
Suggestion: Read How to win friends and influence people and seven habits of highly effective people.

7. Be a Leader!
Leadership is earned... it isn't a title. For a technical person, It consists of having deep technical skills in at least 1 area combined with good morals. People follow people who know what the heck they are talking about and they trust. Knowledge without morals and people don't trust you. Morals without knowledge and you are a really nice guy. It has to be both.

8. Don't be Afraid:
- to say no (especially to things that don't align with you, know what you want)
- to have an opinion or stance on a subject (as soon as you do someone will shoot arrows at you)
- to change your opinion when the evidence dictates... but not based on pressure or opinion
- to make a decision (even in the absence of information)
- to be criticized
Suggestion: Read Who moved my Cheese.

9. Eat like a Bird and poop like an elephant
This is a wonderful phrase by Guy Kawasaki. Birds eat half their weight in food per day... Elephats poop... well they poop a lot. The idea is to read, listen, consume knowledge... then poop... I mean share that knowledge with anyone and everyone. I have a large library (which I read), and an account with Audible and Audio-Tech Book Summaries.
Suggestion: Read made to stick, and rules for revolutionaries.

10. Passion!
Have some... find it! There is nothing more convincing and more contagious than someone with passion. Change you, then change the world!

11. Have fun...
Life is too short

Tuesday, June 9, 2009

BTrace and JStat

I just finished a JavaOne presentation on Debugging your Production JVM. The killer part and climax of the presentation was on BTrace. BTrace just rocks! As good as it is, the documentation and javadoc information is somewhat dated. This post will explain one of my favorite tool sets working together; that being jstat and Btrace.

JStat
Jstat is a tool that has been provided in the jdk bin directory since Java 5. It has a number features, which can be examined by using the flag -options:
jstat -options
The most significant in my opinion being jstat -gcutils which provides an output of jvm memory by compartment; survivor spaces, eden, old and perm space. Here are the steps if this is new to you:
1. start an demo application: java -jar Java2D.jar
2. get the pid: jps
3. lauch jstat: jstat -gcutil 1483
output:

S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 0.00 10.61 59.98 76.81 8 0.057 9 0.460 0.516
BTrace
Btrace is a tool that allows you to inject probes into a running java process to observe and debug an application. I won't go into "how" to use BTrace, as I've blogged on it in the past and the BTrace document does a good job of explaining it.

The area that is lacking is around the @Export annotation for BTrace scripts. There are no details in the documentation and the sample code comments says "// create a jvmstat counter using @Export"... yeah... right... Even the internal code comments and JavaDocs are just as vague. So if you are lost at this point, jstat use to be called jvmstat and was provided as part of the distribution of jvmstat. Even then there is no information on how to leverage the two... until now!!

Creating the Script
Take a look at the ThreadCounter.java file in the sample directory of BTrace. Any value you want to be exposed to jstat will need to be annotated with @Export. Then you will need to assign a value to this exported value with a static method from BtraceUtils. In the ThreadCounter example this is accomplished with perfLong("btrace.com.sun.btrace.samples.ThreadCounter.count"); The next step is to inject this code into a targeted JVM; such as:
btrace 1483 ../samples/ThreadCounter.java 

Accessing BTrace exports with jstat
The undocumented trick for jstat is that you have to specify -J-Djstat.showUnsupported=true and -name with the name of the exported variable defined by the perf statement. Here is the full command-line:
jstat -J-Djstat.showUnsupported=true -name btrace.com.sun.btrace.samples.ThreadCounter.count 1483




Thanks to Sundar at Sun for the enlightenment!

Sunday, May 3, 2009

Closure: I Don't Think That Word Means What You Think It Means

It is curious that my weakest subject in school was English, and that I struggle so much with the miss use of certain words. Words really do matter. As words are misused, it starts to become commonplace which through a domino effect results in the complete lose of their meaning.

The word most in jeopardy today in the domain of programming is Closure.

WikiPedia has a good definition which states: a closure is a first-class function with free variables. The critical phrase is: free variables. I realize how subtle this is... however without the free variables it is just another function... and IMO it doesn't have to be a function that is just the common mechanism. Adding to the list of good examples is the docs on closures in Javascript.

Examples that are wrong or vague are commonplace. First Example: Groovy, which defines it merely as a code block... or a second example, how about Zdeněk Troníček's blog support Closures in Java which defines it as an anonymous function. A Google search will provide a fairly equal number of good and bad examples.

Defining Closure
The point to be made here is that if you pull the "free variables" out of the definition of a closure than all you have an anonymous function and the two words would be synonymous making one redundant to the other. Put another way... Closures are typically anonymous methods, but not all anonymous methods are closures.

Understanding Variable Scope
When defining a variable, our typical scopes are global, part of a class, parameter to a method, or local to a method. This would be complex if we tried to discuss this for all languages, so I will focus on Java. In Java there is no global scope, this is typically handled as a static member of a class. In this case, it lives in memory in a space called perm space. If the variable is part of a class, then the instance is part of heap memory for an instance of that class. If it is a passed parameter or a local member variable, then the storage is part of the stack frame pushed on the stack. As a reminder, as a method returns the stack frame for that method is popped off the stack and the scope has ended for local variables, references, etc. What is interesting about the proposition of a closure, is we want to be able to create a method (method 1) with some "variables" in the scope of another method (method 2). After the return of method 2, which means everything is out of scope, we still want to be able to pass method 1 to other parts of the program with the state of local variables maintaining state. Perhaps you can understand the issue now. If the scope is lost, where and how is the state of the anonymous method 1 being maintained? This is what closure is, and it is up to the language, compiler or run-time to figure that out.

I prefer to consider closures as another level of scope, as oppose to just another function.

Functional Example
It is easier IMO to see this with a functional language, so here is an example:
Function powerFunctionFactory (int power) {
int pwrFunction(int base) {
return pow(base, power);
}

return pwrFunction;
}

Function sqr = powerFunctionFactory (2);
Function cube = powerFunctionFactory (3));

sqr (3);
cube(3);
Looking at this example when the function sqr is defined by invoking powerFunctionFactory(2), the scope of the value of 2 should be lost through normal scoping rules, however due to the feature of closure it is not and this technique is possible for a number of languages. Not only must it maintain state, but it must do it for each instance of closure. Notice that the cube function has a power state all it's own.

In closing I just ran across this great post by Neal Gafter, which is titled the definition of Closures. I didn't get to it in time to reference his material, which looks great. Perhaps there will be a follow up.

If we can't agree on the nature of closures containing variables... we will never understand monads :)