Accessing Linux File Systems from within Windows

Near the end of 2009 I ask a question on superuser.com about accessing a Linux XFS filesystem from within Windows. After investigating coLinux I decided to try the VirtualBox route -- and it worked beautifully! I can now access my standard Linux install (which I dual boot into) from within Windows on those few days where I need to be in Windows.

System Configuration

Machine: Windows 7 and Gentoo Linux. Windows is acting as the boot loader. Here's my drive layout:

/dev/sda1 - NTFS - Windows 7
/dev/sda2 - Extended Partition
/dev/sda5 - Boot Partition (EXT3)
/dev/sda6 - Swap
/dev/sda7 - LVM
/dev/sda8 - LVM

The Windows boot loader needs a copy of the boot record to be used. In my case, I had installed grub into /dev/sda5, my boot partition. I copied the boot loader out of /dev/sda5 using dd which makes it possible to copy the first 512 bytes of the partition:

dd if=/dev/sda5 of=/root/sda5_mbr bs=512 count=1

I then copied /root/sda5_mbr over to a thumb drive so it was available on Windows. We will need this file not only for the Windows boot loader, but also for VirtualBox as you'll see shortly.

My Linux machine is publicly accessible on the internet and I have, as a requirement, that it still be accessible on the internet when I'm in Windows with VirtualBox running. Given that requirement, here's my networking configuration:

Windows 7 Host
- 192.168.1.20 - static IP

Linux Host:
- 192.168.1.10 - static IP

Router:
- 192.168.1.1 - primary gateway, forwards desirable ports to 192.168.1.10

I've purposefully given Windows and Linux different static IP's so each is accessible and they don't run into any IP address conflicts.

Install VirtualBox

First, I installed VirtualBox in Windows. Most application installs are pretty much the same in Windows, and VirtualBox was no different, so I'm not going to document installing VirtualBox. If you're reading this, I'm sure you're more than capable.

Configure VirtualBox Hard Drive Access

Like many applications, VirtualBox has some standard directories that it uses:

%USERDIR%\%USERNAME%\.VirtualBox\HardDisks
%USERDIR%\%USERNAME%\.VirtualBox\Machines

%USERDIR% is likely C:\Users and %USERNAME% is the username of the account that you typically log in as.

You can create the above directories by hand if you like, but I'm going to assume they exist. With those directories in hand, we now need to tell VirtualBox about the raw partitions that we want it to use. Since Windows is our host, we do not want to accidentally access it from within Linux as that could cause corruption. Furthermore, we want to use the grub boot loader so we don't have to bother with the standard Windows MBR. Since we're giving VirtualBox access to the physical drive, we need to be a full administrator. So, run cmd as an administrator. Now let's make VirtualBox aware of the rest of our partitions:

C:\>cd \users\kpederson\.virtualbox\harddisks
C:\Users\kpederson\.VirtualBox\HardDisks>"\Program Files\Oracle\VirtualBox\VBoxManage" \
  internalcommands createrawvmdk -filename rawdisk.vmdk -rawdisk \\.\PhysicalDrive0 \
  -partitions 5,6,7,8 -mbr f:\sda5_mbr -register

The above creates two files, rawdisk.vmdk and rawdisk-pt.vmdk:

-r-------- 2 root root     161280 Sep  1 22:19 rawdisk-pt.vmdk
-r-------- 2 root root        994 Sep  2 00:16 rawdisk.vmdk

The files aren't particularly big. They do little more than tell VirtualBox about the partitions that it can access and embed the MBR. Here's some details about the command:

"\Program Files\Oracle\VirtualBox\VBoxManage" - the command we're calling
internalcommands - the main command we're calling
createrawvmdk - the sub command - create a raw virtual disk file
-filename rawdisk.vmdk - specify name of the virtual disk to be created
-rawdisk \\.\PhysicalDrive0 - creating a raw disk and using the main physical drive
-partitions 5,6,7,8 - only give VirtualBox access to the Linux partitions
-mbr f:\sda5_mbr - install a new MBR, the one I copied onto my thumb drive (f:\)
-register - add the drive virtual media manager

If the above command worked successfully, we should see output similar to the following:

Oracle VM VirtualBox Command Line Management Interface Version 3.2.8
(C) 2005-2010 Oracle Corporation
All rights reserved.

RAW host disk access VMDK file rawdisk.vmdk created successfully.

Create your Virtual Machine

Now that VirtualBox knows about my raw Linux partitions and is setup to use the MBR containing grub, I can now configure a Virtual Machine within VirtualBox. Here's the basic steps:

1. Choose a name and the correct version (note 64-bit option)
2. Select the amount of available RAM to dedicate to the VM
3. Make sure "Boot Hard Disk" is checked and select use existing hard disk: FIXME: insert image here
4. Review settings and select Finish

The system should now boot up when choose start. Hopefully you'll be greeted by a the grub startup screen:

Configure Networking

Most of the configuration is pretty straightforward, but my networking requirements will take a little effort to get right. First, let me show you the obvious correct answer and my first attempt -- bridged networking. Bridge networking should allow the virtual network card presented to Linux by VirtualBox to send out any data it wants on the physical device as if the computer was assigned both IP addresses (192.168.1.10 and 192.168.1.20):

Unfortunately, that didn't work -- I had network access, but it was only local. Windows 7 could communicate with my virtual machine and my virtual machine could communicate with Windows but my virtual machine didn't have internet access and couldn't resolve DNS. I tried for quite a while and couldn't figure out the problem in a reasonable time (I assume it's a bug). However, I found a work around thanks to a post that on the VirtualBox forum that I can no longer find. Instead of letting VirtualBox create the bridge, use Windows to bridge the connections. To do this, we have to configure our virtual machine to use host-only networking:

With host-only networking configured, we'll now boot into Linux. Once up and running, lets bridge the networks. Pull up the network interfaces dialog and select your standard network adapter and the host-only network adapter, then right click and bridge the connection as illustrated below:

That does bring some slight complications, namely, I have to disable the bridge in order to retain internet access when my virtual machine isn't running.

Since the new bridge is in place, you should now be able to resolve network addresses. Here's a quick test using dig:

Configure Samba

Assuming your Linux distribution installs a working samba, the only thing you should need to do is to edit smb.conf to include a share. Here's my most frequently used share definition:

[shared]
   comment = Shared Documents
   path = /home/shared
   force group = smb_users
   guest ok = no
   writable = yes

With that share in place you can restart samba (or have it re-read the configuration file using smbcontrol) to enable the share. And with that, you should be able to access that share from within Windows using your ip address or netbios name.

QMetaEnum Magic - Serializing C++ Enums - Take 2

A few posts ago I described two methods of serializing C++ enums. Of these, method 2 serialized the Qt::Key enum. The approach, however, relied on some behind the scenes magic that I wasn't fully aware of nor did I fully document. This approach remedies that and describes in full the requirements for serializing C++ enums.

Method 2 - Reevaluated

The goal for this method is to take an enum called MyKey in the MyNS namespace and serialize it. In brief, the code for that looks like the following:

namespace MyNS
{
    enum MyKey {
        MyKey_Return = 0,
        MyKey_Enter = 1
        };
}

Since we want Qt to be able to serialize the above enum, it needs to know about the enum, so we're going to need Q_ENUMS(MyKey). But unless moc sees a Q_OBJECT or Q_GADGET macro, our Q_ENUMS macro will result in a compilation error. To get around this, we need to convince moc to process the file as if it were a class. We can do that by adding some preprocessor defines:

#ifndef Q_MOC_RUN
namespace MyNS
#else
class MyNS
#endif
{
#if defined(Q_MOC_RUN)
    Q_GADGET
    Q_ENUMS(MyKey)
public:
#endif
    enum MyKey {
        MyKey_Return = 0,
        MyKey_Enter = 1
    };
}

At this point we've convinced moc to look at and process the file, but that alone isn't enough. The code that moc generates assumes that a const staticMetaObject has been declared, but at this point one hasn't been declared. Although some compilers will let us get away with this, we'll declare it as follows:

    // ... continuing at the enum
    enum MyKey {
        MyKey_Return = 0,
        MyKey_Enter = 1
    };
    extern const QMetaObject staticMetaObject;
}

With that in place, we're ready to serialize the enum. The first step is to get a copy of the QMetaEnum object. We do so by accessing the static reference directly and then calling indexOfEnumerator to get the appropriate index:

    // get the QMetaEnum object
    const QMetaObject &mo = MyNS::staticMetaObject;
    int enum_index = mo.indexOfEnumerator("MyKey");
    QMetaEnum metaEnum = mo.enumerator(enum_index);

With the QMetaEnum instance in hand, we can now serialize the enum as demonstrated in my prior post:

    // convert to a string
    MyNS::MyKey key = MyNS::MyKey_Return;
    QByteArray str = metaEnum.valueToKey(key);
    qDebug() << "Value as str:" << str;

    // convert from a string
    int value = metaEnum.keyToValue("MyKey_Enter");
    key = static_cast(value);
    qDebug() << "key is MyKey_Enter? : " << (key == MyNS::MyKey_Enter);

With all the above in place, we've used a bit of magic to trick moc into thinking MyNS was a class. This causes moc to generate a MyNS::staticMetaObject instance and store the necessary serialization meta data. With everything in place, we get the following output:

Value as str: "MyKey_Return" 
key is MyKey_Enter? :  true 

References:

• "Secrets of Qt Full" developer days presentation
• src/tools/moc/generator.cpp:340
• src/corelib/global/qnamespace.h:53,80

Learning Ruby Symbols

Ruby has at least a couple of different ways of referencing variables. Class instance variables can be created with the @ prefix, like @my_var = "test", there's also an @@ prefix for class-level variables.

For the first few days I was learning Ruby and Ruby on Rails, I kept stumbling upon "variables" like :name. But when I tried to assign one, I quickly discovered that it wasn't a variable -- it was a symbol. My attempt resulted in a failure:

irb(main):001:0> :test = 4
SyntaxError: compile error
(irb):1: syntax error, unexpected '=', expecting $end
:test = 4
       ^
        from (irb):1

The ruby compiler didn't expect anything after the '=', and it complains.

What are Symbols?

According to the Symbol documentation:

Symbol objects represent names and some strings inside the Ruby interpreter. They are generated using the :name and :"string" literals syntax, and by the various to_sym methods. The same Symbol object will be created for a given name or string for the duration of a program's execution, regardless of the context or meaning of that name. Thus if Fred is a constant in one context, a method in another, and a class in a third, the Symbol :Fred will be the same object in all three contexts.

A symbol acts as an alias to a name such as a variable, class, or method. As a simple example, consider the following two hashes:

u1 = { "name" => "Kaleb", "phone" => "private" }
u2 = { "name" => "Larry", "phone" => "private" }

And here's a similarly structured hash using symbols:

u1 = { :name => "Kaleb", :phone => "private" }
u2 = { :name => "Larry", :phone => "private" }

For each hash the compiler keeps copies of the keys "name" and "phone". For a couple of values this doesn't matter, but for thousands of records it adds up to a lot of duplication and wasted memory. A post on glu.ttono.us provides more concrete details on both symbols and memory usage differences.

Other Resources on Symbols

• 13 Ways of Looking at a Ruby Symbol
• The Symbol Documentation
• Google

rspec: undefined method route_for error

I'm now in the process of learning Ruby on Rails and its friends, like rspec. Firmly believing in TDD, I jumped on the rpsec bandwagon as soon as I could. The second thing I started to test was my routes. My test looked something like this:

  describe "should create :page param for / url" do
    route_for(:controller => 'pages', :action => 'index').should == "/"
  end

But sadly this resulted in the following error:

pages_controller_routing_spec.rb:14: undefined method `route_for' for # (NoMethodError)

After some googling I found a thread on the rspec mailing list that revealed a few different causes:

1. spec/controllers in their path (i.e. spec/controllers/foo_controller_spec)
2. describe "route generation", :type => :controller do

My spec class was a controller, so that wasn't the issue. The second option is only required when the controller test is not in spec/controllers, so that wasn't the issue. I had scoured the rspec docs, and everything I read said it should work.

I had missed the most obvious thing of all -- I had said describe when I meant it:

  it "should create :page param for / url" do
    route_for(:controller => 'pages', :action => 'index').should == "/"
  end

So, if you happen to encounter a similar error... don't forget to check the obvious.

QMetaEnum Magic - Serializing C++ enum's

Qt has a number of useful classes and utilities; among these is QMetaEnum which provides the ability to serialize and deserialize C++ enumerations through the use of moc, the meta-object compiler.

Method 1 - Enum's within a QObject (fairly common)

First, let's take a look at a common example of an enum within a class as provided in the "Secrets of Qt Full" developer days presentation:

class Person : public QObject {
    Q_OBJECT
    enum  Qualification { Student, ... };
    Q_ENUMS(Qualification)
};

When moc runs on the above and sees Q_OBJECT it adds a staticMetaObject member (that's static, big surprise) of type QMetaObject. This QMetaObject instance has an indexOfEnumerator and an enumerator member functions that make it possible to access the QMetaEnum representing the Person::Qualification enum.

The code to access the QMetaEnum member looks something like the following:

const QMetaObject &mo = Person::staticMetaObject;
int index = mo.indexOfEnumerator("Qualification"); // watch out during refactorings
QMetaEnum metaEnum = mo.enumerator(index);

We can then use the QMetaEnum object as follows:

// first, let's convert from an enum value to a string
Qualification q = Person::Student;
QByteArray str = metaEnum.valueToKey(q);
// str now contains "Student"

// second, let's convert from a string to an enum value:
int value = metaEnum.keyToValue("Student");
Qualification q = static_cast(value);

Method 2 - Without a QObject-based class (less common)

UPDATE: Based on Vladislaw's comment, I've done further research and added an additional post which demonstrates how to accomplish this in namespaces other than Qt.

Another less well-known feature is that with only minor effort you can use this same feature without needing a QObject-based class as a container. For this example, I'll use the Qt::Key enum. Since we don't want a QObject based class, rather than using Q_OBJECT we'll use the Q_GADGET macro within a container:

class Container {
 Q_GADGET
 Q_PROPERTY(Qt::Key key_enum);
public:
 Qt::Key key_enum;
};

Like Q_OBJECT, Q_GADGET creates a staticMetaObject member that we'll need to use to access the QMetaEnum:

const QMetaObject &mo = Container::staticMetaObject;
int prop_index = mo.indexOfProperty("key_enum");
QMetaProperty metaProperty = mo.property(prop_index);
QMetaEnum metaEnum = metaProperty.enumerator();

Unlike the first example, in this example I referenced the Qt::Key typed key_enum property to get a hold on the QMetaEnum object that I can now use exactly as before:

// convert to a string
Qt::Key key = Qt::Key_Down;
QByteArray str = metaEnum.valueToKey(key);
qDebug() << "Value as str:" << str;

// convert from a string
int value = metaEnum.keyToValue("Key_Up");
key = static_cast(value);
qDebug() << "key is Key_Up: " << (key == Qt::Key_Up);

Which results in the following output:

Value as str: "Key_Down" 
key is Key_Up:  true

And there you have it... an easy way to leverage Qt to serialize C++ enums.

The leak that wasn't a bug (i.e. one reason programming is hard)

Bugs are a pain, they eat time, destroy quality, and seem to almost always happen at inopportune times. What's even worse is when the bug is in a 3rd party library or somebody else's code.

My application needed to monitor a directory for changes to files. Unfortunately, Java 6 doesn't yet have support for directory monitoring, so I started looking around for different libraries. I found two that were good candidates, JNotify and JPoller but ultimately decided to use JPoller because it was a pure Java implementation and was fairly well documented.

It didn't take long to integrate JPoller into my application. Although it wasn't designed for testability in mind, I expected the tests to go fairly quickly, and they did. Everything looked good. After some refactoring I started to implement my next feature. Something was wrong. My file wasn't picked up. The first and second times I just assumed I placed the files in the wrong directory... but I hadn't.

I started looking through the JPoller source. I learned a few things, made some changes and continued the bug hunt. But, to no avail. Despite the logic looking perfectly sound it wasn't working right all the time. It was a race condition.

As luck would have it, I had a startling insight. What if the timestamps were wrong?

After quite a bit of investigation I discovered the insight was correct. Despite the files having a timestamp in milliseconds, it was only accurate up to one second. This allowed the file to be created at 850 milliseconds after some second and to have a timestamp of 850 milliseconds earlier.

The Law of Leaky Abstractions

I had just been struck by the Law of Leaky Abstractions. On a Windows NTFS file system the resolution was accurate to 100 nanoseconds, but on Linux it was accurate to one second. And Windows FAT filesystems were only accurate to two seconds (for modification time).

Despite the cross-platform nature of Java, I had to know and understand file system specifics in order to hunt down and kill this bug that wasn't a bug. Instead, it was a leak, and one that had to be carefully worked around in order to maintain the cross-platform nature of Java.

Programming is hard. It's never enough to know a single language or a single platform. Sooner or later a leaky abstraction forces you into other realms.

The faller's wedge & ubiquitous language

I've recently been reading Domain-Driven Design, by Eric Evans. One of the foremost things Evans discusses is ubiquitous language, which he defines as:

A language structured around the domain model and used by all team members to connect all the activities of the team with the software.

Without that shared language, problems ensue. The following story, told by Samuel T. Whitman, illustrates well what may happen without a common shared language:

“The ice storm [that winter] wasn’t generally destructive. True, a few wires came down, and there was a sudden jump in accidents along the highway. … Normally, the big walnut tree could easily have borne the weight that formed on its spreading limbs. It was the iron wedge in its heart that caused the damage.

“The story of the iron wedge began years ago when the white-haired farmer [who now inhabited the property on which the tree stood] was a lad on his father’s homestead. The sawmill had then only recently been moved from the valley, and the settlers were still finding tools and odd pieces of equipment scattered about. …

“On this particular day, [the lad found] a faller’s wedge—wide, flat, and heavy, a foot or more long, and splayed from mighty poundings. [A faller’s wedge, used to help fell a tree, is inserted in a cut made by a saw and then struck with a sledgehammer to widen the cut.] … Because he was already late for dinner, the lad laid the wedge … between the limbs of the young walnut tree his father had planted near the front gate. He would take the wedge to the shed right after dinner, or sometime when he was going that way.

“He truly meant to, but he never did. [The wedge] was there between the limbs, a little tight, when he attained his manhood. It was there, now firmly gripped, when he married and took over his father’s farm. It was half grown over on the day the threshing crew ate dinner under the tree. … Grown in and healed over, the wedge was still in the tree the winter the ice storm came.

“In the chill silence of that wintry night, … one of the three major limbs split away from the trunk and crashed to the ground. This so unbalanced the remainder of the top that it, too, split apart and went down. When the storm was over, not a twig of the once-proud tree remained.

“Early the next morning, the farmer went out to mourn his loss. …

“Then, his eyes caught sight of something in the splintered ruin. ‘The wedge,’ he muttered reproachfully. ‘The wedge I found in the south pasture.’ A glance told him why the tree had fallen. Growing, edge-up in the trunk, the wedge had prevented the limb fibers from knitting together as they should.”

The Communication Schism

A chasm, or at minimum a schism, often exists between domain experts and developers. Domain experts often use terms that are inexact or ambiguous. Developers use terms specific to themselves and that describe implementation details. This leads to fracture in in both communication and the domain model. This fracture happens at all levels of communication. Without a ubiquitous language used by all team members and the domain experts who largely define the software, a gap between understanding and implementation grows.

Domain experts often use terms that are inexact or ambiguous. Developers use terms specific to programmers and describe implementation details. This leads to fracture in in both communication and the domain model. This fracture happens at all levels. Sometimes it may be a small schism and other times it grows into a chasm.

Without a solid base, a ubiquitous language to meld the domain expert's knowledge with implementation, it is as if there's a faller's wedge stuck in our limbs. Our communication avoids certain aspects of the domain, circumvents necessary detail, and ignores relevant issues.

As this happens time and time again, both developers and domain experts become hardened, they don't see the issues nor understand how to resolve the differences. Rather than build up knowledge, it is skirted and worked around.

In Domain-Driven Design, Eric teaches us how to overcome these problems. I highly recommend it.

Customer Support is about LOVE

Yes, customer support is about love! It's about caring and doing something because you care. Here are some relevant definitions from Dictionary.com:

Love --
1. a profoundly tender, passionate affection for another person
9. affectionate concern for the well-being of others: the love of one's neighbor
10. strong predilection, enthusiasm, or liking for anything: her love of books
11. the object or thing so liked: The theater was her great love

And that's not it, there are other relevant definitions, but the above are sufficient to make my point:

Customer support and business to customer relations should be about love.

I've recently had a few customer support experiences that I'd like to recount:

Pulsar Watches

The battery in my wife's Pulsar watch died years ago. Sometime after taking the battery out to get it replaced I lost the battery. Nothing on the watch helped us identify the correct battery. My wife, as organized as ever, knew exactly where the receipt was and I was able to look up the model number, but no online searches revealed the correct battery, so I e-mailed pulsar customer support.

Even though the watch was over 10 years old, completely out of warranty, and I doubt they had many electronic records about the watch, they e-mailed me back and provided me with the exact battery type.

To be honest, I wouldn't have minded at all if they had said they had no idea where they could find that information, but they answered my question when they could have easily disregarded it. I could say they showed a predilection for their customer referring back to definition 10. Pulsar watches loved their customer enough to go the extra mile.

The Unnamed Tech Company

The Unnamed Tech Company has a great product that many people love. In addition to their main product they have another product that's quite esoteric but that happens to fill an exact need that I have at the time. I e-mailed them, and then I waited... and waited... and waited. Nothing. I decided to e-mail them again, this time at a different e-mail address, explaining my previous e-mail, and clarifying my intention to purchase. And then I waited... and waited.

I did finally receive an e-mail back, but now I'm waiting for a response. Support has been lackluster, perhaps almost dismal. In the spirit of full disclosure they admitted my first e-mail went into junk mail, but still. I'm a potential customer, or could have been an existing customer, and my e-mail was lost. They didn't answer all my questions and still haven't reached a satisfactory conclusion.

Netflix on Proactive Love

Netflix constantly amazes me. Even their company culture is amazing. But, it goes far beyond that. They are passionate about their customers. Yes, that's right. They love their customers, or at least I think so... and I doubt they'd say otherwise.

A few days ago, after having streamed a movie the night before, I received an e-mail asking me how the quality was. I was impressed. Not only were they proactive, but they really wanted to know. And, I'm happy to say that the quality was superb. For a 10-15 year old movie, I can't imagine the picture being any better had it been on Blu-ray.

But... it doesn't stop there. Monday I sent back a DVD which they received early Tuesday morning. Nice and quick. I was impressed. But, it gets better. They then asked me when I mailed it. They actually wanted to know how well their service was doing. Was it taking too long? Would they consider building a distribution center closer to me? I don't know, but I knew they cared. I knew they loved their customer.

Love Your Customer

If you and your business doesn't care about your customers, change jobs, or something. As Gary Vaynerchuk would say, follow your passion. Show your customers you care. Make a difference in their lives. When you do, they'll become your earlyvangelists, your marketers, your best and most devoted customers. And, perhaps most importantly, they'll love you back, and forgive you when you make mistakes.

[Image Courtesy: dolphinsdock on flickr]

Scripting Refactoring -- Overthrowing the GUI (part 4)

Over the last few blog posts I've mentioned a number of different reasons why it would be nice to be able to script the application of refactorings. This post talks more about one possible scripting language that could be used to script refactorings.

Interesting Cases

Although I could just provide the grammar and some sample input, it wouldn't be very instructive or helpful. Here are some more interesting cases and considerations.

Camel Case

Camel Case is a standard Java convention that uses capital letters to separate words that form an identifier. For example, isTrue, shouldValidate, and eatsHotdogsOrHamburgers are all camel case identifier names. The first letter is typically lower case with the first letter in each subsequent word in the identifier being upper case.

Imagine that Jr. Programmer comes along and creates a class named AbstractSyntaxNode that contains methods called complexNode, simpleNode and 20 more similarly named methods. When Mr. Senior Programmer comes along he immediately notices that in order to follow standard Java programming conventions, these should instead be named getComplexNode, getSimpleNode, and so on. Although it would be painful to manually go through and invoke the rename refactoring on all 22 methods, this is a good case for scripted refactorings. Consider the following rename refactoring script:

rename {
    AbstractSyntaxNode{class}::(.*)Node{method},
    AbstractSyntaxNode::get\1Node;
}

This, however, would result in the following method names:

• getcomplexNode
• getsimpleNode
• ...

In the above case, we need an easy way to tell it that the first letter needs to be transformed to an upper case character. Perl provides a \u regular expression escape sequence that capitalizes the following character. Thus, to handle the camel case issue nicely, the substitution engine would need to support similar escape sequences.

Absolute and Relative Position

Absolute and relative position markers also make an interesting case. For example, I might want to rename the variable on the second line after the start of the method. Or, I might have my cursor over a variable within an editor that has a keyboard shortcut for invoking a refactoring. In these cases, I have both relative and absolute position information that is likely desirable in a refactoring scripting language.

Assume for a moment that I want to do an Extract Method refactoring of everything between line 40 and 60, inclusive. I might write the following:

extractMethod {
    40, 60, newMethodName;
}

In this case, there's no need for column information, so using a basic integer for the line information is sufficient. But, what if you wanted to inline the function referenced between column 20 and 30 on line 42? Then I would need to write something like this:

inline {
    42:20, 42:30;
}

As many editors display the line and column information as line:column, the above syntax is fairly familiar. But, how could we go about identifying relative position indicators? Consider the following:

rename {
    @(SomeClass::someMethod{method && definition} + 2), newVariableName;
}

This shows a couple of more interesting cases - first, I need to be able to identify the the method definition which I do by restricting the results of my element query to definitions in my scope clause. And, second, I needed to use the @ operator to make the element reference a position based reference.

Conclusion

Such a scripting language could easily grow unwieldy, but the Pareto Principle suggests that there must be a fairly "easy" solution that will handle 80% of the cases. What would such a solution look like? It's really hard to say until somebody actually implements one and has some real-world users. Nevertheless, my sample input and ANTLR based grammar are linked to below.

Comments and thoughts? I'd love to hear them! Thanks.

A full set of sample inputs is available on GitHub. Similarly, the full grammar is also available.

Scripting Refactoring -- Overthrowing the GUI (part 3)

I've already mentioned why it might be useful to perform refactorings in bulk. Perhaps we've decided we followed a poor naming convention, or perhaps we've moved some classes into their own namespace, so part of the class name is redundant and can be removed.

Another point of textual element identification is to remove the necessity of the GUI. Take, for example, Emacs and Vim. Both are very capable editors and and lack (to the best of my knowledge) complete and capable refactoring support. But, if an element can be identified in text, either through line and column information, or through an element reference, the editors like Vim, Emacs, and Textmate could have macros written for them that call out to a command line refactoring engine.

Two Syntaxes for Scripting Refactoring

Each refactoring could look like a function call as in the following:

rename(OldNamespace::OldClass, NewNamespace::NewClass);

In the above case, the rename refactoring takes in two different ElementReference parameters. In order to perform many rename refactorings, we'd end up with a list of the above:

rename(OldNamespace::FirstClass, NewNamespace::FirstClass);
rename(OldClass, NewClass);
rename(OldClass::oldMethod, OldClass::newMethod);
rename(globalFunction, newGlobalFunction);
//...

Although the above is unreadable, it seemed a little repetitive, so I went with a block structure that allowed the refactoring to be specified once, with a list of parameter sequences:

rename {
    // support named parameters
    // and type restriction after name specifier
    oldName = SomeNamespace{namespace}::/(.*)ElementName/,
    newName = SomeNamespace::\1Node;

    // type restriction within regular expression
    /SomeClass{class}::get(.*)Node/,
    SomeClass::\1Node;

    ::memset, ::customMemSet;
}

The block syntax above isn't as repetitive as the first example. To help the case where an editor would invoke the refactoring, line and column information could be used to identify elements as well.

Conclusion

Although the above is a new syntax, it's built on many different syntaxes that are common to languages. It uses a block syntax for overall structure. Named parameters are supported in many different dynamic languages. It supports regular expressions for matching and backreferences for substitutions. It also has C++ and Java style comments and uses the sometimes-despised semicolon as a terminator.

Yeah, some people aren't going to like the syntax and may avoid it like the plague. But, arguably, there are some very good reasons to to make refactorings scriptable to the average programmer, among which are the easy interface it would provide to editors and the ability to doing certain refactorings in bulk.

Scripting Refactoring -- Overthrowing the GUI (part 2)

Last week I introduced a hokey syntax that could be used to identify various elements that were going to be refactored in bulk. The syntax was ambiguous and incapable of expressing anything of even mild complexity. Now for something better.

The proposed syntax doesn't apply to every language. It would be both pointless and painful to attempt such a task. Rather, the following syntax is intended to work with languages like Java, C/C++, and other similarly structured languages.

A Grammar Proposal

Consider the following grammar defined in EBNF, with non-terminals being lower-cased and terminals being upper-cased and left implicit:

query-element: pattern ( "::" pattern )*;

pattern: ( IDENTIFIER | "/" REGEXP-LITERAL "/" ) type-specifier?;

type-specifier: "{" TYPE-LITERAL "}";

Using the above EBNF, arbitrary elements can be queried and their type constrained when necessary. Here's a few examples:

• All classes named MyClass in any namespace - /.*/::MyClass
• Any class containing the word Node in the AST namespace - AST::/.*Node.*/
• The second class containing the word Node in the AST namespace - AST::/.*Node.*/[1]
• All typedefs named subType contained in functions of the Node class - Node::/.*/::subType{typedef}
• All variables named other contained in functions of the Node class - Node::/.*/{function}::other{variable}
• Any function starting with test in the Temp namespace - Temp::/test.*/{function}

Admittedly, this query language does not support all the queries that one might want to make, but it should be noted that the above query language can be easily extended. For example, instead of using a TYPE-LITERAL terminal symbol, one could use a type-literal production that allowed for negating conditions, alternations, and more.

Function and operator overloading, as supported by C++, also complicate matters. Even though the fully qualified name of a function may be specified, it does not imply that only a single match exists. Any number of functions might exist. This language could be expanded to allow the types of the parameters to be specified. By specifying the parameter types, any function could be fully resolved. In addition, anonymous namespaces and blocks could be identified by augmenting the language to support empty names and array indexing.

Isn't It Complicated?

Yes, now we're starting to get complicated. But honestly, most of those features wouldn't matter. It's like the saying, "20% of your code you handle 80% of the cases." By starting out simple, and only augmenting the grammar when a real, arguable need is established, the grammar could be expanded little-by-little in useful ways. It really goes back to two principles: KISS and YAGNI. Start simple, design when necessary, refactor to your goal

Bonus points to the person who finds the grammar / example mismatch above.

Scripting Refactoring -- Overthrowing the GUI

The most basic and common refactoring in any language is Rename. Whether it's Rename Variable, Rename Method, Rename Class, or Rename Namespace/Package, this simple refactoring helps improve code clarity and, when applied correctly, makes code easier to understand. As reading code is the most frequent activity a programmer undertakes, this makes the Rename refactoring one of the most powerful tools in a developers' arsenal.

Introducing Element Identification

Consider now the Rename refactoring when applied in a batch. In order to apply the Rename refactoring, I first need to be able to unambiguously identify the element being renamed, be it a class, method, variable, etc. Let's entertain for a moment the following notation

[Member::[Member::[Member::...]]]Member

This notation could be used to identify:

• a namespace at any level (e.g., MyNamespace, Namespace1::Namespace2)
• a class within a namespace (e.g., MyNamespace::MyClass, SomeClass)
• a member of a class (e.g., AClass::someVar, ANamespace::MyClass::a)

Although this syntax resembles the C++ scoping syntax, it is important to note that it is ambiguous. Given the search query, Something::someMember, determining whether Something is a class, struct, or namespace is impossible. Similarly, someMember could be a class within a namespace, a class or struct within a namespace, or a member variable within a class or struct.

Overcoming Ambiguity

Since we need to constrain how we select elements, lets extend our syntax a bit by adding an optional { TYPE_LITERAL } constraint to each member:

[Member[{TYPE}]::...]Member[{TYPE}]

More cases can now be handled well:

• Something{namespace}::someMember{class}
• Something{class}::someMember{method}
• Something{method}::someMember{variable}
• Something{namespace}::Something{class}::someMember{method}

But... we're still ambiguous. Consider the following code:

void doSomething() {
    for (int i=0; i<10; ++i) { /* ... */ }
    /* ... more code here ... */
    for (int i=0; i<15; ++i) { /* ... */ }
}

Staying Useful

In the above code we have two different counter variables with a name of i. Does that mean it's pointless to try to support batch refactorings or to try to constrain elements in a refactoring scripting language? No.

Regular expressions don't allow everything to be queried, yet they're still useful. Even after adding positive or negative lookahead (or lookbehind) assertions, they're still limited. Yet, at the same time, they handle a few more cases and become more generally useful.

The same thing should be able to happen for (most) programming languages. It's possible to create a scripting language, or syntax, that will allow us to easily identify most syntactic elements within a programming language. Perhaps this refactoring scripting language could leverage the BNF that defines the programming language, or perhaps there's even something better. Either way, we're one step closer to overthrowing the GUI and enabling bulk and scriptable refactorings.

Yet more excerpts and discussion from my thesis to come.

Enabling Bulk Refactorings

In my last post, I talked about some of the limitations of refactoring IDEs and enumerated three different cases that could better be handled by refactoring IDEs:

• Vendor Branches - can we make refactorings apply like patches?
• Universal Language - can we switch between vocabularies or languages?
• In Concrete - can we overcome some of the small problems presented?

For this post, I'm going to examine the small concrete cases presented in In Concrete.

Bulk Renaming

My first case was to "Rename all my *ElementName classes in a given namespace to *Node where the asterisk represents some wildcard.
Consider over a 100 classes like the following, spread out over at least as many files. Each one might represent a different element in an XML or otherwise non-trivial file format:

package com.example.xml.schemas;

public class ComplexTypeElement { /* ... */ }
public class CompoundTypeElement { /* ... */ }
public class SimpleTypeElement { /* ... */ }
public class AnyElement { /* ... */ }

Here's what the current Rename refactoring dialog looks like in Eclipse:

Augmenting the Rename UI

In this case, our goal is to bulk refactor the classes since our naming convention of *ElementName ended up being a poor convention. What we essentially need is a context sensitive, regex-based find-and-replace refactoring engine. So, let's augment the rename dialog with a couple of other elements that, left alone, wouldn't alter the rename behavior:

Now, let me describe a couple of the elements. First, I introduced a checkbox that allows the programmer to specify whether the refactoring should indeed be a wildcard refactoring. Second, I introduced an "Old name" field. This field represents the entire fully qualified class name to which the refactoring should be applied when a wildcard refactoring is being performed. When a standard (non-bulk or wildcard) refactoring is being performed, this field would display the original class name. Third, I added support for backreferences in the "New name" field. By allowing support for backreferences, the new class name can be based on and derived from the old class name, much like might be done with sed.

To handle the second case presented in my previous blog post, I could now use the following in the "Old name" and "New name" input fields, respectively:

Old name: DataNode::(.*)::get.*Instance

New name: DataNode::\1::instance

In this example, the backreference is used to identify the class but not needed for the member function. Like sed or awk, the syntax and structure would take a bit of learning, but it could be learned far faster than it would likely take to rename over 100 different classes manually.

Conclusion

The above syntax is, of course, not yet sufficient for everything that might need to happen. For example, it might also match DataNode::NestedNamespace::AnotherNestedNamespace::getSomeInstance even though only one namespace should have been considered, but it's a start, and one that I'll elaborate on in later posts.

The main point, however, is that it is possible to overcome many of the current limitations of refactoring engines. Yes, it will take some work, but it will be worth it in the long run. And, in the above cases at least, the implementation is trivial compared with the work already done to support a single rename refactoring.

Limitations of Refactoring IDEs

Coupled with every strength is a weakness. Within an IDE, the ability to leverage the utility of a mouse is a strength. When trying to automate the selection of the next four characters, no matter where you may be in a file, requiring the same mouse-driven approach becomes a weakness. And so it is with refactoring tools today. The current mouse-driven approach to identifying parameters to refactorings has a number of weaknesses:

• Other than through GUI automation tools, such as APIs and macros supported by IDEs, starting the refactoring process cannot be automated.
• It is impossible to apply the same refactoring to many elements at one time. Rather, each element must be selected one at a time and the refactoring applied.
• It is difficult or impossible to script refactoring operations.
• Elements being refactored are often identifi ed by line and column position within a fi le, which usually changes over time.

Although perhaps unseen and unrecognized, the weaknesses stated above are prevalent. The mainstay of element identi cation within existing refactoring tools is the mouse location and current cursor position. Although these function to identify the elements to which refactorings should be applied, they are not appropriate nor reasonable for every situation. Two use cases, detailed below, demonstrate an unmet need in the arena of refactoring tools.

Use Case 1 - Vendor Braches

Background

John is working with a vendor's source code and is maintaining a vendor branch. Unfortunately, the vendor's code has some unpatched bugs and often uses poorly named identi ers which make it hard to work with.

Problem

John has a set of patches which he applies using the standard vendor branch approach to patching. He has two sets of patches. The first set of patches applies the Rename refactoring to many di fferent elements in order to make the code easier to work with. The second set of patches fixes currently unpatched bugs.

Following the vendor branch patching process, after each vendor update John loads the vendor's source code into the source control system and then applies the first and second set of patches. Unfortunately, the first set of patches often fails to compile because any new references to the variables being renamed are not included in the patch. Each new reference must be manually renamed and the patch updated. Once the first patch has been successfully applied, the second bug-fi xing patch usually applies successfully.

Although this approach accomplishes what it needs to, it is less than ideal. With each new vendor release the Rename refactoring must be re-applied to the source code for every newly introduced reference to the variables being renamed.

Use Case 2 - Universal Language

Background

Beth is involved in an Open Source project whose primary developers are French. The source code is written and commented entirely in French, making it hard for Beth, who does not speak French. Although the developers have decided to change to English as their universal language, they have not yet made the change.

Beth worked with the developers to understand the meaning of each variable but needs to work with the French code for quite a while before English becomes the universal language.

Problem

Although Beth could maintain a separate English branch of the source code, that would require that all code be modi ed or committed twice, once for English and once for French. She could convert the source code over once within a separate branch, but each new reference to an existing variable would need to be changed. No good solution exists.

Use Case 3 - In Concrete

I would like to perform the following refactorings, but doing so through a GUI will be painful:

• Rename all my *ElementName classes in a given namespace to *Node where the asterisk represents some wildcard.
• Rename all get*Instance member functions to instance for every class within the DataNode namespace.

Perhaps the above seem meaningless, but I've had to do almost those exact same things. After a while, I discovered that some of the class names and methods that I was using weren't descriptive enough. I had to go rename over 50 classes even more files.

Conclusion

Current refactoring tools are wonderful... for what they were intended for. They make the process of writing code, and cleaning up that code while you're working on it, far easier and more likely to happen that if the process were manual. But, the GUI itself is also a weakness, one for which we need an alternative. We need a way to handle these and other uses cases. We need a way to back-port refactorings, where possible, without forcing the programmer to take manual steps. We need wildcards and sed-like functionality in our refactoring tools.

Much of the above content comes directly from my thesis. I believe there's a solution and future posts will discuss pieces of that solution. I don't believe my solution is complete or perfect, but I hope to further the work in this area.

Image courtesy seizethedave on Flickr

Things I Wish I Knew...

When I was a young child an adult friend of mine took me over to his house and showed me his Commodore 64 computer. From the moment I saw the music software and the other things that he could do on his computer, I was hooked.

From that moment on, I played with computers whenever the opportunity arose. Before long, an uncle of mine, who was a computer programmer, sent me a Radio Shack TRS-80 computer. My interested piqued, I started programming in BASIC, going to the library to get programming books, and saving my money for computer parts. My interest for computers never died.

Years Later

Years later I found myself studying computer science (CS) at Eastern Washington University. I enjoyed it. Aside from some occasional frustration at my inability to easily do a homework assignment, I loved it!

Then, a bit of realization struck, which I didn't fully understand at the time:

• I never heard version control systems mentioned
• The instructor's used #include <iostream.h> instead of #include <iostream.h>
• Unit tests were unheard of
• Test-driven development wasn't considered

I Wish I Knew...

Having reflected on some of the insights I had earlier, I've realized many different things that I wished I knew when I started in school. I almost think they should have been obvious, but I had no mentor to ask and no prior knowledge to inform me. Here's a few of the things I wish I knew long ago:

• Computer Science is not a programming degree. Although much of what is learned in computer science is useful when programming, knowing everything they teach in CS doesn't make you a good programmer.
• Programming is an art. Programmers need a sense of aesthetics. They need to see how different pieces fit together to form a beautiful and functional whole.
• Good code is beautiful code. Just as a good writer's sentences are flowered with adjectives and adverbs that form vivid images in his readers' minds, good code helps its reader understand and envisage the whole intent of the code. Each variable and name is like an adjective or adverb that adds clarifying definition to the whole.
• Programming is rarely engineering. Unfortunately perhaps, requirements are rarely what they seem. A good programmer communicates, questions, and suggests whatever might be the best for her customer; she doesn't assume the customer knows what she wants.
• Universities can't teach you everything. And, they don't even try. A CS degree program doesn't try to teach you everything you'll need to know, because much of it can't be taught. At best, the professors can encourage the student to go off and do research on their own. It takes hard work, and lots of practice to be a good programmer. Although you need not go through trial and error to make a test pass, it takes trial and error to perfect your skills, and there's nothing that you can do about it. Put in your 10,000 hours. Do it fast, and do it right. Read, study and learn on your own.

Countless other details could be added to the list above. I'm sure there are many more which I have yet to consider, but I hope this will help some future student.

What do you wish you had known?

[Image courtesy Guy with the Bolex on Flickr.]

Eliminate Primitive Obsession

Primitive obsession is not just fiddling bits, playing with ints, or tossing around chars. Primitive obsession is using something non-expressive where something better suited is possible. It's about using List and Enum when domain objects would be far better.

In my post "The UI/UX Difference" I mentioned implementation models -- a system wherein the implementation leaks through to the user, forcing the user to understand something about how the system is implemented before it becomes usable. It turns out, I've been doing the same thing in my code.

Primitive: "1.a - not derived, 2.c - belonging to or characteristic of an early stage of development

I'm guilty. My code has too often been nothing more than a code implementation model. How's that you might ask?

Let me explain. Consider the following code:

createUri(
    annotation.getArgument(2)
        .getString(AnnotationConstants.SeparatorDefault)
    )

So now, my question to you -- what does that code do? You probably can't tell me. What if I had written the following:

separatorAnnotation.separatorStringOrDefaultValue().asUri()

The above code should make it much more obvious -- it's converting the separator string, or its default value, in URI form.

The first form uses a general annotation object, one that isn't derived enough. It forces the user, in this case a programmer, to:

• know that the separator string is the third argument in the annotation, and 
• know that each argument is an argument object, and 
• know that the argument object can be converted to different types, and 
• know that the argument should have been a string type, and 
• know where the default value for that annotation parameter was specified.

That's bad, horrible, and awful! I'm forcing all these things upon other programmers because I'm failing to work with domain-level objects.  In this case, the separator annotation is a fundamental part of my domain and my business.  And yet, by working with a generic annotation object, I left it a primitive.  I left it in an early stage of development.

Don't make the same mistake I did.  Primitive obsession is much more than using a bunch of ints, it's about avoiding your domain in code. Use domain code and free others from understanding the implementation.

The UI / UX Difference

As a child, either I or one of my siblings had a conversation with my mom that went something like this:

Mother: Where's that magazine you borrowed?

Child: Oh, it's right under my math book under that stack of clothes.

Mother digs through stack of clothes.

Mother: Amazing! I don't know how you could find anything in here.

When we create something, it becomes second nature to us.  We understand it; we know how it works; we know where we put things. What may have taken the Mom hours to find, the child could do in seconds.  The same problem exists with user-interfaces written without attention to the user.

Integrated Knowledge

As programmers and software developers, some things are instinctively easy to us.  Actually, it goes beyond that, we have habitualized and integrated every related metaphor and paradigm. Computers, applications and their uses are so ingrained in us that it becomes impossible for us to imagine life and computers without that knowledge.

Last year my dad asked for some computer lessons. My dad has no experience with computers -- he had never been interested in computers nor had he touched them since he was in college. And, even then, he was learning some basics of punch-card programming. Certainly not relevant any more.

My wife both laughed and shuttered as I attempted to teach my dad the basics of computers -- menus, toolbars, the difference between a right click and a left click. I tried to teach the generic concepts that would be applicable to most programs.  I was attentive and careful to use only universal idioms and metaphors.

I failed. He didn't understand the metaphors; he didn't see the relevance to life. Every concept was completely foreign.

Importance of UI / UX

I had recently been researching some user interface design issues in About Face 3: Essentials of Interaction Design so I understood a one very important thing:

Most everything we attempted was based on implementation model, i.e., rather than being goal directed and based on user understanding, my dad would have to understand something about how the computer was implemented in order to use the computer.

Resolve to Learn

Admittedly, it's impossible for me to forget everything that I know, but it's not impossible for me to learn much more about how non-programmers and every-day users think. I'm by no means a capable UI / UX person (quite honestly, I'm awful with colors and aesthetics), but I have learned enough to make a difference for the users of my programs.  Resolve to learn some UI and UX basics -- you can make a difference.

Inspired by a tweet by David Siegel.

Think like a Master

Not too long ago I was reading "The Unreasonable Effectiveness of Mathematics," by Richard W. Hamming, the creator of Hamming Code and winner of the Turing award. Amidst many of his great comments stood one that transcends mathematics and fits into the lives of each of us, no matter what our professions or hobbies:

Not too long ago I was trying to put myself in Galileo's shoes, as it were, so that I might feel how he came to discover the law of falling bodies. I try to do this kind of thing so that I can learn to think like the masters did-I deliberately try to think as they might have done.

Do we take time to think like those who came before us? How did Galileo consider gravity? How did Pythagoras develop his well known theorem? How did Isaac Newton discover the inverse square relationship based on Kepler's laws?

I often look back in time and stand amazed at what my forefathers accomplished.  They didn't have the resources I have available to me -- the books, science, technology, and tools.  Yet, despite their limited resources, they developed incredible knowledge and wisdom in diverse and wonderful topics.

Think Like a Master

In order to glean a few insights from the masters, attempt to think like them.  I'll propose a few for programming and software development:

• How would you encode the rules for a chess game? Could you write basic AI for a chess game? Now do both of those... and make the program work in less than 1k of memory.
• How can you optimize reading and writing of files to extremely slow disks in the era of Microsoft Word 1.0?
• How could you write a single-pass compiler that allows mutually recursive declarations?
• What could you do to reduce the risk of an error when using a punch card system?
• How do you avoid parsing an extremely large XML file to read the last N entries in the file?

The human mind is amazing.  It can imagine and discover wonderful things.  Take time to think, to ponder, and to learn from the masters and what they accomplished.

SMART Tests Outwit Bugs

With the new year come the typical thoughts of goals and resolutions.  I have come across the SMART acronym many times.  It states that goals should be:

Specific
Measurable
Attainable
Reasonable
Timely

After some thought, I realized that this also applies to tests.

Specific - A test should target one specific feature or behavior. The name of the test should be representative of what that behavior or feature is. When I name my tests in this way, as soon as the test failure is seen, I know what's broken. By looking at a few lines of code, I should know where the cause of the failure is. If I can't tell by looking at the code, my test is not specific enough. This also relates to the one (conceptual) assert per test idea. Like the high-contrast target above, you want to be able to tell, instantly, what went wrong.

Measurable - If you can't measure what you're attempting to test, it's pointless. For example, I don't test that the screen is drawn pixel perfect. It's not easily measurable. To attempt to test it would be a waste of my time. Manual tests are still tests; use them. TDD isn't a panacea.

Accurate - I consider a test accurate if it meets two requirements. First, it's name must represent the behavior being tested. Second, I must be able to look at the test and very quickly ascertain that it does what it says it does. Uncle Bob Martin's book, Clean Code, describes a good function as one that has a single abstraction -- one that can't be refactored any further.

Responsible - I shouldn't have to say this, but tests should NOT have side-effects. Tests with side-effects destroy any confidence that a later test may, or may not, be correct. Don't write tests with side effects.

Timely - Long-running tests are less-run tests. If a test isn't fast, I'm not going to run it as often. And, when I do run it, I'm likely to get distracted and start answering lots of Stack Overflow questions while I'm waiting for it to run.

Write SMART tests. By doing so, it becomes much easier to outwit bugs.

[Image from: http://www.ctc.com.pt/fotos_20061007.html]