[private method test] Testing Private Methods with JUnit and SuiteRunner

source:http://www.artima.com/suiterunner/privateP.html

Testing Private Methods with JUnit and SuiteRunner
by Bill Venners
May 24, 2004

Summary
This article compares four different approaches to testing private methods in Java classes.

My very first use of JUnit was to build a conformance test kit for the ServiceUI API [1]. The purpose of a conformance test kit is to help ensure that alternate implementations of the same API are compatible with the API's specification. Because an API specification defines only the public interface of the API, not the API's implementation, a conformance test exercises only the public interface. In other words, a conformance test is a "black box" test. It treats the API under test as a black box, whose external interface can be seen, but whose internal implementation cannot. A conformance test of a Java API, therefore, need only access the public members of the packages and classes under test. There is no need to access package-level, protected, or private members.
When I later applied JUnit to the task of writing actual unit tests, as opposed to conformance tests, I found myself wanting to write white box tests—tests that employ knowledge of the internal implementation of the packages and classes under test. Whereas I only wanted to test public methods in my conformance tests, I wanted to write unit tests for package access and occasionally private methods as well as public methods.
Daniel Steinberg [2] showed me the common JUnit technique of using parallel source code trees, which allowed me to place test classes in the same package as the classes under test, but keep them in a different directory. This provided a clean separation of test and production code. By placing both source trees in the CLASSPATH, my test classes could access package-level methods and classes in the package under test. This still left me, however, with the problem of testing private methods.
When I asked Daniel about testing private methods, he gently suggested that I test the private methods indirectly by testing the package-access and public methods that call the private ones. This answer did not quite satisfy me, because on occasion I really did feel the urge to directly test a private method. My initial solution was to just make such private methods package access, which allowed me to test them directly with JUnit from the test classes in the same package in the parallel source tree. This worked fine, but made me feel a bit dirty somehow. Although in general I discovered that thinking about how to design interfaces so they could be easily unit tested helped me design better interfaces, in this case I felt I was making the design slightly worse to make it testable.
When I later ended up participating in the creation of what Frank Sommers, Matt Gerrans, and I eventually released as Artima SuiteRunner [3], I vowed that I would make the testing of private methods easier in SuiteRunner than it is in JUnit. But after investigating the various approaches to testing private methods, I decided not to do anything special in SuiteRunner to support testing private methods. So whether you are using JUnit or SuiteRunner, you have the same four basic approaches to testing private methods:

• Don't test private methods.
• Give the methods package access.
• Use a nested test class.
• Use reflection.

In this article, I will discuss these four approaches to testing private methods in Java. I will look at the advantages and disadvantages of each and attempt to shed some light on when it makes sense to use each approach.

Approach 1: Don't Test Private Methods

As I mentioned in the introduction, I first heard the advice to suppress my occasional urges to test private methods from Daniel Steinberg. But Daniel is not only source of this advice that I have encountered. It seems to be a common attitude in the Java community. For example, the JUnit FAQ [4] states:

Testing private methods may be an indication that those methods should be moved into another class to promote reusability.

Charles Miller expressed a similar point of view in his weblog [5]:

If you have a thorough suite of tests for a class's exposed (non-private) interface, those tests should, by their nature, verify that any private method within the class also works. If this isn't the case, or if you have a private method so complex that it needs to be tested out of the context of its public callers, I would consider that a code-smell.

And Dave Thomas and Andy Hunt, in their book Pragmatic Unit Testing [6], write:

In general, you don't want to break any encapsulation for the sake of testing (or as Mom used to say, "don't expose your privates!"). Most of the time, you should be able to test a class by exercising its public methods. If there is significant functionality that is hidden behind private or protected access, that might be a warning sign that there's another class in there struggling to get out.

I believe all this advice. Most of the time, private methods can be most effectively tested via approach 1, indirectly by testing the package-level, protected, and public methods that call them. But inevitably, some people in some situations will feel that directly testing a private method is the right thing to do.
In my case, I tend to create many private utility methods. These utility methods often do nothing with instance data, they just operate on the passed parameters and return a result. I create such methods to make the calling method easier to understand. It is a way to manage the complexity of the implementation of the class. Now, if I extract the private method out of a method that already works and has good unit test coverage, then those existing unit tests will likely suffice. I needn't write more unit tests just for the private method. But if I want to write the private method before its calling method, and I want to write the unit tests before writing the private method, I'm back to wanting to directly test the private method. In the case of private utility methods, I don't feel my urge to directly test the methods is, as the JUnit FAQ put it, "an indication that those methods should be moved into another class to promote reusability." These methods are really only needed in the class in which they reside, and in fact are often only called by one other method.
Another reason I sometimes feel the urge to test private methods directly is that I tend to think of unit testing as helping me achieve a robust system by building that system out of robust parts. Each part is a "unit" for which I can write "unit tests." The unit tests help me ensure each unit is functioning correctly, which in turn helps me build a system that functions correctly as a whole. The primary unit I think in terms of when programming in Java is the class. I build systems out of classes, and unit tests give me confidence that my classes are robust. But to some extent I also feel the same way about the private methods out of which I compose package-access, protected, and public methods. These private methods are units that can be tested individually. Such unit tests give me confidence that the private methods are working correctly, which helps me build package-access, protected, and public methods that are robust.

Approach 2: Give the Methods Package Access

As I mentioned in the introduction, giving methods package access was my first approach to testing private methods with JUnit. This approach actually works just fine, but it does come with a slight cost. When I see a private access specifier on a method, it tells me something I like to know—that this is part of the implementation of the class. I know I can ignore the method if I am just trying to use the class from another class in the package. I could figure this out about a package-access method by looking more closely at the name, documentation, and code of the method, but the word private communicates this far more efficiently. Moreover, the main problem I have with this approach is philosophical. Although I don't mind "breaking encapsulation for the sake of testing," as Dave and Andy would put it, I just don't feel good about breaking encapsulation in a way that changes the package-level API. In other words, although I am quite enthusiastic to test non-public methods of classes, i.e., to create "white-box" unit tests, I'd rather the API of the classes under test, including the package-level API, not be changed to facilitate those tests.

Approach 3: Use a Nested Test Class

A third approach to testing private methods is to nest a static test class inside the production class being tested. Given that a nested class has access to the private members of its enclosing class, it would be able to invoke the private methods directly. The static class itself could be package access, allowing it to be loaded as part of the white box test.
The downside to this approach is that if you don't want the nested test class being accessible in your deployment JAR file, you have to do a bit of extra work to extract it. Also, some people may not like having test code mixed in the same file as production code, though others may prefer that approach.

Approach 4: Use Reflection

The fourth approach to testing private methods was suggested to me by Vladimir R. Bossicard, who wrote JUnit Addons [7]. One day over lunch, Vladimir enlightened me that the java.lang.reflect API included methods that allowed client code to circumvent access protection mechanism of the Java virtual machine. He also told me that his JUnit Addons project included a class, junitx.util.PrivateAccessor [8], to assist in using the reflection API for just this purpose: to write unit tests that manipulate private members of the classes under test. The JUnit FAQ points to a similar class, called PrivilegedAccessor [9], written by Charlie Hubbard and Prashant Dhotke.
One advantage of using the reflection approach to testing private methods is that it provides a clean separation of test code and production code. The tests need not be nested inside the class under test, as in approach 3. Rather, they can be placed alongside the other tests that exercise the package-level and public methods of the class. In addition, you need not alter the API of the class under test. Unlike approach 2, private methods can remain private. Unlike approach 3, you need not add any extra nested class at package access level. The main disadvantage of this approach is that the test code is far more verbose because it uses the reflection API. In addition, refactoring IDEs such as Eclipse and IntelliJ usually aren't as adept at changing the names of methods where they are referred to as Strings passed to the methods of the reflection API. So if you change the name of the private method with your refactoring IDE, you may still have to make some changes by hand in the test code.

An Example

To give one example of a private method that, in my opinion, merits direct unit testing, I extracted some functionality out of the main method of class org.suiterunner.Runner. Runner.main parses command line arguments and runs a suite of tests, optionally firing up the GUI. The method I extracted, parseArgsIntoLists, does part of the work of parsing of the command line arguments to the SuiteRunner application. Now, to test the public method that calls this private method, I would need to test main. Main, of course, is the entire application, which makes the method rather difficult to test. In fact, I have no existing test for main. At this point, you may be wondering, if I were writing tests first in the style of test-driven development [10], how did I ever end up writing parsing code that had no unit tests? The main reason is that my test infection has come in stages. I did actually catch a unit test flu long before I had heard of JUnit or read Test Infected [11]. Back when I was building Windows applications in C++, for example, I would write a bit of code to test a newly implemented method, then execute that code and watch it execute by stepping through the method under test with the debugger. This kind of unit testing did help me achieve robustness, but the tests themselves didn't check for the proper behavior. I checked for the proper behavior myself by observing via the debugger. The tests weren't automated, and therefore I didn't save them so they could be run again later. When I read Test Infected, I immediately saw the value of automating the tests and keeping them around as a kind of regression test after refactoring, but for a long time it didn't make sense to me to write the tests first. I wanted to write the tests after I implemented the functionality, because that was when I had run the tests with the debugger. A secondary reason I didn't write tests first while developing much of SuiteRunner is that I wanted to write SuiteRunner's tests with SuiteRunner itself, in an effort to eat my own dog food. Until SuiteRunner's basic API settled, I didn't have the testing toolkit I wanted to use to write the tests.
Since that time, however, the testing virus has taken a stronger hold on me, and I do now prefer to write unit tests first most of the time. I prefer to write tests first not so much because I find I end up with cleaner designs, which is usually promoted as the main benefit of test-driven development. Rather, I prefer to write tests first because I find that often if I dive into the code under pressure, with the intent that I'll write the test later, the test doesn't actually ever get written. SuiteRunner itself has very few unit tests at this point for that very reason. Here's the parseArgsIntoLists method:

    private static void parseArgsIntoLists(String[] args, List runpathList,
            List reportersList, List suitesList) {

        if (args == null || runpathList == null
                || reportersList == null || suitesList == null) {

            throw new NullPointerException();
        }

        for (int i = 0; i < args.length; i++) {

            if (args[i].startsWith("-p")) {

                runpathList.add(args[i]);
                runpathList.add(args[i + 1]);
                ++i;
            }
            else if (args[i].startsWith("-g")) {

                reportersList.add(args[i]);
            }
            else if (args[i].startsWith("-o")) {

                reportersList.add(args[i]);
            }
            else if (args[i].startsWith("-e")) {

                reportersList.add(args[i]);
            }
            else if (args[i].startsWith("-f")) {

                reportersList.add(args[i]);
                reportersList.add(args[i + 1]);
                ++i;
            }
            else if (args[i].startsWith("-r")) {

                reportersList.add(args[i]);
                reportersList.add(args[i + 1]);
                ++i;
            }
            else if (args[i].startsWith("-s")) {

                suitesList.add(args[i]);
                do  {
                    ++i;
                    suitesList.add(args[i]);
                } while (i + 1 < args.length);
            }
            else {
                throw new IllegalArgumentException("Unrecognized argument: "
                        + args[i]);
            }
        }
    }

The command line for SuiteRunner contains three types of information used by SuiteRunner to run tests: runpath, reporters, and suites. The parseArgsIntoLists method merely goes through the arguments passed as an array of Strings, and places each argument into one of the lists, runpathList, reportersList, and suitesList.
Before I write a test for this private method, I would ask whether my urge to write this unit test represents a code smell, as Charles Miller put it in his weblog? Does it indicate that parseArgsIntoLists should be moved into another class to promote reusability, as the JUnit FAQ suggests? Would Dave and Andy say its a warning sign that there's another class in there struggling to get out? Well, maybe. I could concievably create an ArgumentsParser class that only holds a few static methods that perform the parsing work. Both the ArgumentsParser class and the methods it contains could be package access, which would make them easy to test. But that just doesn't feel right to me. These methods are only called by Runner.main. They clearly feel like private methods to me. The only reason I would be moving them to an ArgumentsParser class is to be able to test them. I would in fact be using approach number 2: make the private methods package access.
Instead, for this example I decided to take approach number 4, and use reflection. I looked at both Vladimir Bossicard's junitx.utils.PrivateAccessor and Charlie Hubbard and Prashant Dhotke's PrivilegedAccessor, but decided that neither of them helped me quite the way I wanted. For one thing, these classes both have the ability to test fields to make sure they are set correctly. As yet I have never felt any urge to directly access private fields from unit tests. I just want to be able to test private utility methods. The main problem I had with these two classes, however, is how they dealt with the exceptions that may be thrown when trying to invoke the private method via reflection. Each class has one or more methods whose job it is invoke a method with reflection. PrivilegedAccessor's two invokeMethod methods passes any exception back to its caller, including three checked exceptions declared in the throws clause: NoSuchMethodException, IllegalAccessException, and InvocationTargetException. By contrast, PrivateAccessor's two invoke methods catch InvocationTargetException, and extract and throw the target exception, the actual exception thrown by the invoked method. It then catches any other exception, and throws NoSuchMethodException. I didn't like that the caller of PrivilegedAccessor.invokeMethod would always need to handle the three checked exceptions, because I figured the general way to handle any exception would be to let the test fail. I was also concerned that the PrivateAccessor.invoke was throwing away potentially useful stack trace information in its exception handling policy. What I really wanted was a method that attempted to invoke a private method with reflection, which wrapped any thrown exception besides InvocationTargetException in an unchecked TestFailedException. Most of the time this exception would cause the test to fail. In tests that were expecting an exception to be thrown, the exception contained in InvocationTargetException could be extracted and tested for correctness.
Therefore, I wrote invokeStaticMethod. The setAccessible(true) call is what enables the private method to be invoked from outside the class. An corresponding invokeStaticMethod implementation for use with JUnit would throw AssertionFailedError rather than TestFailedException. Here's the code:

    private static void invokeStaticMethod(Class targetClass,
            String methodName, Class[] argClasses, Object[] argObjects)
            throws InvocationTargetException {

        try {
            Method method = targetClass.getDeclaredMethod(methodName,
                    argClasses);
            method.setAccessible(true);
            method.invoke(null, argObjects);
        }
        catch (NoSuchMethodException e) {
            // Should happen only rarely, because most times the
            // specified method should exist. If it does happen, just let
            // the test fail so the programmer can fix the problem.
            throw new TestFailedException(e);
        }
        catch (SecurityException e) {
            // Should happen only rarely, because the setAccessible(true)
            // should be allowed in when running unit tests. If it does
            // happen, just let the test fail so the programmer can fix
            // the problem.
            throw new TestFailedException(e);
        }
        catch (IllegalAccessException e) {
            // Should never happen, because setting accessible flag to
            // true. If setting accessible fails, should throw a security
            // exception at that point and never get to the invoke. But
            // just in case, wrap it in a TestFailedException and let a
            // human figure it out.
            throw new TestFailedException(e);
        }
        catch (IllegalArgumentException e) {
            // Should happen only rarely, because usually the right
            // number and types of arguments will be passed. If it does
            // happen, just let the test fail so the programmer can fix
            // the problem.
            throw new TestFailedException(e);
        }
    }

Next, I created a convenience method that invokes the particular private method I wanted to test:

    private static void invokeParseArgsIntoLists(String[] args,
            List runpathList, List reportersList, List suitesList)
            throws InvocationTargetException {

        // Purposely pass null values to the method, to make sure it throws
        // NullPointerException

        Class[] argClasses = {String[].class, List.class, List.class, List.class };
        Object[] argObjects = {args, runpathList, reportersList, suitesList };

        invokeStaticMethod(Runner.class, "parseArgsIntoLists", argClasses, argObjects);
    }

At last, I could write tests against the private method without too much excess clutter, like this:

    public void testParseArgsIntoLists() throws InvocationTargetException {

        String[] args = new String[0];
        List runpathList = new ArrayList();
        List reportersList = new ArrayList();
        List suitesList = new ArrayList();

        try {
            invokeParseArgsIntoLists(null, runpathList, reportersList, suitesList);
            fail();
        }
        catch (InvocationTargetException e) {

            // throw the InvocationTargetException unless the target
            // exception is NullPointerException, which is expected
            Throwable targetException = e.getTargetException();
            if (!(targetException instanceof NullPointerException)) {
                throw e;
            }
        }

        try {
            invokeParseArgsIntoLists(args, null, reportersList, suitesList);
            fail();
        }
        catch (InvocationTargetException e) {

            // throw the InvocationTargetException unless the target
            // exception is NullPointerException, which is expected
            Throwable targetException = e.getTargetException();
            if (!(targetException instanceof NullPointerException)) {
                throw e;
            }
        }

        try {
            invokeParseArgsIntoLists(args, runpathList, null, suitesList);
            fail();
        }
        catch (InvocationTargetException e) {

            // throw the InvocationTargetException unless the target
            // exception is NullPointerException, which is expected
            Throwable targetException = e.getTargetException();
            if (!(targetException instanceof NullPointerException)) {
                throw e;
            }
        }

        try {
            invokeParseArgsIntoLists(args, runpathList, reportersList, null);
            fail();
        }
        catch (InvocationTargetException e) {

            // throw the InvocationTargetException unless the target
            // exception is NullPointerException, which is expected
            Throwable targetException = e.getTargetException();
            if (!(targetException instanceof NullPointerException)) {
                throw e;
            }
        }

        args = new String[7];
        args[0] = "-p";
        args[1] = "\"mydir\"";
        args[2] = "-g";
        args[3] = "-f";
        args[4] = "test.out";
        args[5] = "-s";
        args[6] = "MySuite";
        runpathList.clear();
        reportersList.clear();
        suitesList.clear();

        invokeParseArgsIntoLists(args, runpathList, reportersList,
                suitesList);

        verify(runpathList.size() == 2);
        verify(runpathList.get(0).equals(args[0]));
        verify(runpathList.get(1).equals(args[1]));

        verify(reportersList.size() == 3);
        verify(reportersList.get(0).equals(args[2]));
        verify(reportersList.get(1).equals(args[3]));
        verify(reportersList.get(2).equals(args[4]));

        verify(suitesList.size() == 2);
        verify(suitesList.get(0).equals(args[5]));
        verify(suitesList.get(1).equals(args[6]));

        args = new String[9];
        args[0] = "-p";
        args[1] = "\"mydir\"";
        args[2] = "-e";
        args[3] = "-o";
        args[4] = "-r";
        args[5] = "MyCustomReporter";
        args[6] = "-s";
        args[7] = "MySuite";
        args[8] = "MyOtherSuite";
        runpathList.clear();
        reportersList.clear();
        suitesList.clear();

        invokeParseArgsIntoLists(args, runpathList, reportersList,
                suitesList);

        verify(runpathList.size() == 2);
        verify(runpathList.get(0).equals(args[0]));
        verify(runpathList.get(1).equals(args[1]));

        verify(reportersList.size() == 4);
        verify(reportersList.get(0).equals(args[2]));
        verify(reportersList.get(1).equals(args[3]));
        verify(reportersList.get(2).equals(args[4]));
        verify(reportersList.get(3).equals(args[5]));

        verify(suitesList.size() == 3);
        verify(suitesList.get(0).equals(args[6]));
        verify(suitesList.get(1).equals(args[7]));
        verify(suitesList.get(2).equals(args[8]));

        args = new String[10];
        args[0] = "-p";
        args[1] = "\"serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar\"";
        args[2] = "-g";
        args[3] = "-s";
        args[4] = "MySuite";
        args[5] = "MySecondSuite";
        args[6] = "MyThirdSuite";
        args[7] = "MyFourthSuite";
        args[8] = "MyFifthSuite";
        args[9] = "MySixthSuite";
        runpathList.clear();
        reportersList.clear();
        suitesList.clear();

        invokeParseArgsIntoLists(args, runpathList, reportersList,
                suitesList);

        verify(runpathList.size() == 2);
        verify(runpathList.get(0).equals(args[0]));
        verify(runpathList.get(1).equals(args[1]));

        verify(reportersList.size() == 1);
        verify(reportersList.get(0).equals(args[2]));

        verify(suitesList.size() == 7);
        verify(suitesList.get(0).equals(args[3]));
        verify(suitesList.get(1).equals(args[4]));
        verify(suitesList.get(2).equals(args[5]));
        verify(suitesList.get(3).equals(args[6]));
        verify(suitesList.get(4).equals(args[7]));
        verify(suitesList.get(5).equals(args[8]));
        verify(suitesList.get(6).equals(args[9]));
    }

Conclusion

Approach 1, testing private methods indirectly by testing the package-level, protected, and public methods that call them, will often be the best approach. In cases where you really do want to test private methods directly, using reflection to test private methods, although rather cumbersome, does provide the cleanest separation of test code from production code, and the least impact on production code. However, if you don't mind making those particular private methods you want to test package access, you could use approach 2. Or if you don't mind placing a nested test class inside your production class under test, approach 3 would at least let you keep the private methods private.
There isn't a perfect answer. But if you adopt approach 4, you will ultimately end up with a handful of methods like invokeStaticMethod that you can reuse. Once you write a convenience method, like invokeParseArgsIntoLists, for a private method, you can write tests against the private method without much difficulty.

Talk Back!

Do you ever test private methods? If so, what is your preferred apprach? Discuss this article in the Articles Forum topic, Testing Private Methods with JUnit and SuiteRunner.

About the Author

Bill Venners is President of Artima Software, Inc. and Editor-In-Chief of Artima.com. He is the author of Inside the Java Virtual Machine (Computing McGraw-Hill), a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project, whose ServiceUI API became the de facto standard for associating user interfaces to Jini services. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community.

Resources

1. The ServiceUI API defines a standard way to attach user interfaces to Jini services:
http://www.artima.com/jini/serviceui/index.html
2. Daniel Steinberg is currently the Editor-In-Chief of Java.NET:
http://www.java.net/
3. Artima SuiteRunner is a free open source testing toolkit and JUnit runner:
http://www.artima.com/suiterunner/index.html
4.JUnit FAQ question about testing private methods:
http://junit.sourceforge.net/doc/faq/faq.htm#tests_10
5. Testing Private Methods (Don't Do It), a weblog post by Charles Miller:
http://fishbowl.pastiche.org/2003/03/28/testing_private_methods_dont_do_it
6. Andy Hunt and Dave Thomas are the authors of Pragmatic Unit Testing, which is available at The Pragmatic Store.
7. JUnit Addons is a collection of helper classes for JUnit created by Vladimar R. Bossicard:
http://sourceforge.net/projects/junit-addons
8. PrivateAccessor is the class from JUnit Addons that facilates testing private members:
http://junit-addons.sourceforge.net/junitx/util/PrivateAccessor.html
9.PrivilegedAccessor class, which you can use to access private members:
http://groups.yahoo.com/group/junit/files/src/PrivilegedAccessor.java
10. Test Driven Development by Example, by Kent Beck, describes the test-first technique:
http://www.amazon.com/exec/obidos/ASIN/0321146530/
11. Test Infected, by Kent Beck and Erich Gamma, introduced JUnit to the world:
http://junit.sourceforge.net/doc/testinfected/testing.htm
Unit Testing Private Methods, a weblog post about nUnit by Ted Graham:
http://weblogs.asp.net/tgraham/archive/2003/12/31/46984.aspx
Subverting Java's Access Protection for Unit Testing, an O'Reilly OnJava.com article by Ross Burton:
http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html
Class RunnerSuite, from which the code snippets in this article were taken, appears in its entirety here:
http://www.artima.com/suiterunner/privateExample.html
Why We Refactored JUnit
http://www.artima.com/suiterunner/why.html
Artima SuiteRunner Tutorial, Building Conformance and Unit Tests with Artima SuiteRunner:
http://www.artima.com/suiterunner/tutorial.html
Getting Started with Artima SuiteRunner, How to Run the Simple Example Included in the Distribution:
http://www.artima.com/suiterunner/start.html
Runnning JUnit Tests with Artima SuiteRunner, how to use Artima SuiteRunner as a JUnit runner to run your existing JUnit test suites:
http://www.artima.com/suiterunner/junit.html
Artima SuiteRunner home page:
http://www.artima.com/suiterunner/index.html
Artima SuiteRunner download page (You must log onto Artima.com to download the release):
http://www.artima.com/suiterunner/download.jsp
The SuiteRunner Forum:
http://www.artima.com/forums/forum.jsp?forum=61