In your pilot, are you testing for usability or just looking for system bugs? If you including usability, there is a lot to consider. More formal information is available in this Wikipedia, but here are a few points off the top of my head:
Despite the URL pointing to a US Government site, this page also has relevant usability testing background information.
Rob makes some excellent points. Pilots can be tricky. The best advise I ever got on conducting pilots (or any kind of comparison experiment) was to do a Design of Experiment (DOE).
DOE is pretty formal and uses statistics to compare different treatments. For your pilot (experiment), you probably don't need all the fancy math. What you do need is to answer these questions:
This is clearly a shorter list than the whole method but is a good snapshot of how to go about proving that your solution adds value. There may be debate over what are "hard" savings as opposed to "soft" savings. Get those discussions done up front and drive toward the definition of success that everyone agrees with.