Interest in software networking has grown significantly since the introduction of Network Function Virtualization (NFV). Software switches are used in NFV to steer traffic between different virtualized network functions and physical Network Interface Cards (NICs). It is becoming more and more important to objectively evaluate and compare the performance of the multiple alternative implementations that have recently been proposed. A comprehensive performance analysis is still missing for two main reasons: (i) the amount of time required to configure and compare all such tools is enormous; (ii) it is very difficult to define a proper methodology to compare different solutions in a fair manner. In this paper, we propose a methodology based on four simple yet representative test scenarios used to evaluate the performance of software switches. We apply this methodology to measure throughput and latency metrics for 6 state-of-the-art software switches namely, OVS-DPDK, snabb, BESS, FastClick, VPP and netmap VALE. Our work constitutes a first step to building a better understanding of design tradeoffs and identifying performance bottlenecks.