About EEMBC
EEMBC Membership
EEMBC Benchmarks
Understanding EEMBC Benchmark Scores
The Certification Process
The EEMBC Test Harness (TH)
Application-Specific Topics

About EEMBC

Q: What is EEMBC's mission?
A: The Embedded Microprocessor Benchmark Consortium (EEMBC - pronounced embassy) is a non-profit organization that works collaboratively with member companies to develop performance benchmarks that target key applications of embedded systems. EEMBC activities include the EEMBC Technology Center, which delivers technical support to members and licensees, certifies benchmark scores, and oversees the development of benchmark software.

Q: Who are EEMBC's members?
A: EEMBC members include semiconductor manufacturers, intellectual property providers, compiler and RTOS vendors, and OEMs.

Q: Why aren't benchmark scores available for all embedded processors?
A: Members must agree to publish their benchmark scores. If a processor score is not available on the EEMBC website, you may request that score on line or directly with the corresponding processor vendor. Customer requests for scores on specific processors are a key mechanism by which more scores are published, and EEMBC encourages you to request certified scores for any processor you are considering for a design.

Q: What steps does EEMBC take to ensure that the published benchmark results are accurate and honest?
A: Only scores that have been verified by the EEMBC Technology Center may be publishedfollowing a rigorous testing process to ensure that benchmark scores are repeatable, obtained fairly, and according to the rules of EEMBC. The EEMBC Technology Center is also responsible for verifying the exact environment under which each test was run; this includes processor frequency, wait states, and compiler version.

Q: What role does EDN play in the consortium?
A: EEMBC traces its origins to a "hands-on project" conducted by Markus Levy at EDN Magazine in early 1996, and since that time EDN has provided EEMBC with the same gracious support it extends to other non-profit organizations in the semiconductor industry. Otherwise, there is no ownership or organizational relationship between EEMBC and EDN.

EEMBC Membership

Q: How do I qualify to be an EEMBC member?
A: You must be a designer or manufacturer for which the end products are microprocessors, microcontrollers or DSPs. Alternatively, you must be a designer or manufacturer of microprocessor development tools or manufacturer of products for which the primary end-product includes hardware, software, or firmware associated with microprocessors.

Q: What are the main steps in becoming a member of EEMBC?
A: First, please fill out the form you’ll find on the EEMBC Membership page and submit the requested information. We will contact you with full details regarding the process of becoming a member. Member companies are required to execute a set of standard agreements, including an Application for Membership, the EEMBC Bylaws, and a Member License Agreement. Membership dues are typically paid by check or wire transfer.

Q: Is my whole company included in our EEMBC membership? Can more than one person in a member company access the benchmark source code?
A: Once you have become an EEMBC member, as many people in your company as you like may download the EEMBC benchmark source code, but each such individual must be advised of the EEMBC rules and the provisions of the EEMBC Member License Agreement. Moreover, each person accessing the members-only area of the EEMBC Web site to download source code or obtain other information must register individually. You are also responsible for seeing that any employees who leave your company do not take the source code outside of your company.

Q: What sort of person should be the representative from our company to the EEMBC Board of Directors?
A: EEMBC Board of Directors meetings combine both technical and marketing-oriented discussions, and our member representatives include individuals whose roles in their respective companies are purely technical, purely marketing, and everything in between. In general, the most technical discussions in EEMBC take place on the subcommittee level by teleconference outside of Board meetings. Once your company has joined the Consortium, you are welcome to review the minutes of past Board meetings (posted in the members-only area of the EEMBC Web site) as a way of understanding the issues that are regularly discussed and decided by the Board.

Q: Do we own the benchmark source code once we have become a member?
A: No, EEMBC owns the source code; your membership makes you a licensee.

Q: What are the fees for joining EEMBC?
A: The fees for a new Board of Directors member are $30,000 for the first year and $16,000 for each subsequent year. However, if you want to join but don't want to be a Board member, you can join individual subcommittees (application area) where the cost ranges between $7,500-$12,000 for the first year ($6,000 for subsequent years) for each technical subcommittee that you participate on. Note that from time to time, dues may be raised or reduced as the Board members deem necessary.

Q: How are membership dues spent?
A: Your member dues go towards the development, maintenance, and marketing of the benchmarks as well as funding certifications and technical support from the EEMBC Technology Center.

Q: What's the difference between a board member and a technical subcommittee member?
A: A board member has full voting rights on all the major elements of the consortium. A board member has full access to all of the EEMBC benchmarks. A subcommittee member (application area) can only help determine the technical direction of the subcommittee(s) for which it has paid a membership fee. A subcommittee member has access to the benchmarks in the application area addressed by the subcommittee.

Q: Does EEMBC have a university program or special rate for academia?
A: Yes. For more information, visit the “Academic Members” area under the “Membership” section of the EEMBC Web site.

Q: Can I just "audit" EEMBC activities?
A: No, you must be an EEMBC member.

EEMBC Benchmarks

Q: Does EEMBC have benchmarks for DSPs?
A: Yes. Our Digital Entertainment and Telecom benchmark suites contain a number of kernels and applications that are highly relevant for DSPs. Furthermore, EEMBC’s rules allow benchmark code optimizations to promote support for specialized DSP instructions and architectures. For more information, visit the Digital Entertainment and Telecom areas on the EEMBC Web site.

Q: EEMBC has such a variety of benchmarks, how do I determine which one to use?
A: Focus on those parameters that most closely match your application needs. A designer typically uses the benchmark scores as a criterion to pick the best processor for the application. For example, if the application does many table look-ups, then that parameter (in the Automotive/industrial benchmark suite) is most important and the designer will like give the most weight to that benchmark. On the other hand, if the application is a modem, then the benchmark scores in the Telecomm suite become the most important parameters.

Q: What process does EEMBC use to select specific benchmark algorithms?
A: Algorithms are selected by discussion, consensus, and ultimately a vote of the member companies. Non-members are also invited to suggest benchmark algorithms that you would like to see the consortium develop.

Q: Does EEMBC perform board-level or system-level benchmarking?
A: The Version 1.0 benchmarks are processor benchmarks. However, the board and/or system can make a significant difference in processor performance (for example, the memory subsystem is one aspect that can dictate performance). EEMBC’s second-generation benchmarks, specifically for Networking and Digital Entertainment are system-level benchmarks. The same is true for EEMBC’s Java benchmarks which can be run directly on a mobile phone platform.

Understanding EEMBC Benchmark Scores

Q: What scoring methods does EEMBC use?
A: EEMBC allows two scoring methods: Out-of-the-box and Full-Fury. Out-of-the-box is where you use the compiler of your choice along with any compiler switches, but you cannot modify the original EEMBC source code. Full-Fury is the optimized version where you can hand tune the code to take advantage of any peripherals or hardware accelerators, special libraries, or assembly language. Several members have provided different levels of optimized scores.

Q: In out-of-the-box benchmarking, a vendor can use any compiler, but cannot alter the source code in any way. Does the compiler have a significant effect on the benchmark score, even in out-of-the-box scores?
A: Absolutely. Feedback from several processor vendors indicates that they have seen performance differences upwards of 40% simply as a result of using a different compiler. This is why vendors such as Green Hills, IAR, and Wind River are actively involved in the EEMBC process.

Q: What do the out of box scores reveal about compiler performance and/or processor performance?
A: Over time, as more processor vendors submit scores from a variety of compilers running on the same processor, this will provide very valuable information. Also, if you held the compiler family constant and varied the processor, this would be a great way to compare a processor's C friendliness.

Q: Must a compiler vendor be an EEMBC member to have scores posted on the EEMBC web site?
A: A compiler vendor's results on the web site are not related to whether or not that vendor is an EEMBC member. The processor vendor member typically makes that decision, and they usually publish scores from the compiler that yields the best scores.

Q: EEMBC benchmarks attempt to resemble real-world applications by allowing members to optimize the source code to take advantage of their architectures. How do you define the optimization guidelines so customers can still make apples to apples comparisons?
A: EEMBC-allowed optimizations are decided on a benchmark-by-benchmark basis. There are some fundamental restrictions in place to ensure fair comparison. EEMBC allows optimizations that change the source code to take advantage of libraries or hardware accelerators. These restrictions will allow you to demonstrate an architecture's capability and facilitate practical comparisons. You can also use intrinsics ( i.e. C code modification), or change the algorithm in some cases. For example, if EEMBC has a radix-4 implementation of an FFT, we will allow it to change to radix-2. So it is still an FFT, but on some DSPs that makes a big difference.

Q: What are the restrictions on information found on EEMBC's web site? How can I use the benchmark scores for various processors?
A: This topic is treated in full in the document Publishing Your Benchmark Scores: Questions and Answers on Procedures and Marketing Opportunities, available from the Members-only section of the EEMBC Web site. If you are not an EEMBC member, please direct your questions to one of the contacts listed on the "Press & Events" page on the EEMBC Web site.

The Certification Process

Q: What are the main steps in obtaining certified benchmark scores for my processor?
A: The major steps are listed here:

Q: Are any additional tools needed to run the benchmarks?
A: No. You need to download the benchmark source code and Test Harness and port these to your board or platform. For a fee, the EEMBC Technology Center can do this for you.

Q: What information goes into the Disclosure Workbook?
A: The Disclosure Workbook contains all the information necessary to recreate the benchmark environment. In addition to all the benchmark scores, the Disclosure Workbook contains board-level information (memory configuration, etc.), compiler information (settings, libraries, etc.), and processor information (core and bus speeds, cache sizes, on-chip peripherals, etc.). It also contains details on the types of optimizations that were used for a Full-Fury score. After certification is complete, and the scores are published, the associated Disclosure Workbook is publicly available alongside the benchmark scores.

Q: What does the certification process cost?
A: As an EEMBC Board member, you are entitled to two free certifications per year. Certifications are available to subcommittee members on a fee basis.

The EEMBC Test Harness (TH)

Q: What is the motivation behind the Test Harness (TH)?
A: The Test Harness (TH) is a necessity to maintain extremely portable, honest benchmarks. The TH provides a standardized interface to each kernel so that you do not have to port each kernel, just the TH. This provides a standard API for dealing with timers and the serial port host-target interface, and for file I/O. Parenthetically, the TH has been ported to more than three dozen architectures so far and has proven a reliable method of doing industry-standard benchmarking. It is analogous to SPEC's test harness (although designed quite differently), BAPCo's test setup, and every other benchmark suite. The TH is one of the key components that allows us to conduct certification work inexpensively, passing that savings on to each vendor, and ensures that EEMBC's test results are truthful and accurate.

Q: What is the basic Test Harness structure?
A: The TH is divided into two parts: the functional layer (in the src directory) and the platform-dependent portion (in the al, or adaptation layer). The TH assumes that you have a small number of C library functions. When the TH was created, all EEMBC members had C libraries, including all the DSP vendors.

Q: Does the Test Harness affect the benchmark score?
A: It is impossible to ensure that the TH and communications with the host have no impact on the results. It is only possible to ensure that it has an inconsequential impact. The issue is the relatively small amount of time it takes for th_signal_start() and th_signal_finished() to execute, and for the '>> START!' and '>> FINSIHED!' messages to be transmitted to the host and seen by the Host Control Program. If tests are only executed for very short time periods, then the communications latency between the target and the host would skew the results. Benchmarks need to run for a relatively long period of time, a minimum of two seconds, but preferably up to thirty seconds.

Q: Should the Test Harness use the target or host timer?
A: Better accuracy is obtained if a TH port implements the target timer. But even this method is problematic. For example, if we mandated that tests run for a short period of time, one second for instance, then we would need a timer accuracy and resolution of at least 1ms to get good results. Running a timer interrupt at 1ms skews the results! In this case, it can actually produce more accurate results by letting the host time the test. Many systems do not have high-accuracy, low-overhead timers. In fact, many target platforms do not have on board timers. This is exactly why the TH supports host-based timing and why we have the Host Control Program.

Application-Specific Topics

Q: Why does EEMBC have an FFT filter benchmark in both the Telecommunications and Automotive/Industrial benchmarks?
A: FFTs are used more often in Telecomm applications, but they are becoming increasingly important in automotive and industrial applications. Sensors used in engine knock detection, vehicle stability control, occupant safety systems and flow control, all require sophisticated filtering of the incoming signal to determine what is really happening among a lot of noise.

Q: Why don't the Telecomm benchmarks use floating point?
A: All of the kernels (Bit Allocation, AutoCorrelation, FFT/iFFT, Viterbi, Biquad IIR, Convolutional Encoder) in the suite are intended to be run (for comparison purposes) using 16-bit fixed point math. There are FP versions of some of the kernels which were used to generate the golden data file sets. At this time, we have no plans to extend the "Out of the Box" code to require/use FP math. You may re-code the kernels, using FP for an optimized version of the benchmark if you wish. At this time, we do not have access to any floating point data sets. The data sets which accompany the AutoCorr kernel are in 16-bit (1.15) fixed point format (that is: 1 sign bit, 15 bits of data) to reduce the precision of the data, you can remove (or zero) bits from the LSBs of each number.