Evaluation

~ ~ SNEAKERS cannot unequivocally be stated to be a success, because only time will tell if it truly meets its primary goal of educating managers and engineers in the CE design methodology. However, there are some more immediate evaluations which were performed, against which SNEAKERS can be said to have succeeded.

The first measure of evaluation is in how well SNEAKERS matches the list of ingredients which were first introduced in Section . The purpose of this thesis, in part, was to develop this list and then to create a Concurrent Engineering tool which would demonstrate as many of these ingredients as possible, while always keeping in mind the educational nature of the project. The ingredients are listed again below, along with a short description of how SNEAKERS meets the needs of each.

1. Design Agents - There are 26 independent agents in SNEAKERS.
2. Multi-disciplinary Goal and Specification Representation - Several requirements and results are multi-disciplinary, but this would be considered marginally included.
3. A Catalog - The possible components are given in a palette during the design.
4. A Design Database - The COOL object list keeps the current design representation.
5. An Accumulated Database - Each of the agents can be accessed individually to see messages which that agent has provided to the user.
6. Shareability of Design Information - The blackboard structure allows all the agents to know everything about the design.
7. Communication - All agents in CLIPS communicate and they also communicate with the user, but there is no support for communication with other people.
8. A Manager - The user serves as the manager of the tower design.
9. A Design History - The scrollable message area has a complete listing of user actions together with agents' reactions.
10. A Checklist - The other information window attempts to serve this purpose, but this, too, may be considered marginal.
11. A Standard Interface - Motif provides a common interface that is familiar to many users, and the system itself has only a single interface which tries to remain consistent throughout the system.
12. Virtual Collocation of People - As this is a single-user system, this ingredient is not satisfied by this system.

Besides this evaluation, SNEAKERS was also evaluated by five graduate students at WPI, who all have experience in industry or with Concurrent Engineering. They were Sundar Victor, Peter McCann, Eric Rasmussen, Jingwen Liu, and Kathy Urbanowicz.

Sundar Victor, who is working toward his master's degree in Computer Science, was a member of WPI's Intelligent, Interactive, and Integrated Design (I3D) team which did Army funded CE research. He has also worked in industry.

Peter McCann holds a bachelor's degree in Computer Science, has worked for five years in industry, and is now pursuing his master's degree in Manufacturing Engineering. He was also a member of the I3D team.

Eric Rasmussen, a third member of the I3D team, is pursuing his master's degree in Manufacturing Engineering and has had significant computer experience.

Jingwen Liu has nearly completed his PhD in Computer Science, with a concentration on Artificial Intelligence in Design.

Kathy Urbanowicz has worked in industry, and has recently returned to school to work full-time on her master's degree in Computer Science.

Each of these people read the user's guide, and were able to ask questions for clarification. Their questions and comments on the user's guide helped to improve that document, and make it less ambiguous. They were also given a short introduction to the domain. They had all seen at least one presentation on SNEAKERS, so they were familiar with the project.

They were allowed to run the system alone, and made comments in an informal setting. Their comments were noted. Some of the comments suggested reasonable changes to the system, including changing the wording of some of the expert systems, changing the messages in the other information area, and guarding against some user actions which could cause errors in the system, but which had not previously been trapped. These comments are considered reasonable because either they could be satisfied with less than an hour of effort, or the consequences of leaving them were detrimental to the goal of the system. All of the reasonable comments were immediately coded into the system in under ten minutes and the evaluator was able to see the system with the change. Some of the more unreasonable comments included forcing the cursor to onto a particular palette choice when an agent suggests or advises that that choice be made and increasing the control over the user's actions.

Aside from these suggested changes, the other comments were that the interface was very helpful for the task; that the rules made sense to the users; that it was easy to use and that there was no need for someone to sit nearby to tell the user what to do; and that the highlighting of the agent's buttons as they make new comments really does convey the needs of other aspects in the design. As a consequence of using SNEAKERS, the members of the I3D team want to build a similar interface for their CE support system.

From these comments, it appears that SNEAKERS has lived up to the goals of being intuitive, easy to use, and educational. It is not perfect, but its flaws are outweighed by the usability of the system.