Abstract

Software Engineering, the art and science of building better software, is a core requirement of most computer science curriculum. Many departments have been renamed from “computer science” to “computer science and software engineering”.  In this paper we present a snapshot of software engineering knowledge and use within academia. Just like the bare footed cobbler, so too do our best and brightest computer science researchers often abandon software engineering when developing software to assist or demonstrate their research. Some believe software engineering is inappropriate for research software, however our research, conducted both qualitatively and quantitatively using interviews, case studies and surveys, has indicated that a moderated approach can have positive benefits both for the researcher and the long term use and further development of their work.

1. Introduction

Computer science is the art and science of problem solving with the aid of computers and the study of how this can be achieved. The history of computer science often starts with the abacus (around 3000BC), however modern Computer Science is a recent field and can be dated from the publication of Alan Turing’s article “Computing machinery and intelligence” which introduced the Turing machine (a mathematical model of a general computing device) in 1950 [1].

Software Engineering, the engineering discipline for developing software, began as a field of computer science but is now often considered a field in its own right. The term was coined as the title of a NATO Science Committee sponsored conference in Garmisch, Germany in 1968 [2]. The conference was aimed at solving the “software crisis” where by software projects were regularly running hugely over budget and over time. A follow-on conference in 1969 discussed ways to make software development more “engineering like” [3].

Today, software projects still regularly run over budget and over time. The scale of the problem in the industry has been greatly reduced through the application of software engineering and better project management. Software engineering is now a core component in Computer Science degrees, as well as being offered as a more specialised degree in its own right.

Our research has examined the level of use that a number of different software engineering methods receive from academic researchers. We have also measured characteristics of the researchers and their projects to find correlations between software engineering usage and particular outcomes such as publication rates and funding. We have discovered some resistance to applying software engineering in academia, and through the use of interviews and case studies we have examined this problem and formulated a new software development life cycle that addresses the specific needs and concerns of academic researchers.

In this paper we present our results showing the current levels of software engineering in academia. We examine why researchers abandon software engineering as taught to undergraduates. We present our RAISER/RESET SDLC (software development life cycle) for academia. Finally we discussing a view of the future for computer science departments, making use of the RAISER / RESET approach to improve research outcomes and efficiency.

2. A brief history of Software Engineering and Academia

In 1950 Turing [1] noted that improvements in both programming and engineering were needed for artificial intelligence research to succeed.

In 1970 Royce [4] introduced the Waterfall. This was the first published SDLC. Royce began with a two-step approach: “analysis” leading into “coding”. He stated this was sufficient for small projects operated only by the developers. His waterfall SDLC then showed how large-scale software development could bring a product to an operational state in a number of well defined steps.

In 1988 Boehm [5] introduced the Spiral SDLC. Past models had discouraged code reuse. The spiral uses prototyping and repeated risk assessment to mirror the evolving nature of software.

In 1989 Software Engineering became a core part of the computer science curriculum as endorsed by the ACM (Association of Computing Machinery).

In 1998 Robillard and Robillard [6] compared industry and student development work. They found that the programming phase dominated university work. Humphrey [7] arrived at the same conclusion, and adding that unless directed, undergraduate students failed to use the software engineering they had been taught. Humphrey did not examine research students, but described the common undergraduate ethic as “ignoring planning, design and quality in a mad rush to start coding”.

In 2000 Cook, Ji and Harrison [8] suggested focusing on design activities and the adaptability of the software process rather than repeatability as with other engineering fields. Cook et al. [8] recommended four steps to software evolvability: analyse to show which parts of the system may need to change, implementation of the change, restabilising the product (to avoid the introduction of new errors), and testing the changed product.

2. The curriculum

Since 1989 the computer science curriculum, as endorsed by the ACM has including “Software Methodology and Engineering” as one of its key requirements. It aimed to teach how software can be designed for understandability and modifiability [9].

More recently, a software engineering curriculum has been developed. Successful completion allows software engineering graduates to qualify as certified engineers.

Despite this emphasis in the curriculum, Software Engineering methods are seldom used by academic staff when developing software as part of their research. Anecdotal evidence suggests the failure to apply software engineering in research development is commonplace and results in expected problems. Prior to our 2002 research [10] no formal work appears to have been done to examining the level of use of Software Engineering use in academia or its appropriateness.

While Software Engineering is seen as a vital industry skill to teach our students, within our departments we have been slow to adopt it. Once adopted it still needs adaptation to better suit the needs of research.

3. Research Methods

Both quantitative and qualitative methods have been used in our research. These including case studies, surveys, interviews and correspondence with software engineers and researchers.

Survey findings were compared to researchers experiences. The case studies were compared to other projects’ experiences as discussed in the interviews as well as to survey trends. The US survey led to alterations and improvements before the Australian survey was released. Full results can be found in [10], some key conclusions from these results can be found in  [11].

3.1 Survey

They survey was Internet based, and advertised via e-mail. Invitations to participate were send to the heads of the computer science (or similar) departments who were then asked to forward it to their staff. 255 department heads were e-mail in the United States and from these 29 survey responses were collected. These came from 17 universities. In Australia 34 department heads were e-mailed and 35 responses were collected from 13 universities. 15 of these were from Monash University.

Correlations were calculated from the survey results using the Spearman rank order correlation (for ranked responses) and the Pearson product moment correlation (for real values). In both cases a t-test was used to calculate the probability that there was no correlation. The results of this analysis will not be discussed in this paper, but have been used to formulate our RAISER/RESET SDLC.

3.2 Interviews

There were 12 formal interviews. All were conducted privately and tape-recorded within a one-week period.

3.3 Case Studies

Case study involved: observations from meetings and presentations, interviews with developers and users, reviews of internal documentation, meeting minutes and published literature and field testing products. The case study projects included:

4.3.1 CaMML (Causal Discovery via MML)

Developed by Wallace and Korb [12], CaMML creates causal networks that would best explain the set of data provided. Four versions have been developed; three were based on the original code (with modifications) and developed without any regard to software engineering. Korb [13] felt the work had suffered for this. The latest implementation (managed by Korb) is making some use of software engineering and plans to address issues including maintainability, readability and extensibility.

3.3.2 CDMS (Core Data Mining Software)

CDMS was inspired as “there was no common platform for people to carry-out data mining …different programs spat out different forms of data and no one knew how to use any of the programs apart from the author” [14]. There is a plan to publicly release software and a manual.

3.3.3 The GIFT (GNU Image Finding Tool)

The GIFT [15] is a framework for content-based image retrieval (CBIR) systems. During development, one of the major goals was to produce a modular, extensible framework based on pluggable components. This was to allow more focus for research students and reduce unneeded overhead. This project is in its maintenance phase and has been a huge success. The GIFT has allowed research students to take on work not possible before due to the limited time frame of their research. In one case a student was able to add support for query-by-region to MRML (Multimedia Retrieval Markup Language) in the relatively short duration of a masters minor thesis. This is seen as a significant contribution to the research area, and one that could not have been contributed by a research student prior to the availability of the GIFT.

4. Results and Discussion

4.1 A Snapshot

In this section we will present our snapshot view of the level of Software Engineering knowledge and use within computer science departments. It is important to note that the US results come from a skewed sample. The original e-mail invitation as sent to the US included the word “Software Engineering” in the title, survey and body of the e-mail. While it also contained the words “computer science” (in all the same places), once department heads read software engineering, many forwarded the e-mail only to their software engineering teaching staff.

This view of a segregation of software engineering into a specific subset of computer science not relevant to other academics is supported by a collection of e-mails from heads of department informing us how they had assisted by filtering the e-mail. It is also supported by the US survey results, where 72% of respondents had taught software engineering compared with 43% in the Australian results. The words “Software Engineering” had been removed from the e-mail as well as the survey prior to its release in Australia.

The field of study of survey participants was fairly consistent between the surveys. In both cases the vast majority, 22 in Australia and 18 in the US had been trained as computer scientists. In both cases the rest of the sample was made of 2 mathematicians, 3 software engineers and the remainder from various other fields.

4.1.1 The Tools and Methods

Before going through specific “software engineering tools”  (described as “tools and methodologies” to the Australians) participants were asked if they had used graphical models to design software. The majority responded yes, 68% in the US survey and 74% in Australia. Note that the US has the lower response to this question (designed to catch almost any planning tool), despite having a much higher percent of Software Engineering educators. The percentages alone suggest that some software engineering educators have never tried to apply what they teach to their own work.

The high positive response to the question of graphical models may look like positive use of software engineering. In fact the question only asks if the respondent has ever used any form of model (positive responsive are therefore so vague that they are just about  meaningless).

We will now examine a number of specific tools. In these questions survey respondents were asked if they knew the tool / methods and if so, how often they used it. Their choices were: Don’t know this, never use it, rarely use it, sometimes use it, use it often, and use it always.

In the US, flow charts were unknown to 1 person (4%), and not used or used rarely by 60%, and only 8% used them often or always.  In Australia 1 person did not know about flow charts (3%), 63% never use flowcharts and a further 20% chose to use flow chart rarely. 9% used flow charts often and no one always used them.

Class diagrams, a key design tool of the Object Oriented methodology (the main stream programming methodology at present, used with languages such as C++ and  Java) was unknown to 24% of US respondents. 28% used it often or always, and 12% chose not to use it. In Australia it was unknown to 20%, and used often or always by 20%. 26% chose not to use it.

These figures show that both older tools like flow charting and current tools like class diagrams are unknown, or ignored by choice by our researchers. The methodologies fair just as badly.

The waterfall model (previously mentioned) was unknown to 28% of US academics. A further 24% chose never to use it. 12% always used it, this is something of concern given that it has been superseded for most types of projects. The spiral model, currently the most popular model in industry, also faired badly. 36% of researchers did not know it. A further 8% chose never to use it.

In Australia 29% did not know the waterfall model and 40% chose not to use it. This gives the appearance of progress until the spiral model is examined. 40% did not know the spiral model; a further 29% chose not to use it. Other software development life cycles (including Rapid application design (RAD) and the fountain SDLC) were also examined with similar results, namely that a majority fell into the category of knowing but not using that SDLC or not knowing it at all. Later in the survey participants were asked which SDLC they most often used. As shown in figure 1, the unplanned approach was most popular in Australia.

In the US, with its skewed sample population, 7 people (35% of those who responded to the question) indicated they usually use the spiral, with the next highest group being those who declined to answer the question. Only 20% indicated they used the unsystematic approach, and a further 20% indicated they used some other systematic approach of their own. 15% used the waterfall and the remaining 10% used RAD. Given the sample population these results should not be taken as typical of US academics.

Returning to the Australian results, the lack of a systematic approach needs some explanation. When asked why they used the tools and methods they did, respondents indicated their motivation was a potential time saving over the life of that particular project and a method for clarifying their ideas. The most popular reasons why other methods were not used included a view that they were inappropriate for the task or would be of greater cost than benefit. This idea was examined further though interviews and case studies.

4.1.2 The benefits of Software Engineering

Interviews with researchers and postgraduate students have highlighted problems that some level of software engineering could mitigate or prevent. In some cases we found software engineering practices like version control were being used by some researchers, inevitably these same researchers had suffered for not having them in the past. In addition to lost work, (or the need to retrace to an earlier version) other problems include:

• Recreating the wheel, the problem of lost code or code so complicated it is cheaper to start over from scratch.
• Reinventing the clock to change the time, the problem of needing to redevelop an entire software tool in order to make a small change for ones own research.
• Understanding research code, in many cases code is obscure and new research students (or even the author after a few months) can have great difficulty deciphering it.
• Authenticity, often code is not publicly available or even if it is it may not be easy to check. This means experimental method, let alone results, can be hard or impossible to check.

Understandable code, helpful design documents and more evolvable i.e. maintainable code are all goals software engineering has been working towards since its inception. In additional we found some strong correlations between certain software engineering practices and research outcomes. Not all of these correlations were positive. Our case studies indicated the importance of keeping the needs of research in mind.

4.1.3 The needs of research

In personal communications prior to the surveys release, Pressman  [16] and Brooks [17] both observed that the aims of research software are different to those of industry, and as a result they would have different requirements. The focus of Software engineering has long been on industry and meeting its needs. Wallace, an emeritus professor with over 40 year experience in computer science research, observed that despite changes in industry he had observed little change in research practice since he started in the field [18]. In some respects he felt we were going backwards and becoming too reliant on trial and error, compile and run, approaches.

Waite observed “the major problem [in applying software engineering to research] is that research projects tend to be opportunistic rather than planned” [19]. This makes planning up front difficult if not impossible. Meyer pointed out that is the exception that something works [20]. The cost of overheads may need to be outlaid many times before “something works”. This combined with the fact that project aims tend to be fairly flexible and subject to change means that traditional approaches have very little chance of success. While some aspects of software engineering may be more appropriate than many think, over all the survey participants were correct to think they were uneconomical if applied strictly as per industry. Pressman [16] suggested that an approach for the research community would need to be “agile”, or would have little chance of adoption. This would overcome many of the overhead costs. These ideas, as well as other benefits of Software Engineering were demonstrated in the case studies.

4.2 Lessons from the Case studies

In the CDMS project design documents (specifically UML) were useful early on, but rapid changes to the research required regular updates to these document. As the project primarily involved two postgraduate students who were “living and breathing it” [14], the effort in maintaining these documents soon far out weighted their usefulness. The documents were abandoned. The need for documents, specifically a user manual is recognised by the developers, but other priorities (much as their theses) took precedence. Their supervisor admits “We’re not sure how this will happen. We were sort of hoping it would happen by magic or delivered by a stork” [21].

In the case of CaMML a functional specification and design documents were developed before redevelopment started. The team was much larger than that of CDMS and shortly after coding started the developers (who had not designed the spec) decided to create their own simplified design as they went along rather than follow the design presented to them. Part of their motivation was a desire to simplify their task and get on with their own research (adaptation to CaMML).

The GIFT was created to reduce the amount of duplicate work taking place at two universities working in the same research area. The project took existing work, rewrote parts of it, and generally redesigned it. At the start of the project a large effort was invested in creating a design that was modular and easily extensible. Initially this effort was seen by some in the project a waste of time. The investment soon paid off when new features requested by a reviewer were added in a matter of days… these changes would normally have taken months and possibly caused the paper to miss its deadline. Design documents were not produced for the GIFT, but an effort at reverse engineering them is currently underway. The GIFT is now part of the GNU project and is used other researchers around the world for their own research. Once such researcher stated that that her reason for using the GIFT was an expectation that it would save time and the softwares platform independence. A “how to” guide for the GIFT does exist and this particular researcher had found it quite helpful.

These case studies have presented three approaches. CDMS tried to maintain documentation during the projects developed (as would be done in industry). CaMML attempted a waterfall style approach with design completed before coding began. The GIFT took existing research and reworked it before developing it further. In addition, the GIFT was design specifically for reuse and future (unspecified) extensibility.

In the research setting it is the basic approach of the GIFT, refined though feedback from the other case studies, survey results and interviews that we recommend for research. This approach, extended for the longer term we have published as the RAISER/ RESET SDLC [10][11]. While a full introduction is beyond the scope of this paper, we will now briefly present the RAISER / RESET methodology.

5 RAISER / RESET

The 1993 EDRC workshop concluded that it was time to “adopt a more comprehensive approach to software development—even within a research setting—and for establishing a better infrastructure for software design, maintenance and reuse” [22]. RAISER / RESET aims to achieve this. In addition we aim to reduce the burden on researchers, during the research phase of a project, to a bare minimum.

The RAISER/RESET approach (see Figure 2) divides work into research and development cycles. The aim is to ensure that the both tasks occur and support, rather than hinder, each other. There is an increased overhead for development, but this already exists and is being paid by research students. The RAISER/RESET SDLC shifts the burden to a more proficient, specialised unit.

5.1 Overview

The RAISER/RESET SDLC is divided into two halves.  Research occurs in the top “RAISER” half, while development takes place in the bottom “RESET” half. A distinction is made between initial research and follow-on research. More than one “initial research project” can go into a single development phase. Likewise, a development phase that produces stable software may spawn many new “follow-on research” projects.

5.1.1 Initial research

Research software begins with an idea. The idea is coded, considered, tested, changed, recoded, etc. and eventually publication occurs. After the initial exploratory research, development is needed to clean code and produce design and user documentation. This “stabilisation” [8] not only makes the work more usable and “evolvable”, but also lowers the barrier for future researchers.

5.1.2 Stable Platform

. The “resetting” of the code aims to produce readable, documented, modular and reusable code. Development should take place after a research project finishes and before a new project wishes to build on it. Researchers may advise during the RESET (development) phase, clarifying their code and ensure any changes to not alter the integrity of the product. The actual changes should be made by a specialised team of software engineers.

5.1.3 Follow-on Research

Follow-on research builds on the existing research. It may include the development of new algorithms or  the addition of complementary new functionality. All researchers wanting to build on existing work should begin with a stable platform. At present a lot of time is regularly wasted as researchers attempt to understand each others undocumented, badly commented and often obscure code. Well-structured, readable code and documentation can greatly reduce this.

5.2 RAISER

RAISER (Reactive Assisted Information Science Enabled Research), aims raise the evolvability and readability of research code without making it a burden to researchers. The four features of RAISER are described below.

Reactive: RAISER is reactive rather than proactive. Changes to research ideas cause changes to code and/or approach. The nature of the project dictates what software engineering tools should be used. (Some idea from our research are presented in the tech report [11]).

Assisted: The code and methodology should assist the researcher, not burden them.

Information Science Enabled: This stresses the theoretical research behind the software development.

Research: The RAISER phase should only occur while new research is being done. Once all or a significant (publishable) part of the research is completed the project should migrate to the RESET phase.

5.3 RESET

RESET will differ somewhat from the usual development cycle as it is always based on research software and the working research software already exists. The features of RESET are:

Research Enabled: Software based on a research might be unique in nature and not fit any common design pattern. There may be an opportunity to abstract new design patterns during the conversion.

Software Engineering: In the RESET phase the prototype is cleaned up and remoulded. The code is  restructured for interface and robustness improvements. This task is mainly one of software engineering, the existing functionality should change. Additional documentation, both for users and developers (i.e. computer scientists who will further develop the research and the group applying RESET who will be maintaining it) may be created.

Techniques: The RESET process aims to produce a certain level of software quality and reusability. How this is a achieved may vary project to project, given that the starting position will depend on the raiser phase and the researchers who were involved.

6. A view of the future

As Computer Science matures a vast array of past research is building up. This will eventually form the basis of future research in the field. Not so long ago Royce’s two step approach could be applied by academic researchers who build software for their own use. Today research is easily shared via the Internet with colleagues all over the globe. Machine independent programming languages and the affordability and wide spread use of computers mean almost anyone can read, compile and run the programs our researchers write. Why then is there such resistance to university created software?

To our mind the answer is one of quality. The research may be first class, but the research product is usually not. Research work is often left in a semi-complete state and shelved. Unless directed to it by a supervisor it is often difficult for research students to find existing code in order to extend past work. When such work is found, continuing past work often becomes inefficient as either part of the code is missing and needs to be recreated, or  if all of the code is there, it is badly written and needlessly hard to understand and modify. Under pressure of time research students, who do most of the actual coding in most computer science departments, end up leaving their own code is a state just as bad (or worse) then that they were originally presented with.

If computer science were resourced like other sciences, rather than on the model of mathematics, as is usually the case, the setting up of experiments would become the task of professional lab assistants not scientists. In the case of computer science where one experiment ideally becomes the equipment for the next, this could be implemented using an in house software development lab to implement the RESET phase of our RAISER/RESET SDLC as already described.

An in house software development lab would decrease the burden on researchers, particularly research students who are being pressured to complete their degrees in shorter time periods than ever before. In addition it would increase code quality and reuse. A professional unit could do this efficiently and ti would be in their interest to make research code maintainable.

We stress that such a department would be a service department within the computer science department and its purpose would be to raise software quality and reduce the burden for researchers extending past work. While there is an over lap with current trends in the commercialisation of universities and computer science departments in particular, the issue of commercialisation and that of software quality for the researchers themselves should be considered separately.

7. Conclusion

In this paper we have investigated the use of software engineering by computer science researchers. We noted a lack of use of software engineering and discussed reasons behind this, including the inappropriateness of software engineering if applied as it is in industry. We discussed three case studies that took different approaches to development and presented our new SDLC for the research environment. Finally we have presented a view of the future for computer science departments in academia. With increased pressure on researchers and particularly research students a change in our approach is needed.

References

[1] Turing, A. (1950). “Computing machinery and intelligence”, Mind LIX(236): 433-60.

[2] Naur, P. and Randell, B. (eds) (1969). “Software Engineering: Report on a conference sponsored by the NATO SCIENCE COMMITTEE”, Garmisch, Germany, 7th to 11th October 1968, Scientific Affairs Division, NATO.

[3] Randell, B. and Buxton, J. (eds) (1970). “Software Engineering Techniques: Report of a conference sponsored by the NATO Science Committee”, Rome, Italy, 27-31 Oct. 1969, scientific Affairs Division, NATO.

[4] Royce, W. W. (1970). « Managing the development of large software systems: Concepts and techniques”, 1970 WESCON Technical Papers, Vol. 14, Western electronic Show and Convention, Los Angeles, pp. 1-9. Reprinted in Proceedings of the Ninth International Conference on Software Engineering, Pittsburgh, PA, USA, ACM Press, 1989, pp.328-338.

[5] Boehm, B. W. (1988). “A spiral model of software development and enhancement”, IEEE Computer 21(5): 61-72.

[6] Robillard, P. N. and Robillard, M. P. (1998) “Improving Academic Software Engineering Projects: A Comparative Study of academic and Industry Projects”, Annals of Software Engineering, 6:343-363.

[7] Humphrey, W. S. (1998). “Why don’t they practice what we preach?”, Annals of Software Engineering, 1(4): 201-222.

[8] Cook, S., Ji, H., and Harrison R. (2000), “Software evolution and software evolvability”, Working paper, University of Reading, UK.

[9]Denning, P., Comer, D., Gries, D., Mulder, M., Tucker, A., Turner A., and Young, P.  (1989), “Computing as a Discipline”, Communications of the ACM, 32(1)

[10] Oboler, A (2002), “Investigating the use of Software Engineering in Computer Science Research”, Honours Thesis, School of Computer Science and Software Engineering, Monash University, Australia. http://www.csse.monash.edu.au/hons/
projects/2002/Andre.Oboler/

[11] Oboler, A., Squire, D. and Korb, K. “Why don’t we practice what we teach? Engineering Software for Computer Science Research in Academia”, Tech. Report 2003/139, School of Computer Science and Software Engineering, Monash University, Australia, 2003.

[12] Wallace, C. S. and Korb, K. B. (1999). “Learning linear causal models by MML sampling”, in A. Gammerman (ed.), Causal Models and Intelligent Data Management, Springer-Verlag.

[13] Korb, Kevin (2002). Interview with Dr Kevin Korb, conducted for the USE CSR project.

[14] Comley, Josh (2002). Interview with Josh Comley, conducted for the USE CSR project.

[15] Squire, David McG., Müller, Henning, Müller, Wolfgang, Marchand-Maillet, Stéphane, and Pun, Thierry, “Design and Evaluation of a Content-based Image Retrieval System”, In Design and Management of Multimedia Information Systems: Opportunities and Challenges, 7, pp. 125-151, Idea Group Publishing, 2001.  http://www.gnu.org/software/gift/gift.html

[16] Pressman, R. (2002). Re: SE practise in Comp. Sci. Research, personal communication.

[17] Brooks, Jr, F. P. (2002). Re: Use of software engineering in computer science research, personal communication.

[18] Wallace, Chris. (2002). Interview with Prof Chris Wallace, conducted for the USE CSR project.

[19] Waite, W. (2002). Re: Use of software engineering in computer science research, personal communication.

[20] Meyer, Bertrand. (2002). Interview with Prof Bertrand Meyer, conducted for the USE CSR project.

[21] Allison, Lloyd (2002). Interview with Dr Lloyd Allison, conducted for the USE CSR project.

[22] Steier, D., Coyne, R. and Subrahmanian, E. (1993). “Software doesn’t transfer, people do—(and other observations from an EDRC workshop on the role of software in disseminating new engineering methods)”, Technical report, Carnegie Mellon University.

http://www.personal.rdg.ac.uk/~sis99scc/desel/desel_result.html

About the Author:

Andre Oboler, School of Computer Science and Software Engineering, Monash University, Australia.

Download the paper, see the presentation slides.

Full citation: Andre Oboler (2003). Examining the use of Software Engineering by Computer Science Researchers. In Proceedings of Education Students’ Third Regional Research Conference, Graduate School in Humanities University of Cape Town, Cape Town, South Africa. pp. 37-45, 26-27 September 2003.