Software Evolution

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

Basic Definitions

Software evolution is the term used in software engineering (specifically software maintenance) to refer to all programming activity that is intended to generate a new software version from an earlier operational version.

— Lehman & Ramil 2000

Software maintenance is the modification of a software product after delivery to correct faults, to improve performance or other attributes, or to adapt the product to a modified environment.

— IEEE 610-12-1990

Evolution and Maintenance

  • Often, the terms evolution and maintenance are used interchangeably.

  • Some authors consider that maintenance preserves software from failure whereas evolution changes the software from a worse to a better state.

  • Others consider maintenance to be set of planned activities whereas evolution concern whatever happens to a system over time.

Types of Software Maintenance

There are 4 types of maintenance, wrt the goal of the maintenance activity: (i) Adaptive, (ii) Corrective, (iii) Perfective, and (iv) Preventive.

Maintenance Chart
Adaptive Maintenance

«Adaptive maintenance is the maintenance activity performed to make a computer program usable in a changed environment.»

  • Occurs as a result of external influences or strategic changes.

  • For instance:

    • The government creates new taxes or changes the VAT rate.

    • A bank buys another one and their system must be integrated into their own system.

Corrective maintenance

«Corrective maintenance is the maintenance activity performed to correct faults in hardware or software.»

  • Occurs when customers detect a deviation of the software from its expected behavior, which was not detected during development.

Perfective Maintenance

«Perfective maintenance is the maintenance activity performed to improve the performance, maintainability, or other properties of software.»

  • Occurs after the deployment, when costumers ask for minor improvements in the software.

  • For instance:

    • Better validation, simpler data input, better performance.

Preventive maintenance

«Preventive maintenance is the maintenance activity performed for the purpose of preventing problems before they occur.»

  • For instance:

    • Improve maintainability: modularity, testability, understandability, portability, etc.

Maintenance classification

Maintenance Classification
legacy code
A legacy software system is any system that significantly resists modifications and changes while remaining of significant value for its owner. The system might have been developed using an outdated programming language or an obsolete development method. Most likely, it has changed hands several times and shows many signs of degradation.
Legacy code often differs from its suggested alternative by actually working and scaling.
FAQ: What is "legacy code"?
— Bjarne Stroustrup's
Reengineering is the examination and alteration of a subject system to reconstitute it in a new form and the subsequent implementation of the new form. Reengineering generally includes some form of reverse engineering (to achieve a more abstract description) followed by some form of forward engineering or restructuring. This may include modifications to fulfill new requirements not met by the original system.
reengineering

«Forward engineering is the traditional process of moving from high-level abstractions and logical, implementation-independent designs to the physical implementation of a system.»

Forward engineering

Reverse engineering is the process of analyzing a subject system to identify its components and their interrelationships and to create representations of the system in another form or at a higher level of abstraction.

Reverse engineering generally involves extracting design artifacts and building or synthesizing abstractions that are less implementation-dependent.

reverse engineering

Reverse engineering with Modisco

Modisco overview

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

Software Aging

Aging

«Programs, like people, get old. We cannot prevent aging, but we can understand its causes, take steps to limit its effects, temporarily reverse some of the damage it has caused, and prepare for the day when the software is no longer viable.»

A sign that the software engineering profession has matured will be that we lose our preoccupation with the first release and focus on the long-term health of our products. Researchers and practitioners must change their perception of the problems of software development. Only then will software engineering deserve to be called engineering.
— Parnas 1994

Why software ages?

  • Maintenance activities.

  • Ignorant surgery and architectural erosion.

  • Inflexibility from the start.

  • Insufficient or inconsistent documentation.

  • Deadline pressure.

  • Duplicated functionality (code duplication)

  • Lack of modularity

Software Change is inevitable!

Changes
  • New requirements emerge when the software is used

  • The business environment changes

  • Errors must be repaired

  • New computers and equipment are added to the system

  • The performance or reliability of the system may have to be improved

  • A key problem for all organizations is implementing and managing change to their existing software systems.

Importance of Evolution

  • Organizations have huge investments in their software systems – they are critical business assets

  • To maintain the value of these assets to the business, they must be changed and updated

  • The majority of the software budget in large companies is devoted to changing and evolving existing software rather than developing new software

Cost of Maintenance in a Software Life

Cost

Thomas M. Pigoski. 1996. Practical Software Maintenance: Best Practices for Managing Your Software Investment (1st ed.). Wiley Publishing.

Changes Induce Technical Debt

  • As a change is started on a software system, often there is a need to make other coordinated changes at the same time in other parts of the software.

  • These other required changes, if not completed immediately, incur a kind of debt that must be paid at some point in the future.

  • This technical debt is “the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution”.

Changes Induce Technical Debt (Cont.)

Living with Technical Debt
  • Just like financial debt, if technical debt is not repaid, the uncompleted changes accumulate interest on top of interest, because of increased entropy.

  • The longer you wait to make the changes, the harder it becomes.

Change is Hard to Anticipate

All Customers Needs
Many changes cannot be anticipated at design time:

The fundamental problem, supported by 40 years of hard experience, is that many changes actually required are those that the original designers cannot even conceive of.

2000
— Bennett and Rajlich

The Uncertainty Principle in Software Engineering

Uncertainty is inherent and inevitable in software development processes and products.

— Ziv and Richardson 1996

Key challenge for organizations: implementing and managing change to their existing software systems

Software Evolution

Python Evolution
Figure 2. Python Evolution Storyline

Ogawa, Michael, and Kwan-Liu Ma. "Software Evolution Storylines." (2010).

Evolution may arise

  • During software development, where the design evolves and matures as the understanding of the problem to be solved and how to solve it gradually increases

  • During software maintenance: after deployment, in the continuing process of adapting the system to the changing needs of its users and usage environment

  • When the system’s continuous evolution made it too complex to maintain: the system may requiring a more extensive restructuring, redesign or even a full reimplementation or replacement

Static vs. Dynamic Software Evolution

Static vs Dynamic Evolution
Static evolution
  • changes are applied manually by a human programmer

  • part of the software gets adapted or replaced by a programmer and the evolved software is redeployed

Dynamic evolution
  • changes are applied automatically at runtime

  • to better suit the current needs of the software system

  • by automatically generating, adapting, transforming or selecting parts of the software

    • e.g., self-adaptive systems, context-oriented programming, etc.

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

Laws of Software Evolution

  • In 1965, Mark Halpern used the term evolution to define the dynamic growth of software.

  • The term evolution in relation to application software took gradually in the 1970s.

  • Lehman and his collaborators from IBM are generally credited with pioneering the research field of software evolution.

  • Lehman and Belady formulated a set of observations that he called laws of evolution.

Laws of Software Evolution (Cont.)

Law books
  • These laws are the results of studies of the evolution of large-scale proprietary or closed source system (CSS).

  • The laws concern what Lehman called E-type systems: «Monolithic systems produced by a team within an organization that solve a real world problem and have human users.»

Laws of Software Evolution (Cont.)

  • In 2009, Xie et al. applied Lehman’s laws to 7 different open source projects.

Projects

Guowu Xie, Jianbo Chen and Iulian Neamtiu, "Towards a better understanding of software evolution: An empirical study on open source software," 2009 IEEE International Conference on Software Maintenance, Edmonton, AB, 2009, pp. 51-60.

Continuing change (1st law)

A software system that is used in a real-world environment must necessarily change, or else become progressively less useful in that environment.

Changes
Figure 3. Cumulative changes and ratio of changes to Samba.

Increasing complexity (2nd)

As an evolving software system changes, its structure tends to become more complex. Extra resources must be devoted to preserving and simplifying the structure.

xie second
Figure 4. Slope and correlation coefficients showing how program complexity changes over time.

Self-regulation (3rd)

Software evolution is a self-regulating process. Software properties such as size, time between releases, and the number of reported errors is approximately invariant for each system release.

xie third
Figure 5. Incremental module growth for OpenSSH.

Conservation of organizational stability (4th)

Over a software system’s lifetime, its rate of development is approximately constant and independent of the resources devoted to its development.

xie fourth
Figure 6. Change and growth rates for Samba

Conservation of familiarity (5th)

The amount of incremental change in each successive release of a software system tends to stay constant or decrease over time.

Continuing growth (6th)

The amount of functionality in a software system will increase over time, in order to satisfy its costumers.

xie fifth
Figure 7. Evolution of application size

Declining quality (7th)

A software system will be perceived as losing quality over time, unless its design is carefully maintained and adapted to new operational constraints.

Feedback system (8th)

Successfully evolving a software system requires recognition that the development process is a multi-loop, multi-agent, multi-level feedback system.

Applicability of Lehman’s laws

  • Lehman’s laws seem to be generally applicable to large, tailored systems developed by large organizations.

  • Confirmed in early 2000’s by work by Lehman on the FEAST project.

However

It is not clear how they should be modified for:
  • Shrink-wrapped software products;

  • Systems that incorporate a significant number of COTS components;

  • Small organizations;

  • Medium sized systems.

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

Maintenance

  • In hardware, maintenance is the replacement of used parts

  • In software:

    • Source code doesn’t wear out

    • Maintenance is mainly evolution

Software is knowledge

Truth can only be found in one place: the code.

Clean Code
— Robert C. Martin

The code is the truth, but it is not the whole truth.

— Grady Booch

Maintenance needs knowledge

  • Software code contains a lot of knowledge about the world

  • 40% to 60% of maintenance is spent reading (understanding) the code

  • Maintenance needs this knowledge

Types of knowledge

Application Domain

Business rules, processes, etc.

Organizational structure

Positions, responsibilities, roles, hierarchy, etc.

Processes

Activities, generated artifacts.

Software system itself

Documents, Components, Architecture

Techniques and tools

OOP, AOP, SPL, MDE, IDE, compilers, etc.

Saving knowledge

  • A good part of software development is spent on acquiring this knowledge and putting it in the system.

  • Then, developers forget or go away: the knowledge only exists in the software.

  • A good part of maintenance is spent on re-acquiring this knowledge, to understand the system to change it

  • It is important to try to save this knowledge: code comments, architecture and design models, wiki, formal specifications, etc.

Software Metrics

  • An objective view of a software system is essential

  • Often, maintainers have a biased view (they only see problems)

  • Metrics give an unbiased view of the system

You cannot manage what you cannot control, and you cannot control what you do not measure.

Controlling Software Projects
— Tom DeMarco 1982

Some Metrics for Maintenance

guava
Guava’s view from Code City
  • System size (eg: LOC)

  • Effort (eg: man/hour)

  • Schedule (eg: start/end dates)

  • Quality (eg: number of errors)

Be Careful with Metrics

  • Need to be analyzed in context

  • Not an end but a mean (not a goal but a tool)

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

maintenance process
Figure 8. Software Maintenance Process

ISO/IEC 14764:2006, Software Engineering — Software Life Cycle Processes — Maintenance

Activities

maintenance process
The maintenance process comprises the following high- level activities:
  1. Process Implementation.

  2. Problem and Modification Analysis.

  3. Modification Implementation.

  4. Maintenance Review and Acceptance.

  5. Migration.

  6. Retirement.

Process Implementation

process implementation
  • Software and organization need to be prepared

  • Prepare maintenance when development starts

  • Prepare software:

    • Where is the source code?

    • Is there documentation? How to make change (VCS)?

    • Prepare organization:

    • What is the maintenance process?

    • Who is responsible for what?

Problem and Modification Analysis

problem and modification analysis
  1. Analyze and confirm request

    • Reproduce (with a test - TDD)

    • Classify (adaptive, corrective, emergency corrective, etc.)

    • Check its validity (Real bug? Needed new feature?)

  2. Propose a solution

    • Estimate costs, plan (schedule)

    • Consider several possible solutions and compare them objectively

Problem and Modification Analysis (Cont.)

problem and modification analysis
  1. Document the request and the solution proposal

    • Software is knowledge!

    • Documentation can take several forms (models, comments, wiki, formal document, commit message, etc.)

  2. Obtain authorization to apply solution

    • You might be the authorizing authority (simplified process)

Modification Implementation

Modification Implementation
  • Implement solution

    • Bug fix for corrective maintenance

    • Development process for evolution

  • Test modification and system

    • Regression testing (did not break some other part of the system)

Maintenance Review and Acceptance

Problem and Modification Analysis
  • Review and approval

  • Might require formal approval (ex: Quality Assurance authority)

Migration

Problem and Modification Analysis
  • Large modification

    • Potentially affects all users

  • Might change more than the software

    • Hardware, OS, DB, …​

Migration

Problem and Modification Analysis
Plan
  • When, who, why

  • Parallel operations

  • Data conversion

  • Backup old version, old data

Notify
  • Identify affected people

  • Notify in advance

  • Notify when finished

Retirement

retirement
End of life of software system
  • Similar to migration

  • Plan, Notify, Parallel operation, Archive

Plan

  • Basic Definitions

  • Evolution

  • Evolution Laws

  • Maintenance

  • Maintenance Process

  • Conclusion

Maintenance Misconceptions

  • Perfect development can eliminate maintenance.

  • New technologies and tools will reduce maintenance.

  • Maintenance is complex and boring

From Nicolas Anquetil presentation about Software Evolution]

Maintenance is complex

Intrinsically more difficult than development:
  • Information missing on existing system

  • Must preserve some backward compatibility (existing data, user habits, …)

  • More chaotic (reaction to external events)

  • Less resources

Maintenance is boring

sudoku simple
sudoku complex

Sometimes, complex things are more interesting than simple ones.

Is maintenance really boring?

The problem is cultural first:
  • Maintenance is not taught (implies it is not important?

  • Computer science evolves fast (“newer is better”)

  • Seen as punishment/probation

Maintenance is complex
  • Can be seen as an interesting challenge

  • A good way to learn many things (e.g. programming tricks)

Conclusion

  • Software evolution is very important

  • Typically ill-perceived

  • There is an important cultural problem (misunderstanding)

References

  • ISO/IEC 74764 IEEE Std. 74764-2006, Software Engineering—Software Life Cycle Processes—Maintenance, IEEE, 2006; doi: 10.1109/IEEESTD.2006.235774. K.H. Bennett and V.T. Rajlich, “Software Maintenance and Evolution: A Roadmap,” in Proc. Conf. Future of Software Eng., ACM Press, 2000, pp. 73-87; doi: 10.1145/336512.336534.

  • N. Chapin , “Types of Software Evolution and Software Maintenance,” J. Software Maintenance and Evolution: Research and Practice, vol. 13, no. 1, 2001, pp. 3-30; doi: 10.1002/smr.220.

References (Cont.)

  • E.J. Chikofsky and J.H. Cross, “Reverse Engineering and Design Recovery: A Taxonomy,” IEEE Software, vol. 7, no. 1, 1990, pp. 13-17; doi: 10.1109/52.43044.

  • M. M. Lehman, L. Belady. Program Evolution: Processes of Software Change. Academic Press, London, 1985, 538pp. https://dl.acm.org/citation.cfm?id=7261