Solutions
Introduction to Versioning, Continuous Integration, Life-Cycle and Approval management for Subversion
1. Running an EJB3 project with ContinousIntegration and Life-Cycle Management
The following document describes how SCM4ALL extends the functionality of Subversion by offering automation, control and visibility throughout the whole life-cycle, from managing a Continuous Integration build close to the developers, to automating the deployment into test, QA, staging and production and organizing the related authentication and approval processes.
We have published a technical paper, which includes a small EBJ 3.0 Oracle JDeveloper demo project, called MusicLib. This example describes a three stage life-cycle with Subversion for an EJB3 project, which includes ContinuousIntegration, build and deploy on a test environment and finally a deploy of an OC4J container to the production environment with an approval mechanism
Introduction
Building a software product involves many complex processes, roles and deliverables, which need to be managed to fit together. Streamlining these processes is a major effort, particularly when there are many people involved. A truly comprehensive software configuration management (SCM) solution manages not just the simple versioning of your source code files - it also facilitates support for Continuous Integration, accurate and predictable build management, provides the ability to deploy the end result as well as offering approval processes and manages the complex run-time dependencies of many applications today.
Software development, seen in a historical perspective, is a very young discipline. In the early stages, software development was seen more as an art (the developer is always right, even if he was wrong, and beauty won over effectiveness and efficiency), as it was seen as supporting business objectives (who pays the bill and who wants to profit?). In the recent decades however, software development started to organize itself: initially to make life easier for itself, finally to answer internal and external driven productivity and compliancy needs.
The first generation productive tools were called library management tools: just like you store a book in a library, you stored the software code of a program in a software library. In worst (or best) case, you had all versions (it knew the one that was the latest and/or corresponding with the version used in production), or you just had the latest. To keep a long story short: library management has evolved over managing just sources, to a full process driven SCM approach.
What are organizations looking for?
Every role within the organization has a different area of interest. For instance, a developer wants to have early feedback on the code he committed in the trunk stream (Continuous Integration) and wants to build the Project in his IDE with the correct latest sources and common libraries. A project manager wants to have a clear overview of the project status: is the latest code in the trunk buildable, do the unit tests run successfully, what are the guys of QA testing, which is the current production version? Production operators prefer an automated deployment process, where they can control the environment variables. Finally the CIO and CEO of a corporation would like to see an automated and repeatable process with an audit trail.
What does SCM comprise?
True SCM is process driven and is composed of several steps: it starts with versioning and finally ends with deployment.
| Developer | IT Management | Production | Management/audit | |
|---|---|---|---|---|
| Why Versioning? | To keep track of the changes | Safe storage of all historic data | Easily revert to a prior version | No loss of data |
| Why implement a continuous integration process? |
Concentrate on developing software Get early feedback on committed code |
Early feedback on code quality Find weak spots |
Get high quality production code |
Fewer errors Repeatable process Faster and shorter release cycle |
| Why have an automated build? | No loss of valuable time trying to build manually | Allows to do more builds and gives rapid feedback | Everything is coordinated by a script | Prevents mistakes |
| Why approval management? | Improves communication accross the project team | Control the evolution in the different stages of the life-cycleBuild in audit moments | Control deployment to the production servers | Traceability Who authorized |
| Why have an automated deploy? | Guarantee that production will receive the quality code that I created | To speed up the process and help reduce errors | No manual intervention reduces risk | Increases the possible release cycle frequency and productivity |
| Why rollback process? | Have more time to fix defects | Can always revert to the latest good release | Refuse or eliminate risk of service outage |
Ensure return to exact prior state Quickly resolve errors in production |
| Why life-cycle management? | No worries of building code for the test or production level | Have a clear view of the development process and status |
Automate production deployment Reduces rework |
Answers questions of Who, When, Why and What occurred? |
Versioning
One of the first steps is versioning. Versioning sources is a best practice, which is considered evident for controlling the software development process. Whenever people startup a new software project, setting up a version control repository is one of the first steps and is done even before the first code is created. When you ask developers what they need, they will answer you that a versioning tool like Subversion perfectly answers their needs, as it versions their code. Subversion is one of many versioning tools. It has rapidly become the de facto standard free/open-source version control system. The Subversion project was started by CollabNet, the organization that still funds a large part of the work, in order to make a valid and better replacement for the commonly used CVS system. Today Subversion is considered one of the best versioning systems available.
Subversion
In general Subversion is a free/open-source version control system. That is, Subversion manages files and directories over time. A tree of files is placed into a central repository. The repository is much like an ordinary file server, except that it remembers every change ever made to your files and directories. This allows you to recover older versions of your data, or to examine the history of how your data changed. Subversion can access its repository across networks, which allows it to be used by people on different computers. At some level, the ability for various people to modify and manage the same set of data from their respective locations fosters collaboration. Progress can occur more quickly without the constraints of having a single conduit through which all modifications must occur. And because the work is versioned, you need not fear that quality is the trade-off for replacing that conduit - if some incorrect change is made to the data, just undo that change.
Although most developers are happy with just version management, in today's world where formally basic users have become heavily involved users, production people, auditors to compliance officers and finally shareholders, compounded by the global context in which we operate, we are faced with new challenges: offshore, virtual or outsourced development to name just a few. We now need to answer questions like who did what, where and when, and who has control, what are the fall back options, and how to comply with the new laws on corporate governance (who did it, why, how and who authorized it ?). These new requirements demand that SCM (Software Change Management or version management in the narrower sense) not only keeps versions of software code, but that it also offers developers independent build management, deployment, life-cycle management, approval mechanisms and comprehensive tracking of all activities.
Some version control systems expand their versioning facilities by also delivering SCM functionality. They typically have many features to support all phases in the software development process, such as providing tools for building and deploying software and/or doing life-cycle management. Subversion, however, is not such a system. It is a general system that can be used to manage any collection of files. However, it does not guarantee that a controlled build has been executed properly, nor does it offer complete life-cycle management to provide a traceable and auditable development process. That's where SCM4ALL comes into play. SCM4ALL extends the functionality of Subversion by offering automation, control and visibility throughout the whole life-cycle, from managing a Continuous Integration build close to the developers, to automating the deployment into test, QA and production and organizing the related authentication and approval processes.
Build
Build management orchestrates the complex software assembly, testing and packaging processes that go into producing the product executable. Automated environments for builds are a common feature of systems today. A centralized, reliable build strategy is therefore crucial for an effective SCM approach, and Continuous Integration is one way of doing this.
Continuous Integration
Continuous Integration is an extreme Programming (XP) development practice where members of a team integrate their work frequently; usually each person commits their code changes at least once daily - leading to multiple integrations per day. Each integration is verified by an automated build (preferably also including unit tests and code checks) to detect integration errors as quickly as possible such as, for instance, a failing unit test. Early feedback on broken builds or failed tests is very important since such information will dramatically reduce the time to fix such errors, and, consequently, will also cut down on overall development time. It's obvious that it will take a lot more time fixing code that was written a week ago, compared to code that was just committed to the version control repository only minutes ago. Getting the sources turned into a running system can often be a complicated process involving compilation, moving files around, loading schemas into a database, and so on. It is clear that these tasks should be automated to run on a dedicated server, so that a repeatable build process can be started as soon as possible after the code-base in the versioning system has been changed.
SCM4ALL can complement Subversion with Continuous Integration support in different ways: it is possible to add a post-commit hook script in the Subversion repository that will create a build request each time code is committed in a project controlled by the SCM4ALL. Another way of implementing Continuous Integration is by setting up a schedule that will check on a predefined interval if there are changes in the project code-base of the Subversion repository.
Please see the demo project, which describes in detail how you can Setup your JDeveloper Project, import it into Subversion, and how to create a ContinuousIntegration project.
Automated Build and QA
Continuous Integration gives projects early feedback on the quality of their code. As a result this will improve the end result placed in production and cut down the complete development time. But the development process does not end with the integration build. The result must be tested on a QA level, rebuilt with other compilers or operating systems, deployed under different application servers, and tested on distinct versions of the underlying application database. These are error-prone processes and thus ideal candidates for automation. This is the only way to guarantee that the code used to create the build result after a Continuous Integration process is the same as the code deployed and running in a test environment. During the test you should identify the problems that also might occur during production. It is therefore important that your test environment is similar to the production environment. If this is not the case, it's difficult to predict what will happen in production.
In the sample we will expand the life-cycle of the MusicLib Head ProjectStream by adding a Test level. This test level will contain a build environment on which the code will be rebuilt, a database will be updated and JUnit test will run. It will also contain a Deploy Environment, so that the resulting ear can be deployed under the JEE OC4J container.
Deployment and Approval Process
Traditional change management systems have focused on activities close to the developer, such as source code control, but few mechanisms were in place to support software deployment activities. To be able to know which build version of the application must be retrieved from the build archive to deploy onto the designated target platform you need to automate the deployment of all your software assets. When you have automated the Continuous Integration and promotion to the Test level, it's obvious that the last (and most important) step in the development life-cycle is also a controlled process: deploy to Production. The production environment will probably resemble the test environments. However, you probably will not rebuild your application on the production server, and there will be probably more audit, notification and authorization involved before code is actually deployed. SCM4ALL will solve complex deployment issues, like keeping clustered servers synchronized, or synchronizing applications with their database.
To improve communication within the project team it is advisable to set up approval and notification processes for this last step of the life-cycle. It's a good practice to notify project members that a delivery to production will happen. It's a better practice to also setup an approval process before the production deployment, so that production operators, QA people and project management can audit and control this important stage in the roll-out process.
Even if you have a well planned and executed QA and audit trail, from time to time things may go wrong. Therefore it is not a luxury to ensure that there is a way back, so that in the worst-case scenario production service levels are minimally disturbed. A well-defined and tested rollback process will protect you from nightmares like production shutdown.
In the example, we will expand the base life-cycle with one production Level containing one OC4J container that is running on a remote server. We do not create a new build result, but deploy the ear which we built on the test Build Environment in the previous stage (test) of the Life-cycle. To control the production server an approval is necessary before the deploy action will actually happen.
Life-Cycle Management
Developing applications is not just about writing code. Once it has been developed, code must be tested, approved and delivered to the live environment. Depending on his\her profile, some users will expect their code migration to the live environment to be complete as soon as possible, whereas for others it will only happen when the time is right. Therefore, a correct software change workflow process must be in place to fulfill the needs of the entire project team. SCM4ALL provides this assistance by auditing what has changed, when it changed, who changed it and who approved the change for promotion. SCM4ALL is both the glue that holds together the various components and phases of the application life-cycle and the oil that lubricates the smooth and efficient interaction of those components. It delivers an automated workflow to drive a continuous flow of activity through the development life-cycle and efficiently coordinate and streamline development changes.
Conclusion
Extending Subversion with Continuous Integration, Life-Cycle Management and approval processes provides a flexible framework for controlling and automating the compilation and distribution of software release packages from level to level (stage to stage), site to site and across networks, as well as facilitating versioning at object and stream level and creating an automated routine of building and releasing objects. With SCM4ALL you can guarantee the quality of the code and the deployment process and provide a traceable and auditable logging for each step in the life cycle.
The final goal is to offer a comprehensive framework, where not only the needs of the developers are met, but also the interests of all other stakeholders.
2. Technical paper for a Subversion demo project
MusicLib demo project with Subversion
We created a small EBJ 3.0 JDeveloper project, called MusicLib.. The aim of this web application is to organize your favorite artists and the albums they have published. The technical paper will guide you through the different steps of this project.
Read the technical paper (PDF, 5,8 Mb).
