Source Code Maturity levels

Have you jumped into DevOps wagon already? You probably have. But perhaps you still not sure if you are lacking a certain tool in your toolbox if you are working currently with DevOps.

Or maybe your organization or team is starting to plan to fully embrace DevOps and your team is researching what is exactly what to need to install in order to have the perfect toolchain. Perhaps you have a gap in some processes that you are not even aware of. Establishing a good and solid DevOps toolchain will help determine ahead of time the grade of the success of your DevOps practices.

In this blog post, we will be exposing maturity level checklists for different DevOps areas so you have an idea where you at in terms of Continuous Delivery.

We will review the maturity levels from the following DevOps aspects:

  • Source code management
  • Build automation
  • Testing
  • Managing database changes
  • Release management
  • Orchestration
  • Deployment and provisioning
  • Governance, with insights

Source code management tool

Commonly known as repositories. It works as a version control and can be used to keep track of changes in any set of files. As a distributed revision control system it is aimed at speed, data integrity, and support for distributed, non-linear workflows.

This is the maturity level checklist. we go from a none or low maturity level to a high maturity state:

  • No version control
  • Basic version control
  • Source/library dependency management
  • Topic branches flow
  • Sprint/project to branch traceability

Source Code Maturity levels

Build automation tool

Continuous Integration (CI) is a software development practice that aims for a frequent integration of individual pieces of work. Commonly each person integrates at least once per day giving place to several integrations during the day. Each integration should be verified by an automated Build Verification Test (BVT). These automated tests can detect errors just in time so they can be fixed before they create more problems in the future. This helps to reduce a lot of integration issues since this practice allows to develop faster and in a more efficient way.
This is the automation maturity checklist to see how you are doing in your CI:

  • No build automation. Built by hand. Binary check-in.
  • Build automated by central system
  • Reusable build across apps/projects
  • Continuous/nightly builds
  • Feedback loop for builds

Automation Maturity Levels

Testing framework

Testing automatization can be in code, systems, service etc. This will allow the testing each modification made in order to guarantee a good QA. Even the daily or weekly release of code will produce a report that will be sent every early morning. To accomplish this you can install the Selenium app in Clarive.

This checklist will help to determine your testing practices level:

  • No tests
  • Manual tests
  • Automated unit/integration tests
  • Automated interface tests
  • Automated and/or coordinated acceptance tests
  • Test metrics, measurements, and insights
  • Continuous feedback loop and low test failure

Testing Maturity Levels

Database Change Management

It’s important to make sure database changes be taken into consideration when releasing to production. Otherwise, your release team will be working late at night trying to finish up a release with manual steps that are error-prone and nearly impossible to rollback.

Check what is your team’s database management current state:

  • Manual data/schema migrations
  • Automated un-versioned data/schema migrations
  • Versioned data/schema migrations
  • Rollback-enabled data/schema migrations

Database matutity levels

Since databases schema changes are sometimes delicate, make sure to include your DBA team into the peer review process, so that changes are 1) code; 2) can be merged and patched; 3) can be code reviewed.

Release Management and Orchestration

You can fully orchestrate tools that are involved in the process and manage your release milestones and stakeholders with Clarive.

Imagine that a developer makes a change in the code after this happens you need to promote the code to the integration environments, send notifications to your team members and run the testing plan.

Are you fully orchestrating your tools? Find out with this checklist:

  • Infrequent releases, releases need manual review and coordination
  • Releases are partially automated but require manual intervention
  • Frequent releases, with defined manual and automated orchestration and calendaring
  • Just-in-time or On-demand releases, every change is deployed to production

Orchestration Maturity Levels

Deployment tool

Deploying is the core of how you release your application changes.

How is your team deploying?:

  • Manual deployment
  • Deployment with scripts
  • Automated deployment server or tool
  • Automated deployment and rollback
  • Continuous deployment with canary, blue-green and feature-enabling technology

Deployment Maturity Levels

Provisioning

As part of deployment, you should also review your provisioning tasks and requirements. Remember that it’s important to provision the application infrastructure for all required environments, keep environment configuration in check and dispose of any intermediate environments in the process.

Yes, provision has also several maturity levels:

  • You provision environments by hand
  • Environment configuration with scripts as part of deployment
  • Provisioning of disposable environments with every deployment
  • Full provisioning, disposing and infrastructure configuration as part of deployment
  • Full tracking of environment-application dependencies and cost management

Provisioning Maturity Levels

We have come a long way doing this with IaC (Infrastructure as Code). Nowadays a lot can be accomplished with less pain using technologies such as containers and serverless, but you still need to coordinate all cloud (private and public) and related dependencies, such as container orchestrators.

In your path to provision automation and hands-free infrastructure, make sure you have a clear (and traceable) path to the Ops part of your DevOps team or organization, making sure to avoid bottlenecks when infrastructure just needs a magic touch of the hand. One way of accomplishing that is to have a separate stream or category of issues assigned to the DevOps teams in charge of infrastructure provisioning. We’ll cover that on a later blog post.

With the right reports, you’d be amazed by how many times releases get stuck in infrastructure provisioning hell…

Governance

Clarive has also productivity and management tools such as with Kaban swimlanes, planning, reports and dashboards that give managers tools to identify problems and teams a way to quickly check overall performance of the full end-to-end process.

Here are the key points to make sure you evolve the overall governance of your DevOps process:

  • There is no end-to-end relationship between request (why) and release (when, how, what)
  • Basic Dev-to-Ops traceability, with velocity and release feedback
  • Full traceability from request to deployment
  • Immediate feedback and triggers

Maturity Levels of Source Code Management

There you go, let’s devops like the grownups do

In this post, we have exposed the main Continuous Delivery aspects that every DevOps team should be looking forward to improve and their respective readiness levels. So go with your team and start planning a good DevOps adoption plan 😉


Schedule a demo with one of our specialists and start improving your devops practices.


Releases Across Pipeline

ARA is a key DevOps discipline that deals with managing the software and infrastructure changes brought about by a new release. In order to gain a clearer understanding of ARA, we could say that the development lifecycle involves two main areas:

  1. Component packaging release and environment planning

  2. Readiness and delivery pipeline automation

Efficiency in moving releases between environments is an important feature, which we can break down into the following seven subcategories.

1. Deployment Flexibility

Deployment needs to be flexible enough to accommodate options (across all platforms and for all environments) such as the following: mapping of dependencies, notifications, creation of shared components, handling of possible manual steps, dealing with concurrent releases, prioritization, window management and integration.

Clarive’s rule-driven deployment model approach, which is based on decision trees, allows the system to keep track of the relationship between all artifacts and their configuration, thanks to its graph database. Decision trees may include other releases.

Clarive integrated graph database keeps track of the relationship between all artifacts and their configuration. Rules support orchestration of both automated and of manual activities.

Clarive features a rule design palette containing the necessary controls for configuring the orchestration scenarios needed: pause job for manual tasks, add approval steps in the job pipeline, Clarive traps an error during the job pipeline rule execution and ask for manual input decisions from the Ops team or equivalent (retry, abort and rollback, skip and continue…)

2. Release Readiness

Release readiness features include deployment risk analytics, impact analysis, integration with testing tool results and automated triggering of quality gates based on test case results and pipeline execution governance.

Clarive provides support for release readiness, including impact analysis, deployment risk analytics, integration with testing tool results and automated triggering of quality gates based on test cases. It allows users to customize their experience using dashboards, rules, pipelines and custom reports.

Orchestration tracking readiness, which is used to track how close a release is to being production-ready is only possible in an end-to-end orchestration tool model.

3. Deployment Plan Management

There should feature a complete GUI for creation and management of role-based deployment plans, and the underlying logic of these should be displayed as transparently as possible.

Support for application deployment plan management that should include the following:

  • Control Logic – to enable unrestricted and flexible automation

  • Pipeline Diagram Support – graphical representation of logic

  • DevOps IDE Platform – an Integrated Development Environment for designing development plans

A decision tree structure/rule design, containing collapsible sections for each phase, is used in order to provide an overview of the end-to-end build, deployment, and post-deployment process. Should more sophisticated activities be required, nested deployment plans (reusing rules) can help promote readability and logic reuse.

4. Pipeline Governance

Pipeline governance includes support for root-cause analysis (RCA) and execution profiling, which serves as an aid when improving software performance.

RCA is a method used for problem solving by identifying root causes of faults or problems, and its implementation should be capable not only of tracing a failure to a specific version of a specific component, but also of incrementally patching the component, while highlighting any other components that require similar patching.


Clarive Root Cause Analysis Graph

Clarive Root Cause Analysis Graph

Deployment pipelines may be monitored through the built-in monitoring service, and individual steps can be traced and analyzed for execution failures, together with any details available from the OS or tool with which interfacing is performed.

Clarive comes with a root cause analysis (RCA) algorithm, whereby the tool contains an extensible database of root causes, and can match failures to said database.

5. Pipeline Recreation

Pipeline recreation refers to the ability to rerun a deployment using a previous version of an environment model and/or rule logic, and should also be a supported feature of an ARA application.

Clarive can determine which versions of which components were used during the delivery process by capitalizing on our software’s configuration management capabilities. The system can use the exact version of the delivery rules that was used for the deployment at the requested time.

6. Deployment Scenarios

Application Release Automation tools should offer flexibility in their support for full-featured deployment styles, thereby ensuring extensibility.

Some typical deployment scenarios one should take into consideration:

  • Canary deployments

  • Blue/Green deployments

  • Rolling or phased deployments

  • Hot deployments

  • Dark launch modes

Clarive supports any deployment scenario/strategy, including partial deployments, rolling and phased, as well as canary and hot, blue, green, and dark launch modes.

7. Test Data Management

Test integration is an important feature, therefore automation of test data preparation and integration with test management suites need to be supported.

Clarive release models can include test case and test data selection, with the ability to select the correct test strategy according to the type of release and release contents, as well as control which test data and test sets are to be deployed into an environment, and to scheduling when said deployment is to take place.

It is also capable of orchestration and working with any virtualization, automated testing tool and test data management tool featuring a CLI, REST or SOAP interface.

How Clarive Lives Up to Moving Releases

Clarive greatest strength is moving releases between environments while offering enhanced deployment flexibility, tracking release readiness and management of the release pipeline.

clarive changeset timeline

Clarive tracks every stage of releasing a change

There also features pipeline governance and recreation coverage, with built-in support for rerunning deployments using an earlier version of a given environment model.

In short

Application Release Automation has a very defined set of criteria for release pipeline management. Clarive matches every one of the standard criteria, and indeed surpasses them if we take into account extra features such as Kanban boards, advanced error control and a complete DevOps IDE platform for creating end-to-end automation, orchestration and collaboration of all the teams and assets involved in the organization of DevOps initiatives.


Want to know more about ARA? Check our presentation: Why ARA Matters and learn how to to take control over complexity to deliver a great workflow.


Elixir logo

As you can see in our previous posts, having a complete pipeline to introduce DevOps in your day-to-day live is easy with Clarive’s Rulebook.

You just have to follow these three simple steps:
1. Get your free Clarive cloud instance
2. Upload your code to your Clarive Project repository.
3. Prepare your rulebook, push your commit and enjoy! (oops, maybe four steps would have been better :))

So let’s get down to business: we will detail the needed code.

Defining our variables

First, we declare the variables that will be used throughout our pipeline process.

vars:
  - workspace: "${project}/${repository}"
  - server:  https://<my-clarive>.clarive.io
  - art_path: ${server}/artifacts/repo/${project}

Building our application

In this step, we choose Elixir Docker image using Mix as a building tool.

build:
  do:
    - image:
         name: 'elixir'
         runner: 'bash'

     - shell [Compile application]: |
         cd {{ workspace }}
         mix compile
         tar cvf ${project}_${job}.tar _build/dev/lib/

And publish the compiled application to our artifact repository.

    - artifact_repo = publish:
        repository: Public
        to: '${art_path}'
        from: '{{ workspace }}/${project}_${job}.tar'
    - log:
        level: info
        msg: Application build finished

Ready to test

As long as we have our own application tests, this step is as simple as running the right command.

test:
  do:
    - image:
         name: 'elixir'
         runner: 'bash'

     - shell: |
         cd {{ workspace }}
         mix test

Deploy wherever we want

Now, it’s time to choose where our app will run. For example, send the tar file to another server and run the app.

deploy:
  do:
   - ship:
       from: '${art_path}/${project}_${job}.tar'
       to: /tmp/remotepath/
       host: ${remote_server}
  - shell:
      cd /tmp/remotepath/
      tar -xvf ${project}_${job}.tar
      mix run

This remote_server could be an AWS instance in PROD environment or another Docker container just to QA.

Happy Ending

Now with our .yml file already prepared, we can use Clarive’s interface to visualize the steps that will follow the finalizing of the rulebook.
To start the deployment, we only need to perform the push on the repository, just as we have seen before in other posts. When performing the push, Clarive automatically creates a deployment (Continuous Integration) and executes all the code found in the .clarive.yml.


Visit Clarive documentation to learn more about the features of this tool.



Increase quality and security of node.js applications which rely on NPM packages.


Developers often create small building blocks of code that solve one particular problem and then “package” this code into a local library following NPM guidelines. A typical application, such as a website, often consists of dozens or hundreds such small node.js packages. Development teams often use these packages to compose larger custom solutions.

While NPM allow teams to exploit the expertise of people who have focused on a particular problem area, residing either inside or outside the local organization, and support teams work together better, sharing talent across projects, we often see companies struggling with the quality of packages being used and as a result looking for ways to control usage better.

Better ways to manage and control what packages are being deployed in their cloud and/or their data centers is vital.

Organizations want to reduce the risk of failure or instability resulting from downloads of the latest version of a required NPM package from the internet, potentially improperly tested.

This video shows how Clarive can help you, making your node.js applications that use NPM packages more secure and stable.


Get an early start and try Clarive now. Get your custom cloud instance for free.


In this video we can see how Clarive checks the availability of certain versions of applications (BankingWeb 2.1.0 and ComplexApp 1.2.0) needed in an environment before deploying a version of another application (ClientApp 1.1.0).

We could consider the applications(BankingWeb and ComplexApp) as pre-requisites for the actual application (ClientApp) for which the deployment is requested.

When the required applications versions (BankingWeb 2.1.0 and ComplexApp 1.2.0) are not in the target environment yet, Clarive will block the deployment until the applications are either deployed first (in a separate job) or added to the same deployment job as the application requiring the pre-requisite applications.


Get an early start and try Clarive now. Get your custom cloud instance for free.



Given the fact that the EU’s GDPR law (replacing the 1995 Data Protection Directive) will go into effect on May 25, almost every software vendor is jumping on the bandwagon to explain how they can help.


Let me be compliant, and follow the stream.

gdpr and Clarive

When reading about the subject online, I noticed that most articles center around the specific business and legal obligations regarding personal data. These articles focus on physical data processing and the data controller obligations to manage processing. Of course!

This is not what I want to write about in this blog however. The GDPR is also expected to impact the software delivery life cycle and its related IT-development processes for organizations that plan to rollout IT projects within the EU.

There are many lifecycle flavors to deliver software today on the market, all waterfall, iterative, or agile based. All of them define the way to manage and control the IT project, from planning to rollout, across the different application layers or modules, and platforms. Common software layers that will be directly impacted by the new GDPR law include of course databases as well as their related architecture, but also data transport, data security, presentation, and application layers… basically potentially every software aspect could be affected!

The impact of GDPR on application delivery

This means if your company intends to continue to roll out systems in the EU, you will have to deal with the new functional and technical requirements introduced by the GDPR like the following (this is not an exhaustive list, only some important ones to make the point):

  • Ensure data protection in the system and the organization, by design and by default (Recital 78 and Article 25)
  • Use data encryption when possible (Recitals 83 and Articles 6-4(e), 32-1(a))
  • Use Data pseudonymization when possible (Recitals 26, 28, 29, 78 and Articles 6-4(e), 25-1, 32-1(a))
  • Anonymize data when possible (Recital 26)
  • Share processing attributes and steps to the data subject in an easy to understand form at the time of data collection, electronically or in writing (Recitals 39, 58 and Articles 12-1, 13-2(a-f))
  • Make data portable to another provider (maybe competitor) (Recital 68 and Articles 13-2(b), 14-2(c), 20)
  • Ensure data is secured, and integrity and confidentiality are maintained, using technical and organizational means under the management of the controller (Recital 49 and Articles 5-1(f), 32-1(b-d))

While a number of these new requirements might be seen as “no-brainer” as they were already part of your software design, others will trigger new requirements that need to be implemented fast and with quality before the law is enforced.

Failing seems to be not really an option. Not complying with the GDPR requirements could result in very serious penalties! As I could read, the worst-case scenario could a fine of €20 million or 4 percent of the company’s previous year’s total global revenue, whichever is greater. Ouch!

The clock is ticking, how do you track progress and ensure compliance?

With only a few more months left, how are you progressing with the delivery of these new requirements? Can you truly track requirement progress throughout your software delivery chain? How confident are you all policies are correctly implemented?

When speaking to bigger clients, this is often their biggest challenge: They have deployed multiple tools to support software delivery. Coding is fragmented, and the delivery toolchain is often poorly integrated, leading to extensive manual activities within the delivery process and a lack to end-to-end visibility and traceability.

At Clarive we believe in SIMPLICITY. Your software delivery toolchain should be as simple as possible, requiring the minimal set of tools to get the work done fast, with quality, and in a transparent way. For smaller organizations and startups, this can be a single tool: Clarive! Bigger organizations often do need multiple tools to support multiple platforms, but they miss overall orchestration and automation. Not those that use Clarive!

As a simple, lean application delivery platform, Clarive will deliver you the traceability you need to track progress on your GDPR (and other) requirements with ease.

Clarive not only helps with end-to-end tracking, its powerful role-based, ruling and workflow automation system also offers capabilities that will help you to ensure everyone on the team remains in compliance with the company’s legal and other requirements, like those for GDPR. For example:

  • Workflow rules: Workflow rules allows you to accept/reject code or actions that do not comply with company policies. For example, our support for code reviews ranges from static code analysis-based decision tree rules to multi-level acceptance approvals within the delivery process.
  • Role based security: Permissions can be set very granularly according to the role members have in respect to the project.
  • Cross platform & process Automation: The best way to ensure compliance it to AVOID manual interventions. Clarive allows you to automate every delivery execution step (apart from the coding itself of course) and process workflow. We support this across teams and platforms, making manual activities (other than just approvals) redundant.

Sounds great? Why don’t you take a look at Clarive now? As our customers witness, you can get started quickly. Just download Clarive for free here and try it out yourself.


Get an early start and try Clarive now. Get your custom cloud instance for free.



Today we’re going to see how to deploy an application in Google Play Store with Clarive.


In the following post we’ll see how to automate the compilation of our applications, as well as making its subsequent upload to the Play Store from a mobile app completely automatic. Clarive will be the only tool we’ll use throughout the whole process.
All of this will save you costs, as you will avoid the costs of manually carrying out the compilation and deployment each time a new version of an application is launched.

To develop this, we will use a free Clarive instance. Through the use of Docker containers we will be able to compile and deploy an Android application in Google Play Store by using the rulebook that we will explain below.
The whole process will be managed through what we have configured in the file .clarive.yml that is in the root folder of our repository.

Requirements

In order to complete this process there are some requtirements, which are as follows:
– The Play Store .json file in order to be able to upload the application automatically.
– The application ready to be compiled and signed automatically.
– A Clarive instance, which you can request for free here

Designing our .clarive.yml

The .clarive.yml file will be where we will define the steps that should be followed for the compilation and deployment process.

Defining our variables

First we will declare the variables that will be used throughout our pipeline process.

vars:
  artifact_path: http:///artifacts/repo/
  # Root path to our artifact repository.

  artifact_repo: "public"
  # Name of repository

  artifacts_store_path: "android/app/app-release-{{ ctx.job('change_version') }}.apk"
  # Path inside the repository where the generated APK will be stored

  json_file: "clarive-rulebook.json"
  # Name of our JSON file for its uploading to Play Store

  workspace: "{{ ctx.job('project') }}/{{ ctx.job('repository') }}"
  # Workspace where our development files will be stored

  package_name: codepath.apps.demointroandroid2clarive
  # Pack name for our app in the Play Store

Building our application

Next is the BUILD phase, and here we are going to compile the application and save the generated file in our artifact repository.

Our build.gradle file must be prepared for the automatic digital signing of the application and and its subsequent uploading to the Play Store. In the same way, we must have also previously manually uploaded a first version of the application to the Play Store.

An image with gradle and the SDK source of Android should be enough to make the compilation.

After specifying the Docker image that we will use, we will execute the gradle command within our working directory, so that the compilation and signature of the application can be carried out.

In our particular case, we should use the root user and the sh shell of the image we are using in order to perform the compilation of the application.

build:
  do:
    - image:
         name: 'knsit/gradle-android'
         user: 'root'
         runner: 'sh'

     - shell [Compile application]: |
         cd {{ workspace }}/Application_code/app/
         gradle AssembleRelease

To complete the BUILD phase, in which we have compiled the application, we need to save the generated APK file, where the compiled application is located, in our artifact repository:

    - artifact_repo = publish [Store APK in artifacts repository]:
        repository: Public
        to: '${artifacts_store_path}'
        from: '{{ workspace }}/Application_code/app/build/outputs/apk/app-release.apk'
    - log:
        level: info
        msg: Application build finished

Once the file has been saved and stored we will have terminated our BUILD phase, and we will now move on to the DEPLOY phase in order to carry out the deployment of the file.

Deploying to Play Store

In this phase, we need our APK file with the compiled application, and our .json Play Store authentication file in order to carry out the upload automatically.

In this case, the image that we need to make the deployment will be a Docker image using the installed Fastlane, and here we are going to prepare the command we need to execute in order to deploy our application in Play Store.
In this case we will be uploading it to the Alpha phase of the applications.

Our .json file is located within our development files where the application is, which means we can place it within our workspace.

deploy:
  do:
    - image:
        name: 'levibostian/fastlane'

    - shell [Upload application with Fastlane]: |
        fastlane supply --apk .artifacts/{{ artifact_repo }}/{{ artifacts_store_path }} -p {{ package_name }} --json_key {{ workspace }}/{{ json_file }} -a alpha

    - log:
        level: info
        msg: The app has been deployed to the Play Store

In this way, if we carry out the push to our repository, Clarive will automatically run a deployment (CI) and our application will be uploaded onto our Play Store.

Notifications

Finally, in the POST step we will email the user that has launched the deployment to inform them that it has been completed, and in order for them to check the results.

post:
  do:
    - log:
        level: info
        msg: Deploy finished
    - email:
        body: |
          Hello {{ctx.job("user") }},
          <b>{{ ctx.job('name') }}</b> has finished and your app has been deployed. Check it out in <a href= "https://play.google.com/">your Play Console</a>.

          Also your apk file has been stored in your artifacts repository:
              ${ artifact_path }${artifact_repo}/${artifacts_store_path}.

        subject: Application deployed to Play Store
        to:
           - ${ctx.job("user")}

To conclude

Now with our .yml file already prepared, we can use Clarive’s interface to visualize the steps that will follow the finalizing of the rulebook.
To start the deployment, we only need to perform the push on the repository, just as we have seen before in other posts. When performing the push, Clarive automatically creates a deployment (Continuous Integration) and executes all the code found in the .yml.

If everything has run correctly, we should be able to see on the monitor how our deployment is being executed.

Job succesfully finished

By doing all of this, we have carried out the whole process of compiling and uploading our application to Google Play Store through the use of Clarive’s rulebooks and different Docker containers.
If we look at our page on the Play Store console, we’ll be able to see a message indicating that we have an application ready and waiting.

APK v2 deployed

APK details

This is a brief example that can serve as a reference. You can configure different environments to deploy on. You can also change the type of operation to be carried out in each phase in a completely customizable way, in such a way that it adjusts to what each person needs for their development work and deployments.


Get an early start and try Clarive now. Get your custom cloud instance for free.



In previous video’s we have deployed Mainframe Git managed application, Java application and Mainframe Endevor packaged up till the QA environment.


All of the deployments for each of the technologies and target platforms were done with Jobs created by the same versioned pipeline rule. Now that we have reached the final deployment to the production environment, we don’t want to deploy individual changesets anymore. We want to group them into a release/sprint and deploy them together.

We will do that in a single job created by the same pipeline as the one used for the changesets.
In summary, with Clarive we deploy in a single job from a single pipeline, multiple technologies to multiple environments on multiple platforms. This assures us a consistent way of deploying.


Get an early start and try Clarive now. Get your custom cloud instance for free.