Entrega de aplicaciones móviles en 5 minutos En esta entrada vamos a explicar cómo realizar despliegues de aplicaciones utilizando Clarive EE tanto a la tienda Google Play Store, como a la tienda de iOS (Apple Store) gracias a los Clarive Plugins. Este proceso, no requerirá ningún conocimiento de programación adicional. Gracias a la interfaz que nos ofrece el diseñador de reglas podremos configurar los despliegues. Para más información acerca de Clarive y los elementos que utilizamos, consulta nuestro Docs. Desplegando a Google Play Store Aspectos generales Para el caso de las aplicaciones Android, se utiliza el plugin de Gradle para la compilación de la aplicación, y el plugin de Fastlane para el envío a la Play Store automáticamente. Configuración Previamente, es necesario que ya exista una primera versión de la aplicación subida para que se pueda realizar el despliegue de forma automática. Creamos un Generic Server desde el panel de Resources->Server. En este servidor tenemos la aplicación junto con Gradle y Fastlane instalados. Una vez hemos configurado el servidor, vamos al diseñador de reglas y creamos una nueva de tipo Pipeline en el panel de Admin. Utilizamos la fase PRE para compilar la aplicación. Para ello arrastramos la operación Gradle compile: Dentro de la configuración de la operación, seleccionamos el servidor anteriormente configurado, y completamos los campos para que realice la compilación. A continuación, arrastramos la operación Fastlane task a la fase RUN para configurar el envío de la aplicación a la Play Store: Completamos el formulario de la operación con todos los datos necesarios para el envío. Con esto ya tenemos el despliegue a la Play Store preparado. A continuación, añadimos las operaciones para desplegar en la Apple Store con el mismo Pipeline. Desplegando a Apple Store Aspectos generales En este caso, se utiliza el plugin de Fastlane para compilar y enviar la aplicación a la Apple Store automáticamente. Es necesario que Xcode esté instalado junto con Fastlane. Configuración Creamos un Generic Server, al igual que se hizo en el caso anterior. En este servidor tenemos la aplicación junto con Fastlane y Xcode instalados. También, debemos configurar las credenciales de acceso a nuestra cuenta de la Apple Store en el Recurso iOSCredentials, desde el panel de Resources->iOS. Al igual que en el caso para Android, vamos a compilar nuestra aplicación en la fase PRE, con la operación Fastlane task: Dentro de la configuración de la operación, seleccionamos el servidor, las credenciales, y rellenamos el resto de campos necesarios. A continuación, configuramos el envío de la aplicación en la Apple Store. Usando la misma operación Fastlane task en la fase RUN. Seleccionamos la opción ´Upload App´, y completamos los campos. Con este Pipeline ya tenemos configurada la publicación automática tanto en la Play Store como en la Apple Store. Para cualquier duda, puedes ponerte en contacto con nosotros en Clarive Community.
Why continuous delivery should run in containers The problem at hand The situation with the DevOps toolchain is that it just has too many moving parts. And these moving parts have become a cumbersome part of delivering applications. Have you stopped to think how many different tools are part of your pipeline? And how this is causing your delivery to slow-down? These might be some of the problems you could be facing when setting up your continuous delivery pipeline: Changes in the application require changes in the way it’s built/deployed New components require new tools Many build, test, and deploy tools have plenty of dependencies The container bliss Containers are basically lightweight kits that include pieces of software ready to run the tasks in your pipeline. When containers are used as part of the pipeline, they can include all the dependencies: code, runtime, system tools, system libraries, settings. With containers, your software will run the same pipeline no matter what is your environment. You can run the same container in development and staging environments without opening Pandora’s box. Containers are the way to be consistent in your CI/CD and releasing/provisioning practices. Other advantages of containers are: Containers can be versioned Containers can hold the most relevant DevOps infrastructure Containers are cheap and fast to run Ops can let Dev drive CI/CD safely (by giving Devs templatized containers) Clarive and Docker: what a combo! Docker is therefore a great companion to your DevOps stack. Docker containers allow your project and repository rulebooks to run pipelines alongside any necessary infrastructure without requiring additional software packages to be installed in the Clarive server. Clarive runs your DevOps pipelines within managed containers. By using containers in Clarive you can: Isolate your users from the server environment so that they cannot break anything. Version your infrastructure packages, so that different versions of an app can run different versions of an image. Simplify your DevOps stack by having most of your build-test-deploy continuous delivery workflows to run in one server (or more, if you have a cluster of Clarive servers), instead of having to install runners for every project everywhere. Clarive and Docker flowchart Curating a library of DevOps containers Using a registry is a good way of keeping a library of containers that target your continuous delivery automation. With Clarive you can maintain a copy of a local registry that is used exclusively for your DevOps automation. Defining “natures” Each repository in your project belongs or implement one or more natures. The nature of your code or artifacts define how they are going to be implemented. A nature is a set of automation and templates. These templates can use different Docker containers to run. For example, your application may require Node + Python, so two natures. If you have these natures in templates they will be consistent and will help developers comply to a set of best practices on how to build, validate, lint, test and package new applications as they move to QA and live environments. Running commands on other servers Clarive uses Docker for running shell commands locally. That guarantees that rulebooks (in the projects .clarive.yml file) will not have access to the server(s) running your pipelines. But you can still run shell commands in other servers and systems, such as Linux, Windows, various Unixes flavors and other legacy systems (including the mainframe!) using the host: option in the shell: command. How do I use my own containers If the container is not available on the Clarive server, the Clarive rulebook downloads the container from Docker Hub. So, to use your own containers, you have two options: Upload them to Docker Hub. Then use them from your rulebook. Clarive will download it on the first run. Install it on your Clarive server. On the first run Clarive will build another version of your container based on Clarive’s default Dockerfile, called clarive/your container. You don’t need to prefix clarive/ into the name, that’s done for you automatically. Manage all active Docker containers in your pipeline from within Clarive Getting started today Using containers is an important step in implementing a continuous delivery and continuous deployment process that is streamlined and avoids environment clutter. Head over to our 30-day trial and let Clarive to run your DevOps automation in docker containers for better consistency and easy setup of your temporary environments. Learn more about Clarive Docker admin interface with this blog post and learn how to manage containers and docker images.
Continuous release readiness checklist ARA is a key DevOps discipline that deals with managing the software and infrastructure changes brought about by a new release. In order to gain a clearer understanding of ARA, we could say that the development lifecycle involves two main areas: Component packaging release and environment planning Readiness and delivery pipeline automation Efficiency in moving releases between environments is an important feature, which we can break down into the following seven subcategories. 1. Deployment Flexibility Deployment needs to be flexible enough to accommodate options (across all platforms and for all environments) such as the following: mapping of dependencies, notifications, creation of shared components, handling of possible manual steps, dealing with concurrent releases, prioritization, window management and integration. Clarive’s rule-driven deployment model approach, which is based on decision trees, allows the system to keep track of the relationship between all artifacts and their configuration, thanks to its graph database. Decision trees may include other releases. Clarive integrated graph database keeps track of the relationship between all artifacts and their configuration. Rules support orchestration of both automated and of manual activities. Clarive features a rule design palette containing the necessary controls for configuring the orchestration scenarios needed: pause job for manual tasks, add approval steps in the job pipeline, Clarive traps an error during the job pipeline rule execution and ask for manual input decisions from the Ops team or equivalent (retry, abort and rollback, skip and continue…) 2. Release Readiness Release readiness features include deployment risk analytics, impact analysis, integration with testing tool results and automated triggering of quality gates based on test case results and pipeline execution governance. Clarive provides support for release readiness, including impact analysis, deployment risk analytics, integration with testing tool results and automated triggering of quality gates based on test cases. It allows users to customize their experience using dashboards, rules, pipelines and custom reports. Orchestration tracking readiness, which is used to track how close a release is to being production-ready is only possible in an end-to-end orchestration tool model. 3. Deployment Plan Management There should feature a complete GUI for creation and management of role-based deployment plans, and the underlying logic of these should be displayed as transparently as possible. Support for application deployment plan management that should include the following: Control Logic – to enable unrestricted and flexible automation Pipeline Diagram Support – graphical representation of logic DevOps IDE Platform – an Integrated Development Environment for designing development plans A decision tree structure/rule design, containing collapsible sections for each phase, is used in order to provide an overview of the end-to-end build, deployment, and post-deployment process. Should more sophisticated activities be required, nested deployment plans (reusing rules) can help promote readability and logic reuse. 4. Pipeline Governance Pipeline governance includes support for root-cause analysis (RCA) and execution profiling, which serves as an aid when improving software performance. RCA is a method used for problem solving by identifying root causes of faults or problems, and its implementation should be capable not only of tracing a failure to a specific version of a specific component, but also of incrementally patching the component, while highlighting any other components that require similar patching. Clarive Root Cause Analysis Graph Deployment pipelines may be monitored through the built-in monitoring service, and individual steps can be traced and analyzed for execution failures, together with any details available from the OS or tool with which interfacing is performed. Clarive comes with a root cause analysis (RCA) algorithm, whereby the tool contains an extensible database of root causes, and can match failures to said database. 5. Pipeline Recreation Pipeline recreation refers to the ability to rerun a deployment using a previous version of an environment model and/or rule logic, and should also be a supported feature of an ARA application. Clarive can determine which versions of which components were used during the delivery process by capitalizing on our software’s configuration management capabilities. The system can use the exact version of the delivery rules that was used for the deployment at the requested time. 6. Deployment Scenarios Application Release Automation tools should offer flexibility in their support for full-featured deployment styles, thereby ensuring extensibility. Some typical deployment scenarios one should take into consideration: Canary deployments Blue/Green deployments Rolling or phased deployments Hot deployments Dark launch modes Clarive supports any deployment scenario/strategy, including partial deployments, rolling and phased, as well as canary and hot, blue, green, and dark launch modes. 7. Test Data Management Test integration is an important feature, therefore automation of test data preparation and integration with test management suites need to be supported. Clarive release models can include test case and test data selection, with the ability to select the correct test strategy according to the type of release and release contents, as well as control which test data and test sets are to be deployed into an environment, and to scheduling when said deployment is to take place. It is also capable of orchestration and working with any virtualization, automated testing tool and test data management tool featuring a CLI, REST or SOAP interface. How Clarive Lives Up to Moving Releases Clarive greatest strength is moving releases between environments while offering enhanced deployment flexibility, tracking release readiness and management of the release pipeline. Clarive tracks every stage of releasing a change There also features pipeline governance and recreation coverage, with built-in support for rerunning deployments using an earlier version of a given environment model. In short Application Release Automation has a very defined set of criteria for release pipeline management. Clarive matches every one of the standard criteria, and indeed surpasses them if we take into account extra features such as Kanban boards, advanced error control and a complete DevOps IDE platform for creating end-to-end automation, orchestration and collaboration of all the teams and assets involved in the organization of DevOps initiatives. Want to know more about ARA? Check our presentation: Why ARA Matters and learn how to to take control over complexity to deliver a great workflow.
Bringing DevOps to the Mainframe (3/4): Source code versioned in z/OS In this installment of this series we will review how Clarive drive popular z/OS SCM tools such as CA Endevor or Serena ChangeMan as part of your global DevOps pipeline. Source code versioned in z/OS (Endevor or Changeman) In this scenario, the source code to be deployed resides in the mainframe. Selection of objects to deploy Clarive will allow the development user to attach z/OS packages (Endevor orChangeMan) to the changesets for further processing. z/OS packages z/OS repository (Changeman ZMF) Preparing Payload Elements Clarive rules will define the logic to prepare the z/OS packages by executing either online or batch operations that are needed to prepare them (freeze,inspect, generate, execute, etc.) Inspect package operation (Endevor) Deployment of the elements The deployment of the packages will be executed by Clarive by submitting the specific JCL to ship/promote/approve the packages included in the job changesets. Ship package (Endevor) Endevor Ship package sample output Rollback Clarive rule will allow the administrator to define the way to rollback changes. In either Endevor and ChangeMan ZMF it will execute a Backout operation on each package included in the job. Backout package (Endevor) Conclusion Endevor and Changeman are powerful version control tools used in mainframe environments. Fundamentally, they have similar or equivalent workflows for the software configuration lifecycle but often these tools are used independently of the overall enterprise DevOps pipeline. DevOps implementations often leave them out, however they remain critical to deliver and run the critical areas of the business with much newer technologies. With its mainframe orchestration capabilities, Clarive enables organizations with either tool to build better integrated pipelines that brings mainframe changes into the main Dev to Ops stream. Stay tuned for the forth and final installment of these series! Read the previous post of this series and learn about the mainframe features you can find in Clarive and how they are integrated into the system.
Scheduling your pipeline stages Running pipeline jobs to critical environments often requires a scheduled execution to take place. Clarive scheduled jobs run always in 3 main steps, called “Pre”, “Run” and “Post”. Why run a pipeline in phases? Most of the time the entire deployment job should not run all of its tasks at the scheduled time but as soon as the job is scheduled. There are several phases or stages to every scheduled deployment, most of which can run as early as it’s scheduled. Tasks such as building, packaging, testing and even provisioning infrastructure can take place earlier if they do not impact on the productive environments. When defining a pipeline, always think of what can be detected in earlier phases so that the evening deployment run will not fail on something that could have been checked previously. Separating your pipeline into different stages Pipeline phases Here are the different pipeline deployment phases you can run in Clarive. Deployment preparation (PRE) The deployment pipeline will take care of: Creating the temporary deployment directory Identifying the files changed to be deployed (BOM) Checking out the files related to the changesets to be deployed from either code repositories (code) or directly attached to the changesets (SQL). Renaming environment specific files (i.e web{PROD}.properties will be used just for deployments to the PROD environment) Replacing variables inside the files (i.e. it will replace ${variable} with the real value of the variable configured for the project affected for the target environment) Nature detection Clarive will identify the technologies to deploy analyzing the BOM Common deployment operations Deployment operations (RUN) Actual deployment operations will be executed in this phase of the Clarive job (movement of binaries, restart of servers, check the proper installation, etc.) Deployment operations Post-deployment operations (POST) After the deployment is finished Clarive will take care of any task needed depending on the final status. Some of the typical operations performed then are: Notify users Transition changesets states Update repositories (i.e. tag Git repositories with an environment tag) Synchronize external systems Cleanup temporary directories locally and remotely Synchronize related topics statuses (defects, releases, etc.) Post deployment operations Recap Regardless of if you use Clarive or not, when defining your deployment pipelines always think on these three stages or phases. PRE – everything that can run as soon as it has been scheduled RUN – crucial changes that run at the scheduled time POST – post activities or error recovery procedures Happy DevOps! Try Clarive now and start with your DevOps journey to continuous delivery.
Bringing DevOps to the Mainframe (2/4): Tooling In this second part of this blog series we will detail the mainframe features you can find in Clarive and how they are integrated into the system. Clarive Mainframe Features Clarive manages all aspects of the z/OS code lifecycle: Sending files to z/OS partitions Character translation maps and codepages Identify relationships – impact analysis JCL Template Management Submit JCL Nested JCL Management and synchronous – asynchronous queue control Retrieve Job Spool output and parse results Integration rules Clarive z/OS features 3 entirely different integration points with the mainframe. Each integration feature serves a specific purpose Job queue access – to ship files and submit jobs into the z/OS job queue in batch mode. Clarive will track all nested jobs and parse results into the job tree. ClaX Agent – for delivering files into Datasets and/or OMVS partitions and executing z/OS processes online. This is the preferred way of running REXX scripts sent from Clarive to the mainframe. Access z/OS facilities such as SDSF®,ISPF®, VSAM® data records or RACF®. Webservices Library – for writing code that initiates calls from the mainframe directly into Clarive using TCP/IP sockets and the RESTful webservices features of the Clarive rules. Clarive to Mainframe Integration Point Tool considerations Clarive is a tool that allows enterprise companies to implement an end-to-end solution to control the software lifecycle providing countless out-of-the-box functionalities that help solving any complex situation (automation, integration with external tools, critical regions, manual steps through the process, collaboration, etc.) CCMDB – Configuration Items In Clarive any entity that is part of the physical infrastructure or the logical lifecycle is represented as a configuration item (CI). Servers, projects/applications, sources repositories, databases, users, lifecycle states, etc. are represented as CIs in Clarive under the name Resource. Any resource can have multiple relationships with other resources (i.e. an application is installed in one server in production, a user is developer of an application, the Endevor “system x/subsystem y” combination is the source code repository related to an application, etc.) The graph database made of this entities and relationships is Clarive’s Change oriented Configuration Management Database (CCMDB). The CCMBD is used to keep the whole system configuration as well as to do impact analysis, infrastructure requests management, etc. CCMBD navigation Natures / Technologies Clarive natures are special CIs in Clarive that automate the identification of technologies to be deployed by a deployment job. A nature can be detected by file path/name (i.e. Nature SQL: *.sql), by project variable values (i.e. ${weblogic}:Y) or by parsing the changed files code (i.e. COBOL/DB2: qr/EXEC SQL/) Natures list JES spool monitoring Clarive will take care of downloading and parsing the spool output when submitting a JCL in z/OS to split the DDs available in the output and to identify and use the return codes of all steps executed in the JCL. JES output viewer Calendaring / Deployment scheduling – Calendar slots Any deployment job in Clarive will be scheduled and Clarive will provide the available slots depending on the infrastructure affected by the deployment. Infrastructure administrators can define these slots related to any CCMDB level (environment, project, project groups, server, etc…) Calendar slots definition Rollback Clarive rule will allow the administrator to define the way to rollback changes. In either Endevor and ChangeMan ZMF it will execute a Backout operation on each package included in the job. Rollback control Next Steps Features are an important step when picking the right tool to bring DevOps to the mainframe. In the next 2 installments of this series we will review how Clarive can deploy mainframe artifacts (or elements), either by driving popular z/OS SCM tool such as CA Endevor or Serena ChangeMan, or replacing them with Clarive’s own z/OS deployment agents. Read the first post of this series and learn more on how to bring DevOps to the mainframe.
QA Automation: building test plans with Clarive A test plan is an essential part of software development. It is a must if you want to get devs and ops people on the same page. As a guide and a workflow, it reinforces your projects success detecting potential flaws in advance. Tracking is also an important part of the testing process, as changes are applied, the test plan should be updated. Unit test, code QA, performance test, integration and regression test are some of the most common types of software testing, and in some cases they are even compulsory to apply as a previous step to a production deployment. With Clarive you can create a QA workflow that combines manual and automated steps, creating test plans automatically on a pull-request/merge-request, which in Clarive can actually be any changeset (user stories, features, bugfixes, etc.). Test plans can then be used to automate test validation. Like in all software development processes, it’s necessary that each code revision go through a proper testing process to ensure the quality of the product. Automating test plan creation You can also create test plans automatically along with the included changes in the version to be released. This feature is capable of detecting the modified files by the developer, create a plan with the test cases (automated and manual tests) that have impact over the modified/created functions and link them to the release. Clarive will only add the test cases that are directly affected by one of the functionalities modified in the release. This way Clarive creates a test plan suitable for each release and if the user wishes to, automate the test cases and depending on the results execute the required actions. Download our whitepaper: The Value of DevOps for Test & Quality Managers and learn more about how to minimize the risk of product failure with Clarive.
Bringing DevOps to the Mainframe (1/4) The DevOps movement in general, tends to exclude any technologies that are outliers to the do-it-yourself spirit of DevOps. This is due to the nature of how certain technologies are closed to developer-driven improvements, or roles are irreversibly inaccessible to outsiders. That’s not the case in the mainframe. The mainframe is armed with countless development tools and programmable resources that rarely failed to enable Dev to Ops processes. Then why DevOps practices have not prospered in the mainframe? Ops are already masters of any productive or pre-productive environments – so changing the way developer teams interact with those environments require more politics than technology and are vetted by security practices already in place. New tools don’t target the mainframe – the market and open source communities have focused first on servicing Linux, Windows, mobile and cloud environments. Resistance to change – even if there were new tools and devs could improve processes themselves, management feels that trying out new approaches, especially those that go “outside the box”, could end up putting these environments, and mission-critical releases at risk. Organizations want to profit from DevOps initiatives that are improving the speed and quality of application delivery in the enterprise at a vertiginous pace. But how can they leverage processes that are already in place with the faster and combined pipelines setup in the open side of the house? Enter Clarive for z/OS Our clients have been introducing DevOps practices to the mainframe for many years now. This has been made possible thanks to the well-known benefits of accepting and promoting the bimodal enterprise. There are two approaches that can be used simultaneously in accomplishing: Orchestrate mainframe tools and processes already in place – driving and being driven by the organization’s delivery pipeline Launch modernization initiatives that change the way Dev and Ops deliver changes in the mainframe Business Benefits bringing DevOps to the Mainframe The benefit is simple. Code that runs in the mainframe is expensive and obscure. By unearthing practices and activities, organizations gain valuable insight that can help transform the z/OS-dependent footprint into a more contained and flexible part of the pipeline with these key benefits: Coordinate and Speed-up Application Delivery Mainframe systems don’t run in isolation. The data it manages and the logic it implements are shared as a single entity throughout the enterprise by applications in the open, cloud and even mobile part of the organization. Making changes that disrupt different parts of this delicate but business-critical organism needs to be coordinated at many phases, from testing to UATs to production delivery. Delivering change as a single transactional pipeline has to be a coordinated effort both forward and backwards. End-to-End Visibility DevOps practices perceive the mainframe as a closed box that does not play well with activities that target better visibility and end-to-end transparency. Having dashboards and reports that can work as input and output from the mainframe release processes into other pipelines will help deliver change. Run a Leaner Operation and Avoid Waste Creating mainframe processes that are part of the bigger picture help determine where constraints may lay and what parts of the pipeline may be deemed obsolete or become bottlenecks. Lower Release Costs Mainframe tools are expensive and difficult to manage. MIPS and processing in the mainframe may be capped and new processes could create unwanted expenses. Relying more on tools that drive the mainframe from Linux may in return translate into significant per release cost savings, encouraging a more continuous release process. Use Cases The following is a list of the most relevant benefits of Clarive z/OS and popular use cases that our clients have implemented using the Clarive z/OS platform and tools: Compile and link programs using JCL preprocessed templates. Deploy DB2 items directly to the database. Compile related COBOL programs when Copybooks change Total control what is deployed to each environment at a given time Schedule jobs according to individualized release and availability calendars and windows Request approval for critical deployment windows or sensitive applications or items Keep the lifecycle in sync with external project and issue management applications Run SQA on the changes promoted. Block deployment if a minimum score has not been reached Reliably rollback changes in Production, replacing previous PDS libraries with the correct ones Provision CICS resources on request by users Stay tunned for more of these DevOps for mainframe blog series! Try Clarive now and start bringing DevOps to the mainframe now.