Giovanni Fontana

Cert expired again! How I quickly renewed my OpenShift cluster's certificate and automated renewals with Claude Code!

Mon, 16 Mar 2026 01:00:00 +0000

Ruining My First Coffee!

Nothing ruins a peaceful morning quite like a certificate expiration. There I was, coffee in hand, when I noticed certificate alerts for many of the apps running in my lab. My thoughts:

“Ouch, I forgot to renew it again!”
“I have an important meeting tomorrow, I may need this environment running properly”
“I will have meetings until 11am, can’t do it before then…”
“Wait, I do have Claude Code now… what if I ask it to do it for me?!”

Here’s how it all went down!

For the past year, I’d been managing this certificate manually through a tedious process:

SSH into a remote server to request new certificates from Let’s Encrypt
Download the certificates and encrypt them with Sealed Secrets
Create a new SealedSecret manifest
Commit to Git and wait for Argo CD to sync
Hope everything works without breaking ingress

Every. Single. 90 days.

I wanted to automate this process with cert-manager, but I’d never had the time to create the necessary GitOps configuration files for deployment and setup (I try to deploy and manage everything in my lab using GitOps wherever possible, so I can quickly re-build it from scratch if I have to).

So, this was the last time. I decided this morning would be different. I’d use Claude Code to not only handle this renewal quickly but finally implement the cert-manager with GitOps that I wanted to do long ago.

Act 1: The Quick Fix (Morning Emergency)

First things first—I needed to renew the expiring certificate before thinking about automation. With Claude Code as my AI pair programming partner, what usually took me an hour was done in minutes.

Claude Code helped me:

Navigate to the remote server and run the certificate renewal commands
Properly encrypt the new certificate with kubeseal
Update the SealedSecret manifest in my GitOps repository
Verify the deployment through Argo CD

Time saved: What normally took me 20-30 minutes of manual steps was reduced to no more than 10 minutes (well, honestly it took basically 1 or 2 minutes of my time, the rest was observing Claude to do it and approving execution of codes and commands).

But as I watched Argo CD sync the new certificate, I knew this was just postponing the inevitable. In 90 days, I’d be back here doing the same thing. It was time to automate.

Act 2: The Automation Decision

Over my second coffee, I outlined what I needed:

Automatic certificate renewal: Use cert-manager to automatically renew OpenShift Default Ingress certificate.
GitOps-native: Everything managed declaratively through Argo CD.
Let’s Encrypt integration: Continue using free, trusted certificates.
DNS-01 challenge: For wildcard certificates with Cloudflare DNS.

I told Claude Code: Now, can you read the repo at <folder for my gitops manifests> and suggest changes to be able to use cert-manager to auto renew default ingress cert? cert-manager operator needs to be installed and configure through GitOps (Argo CD)

What followed was an impressive demonstration of how helpful Claude can be to automate repetitive tasks very quickly.

Act 3: The Implementation Journey

Understanding the Environment

Claude Code immediately understood my GitOps repository structure:

Repository: gitops-ocp-infra
Pattern: Cluster-specific configurations with shared operator bases
Target: simpsons cluster at *.apps.simpsons.lab.gfontana.me (Curious why this cluster is named “Simpsons”? Check the reason in the end of this article).
GitOps Engine: Argo CD managing all cluster resources

Phase 1: Leveraging my current GitOps standards

Instead of creating operator manifests from scratch, Claude Code recognized the pattern I use for GitOps and suggested the implementation of cert-manager following the same pattern.

My GitOps follows a layered based approach, in which I separate shared operators and manifests from cluster specific configurations. I also use an Aggregation layer and App-of-Apps to bootstrap all apps. It is not a simple framework, it took me a very long time to develop it and Claude was able to understand it in a matter of seconds. I am planning to publish an article specifically about the way I structure and use GitOps, stay tuned!

One thing that Claude wasn’t able to detect initially was how I re-use manifests from gitops-catalog. Initially it created all manifests to deploy cert-manager operator (OperatorGroup, Subscription, etc.). I had to ask Claude the following:

Note: This folder points to gitops-catalog, which is a fork of this repo.

Claude understood what I wanted and updated the cert-manager operator kustomization YAML.

File: /gitops/manifests/operators/cert-manager-operator/overlays/stable/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://github.com/giofontana/gitops-catalog//openshift-cert-manager-operator/operator/overlays/stable-v1?ref=main

This approach:

Reused well-tested configurations
Maintained a single source of truth
Eliminated duplicate manifests
Referenced the stable-v1 channel automatically

Phase 2: Cloudflare DNS-01 Configuration

For wildcard certificates, DNS-01 challenges are required. Claude Code helped me configure the ClusterIssuer with Cloudflare integration:

File: /gitops/manifests/clusters/simpsons/configuration/cert-manager/clusterissuer-letsencrypt-prod.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@gfontana.me
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - dns01:
        cloudflare:
          email: admin@gfontana.me
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
        - "gfontana.me"

Security note: The Cloudflare API token was stored as a SealedSecret, encrypted and safe for Git storage.

Phase 3: Certificate Resource Definition

Here’s where things got interesting. Claude Code initially created the Certificate resource. We deployed it, watched the logs, and… error.

The commonName Bug Fix

I messed it up! Claude correctly set the commonName and dnsNames. However, I changed it to add some other aliases and mistakenly removed gfontana.me from dnsNames, which caused the certificate request to fail with a validation error.

File: gitops/manifests/clusters/simpsons/configuration/cert-manager-certs/certificate-wildcard-ingress.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-apps-cert
  namespace: openshift-ingress
spec:
  secretName: wildcard-apps-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: "gfontana.me"
  dnsNames:
  - "gfontana.me"  # <-- This line was missing, causing the error
  - "*.gfontana.me"
  - "*.simpsons.lab.gfontana.me"
  - "*.apps.simpsons.lab.gfontana.me"
  usages:
  - digital signature
  - key encipherment
  - server auth
  renewBefore: 720h  # Renew 30 days before expiry

Claude Code not only identified the failure, but also fixed it and asked my permission:

Within minutes of the fix, the certificate was issued successfully by Let’s Encrypt.

Phase 4: IngressController Integration

The final piece was updating OpenShift’s IngressController to use the new certificate. Claude Code created an Argo CD PostSync hook job:

apiVersion: batch/v1
kind: Job
metadata:
  name: patch-ingress-cert-manager
  namespace: openshift-ingress-operator
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
      - name: patch
        image: registry.redhat.io/openshift4/ose-cli:latest
        command:
        - /bin/bash
        - -c
        - |
          oc patch ingresscontroller default \
            -n openshift-ingress-operator \
            --type=merge \
            -p '{"spec":{"defaultCertificate":{"name":"wildcard-apps-tls"}}}'
      restartPolicy: Never
      serviceAccountName: cli-job-sa
  backoffLimit: 4

This job runs automatically after Argo CD syncs, ensuring the IngressController always uses the cert-manager-issued certificate.

Phase 5: Argo CD Application Definitions

Finally, Claude Code created the Argo CD Application resources to manage everything. My initial thinking was to have only a single Argo CD Application for it. However, Claude created two, the first for the operator itself and second for ingress certificate - which makes a lot of sense and even better than I thought initially!

cert-manager Application (Wave 1):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: openshift-gitops
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  destination:
    namespace: openshift-gitops
    server: https://kubernetes.default.svc
  project: default
  source:
    path: gitops/manifests/clusters/simpsons/aggregate/infra/cert-manager/
    repoURL: https://github.com/giofontana/gitops-ocp-infra.git
    targetRevision: main
  syncPolicy:
    automated:
      prune: false
      selfHeal: true

cert-manager-certs Application (Wave 3):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager-certs
  namespace: openshift-gitops
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  destination:
    namespace: openshift-gitops
    server: https://kubernetes.default.svc
  project: default
  source:
    path: gitops/manifests/clusters/simpsons/aggregate/infra/cert-manager-certs/
    repoURL: https://github.com/giofontana/gitops-ocp-infra.git
    targetRevision: main
  syncPolicy:
    automated:
      prune: false
      selfHeal: true

Sync waves ensure proper ordering: operator and issuer first (wave 1), then certificates and ingress configuration (wave 3).

Act 4: “It’s Magic!”

After pushing all changes to Git, I watched Argo CD work its magic:

$ oc get certificate wildcard-apps-cert -n openshift-ingress
NAME                 READY   SECRET              AGE
wildcard-apps-cert   True    wildcard-apps-tls   5m

True means the magic worked! Let me verify:

$ oc get secret wildcard-apps-tls -n openshift-ingress -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -issuer

notBefore=Mar 16 12:00:00 2026 GMT
notAfter=Jun 14 12:00:00 2026 GMT
issuer=C = US, O = Let's Encrypt, CN = R12

Perfect. A legitimate Let’s Encrypt certificate, valid for 90 days, with automatic renewal scheduled for May 15, 2026 (30 days before expiry). The OpenShift console was now serving the Let’s Encrypt certificate. The router pods had restarted automatically, and everything just worked.

What Made Claude Code Special

This wasn’t just about saving time—it was about the quality of the collaboration. Here’s what impressed me:

1. Pattern Recognition

Claude Code immediately understood my GitOps repository structure and created all the new manifests following it. This kind of contextual awareness is exactly what you want in a pair programming partner.

2. Real-Time Debugging

When the certificate request failed due to the missing gfontana.me entry in dnsNames, Claude Code:

Analyzed the error logs
Identified the root cause
Proposed the fix
Implemented it immediately

No googling, or searching Stack Overflow. No reading through cert-manager documentation for 30 minutes.

3. Best Practices Built-In

Sync waves for proper resource ordering
SealedSecrets for sensitive data
PostSync hooks for IngressController patching
Proper RBAC for the patch job

Claude Code knew these patterns and applied them without being told.

4. Multi-Layer Understanding

The implementation required knowledge of:

OpenShift IngressController architecture
cert-manager CRDs and workflows
Kustomize overlays and aggregation
Argo CD application patterns
Kubernetes RBAC and ServiceAccounts
Let’s Encrypt ACME protocol and DNS-01 challenges
Cloudflare API integration

Claude Code navigated all these layers seamlessly and very quickly.

5. Adaptive Problem-Solving

When initial approaches didn’t work, Claude Code didn’t retry the same commands. It analyzed, adapted, and found alternative solutions. That’s intelligent automation.

The icing on the cake - This article!

To end my engagement with Claude in the best way I asked it the following:

Now use all the procedure executed here and write an article about how I used Claude Code this morning to quickly renew OpenShift certificates for my lab and also deployed cert-manager and configure it with Argo CD to auto renew the certificates. Write an article in my github pages at: <my github pages>

It learned my GitHub pages structure and wrote a big part of this article for me. I just had to review it, add my personal touch and style, and voilà—a new blog post ready to publish!

Conclusion: A Morning Well Spent

What started as a frustrating certificate expiration alert turned into a nice experimentation experience with Claude Code. With Claude Code as my AI pair programming partner, I:

Quickly renewed an expiring certificate in minutes instead of hours
Deployed cert-manager using GitOps best practices
Automated renewals: certificates now auto-renew 30 days before expiration
Eliminated toil and saved a few hours annually
Learned patterns I can apply to other infrastructure automation
Shared my experience by writing this article

The real revelation wasn’t just the time saved—it was the quality of the collaboration. Claude Code understood my environment, adapted to my patterns, debugged issues in real-time, and applied best practices throughout.

What about you? Are you using AI assistants like Claude Code to automate repetitive tasks? Share your thoughts! I’d love to hear your experiences!

Resources

Repository: gitops-ocp-infra
cert-manager Documentation: cert-manager.io
Claude Code: Anthropic’s AI-Powered CLI
Argo CD: argoproj.github.io/cd

Why “Simpsons”?

A couple of years back I struggled a lot to find good names for my lab resources and easily remember them. So I decided to name all my lab resources after cartoons. I love Simpsons, so decided to name my main cluster as simpsons. This cluster has 4 nodes: Homer, Marge, Bart and Lisa. The node with more resources - biggest or fatter - is obviously Homer. The smartest (which has a GPU) is Lisa. And so on. I have another small cluster that sits next to Simpsons, what is its name? Flanders, of course!

So, that way I gave names that are meaningful and help me remember what they have… and it is also kind of funny! lol

The Cost of Generative AI

Mon, 02 Feb 2026 21:05:55 +0000

Understanding the True Cost of AI Systems: Lessons from MIT Professional Education

Recently, I started the course “Applied Agentic AI for Organizational Transformation” with MIT Professional Education to learn more about Agentic AI and all the buzzwords surrounding AI these days. To my surprise, the first module is not about Agentic AI itself, but about the cost of AI Systems. My first reaction was, “Wait, what?! Why start with costs?!”

As it turns out, it was quite eye-opening to learn how complex the cost structure of Large Language Models (LLMs) truly is. There are numerous components you must consider to keep your budget under control. This article is inspired by the first assignment of this course, featuring a hypothetical AI assistant called DIYgpt. I hope this helps shed light on what you need to consider for your own applications.

The Use Case: DIYgpt

To evaluate costs, I envisioned DIYgpt, an AI assistant designed to guide DIY enthusiasts through home projects like carpentry and repairs. Users describe a task, and the AI generates a step-by-step tutorial with a parts list.

To get more realistic data, I tested prompts across different models (like ChatGPT and Google Gemini) to find the average token counts for inputs and outputs.

The Data Points

The prompts below were utilized for testing and establishing the average token consumption. It should be noted that a real-world use case would typically involve a greater number of prompts, more iterations, and a wider variety of models.

1) “I want to build a simple wooden bookshelf for my home office. It needs to be 4 feet tall and 3 feet wide with three shelves. What materials do I need to buy at the hardware store, and what are the step-by-step assembly instructions?”

2) “My kitchen faucet is leaking from the base of the handle whenever I turn the water on. I have a standard hex wrench set and a screwdriver. Can you walk me through how to diagnose and fix this?”

3) “I just bought my first power drill. I see a bunch of numbers on a dial near the tip (1 through 20). What those numbers mean, and which one should I use if I’m just driving a screw into a piece of pine?”

The result is shown below.

Model	Question	Prompt (tokens)	Response (tokens)
ChatGPT	#1	52	834
		52	828
		52	711
Google Gemini Fast		52	620
		52	643
		52	704
ChatGPT	#2	41	630
		41	541
		41	516
Google Gemini Fast		41	622
		41	675
		41	628
ChatGPT	#3	50	402
		50	288
		50	272
Google Gemini Fast		50	381
		50	465
		50	434
Average		47	566

Therefore, the following average token counts was used in this execise:

Input: 47 tokens (average of all inputs).
Output: 566 tokens (average of all outputs).

Projecting the “Burn”: Daily and Monthly Costs

To estimate usage of the DIYgpt, the following assumptions were made:

Daily Active Users (DAU): 5,000
Sessions per User: 2
Queries per Session: 3

Total: 5,000 users * 2 sessions * 3 queries = 30,000 API calls per day

Additionally, an average of 10% of inputs were assumed to be cached.

1. Daily Token Volume

Total volume of data being processed by the API:

Total Input Tokens: 30,000 calls * 47 tokens = 1,410,000 tokens/day
Standard Input Tokens (90%): 1,269,000 tokens
Cached Input Tokens (10%): 141,000 tokens
Total Output Tokens: 30,000 calls * 566 tokens = 16,980,000 tokens/day

2. Daily Cost Breakdown

Applying the GPT-4.1 pricing model, as an example:

Standard Input Cost: 1.269M tokens * $3.00 = $3.81
Cached Input Cost: 0.141M tokens* 0.75 = $0.11
Output Cost: 16.98M tokens * $12.00 = $203.76
Total Daily Cost: $207.68

3. Final Monthly Totals

Assuming a standard 30-day billing cycle:

Metric	Calculation	Result
Daily Burn	Sum of all token costs	$207.68
Monthly Burn	$207.68 * 30 days	$6,230.40
Cost Per Query	$207.68 / 30,000 calls	$0.0069

4. Cost of the models:

Following the same line of reasoning for the remaining models, I compared several models. The difference in “daily burn” is staggering:

Model	Daily Cost	Monthly Cost (30 Days)	Cost Per Query
o4-mini	$276.90	$8,306.91	$0.0092
GPT-4.1	$207.67	$6,230.18	$0.0069
Gemini 3 Pro Preview	$206.33	$6,189.79	$0.0069
GPT-4.1 mini	$55.38	$1,661.38	$0.0018
Gemini 3 Flash Preview	$51.58	$1,547.45	$0.0017
GPT-4.1 nano	$13.84	$415.35	$0.0005

The table and chart above illustrate the monthly “burn”. As you can see:

o4-mini sits at the top, representing the premium price for reasoning capabilities (approx. $8,306 per month). This is sort of expected, as o4-mini is a reasoning model (the “o” series) which usually charges a premium for its “hidden” reasoning tokens - you are paying for the quality of the thought process, not just the final words.
GPT-4.1 nano occupies the lowest bracket, demonstrating how architectural choices can reduce costs by over 95% compared to the most expensive model.The mid-range models (GPT-4.1 and Gemini 3 Pro) are nearly identical in cost, making their selection a matter of latency and accuracy rather than budget.
The mid-range models (GPT-4.1 and Gemini 3 Pro) are nearly identical in cost, making their selection a matter of latency and accuracy rather than budget.

The staggering difference in cost between these models is a great segue to what I want to discuss next.

Beyond the Numbers: A Comprehensive Decision

While cost is a primary driver, choosing a model based solely on the “monthly burn” can be a mistake. To make a truly informed decision for an organization, you must consider additional factors, such as:

1. Accuracy and Domain Expertise

When evaluating AI systems, accuracy is paramount; a model with unacceptably low accuracy is essentially useless, regardless of its low cost. Therefore, you must define a strict minimum accuracy threshold and discard any model that fails to meet it. On the other hand, many applications do not require a costly, super-powerful reasoning model. The challenge lies in balancing performance, tailored to your specific domain, with cost. This is an ever-evolving field - recent developments include “LLM-as-a-judge” concepts, but you can start with a simpler approach: use a “golden set” of questions and employ keyword detection in the responses to gauge if they align with your desired outcome.

2. Latency

In a “DIY” scenario, users don’t want to wait 1 minute for a response while holding a power drill. Higher-performing models are often slower, whereas “Nano” or “Flash” models are optimized for speed, but may sacrifice some accuracy. Therefore, when selecting a model, you must prioritize response time for the end-user while balancing overall cost, efficiency, and performance.

3. Context Length and Memory

As users engage in longer sessions, the “memory” of the conversation grows. Models have different limits on how much information they can “remember” (context length). If your project requires analyzing long manuals or maintaining a multi-day conversation, you may need a model with a larger context window, which often comes at a higher price point, as well.

4. Caching Efficiency

Modern APIs allow you to save money by “caching” frequent inputs. In our DIYgpt exercise, I assumed 10% of inputs were cached, which significantly reduced the cost for repetitive instructions. The actual caching percentage is highly dependent on the domain and can fluctuate. Continuous monitoring is essential to establish a more accurate, realistic number over time.

Conclusion

Starting with costs wasn’t just an academic exercise - it was a lesson in viability. Whether you are building a simple assistant or a complex agentic system, understanding your “cost per query” is the first step in ensuring your AI transformation is sustainable in the long run. The DIYgpt case study demonstrated that the cost of AI is not merely a background detail but a decisive factor in a project’s survival. As our financial analysis revealed, the choice of model can result in a “staggering” disparity in daily burn rates - ranging from as low as $13.84 for GPT−4.1 nano to $276.90 for o4-mini, daily cost.

However, the lowest price tag does not guarantee the best outcome. True optimization requires looking “beyond the numbers” to balance cost efficiency with user experience. In a real-world scenario - such as a user waiting for instructions while holding a heavy power drill - metrics like latency and accuracy become just as valuable as the raw cost per token.

Ultimately, this exercise proves that sustainable AI transformation is not just about code; it is about architecture and economics. By leveraging technical efficiencies like caching and selecting models that offer the right accuracy for the specific domain, developers can bridge the gap between a cool prototype and a scalable business solution. As we move toward more complex Agentic systems, where autonomous loops increase token consumption, this ability to forecast and manage the “burn” will be the defining characteristic of successful organizational transformation.

References:

The following price list is reported on the OpenAI pricing website as of the time of writing this document.

Model	Input ($/1M tokens)	Cached Input ($/1M tokens)	Output ($/1M tokens)
GPT-4.1	3.00	0.75	12.00
GPT-4.1 mini	0.80	0.20	3.20
GPT-4.1 nano	0.20	0.05	0.80
o4-mini	4.00	1.00	16.00

This is the pricing reported for Gemini models through Vertex AI platform. It was assumed that the query input context is less than 200K tokens.

Model	Price (/1M tokens) <= 200K input tokens	Price (/1M tokens) <= 200K cached input tokens	Output ($/1M tokens)
Gemini 3 Pro Preview	2	0.2	12
Gemini 3 Flash Preview	0.5	0.05	3

Automate VM golden image builds for OpenShift with Packer

Thu, 06 Nov 2025 21:05:55 +0000

Originally published at https://developers.redhat.com on November 07, 2025.

In any virtualized environment, maintaining consistency across virtual machines (VMs) is a major challenge. Golden images (pre-configured VM templates) are the industry-standard solution. They ensure every new VM comes with the correct OS settings, security patches, and monitoring tools baked in. But how do you create and manage these images without tedious manual work?

This is where Packer comes into the picture. Packer is a tool that automates the creation of identical machine images for multiple platforms from a single template. In this article, we’ll show you how to use Packer with the KVM plugin to build golden images specifically for Red Hat OpenShift Virtualization.

Packer

Packer allows you to create images from a single source template, described using either HashiCorp Configuration Language (HCL) or JSON. It provides a large collection of plugins to create machines and images for different platforms, such AWS, Azure, VMware, KVM, and others. Refer to the Packer Integrations page for more information.

Red Hat OpenShift Virtualization

Red Hat OpenShift Virtualization, built on KubeVirt, is a feature of Red Hat OpenShift that allows you to run and manage traditional virtual machines alongside containers on a single, hybrid cloud platform. This unification of both containerized and virtualized workloads simplifies management and enables a consistent operational experience across your entire infrastructure.

Packer + OpenShift Virtualization: Better together

Through the KVM plugin and with a few simple commands, you can make your golden image ready on OpenShift Virtualization. The GitHub repository includes a collection of examples for creating images for various operating systems, including Linux (Fedora, RHEL 9) and Windows.

Later in this article, we will dig into one of these examples, but the general workflow is simple. You define a Packer template (HCL) that specifies the operating system, configuration scripts, and other customizations. Packer then uses this template to build the VM image automatically using KVM. The output will be a qcow2 image file for you to upload to OpenShift Virtualization and use as a golden image for your VM provisioning.

Once Packer completes the qcow2 file creation, uploading it to OpenShift and setting it as a bootable image are steps easily automated as part of a pipeline or with automation tools like Red Hat Ansible Automation Platform or Terraform.

Key features and benefits:

Automation: Eliminate tedious manual image creation, saving time and preventing configuration drift.
Customization: Easily embed specific software, user configurations, and security settings directly into the image.
Performance optimization: Build images with the best drivers for your environment, like using virtio drivers for Windows on KVM to achieve near-native performance.
Reduced risk: Automating the image creation process ensures a standardized and less error-prone approach compared to manual methods.
Simple integration: The build process outputs a qcow2 file that can upload to OpenShift Virtualization with a few simple commands, perfect for CI/CD pipelines.

Getting started

The project’s GitHub repository provides detailed instructions on how to get started. Here’s a quick overview of the steps involved:

Clone the repository: Obtain a local copy of the project.
Install Packer and KVM: Install Packer on your system.
Choose an example: The repository includes examples for different operating systems.
Build the image: Run the Packer build command to create your custom VM image.
Upload to OpenShift: Use the virtctl and oc command-line tool to upload the image to OpenShift Virtualization.

Let’s walk through the Windows Server 2019 example to see how the automation comes together. The windows2019.pkr.hcl Packer template defines the process and orchestrates the entire build.

The source block defines the crucial first step, where Packer creates and attaches a virtual CD-ROM to the new VM. This CD contains all the files needed for an unattended installation:

# This creates a virtual CD with our automation files and drivers.
cd_files = [
  "./autounattend.xml",
  "./scripts/Configure-WinRM.ps1",
  "./virtio/"
]
cd_label = "PACKERDRV" # A label for our driver CD

The Windows installer then uses the autounattend.xml file from this virtual CD as its answer file to automate the setup. Inside this XML file, we point the installer to the virtio drivers, also located on the virtual CD (which typically mounts as the E: drive). This allows Windows to use the high-performance KVM-native storage and network drivers right from the start.

<DriverPaths>
    <PathAndCredentials wcm:action="add" wcm:keyValue="1">
        <Path>E:\virtio\viostor\2k19\amd64</Path>
    </PathAndCredentials>
    <PathAndCredentials wcm:action="add" wcm:keyValue="2">
        <Path>E:\virtio\NetKVM\2k19\amd64</Path>
    </PathAndCredentials>
</DriverPaths>

After the OS installation is complete, the autounattend.xml file has one more critical job. The FirstLogonCommands section automatically runs our PowerShell script.

<FirstLogonCommands>
    <SynchronousCommand wcm:action="add">
        <CommandLine>powershell.exe -ExecutionPolicy Bypass -File E:\Configure-WinRM.ps1</CommandLine>
        <Description>Install and Configure SSH</Description>
        <Order>1</Order>
    </SynchronousCommand>
</FirstLogonCommands>

This script configures Windows Remote Management (WinRM), which opens a communication channel back to Packer. Once Packer can connect to the VM via WinRM, it takes over to run post-install configurations. In this example, it executes scripts to install Windows updates before finalizing the qcow2 image.

From image to OpenShift VM

Once Packer creates the qcow2 file, the final step is to make it available in OpenShift Virtualization:

Upload the image: First, you upload the qcow2 file, which creates a Persistent Volume Claim (PVC) that serves as a bootable volume for your new virtual machines.
Create a template (optional): For better reusability, you can create a custom VM template that points to this new bootable volume. This allows developers and operators to provision new, fully configured VMs with just a few clicks from the OpenShift console.

For organizations managing multiple OpenShift clusters, distributing and managing these golden images can be streamlined using Red Hat Advanced Cluster Management (RHACM). RHACM policies can automate the distribution of these templates and their associated resources across your entire fleet, ensuring consistency and saving significant operational effort.

The repository’s README.md file provides detailed instructions for this, including an example of how to use Red Hat Advanced Cluster Management policies to distribute these images across multiple clusters.

Simplify your VM images on OpenShift

By automating the VM image creation process, teams improve efficiency, consistency, and scalability. Whether you’re a developer, a system administrator, or a DevOps engineer, the powerful combination of Packer and OpenShift Virtualization can help you streamline your workflows and make the most of your platform.

If you’re ready to streamline how you manage VM images on OpenShift, dive into the openshift-virt-packer repository. Clone it, run an example, and start building your own automated image pipeline today with Packer and OpenShift Virtualization. You can also find more information in the Packer and OpenShift Virtualization - Managing VMs documentation.

To learn more about Red Hat OpenShift Virtualization, check out the 15 reasons to adopt Red Hat OpenShift Virtualization e-book and explore how to build a migration plan with Red Hat experts through the Virtualization Migration Assessment.

Effective observability with Red Hat build of OpenTelemetry | Red Hat Developer

Tue, 02 Sep 2025 21:05:55 +0000

Originally published at https://developers.redhat.com on September 3, 2025.

In today’s complex IT environments, monitoring and understanding the health and performance of your applications and infrastructure is critical. The Red Hat build of OpenTelemetry, which can be installed in Red Hat OpenShift, provides a powerful framework for collecting and exporting telemetry data, enabling comprehensive metrics and logs reporting. In this article, we will explore the benefits and capabilities of using the Red Hat build of OpenTelemetry for effective observability.

OpenTelemetry is an open source project under the Cloud Native Computing Foundation (CNCF) that provides a set of APIs, libraries, agents, and collectors to capture distributed traces, metrics, and logs. Red Hat build of OpenTelemetry is a distribution of the upstream OpenTelemetry project, built and supported by Red Hat.

Collecting metrics with Red Hat build of OpenTelemetry

Metrics provide insights into the performance and health of your applications. The Red Hat build of OpenTelemetry can collect various types of metrics, including:

System metrics: CPU usage, memory consumption, disk I/O, etc.
Application metrics: Request latency, error rates, and throughput.
Custom metrics: Business-specific metrics tailored to your application needs.

You can export these metrics to various monitoring systems, such as Prometheus, for visualization and analysis.

Key benefits of Red Hat build of OpenTelemetry

The Red Hat build of OpenTelemetry simplifies the collection and management of telemetry data, offering several key advantages:

Standardization: Provides a unified way to collect and export data to third-party tools, reducing vendor lock-in.
Scalability: Designed to handle large volumes of data in complex environments.
Flexibility: Supports multiple data formats and backends, allowing integration with existing monitoring tools.
Red Hat support: Offers enterprise-grade support and maintenance from Red Hat, integration with Red Hat Observability UI and supported integration with third-party vendors.

Using Red Hat build of OpenTelemetry

You can seamlessly integrate Red Hat build of OpenTelemetry with existing monitoring and logging systems. The OpenTelemetry collector acts as a central hub for receiving, processing, and exporting telemetry data. OpenTelemetry Collector is composed of the following components.

Receivers:

Receives telemetry data from different sources. Supported receivers include: OTLP, Jaeger, Host Metrics, Kubernetes Objects Receiver, and many others. Refer to the full list of supported receivers.

Exporters:

Send telemetry data to various backends. Supported exporters include: OTLP, OTLP HTTP, Prometheus, and others. Refer to the full list of supported exporters.

Processors:

Manipulate data between received and sent (optional but very helpful). Supported processors include: batch, attributes, kubernetes attributes, and others. Refer to the full list of supported processors.

Check out the OpenTelemetry architecture documentation for more information.

Getting started with Red Hat build of OpenTelemetry

To get started with the Red Hat build of OpenTelemetry, you can follow these general steps.

Deploy the Red Hat build of OpenTelemetry operator: Deploy the Red Hat build of OpenTelemetry operator in your OpenShift cluster.
Deploy the OpenTelemetry collector: Deploy the OpenTelemetry Collector in your environment.
Visualize your data: Use tools like Grafana to visualize and analyze your metrics and logs.

Deploy of Red Hat build of OpenTelemetry operator

The deployment process of the Red Hat build of OpenTelemetry operator is very straightforward. In your OpenShift cluster, follow these steps:

Go to Operators OperatorHub and search for the Red Hat build of OpenTelemetry operator.
Select the Red Hat build of OpenTelemetry Operator that is provided by Red Hat and click in Install. Leave the default presets: a) Update channel: stable b) All namespaces on the cluster c) Installed Namespace: openshift-operators d) Update approval: Automatic
In the Details tab of the Installed Operator page, under ClusterServiceVersion details, verify that the installation Status is Succeeded.

Deploy OpenTelemetry collector

The OpenTelemetry collector is where all the magic happens. In the collector, you will define all the receivers, processors, and exporters you want in your environment. Let’s dig into the receivers that make more sense to use in a Kubernetes environment.

Host metrics receiver: Designed to collect performance metrics directly from the underlying host operating system. This includes vital information about CPU utilization, memory consumption, disk I/O, network traffic, and other system-level statistics. By monitoring these metrics, users gain insights into the health and resource utilization of the nodes running their Kubernetes cluster. This receiver is crucial for identifying potential bottlenecks or resource exhaustion at the host level.
Kubernetes objects receiver: Provides a way to observe the state of Kubernetes resources. It can monitor various Kubernetes objects such as pods, nodes, deployments, services, and more. The receiver captures metadata and status information about these objects, allowing users to track their health, configuration, and lifecycle events. This is essential for understanding the overall health and configuration of the Kubernetes cluster itself.
Kubelet stats receiver: Focuses on collecting detailed performance metrics directly from the Kubelet, the primary node agent in Kubernetes. These metrics offer granular insights into the resource usage and performance of individual pods and containers running on a specific node. Information such as CPU and memory usage per container, network statistics, and disk I/O at the pod level can be obtained through this receiver, enabling fine-grained monitoring and troubleshooting of application performance within the Kubernetes environment.
Kubernetes cluster receiver: The Kubernetes cluster receiver offers a broader view of the Kubernetes cluster’s health and performance. It aggregates information from various Kubernetes components, providing insights into the control plane’s operation, such as the API server’s health, scheduler performance, and controller manager activity. This receiver helps in understanding the overall stability and responsiveness of the Kubernetes control plane.
Filelog receiver: For log collection, the filelog receiver is a versatile option. It allows the OpenTelemetry collector to tail log files present on the file system of the nodes. This is particularly useful for collecting application logs written to files, as well as logs from other services running on the Kubernetes nodes. The receiver supports various configuration options for specifying file paths, log formats, and multiline log processing.
Journald receiver: Specifically designed for systems utilizing systemd’s journal for logging. It enables the OpenTelemetry collector to directly ingest logs from the systemd journal. This is beneficial in modern Linux distributions where systemd is the default init system, providing a centralized and structured way to collect system and application logs.
Kubernetes events receiver: Allows the OpenTelemetry collector to capture Kubernetes events. These events provide valuable insights into significant occurrences within the cluster, such as pod creation, deletion, scaling events, and resource issues. By collecting and analyzing Kubernetes events, users can gain a better understanding of the dynamics of their cluster and troubleshoot issues proactively.

Strategically select the receivers that make sense to your use case. You might not need all of them, but enabling the receivers mentioned previously will provide a very comprehensive metrics and logs reporting system tailored to Kubernetes environments. By selecting only specific receivers, you will keep the footprint of opentelemetry-collector low, allowing you to deploy the collector even in environments with limited resources.

Pipelines

A pipeline defines the complete lifecycle of telemetry data. This journey begins with the reception of data from various sources, continues through optional processing stages where the data can be transformed, enriched, or filtered, and culminates in the export of the data to one or more back-end destinations for storage, visualization, or analysis. A pipeline typically looks like this:

    service:
      pipelines:
        metrics:
          receivers:
            - hostmetrics
            - kubeletstats
            - k8s_cluster
          processors:
            - k8sattributes
          exporters:
            - debug

There are various types of pipelines:

Metrics: These pipelines are designed to handle numerical measurements captured at specific points in time. Metrics are essential for monitoring the performance and resource utilization of systems and applications. They often include aggregated data and can be used to create dashboards and set up alerts.
Logs: These pipelines deal with textual records of events that occur within a system or application. Logs provide detailed information about the behavior and state of software and infrastructure. Effective log management involves collecting, processing, and storing logs in a structured and searchable manner.
Traces: These pipelines are concerned with tracking the execution of requests as they propagate through distributed systems. A trace represents the end-to-end journey of a request, consisting of individual spans that represent specific operations or calls. Tracing is crucial for understanding the flow of execution, identifying performance bottlenecks, and diagnosing issues in microservice architectures.

Each of these pipeline types has its own set of specialized receivers, processors, and exporters that are tailored to the specific characteristics of the telemetry data they handle. By configuring pipelines appropriately for metrics, logs, and traces, users can gain comprehensive observability into their applications and infrastructure using the Red Hat build of OpenTelemetry.

Now that we know how a collector works, let’s define a collector that will have all the receivers previously mentioned.

Receivers

The full collector is available here. The most important pieces are the following.

Host metrics receiver

Available scrapers for host metrics:

cpu: This scraper collects CPU utilization metrics, including information about user, system, idle, and wait times. Analyzing CPU metrics helps identify periods of high CPU load, which can indicate performance issues or the need for resource scaling.
memory: The memory scraper gathers information about memory usage, such as total, used, free, cached, and buffered memory. Monitoring memory metrics is critical for detecting memory leaks, insufficient memory allocation, or excessive memory consumption by applications.
disk: This scraper collects disk I/O metrics, providing insights into read and write operations, throughput, and latency for the host’s disks. Analyzing disk metrics can help identify slow disks or I/O bottlenecks that might be affecting application performance.
load: The load scraper collects system load averages (1-minute, 5-minute, and 15-minute averages), which provide a general indication of system utilization. High load averages can suggest that the system is overutilized.
filesystem: This scraper gathers metrics related to file system usage, including total size, used space, and available space for each mounted file system. Monitoring file system usage helps prevent disks from becoming full, which can lead to application failures.
paging: The paging scraper collects metrics about the host’s swap or page file usage, such as swap in and swap out rates. Excessive paging activity can indicate that the system is experiencing memory pressure.
processes: This scraper provides a count of the total number of processes running on the host. process: This scraper allows for the collection of detailed metrics about specific processes running on the host. It typically requires further configuration to specify which processes to monitor and the specific metrics to collect (e.g., CPU usage, memory usage, I/O).

Example:

    receivers:
      hostmetrics:
        collection_interval: 60s
        initial_delay: 1s
        root_path: /
        scrapers: 
          cpu: {}
          memory: {}
          disk: {}
          load: {}
          filesystem: {}
          paging: {}
          processes: {}
          process: {}    

Kubernetes objects

Monitored objects:

name: pods: This configuration instructs the receiver to collect information about Kubernetes pod objects.
- mode: pull: The pull mode indicates that the receiver will periodically query the Kubernetes API server to retrieve the current state of all pod objects in the cluster.
- interval: 60s: This parameter specifies how frequently the receiver will poll the API server for pod information. In this case, it will check every 60 seconds.
name: events: This configuration tells the receiver to collect Kubernetes event objects.
- mode: watch: The watch mode utilizes the Kubernetes watch API, which allows the receiver to receive near real-time notifications whenever events occur in the cluster. This is more efficient than periodically polling, as the receiver only receives updates when changes happen.

Example:

     k8sobjects:
        auth_type: serviceAccount
        objects:
          - name: pods
            mode: pull 
            interval: 60s
          - name: events
            mode: watch

Kubeletstats

Configuration parameters:

endpoint: This parameter defines the URL of the Kubelet API endpoint. The configuration uses the environment variable K8S_NODE_NAME to dynamically determine the hostname or IP address of the current node. The standard Kubelet port for secure communication is 10250. The https:// scheme ensures that communication with the Kubelet is encrypted.

Example:

      kubeletstats:
        collection_interval: 60s
        auth_type: "serviceAccount"
        endpoint: "https://${env:K8S_NODE_NAME}:10250"
        insecure_skip_verify: true

K8s_cluster

Configuration parameters:

distribution: This parameter specifies the type of Kubernetes distribution being monitored. In this example, it is set to openshift, indicating that the metrics and logs are being collected from an OpenShift cluster. This information can be crucial for tailoring collection methods and interpreting cluster-specific data.

Example:

      k8s_cluster:
        distribution: openshift
        collection_interval: 60s  

K8s_events

To limit the scope of event collection to specific namespaces, the namespaces parameter can be defined as a list of namespace names. This allows for focused monitoring and reduces the volume of event data being processed. In this example, the namespaces are commented out, so all namespaces will be collected.

Example:

      k8s_events:
        namespaces: [project1, project2]

Journald configuration breakdown:

files: /var/log/journal/*/: This parameter specifies the path to the journal files that the collector should monitor. The wildcard characters * allow for matching across different journal file directories and individual journal files within those directories. This ensures comprehensive log collection from all persistent journal logs.
priority: info: This setting defines the minimum severity level of logs to be collected. By setting it to info, the collector will gather all logs with a priority of info and higher (e.g., warning, err, crit, alert, emerg). This allows for filtering out less critical debug-level messages, reducing noise and focusing on more important events.
units: This section allows for filtering logs based on specific systemd units.
- kubelet: This entry specifies that logs originating from the kubelet systemd unit should be collected. The kubelet is the primary “node agent” in Kubernetes, responsible for running containers on a worker node. Collecting its logs is crucial for monitoring the health and status of pods and the node.
- crio: This entry indicates the collection of logs from the crio (Container Runtime Interface) unit. CRI-O is a lightweight container runtime for Kubernetes, and its logs provide insights into container lifecycle events, image management, and other runtime-related activities.
- init.scope: This entry configures the collection of logs from the init.scope unit. init.scope represents the system’s initialization process and often contains essential boot-related information and early system messages.
- dnsmasq: This entry specifies the collection of logs from the dnsmasq unit. dnsmasq is a lightweight DNS forwarder and DHCP server commonly used in Kubernetes clusters for internal DNS resolution. Monitoring its logs can be helpful for diagnosing DNS-related issues within the cluster.
all: true: This boolean parameter, when set to true, instructs the collector to gather logs from all other journal entries that are not explicitly excluded by other filters (i.e., the units list). This acts as a catch-all to ensure that important system-level logs beyond the specified units are also captured.
retry_on_failure: This section configures how the collector should handle failures during log collection.

Example:

      journald:
        files: /var/log/journal/*/*
        priority: info 
        units:
          - kubelet
          - crio
          - init.scope
          - dnsmasq
        all: true
        retry_on_failure:
          enabled: true
          initial_interval: 1s
          max_interval: 60s
          max_elapsed_time: 5m

You can find the complete OpenTelemetry collector on GitHub.

Depending on the receivers used, the collector needs different permissions. You will find the permissions in the documentation of each receiver. For the receivers listed above, the necessary permissions are available at the repo.

To install the collector with all necessary permissions, run the following:

git clone https://github.com/giofontana/rh-build-opentelemetry.git
cd rh-build-opentelemetry
oc apply -k manifests/overlays/debug

Testing the OpenTelemetry collector

Initially, our collector only logs the metrics collected by the receivers using the debug exporter. To fully leverage the collected metrics, we will now change the configuration to use OTLP/HTTP exporter, enabling the collector to send the gathered metrics data to a remote system accessible via HTTP using the OpenTelemetry Protocol (OTLP).
For demonstration and testing of the OpenTelemetry setup, we will deploy the following systems in a Red Hat Enterprise Linux virtual machine. These systems are essential for illustrating the complete data flow, from collection through processing to visualization and storage. The following systems are deployed in the VM:

Grafana: Grafana provides a robust platform for creating and displaying dynamic dashboards. It will be configured to connect to the metrics storage solution and present the collected data in a visually intuitive format, enabling detailed analysis and monitoring.
Mimir: Mimir is an open source time-series database system that serves as the primary storage for the metrics exported by the OpenTelemetry collector. Mimir’s scalable architecture and efficient query engine make it ideal for handling large volumes of time-series data.
Loki: Loki is a horizontally scalable, highly available, multi-tenant log aggregation system. It will be used to manage and store log data, complementing the metrics data stored in Mimir.
Promtail: Promtail is an agent that collects logs from various sources and sends them to Loki. It will be configured to scrape logs from the OpenTelemetry collector and other relevant system components, ensuring that all relevant log data is captured and accessible in Loki.

Detailed, step-by-step instructions on how to deploy these systems in a virtual machine are not the scope of this article, but you can refer to the documentation for more information.
With Mimir up and running, we will now reconfigure the OpenTelemetry collector to send metrics to it. To do so, change the endpoint of spec.config.exporters.otlphttp.endpoint to reflect your environment:

#Change line highlighted below:
vi manifests/overlays/all/opentelemetry-collector.yaml
   exporters:
      debug:
        verbosity: basic
      otlphttp:
        endpoint: 'http://10.1.1.100:9009/otlp' #CHANGE IP
        tls:
          insecure: true
      otlphttp/logs:
        endpoint: 'http://10.1.1.100:3100/otlp' #CHANGE IP
        tls:
          insecure: true

You might need to add other parameters depending on TLS and other configurations you have on your external system. Check this documentation for more information about configuration parameters.
With the opentelemetry-collector.yaml properly configured, you can deploy the new collector.

#Delete the existing one, if exists
oc delete OpenTelemetryCollector otel -n k8s-otel
#Deploy the new one
oc apply -k manifests/overlays/all/

To verify the setup, add a new data source to Grafana for Mimir as follows:

Access Grafana at http://:3000/
Use admin / admin as the initial password. Set a new password for the admin user.
Go to Connections -> Add new connection.
Choose Prometheus and click Add new data source.
Enter the following details:
- Name: Mimir
- Prometheus server URL: http://:9009/prometheus
- Authentication: None
- Skip TLS certificate validation

Then, use the Explore function. You should now see available metrics, like container_cpu_time (Figure 1).

Figure 1: The Grafana Explore feature, with Mimir data source, shows a metric collected and sent by OpenTelemetry.

Do the same for Loki:

Go to “Connections” -> “Add new connection”.
Choose Prometheus and click “Add new data source”.
Enter the following details:
- Name: Loki
- Prometheus server URL: http://:3100
- Authentication: None
- Skip TLS certificate validation

Then, use the Explore function. Select Loki, any filter (e.g., k8s_namespace_name=k8s-otel), and click on Run query, as shown in Figure 2.

Figure 2: The Grafana Explore feature, with Loki datasource, shows logs collected and sent by OpenTelemetry.

To test OpenTelemetry, you can import this dashboard: https://grafana.com/grafana/dashboards/20376-opentelemetry-collector-hostmetrics-node-exporter/

Navigate to Dashboards -> New -> Import.
Enter 20376 and click the Load button, as shown in Figure 3.

Figure 3: This example of a Grafana dashboard shows graphs and statistics from metrics collected by OpenTelemetry.

Summary

Red Hat build of OpenTelemetry provides a powerful and flexible solution for comprehensive metrics and logs reporting in complex environments. By standardizing data collection, offering robust scalability, and providing enterprise-grade support from Red Hat, it simplifies observability and enables deeper insights into application and infrastructure health. With its seamless integration capabilities and a rich set of receivers, processors, and exporters, the Red Hat build of OpenTelemetry allows you to tailor your monitoring setup to meet your specific needs.
To fully leverage the capabilities of the Red Hat build of OpenTelemetry and enhance your monitoring strategy, explore the official documentation and community resources. Dive deeper into configuring collectors, setting up pipelines, and integrating with visualization tools like Grafana. You can find detailed information and getting started guides to help you implement and optimize your observability practices. Find out more about Red Hat build of OpenTelemetry and Red Hat OpenShift observability.

Applied Data Science Program: Levaraging AI for Effective Decision Making

Mon, 29 Jul 2024 21:05:55 +0000

🎉 I did it! Thrilled to share this milestone with all of you! Now, I won’t be totally lost in AI conversations! 😂

On a serious note, this course has truly transformed my understanding of artificial intelligence. Generative AI often feels like a magic box, and without grasping the fundamentals, it’s easy to get lost. This course has been a game-changer for me, covering everything from the foundations of Data Science, Data Analysis, and Visualization to Neural Networks and Recommendation Systems.

Feel free to reach out if you’d like to know more about my experience!

New Book Announcement - OpenShift Multi-Cluster Management Handbook

Thu, 10 Nov 2022 21:05:55 +0000

It’s been a long road to get here, but we can finally make this announcement! Our new book OpenShift Multi-Cluster Management Handbook is going to be released soon! It is available for preorder on Amazon already.

Our initial goal with OpenShift Multi-Cluster Management Handbook was to cover a wide range of topics that we found relevant while performing OpenShift-related activities in our career. We have been working for years with a variety of different tasks, such as clusters deployment, architectural design, troubleshooting, and others related to OpenShift, during this time we were able to accumulate experiences and lessons learned, such as:

The main challenges many companies face when adopting public cloud;
Common pitfalls with OpenShift adoption;
Architectural best practices;
Security concerns and best practices;
Main personas that work with OpenShift, their expectations, challenges, and key responsibilities;

Our intention with this book was not only to provide technical content to help you learn about new topics but also to cover some topics that you can usually only learn on the job, with practical experience. Additionally, you will discover some up-to-date materials that will assist you to survive in this world of several OpenShift clusters living on multiple platforms, from on-premise to the cloud, using tools that make up the OpenShift Plus offering - Red Hat Advanced Cluster Management, Advanced Cluster Security, and Quay.

It was our first time writing a book, and we are not even native English speakers, so we can definitely say it was a real challenge for us! This book is the result of many late nights and intensive research. But writing was not the only challenge: our family and closest friends know some of the real challenges we had during this journey, just to name a few: we both have our roles changed at Red Hat, one of the writers was relocated to a different city, and the other moved even farther away - to another country! So many new things in our personal and professional lives - not once or twice that we considered giving up, but, as the wise Master Yoda says: “Do or do not, there is no try.”

We persisted, and after several months, we finally can say we succeeded! Here it is, the finished book, we hope you enjoy it and find it useful on your journey! Although we would be quite grateful if it becomes a good-selling book, our major goal will have been accomplished and we will feel that the several late hours were worthwhile if it helped even just one person get better outcomes.

So, we encourage you to check out our book on Amazon. If you want to get in touch, add us to your social network, GitHub, and join Packt Community Platform!

Our social network: Giovanni Fontana (@giofontana) / Twitter | Rafael Pécora (@rvspecora) / Twitter Giovanni Fontana - OpenShift Specialist Solution Architect - Red Hat / LinkedIn | Rafael Pécora - Cloud Success Architect - Red Hat / LinkedIn

GitHub: giofontana (Giovanni Fontana) · GitHub

pecorawal (Rafael Pécora) · GitHub

Nothing is permanent except change!

Mon, 17 Oct 2022 21:05:55 +0000

My family and I made a significant change in our lives in May of this year: We relocated from my home country of sunny and warm Brazil to the lovely, historical, and cold — at least for me! — New England, more specifically Massachusetts. It is difficult to leave behind friends, family, and my country, but we have decided to take on this challenge.

During this transition period, we faced numerous challenges — I’m still wondering whether or not to write a separate post about it (what do you think?) — but difficulties ranging from financing and rental due to a lack of a credit score to unexpected emotional instability due to the distance from family and friends. I left home when I was 18 years old to attend university, and a few years later I moved to another city to work. I also had to travel quite frequently for work at some point in my life, and I assumed that because of these and other factors, it would not be as difficult, emotionally speaking, to deal with the many things that a change like this generates. But this time was different: seeing my son cry because he can’t play with his friends or stay close to our family hurts so much! I felt the weight of responsibility on my shoulders and wondered many times if I had made the right decision.

But as time passes, we gradually overcome the challenges and adapt to the new reality. I feel that we will always miss our home country, but the experiences we have had here in these few months have been priceless! We were able to experience the intensity of summer here, as well as the splendor of nature in the fall, with its incredible red, yellow, and colorful leaves!

We are now eagerly awaiting the arrival of winter with a mixture of excitement and anxiety; we are from a warm country, so perhaps this will be our most difficult challenge here?!?! I’m not sure, but that’s what we’re going to look into soon! :)

So, how about you? How do you deal with life changes? Comment and share!

Deploying a Private OpenShift Cluster on AWS using Transit Gateway

Tue, 30 Aug 2022 21:05:55 +0000

These are the steps we are going to follow:

VPCs creation;
Transit Gateway and route table configuration;
Deployment of a Red Hat OpenShift on AWS (ROSA) cluster using the OpenShift Cluster Manager on console.redhat.com;
Publishing DNS in the public VPC;
Deployment of a Jump server to access the private subnets.

VPCs Creation

We are going to create 3 VPCs:

Egress VPC: This VPC is the only public one, which has an Internet and NAT Gateway. All egress traffic flows through this VPC. Our jump server will also be in this VPC, as it is the only one with public internet access.
ROSA VPC: Private VPC in which we are going to install the Red Hat OpenShift on AWS (ROSA) cluster.
Other VPC: This VPC would represent other services that applications on OpenShift would be interacting with, such as Databases, APIs, etc.

The following table has the details of the VPCs we are going to create:

VPC Name	CIDR block	Subnets
egress-vpc	10.0.0.0/24	us-east-1a egress-subnet-public1-us-east-1a - 10.0.0.0/28 egress-subnet-private1-us-east-1a - 10.0.0.128/28
ocp-vpc	10.1.0.0/16	us-east-1a ocp-subnet-private1-us-east-1a - 10.1.128.0/20 us-east-1b ocp-subnet-private2-us-east-1b - 10.1.144.0/20 us-east-1c ocp-subnet-private3-us-east-1c - 10.1.160.0/20
integration-vpc	10.2.0.0/16	us-east-1a integration-subnet-private1-us-east-1a - 10.2.128.0/20 us-east-1b integration-subnet-private2-us-east-1b - 10.2.144.0/20 us-east-1c integration-subnet-private3-us-east-1c - 10.2.160.0/20

Follow the steps below to create the VPCs:

Access your AWS account and navigate to the VPC feature of the desired region.

Click on Create VPC button on the top right corner of the screen.

Fill out the form for the egress-vpc:
- Resources to create: VPC and more
- Name tag auto-generation: egress
- IPv4 CIDR block: 10.0.0.0/24 (*)
- IPv6 CIDR block: No IPv6 CIDR block
- Number of Availability Zones (AZs): 1
- Number of public subnets: 1
- Number of private subnets: 1
- Public subnet CIDR block in us-east-1a: 10.0.0.0/28 (*)
- Private subnet CIDR block in us-east-1a: 10.0.0.128/28 (*)
- NAT gateways ($): In 1 AZ
- VPC endpoints: S3 Gateway
- Enable DNS hostnames: enabled
- Enable DNS resolution: enabled

(*) You may customize the CIDR ranges as you need, just make sure there are no overlaps between the CIDRs of each VPCs and subnets.

Repeat the same thing to create the OpenShift VPC (ocp-vpc):
- Resources to create: VPC and more
- Name tag auto-generation: ocp
- IPv4 CIDR block: 10.1.0.0/16 (*)
- Number of Availability Zones (AZs): 3
- Number of public subnets: 0
- Number of private subnets: 3
- Private subnet CIDR block (1a, 1b, 1c): 10.1.128.0/20, 10.1.144.0/20, 10.1.160.0/20
- NAT gateways ($): None
- VPC endpoints: S3 Gateway
- Enable DNS hostnames/resolution: enabled
Finally, create the integration-vpc:
- Resources to create: VPC and more
- Name tag auto-generation: integration
- IPv4 CIDR block: 10.2.0.0/16 (*)
- Number of Availability Zones (AZs): 3
- Number of public subnets: 0
- Number of private subnets: 3
- Private subnet CIDR block (1a, 1b, 1c): 10.2.128.0/20, 10.2.144.0/20, 10.2.160.0/20
- NAT gateways ($): None

Transit Gateway and route table configuration

With VPCs created we can go ahead and create the Transit Gateway. Access the Transit Gateway menu and click on Create transit gateway. Give it a name and wait until the state is Available.

On the screen give it a name and click on Create transit gateway attachments.

Wait one minute or two until you see the transit gateway state as Available:

Now we need to create Transit gateway attachments for each VPC. Access the Transit gateway attachments menu and click on Create transit gateway attachment button:

Give the name egress-tga, select the transit gateway that has been just created, egress-vpc, leave all subnets enabled, and click on Create transit gateway attachment button.

Repeat the same step for ocp-vpc and integration-vpc. Wait some minutes until all TGA state is Available.

Now access Transit gateway route tables and click on the one that has been automatically created with the TGW.

Access the Routes tab and click on Create static route button.

Add the following route:
- CIDR: 0.0.0.0/0
- Attachment: egress-tga

We need to configure the egress private subnet route table to allow network packages to return back to the VPCs using the transit gateway. To do so, access one of the egress public subnets and click over the route table link:

Then click on the Routes tab and Edit routes button.

Add the following routes:
- 10.1.0.0/16 - Transit Gateway
- 10.2.0.0/16 - Transit Gateway

Finally, we need to add the following rule on all subnets of ocp and integration VPCs to enable these subnets to use the Transit Gateway. To do so, click on the subnet, access the Route table tab, and click over the Route table link:
- 0.0.0.0/0 - Transit Gateway

Click on Edit routes:

Add the rule:

This concludes the configuration required on Transit Gateway and route tables. See next how to install Red Hat OpenShift for AWS (ROSA) on this infrastructure.

Deployment of a Red Hat OpenShift on AWS (ROSA) cluster

First, access the Red Hat Hybrid Cloud console: https://console.redhat.com/openshift/create. If you don’t have an account use the link Register for a Red Hat account to create one. Click on Create cluster button next to the Red Hat OpenShift Service on AWS (ROSA).

The first step required is to link your AWS account to your Red Hat console. To do so you will need a workstation with the AWS CLI installed and configured beforehand. Look at the references at the end of this article if you need instructions on how to install and use the AWS CLI. On the first page, click on the Associated AWS account combo box and Associate AWS account button.

Follow the instructions on the screen to download the rosa cli:

Copy the rosa login command and run it from your workstation.

$ rosa login --token="eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2*********"
I: Logged in as '******' on 'https://api.openshift.com'

$ rosa create ocm-role --admin
I: Creating ocm role
? Role prefix: ManagedOpenShift
? Permissions boundary ARN (optional): 
? Role creation mode: auto
I: Creating role using 'arn:aws:iam::********:user/*****@******-admin'
? Create the 'ManagedOpenShift-OCM-Role-11009103' role? Yes
I: Created role 'ManagedOpenShift-OCM-Role-11009103' with ARN 'arn:aws:iam::839138491912:role/ManagedOpenShift-OCM-Role-11009103'
I: Linking OCM role
? OCM Role ARN: arn:aws:iam::*********:role/ManagedOpenShift-OCM-Role-11009103
? Link the 'arn:aws:iam::*********:role/ManagedOpenShift-OCM-Role-11009103' role with organization '**********'? Yes
I: Successfully linked role-arn 'arn:aws:iam::*********:role/ManagedOpenShift-OCM-Role-11009103' with organization account '**********'

$ rosa create user-role
I: Creating User role
? Role prefix: ManagedOpenShift
? Permissions boundary ARN (optional): 
? Role creation mode: auto
I: Creating ocm user role using 'arn:aws:iam::*******:user/******@*****-admin'
? Create the 'ManagedOpenShift-User-******-Role' role? Yes
I: Created role 'ManagedOpenShift-User-******-Role' with ARN 'arn:aws:iam::*******:role/ManagedOpenShift-User-******-Role'
I: Linking User role
? User Role ARN: arn:aws:iam::******:role/ManagedOpenShift-User-******-Role
? Link the 'arn:aws:iam::839138491912:role/ManagedOpenShift-User-******-Role' role with account '*****'? Yes
I: Successfully linked role ARN 'arn:aws:iam::******:role/ManagedOpenShift-User-******-Role' with account '*********'

After you run these commands successfully, you should now see your AWS account listed in the Associated AWS account combobox:

You will still see the following message, indicating that you need to create the account roles.

To do so, run the command as described in the message:

$ rosa create account-roles
I: Logged in as '*****' on 'https://api.openshift.com'
I: Validating AWS credentials...
I: AWS credentials are valid!
I: Validating AWS quota...
I: AWS quota ok. If cluster installation fails, validate actual AWS resource usage against https://docs.openshift.com/rosa/rosa_getting_started/rosa-required-aws-service-quotas.html
I: Verifying whether OpenShift command-line tool is available...
(...)
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-machine-api-aws-cloud-credentials'
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-cloud-credential-operator-cloud-crede'
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-image-registry-installer-cloud-creden'
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-ingress-operator-cloud-credentials'
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-cluster-csi-drivers-ebs-cloud-credent'
I: Created policy with ARN 'arn:aws:iam::****:policy/ManagedOpenShift-openshift-cloud-network-config-controller-cloud'
I: To create a cluster with these roles, run the following command:
rosa create cluster --sts

Now click on Refresh ARNs button and we should be good to go:

On the next page fill out the Cluster name and set the Availability to Multi-zone.

Change the machine pool size if you want or leave it as-is:

Change the Cluster privacy to Private:

Now copy the private subnet IDs from the ocp VCP we created before and paste here:

Set the Machine CIDR to the same range you used with the ocp VPC:

You don’t need to change the Cluster roles and policies.

Set the update strategy according to what you need:

Now review the information provided and start the cluster provisioning:

You will need to wait from 40 minutes to 1 hour to have your cluster available:

When the cluster is available you will be automatically redirected to a page with the main details of your cluster.

To access our cluster we need to add an identity provider and a cluster-admin user. Click in the Access control tab, then Identity providers and select the HTPasswd:

Now set the desired admin user and password:

Now go back to the Access control tab and click on Add user button at Cluster Roles and Access feature.

Add the user you just created as a cluster-admin.

To check the console URL, click on the Open console button. You will not be able to access the console, as expected, as this is a private cluster.

We are going to use a jump server to access the console.

Publishing DNS in the public VPC

Our cluster is private, so the DNS domain created by ROSA is. To be able to access the console and applications from the egress VPC, we should add this VPC to the DNS domain created by ROSA. To do so, access the AWS Route-53 of your AWS account, access the domain created by ROSA (it ends with openshiftapps.com), and click on the Edit hosted zone button.

Now add the egress VPC in this domain:

Deployment of a Jump server to access the private subnets

Now launch a new instance to be our jump server. You can provision any OS you prefer, such as Windows, Fedora, Red Hat, or Ubuntu that has a GUI with a supported browser version (Firefox, Chrome, or Edge). Make sure you select the egress VPC and the public subnet. Assign a public IP and use it to connect to that instance.

Connect to the instance using your preferred remote desktop tool, you should be able to access the OpenShift console from there and login using the user you defined before.

In this article we created from scratch the AWS VPCs infrastructure, connected them using a Transit Gateway and installed a Red Hat OpenShift on AWS (ROSA) in one of the private VPC. If you are interested in deploying an OpenShift cluster in a private AWS VPC, this is one of the ways to do that.

References:

Installing or updating the latest version of the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
AWS CLI Configuration: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html

Windows Containers on Red Hat OpenShift: Does That Make Sense?

Sun, 27 Jun 2021 21:05:55 +0000

Originally published at https://redhat.com/en/blog on June 28, 2021.

Why Windows on OpenShift?

Red Hat OpenShift workers with … Windows? Perhaps it may seem strange to you; it seemed a bit strange to me at first as well. However, after seeing some numbers, I realized that this is a natural growing movement for the Containers ecosystem, in which Red Hat is one of the most active collaborators. Let’s see some numbers to understand that better.

Although the growth rate of Red Hat Enterprise Linux is much greater than Windows, Windows still has a very strong presence among server operating systems in the data center, as you can see in the pie chart below, from research by IDC in 2018 (see here):

This data, combined with the digital transformation movement we have been experiencing in recent years, gives us a real notion of the huge amount of effort that the market probably will demand from transformation and modernization in the next few years. Research in 2019 with more than 1,000 respondents from all regions stated that almost half of them were still in the early phase of the digital transformation:

Source: 2020 Red Hat Global Customer Tech Outlook survey. Conducted by Red Hat via Qualtrics, August-September 2019. Q1. Currently, where is your company in its journey to digital transformation? n=873.

This same research pointed out that “Optimizing existing IT” was still a top priority at that time. Of course, this scenario may have changed a bit after the pandemic started in 2020, but that only complicates the IT situation because t it is hard to innovate without getting existing IT in order first. Many companies are struggling to optimize their existing legacy applications and infrastructure while, at the same time, the industry is pushing them to innovate more. If you want to see more of this report, access it here.

Source: 2020 Red Hat Global Customer Tech Outlook survey. Conducted by Red Hat via Qualtrics, August-September 2019. Q4. Over the next 12 months, what are your company’s top IT technology funding priorities? Please select up to 3 of the top areas your company is investing. n=674.

Here is when Windows on OpenShift makes real sense. A huge amount of existing IT workloads run on Windows Servers only. With this movement, Red Hat allows many customers to move existing Windows workloads into OpenShift with very low friction and optimize them with many benefits of the containerization, as part of an Enterprise Kubernetes cluster.

Have you considered the benefits of having standard Kubernetes/OpenShift features with your existing Windows workloads? Features that are standard with OpenShift for many years now are available for Windows Server workloads also. Now you may have the benefits of features like self-healing, secret and configuration management, scaling, resilience, decoupled applications (from infrastructure), service discovery, and load balancing by moving the Windows workloads as containers on OpenShift.

However, keep in mind that Windows on OpenShift is still evolving, and some features are not available or not under development yet. Here are a few of them:

Serverless
OpenShift Pipelines
Service Mesh

Now that we have already discussed why Red Hat is making this investment and the benefits you may have with that, let’s look briefly at how to move a Windows workload to OpenShift.

How to Move Windows Workloads to OpenShift

There is not only one strategy to move Windows workloads to OpenShift. Let’s discuss the main strategies to do that.

Rehost

Red Hat OpenShift has the ability to host VMs with the OpenShift Virtualization feature based on the KubeVirt upstream project in which you can collocate VMs with Containers in the same OpenShift cluster. This is the lowest friction (lift and shift) you have to migrate to OpenShift as you will not need to refactor the application to run as a container; however, you will not have many benefits that the containerization might bring, like a lighter footprint and decoupled applications.

Refactor

If the application is compatible with Windows Server 2019, you may choose to refactor it as a Windows container. As such, you will be able to have all the benefits of containerization on OpenShift, as we have covered in this article.

Rearchitect/Rebuild

With this strategy, the application will be rebuilt using .Net Core, which is Linux compatible, and you will be able to deploy it in any OpenShift cluster. You may also decide to rearchitect the application entirely using some modern cloud-native framework. In general, these applications are fragmented into micro-services that use API gateway, in-memory databases, serverless functions, and many other modern middleware layers. As such and in general, this strategy requires much more effort and is time-consuming, but it does make sense for “Mode 2” applications that require a rapid path (or IT “fast lane”) to transform business ideas into features and changes.

Red Hat OpenShift: The Complete Set of Workloads

In this article, we saw why it makes sense to bring more different kinds of workloads to OpenShift. Red Hat is making this a reality by bringing together the regular Red Hat CoreOS/RHEL worker nodes, Virtual Machines with OpenShift Virtualization, and now the Windows Containers running on MS Windows Servers 2019.

Ready to move your Windows workloads to Red Hat OpenShift? See here how to try it or talk to a Red Hatter by filling this form.

The start of the business on the multicloud journey

Sun, 13 Dec 2020 21:05:55 +0000

Originally published at https://tiinside.com.br/ on December 14, 2020.

Moving from one stage to another within IT is no simple step. It’s a journey, a veritable journey through the vast universe of technology, which today offers an infinite range of possibilities. Rising trends, such as hybrid cloud and multicloud, are just some of the leveraging tools for transformation and increased agility. Implementing these solutions and changing systems and behaviors to utilize them to their full potential, however, requires clarity, method, persistence, and, above all, alignment with the business.

The adoption of clouds, whether public or private, for example, is already a reality in most companies. However, not all organizations are clear about their goals with this implementation, have defined the method to follow, or have managed to connect the use of these solutions with business objectives. Others still fail to persist on the journey when they begin to face the first challenges. As with any journey, it’s possible to correct routes, and knowing the path is crucial in this process.

Setting a direction

Every first step in life, whether personal or professional, requires clarity, and paradigm shifts in IT are no exception. It’s important to begin by understanding the business context and answering basic questions, from defining goals and efforts to achieve them to analyzing available infrastructure and investments. In other words, it’s essential to understand exactly what role IT plays as a support system within the business, viewing it comprehensively and not just as a silo.

This step also involves understanding the expected results of cloud adoption. If the answer is to implement it because the entire market is using it, that’s a red flag. While beneficial to all companies, regardless of size, sector, or level of digital maturity, the cloud needs to be purposefully integrated into organizations’ IT to support day-to-day demands and improve processes.

The commitment of the organization and the team must also be considered. As with any journey, results won’t be achieved immediately and will require everyone’s commitment. The team, across its various hierarchical layers, must be open to change and ready to undergo a cultural and behavioral revolution to adapt to the transformations these technologies impose. Knowledge and motivation for implementing the cloud, or any other tool, cannot be restricted to senior management. It’s crucial that they be shared with the operational team, the driving force behind (or hindering) adoption.

Following routes

Once the directions have been defined and the “whys” understood, it’s time to move on to the next stage of the journey, which includes the methods and strategies that will be used to adopt the cloud in an efficient, organized, and standardized manner. While there’s no one-size-fits-all recipe, there are tips that apply to any organization. The first—and perhaps most fundamental of all—is to prioritize people. It’s people, not tools, who will truly determine the success (or failure) of cloud adoption.

Another important point involves experimentation, as there is a huge range of different processes and tools involved in cloud migration. Therefore, it’s very difficult to determine which ones will be most appropriate for a given company’s needs without some level of experimentation. Adopting a process that combines experimentation and rapid feedback helps determine with greater confidence and assertiveness what should be used in each journey.

Opening paths

The journey to the cloud requires adopting several new technologies. All public clouds offer a service platform that supports application development and delivery, known as PaaS (Platform as a Service). These tools are often tempting, as they effectively increase productivity and add a series of features to application development.

However, it’s important to always understand whether adopting a particular cloud provider’s tool will lock the company into that provider, known as service lock-in. To avoid this kind of surprise, it’s worth seeking out and adopting open-source solutions developed by open communities and portable to any cloud or infrastructure. Most open-source tools have procedures and automations already built in for installation on major clouds.

The cloud operating model also needs to be distinct. Ideally, the cloud shouldn’t be operated in the same way a company operates in an on-premises data center. Managing a cloud environment is, or should be, different from managing a traditional virtualization environment.

Crossed paths

Legacy infrastructure can (and should!) be integrated with cloud infrastructure, but without losing its autonomy. Companies often want to use the cloud as an overflow for their on-premises data centers. And that’s perfectly fine. However, care must be taken not to create couplings between on-premises and cloud services, creating dependencies between them, which could lead to scalability and availability issues.

The journey also requires the creation of clear criteria for migrating applications to the cloud. Several strategies are available, such as lift and shift, modernization, rebuild, or even switching to a ready-made SaaS service. Each application needs to be analyzed to find the appropriate migration path. A one-size-fits-all approach won’t be efficient.

Continuous cycle

The journey toward IT transformation has a starting point, an imagined destination (a clearly defined objective), but never an end point. It’s a continuous cycle, a journey that allows for changes of direction, changes of reference, adaptations, and improvements. The important thing is to enjoy the beauty of the lessons learned along the way, the exchange of experiences, the contexts, and, above all, the opportunity to meet and value people.

Technologies are important, and evolution is essential. But when starting a business in the multicloud era and ensuring the sustainable continuity of any organization’s projects, people are and always will be the most important asset to consider.

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

Sun, 25 Oct 2020 21:05:55 +0000

Originally published at https://redhat.com/en/blog on October 10, 2020.

Therefore, our Non-Production cluster will have the Dev and QA projects:

On other hand, in our Production and DR clusters we are going to have exactly the same workload, which is the application production version - in this step we will see how to use the PlacementRule to deploy an application on multiple clusters at once!

In the last article we explored the different RH ACM object types: Application, Channel, PlacementRule and Subscription. We also deployed all three stages on a single cluster, therefore the PlacementRule is equal for all of them. At this time we will change them to be able to deploy the application on different clusters. Let’s check them:

PlacementRule for Dev and QA:

---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: nonprod-cluster
  namespace: etherpad-acm-dev
spec:
  clusterConditions:
    - type: ManagedClusterConditionAvailable
      status: "True"
  clusterSelector:
    matchLabels:
      environment: nonprod

PlacementRule for Prod and DR:

---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: prd-and-dr-cluster
  namespace: etherpad-acm-prd
spec:
  clusterConditions:
    - type: ManagedClusterConditionAvailable
      status: "True"
  clusterSelector:
    matchExpressions:
    - key: environment
      operator: In
      values:
      - prod
      - dr

NOTE: My clusters have been tagged with the following labels during their import to RH ACM:

Non Production Cluster: environment=nonprod
Production Cluster: environment=prod
DR Cluster: environment=dr

Let’s run it? The yaml files are in this github repo: https://github.com/giofontana/rhacm-pipelines/tree/master/multipleclusters-multiple-envs

Note that when you create the subscription for prod environment, RH ACM starts the deployment of the etherpad application on two clusters at the same time (Prod and DR).

Therefore, we already addressed how to deploy an application on multiple namespaces in multiple clusters using RH ACM. Now we need to inject what we already have on a Tekton pipeline.

Our pipeline now will cover the following process:

[TEKTON - TASK creating-namespaces] Create the namespaces on the OpenShift Hub Cluster (where RH ACM is deployed);
[TEKTON - TASK etherpad-dev-deployment] Create Application, Channel, PlacementRule for NON-PRODUCTION and Subscription;
[RH ACM - DEV DEPLOYMENT] Deploy etherpad application on etherpad-dev namespace in NON-PRODUCTION cluster
[TEKTON - TASK etherpad-qa-deployment] Create Application, Channel, PlacementRule for NON-PRODUCTION and Subscription;
[RH ACM - QA DEPLOYMENT] Deploy etherpad application on etherpad-qa namespace in NON-PRODUCTION cluster
[TEKTON - TASK etherpad-prd-deployment] Create Application, Channel, PlacementRule for PRODUCTION + DR and Subscription;
[RH ACM - PROD + DR DEPLOYMENT] Deploy etherpad application on etherpad-prod namespace in PRODUCTION and DR cluster.

Here we can notice something that in my perspective is really valuable about using RH ACM as your deployment tool: You don’t need to change anything in your pipeline because now you need to deploy your application in more clusters - you just need to change the PlacementRule object of the environment you need to change. In the last article we deployed the Dev, QA and Prod application in a single cluster, we are now deploying this same application using three different clusters and what have we changed? Only the PlacementRule object! Very nice right?

To finish our article, let’s see our pipeline in action!

Conclusion

In this article, we improved the application deployment to simulate a more real world scenario using RH ACM and Tekton: Deploy of Dev and QA version of an application in a NON-PRODUCTION cluster and deploy of the Production version in two different clusters (PROD and DR) at the same time. We also saw that, by using RH ACM, we “decouple” the deployment logic from the pipeline and we were able to change the deployment targets by only changing the PlacementRule objects and no changes were needed on the Tekton side.

What did you think? Interesting, right? If you are interested in knowing more about OpenShift Pipelines and Red Hat Advanced Cluster Management for Kubernetes talk to a Red Hatter by filling this form or talking to your Red Hat representative. You can also see more about it on the Red Hat Videos YouTube channel.

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

Thu, 03 Sep 2020 21:05:55 +0000

Originally published at https://redhat.com/en/blog on September 4, 2020.

Use Cases:

In general, each organization defines its application life cycle strategy differently based on its own business requirements. Having that in mind, I will try to reproduce some general use cases, from simpler ones to more complex use cases.

One single cluster: multiple environments using different namespaces.
Two clusters: one for non-production workloads and another one for production.
Three or more clusters: one for non-production workloads and two or more clusters for production workloads (for example, Production and DR cluster).

Also, first we will simulate a simpler scenario that focuses only on the deployment of an image (essentially the CD stage) and finally a more complete pipeline including the building phase (entire CI/CD workflow).

In this article, I am going to show only the first use case. Use cases 2 and 3 will be covered in the next posts.

Preparation:

I will not cover the installation of RH ACM and OpenShift Pipelines. The deployment is very straightforward using Operators, and you can find the instructions in the official documentation:

[Installing OpenShift Pipelines

Pipelines

OpenShift Container Platform 4.5](https://docs.openshift.com/container-platform/4.5/pipelines/installing-pipelines.html)

Install Red Hat Advanced Cluster Management for Kubernetes 2.0

Both tools have been installed in a dedicated OpenShift cluster, which RH ACM names as “hub cluster”. Also, the managed-cluster has been imported in RH ACM with name “prd”:

GitOps in a Single OpenShift Cluster Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

In this use case, I will be using the etherpad, a sample application from rhacm-labs.

Resources:

Basically, we have three types of resources we will need to handle:

Application resources: YAML files that describe the k8s application, such as the application deployment, route, services, and so on.

RH ACM resources: Files that will create the objects that RH ACM uses to manage an application. The following objects will be used:

Channels (channel.apps.open-cluster-management.io): define the source repositories that a cluster can subscribe to with a subscription, and can be the following types: GitHub repositories, Helm release registries, object stores, and resource template (deployable) namespaces on the hub cluster. In our case, we are using a GitHub channel.
Subscriptions (subscription.apps.open-cluster-management.io): allow clusters to subscribe to a source repository (channel). Subscriptions can be applied locally to the hub or to managed-clusters.
Placement rules (placementrule.apps.open-cluster-management.io): define the target clusters where subscriptions deploy and maintain the Kubernetes resources. You can use placement rules to help you facilitate the multicluster deployment. Placement rules can be shared across subscriptions.
Applications (application.app.k8s.io): Used in Red Hat Advanced Cluster Management for Kubernetes for grouping Kubernetes resources that make up an application.

Tekton resources: Files that will create the tasks and pipeline to deploy the application using RH ACM in three different stages: development, QA, and production.

All source code used in this demo is in this git repository: https://github.com/giofontana/rhacm-pipelines/tree/master/onecluster-multiple-envs

Deploying an application using RH ACM

To begin our process, let’s look first on how to deploy the sample application using RH ACM. The first thing we will need to do is to create namespaces on the hub cluster that will manage the application:

---
apiVersion: v1
kind: Namespace
metadata:
  name: etherpad-acm-dev
---
apiVersion: v1
kind: Namespace
metadata:
  name: etherpad-acm-qa
---
apiVersion: v1
kind: Namespace
metadata:
  name: etherpad-acm-prd

We are creating a project for each stage of our application life cycle (DEV/QA/PRD). This is a sample; in a real-world scenario, you should probably use some annotations and labels in your namespaces also.

We also will have an application, channel, placement-rule, and subscription YAML file for each stage:

Example for dev:

---
apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
  name: etherpad-acm-dev
  namespace: etherpad-acm-dev
spec:
  componentKinds:
  - group: apps.open-cluster-management.io
    kind: Subscription
  descriptor: {}
  selector:
    matchExpressions:
    - key: app
      operator: In
      values:
      - etherpad-acm-dev
---
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
  name: etherpad-app-latest
  namespace: etherpad-acm-dev
spec:
  type: GitHub
  pathname: https://github.com/giofontana/rhacm-pipelines.git 
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: dev-cluster
  namespace: etherpad-acm-dev
spec:
  clusterConditions:
    - type: ManagedClusterConditionAvailable
      status: "True"
  clusterSelector:
    matchLabels:
      environment: prd
---
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: etherpad-acm-dev
  namespace: etherpad-acm-dev
  labels:
    app: etherpad-acm-dev
  annotations:
    apps.open-cluster-management.io/github-path: onecluster-multiple-envs/app-resources/etherpad/dev
spec:
  channel: etherpad-acm-dev/etherpad-app-latest
  placement:
    placementRef:
      kind: PlacementRule
      name: dev-cluster

Note: You will notice that the clusterSelector for all stages are equal. This is because for while, we are using one single managed-cluster. You can switch it according to as many clusters as you need.

This is the folder structure we will have in the end:

onecluster-multiple-envs/rhacm
├── namespaces.yaml
├── dev
│   ├── application.yaml
│   ├── channel.yaml
│   ├── placement-rule.yaml
│   └── subscription.yaml
├── prd
│   ├── application.yaml
│   ├── channel.yaml
│   ├── placement-rule.yaml
│   └── subscription.yaml
└── qa
    ├── application.yaml
    ├── channel.yaml
    ├── placement-rule.yaml
    └── subscription.yaml

Now, let’s test it. To deploy this structure, you just need to run each YAML file against the RH ACM hub cluster.

On the right side, you will see the managed cluster. Note that as soon as the subscription is created on the hub cluster, the deployment starts on the managed cluster: first DEV, then QA, and finally PRD.

Creating a Pipeline Using RH ACM Resources

Now that we already have our three projects for each stage (DEV/QA/PRD) being deployed and managed using RH ACM, we will create a pipeline to create the RH ACM resources automatically using Tekton.

To do that, we have the following tasks in Tekton:

create_namespaces: Create the namespaces needed in the hub cluster.
etherpad-dev-deployment: Deployment of DEV stage of etherpad.
etherpad-qa-deployment: Deployment of QA stage of etherpad.
etherpad-prd-deployment: Deployment of PRD stage of etherpad.

Also, we created a pipeline with the following structure:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: deploy-using-acm
spec:
  workspaces:
  - name: shared-workspace
  params:
  - name: deployment-name
    type: string
    description: name of the deployment to be patched
  - name: git-url
    type: string
    description: url of the git repo for the code of deployment
    default: "https://github.com/giofontana/rhacm-pipelines.git"
  - name: git-revision
    type: string
    description: revision to be used from repo of the code for deployment
    default: "master"
  tasks:
  - name: fetch-repository
    taskRef:
      name: git-clone
      kind: ClusterTask
    workspaces:
    - name: output
      workspace: shared-workspace
    params:
    - name: url
      value: $(params.git-url)
    - name: subdirectory
      value: ""
    - name: deleteExisting
      value: "true"
    - name: revision
      value: $(params.git-revision)
        - name: create-namespaces
    taskRef:
      kind: Task        
      name: create-namespaces
    workspaces:
    - name: source
      workspace: shared-workspace
    runAfter:
    - fetch-repository     
  - name: etherpad-dev-deployment
    taskRef:
      kind: Task        
      name: etherpad-dev-deployment
    workspaces:
    - name: source
      workspace: shared-workspace
    runAfter:
    - create-namespaces      
  - name: etherpad-qa-deployment
    taskRef:
      kind: Task        
      name: etherpad-qa-deployment
    workspaces:
    - name: source
      workspace: shared-workspace
    runAfter:
    - etherpad-dev-deployment  
  - name: etherpad-prd-deployment
    taskRef:
      kind: Task        
      name: etherpad-prd-deployment
    workspaces:
    - name: source
      workspace: shared-workspace
    runAfter:
    - etherpad-qa-deployment 

We can now deploy the Tekton objects:

> oc new-project etherpad-cicd
Now using project "etherpad-cicd" on server "https://api.acmhub.rhbr-labs.com:6443".
You can add applications to this project with the 'new-app' command. For example, try:
    oc new-app ruby~https://github.com/sclorg/ruby-ex.git
to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:
    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
>  oc get serviceaccount pipeline
NAME       SECRETS   AGE
pipeline   2         29s
> cd onecluster-multiple-envs/tekton
> oc create -f tasks/01_create_namespaces.yaml
task.tekton.dev/create-namespaces created
> oc create -f tasks/02_create_dev_app_using_acm.yaml
task.tekton.dev/etherpad-dev-deployment created
> oc apply -f tasks/03_create_qa_app_using_acm.yaml
task.tekton.dev/etherpad-qa-deployment created
> oc apply -f tasks/04_create_prd_app_using_acm.yaml
task.tekton.dev/etherpad-prd-deployment created
> oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:etherpad-cicd:pipeline
clusterrole.rbac.authorization.k8s.io/cluster-admin added: "system:serviceaccount:etherpad-cicd:pipeline"
> oc apply -f pipeline-acm.yaml
pipeline.tekton.dev/deploy-using-acm created
> oc apply -f prepare/tekton-source-pvc.yaml
persistentvolumeclaim/source-pvc created

And run our pipeline:

> oc apply -f pipelinerun.yaml
pipelinerun.tekton.dev/deploy-using-acm-run-1 created
> tkn pipelinerun logs deploy-using-acm-run-1 -f
[fetch-repository : clone] + CHECKOUT_DIR=/workspace/output/
[fetch-repository : clone] + '[[' true '==' true ]]
[fetch-repository : clone] + cleandir
[fetch-repository : clone] + '[[' -d /workspace/output/ ]]
[fetch-repository : clone] + rm -rf /workspace/output//onecluster-multiple-envs
[fetch-repository : clone] + rm -rf /workspace/output//.git
[fetch-repository : clone] + rm -rf '/workspace/output//..?*'
[fetch-repository : clone] + test -z 
[fetch-repository : clone] + test -z 
[fetch-repository : clone] + test -z 
[fetch-repository : clone] + /ko-app/git-init -url https://github.com/giofontana/rhacm-pipelines.git -revision master -refspec  -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[fetch-repository : clone] {"level":"info","ts":1597794549.2025642,"caller":"git/git.go:136","msg":"Successfully cloned https://github.com/giofontana/rhacm-pipelines.git @ f111d89c17e5647d3cd4dedca6968f20b73cfb0d (grafted, HEAD, origin/master) in path /workspace/output/"}
[fetch-repository : clone] {"level":"info","ts":1597794549.240191,"caller":"git/git.go:177","msg":"Successfully initialized and updated submodules in path /workspace/output/"}
[fetch-repository : clone] + cd /workspace/output/
[fetch-repository : clone] + git rev-parse HEAD
[fetch-repository : clone] + tr -d '\n'
[fetch-repository : clone] + RESULT_SHA=f111d89c17e5647d3cd4dedca6968f20b73cfb0d
[fetch-repository : clone] + EXIT_CODE=0
[fetch-repository : clone] + '[' 0 '!=' 0 ]
[fetch-repository : clone] + echo -n f111d89c17e5647d3cd4dedca6968f20b73cfb0d
[create-namespaces : creating-namespaces] **** Creating namespaces ****
[create-namespaces : creating-namespaces] namespace/etherpad-acm-dev unchanged
[create-namespaces : creating-namespaces] namespace/etherpad-acm-qa unchanged
[create-namespaces : creating-namespaces] namespace/etherpad-acm-prd unchanged
[etherpad-dev-deployment : etherpad-dev-deployment] **** DEV ENVIRONMENT: Creating application on RH ACM ****
[etherpad-dev-deployment : etherpad-dev-deployment] application.app.k8s.io/etherpad-acm-dev unchanged
[etherpad-dev-deployment : etherpad-dev-deployment] **** DEV ENVIRONMENT: Creating channel on RH ACM ****
[etherpad-dev-deployment : etherpad-dev-deployment] channel.apps.open-cluster-management.io/etherpad-app-latest unchanged
[etherpad-dev-deployment : etherpad-dev-deployment] **** DEV ENVIRONMENT: Creating placementrule on RH ACM ****
[etherpad-dev-deployment : etherpad-dev-deployment] placementrule.apps.open-cluster-management.io/dev-cluster unchanged
[etherpad-dev-deployment : etherpad-dev-deployment] **** DEV ENVIRONMENT: Creating subscription on RH ACM ****
[etherpad-dev-deployment : etherpad-dev-deployment] subscription.apps.open-cluster-management.io/etherpad-acm-dev unchanged
[etherpad-qa-deployment : etherpad-qa-deployment] **** QA ENVIRONMENT: Creating application on RH ACM ****
[etherpad-qa-deployment : etherpad-qa-deployment] application.app.k8s.io/etherpad-acm-qa unchanged
[etherpad-qa-deployment : etherpad-qa-deployment] **** QA ENVIRONMENT: Creating channel on RH ACM ****
[etherpad-qa-deployment : etherpad-qa-deployment] channel.apps.open-cluster-management.io/etherpad-app-latest unchanged
[etherpad-qa-deployment : etherpad-qa-deployment] **** QA ENVIRONMENT: Creating placementrule on RH ACM ****
[etherpad-qa-deployment : etherpad-qa-deployment] placementrule.apps.open-cluster-management.io/qa-cluster unchanged
[etherpad-qa-deployment : etherpad-qa-deployment] **** QA ENVIRONMENT: Creating subscription on RH ACM ****
[etherpad-qa-deployment : etherpad-qa-deployment] subscription.apps.open-cluster-management.io/etherpad-acm-qa unchanged
[etherpad-prd-deployment : etherpad-qa-deployment] **** PRD ENVIRONMENT: Creating application on RH ACM ****
[etherpad-prd-deployment : etherpad-qa-deployment] application.app.k8s.io/etherpad-acm-prd unchanged
[etherpad-prd-deployment : etherpad-qa-deployment] **** PRD ENVIRONMENT: Creating channel on RH ACM ****
[etherpad-prd-deployment : etherpad-qa-deployment] channel.apps.open-cluster-management.io/etherpad-app-latest unchanged
[etherpad-prd-deployment : etherpad-qa-deployment] **** PRD ENVIRONMENT: Creating placementrule on RH ACM ****
[etherpad-prd-deployment : etherpad-qa-deployment] placementrule.apps.open-cluster-management.io/prd-cluster unchanged
[etherpad-prd-deployment : etherpad-qa-deployment] **** PRD ENVIRONMENT: Creating subscription on RH ACM ****
[etherpad-prd-deployment : etherpad-qa-deployment] subscription.apps.open-cluster-management.io/etherpad-acm-prd unchanged

Note that the tasks have “unchanged” results because we already have our application deployed.

Here is how we see this pipeline in the web console:

Conclusion

In this article, we saw how to create three applications on RH ACM in your hub cluster to deploy a sample application for each stage of its life cycle: development, QA, and production. We also learned how to create a pipeline using Tekton to make those deployments automatically.

In the next article, we are going to explore how to have a similar case, but deploying the application in more than one cluster (use cases 2 and 3). In a final article, I am planning to share an entire CI/CD pipeline, including the build phase that we are not mentioning yet.

I encourage you to try out Red Hat Advanced Cluster Management for Kubernetes if you are looking for a solution for multicluster; see here how to try it. Combining it with Red Hat OpenShift Pipelines (Tekton) will give you a powerful native Kubernetes solution for GitOps. You can find more information about it in the OpenShift documentation page.

How to Restore All Masters in OpenShift 4.x

Wed, 27 May 2020 21:05:55 +0000

In this article I am going to show how to restore all masters servers, in case you need to restore your cluster of masters and etcds.

This procedure was created with my friend Rafael Pécora and it is based on OpenShift documentation.

NOTE: This procedure has been tested on OpenShift 4.3 and it does not apply to version 4.4 and above.

To do so, you will need to have at least one functional master/etcd server still working or at least you should have a recent backup to recover one of the masters and start the recovery process.

Therefore we should run the following tasks:

ETCD Backup: Backup one of the remaining funcional master nodes. In our case this will be the server master-0 (etcd-0).
Restore backup and etcd cluster: Restore backup selecting one etcd member only in the cluster (in our case will be the master-0/etcd-0)
Remove 2 masters nodes: Remove other 2 masters nodes (master-1/etcd-1 and master-2/etcd-2)
Recreate masters: Recreate both machines (master-1 and master-2)
Update DNS and LB records
Run etcd tokenize on the remaining funcional master node (master-0/etcd-0)
Run etcd-member-recover for new members (master-1/etcd-1 and master-2/etcd-2)

Now you already have your cluster of masters entirely functional and recovered. Only to exercise this process a little bit more, we will also replace the first master node (master-0/etcd-0). In order to do that, we will add the following tasks to our procedure:

Remove the etcd member etcd-0 (master-0/etcd-0)
Remove server master-0
Create a new server master-0 (master-0/etcd-0)
Update DNS and LB records
Run etcd tokenize on one of remaining funcional master node (now can be either master-1/etcd-1 or master-2/etcd-2)
Run etcd-member-recover for new the member (master-0/etcd-0)

ETCD Backup

The first step we should follow is to have a funcional backup of the cluster etcd. If you lost your entire cluster, you should restore it from an existing etcd backup that you should already have (in this case you already have your backup and you can skip this step). This process is documented in the OpenShift official process.footnote:[https://docs.openshift.com/container-platform/4.3/backup_and_restore/backing-up-etcd.html]

In our case, we elect the master-0 as the remaining funcional node.

Procedure:

# SSH to master-0
[root@ocp-bastion install]# ssh -i ~/.ssh/id_rsa core@10.0.140.240
Red Hat Enterprise Linux CoreOS 43.81.202005180953.0
  Part of OpenShift 4.3, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.3/architecture/architecture-rhcos.html

Last login: Fri May 22 10:32:14 2020 from 172.31.36.49
[core@ip-10-0-140-240 ~]$ sudo -i
[root@ip-10-0-140-240 ~]# /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup
Creating asset directory ./assets
683bef0adcff2cce1214a3cd00b78a06fedf6866f02d020cedc8ca89bcd902d3
etcdctl version: 3.3.17
API version: 3.3
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6 backing up to ./assets/backup/
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Trying to backup latest static pod resources..
{"level":"warn","ts":"2020-05-22T10:35:19.130Z","caller":"clientv3/retry_interceptor.go:116","msg":"retry stream intercept"}
Snapshot saved at ./assets/backup/snapshot_2020-05-22_103518.db
snapshot db and kube resources are successfully saved to ./assets/backup!

Copy backup to a bastion host

[root@ip-10-0-140-240 ~]# tar -cvzf etcd-20200522-bkp.tgz assets 
assets/
assets/backup/
assets/backup/etcd-ca-bundle.crt
assets/backup/etcd-client.crt
assets/backup/etcd-client.key
... omitted ...
[root@ip-10-0-140-240 ~]# mv etcd-20200522-bkp.tgz /home/core/
[root@ip-10-0-140-240 ~]# chown core:core /home/core/etcd-20200522-bkp.tgz 
[root@ip-10-0-140-240 ~]# exit
logout
[core@ip-10-0-140-240 ~]$ exit
logout
Connection to 10.0.140.240 closed.

[root@ocp-bastion install]# scp -i ~/.ssh/id_rsa core@10.0.140.240:/home/core/etcd-20200522-bkp.tgz .
etcd-20200522-bkp.tgz                                                                                                                   100%   23MB  83.0MB/s   00:00    

Restore backup and etcd cluster

Now we should restore the backup selecting etcd-0 as the only etcd member in the cluster. We should do this to restore the etcd cluster quorum and, as such, restoring access to our OpenShift api and console.

This procedure should be done in our remaining funcional node or the first member selected to be recovered. In our case this is the server master-0/etcd-0.

Procedure:

# Copy the backup to our remaining funcional node (or the first member selected to be recovered):
[root@ocp-bastion recover-master-2]# scp -i ~/.ssh/id_rsa etcd-20200522-bkp.tgz core@10.0.140.240:/home/core/
etcd-20200522-bkp.tgz                                                                                                                   100%   23MB 118.8MB/s   00:00    

# SSH to master-0 (our remaining funcional node)
[core@ip-10-0-140-240 ~]$ tar -xvzf  etcd-20200522-bkp.tgz
assets/
assets/backup/
assets/backup/etcd-ca-bundle.crt
assets/backup/etcd-client.crt
assets/backup/etcd-client.key
... omitted ...
[core@ip-10-0-140-240 ~]$ export INITIAL_CLUSTER="etcd-member-ip-10-0-140-240.ec2.internal=https://etcd-0.ocp.rhbr-labs.com:2380"
[core@ip-10-0-140-240 ~]$ sudo -E /usr/local/bin/etcd-snapshot-restore.sh /home/core/assets/backup $INITIAL_CLUSTER
e72fde15c357a44bd242d28853e8ad66fba2e0ce472c4602f55b9634c8dadf46
etcdctl version: 3.3.17
API version: 3.3
etcd-member.yaml found in ./assets/backup/
Stopping all static pods..
..stopping etcd-member.yaml
..stopping kube-scheduler-pod.yaml
..stopping kube-controller-manager-pod.yaml
..stopping kube-apiserver-pod.yaml
... omitted ...
Restoring etcd member etcd-member-ip-10-0-140-240.ec2.internal from snapshot..
2020-05-22 10:47:08.134229 I | pkg/netutil: resolving etcd-0.ocp.rhbr-labs.com:2380 to 10.0.140.240:2380
2020-05-22 10:47:09.775887 I | mvcc: restore compact to 25232
2020-05-22 10:47:09.814652 I | etcdserver/membership: added member 890a07c73df999b0 [https://etcd-0.ocp.rhbr-labs.com:2380] to cluster ea5a775da961a326
Starting static pods..
..starting etcd-member.yaml
..starting kube-scheduler-pod.yaml
..starting kube-controller-manager-pod.yaml
..starting kube-apiserver-pod.yaml
Starting kubelet..

Some minutes after above procedure you should have your cluster functional again. After that, check if we only have etcd-0 in our etcd cluster:

[root@ocp-bastion install]# oc -n openshift-etcd rsh $(oc get pods -n openshift-etcd | grep ip-10-0-140-240 | awk '{print $1}') # <1>
Defaulting container name to etcd-member.
Use 'oc describe pod/etcd-member-ip-10-0-140-240.ec2.internal -n openshift-etcd' to see all of the containers in this pod.
sh-4.2# export ETCDCTL_API=3 ETCDCTL_CACERT=/etc/ssl/etcd/ca.crt ETCDCTL_CERT=$(find /etc/ssl/ -name *peer*crt) ETCDCTL_KEY=$(find /etc/ssl/ -name *peer*key)
sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
|        ID        | STATUS  |                   NAME                   |              PEER ADDRS               |       CLIENT ADDRS        |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
| 890a07c73df999b0 | started | etcd-member-ip-10-0-140-240.ec2.internal | https://etcd-0.ocp.rhbr-labs.com:2380 | https://10.0.140.240:2379 |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
sh-4.2# 

<1> Change ip-10-0-140-240 for the ip of your master-0

Remove 2 masters nodes

(master-1/etcd-1 and master-2/etcd-2)

Now we should remove problematic master nodes to replace them further. Before deleting them let’s export their machine configurations to be used to recreate them.

Procedure:

# Export master-0
[root@ocp-bastion install]# oc get machine $(oc get machines -n openshift-machine-api -o wide | grep master-0 | awk '{ print $1 }') -n openshift-machine-api -o yaml > new-master-0.yaml
# Export master-1
[root@ocp-bastion install]# oc get machine $(oc get machines -n openshift-machine-api -o wide | grep master-1 | awk '{ print $1 }') -n openshift-machine-api -o yaml > new-master-1.yaml
# Export master-2
[root@ocp-bastion install]# oc get machine $(oc get machines -n openshift-machine-api -o wide | grep master-2 | awk '{ print $1 }') -n openshift-machine-api -o yaml > new-master-2.yaml

Now we can remove masters 1 and 2.

Procedure:

# Remove master-1
[root@ocp-bastion install]# oc delete machine -n openshift-machine-api $(oc get machines -n openshift-machine-api -o wide | grep master-1 | awk '{ print $1 }')
machine.machine.openshift.io "ocp-w2lhz-master-1" deleted
# Remove master-2
[root@ocp-bastion install]# oc delete machine -n openshift-machine-api $(oc get machines -n openshift-machine-api -o wide | grep master-2 | awk '{ print $1 }')
machine.machine.openshift.io "ocp-w2lhz-master-2" deleted

At this time, your cluster may become nonfunctional again:

[root@ocp-bastion install]# oc get machines -n openshift-machine-api 
The connection to the server api.ocp.rhbr-labs.com:6443 was refused - did you specify the right host or port?

If you get this problem you can remove the servers manually on AWS and restore the backup again in the master-0 (step above).

[core@ip-10-0-140-240 ~]$ export INITIAL_CLUSTER="etcd-member-ip-10-0-140-240.ec2.internal=https://etcd-0.ocp.rhbr-labs.com:2380"
[core@ip-10-0-140-240 ~]$ sudo -E /usr/local/bin/etcd-snapshot-restore.sh /home/core/assets/backup $INITIAL_CLUSTER                                                      
b3dd01ff7158ee8809b761d97be626d1e42760f2de505fcc4e950a951d213f1b
etcdctl version: 3.3.17
... omitted ...
2020-05-22 11:12:16.784231 I | etcdserver/membership: added member 890a07c73df999b0 [https://etcd-0.ocp.rhbr-labs.com:2380] to cluster ea5a775da961a326
Starting static pods..
..starting etcd-member.yaml
..starting kube-scheduler-pod.yaml
..starting kube-controller-manager-pod.yaml
..starting kube-apiserver-pod.yaml
Starting kubelet..

After some minutes that you recovered your backup you will see that the api will become functional again, but still showing master-1 and master-2 (because at the time of your backup both servers still exists). Wait up to 10 minutes and the OpenShift will move those masters to “Failed” state:

[root@ocp-bastion install]# oc get machines -n openshift-machine-api 
NAME                                PHASE     TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   94m
ocp-w2lhz-master-1                  Failed    m4.xlarge   us-east-1   us-east-1b   94m
ocp-w2lhz-master-2                  Failed    m4.xlarge   us-east-1   us-east-1c   94m
ocp-w2lhz-worker-us-east-1a-nq6g2   Running   m4.large    us-east-1   us-east-1a   89m
ocp-w2lhz-worker-us-east-1b-chzl6   Running   m4.large    us-east-1   us-east-1b   89m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running   m4.large    us-east-1   us-east-1c   89m

Now remove the machines again:

[root@ocp-bastion install]# oc delete machine -n openshift-machine-api $(oc get machines -n openshift-machine-api -o wide | grep master-1 | awk '{ print $1 }')
machine.machine.openshift.io "ocp-w2lhz-master-1" deleted
[root@ocp-bastion install]# oc delete machine -n openshift-machine-api $(oc get machines -n openshift-machine-api -o wide | grep master-2 | awk '{ print $1 }')
machine.machine.openshift.io "ocp-w2lhz-master-2" deleted
[root@ocp-bastion install]# oc get machines -n openshift-machine-api
NAME                                PHASE     TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   94m
ocp-w2lhz-worker-us-east-1a-nq6g2   Running   m4.large    us-east-1   us-east-1a   90m
ocp-w2lhz-worker-us-east-1b-chzl6   Running   m4.large    us-east-1   us-east-1b   90m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running   m4.large    us-east-1   us-east-1c   90m
[root@ocp-bastion install]# oc get nodes
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-129-242.ec2.internal   Ready    worker   86m   v1.16.2
ip-10-0-140-240.ec2.internal   Ready    master   94m   v1.16.2
ip-10-0-148-0.ec2.internal     Ready    worker   86m   v1.16.2
ip-10-0-161-99.ec2.internal    Ready    worker   86m   v1.16.2

Recreate masters

(master-1 and master-2)

Now we will use the machine config yaml files that you exported before.

Procedure:

[root@ocp-bastion install]# vi new-master-1.yaml
# Remove entire "status" section
# Remove the providerID field
[root@ocp-bastion install]# vi new-master-2.yaml
# Remove entire "status" section
# Remove the providerID field

# Now recreate the masters
[root@ocp-bastion install]# oc create -f new-master-1.yaml
machine.machine.openshift.io/ocp-w2lhz-master-1 created
[root@ocp-bastion install]# oc create -f new-master-2.yaml
machine.machine.openshift.io/ocp-w2lhz-master-2 created
[root@ocp-bastion install]# oc get machines -n openshift-machine-api
NAME                                PHASE          TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-0                  Running        m4.xlarge   us-east-1   us-east-1a   96m
ocp-w2lhz-master-1                  Provisioning   m4.xlarge   us-east-1   us-east-1a   13s
ocp-w2lhz-master-2                  Provisioning   m4.xlarge   us-east-1   us-east-1a   8s
ocp-w2lhz-worker-us-east-1a-nq6g2   Running        m4.large    us-east-1   us-east-1a   91m
ocp-w2lhz-worker-us-east-1b-chzl6   Running        m4.large    us-east-1   us-east-1b   91m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running        m4.large    us-east-1   us-east-1c   91m

Wait until the new masters (master-1 and master-2) to be in Running state:

[root@ocp-bastion recover-master-2]# oc get machines -n openshift-machine-api
NAME                                PHASE     TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   118m
ocp-w2lhz-master-1                  Running   m4.xlarge   us-east-1   us-east-1a   22m
ocp-w2lhz-master-2                  Running   m4.xlarge   us-east-1   us-east-1a   22m
ocp-w2lhz-worker-us-east-1a-nq6g2   Running   m4.large    us-east-1   us-east-1a   114m
ocp-w2lhz-worker-us-east-1b-chzl6   Running   m4.large    us-east-1   us-east-1b   114m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running   m4.large    us-east-1   us-east-1c   114m

Update DNS and LB records

Now we already have our 3 masters online again, however we still have only one etcd running on the cluster (on master-0). In the next steps we will deploy etcd on the new masters and add them to the etcd cluster.

The first step we need to update is the DNS and LoadBalancer in the AWS console.

Procedure:

To do so, get the IP address of the new servers in the AWS Console:

Now open the HostedZone for OCP in Route53:

Finally update the records etcd-1 and etcd-2 for this cluster:

LoadBalancer is automatically updated if you are running an IPI cluster. If you are using an UPI cluster, you should manually update your LB also.

Run etcd tokenize

Now we need to start the etcd-signer in the master-0.

Procedure:

[root@ocp-bastion recover-master-2]# ssh -i ~/.ssh/id_rsa core@10.0.140.240
... omitted ...
Last login: Fri May 22 11:08:03 2020 from 172.31.36.49
[core@ip-10-0-140-240 ~]$ sudo oc login https://api-int.ocp.rhbr-labs.com:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api-int.ocp.rhbr-labs.com:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.
[core@ip-10-0-140-240 ~]$ export KUBE_ETCD_SIGNER_SERVER=$(sudo oc adm release info --image-for kube-etcd-signer-server --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-140-240 ~]$ sudo -E /usr/local/bin/tokenize-signer.sh ip-10-0-140-240 <1>
[core@ip-10-0-140-240 ~]$ sudo oc create -f ./assets/manifests/kube-etcd-cert-signer.yaml
pod/etcd-signer created
[core@ip-10-0-140-240 ~]$ sudo oc get pods -n openshift-config
NAME          READY   STATUS    RESTARTS   AGE
etcd-signer   1/1     Running   0          15s

<1> Hostname of our functional master. In our case it is master-0

Run etcd-member-recover for new members

// TODO: SPEAK ABOUT THE BUG IN etcd-member-recover.sh SCRIPT

Procedure:

# Copy etcdctl bin due bug in etcd-member-recover script
[root@ocp-bastion recover-master-2]# tar -xzf etcd-20200522-bkp.tgz
[root@ocp-bastion recover-master-2]# scp -i ~/.ssh/id_rsa assets/bin/etcdctl core@ip-10-0-136-143:~/
etcdctl                                                                                                                                 100%   24MB  55.4MB/s   00:00    

# SSH to master-1
[root@ocp-bastion recover-master-2]# ssh -i ~/.ssh/id_rsa core@ip-10-0-136-143.ec2.internal
... omitted ...
[core@ip-10-0-136-143 ~]$ mkdir -p ~/assets/bin ~/assets/backup ~/assets/tmp
[core@ip-10-0-136-143 ~]$ mv etcdctl ~/assets/bin/

[core@ip-10-0-136-143 ~]$ sudo vi /usr/local/bin/etcd-member-recover.sh
### COMMENT LINE BELOW (dl_etcdctl)
function run {
  init
#  dl_etcdctl <-- THIS LINE
  backup_manifest
  DISCOVERY_DOMAIN=$(grep -oP '(?<=discovery-srv=).*[^"]' $ASSET_DIR/backup/etcd-member.yaml ) || true
  if [ -z "$DISCOVERY_DOMAIN" ]; then
    echo "Discovery domain can not be extracted from $ASSET_DIR/backup/etcd-member.yaml"
    exit 1
  fi

[core@ip-10-0-136-143 ~]$ sudo oc login https://api-int.ocp.rhbr-labs.com:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api-int.ocp.rhbr-labs.com:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.

[core@ip-10-0-136-143 ~]$ export SETUP_ETCD_ENVIRONMENT=$(sudo oc adm release info --image-for machine-config-operator --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-136-143 ~]$ export KUBE_CLIENT_AGENT=$(sudo oc adm release info --image-for kube-client-agent --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-136-143 ~]$ sudo -E /usr/local/bin/etcd-member-recover.sh 10.0.140.240 etcd-member-ip-10-0-136-143.ec2.internal
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Backing up /etc/etcd/etcd.conf to ./assets/backup/
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6 backing up to ./assets/backup/
Stopping etcd..
Waiting for etcd-member to stop
... omitted ...
Waiting for generate-certs to stop
Patching etcd-member manifest..
Updating etcd membership..
Removing etcd data_dir /var/lib/etcd..
Member 2517d85f40558b47 added to cluster ea5a775da961a326

ETCD_NAME="etcd-member-ip-10-0-136-143.ec2.internal"
ETCD_INITIAL_CLUSTER="etcd-member-ip-10-0-136-143.ec2.internal=https://etcd-1.ocp.rhbr-labs.com:2380,etcd-member-ip-10-0-140-240.ec2.internal=https://etcd-0.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-1.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
Starting etcd..

<1> IP 10.0.136.116 is the server functional master, where etcd-signer is running. In our case this is the master-0.

Now you have etcd functional in two masters (master-0 and master-1):

[core@ip-10-0-136-143 ~]$ sudo oc get pods -o wide -n openshift-etcd
NAME                                       READY   STATUS     RESTARTS   AGE    IP             NODE                           NOMINATED NODE   READINESS GATES
etcd-member-ip-10-0-136-143.ec2.internal   2/2     Running    0          28s    10.0.136.143   ip-10-0-136-143.ec2.internal   <none>           <none>
etcd-member-ip-10-0-137-235.ec2.internal   0/2     Init:1/2   5          38m    10.0.137.235   ip-10-0-137-235.ec2.internal   <none>           <none>
etcd-member-ip-10-0-140-240.ec2.internal   2/2     Running    0          136m   10.0.140.240   ip-10-0-140-240.ec2.internal   <none>           <none>
[core@ip-10-0-136-143 ~]$ sudo oc -n openshift-etcd rsh etcd-member-ip-10-0-136-143.ec2.internal
Defaulting container name to etcd-member.
Use 'oc describe pod/etcd-member-ip-10-0-136-143.ec2.internal -n openshift-etcd' to see all of the containers in this pod.
sh-4.2# export ETCDCTL_API=3 ETCDCTL_CACERT=/etc/ssl/etcd/ca.crt ETCDCTL_CERT=$(find /etc/ssl/ -name *peer*crt) ETCDCTL_KEY=$(find /etc/ssl/ -name *peer*key)
sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
|        ID        | STATUS  |                   NAME                   |              PEER ADDRS               |       CLIENT ADDRS        |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
| 2517d85f40558b47 | started | etcd-member-ip-10-0-136-143.ec2.internal | https://etcd-1.ocp.rhbr-labs.com:2380 | https://10.0.136.143:2379 |
| 890a07c73df999b0 | started | etcd-member-ip-10-0-140-240.ec2.internal | https://etcd-0.ocp.rhbr-labs.com:2380 | https://10.0.140.240:2379 |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+

Repeat procedure above to master-2:

[root@ocp-bastion recover-master-2]# scp -i ~/.ssh/id_rsa assets/bin/etcdctl core@ip-10-0-137-235.ec2.internal:~/
Warning: Permanently added 'ip-10-0-137-235.ec2.internal,10.0.137.235' (ECDSA) to the list of known hosts.
etcdctl                                                                                                                                 100%   24MB  51.6MB/s   00:00   
[core@ip-10-0-137-235 ~]$ mkdir -p ~/assets/bin ~/assets/backup ~/assets/tmp
[core@ip-10-0-137-235 ~]$ mv etcdctl ~/assets/bin/
[core@ip-10-0-137-235 ~]$ sudo vi /usr/local/bin/etcd-member-recover.sh
[core@ip-10-0-137-235 ~]$ sudo oc login https://api-int.ocp.rhbr-labs.com:6443
... omitted ...
[core@ip-10-0-137-235 ~]$ export SETUP_ETCD_ENVIRONMENT=$(sudo oc adm release info --image-for machine-config-operator --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-137-235 ~]$ export KUBE_CLIENT_AGENT=$(sudo oc adm release info --image-for kube-client-agent --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-137-235 ~]$ sudo -E /usr/local/bin/etcd-member-recover.sh 10.0.140.240 etcd-member-ip-10-0-137-235.ec2.internal
[core@ip-10-0-137-235 ~]$ sudo -E /usr/local/bin/etcd-member-recover.sh 10.0.140.240 etcd-member-ip-10-0-137-235.ec2.internal
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Backing up /etc/etcd/etcd.conf to ./assets/backup/
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6 backing up to ./assets/backup/
Stopping etcd..
... omitted ...
Member f03ec7613f440ab8 added to cluster ea5a775da961a326

ETCD_NAME="etcd-member-ip-10-0-137-235.ec2.internal"
ETCD_INITIAL_CLUSTER="etcd-member-ip-10-0-136-143.ec2.internal=https://etcd-1.ocp.rhbr-labs.com:2380,etcd-member-ip-10-0-140-240.ec2.internal=https://etcd-0.ocp.rhbr-labs.com:2380,etcd-member-ip-10-0-137-235.ec2.internal=https://etcd-2.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-2.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
Starting etcd..
[core@ip-10-0-137-235 ~]$ sudo oc get pods -o wide -n openshift-etcd
NAME                                       READY   STATUS    RESTARTS   AGE     IP             NODE                           NOMINATED NODE   READINESS GATES
etcd-member-ip-10-0-136-143.ec2.internal   2/2     Running   0          8m28s   10.0.136.143   ip-10-0-136-143.ec2.internal   <none>           <none>
etcd-member-ip-10-0-137-235.ec2.internal   2/2     Running   0          42s     10.0.137.235   ip-10-0-137-235.ec2.internal   <none>           <none>
etcd-member-ip-10-0-140-240.ec2.internal   2/2     Running   0          144m    10.0.140.240   ip-10-0-140-240.ec2.internal   <none>           <none>
[core@ip-10-0-137-235 ~]$ sudo oc -n openshift-etcd rsh etcd-member-ip-10-0-137-235.ec2.internal
Defaulting container name to etcd-member.
Use 'oc describe pod/etcd-member-ip-10-0-137-235.ec2.internal -n openshift-etcd' to see all of the containers in this pod.
sh-4.2# export ETCDCTL_API=3 ETCDCTL_CACERT=/etc/ssl/etcd/ca.crt ETCDCTL_CERT=$(find /etc/ssl/ -name *peer*crt) ETCDCTL_KEY=$(find /etc/ssl/ -name *peer*key)
sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
|        ID        | STATUS  |                   NAME                   |              PEER ADDRS               |       CLIENT ADDRS        |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
| 2517d85f40558b47 | started | etcd-member-ip-10-0-136-143.ec2.internal | https://etcd-1.ocp.rhbr-labs.com:2380 | https://10.0.136.143:2379 |
| 890a07c73df999b0 | started | etcd-member-ip-10-0-140-240.ec2.internal | https://etcd-0.ocp.rhbr-labs.com:2380 | https://10.0.140.240:2379 |
| f03ec7613f440ab8 | started | etcd-member-ip-10-0-137-235.ec2.internal | https://etcd-2.ocp.rhbr-labs.com:2380 | https://10.0.137.235:2379 |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+

Approve pending certificates:

[root@ocp-bastion recover-master-2]# oc get csr    
NAME                                          AGE    REQUESTOR                                                                   CONDITION                                
csr-6dns4                                     69m    system:node:ip-10-0-136-143.ec2.internal                                    Approved,Issued                         
csr-7g7b9                                     168m   system:node:ip-10-0-163-43.ec2.internal                                     Approved,Issued                         
csr-829pm                                     160m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued                         
csr-d5q44                                     160m   system:node:ip-10-0-148-0.ec2.internal                                      Approved,Issued                         
csr-gxdhg                                     168m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued          
... omitted ...
system:etcd-server:etcd-1.ocp.rhbr-labs.com   46m    system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending                                 
system:etcd-server:etcd-2.ocp.rhbr-labs.com   47m    system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending           

[root@ocp-bastion recover-master-2]# oc get csr -o name | xargs oc adm certificate approve    
certificatesigningrequest.certificates.k8s.io/system:etcd-server:etcd-1.ocp.rhbr-labs.com approved                                                                       
certificatesigningrequest.certificates.k8s.io/system:etcd-server:etcd-2.ocp.rhbr-labs.com approved                  

That’s all you need to recover a cluster of masters. The steps below is only to exercise this process a little bit more.

Replace master-0/etcd-0

This step is not required. This is only for testing purposes.

Remove the etcd member etcd-0

Procedure:

# SSH to master-1 or master-2
[root@ocp-bastion recover-master-2]# ssh -i ~/.ssh/id_rsa core@ip-10-0-137-235.ec2.internal
# Run script to remove etcd-0 from the cluster
[core@ip-10-0-137-235 ~]$ sudo -E /usr/local/bin/etcd-member-remove.sh etcd-member-ip-10-0-140-240.ec2.internal
22e5bfd54e0e25533d7fb5214561968d112d4ab5dc2cb2db993783e78a535e6e
etcdctl version: 3.3.17
API version: 3.3
Trying to backup etcd client certs..
etcd client certs already backed up and available ./assets/backup/
Member 890a07c73df999b0 removed from cluster ea5a775da961a326
etcd member etcd-member-ip-10-0-140-240.ec2.internal with 890a07c73df999b0 successfully removed..

Remove server master-0

Procedure:

[root@ocp-bastion recover-master-2]# oc delete machine -n openshift-machine-api $(oc get machines -n openshift-machine-api -o wide | grep master-0 | awk '{ print $1 }')
machine.machine.openshift.io "ocp-w2lhz-master-0" deleted

In the step below, sometimes you will see that OpenShift will be stuck in Deleting phase.

[root@ocp-bastion recover-master-2]# oc get machines -n openshift-machine-api
NAME                                PHASE      TYPE        REGION      ZONE         AGE                                                                                  
ocp-w2lhz-master-0                  Deleting   m4.xlarge   us-east-1   us-east-1a   157m                                                                                 
ocp-w2lhz-master-1                  Running    m4.xlarge   us-east-1   us-east-1a   61m                                                                                  
ocp-w2lhz-master-2                  Running    m4.xlarge   us-east-1   us-east-1a   61m                                                                                  
ocp-w2lhz-worker-us-east-1a-nq6g2   Running    m4.large    us-east-1   us-east-1a   153m                                                                                 
ocp-w2lhz-worker-us-east-1b-chzl6   Running    m4.large    us-east-1   us-east-1b   153m                                                                                 
ocp-w2lhz-worker-us-east-1c-6zdpt   Running    m4.large    us-east-1   us-east-1c   153m 

If you get this problem, terminate the server manually on AWS and it will disappear in OpenShift after some minutes.

[root@ocp-bastion recover-master-2]# oc get machine -n openshift-machine-api 
NAME                                PHASE     TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-1                  Running   m4.xlarge   us-east-1   us-east-1a   71m
ocp-w2lhz-master-2                  Running   m4.xlarge   us-east-1   us-east-1a   71m
ocp-w2lhz-worker-us-east-1a-nq6g2   Running   m4.large    us-east-1   us-east-1a   162m
ocp-w2lhz-worker-us-east-1b-chzl6   Running   m4.large    us-east-1   us-east-1b   162m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running   m4.large    us-east-1   us-east-1c   162m

Create a new server master-0

Procedure:

[root@ocp-bastion recover-master-2]# vi new-master-0.yaml
# Remove entire "status" section
# Remove the providerID field

[root@ocp-bastion recover-master-2]# oc create -f new-master-0.yaml

# Wait until you get it in Running state:
[root@ocp-bastion recover-master-2]# oc get machine -n openshift-machine-api 
NAME                                PHASE          TYPE        REGION      ZONE         AGE
ocp-w2lhz-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   9s
ocp-w2lhz-master-1                  Running        m4.xlarge   us-east-1   us-east-1a   77m
ocp-w2lhz-master-2                  Running        m4.xlarge   us-east-1   us-east-1a   77m
ocp-w2lhz-worker-us-east-1a-nq6g2   Running        m4.large    us-east-1   us-east-1a   169m
ocp-w2lhz-worker-us-east-1b-chzl6   Running        m4.large    us-east-1   us-east-1b   169m
ocp-w2lhz-worker-us-east-1c-6zdpt   Running        m4.large    us-east-1   us-east-1c   169m

Update DNS and LB records

Procedure:

Now open the HostedZone for OCP in Route53:

Finally update the records etcd-1 and etcd-2 for this cluster:

LoadBalancer is automatically updated if you are running an IPI cluster. If you are using an UPI cluster, you should manually update your LB also.

Run etcd tokenize

Now we need to start the etcd-signer in the master-1.

Procedure:

[root@ocp-bastion recover-master-2]# ssh -i ~/.ssh/id_rsa core@ip-10-0-136-143.ec2.internal
[core@ip-10-0-136-143 ~]$ sudo oc login https://api-int.ocp.rhbr-labs.com:6443
Authentication required for https://api-int.ocp.rhbr-labs.com:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
[core@ip-10-0-136-143 ~]$ export KUBE_ETCD_SIGNER_SERVER=$(sudo oc adm release info --image-for kube-etcd-signer-server --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-136-143 ~]$ mkdir -p ./assets/manifests
[core@ip-10-0-136-143 ~]$ sudo -E /usr/local/bin/tokenize-signer.sh ip-10-0-136-143
Populating template /usr/local/share/openshift-recovery/template/kube-etcd-cert-signer.yaml.template
Populating template ./assets/tmp/kube-etcd-cert-signer.yaml.stage1
Tokenized template now ready: ./assets/manifests/kube-etcd-cert-signer.yaml
[core@ip-10-0-136-143 ~]$ sudo oc create -f ./assets/manifests/kube-etcd-cert-signer.yaml
pod/etcd-signer created
[core@ip-10-0-136-143 ~]$ sudo oc get pods -n openshift-config
NAME          READY   STATUS    RESTARTS   AGE
etcd-signer   1/1     Running   0          16s

Run etcd-member-recover for new the member

// TODO: SPEAK ABOUT THE BUG IN etcd-member-recover.sh SCRIPT

Procedure:

# Copy etcdctl bin due bug in etcd-member-recover script
[root@ocp-bastion recover-master-2]# tar -xzf etcd-20200522-bkp.tgz
[root@ocp-bastion recover-master-2]# scp -i ~/.ssh/id_rsa assets/bin/etcdctl core@ip-10-0-130-22.ec2.internal:~/
etcdctl                                                                                                                                 100%   24MB  56.3MB/s   00:00    

# SSH to master-0
[root@ocp-bastion recover-master-2]# ssh -i ~/.ssh/id_rsa core@ip-10-0-130-22.ec2.internal   
... omitted ...
[core@ip-10-0-130-22 ~]$ mkdir -p ~/assets/bin ~/assets/backup ~/assets/tmp
[core@ip-10-0-130-22 ~]$ mv etcdctl ~/assets/bin/

[core@ip-10-0-130-22 ~]$ sudo vi /usr/local/bin/etcd-member-recover.sh
### COMMENT LINE BELOW (dl_etcdctl)
function run {
  init
#  dl_etcdctl <-- THIS LINE
  backup_manifest
  DISCOVERY_DOMAIN=$(grep -oP '(?<=discovery-srv=).*[^"]' $ASSET_DIR/backup/etcd-member.yaml ) || true
  if [ -z "$DISCOVERY_DOMAIN" ]; then
    echo "Discovery domain can not be extracted from $ASSET_DIR/backup/etcd-member.yaml"
    exit 1
  fi

[core@ip-10-0-130-22 ~]$ sudo oc login https://api-int.ocp.rhbr-labs.com:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api-int.ocp.rhbr-labs.com:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.

[core@ip-10-0-130-22 ~]$ export SETUP_ETCD_ENVIRONMENT=$(sudo oc adm release info --image-for machine-config-operator --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-130-22 ~]$ export KUBE_CLIENT_AGENT=$(sudo oc adm release info --image-for kube-client-agent --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-130-22 ~]$ sudo -E /usr/local/bin/etcd-member-recover.sh 10.0.136.143 etcd-member-ip-10-0-130-22.ec2.internal <1>
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Backing up /etc/etcd/etcd.conf to ./assets/backup/
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-6 backing up to ./assets/backup/
Stopping etcd..
... omitted ...
Waiting for generate-certs to stop
Patching etcd-member manifest..
Updating etcd membership..
Removing etcd data_dir /var/lib/etcd..
Member 2edf74688e8d0666 added to cluster ea5a775da961a326

ETCD_NAME="etcd-member-ip-10-0-130-22.ec2.internal"
ETCD_INITIAL_CLUSTER="etcd-member-ip-10-0-136-143.ec2.internal=https://etcd-1.ocp.rhbr-labs.com:2380,etcd-member-ip-10-0-130-22.ec2.internal=https://etcd-0.ocp.rhbr-labs.com:2380,etcd-member-ip-10-0-137-235.ec2.internal=https://etcd-2.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-0.ocp.rhbr-labs.com:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
Starting etcd..

<1> IP 10.0.136.143 is the server functional master, where etcd-signer is running. In our case this is the master-2.

Now you have etcd functional all 3 masters again:

[core@ip-10-0-130-22 ~]$ sudo oc get pods -o wide -n openshift-etcd
NAME                                       READY   STATUS    RESTARTS   AGE   IP             NODE                           NOMINATED NODE   READINESS GATES
etcd-member-ip-10-0-130-22.ec2.internal    2/2     Running   0          80s   10.0.130.22    ip-10-0-130-22.ec2.internal    <none>           <none>
etcd-member-ip-10-0-136-143.ec2.internal   2/2     Running   0          54m   10.0.136.143   ip-10-0-136-143.ec2.internal   <none>           <none>
etcd-member-ip-10-0-137-235.ec2.internal   2/2     Running   0          46m   10.0.137.235   ip-10-0-137-235.ec2.internal   <none>           <none>
[core@ip-10-0-130-22 ~]$ sudo oc -n openshift-etcd rsh etcd-member-ip-10-0-130-22.ec2.internal
Defaulting container name to etcd-member.
Use 'oc describe pod/etcd-member-ip-10-0-130-22.ec2.internal -n openshift-etcd' to see all of the containers in this pod.
sh-4.2# export ETCDCTL_API=3 ETCDCTL_CACERT=/etc/ssl/etcd/ca.crt ETCDCTL_CERT=$(find /etc/ssl/ -name *peer*crt) ETCDCTL_KEY=$(find /etc/ssl/ -name *peer*key)
sh-4.2# etcdctl member list -w table
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
|        ID        | STATUS  |                   NAME                   |              PEER ADDRS               |       CLIENT ADDRS        |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+
| 2517d85f40558b47 | started | etcd-member-ip-10-0-136-143.ec2.internal | https://etcd-1.ocp.rhbr-labs.com:2380 | https://10.0.136.143:2379 |
| 2edf74688e8d0666 | started |  etcd-member-ip-10-0-130-22.ec2.internal | https://etcd-0.ocp.rhbr-labs.com:2380 |  https://10.0.130.22:2379 |
| f03ec7613f440ab8 | started | etcd-member-ip-10-0-137-235.ec2.internal | https://etcd-2.ocp.rhbr-labs.com:2380 | https://10.0.137.235:2379 |
+------------------+---------+------------------------------------------+---------------------------------------+---------------------------+

Remove etcd-signer pod:

[root@ocp-bastion recover-master-2]# oc delete pod etcd-signer -n openshift-config
pod "etcd-signer" deleted

Validating cluster

Approve any pending certificates:

[root@ocp-bastion recover-master-2]# oc get csr | grep Pending
system:etcd-server:etcd-0.ocp.rhbr-labs.com   13m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
[root@ocp-bastion recover-master-2]# oc get csr -o name | xargs oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/system:etcd-server:etcd-0.ocp.rhbr-labs.com approved  

Check nodes status:

[root@ocp-bastion recover-master-2]# oc get nodes
NAME                           STATUS   ROLES    AGE     VERSION
ip-10-0-129-242.ec2.internal   Ready    worker   3h10m   v1.16.2
ip-10-0-130-22.ec2.internal    Ready    master   21m     v1.16.2
ip-10-0-136-143.ec2.internal   Ready    master   99m     v1.16.2
ip-10-0-137-235.ec2.internal   Ready    master   99m     v1.16.2
ip-10-0-148-0.ec2.internal     Ready    worker   3h10m   v1.16.2
ip-10-0-161-99.ec2.internal    Ready    worker   3h10m   v1.16.2
[root@ocp-bastion recover-master-2]# oc get machine -n openshift-machine-api -o wide
NAME                                PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
ocp-w2lhz-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   25m     ip-10-0-130-22.ec2.internal    aws:///us-east-1a/i-02040b6466f85e292   running
ocp-w2lhz-master-1                  Running   m4.xlarge   us-east-1   us-east-1a   102m    ip-10-0-136-143.ec2.internal   aws:///us-east-1a/i-06626c50318669add   running
ocp-w2lhz-master-2                  Running   m4.xlarge   us-east-1   us-east-1a   102m    ip-10-0-137-235.ec2.internal   aws:///us-east-1a/i-07037feaed2c7af5c   running
ocp-w2lhz-worker-us-east-1a-nq6g2   Running   m4.large    us-east-1   us-east-1a   3h14m   ip-10-0-129-242.ec2.internal   aws:///us-east-1a/i-01730585eb2f8c877   running
ocp-w2lhz-worker-us-east-1b-chzl6   Running   m4.large    us-east-1   us-east-1b   3h14m   ip-10-0-148-0.ec2.internal     aws:///us-east-1b/i-0c7998b00f3be1800   running
ocp-w2lhz-worker-us-east-1c-6zdpt   Running   m4.large    us-east-1   us-east-1c   3h14m   ip-10-0-161-99.ec2.internal    aws:///us-east-1c/i-0b771047194746d16   running

Check the ClusterOperator status:

[root@ocp-bastion recover-master-2]# oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.3.22    True        False         False      179m
cloud-credential                           4.3.22    True        False         False      3h15m
cluster-autoscaler                         4.3.22    True        False         False      3h7m
console                                    4.3.22    True        False         False      3h2m
dns                                        4.3.22    True        False         False      3h11m
image-registry                             4.3.22    True        False         False      3h5m
ingress                                    4.3.22    True        False         False      3h5m
insights                                   4.3.22    True        False         False      3h7m
kube-apiserver                             4.3.22    True        False         False      3h9m
kube-controller-manager                    4.3.22    True        False         False      3h9m
kube-scheduler                             4.3.22    True        False         False      3h9m
machine-api                                4.3.22    True        False         False      3h11m
machine-config                             4.3.22    True        False         False      3h10m
marketplace                                4.3.22    True        False         False      37m
monitoring                                 4.3.22    True        False         False      102m
network                                    4.3.22    True        False         False      3h11m
node-tuning                                4.3.22    True        False         False      38m
openshift-apiserver                        4.3.22    True        False         False      31m
openshift-controller-manager               4.3.22    True        False         False      3h10m
openshift-samples                          4.3.22    True        False         False      3h6m
operator-lifecycle-manager                 4.3.22    True        False         False      3h8m
operator-lifecycle-manager-catalog         4.3.22    True        False         False      3h8m
operator-lifecycle-manager-packageserver   4.3.22    True        False         False      38m
service-ca                                 4.3.22    True        False         False      3h11m
service-catalog-apiserver                  4.3.22    True        False         False      3h8m
service-catalog-controller-manager         4.3.22    True        False         False      3h8m
storage                                    4.3.22    True        False         False      3h7m

Check etcd status:

[root@ocp-bastion recover-master-2]# oc get pods -n openshift-etcd
NAME                                       READY   STATUS    RESTARTS   AGE
etcd-member-ip-10-0-130-22.ec2.internal    2/2     Running   0          6m32s
etcd-member-ip-10-0-136-143.ec2.internal   2/2     Running   0          59m
etcd-member-ip-10-0-137-235.ec2.internal   2/2     Running   0          51m

Check API pods:

[root@ocp-bastion recover-master-2]# oc get pods -n openshift-apiserver
NAME              READY   STATUS    RESTARTS   AGE
apiserver-4wq9s   1/1     Running   0          21m
apiserver-ljz4t   1/1     Running   0          98m
apiserver-zrntw   1/1     Running   0          98m
[root@ocp-bastion recover-master-2]# oc get pods -n openshift-kube-apiserver
NAME                                             READY   STATUS      RESTARTS   AGE
installer-6-ip-10-0-130-22.ec2.internal          0/1     Completed   0          21m
installer-6-ip-10-0-136-143.ec2.internal         0/1     Completed   0          96m
installer-6-ip-10-0-137-235.ec2.internal         0/1     Completed   0          98m
kube-apiserver-ip-10-0-130-22.ec2.internal       3/3     Running     0          20m
kube-apiserver-ip-10-0-136-143.ec2.internal      3/3     Running     1          96m
kube-apiserver-ip-10-0-137-235.ec2.internal      3/3     Running     1          97m
revision-pruner-6-ip-10-0-130-22.ec2.internal    0/1     Completed   0          19m
revision-pruner-6-ip-10-0-136-143.ec2.internal   0/1     Completed   0          96m
revision-pruner-6-ip-10-0-137-235.ec2.internal   0/1     Completed   0          96m

CloudForms Database High Availability Explained

Thu, 22 Feb 2018 21:05:55 +0000

The following diagram describes our setup.

In this architecture, server A is the primary VMDB, server B is the standby, and C1 to C4 are workers connected to server A database. The CloudForms HA mechanism is based on two elements: repmgr and evm failover monitor.

The repmgr tool (see https://repmgr.org/) runs on both VMDB appliances to keep the databases synchronized.

The evm-failover-monitor tool runs on all workers and identifies when the primary VMDB is down to point workers to the standby VMDB.

Let’s simulate a failure on our primary database.

Repmgr tool in server B identifies that server A is down and promotes itself as primary VMDB.

The evm-failover-monitor services, running on worker appliances C1 to C4, notice that server A is down and that standby server B was promoted. evm-failover-monitor reconfigures each worker’s database setup by modifying their configuration file database.yml to point to the new primary server B.

Server B is now our primary database. Server A needs to be manually reintroduced to the cluster as the standby vmdb.

Additional details on HA deployment procedures and configuration can be found on the Red Hat CloudForms High Availability Guide.

Conclusion

The Red Hat CloudForms database high availability feature explained in this article is active/passive. The solution uses simple tools, is easy to configure, and is cost effective (as opposed to solutions using HAProxy, VIP, keepalived, etc).

Being active/passive means that we might observe few seconds of unavailability (*) while repmgr and evm-failover-monitor tools detect the primary vmdb failure and promote the standby vmdb as primary. This minor downtime is acceptable in most cases.

(*) The frequency with which CloudForms checks for database failover can be configured in the vmdb/config/ha_admin.yml configuration file.

ACM

Posts by Category : ACM

25 Oct 2020 GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

03 Sep 2020 GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

CloudForms

CloudForms Database High Availability Explained

OpenShift

Cert expired again! How I quickly renewed my OpenShift cluster's certificate and automated renewals with Claude Code!

Automate VM golden image builds for OpenShift with Packer

Effective observability with Red Hat build of OpenTelemetry | Red Hat Developer

New Book Announcement - OpenShift Multi-Cluster Management Handbook

Deploying a Private OpenShift Cluster on AWS using Transit Gateway

Windows Containers on Red Hat OpenShift: Does That Make Sense?

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

How to Restore All Masters in OpenShift 4.x

ACM

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

Tekton

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

GitOps

Cert expired again! How I quickly renewed my OpenShift cluster's certificate and automated renewals with Claude Code!

GitOps Using Red Hat OpenShift Pipelines (Tekton) and Red Hat Advanced Cluster Management

Cloud

The start of the business on the multicloud journey

AWS

Deploying a Private OpenShift Cluster on AWS using Transit Gateway

My Life

Nothing is permanent except change!

Artificial Inteligence

The Cost of Generative AI

Applied Data Science Program: Levaraging AI for Effective Decision Making

Observability

Effective observability with Red Hat build of OpenTelemetry | Red Hat Developer

OpenTelemetry

Effective observability with Red Hat build of OpenTelemetry | Red Hat Developer

Kubevirt

Automate VM golden image builds for OpenShift with Packer

Packer

Automate VM golden image builds for OpenShift with Packer

Generative AI

The Cost of Generative AI

DevOps

Cert expired again! How I quickly renewed my OpenShift cluster's certificate and automated renewals with Claude Code!

AI

Cert expired again! How I quickly renewed my OpenShift cluster's certificate and automated renewals with Claude Code!

Artificial Inteligence

Posts by Category : Artificial Inteligence

02 Feb 2026 The Cost of Generative AI

29 Jul 2024 Applied Data Science Program: Levaraging AI for Effective Decision Making

AWS

Posts by Category : AWS

30 Aug 2022 Deploying a Private OpenShift Cluster on AWS using Transit Gateway

Cloud

Posts by Category : Cloud

13 Dec 2020 The start of the business on the multicloud journey

CloudForms

Posts by Category : CloudForms

22 Feb 2018 CloudForms Database High Availability Explained

Generative AI

Posts by Category : {{ page.title }}

{% assign posts = site.categories[page.title] %}

{% for post in posts %}

{% endfor %}

GitOps

Posts by Category : {{ page.title }}

{% assign posts = site.categories[page.title] %}

{% for post in posts %}

{% endfor %}

Guides

Posts by Category : {{ page.title }}

{% assign posts = site.categories[page.title] %}

{% for post in posts %}

{% assign posts = site.categories[page.title] %}

{% for post in posts %}

{% endfor %}

Our Sponsors

Thanks to all the amazing contributors and our Backers for the support. - [Dirish Mohan](https://dirishmohan.com)

Styleguide

### devlopr - Styleguide

Lets try the different text styles Bold , Strong , Emphasis , Italic

Now, lets try different heading styles :

Hello in h1 !

Hello in h2 !

Hello in h3 !

Hello in h4 !

Hello in h5 !

Hello in h6 !

Unordered List

List Item 1
List Item 2
List Item 3
List Item 4
List Item 5

Ordered List

List Item 1
List Item 2
List Item 3
List Item 4
List Item 5

This is a Block Quote, It can Expand Multiple Lines

You can use the mark tag to highlight text.

~~This line of text is meant to be deleted text~~

This line of text will render as underlined

This line of text is meant to be treated as fine print.

This line rendered as bold text.

This line rendered as italicized text.

attr

HTML

#	Heading	Heading	Heading	Heading	Heading	Heading	Heading	Heading	Heading
1	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell
2	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell
3	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell

Instagram Embed

View this post on Instagram

A post shared by Sujay (@sujaykundu777)

Twitter Embed

I just published “Deploying a blog using Jekyll and Github Pages with SSL certificate for Free” https://t.co/B3T3IQVU93
— Sujay Kundu (@SujayKundu777) June 29, 2018

YouTube Responsive Embed

Vimeo Responsive Embed

TED Responsive Embed

Twitch Responsive Embed

SoundCloud Embed

CodePen Embed

Syntax Highlighting

'use strict';
var markdown = require('markdown').markdown;
function Editor(input, preview) {
  this.update = function() {
    preview.innerHTML = markdown.toHTML(input.value);
  };
  input.editor = this;
  this.update();
}

You can add inline code just like this, E.g. .code { color: #fff; }

pre {
  background-color: #f4f4f4;
  max-width: 100%;
  overflow: auto;
}

GitHub gist Embed

Input Style

Tekton

Posts by Category : {{ page.title }}

{% assign posts = site.categories[page.title] %}

{% for post in posts %}

{% endfor %}

Security Policy

# Security Policy ## Supported Versions Use this section to tell people about which versions of your project are currently being supported with security updates. | Version | Supported | | ------- | ------------------ | | 5.1.x | :white_check_mark: | | 5.0.x | :x: | | 4.0.x | :white_check_mark: | | < 4.0 | :x: | ## Reporting a Vulnerability Use this section to tell people how to report a vulnerability. Tell them where to go, how often they can expect to get an update on a reported vulnerability, what to expect if the vulnerability is accepted or declined, etc.

Change Log

# Change Log All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/). **Note that references to the Font-Awesome-Pro repository refer to a GitHub repository that is by invitation only. You will get a 404 - Not Found if you do not have access** --- ## [5.1.0](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.1.0) - 2018-06-20 **Minor version upgrade notice: there are some backward-incompatible changes to this release. See the [UPGRADING.md guide](https://github.com/FortAwesome/Font-Awesome/blob/master/UPGRADING.md) for more information.** ### Added * New Emoji, Design, and Travel category pack * Another group of requested and commissioned icons * Version 4 shim for Web Fonts with CSS * New simplified download and NPM packages * @fortawesome/fontawesome-free and @fortawesome/fontawesome-pro NPM packages that match what's available in the CDN and .ZIP files * Brand icons rev, nimblr, megaport, mailchimp, hornbill, wix, weebly, themeco, squarespace, aws, shopware * API method toHtml() for converting abstract objects to HTML * API method counter() to generate Layers Counters * API method watch() to configure MutationObserver and watch DOM for icon changes and additions ### Changed * Relocating sponsor data to a separate sponsors.yml * Updated teamspeak brand icon * No more default exports in the CommonJS/ES packages (anything installed from NPM) * Greatly improved performance and rendering of CSS pseudo-elements with SVG and JavaScript * Configuration of SVG with JavaScript can now be done with attributes on the script tag * SVG with JavaScript pseudo-elements now match syntax (font-family, font-weight) of Web Fonts with CSS ### Fixed * Tree shaking of all NPM packages by default * Alignment of the book-open and dice-six icon * Correcting creative-commons * Incorrect license on the fontawesome-common-types package * Improve ligatures that share a base name with another ligature * Correcting solid style of the digital-tachograph icon * Prevent duplicating classes in some scenarios with SVG with JavaScript * Duplicate insertion of CSS when insertCss() method was called * Missing TypeScript definitions for the free-brands-svg-icons package --- ## [5.0.13](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.13) - 2018-05-10 ### Added * 68 icons to Free and 165 to Pro of the most requested icons in Font Awesome --- ## [5.0.12](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.12) - 2018-05-03 ### Added * A long time ago in a galaxy far, far away some icons were added ### Fixed * Renamed the r brand to r-project to prevent ligature collision with the "r" glyph --- ## [5.0.11](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.11) - 2018-05-01 ### Added * 16 new user icons * Full set of Creative Commons symbols * Regular style comment-dots used for v4 comment-alt in shim * Top 6 brand icons: r, ebay, mastodon, researchgate, keybase, teamspeak ### Changed * Revised slider icons FortAwesome/Font-Awesome#11872 * Make desktop typeface easier to find in apps that support ligature previews ### Fixed * Remove errant XML entity from the lastfm-square icon FortAwesome/Font-Awesome#12847 * Correcting paths in cloud icons FortAwesome/Font-Awesome-Pro#920 --- ## [5.0.10](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.10) - 2018-04-10 ### Added * New java brand icon FortAwesome/Font-Awesome#386 ### Changed * Updating depth of dna icon * Updating pied-piper, adding pied-piper-hat ### Fixed * Correcting path errors on readme icon FortAwesome/Font-Awesome#12754 * Light style of lamp icon FortAwesome/Font-Awesome#12725 --- ## [5.0.9](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.9) - 2018-03-27 ### Added * New Chat icon pack and category * New Charity icon pack and category * New Moving icon pack and category * New icons hands and hand-holding ### Changed * Updated flipboard, readme, and houzz brand icon * Making all solid icons in the medical icon pack free * Updated hand-holding-box and hand-receiving in the Light style ### Fixed * Missing box-sizing CSS property for fa-layers-counter --- ## [5.0.8](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.8) - 2018-03-01 ### Fixed * OTF font files missing ligatures for Pro styles FortAwesome/Font-Awesome#12486 FortAwesome/Font-Awesome-Pro#1034 --- ## [5.0.7](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.7) - 2018-02-26 ### Added * New Logistics category * New Medical category * Individual SVG files available from the Font Awesome CDN * Additional search terms ### Changed * Apple brand icon update FortAwesome/Font-Awesome#12337 * Disable mutation observers with fontawesome.noAuto() is called * License information now references https URL scheme ### Fixed * Missing TypeScript names FortAwesome/react-fontawesome#83 * Adding categories metadata FortAwesome/Font-Awesome#12034 * TypeScript improvement for fontawesome.layer() * Correcting a melting, wobbling, weird-looking whistle --- ## [5.0.6](https://github.com/FortAwesome/Font-Awesome/releases/tag/5.0.6) - 2018-01-25 ### Fixed * @fortawesome/fontawesome-pro-light missing submodules --- ## [5.0.5](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.5) - 2018-01-25 ### Added * New Sports category * New Chess category * Added brand icons for flipboard, php, quinscape, and hips ### Fixed * Sass and Less mixin fa-icon() now uses ems instead of percentage * Corrected misspelling of "Alternate" in category labels * Improved TypeScript definitions for @fortawesome/fontawesome * Server-side rendering was failing due to DOM-specific object access * SVG attributes "data-fa-processed" renamed to "data-fa-i2svg", only applies if rendered with i2svg() method --- ## [5.0.4](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.4) - 2018-01-10 ### Changed * Updating all NPM package READMEs ### Fixed * Improving TypeScript exports and fixing some incorrect definitions * TypeScript error when importing entire style Fort-Awesome/Font-Awesome#12072 * Pseudo-elements erasing text contents in parent container Fort-Awesome/Font-Awesome-Pro#11995 * fa-layers-text misalignment when using Bootstrap Fort-Awesome/Font-Awesome#11871 --- ## [5.0.3](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.3) - 2018-01-08 ### Added * Adding elementor, youtube-square brand icons * Adding window-minimize to the Free subset * TypeScript support for all NPM packages ### Fixed * Corrected uneven spacing in university, address-book, address-card, id-badge, id-card, mouse-pointer, phone-volume, portrait, user-alt, user-circle, user-md, user-plus, user-times, user , users * Corrected uneven spacing in brand icons behance-square, dashcube, discourse, ember, erlang, fort-awesome, js-square, laravel, mix, patreon, palfed, phoenix-framework, node-js, skyatlas, stack-exchange, stripe, viber, weixin, yahoo , yoast --- ## [5.0.2](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.2) - 2017-12-19 ### Added * Adding amazon-pay, cc-amazon-pay, korvue, ethereum brand icons * Adding stopwatch to Free version ### Changed * Ligatures now support capital case, all caps, and title case ### Fixed * NPM packages now behave the same way as CDN and browser-specific packages FortAwesome/Font-Awesome-Pro#727 FortAwesome/Font-Awesome-Pro#896 FortAwesome/Font-Awesome-Pro#891 * Icon doesn't change when pseudo-element content changes FortAwesome/Font-Awesome-Pro#839 * Invalid XML in sprites FortAwesome/Font-Awesome-Pro#927 * Incorrect version in Sass and Less variable files --- ## [5.0.1](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.1) - 2017-12-08 ### Added * Adding font-awesome-flag, lock-open, redo-alt, sync-alt, undo-alt to the Free version * New NPM packages `fontawesome-free-webfonts` and `fontawesome-pro-webfonts` * Adding old icon names to search terms for renamed icons * Extensive metadata added to the `advanced-options` directory * Adding stripe-s brand icon * Adding typo3 brand icon ### Changed * Updated dropbox brand icon to match new branding guidelines * Updated firefox brand icon * Updated strava brand icon * OTF font file now include a space character ### Fixed * OTF font file now supports different styles in Windows * OTF font file "j" character now has correct space on the right * Modifying the `class` attribute on an existing `` will now precede other link and style definitions * BREAKING `fontawesome.text` and `fontawesome.icon` now use `styles` param instead of `style` * Updated sizing for twitter, discord, youtube * Class fa-li now respects line-height and has new recommended markup (see included docs) ### Fixed * Duplicate `style` tags being added in the head FortAwesome/Font-Awesome-Pro#858 * Error with icon composition/masking that caused a confusing error message * An error when using pseudo elements and the element is empty (Array.reduce error) * Icons not being replaced with SVG if the text content is not empty --- ## [5.0.0-rc4](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-rc4) - 2017-10-27 ### Added * Ligature support in the OTF font * Vue.js brand icon * Sass and Less brand icons * Autoprefixer brand icon * Individual icon imports in icon packages FortAwesome/Font-Awesome-Pro#808 ### Changed * Better poo eyes * Renamed HTML status classes to `fontawesome-i2svg-active`, `fontawesome-i2svg-pending`, `fontawesome-i2svg-complete` * HTML status class for active is added only after the first batch of icon replacements occur * Added mention of newer versions of iOS in documentation FortAwesome/Font-Awesome-Pro#810 ### Fixed * Performance and missing features with mutation observer (should fix FortAwesome/Font-Awesome-Pro#813) * Incorrect handling of icon class and style attributes when using autoReplace = 'nest' FortAwesome/Font-Awesome-Pro#809 * Pseudo elements not added or removed when class mutations occur FortAwesome/Font-Awesome-Pro#821 --- ## [5.0.0-rc3](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-rc3) - 2017-10-13 ### Added * Node.js brand icon FortAwesome/Font-Awesome-Pro#779 * React brand icon FortAwesome/Font-Awesome-Pro#780 * OSI brand icon FortAwesome/Font-Awesome-Pro#748 * Add a class to the html element when icon replacement is complete FortAwesome/Font-Awesome-Pro#778 * Add support for symbols in API including ability to name the symbol * Use CSS pseudo elements (:before and :after) to make trigger SVG replacements ### Changed * Switched the locations of fork and knife in utensils-alt FortAwesome/Font-Awesome-Pro#466 * Updated the AWS brand icon FortAwesome/Font-Awesome-Pro#735 * Updated Apple App Store icon FortAwesome/Font-Awesome-Pro#728 ### Fixed * Do not throw an error if icon is missing when calling icon() method in API * Ensure that unicode values do not change between releases * Version field is missing in fontawesome-pro-brands/package.json FortAwesome/Font-Awesome-Pro#781 * Repeated commenting out of fa-layers when i2svg is called FortAwesome/Font-Awesome-Pro#788 * Title not showing up correctly for SVG FortAwesome/Font-Awesome-Pro#786 --- ## [5.0.0-rc2](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-rc2) - 2017-09-22 ### Added * Brand icons: accusoft, ns8, uniregistry ### Fixed * Link to the npm package in the docs FortAwesome/Font-Awesome-Pro#729 * Incorrect reference to fontawesome-pro.js in docs --- ## [5.0.0-rc1](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-rc1) - 2017-09-15 ### Changed * New Bitbucket logo FortAwesome/Font-Awesome-Pro#720 * Modifed the star icons to match use case better FortAwesome/Font-Awesome-Pro#710 * Switched names of css3 and css3-alt to reflect correct branding ### Fixed * Correct whitespace with the Visa logo FortAwesome/Font-Awesome-Pro#719 * Improve OTF support by passing through FontForge FortAwesome/Font-Awesome-Pro#565 * Fonts with "undefined" name FortAwesome/Font-Awesome-Pro#711 * Shims will only function if using old prefix of "fa" FortAwesome/Font-Awesome-Pro#692 * Added missing "youtube" icon to categories --- ## [5.0.0-beta7](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta7) - 2017-09-11 ### Added * Ability to nest the `` FortAwesome/Font-Awesome-Pro#624 * Define icons as symbols and leverage SVG sprites FortAwesome/Font-Awesome-Pro#629 * Added alternative CSS3 logo FortAwesome/Font-Awesome-Pro#682 ### Changed * Power Transforms now execute inside the SVG instead of on the root element * Filenames have changed to reflect a better division between Font Awesome Free and Pro ### Fixed * More improvements to the version 4 shim FortAwesome/Font-Awesome-Pro#673 FortAwesome/Font-Awesome-Pro#678 FortAwesome/Font-Awesome-Pro#686 FortAwesome/Font-Awesome-Pro#687 FortAwesome/Font-Awesome-Pro#692 * Animation support for inline SVG now works as expected FortAwesome/Font-Awesome-Pro#662 --- ## [5.0.0-beta6](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta6) - 2017-09-01 ### Added * Ability to flip horizontal and vertical with CSS classes fa-flip-horizontal and fa-flip-vertical * New film-alt icon that allows for layering other icons * Microsoft brand ### Changed * New YouTube branding FortAwesome/Font-Awesome-Pro#646 ### Fixed * Fixed a bunch of shim-related issues * Cogs off center FortAwesome/Font-Awesome-Pro#663 * Corrected icons/categories.yml with canonical names --- ## [5.0.0-beta5](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta5) - 2017-08-25 ### Added * Full parity with Font Awesome 4! 616 total core icons in each style * 297 total brand and logo icons * Separate CSS file to accompany the SVG Framework FortAwesome/Font-Awesome-Pro#627 * Alternative to the dots icon FortAwesome/Font-Awesome-Pro#608 * Made window icons consistent FortAwesome/Font-Awesome-Pro#611 ### Fixed * Production builds not correctly being detected FortAwesome/Font-Awesome-Pro#631 --- ## [5.0.0-beta4](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta4) - 2017-08-18 ### Added * 590 total core icons in each style * 291 total brand and logo icons ### Fixed * Reduced the size of JS file from 66 to 22 kb * Regression caused by with web font alignment FortAwesome/Font-Awesome-Pro#460 --- ## [5.0.0-beta3](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta3) - 2017-08-15 ### Added * 583 total core icons in each style ### Fixed * Documentation improvements and fixes FortAwesome/Font-Awesome-Pro#586 * Vertical alignment of TTF and OTF fonts FortAwesome/Font-Awesome-Pro#460 * The "fa_500px" icon should be named "fa500px" FortAwesome/Font-Awesome-Pro#578 --- ## [5.0.0-beta2](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta2) - 2017-08-11 ### Added * 570 total core icons in each style * 291 total brand and logo icons * NPM (ES6, CommonJS, AMD) packages for use with other JavaScript libraries and tools FortAwesome/Font-Awesome-Pro#574 * Added a guide to choosing which implementation is best for you FortAwesome/Font-Awesome-Pro#532 ### Changed * Showing a missing icon is now configurable FortAwesome/Font-Awesome-Pro#569 ### Fixed * Composition framework now works in browsers that do not support transform-origin FortAwesome/Font-Awesome-Pro#564 --- ## [5.0.0-beta1](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-beta1) - 2017-08-04 ### Added * 524 total core icons in each style * 289 total brand and logo icons * New composition framework FortAwesome/Font-Awesome-Pro#537 * Animated indicator if you use an icon that does not exist ### Changed * Basic linting for Sass and Less files * Add JavaScript guard block to prevent leaking errors * Add support for automatic accessibility to SVG Framework Layers ### Fixed * Regression where stacks and pulled and bordered were not working in SVG Framework * SVG sprite example had confusing inline styles FortAwesome/Font-Awesome-Pro#549 * Make getting started page more consistent between examples FortAwesome/Font-Awesome-Pro#544 * Added missing sizes fa-[6-10], xs, sm FortAwesome/Font-Awesome-Pro#546 * Title tag missing in SVG sprites FortAwesome/Font-Awesome-Pro#536 --- ## [5.0.0-alpha7](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha7) - 2017-07-28 ### Added * 451 total core icons in each style * 281 total brand and logo icons * Less support is back! * OpenType (.otf) file formats for web fonts ### Changed * Changes the fa-spin animation to go from 0deg to 360deg to eliminate hitch FortAwesome/Font-Awesome-Pro#522 * Improved mutation handling FortAwesome/Font-Awesome-Pro#517 ### Fixed * fa-fw now works correctly with the SVG framework FortAwesome/Font-Awesome-Pro#530 * Removed execute bit on some icon files FortAwesome/Font-Awesome-Pro#520 --- ## [5.0.0-alpha6](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha6) - 2017-07-21 ### Added * 410 total core icons in each style * 270 total brand and logo icons * All new Font Awesome 4 shim file * Beginnings of a public JS API FortAwesome/Font-Awesome-Pro#512 ### Changed * Added Firefox ESR and Chrome for Businesses to browser compatibility FortAwesome/Font-Awesome-Pro#506 ### Fixed * Ensure that SVG title attributes are unique * Fixed incorrect viewBox sizes FortAwesome/Font-Awesome-Pro#492 * Fix chart-area alignment in the solid style FortAwesome/Font-Awesome-Pro#508 * Add missing xmlns attributes in some SVGs FortAwesome/Font-Awesome-Pro#509 --- ## [5.0.0-alpha5](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha5) - 2017-07-14 ### Added * 228 total brand and logo icons * New transform framework for sizing, moving, rotating, and flipping icons * New icon counters * New layers framework * New text overlays * Auto-comments with the original source icons alongside SVG replacements ### Changed * Autoprefixer to correctly add browser prefixes for supported browsers * Removed browser-specific CSS properties in Sass source files (now relies on autoprefixer) ### Fixed * The rotation on checkmark icons * Other icon feedback from previous weeks * Correct fixed width settings to 1.25em (based on the new 16px grid) * Icons displaying as block instead of inline-block in IE and older Safari --- ## [5.0.0-alpha4](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha4) - 2017-07-07 ### Added * 93 brand icons --- ## [5.0.0-alpha3](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha3) - 2017-06-30 ### Added * 95 additional icons; including file types, directional, and some existing and new brand icons ### Fixed * Wrong content type in generated CSS FortAwesome/Font-Awesome-Pro#458 * Removal of query string from static resources FortAwesome/Font-Awesome-Pro#458 * SVG font ID's are incorrect in webfont implementation FortAwesome/Font-Awesome-Pro#474 --- ## [5.0.0-alpha2](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha2) - 2017-06-27 ### Added * How/When to upgrade from FA4 to FA5 FortAwesome/Font-Awesome-Pro#454 ### Fixed * Links to SVG files broken in the example files FortAwesome/Font-Awesome-Pro#456 * Misnamed icon names in examples FortAwesome/Font-Awesome-Pro#445 * Mangled HTML in the Getting Started example FortAwesome/Font-Awesome-Pro#442 * Bad grammar and typos FortAwesome/Font-Awesome-Pro#443 * fas-arrow-to-top is identical to fas-arrow-to-right FortAwesome/Font-Awesome-Pro#423 * Vertical alignment issues with webfont implementation FortAwesome/Font-Awesome-Pro#444 * Add browser compatibility tables to demo FortAwesome/Font-Awesome-Pro#435 * Remove MAC OS feces from builds FortAwesome/Font-Awesome-Pro#437 * TTF naming issues that prevent correct usage/installation FortAwesome/Font-Awesome-Pro#450 * Correct CSS for SVG framework stacking, was reversed from normal FortAwesome/Font-Awesome-Pro#452 --- ## [5.0.0-alpha1](https://github.com/FortAwesome/Font-Awesome-Pro/releases/tag/5.0.0-alpha1) - 2017-06-23 ### Added * 300+ more icons * Brands pack * New JavaScript based SVG Framework * New SVG Sprites based framework * Source SVGs * Documentation with a convenient build-in web server ### Changed * New directory structure

Upgrading Guide
# Upgrading Guide See the [CHANGELOG.md](/assets/bower_components/font-awesome/CHANGELOG/) for detailed information about what has changed between versions. This guide is useful to figure out what you need to do between breaking changes. As always, [submit issues](https://github.com/FortAwesome/Font-Awesome/issues/new) that you run into with this guide or with these upgrades to us. ## 5.0.x to 5.1.0 ### New packages available for browser-only integration **If you were previously using @fortawesome/fontawesome you need to switch to one of the new packages.** Our Free and Pro CDN provide access to JS, CSS, sprites, and separate SVG files. We've now made these files conveniently available through NPM. * [@fortawesome/fontawesome-free](https://www.npmjs.com/package/@fortawesome/fontawesome-free) * @fortawesome/fontawesome-pro (private package, requires Pro subscription) If you are familiar with the paths and options available with the CDN these packages should be familiar. Information about [Font Awesome Pro subscriptions](https://fontawesome.com/pro) can be found in your [Font Awesome awesome account](https://fontawesome.com/account/services). ### Renamed packages The following packages have been renamed as part of 5.1.0 of Font Awesome. _All packages are in the [@fortawesome NPM scope](https://www.npmjs.com/search?q=scope:fortawesome&page=1&ranking=optimal)_ | Old package(1) | New package | |---------------------------|------------------------| | fontawesome-free-webfonts | fontawesome-free | | fontawesome-pro-webfonts | fontawesome-pro | | fontawesome-free-solid | free-solid-svg-icons | | fontawesome-free-regular | free-regular-svg-icons | | fontawesome-free-brands | free-brands-svg-icons | | fontawesome-pro-solid | pro-solid-svg-icons | | fontawesome-pro-regular | pro-regular-svg-icons | | fontawesome-pro-light | pro-light-svg-icons | (1) Old packages have now been deprecated. They are still available but will only receive high priority patch release fixes. **You'll need to update your package.json file with the renamed packages and new versions.** ### No more default imports Recently we spent a good deal of time supporting TypeScript to enable us to create the Angular Font Awesome component. During that adventure we [were](https://basarat.gitbooks.io/typescript/docs/tips/defaultIsBad.html) [convinced](https://blog.neufund.org/why-we-have-banned-default-exports-and-you-should-do-the-same-d51fdc2cf2ad) that we were going to remove default exports from all of our components, libraries, and packages. This is complete with the umbrella release of `5.1.0` of Font Awesome. What does that mean? ~~Old way:~~ ```javascript import fontawesome from '@fortawesome/fontawesome' import solid from '@fortawesome/fontawesome-free-solid' import faTwitter from '@fortawesome/fontawesome-free-brands/faTwitter' import FontAwesomeIcon from '@fortawesome/vue-fontawesome' library.add(solid, faTwitter) ``` New way: ```javascript import { library, dom } from '@fortawesome/fontawesome-svg-core' import { fas } from '@fortawesome/free-solid-svg-icons' import { faTwitter } from '@fortawesome/free-brands-svg-icons' import { FontAwesomeIcon } from '@fortawesome/vue-fontawesome' library.add(fas, faTwitter) // Kicks off the process of finding tags and replacing with

Mouse Wheel ChangeLog
# Mouse Wheel ChangeLog ## 3.1.13 * Update copyright notice and license to remove years * Create the correct compressed version * Remove the obsolete jQuery Plugin Registry file ## 3.1.12 * Fix possible 0 value for line height when in delta mode 1 ## 3.1.11 * Fix version number for package managers... ## 3.1.10 * Fix issue with calculating line height when using older versions of jQuery * Add offsetX/Y normalization with setting to turn it off * Cleans up data on teardown ## 3.1.9 * Fix bower.json file * Updated how the deltas are adjusted for older mousewheel based events that have deltas that are factors of 120. * Add $.event.special.mousewheel.settings.adjustOldDeltas (defaults to true) to turn off adjusting of old deltas that are factors of 120. You'd turn this off if you want to be as close to native scrolling as possible. ## 3.1.8 * Even better handling of older browsers that use a wheelDelta based on 120 * And fix version reported by `$.event.special.mousewheel` ## 3.1.7 * Better handle the `deltaMode` values 1 (lines) and 2 (pages) * Attempt to better handle older browsers that use a wheelDelta based on 120 ## 3.1.6 * Deprecating `delta`, `deltaX`, and `deltaY` event handler arguments * Update actual event object with normalized `deltaX `and `deltaY` values (`event.deltaX`, `event.deltaY`) * Add `deltaFactor` to the event object (`event.deltaFactor`) * Handle `> 0` but `< 1` deltas better * Do not fire the event if `deltaX` and `deltaY` are `0` * Better handle different devices that give different `lowestDelta` values * Add `$.event.special.mousewheel.version` * Some clean up ## 3.1.5 * Bad release because I did not update the new `$.event.special.mousewheel.version` ## 3.1.4 * Always set the `deltaY` * Add back in the `deltaX` and `deltaY` support for older Firefox versions ## 3.1.3 * Include `MozMousePixelScroll` in the to fix list to avoid inconsistent behavior in older Firefox ## 3.1.2 * Include grunt utilities for development purposes (jshint and uglify) * Include support for browserify * Some basic cleaning up ## 3.1.1 * Fix rounding issue with deltas less than zero ## 3.1.0 * Fix Firefox 17+ issues by using new wheel event * Normalize delta values * Adds horizontal support for IE 9+ by using new wheel event * Support AMD loaders ## 3.0.6 * Fix issue with delta being 0 in Firefox ## 3.0.5 * jQuery 1.7 compatibility ## 3.0.4 * Fix IE issue ## 3.0.3 * Added `deltaX` and `deltaY` for horizontal scrolling support (Thanks to Seamus Leahy) ## 3.0.2 * Fixed delta being opposite value in latest Opera * No longer fix `pageX`, `pageY` for older Mozilla browsers * Removed browser detection * Cleaned up the code ## 3.0.1 * Bad release... creating a new release due to plugins.jquery.com issue :( ## 3.0 * Uses new special events API in jQuery 1.2.2+ * You can now treat `mousewheel` as a normal event and use `.bind`, `.unbind` and `.trigger` * Using jQuery.data API for expandos ## 2.2 * Fixed `pageX`, `pageY`, `clientX` and `clientY` event properties for Mozilla based browsers ## 2.1.1 * Updated to work with jQuery 1.1.3 * Used one instead of bind to do unload event for clean up ## 2.1 * Fixed an issue with the unload handler ## 2.0 * Major reduction in code size and complexity (internals have change a whole lot) ## 1.0 * Fixed Opera issue * Fixed an issue with children elements that also have a mousewheel handler * Added ability to handle multiple handlers

Fizzy UI utils
# Fizzy UI utils UI utility & helper functions Used in [Flickity](http://flickity.metafizzy.co), [Isotope](http://isotope.metafizzy.co), [Masonry](http://masonry.desandro.com), [Draggabilly](http://draggabilly.desandro.com) ## Install Bower: `bower install fizzy-ui-utils --save` npm: `npm install fizzy-ui-utils --save` ## API ``` js // fizzyUIUtils is the browser global var utils = fizzyUIUtils; // ---- ---- // utils.extend( a, b ) // extend object utils.modulo( num, div ) // num [modulo] div utils.makeArray( obj ) // make array from object utils.removeFrom( ary, obj ) // remove object from array utils.getParent( elem, selector ) // get parent element of an element, given a selector string utils.getQueryElement( elem ) // if elem is a string, use it as a selector and return element Class.prototype.handleEvent = utils.handleEvent; // enable Class.onclick when element.addEventListener( 'click', this, false ) utils.filterFindElements( elems, selector ) // iterate through elems, filter and find all elements that match selector utils.debounceMethod( Class, methodName, threhold ) // debounce a class method utils.docReady( callback ) // trigger callback on document ready utils.toDashed( str ) // 'camelCaseString' -> 'camel-case-string' utils.htmlInit( Class, namespace ) // on document ready, initialize Class on every element // that matches js-namespace // pass in JSON options from element's data-options-namespace attribute ``` --- [MIT license](http://desandro.mit-license.org/). Have at it. By [Metafizzy](http://metafizzy.co)

@fortawesome/free-regular-svg-icons - SVG with JavaScript version
# @fortawesome/free-regular-svg-icons - SVG with JavaScript version > "I came here to chew bubblegum and install Font Awesome 5 - and I'm all out of bubblegum" [![npm](https://img.shields.io/npm/v/@fortawesome/free-regular-svg-icons.svg?style=flat-square)](https://www.npmjs.com/package/@fortawesome/free-regular-svg-icons) ## Installation ``` $ npm i --save @fortawesome/free-regular-svg-icons ``` Or ``` $ yarn add @fortawesome/free-regular-svg-icons ``` ## Documentation Get started [here](https://fontawesome.com/get-started/svg-with-js). Continue your journey [here](https://fontawesome.com/how-to-use/svg-with-js). Or go straight to the [API documentation](https://fontawesome.com/how-to-use/font-awesome-api). ## Issues and support Start with [GitHub issues](https://github.com/FortAwesome/Font-Awesome/issues) and ping us on [Twitter](https://twitter.com/fontawesome) if you need to.

jQuery Easing Plugin
# jQuery Easing Plugin What is it? A jQuery plugin from GSGD to give advanced easing options. More info [here](http://gsgd.co.uk/sandbox/jquery/easing) For CDN please use CloudFlare [`https://cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.4.1/jquery.easing.min.js`](https://cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.4.1/jquery.easing.min.js) to help my host. Thank you. # AMD or CommonJS usage ```js // CommonJS var jQuery = require('jquery'); require('jquery.easing')(jQuery); // AMD define(['jquery', 'jquery.easing'], function (jQuery, easing) { easing(jQuery) }) ``` # Building and testing * Clone the repo * `npm install` * Make changes * Test against files in `/examples` * Build minified version with `npm run build`

jQuery
# jQuery > jQuery is a fast, small, and feature-rich JavaScript library. For information on how to get started and how to use jQuery, please see [jQuery's documentation](http://api.jquery.com/). For source files and issues, please visit the [jQuery repo](https://github.com/jquery/jquery). If upgrading, please see the [blog post for 3.3.1](https://blog.jquery.com/2017/03/20/jquery-3.3.1-now-available/). This includes notable differences from the previous version and a more readable changelog. ## Including jQuery Below are some of the most common ways to include jQuery. ### Browser #### Script tag ```html ``` #### Babel [Babel](http://babeljs.io/) is a next generation JavaScript compiler. One of the features is the ability to use ES6/ES2015 modules now, even though browsers do not yet support this feature natively. ```js import $ from "jquery"; ``` #### Browserify/Webpack There are several ways to use [Browserify](http://browserify.org/) and [Webpack](https://webpack.github.io/). For more information on using these tools, please refer to the corresponding project's documention. In the script, including jQuery will usually look like this... ```js var $ = require("jquery"); ``` #### AMD (Asynchronous Module Definition) AMD is a module format built for the browser. For more information, we recommend [require.js' documentation](http://requirejs.org/docs/whyamd.html). ```js define(["jquery"], function($) { }); ``` ### Node To include jQuery in [Node](nodejs.org), first install with npm. ```sh npm install jquery ``` For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as [jsdom](https://github.com/tmpvar/jsdom). This can be useful for testing purposes. ```js require("jsdom").env("", function(err, window) { if (err) { console.error(err); return; } var $ = require("jquery")(window); }); ```

Outlayer
# Outlayer _Brains and guts of a layout library_ Outlayer is a base layout class for layout libraries like [Isotope](http://isotope.metafizzy.co), [Packery](http://packery.metafizzy.co), and [Masonry](http://masonry.desandro.com) Outlayer layouts work with a container element and children item elements. ``` html

...
``` ## Install Install with [Bower](http://bower.io): `bower install outlayer` [Install with npm](http://npmjs.org/package/outlayer): `npm install outlayer` ## Outlayer.create() Create a layout class with `Outlayer.create()` ``` js var Layout = Outlayer.create( namespace ); // for example var Masonry = Outlayer.create('masonry'); ``` + `namespace` _{String}_ should be camelCased + returns `LayoutClass` _{Function}_ Create a new layout class. `namespace` is used for jQuery plugin, and for declarative initialization. The `Layout` inherits from [`Outlayer.prototype`](docs/outlayer.md). ``` var elem = document.querySelector('.selector'); var msnry = new Masonry( elem, { // set options... columnWidth: 200 }); ``` ## Item Layouts work with Items, accessible as `Layout.Item`. See [Item API](docs/item.md). ## Declarative An Outlayer layout class can be initialized via HTML, by setting an attribute of `data-namespace` on the element. Options are set in JSON. For example: ``` html
...
``` The declarative attributes and class will be dashed. i.e. `Outlayer.create('myNiceLayout')` will use `data-my-nice-layout` as the attribute. ## .data() Get a layout instance from an element. ``` var myMasonry = Masonry.data( document.querySelector('.grid') ); ``` ## jQuery plugin The layout class also works as jQuery plugin. ``` js // create Masonry layout class, namespace will be the jQuery method var Masonry = Outlayer.create('masonry'); // rock some jQuery $( function() { // .masonry() to initialize var $grid = $('.grid').masonry({ // options... }); // methods are available by passing a string as first parameter $grid.masonry( 'reveal', elems ); }); ``` ## RequireJS To use Outlayer with [RequireJS](http://requirejs.org/), you'll need to set some config. Set [baseUrl](http://requirejs.org/docs/api.html#config-baseUrl) to bower_components and set a [path config](http://requirejs.org/docs/api.html#config-paths) for all your application code. ``` js requirejs.config({ baseUrl: 'bower_components', paths: { app: '../' } }); requirejs( [ 'outlayer/outlayer', 'app/my-component.js' ], function( Outlayer, myComp ) { new Outlayer( /*...*/ ) }); ``` Or set a path config for all Outlayer dependencies. ``` js requirejs.config({ paths: { 'ev-emitter': 'bower_components/ev-emitter', 'get-size': 'bower_components/get-size', 'matches-selector': 'bower_components/matches-selector' } }); ``` ## MIT license Outlayer is released under the [MIT license](http://desandro.mit-license.org).