Legal

Requirements for Customer Cloud Subscriptions

LAST UPDATED: March 17, 2019

Red Hat OpenShift Dedicated Customer Cloud Subscription Requirements

Overview

Red Hat OpenShift Dedicated (OSD) provides a Customer Cloud Subscription (CCS) model that allows Red Hat to deploy and manage OpenShift Dedicated into a customer's AWS account. Red Hat requires several prerequisites be met in order to provide this service. This document is intended to outline our prerequisites,  procedural requirements, and also provide a guide to the AWS resources and Identity and Access Management (IAM) users, roles, policies, and identity providers that will be provisioned as part of this service. This service is supported by Red Hat Site Reliability Engineers (SRE).

image1
Figure 1. Red Hat recommended AWS Organization and Account structure

Red Hat recommends the usage of an AWS Organization to manage multiple AWS accounts.

The AWS Organization managed by the customer hosts multiple AWS accounts. There is a root account in the organization that all accounts ultimately refer to in the account hierarchy. 

Best practices for OSD CCS is for the cluster to be hosted in an AWS account within an AWS Organizational Unit. A SCP is created and applied to the Organizational Unit that manages what services the AWS sub-accounts are permitted to access. The SCP applies only to available permissions within a single AWS account for all AWS sub-accounts within the Organizational Unit. It is also possible to apply a SCP to a single AWS account. All other accounts in the customer’s AWS Organization are managed in whatever manner the customer requires. Red Hat SRE will not have any control over SCPs within the AWS Organization.

Customer Requirements

Account Requirements

  • All CCS accounts must be for AWS.
  • Customer ensures AWS limits are sufficient to support managed services provisioned within the customer-provided AWS account.
  • Customer-provided AWS account should be in the customer's AWS Organization with the applicable Service Control Policy applied. 
    • Note: It is not a requirement that the customer-provided account be within an AWS Organization or for the Service Control Policy to be applied, however Red Hat must be able to perform all the actions listed in the Service Control Policy without restriction.
  • Customer-provided AWS account should not be transferable to Red Hat.
  • Customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions will severely hinder Red Hat's ability to respond to incidents.
  • Red Hat will deploy monitoring into AWS to alert Red Hat when a highly privileged AWS account, e.g., a root account, logs into the customer-provided AWS account.
  • Customer may deploy native AWS services within the same customer-provided AWS account
    • Note: Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting OpenShift Dedicated and other Red Hat supported services. 

Access Requirements

  • Red Hat must have the AdministratorAccess policy applied to the "admin" role at all times in order to appropriately manage the OSD service. 
    • Note: This only provides Red Hat with permissions and capabilities to affect resources in the customer-provided AWS account.
  • Red Hat must have AWS Console access to the customer-provided AWS account. This access is protected and managed by Red Hat.
  • Customer must not utilize the AWS account to elevate their permissions within the OSD cluster.
  • Actions available in the OpenShift Cluster Manager must not be directly performed in the customer-provided AWS account.
  • OSD CCS v3 customers will have customer-admin and read-only roles federated to a separate AWS account specified by the customer. It is the customer's responsibility to manage authentication and authorization to these roles via AWS account federation
  • OSD CCS v4 customers will be able to self-service role federation via OpenShift Cluster Manager (OCM).

Support Requirements

  • Red Hat recommends that the customer have at least Business Support from AWS.
  • Red Hat has authority from customer to request AWS support on behalf of the customer.
  • Red Hat has authority from customer to request AWS resource limit increases on the customer-provided account.
  • Red Hat manages all OSD clusters in the same manner including the same restrictions, limitations, expectations, and defaults unless otherwise specified in this requirements section.

Security Requirements

  • Customer-provided IAM credentials should be unique to the customer-provided AWS account and should not be stored anywhere in the customer-provided AWS account.
  • Volume snapshots will remain within the customer-provided AWS account and customer-specified region.
  • Red Hat must have ingress access to EC2 hosts and the API server via white-listed Red Hat machines.
  • Red Hat must have egress allowed in order to forward system and audit logs to a Red Hat managed central logging stack. 

Customer Procedure

image2
Figure 2. AWS configuration procedure

  1. If customer is utilizing AWS Organizations, you must use an AWS account within your organization or create a new one.
  2. In order to ensure that Red Hat can perform the necessary actions, you must either create a Service Control Policy or ensure that none is applied to the AWS account.
  3. Attach the Service Control Policy to the AWS account.
  4. Within the AWS account, you must create an osdCcsAdmin IAM user:
    • This user needs at least Programmatic access access type enabled.
    • This user must have the AdministratorAccess policy attached to it.
  5. Provide the IAM user credentials to Red Hat:
    • For OSD v3 CCS, you must provide the access key ID and secret access key GPG encrypted with a key provided by Red Hat.
    • For OSD v4 CCS, you must provide the access key ID and secret access key in OpenShift Cluster Manager creation form.

Minimum Required Service Control Policy

Service Control Policy (SCP) management is the responsibility of the customer. These policies are maintained in the AWS Organization and control what services are available within the attached AWS accounts.

  Service Actions Effect
Required Amazon EC2 All Allow
Amazon EC2 Auto Scaling All Allow
Amazon S3 All Allow
Identity And Access Management All Allow
Elastic Load Balancing All Allow
Elastic Load Balancing V2 All Allow
AWS Direct Connect All Allow
Amazon CloudWatch All Allow
Amazon CloudWatch Events All Allow
Amazon CloudWatch Logs All Allow
AWS Support All Allow
AWS Key Management Service All Allow
AWS Security Token Service All Allow
AWS Cost and Usage Report All Allow
AWS Cost Explorer Service All Allow
AWS Resource Tagging All Allow
AWS Route53 DNS All Allow
AWS Billing ViewAccount
ViewBilling
ViewUsage		 Allow

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1543327396000",
            "Effect": "Allow",
            "Action": [
                "ec2:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327408000",
            "Effect": "Allow",
            "Action": [
                "autoscaling:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327417000",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327428000",
            "Effect": "Allow",
            "Action": [
                "iam:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327656000",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1546616571000",
            "Effect": "Allow",
            "Action": [
                "directconnect:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327666000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327671000",
            "Effect": "Allow",
            "Action": [
                "events:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327675000",
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327772000",
            "Effect": "Allow",
            "Action": [
                "support:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327781000",
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1548175905000",
            "Effect": "Allow",
            "Action": [
                "sts:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327792000",
            "Effect": "Allow",
            "Action": [
                "cur:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327798000",
            "Effect": "Allow",
            "Action": [
                "ce:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "AllowTagging",
            "Effect": "Allow",
            "Action": [
                "tag:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "AllowRoute53",
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1543327831000",
            "Effect": "Allow",
            "Action": [
                "aws-portal:ViewAccount",
                "aws-portal:ViewBilling",
                "aws-portal:ViewUsage"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Red Hat Managed AWS Resources

The below sections define AWS resources that Red Hat is responsible for creating.

OSD CCS v3

These references only apply to OpenShift Dedicated Customer Cloud Subscription accounts that are running on OpenShift 3.11. 

IAM References

IAM Policies

These policies are subject to modification as the capabilities of OpenShift Dedicated change.

AdministratorAccess

The AdministratorAccess policy is used by the "admin" role. It provides Red Hat the access necessary to administer the OSD cluster in the customer-provided AWS account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "*",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

CustomerAdministratorAccess

The CustomerAdministatorAccess role provides the customer access to administer a subset of services within the AWS account. At this time, the following are allowed:

  • VPC Peering
  • VPN Setup
  • AWS Support
  • Direct Connect
  • AWS GuardDuty
  • AWS Transit Gateway
  • AWS Route53 Resolver

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AcceptVpcPeeringConnection",
                "ec2:AttachVpnGateway",
                "ec2:CreateRoute",
                "ec2:CreateTags",
                "ec2:CreateVPCEndpoint",
                "ec2:CreateVpcPeeringConnection",
                "ec2:CreateVpnConnection",
                "ec2:CreateVpnConnectionRoute",
                "ec2:CreateVpnGateway",
                "ec2:DeleteVpcPeeringConnection",
                "ec2:DeleteVpnConnection",
                "ec2:DeleteVpnConnectionRoute",
                "ec2:DeleteVpnGateway",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSubnets",
                "ec2:DescribeTransitGatewayAttachments",
                "ec2:DescribeTransitGatewayVpcAttachments",
                "ec2:DescribeTransitGateways",
                "ec2:DescribeVPCEndpoint",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpnConnections",
                "ec2:DescribeVpnGateways",
                "ec2:DetachVpnGateway",
                "ec2:ModifyVpcPeeringConnectionOptions",
                "ec2:RejectVpcPeeringConnection",
                "guardduty:GetDetector",
                "guardduty:GetFindings",
                "guardduty:GetFindingsStatistics",
                "guardduty:GetFreeTrialStatistics",
                "guardduty:GetIPSet",
                "guardduty:GetInvitationsCount",
                "guardduty:GetMasterAccount",
                "guardduty:GetMembers",
                "guardduty:GetThreatIntelSet",
                "guardduty:ListDetectors",
                "guardduty:ListFilters",
                "guardduty:ListFindings",
                "guardduty:ListIPSets",
                "guardduty:ListInvitations",
                "guardduty:ListMembers",
                "guardduty:ListThreatIntelSets",
                "ram:AcceptResourceShareInvitation",
                "ram:DeleteResourceShare",
                "ram:GetResourcePolicies",
                "ram:GetResourceShareAssociations",
                "ram:GetResourceShareInvitations",
                "ram:GetResourceShares",
                "ram:ListPendingInvitationResources",
                "ram:ListPrincipals",
                "ram:ListResources",
                "ram:RejectResourceShareInvitation",
                "route53resolver:AssociateResolverRule",
                "route53resolver:DeleteResolverRule",
                "route53resolver:DisassociateResolverRule",
                "route53resolver:GetResolverRule",
                "route53resolver:GetResolverRuleAssociation",
                "route53resolver:ListResolverRuleAssociations",
                "route53resolver:ListResolverRules",
                "support:*",
                "directconnect:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ram:*",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ram:RequestedResourceType": "route53resolver:ResolverRule"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateTransitGatewayVpcAttachment",
            "Resource": [
                "arn:aws:ec2:*:*:subnet/*",
                "arn:aws:ec2:*:*:transit-gateway/*",
                "arn:aws:ec2:*:*:vpc/*",
                "arn:aws:ec2:*:*:transit-gateway-attachment/*"
            ]
        }
    ]
}

BillingReadOnlyAccess

The BillingReadOnlyAccess role provides read-only access to view billing and usage information for the account if it is enabled.

Billing and usage access is only granted if the root account in the AWS Organization has it enabled. This is an optional step the customer must perform to enable read-only billing and usage access and does not impact the creation of this profile and the role that uses it. If not enabled, the impact is that users will not see billing and usage information. See this tutorial on how to enable access to billing data.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aws-portal:ViewAccount",
                "aws-portal:ViewBilling"
            ],
            "Resource": "*"
        }
    ]
}

IAM Identity Providers

ops_sso_saml

This is the SAML provider used to federate access for Red Hat administration of the account.

IAM Roles

admin

This role provides Red Hat administrative access to the AWS account via SAML federation.

Detail

Type:  SAML 2.0 federation
Trust Relationship:  arn:aws:iam::<Account ID>:saml-provider/ops_sso_saml
Policies:

    • AdministratorAccess
customer-admin

This role provides customer-federated administrative access to the AWS account via a separate AWS account. It also has the same access as read-only role.

Detail

Type: AWS account
Policies:

    • AWSAccountUsageReportAccess
    • AmazonEC2ReadOnlyAccess
    • AmazonS3ReadOnlyAccess
    • IAMReadOnlyAccess
    • BillingReadOnlyAccess
    • CustomerAdministratorAccess
read-only

This role provides customer-federated read-only access to the AWS account via a separate AWS account.

Detail

Type: AWS account
Policies:

    • AWSAccountUsageReportAccess
    • AmazonEC2ReadOnlyAccess
    • AmazonS3ReadOnlyAccess
    • IAMReadOnlyAccess
    • BillingReadOnlyAccess

Provisioned AWS Infrastructure

EC2 Instances

AWS EC2 instances are required for deploying control plane and data plane functions of OpenShift Dedicated in the AWS public cloud.

Detail
    • 3 m5.xlarge minimum (Masters Nodes)
    • 3 r5.2xlarge minimum (Infrastructure Nodes)
    • 4 m5.xlarge minimum but highly variable (Compute Nodes)

EBS Storage 

Amazon EBS block storage is used for both local node storage and persistent volume storage.

Detail

Volume requirements for each EC2 instance:  

    • Master Nodes
      • /dev/sda1 
        • size: 100GB
        • type: gp2
      • /dev/sdb
        • size: 200GB
        • type: gp2
      • /dev/sdc
        • size: 50GB
        • type: io1
        • iops: 2000
    • Infrastructure Nodes
      • /dev/sda1
        • size: 100GB
        • type: gp2
      • /dev/sdb
        • size: 200GB
        • type: gp2
    • Compute Nodes
      • /dev/sda1
        • size: 100GB
        • type: gp2
      • /dev/sdb
        • size: 200GB
        • type: gp2

Elastic Load Balancers

Three Classic Elastic Load Balancers (ELBs). See here for more details on ELBs.

S3 storage 

The registry is backed by AWS S3 storage. Aggressive pruning of resources is performed regularly to optimize S3 usage and cluster performance. S3 storage is also used for logging. Logs are retained for 120 days. 

Detail

Buckets required: 2 
Typical size: 2TB each

VPC

Customers should expect to see one VPC per cluster. Additionally, the VPC will need following configurations:

  • Subnets: 1 subnet if cluster with single availability zone is preferred, 3 subnets if multi-az is preferred
  • Router Tables: 1 per cluster
  • Internet Gateways: 1 per cluster

CloudTrail

One AWS CloudTrail instance is required for audit logging. Refer here for documentation on CloudTrail. 

Launch Configuration and Auto Scaling groups

A launch configuration is an instance configuration template that an Auto Scaling group uses to launch EC2 instances. Refer to AWS documentation on launch configuration and Auto Scaling groups for more details. When you create a launch configuration, you specify information for the instances. OpenShift Dedicated will require one instance of Infrastructure EC2 and one instance of Compute EC2 in the launch configuration and AWS Auto Scaling groups. 

Security Groups

AWS security groups are associated with EC2 instances and provide security at the protocol and port access level. Each security group, working much the same way as a firewall, contains a set of rules that filter traffic coming into and out of an EC2 instance. You must ensure the ports required for OpenShift installation listed here are open on your network and configured to allow access between hosts. 

OSD CCS v4

These references only apply to OpenShift Dedicated Customer Cloud Subscription accounts that are running on OpenShift 4.x.

IAM References

IAM Policies

These policies are subject to modification as the capabilities of OpenShift Dedicated change.

AdministratorAccess

The AdministratorAccess policy is used by the "admin" role. It provides Red Hat the access necessary to administer the OSD cluster in the customer-provided AWS account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "*",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

CustomerAdministratorAccess

The CustomerAdministatorAccess role provides the customer access to administer a subset of services within the AWS account. At this time, the following are allowed:

  • VPC Peering
  • VPN Setup
  • Direct Connect

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
          "ec2:AttachVpnGateway",
          "ec2:DescribeVpnConnections",
          "ec2:AcceptVpcPeeringConnection",
          "ec2:DeleteVpcPeeringConnection",
          "ec2:DescribeVpcPeeringConnections",
          "ec2:CreateVpnConnectionRoute",
          "ec2:RejectVpcPeeringConnection",
          "ec2:DetachVpnGateway",
          "ec2:DeleteVpnConnectionRoute",
          "ec2:DeleteVpnGateway",
          "ec2:DescribeVpcs",
          "ec2:CreateVpnGateway",
          "ec2:ModifyVpcPeeringConnectionOptions",
          "ec2:DeleteVpnConnection",
          "ec2:CreateVpcPeeringConnection",
          "ec2:DescribeVpnGateways",
          "ec2:CreateVpnConnection",
          "ec2:DescribeRouteTables",
          "ec2:CreateTags",
          "ec2:CreateRoute",
          "directconnect:*"
            ],
            "Resource": "*"
        }
    ]
}

BillingReadOnlyAccess

The BillingReadOnlyAccess role provides read-only access to view billing and usage information for the account if it is enabled.

Billing and usage access is only granted if the root account in the AWS Organization has it enabled. This is an optional step the customer must perform to enable read-only billing and usage access and does not impact the creation of this profile and the role that uses it. If not enabled, the impact is that users will not see billing and usage information. See this tutorial on how to enable access to billing data.

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aws-portal:ViewAccount",
                "aws-portal:ViewBilling"
            ],
            "Resource": "*"
        }
    ]
}

IAM Users

osdManagedAdmin

This user is created immediately after taking control of the customer-provided AWS account. This is the user that will be performing the OSD cluster install.

IAM Roles

network-mgmt

This role provides customer-federated administrative access to the AWS account via a separate AWS account. It also has the same access as read-only role.

Detail

Type: AWS account
Policies:

    • AmazonEC2ReadOnlyAccess
    • CustomerAdministratorAccess
read-only

This role provides customer-federated read-only access to the AWS account via a separate AWS account.

Detail

Type: AWS account
Policies:

    • AWSAccountUsageReportAccess
    • AmazonEC2ReadOnlyAccess
    • AmazonS3ReadOnlyAccess
    • IAMReadOnlyAccess
    • BillingReadOnlyAccess

Provisioned AWS Infrastructure

This is an overview of the OpenShift Dedicated specific deployments. For a more detailed listing of all AWS components provisioned, you can refer to the OpenShift Container Platform documentation.

EC2 Instances

AWS EC2 instances are required for deploying control plane and data plane functions of OpenShift Dedicated in the AWS public cloud.

Detail
    • 3 m5.xlarge minimum (Masters Nodes)
    • 3 m5.xlarge minimum (Infrastructure Nodes)
    • 4 m5.xlarge minimum but highly variable (Compute Nodes)

EBS Storage 

Amazon EBS block storage is used for both local node storage and persistent volume storage.

Detail

Volume requirements for each EC2 instance:  

    • Master Volume
        • size: 350GB
        • type: io1
        • iops: 1000
    • Infrastructure Volume
        • size: 300GB
        • type: gp2
        • Iops: 100
    • Compute Volume
        • size: 300GB
        • type: gp2
        • Iops: 100

Elastic Load Balancers

Up to two Network Elastic Load Balancers (ELBs) for API and up to two Classic ELBs for application router. See here for more details on ELBs.

S3 storage 

The image registry and EBS volume snapshots are backed by AWS S3 storage. Pruning of resources is performed regularly to optimize S3 usage and cluster performance. 

Detail

Buckets required: 2 
Typical size: 2TB each

VPC

Customers should expect to see one VPC per cluster. Additionally, the VPC will need following configurations:

  • Subnets: 2 subnets if cluster with single availability zone is preferred, 6 subnets if multi-az is preferred
  • Router Tables: 1 per private subnet, and 1 additional table per cluster
  • Internet Gateways: 1 per cluster
  • NAT Gateways: 1 per public subnet

Security Groups

AWS security groups are associated with EC2 instances and Elastic Load Balancers and provide security at the protocol and port access level. Each security group, working much the same way as a firewall, contains a set of rules that filter traffic coming into and out of an EC2 instance. You must ensure the ports required for OpenShift installation listed here are open on your network and configured to allow access between hosts.