Configure Execution Environment
Failure injection is a means of testing resiliency by which a specific failure type is simulated on a service and its response is assessed.
You have a choice of environments from which to execute the failure injections for this lab. Bash scripts are a good choice and can be used from a Linux command line. If you prefer Python, Java, Powershell, or C#, then instructions for these are also provided.
In addition to custom scripts, you can also perform failure injection experiments using AWS Fault Injection Simulator (FIS).
2.1 Setup AWS CloudShell
If you will be using bash, Java, or Python, and are comfortable with Linux, it is highly recommended you use AWS CloudShell for this lab. If you will not be using AWS CloudShell, then skip to Step 2.2
Go to the AWS CloudShell console here
If this is your first time running CloudShell, then it will take less than a minute to create the environment. When you see a prompt like [cloudshell-user@ip-10-0-49-48 ~]$, then you can continue
Validate that credentials are properly setup.
- execute the command
aws sts get-caller-identity - If the command succeeds, and the Arn contains assumed-role/TeamRole/MasterKey, then you can continue
Adjust font size and theme using the gear icon on the upper right
Explore the Actions menu (upper-right) - you can upload/download files or create new tabs
Skip to Step 2.3
2.2 Setup AWS credentials and configuration
If you have chosen to use AWS CloudShell or Windows PowerShell, then skip this step
Otherwise, your execution environment needs to be configured to enable access to the AWS account you are using for the workshop. This includes:
Note: us-east-2 is the Ohio region
- If you already know how to configure these, please do so now.
- If you need help, then follow these instructions
- If you are using PowerShell for this lab, skip this step and continue to Step 2.3
2.3 Set up the programming language environment
Choose the appropriate section below for your language
Using bash is an effective way to execute the failure injection tests for this workshop. The bash scripts make use of the AWS CLI. Or if you wish, you may choose one of the other languages and scripts.
Click here for instructions if using bash:
Prerequisites
awscli AWS CLI installed
$ aws --version
aws-cli/2.2.15 Python/3.8.8...
jq command-line JSON processor installed.
$ jq --version
jq-1.5-1-a5b5cbe
Download the resiliency bash scripts from GitHub
to your execution location (this is AWS CloudShell if you are using that for this lab). You can use the following links to download the scripts:
If using Linux, then from your execution location (AWS CloudShell if you are using that for this lab):
- Copy the link URL (from the links above)
- Use this command to download the script:
wget <url>
Set the scripts to be executable.
chmod u+x fail_instance.sh
chmod u+x failover_rds.sh
chmod u+x fail_az.sh
Click here for instructions if using Python:
Check that python 3 is installed. This is already installed with AWS CloudShell or Amazon Linux.
$ python3 --version
Python 3.7.10
The scripts are written in python with boto3. This is already installed with AWS CloudShell or Amazon Linux.
Download the resiliency Python scripts from GitHub
to your execution location (this is AWS CloudShell if you are using that for this lab). You can use the following links to download the scripts:
If using Linux, then from your execution location (AWS CloudShell if you are using that for this lab):
- Copy the link URL (from the links above)
- Use this command to download the script:
wget <url>
Click here for instructions if using Java:
Java and Maven must be installed
$ mvn -version
Apache Maven 3.0.5 (Red Hat 3.0.5-17)
Maven home: /usr/share/maven
Java version: 1.8.0_302, vendor: Red Hat, Inc.
...
If Maven is not installed, or Java is not 1.8 or higher, then install Maven and Java
Next choose one of the following options: Option A or Option B.
Build: sudo mvn clean package shade:shade
cd target - this is where your jar files were built and where you can run from the command line
Click here for instructions if using C#:
Download the zipfile of the executable at the following URL. https://s3.us-east-2.amazonaws.com/aws-well-architected-labs-ohio/Reliability/csharpresiliency.zip
Unzip the folder in a location convenient for you to execute the command line program.
- The executable is called
AppResiliency.exe - It is in the
win10-x64 folder - Later, when it is time to run you your experiments, you should run the executable from a Windows command prompt
Avoid potential credentials problems
- Check to see if you have a file
%USERPROFILE%\AppData\Local\AWSToolkit\RegisteredAccounts.json - If so, then this file may override the AWS credentials you need to run the lab
- You can rename it so that it does not interfere with the lab
move %USERPROFILE%\AppData\Local\AWSToolkit\RegisteredAccounts.json %USERPROFILE%\AppData\Local\AWSToolkit\RegisteredAccounts-BAK.json
- and then move it back if you want, after the lab
To view or download the source code, see this github repo
Click here for instructions if using PowerShell:
To install the necessary AWS Tools for Powershell packages, and to setup AWS credentials for PowerShell follow the instructions here
Download the resiliency PowerShell scripts from GitHub
to a location where you can run them from within PowerShell. You can use the following links to download the scripts:
To download the script using PowerShell:
- Copy the link URL (from the links above)
- Use this command to download the script:
Invoke-WebRequest "<url>" -OutFile "<filename>"- Replace
<url> and <filename> - Keep the quotation marks
2.4 IAM Role for FIS
In this lab, some of the experiments will be executed AWS Fault Injection Simulator (FIS) in addition to using custom scripts. FIS needs a service role to inject failures for various components of a workload.
This IAM Role has already been created for you as part of the infrastructure deployment
You may proceed to the next step.
If you would like to view the instructions on how to create the IAM Role for FIS (for your information), then click here:
These instructions are here for informational purposes only. You DO NOT need to execute these as this IAM Role was created for you as part of the infrastructure deployment
Navigate to the AWS Identity and Access Management (IAM) console.
Click on Policies from the menu on the left and then click Create Policy.
On the Create policy wizard, click on the JSON tab and replace the contents with the following policy. Click Next: Tags.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowFISExperimentRoleReadOnly",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ecs:DescribeClusters",
"ecs:ListContainerInstances",
"eks:DescribeNodegroup",
"iam:ListRoles",
"rds:DescribeDBInstances",
"rds:DescribeDbClusters",
"ssm:ListCommands"
],
"Resource": "*"
},
{
"Sid": "AllowFISExperimentRoleEC2Actions",
"Effect": "Allow",
"Action": [
"ec2:RebootInstances",
"ec2:StopInstances",
"ec2:StartInstances",
"ec2:TerminateInstances"
],
"Resource": "arn:aws:ec2:*:*:instance/*"
},
{
"Sid": "AllowFISExperimentRoleECSActions",
"Effect": "Allow",
"Action": [
"ecs:UpdateContainerInstancesState",
"ecs:ListContainerInstances"
],
"Resource": "arn:aws:ecs:*:*:container-instance/*"
},
{
"Sid": "AllowFISExperimentRoleEKSActions",
"Effect": "Allow",
"Action": [
"ec2:TerminateInstances"
],
"Resource": "arn:aws:ec2:*:*:instance/*"
},
{
"Sid": "AllowFISExperimentRoleFISActions",
"Effect": "Allow",
"Action": [
"fis:InjectApiInternalError",
"fis:InjectApiThrottleError",
"fis:InjectApiUnavailableError"
],
"Resource": "arn:*:fis:*:*:experiment/*"
},
{
"Sid": "AllowFISExperimentRoleRDSReboot",
"Effect": "Allow",
"Action": [
"rds:RebootDBInstance"
],
"Resource": "arn:aws:rds:*:*:db:*"
},
{
"Sid": "AllowFISExperimentRoleRDSFailOver",
"Effect": "Allow",
"Action": [
"rds:FailoverDBCluster"
],
"Resource": "arn:aws:rds:*:*:cluster:*"
},
{
"Sid": "AllowFISExperimentRoleSSMSendCommand",
"Effect": "Allow",
"Action": [
"ssm:SendCommand"
],
"Resource": [
"arn:aws:ec2:*:*:instance/*",
"arn:aws:ssm:*:*:document/*"
]
},
{
"Sid": "AllowFISExperimentRoleSSMCancelCommand",
"Effect": "Allow",
"Action": [
"ssm:CancelCommand"
],
"Resource": "*"
}
]
}
Click Next: Review.
On the Review policy page, enter WALab-FIS-policy under Name and click Create policy.
Click on Roles from the menu on the left and then click Create role.
FIS is currently not listed in the list of services under use cases. For the time being, Select EC2 and click Next: Permissions.
Under Attach permissions policies, enter WALab-FIS-policy and select the WALab-FIS-policy. This is the policy that was created in the previous steps.
Click Next: Tags.
Click Next: Review.
Enter WALab-FIS-role for Role name. Update the description to Allows FIS to call AWS services on your behalf. and click Create role.
Search for the newly created role WALab-FIS-role and click on it to view details.
On the Trust relationships tab, click Edit trust relationship.
Replace the existing Policy Document with the following and click Update Trust Policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "fis.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
The change should be reflected in the Trust relationships tab,
