AWS DevOps Exam preparation: Code Pipeline

Okay, so now let's talk about CodePipeline, which is a visual workflow tool that will allow you to orchestrate your CI/CD within AWS. So with CodePipeline, you can show sources such as hey, my code is in CodeCommit or we have a Docker image in ECR, or my code is in Amazon S3. Or even external tools such as Bitbucket or GitHub. Then you can move on to the build phase, which is, hey, now that we have this code, let's build it.

So CodeBuild, Jenkins, CloudBees, TeamCity are all options. Then once you have the build phase, you can have the test phase. So again, testing your code. So CodeBuild or Device Farm if you have an app, for example, an iOS, an Android app, or any third-party tools you want. Then once the code is tested, you want to deploy it. So CodeDeploy, Beanstalk, CloudFormation, ECS, S3, all these options can be handled by CodePipeline. And Invoke to invoke a Lambda function or a step function. And then when you have all these building blocks, you can build stages.

So each stage can have sequential actions and/or parallel actions. You can do a lot of things. Here's a very simple example. Build, test, deploy onto staging, then load testing to make sure staging is doing fine. And then once the load testing is done, maybe deploy again to production. Now, there's also a way for you to define manual approval at any stage in your pipeline, really allowing you, for example, just before deploying to prod to have someone review the results of the load testing, for example, if it was there. And make sure it says, yeah, looks good. Green light, deploy to prod.

Now, how does CodePipeline work inside?

So let's assume we have a source, a build and a deploy phase. So source is CodeCommit. Build is CodeBuild, even though we haven't seen CodeBuild yet, and Deploy is CodeDeploy even though we haven't seen CodeDeploy yet. So each pipeline can create artifacts. Artifacts is whatever is created out of the pipeline and the artifacts are going to be stored in S3 buckets to be passed onto the next stage.

And this is how the next stage is going to be able to do what it needs to do. So let's do a concrete example. Developer is going to push some code into CodeCommit, right? And then CodeCommit is going to be orchestrated by CodePipeline, which is going to extract all the code and create an artifact out of it and place that artifact into an S3 bucket. Now, when CodeBuild is invoked, the same artifacts that were extracted are going to be inputted directly into CodeBuild, and that's why CodeBuild doesn't need to have straight access into CodeCommit. Actually, it's CodePipeline that will be pushing the code to CodeBuild through Amazon S3. Then when CodeBuild is building the code, it's going to create some deployment artifacts. So these artifacts are going to be stored yet again in your S3 bucket by CodePipeline, and CodePipeline will push these artifacts yet again to CodeDeploy and CodeDeploy says hey, I have these artifacts, I need to deploy them.

Let's go ahead and deploy them. So as you can see, these stages interact with each other through Amaz on S3, and this is why we have artifacts in CodePipeline.

Now, some troubleshooting for CodePipeline

So if you need to have a look at all these things, for example, you need to look at like CodePipeline action or stage execution state changes, you can use CloudWatch Events, EventBridge to have a look at them. So for example, you can create events for failed pipelines, events for cancelled stages and then receive an email notification, for example. And then if CodePipeline has a failure at a stage, you will see it visually and you can get information through the console. And if there is no way for CodePipeline to perform a specific action, for example, to invoke some code in CodeBuild or to pull the code from CodeCommit, then check the IAM service role of CodePipeline and make sure it has right IAM permissions. Also, if you need to have a look at maybe some denied API calls within your infrastructure, you can use CloudTrail, which is a service used to audit AWS API calls. So let's say for CodePipeline,Now, some troubleshooting for CodePipeline. So if you need to have a look at all these things, for example, you need to look at like CodePipeline action or stage execution state changes, you can use CloudWatch Events, EventBridge to have a look at them.

So for example, you can create events for failed pipelines, events for cancelled stages and then receive an email notification, for example. And then if CodePipeline has a failure at a stage, you will see it visually and you can get information through the console. And if there is no way for CodePipeline to perform a specific action, for example, to invoke some code in CodeBuild or to pull the code from CodeCommit, then check the IAM service role of CodePipeline and make sure it has right IAM permissions.

Also, if you need to have a look at maybe some denied API calls within your infrastructure, you can use CloudTrail, which is a service used to audit AWS API calls.

CodePipeline - Events vs webhooks vs polling

So let's talk about some few additional points about CodePipeline.

The first one is that you have the concept to start a pipeline of events, webhooks and polling. So let's have a look at all of them and see which one is best in which situation.

So events is going to be the preferred way in CodePipeline and this says to start a pipeline whenever we have an event.So for example, when we have CodeCommit, for example, we know that on a new commit, an event will happen in EventBridge, and the EventBridge rule associated with that event can trigger and start a CodePipeline. This is the preferred way, and it's not only CodeCommit, but could be any kind of events within AWS.

With GitHub, which is not part of AWS, how do you have event driven starting of CodePipeline? Well, to do so, you're going to use what's called a CodeStarSourceConnection. It's just a fancy name for a GitHub application

that is going to connect GitHub into AWS. And then from this, you're going to trigger CodePipeline. So these things are event driven and they are very fast because as soon as the event happens, CodePipeline is going to be triggered

An older way of triggering CodePipeline is using webhook. So if you choose that option, then CodePipeline is going to expose an HTTP endpoint and that endpoint can be triggered by a script, whatever you want. If that script sends a payload to CodePipeline on that webhook, then CodePipeline is going to start.

And finally, you can have CodePipeline poll the source, in which case you have regular checks, for example, from CodePipeline onto GitHub, but that is not recommended because it's not as efficient as events.

So events are the default and recommended way to start a CodePipeline.

Manual Approval on Deployment

So when you have a CodePipeline and you have a manual approval, the important part is that the owner is AWS because it says related to within AWS. It's a capability offered by an AWS service and the action is going to be manual because it says a manual approval. In which case when you have a manual approval, what's going to happen is that you can trigger an SNS topic which is in turn you can send in emails to a user and the user will have an IAM user on AWS.

And then you will have to approve this stage. And to approve this stage, it needs permissions. And the permissions the user must have are twofold. Number one is get pipeline because while the user must be able to get to your pipeline to actually view it and to find that manual approval step, so we want the get pipeline star type of action. And then we want the put approval result action on the approval action itself because we want to be able to say yes or no, we approve or we deny.

So that's all you need to know. But it's important to understand the IAM user permissions as well as the fact that the step owner is AWS and the action of this step is manual.

CloudFormation as a target

So let's talk about using CloudFormation as a target for CodePipeline. So CloudFormation is a deploy action and it can be used to deploy any kind of AWS resources and including Lambda functions, either as pure CloudFormation or using the CDK framework or using SAML. So the idea is that from CodePipeline, you have your code in CodeCommit.

The code could be directly the template itself. You can create a change set from CloudFormation, then a manual approval to make sure that the change set is what we want it to be. And then execute the change sets. This is one option.

You can just go ahead also and deploy things right directly without doing a manual approval. So on top of supporting CloudFormation in one region, you support CloudFormation's StackSets to deploy across multiple AWS accounts and AWS regions. On top of things, you can configure multiple settings, such as the stack name, the change set name, the template input, the parameters if you want to override them, the IAM role, the action mode, and so on.

So let's have a look at an architecture and a use case for using CodePipeline with CloudFormation. So we have CodePipeline and we build our app in CodeBuild and we create a template.yaml file out of it. Next, we use CloudFormation to actually deploy the infrastructure in our app and we use the CREATE_UPDATE mode to create or update an existing stack.

So our entire stack will be deployed. For example, this is an ALB with an auto scaling group and EC2 instances. And then we have a new stage to test the application using CodeBuild. So CodeBuild can run, for example, test against the ALB by using the HTTP protocol and make sure our application is functioning as expected. It could be a functional test, it could be load testing, it could be whatever you want. If all of this is working, that means that our application version is working as expected and we can, for example, delete the test infrastructure using the DELETE_ONLY action for CloudFormation to delete the stack.

And then another CloudFormation action, this time to deploy the prod infrastructure in prod by using again the CREATE_UPDATE action, which in this time is going to actually update our prod infrastructure. So hope that makes sense, but that makes it very easy for you to create infrastructure and destroy them as part of your CI/CD pipeline.

So, let's assume we want to deploy a Lambda function into two regions.

The way we're going to do this, that first of all, we define our CodePipeline, we have multiple regions, so we will have multiple Artifact Store. The first one is eu-west-1,

this is where our pipeline is located. So, the code is in CodeCommit, it gets sent to CodeBuild, and CodeBuild is going to build our code, and also create the output yaml file that will be used by CloudFormation to deploy the Lambda function. And so, because we are deploying into multiple regions, we need to create multiple templates to make sure we have one template per region we're deploying to. So, once this template is done, then we can have a CloudFormation action in the region of our pipeline, which is going to reference as an input the template we have created from before from CodeBuild, and deploy a Lambda function.

So far so good. This is just a normal pipeline we've seen in one specific region. But because we are deploying cross region, we can also define a CloudFormation action in another region. So, this one is us-east-2. And so, because we have multiple regions now for CodePipeline, we must create an Artifact Store in us-east-2 as well, an S3 bucket. And this Artifact Store is going to get the input artifacts copied automatically by CodePipeline so that, for example, my template-us-east-2.yaml file, is copied, and then what's going to happen is that CloudFormation is going to take that file as an input, and then use that file automatically from the correct Artifact Store. So, from a configuration perspective, we just told CodePipeline to define a CloudFormation action in another region, and that the input artifact was this template-us-east-2.yaml, the location of which is going to be automatically handled and copied behind the scenes by CodePipeline. And so, therefore, a Lambda function in another region will be deployed. And that's it.

Some best practices

When code commit fails, it generates event in the eventbridge and that can trigger or invoke .

We can invoke in these 2 ways:

When we want to run CodePipeline in multiple regions,

codebuild creates multiple artifacts (each artifact for each region), then the destined artifact is copied to the region and that helps to work cloudformation.

That means, we don't need to use codecommit and codebuild again in the second region.

That's it!